What should you focus on for a Spark, PySpark and Databricks interview?

Which topics should you start with?

Start with pipelines and performance. Pipeline architecture appeared in 23 of 74 interviews, and Spark performance appeared in 22 of 74 interviews. These were the leading topics in these interviews.

Can you explain how you would organize medallion layers, schedule pipeline work and recover from a failure? For performance, can you describe how you would find a bottleneck, choose a change and check that the results are still correct?

Also check your Spark basics. Can you explain lazy evaluation, how Spark executes work, and how RDDs differ from DataFrames?

Which repeated questions can you answer clearly?

The Delta Lake benefits question and the question about partition counts, repartition and coalesce each appeared in 6 of 74 interviews. Start by checking whether you can answer both without only listing terms.

  • What do Delta Lake's ACID transactions, schema enforcement and time travel provide?
  • When would you change partition counts, and how does repartition differ from coalesce?
  • How would you handle skew and verify that your change helped?
  • How does a broadcast join work, and when would you use one?
  • How would you diagnose a Spark job that has become much slower?

For the performance questions, include how you would check the results as well as the speed.

What should you be able to write in PySpark?

Coding tasks appeared in 19 of 74 interviews. Use the recorded tasks to check whether you can write a complete read, transform and write flow:

  • Read a CSV file from Azure Data Lake Storage into a DataFrame.
  • Remove rows with missing values in a specified column.
  • Group by another column and calculate an average.
  • Write the results to a Delta table with schema enforcement and partitioning.

Can you also read nested JSON, flatten it and write it to Delta? Explain how you would handle schema inference and performance. For joins, check whether you can write a broadcast join and aggregate total spend.

Can you explain the work around your Spark code?

Use the deployment, streaming and governance questions to check these choices:

  • How would you choose a cluster and troubleshoot runtime problems?
  • How would you handle checkpoints, late data and incremental loads?
  • How would you manage access and secrets, then validate source data against target data?

For pipeline architecture, can you explain a migration or delivery plan that includes team training and human review?

Who gets asked Spark, PySpark and Databricks

Data engineer64 interviews
BI developer4 interviews
DevOps engineer2 interviews
ML engineer1 interviews
Operations1 interviews
Backend developer1 interviews

The 20 most asked Spark, PySpark and Databricks questions

Counted by interview, not by mention: a question asked twice in one conversation counts once.

  1. What is Delta Lake, and what benefits do its ACID transactions, schema enforcement, and time travel provide?

    Asked in 6 of 74 interviews · last asked Oct 1 · Concept · Junior

    Show answer

    Delta Lake is an open-source storage layer that adds reliable tables to data lakes. ACID transactions prevent partial writes and keep concurrent reads and writes consistent. Schema enforcement rejects writes that do not match the table schema unless allowed changes are enabled. Time travel lets you read older table versions for debugging, auditing, and reproducing results.

    • Delta tables use Parquet data files and a transaction log to track committed changes.
    • ACID guarantees apply to table transactions, not automatically to an entire pipeline.
    • Schema evolution allows supported schema changes when explicitly enabled.
    • Time travel requires retained log history and data files; VACUUM can remove files needed by older versions.

    What the interviewer checks: The interviewer is checking whether you understand how Delta Lake makes data lake tables reliable and queryable over time.

    Other ways it was asked
    • “What is Delta Lake, and how does it support ACID transactions, schema enforcement, and time travel?”
    • “Can you tell me what advantages Delta Tables bring?”
    • “What are Delta Live Tables in Databricks?”

  2. When should you change Spark partition counts, and how do repartition and coalesce differ?

    Asked in 6 of 74 interviews · last asked Sep 18 · Concept · Mid-level

    Show answer

    Change partition counts when partitions are too large, tasks are too small, or you need fewer output files. For DataFrames, repartition can increase or decrease partitions and redistributes data through a shuffle. Coalesce reduces partitions without a shuffle, but can leave uneven partitions. Use repartition for more parallelism or redistribution; use coalesce for a cheaper reduction after filtering.

    • Too few partitions limit parallelism; too many add task overhead.
    • DataFrame coalesce cannot increase the partition count.
    • Coalesce combines existing partitions rather than evenly redistributing rows.
    • Avoid coalesce(1) for large data: one task becomes a bottleneck.

    What the interviewer checks: The interviewer checks whether you understand partition sizing, shuffle costs, and the trade-off between parallelism and fewer output files.

    Other ways it was asked
    • “What is the difference between repartition and coalesce?”
    • “What is the difference between repartition and coalesce in Spark?”
    • “What is the difference between repartition and coalesce in Spark, and when would you use each?”

  3. How do you handle Spark data skew using built-in features or other techniques, and verify improvements?

    Asked in 5 of 74 interviews · last asked Sep 10 · Scenario · Senior

    Show answer

    I use the Spark UI to find tasks with much larger shuffle reads and longer runtimes than others. I use AQE skew join handling, broadcast a small join side, or split hot keys with salting. For aggregations, I reduce data early where possible. I rerun the same workload and compare task times, shuffle sizes, spill, and total runtime.

    • In Spark 3+, AQE can split skewed join partitions for supported joins when enabled.
    • Broadcast joins avoid shuffling the large side, but the broadcast side must fit safely in memory.
    • Salting spreads hot keys; joins need matching salt logic, and aggregations need a final combine step.
    • Adding partitions alone does not split rows with the same hash-partitioned key.

    What the interviewer checks: The interviewer checks whether you can identify skew, choose a suitable fix, and verify improvement with measured results.

    Other ways it was asked
    • “Besides salting, what are other ways to handle data skewness in Spark?”
    • “How would you overcome data skew in Spark?”
    • “How would you handle data skew in Spark so that processing tasks finish at roughly the same time?”

  4. How would you identify bottlenecks and optimize a large Spark workload while keeping results correct?

    Asked in 5 of 74 interviews · last asked Oct 9 · Scenario · Senior

    Show answer

    I use the Spark UI and query plan to find slow stages, large shuffles, skew, spills, and expensive scans. I then target the cause with changes such as early filtering, better joins, or partition tuning. I change one thing at a time, measure performance, and compare results on the same input.

    • Read only needed columns and rows to reduce scan and shuffle work.
    • Broadcast joins help when one side safely fits in executor memory; oversized broadcasts can fail.
    • Adaptive Query Execution can reduce shuffle overhead and handle supported skewed joins; check whether it is enabled.
    • Check schema, duplicates, nulls, and output values; matching row counts alone does not prove correctness.

    What the interviewer checks: The interviewer checks whether you diagnose measured bottlenecks, explain why changes help, and verify correctness.

    Other ways it was asked
    • “What Databricks or PySpark optimizations have you carried out?”
    • “How would you identify and improve performance bottlenecks in Spark code, and why would your changes make it run faster?”
    • “How have you optimized Spark jobs in Databricks when dealing with large datasets?”

  5. How does a Spark broadcast join work, and how do you decide whether to use one?

    Asked in 4 of 74 interviews · last asked Aug 14 · Concept · Mid-level

    Show answer

    A broadcast join sends the smaller side to each executor, avoiding a shuffle of the larger side for that join. I use it when the broadcast side fits safely in executor memory and the driver can prepare it. I check its size after filtering, memory headroom, and whether the join type supports broadcasting that side.

    • Spark can choose broadcasting automatically using size estimates and a configurable broadcast threshold.
    • Check size in bytes, not row count; allow extra memory for hash tables and concurrent tasks.
    • Broadcast hash joins need equality keys; supported broadcast sides depend on the join type.
    • A broadcast hint can bypass the size threshold, but oversized broadcasts can cause memory failures or timeouts.

    What the interviewer checks: The interviewer checks whether you understand how broadcasting avoids shuffle and how to judge its memory risk.

    Other ways it was asked
    • “What does a broadcast join do in Spark?”
    • “What is a broadcast join in Apache Spark, and when would you use one?”
    • “What exactly happens when you use a broadcast join?”

  6. What is lazy evaluation in Spark, and why is it useful for big data?

    Asked in 3 of 74 interviews · last asked Sep 17 · Concept · Mid-level

    Show answer

    Lazy evaluation means Spark waits until an action, such as count() or collect(), needs a result before running transformations. It builds an execution plan instead of processing data at each step. This lets Spark optimize the work, which can reduce data reads, data movement, and processing time on large datasets.

    • Transformations such as filter() and select() build a plan; they do not immediately process data.
    • Actions trigger the computation needed to produce their results.
    • For DataFrames and SQL, Spark can push supported filters down and read only needed columns.

    What the interviewer checks: The interviewer is checking whether you understand transformations, actions, and how delayed execution enables optimization.

    Other ways it was asked
    • “What is lazy evaluation in Spark, and how does it work?”
    • “Can you explain the concept of lazy evaluation in Spark?”
    • “What is lazy evaluation in Apache Spark?”

  7. What is Apache Spark, and when would you use it?

    Asked in 3 of 74 interviews · last asked Sep 17 · Concept · Junior

    Show answer

    Apache Spark is a distributed engine that processes data across multiple machines. I would use it for large data transformations, batch jobs, streaming pipelines, or machine learning when one machine is too slow or cannot handle the data. For small tasks, simpler tools often work better because Spark adds overhead.

    • Supports SQL, DataFrames, streaming, and machine learning.
    • Can cache data in memory, but also reads from and writes to disk.
    • Splits work across machines and can recover from many task failures.
    • Best suited to large-scale processing, not small queries that need very fast responses.

    What the interviewer checks: The interviewer wants to know whether you understand distributed processing and when Spark is worth its overhead.

    Other ways it was asked
    • “What is Apache Spark?”
    • “When would you choose Apache Spark for data processing?”

  8. How do you read a CSV file from Azure Data Lake Storage into a PySpark DataFrame?

    Asked in 3 of 74 interviews · last asked Sep 18 · Write code · Mid-level

    Show answer

    Use spark.read.csv with the file path, CSV options, and a schema. For Azure Data Lake Storage Gen2, use an abfss:// path. Configure storage access first, using an approved identity or credentials.

    • Set header=True if the first row contains column names.
    • An explicit schema avoids the extra scan needed to infer column types.
    • Spark reads data across workers; do not collect large datasets to the driver.
    • Keep credentials in secure configuration, not in your script.
    Python
    # Assumes a Spark session named spark and configured storage access.
path = "abfss://container@account.dfs.core.windows.net/input/data.csv"
# Match this schema to the CSV columns and their order.
schema = "id LONG, name STRING, amount DOUBLE"
df = spark.read.csv(
    path, header=True, schema=schema, mode="FAILFAST"
)

    What the interviewer checks: The interviewer checks whether you can use an ADLS path, configure access, and read CSV data with suitable options.

    Other ways it was asked
    • “Write a PySpark script to read a large batch of JSON files from an Azure Data Lake Gen2 container, validate the JSON schema, write to a Delta Lake table partitioned by a column, and include error handling for schema mismatches with a summary of invalid records.”
    • “Write a PySpark script to read a large batch of JSON files from an Azure Data Lake Gen2 container, validate the JSON schema, write to a Delta Lake table partitioned by a column, and handle schema mismatches by summarizing invalid records.”
    • “Write a PySpark script to read a large CSV file from Azure Data Lake Storage.”

  9. How would you diagnose and fix a Spark job that has become much slower?

    Asked in 3 of 74 interviews · last asked Sep 17 · Scenario · Senior

    Show answer

    I would compare the slow run with a normal run, checking data volume, code, cluster resources, and settings. I would use the Spark UI and query plan to find slow stages, skew, large shuffles, spills, or excess file reads. I would fix the measured bottleneck, then rerun with comparable data to check performance and correctness.

    • A few unusually slow tasks can signal skew; check task input sizes and shuffle sizes.
    • Reduce shuffles by filtering early; broadcast a join input only when it safely fits in memory.
    • Tune partition counts and sizes; use adaptive query execution where supported, and handle skewed keys.
    • Compact small files and check partition pruning; add resources only when metrics show a resource limit.

    What the interviewer checks: The interviewer is checking whether you use evidence to find the bottleneck before choosing and testing a fix.

    Other ways it was asked
    • “When you run ETL on Spark or a cloud warehouse, what steps do you take to diagnose and fix performance regressions involving skew, shuffles, partitioning, or file sizes?”
    • “If a Databricks job normally takes twenty minutes but now takes two hours, what would be your key steps to investigate and resolve it?”
    • “How do you optimize a slow-running Spark job?”

  10. What is Unity Catalog, and is it exclusive to Databricks?

    Asked in 3 of 74 interviews · last asked Sep 18 · Concept · Mid-level

    Show answer

    Unity Catalog is a catalog and governance system for data and AI assets. In Databricks, it provides central access control, auditing, and data lineage. It is not exclusive to Databricks: Unity Catalog also has an open-source project, but the managed Databricks service includes additional features and integrations.

    • It governs assets such as tables, files, and machine learning models.
    • In Databricks, it applies consistent permissions across workspaces connected to the same metastore.
    • The open-source project and the managed Databricks service do not have identical capabilities.

    What the interviewer checks: The interviewer checks whether you understand central governance and distinguish the managed Databricks service from open-source Unity Catalog.

    Other ways it was asked
    • “What is the role of Unity Catalog in Databricks?”
    • “What is Unity Catalog?”

  11. How do partitioning and clustering work in Spark, and when do shuffles happen?

    Asked in 2 of 74 interviews · last asked Sep 17 · Concept · Senior

    Show answer

    Partitioning splits data into pieces that Spark tasks process in parallel. Clustering groups rows by key values; its meaning depends on the operation or table feature. A shuffle moves data between partitions when Spark needs a different distribution, often for joins, aggregations, sorting, or repartitioning.

    • Hash partitioning groups equal keys; range partitioning groups ranges of key values.
    • Writing with partitionBy creates table directories; these are different from Spark's execution partitions.
    • Spark SQL CLUSTER BY redistributes rows by key and sorts within each partition, not globally.
    • Broadcast joins avoid shuffling the large side; coalesce usually reduces partition count without a shuffle.

    What the interviewer checks: The interviewer checks whether you distinguish data layout from execution partitions and understand when data must move.

    Other ways it was asked
    • “Explain Spark partitioning, clustering, and when shuffles happen.”
    • “Explain Spark partitioning and clustering in more detail.”
    • “Can you explain partitioning in Spark?”

  12. How do Spark and pandas DataFrames differ, and when would you choose each?

    Asked in 2 of 74 interviews · last asked Sep 10 · Concept · Junior

    Show answer

    A Spark DataFrame is a table split into partitions that Spark can process across many machines. A pandas DataFrame runs on one machine and usually holds its data in memory. Choose pandas for data that fits comfortably in memory and quick local analysis. Choose Spark for larger workloads or distributed processing, accepting extra setup and processing overhead.

    • Spark usually delays computation until an action runs; pandas usually executes operations immediately.
    • Spark can recover lost partitions after worker failures.
    • pandas often has less overhead for small datasets.
    • Spark DataFrames have no pandas-style row index and no guaranteed row order without sorting.

    What the interviewer checks: The interviewer is checking whether you can choose based on data size, memory, execution model, and processing overhead.

    Other ways it was asked
    • “Could you explain what a Spark DataFrame is and how it differs from a pandas DataFrame? In the context of big data, why is Spark often preferred over pandas?”
    • “How do you decide whether to use Spark or pandas for a job?”

  13. Can you explain Spark architecture?

    Asked in 2 of 74 interviews · last asked Sep 10 · Concept · Mid-level

    Show answer

    Spark uses a driver and executors to run work across a cluster. The driver builds execution plans and schedules tasks. A cluster manager allocates resources, while executors run tasks and cache data. Actions trigger jobs, which Spark splits into stages and tasks.

    • Transformations are lazy: Spark records them and waits for an action to execute them.
    • Shuffle boundaries separate stages; each task processes one partition within a stage.
    • Cluster managers include Spark Standalone, YARN, and Kubernetes.
    • Spark can retry failed tasks and recompute lost partitions using lineage.

    What the interviewer checks: The interviewer checks whether you understand Spark’s main components and how distributed work gets planned and executed.

    Other ways it was asked
    • “How does Spark work under the hood, in a nutshell?”

  14. How do you decide whether to use SQL or PySpark for notebook transformations?

    Asked in 2 of 74 interviews · last asked Sep 9 · Concept · Mid-level

    Show answer

    I use SQL for clear, table-based transformations such as joins and aggregations. I use PySpark when I need reusable functions, dynamic logic, or a larger processing workflow. In Spark notebooks, including Microsoft Fabric, I choose based on readability and team skills, not an assumption that one is faster.

    • Spark SQL and PySpark DataFrame operations use the same optimizer; equivalent operations often have similar performance.
    • Prefer built-in Spark functions over Python UDFs, which can add overhead.
    • You can mix both approaches by exposing DataFrames as temporary views.

    What the interviewer checks: The interviewer checks whether you choose tools for maintainability and understand how Spark executes transformations.

    Other ways it was asked
    • “Do you prefer SQL or PySpark, and how do you decide which to use?”
    • “How do you choose between PySpark and SQL when creating notebooks in Microsoft Fabric?”

  15. Can you explain what Databricks is?

    Asked in 2 of 74 interviews · last asked Aug 15 · Concept · Junior

    Show answer

    Databricks is a cloud platform for data engineering, analytics, and machine learning. It provides managed Apache Spark, SQL tools, notebooks, and workflows. It uses a lakehouse approach, combining data lake storage with features such as reliable tables and access controls. It runs on AWS, Azure, and Google Cloud.

    • Apache Spark is a processing engine; Databricks is a broader managed platform.
    • Teams can build data pipelines, run SQL queries, and train models in one platform.
    • Delta Lake adds transactions and schema checks to tables stored in cloud object storage.
    • Unity Catalog manages data permissions, discovery, and lineage.

    What the interviewer checks: The interviewer checks whether you understand Databricks, its main uses, and how it relates to Spark.

    Other ways it was asked
    • “What is it that you know about the Databricks platform?”
    • “How does Databricks differ from other cloud data platforms?”

  16. What is the difference between PySpark and Databricks?

    Asked in 2 of 74 interviews · last asked Sep 18 · Concept · Junior

    Show answer

    PySpark is the Python API for Apache Spark, used to process large datasets across multiple machines. Databricks is a managed platform for data engineering, analytics, and AI that supports Spark and PySpark. PySpark is a tool you write code with; Databricks provides an environment where that code can run.

    • You can use PySpark without Databricks, locally or on a Spark cluster.
    • Databricks adds managed compute, shared notebooks, job scheduling, and data governance.
    • Databricks supports more than PySpark, including SQL and other tools.

    What the interviewer checks: The interviewer is checking whether you can distinguish a programming API from a managed platform.

    Other ways it was asked
    • “What is the main difference between Snowflake and Databricks?”

  17. What are Delta Lake deletion vectors, and when would you use them?

    Asked in 2 of 74 interviews · last asked Aug 20 · Concept · Senior

    Show answer

    Delta Lake deletion vectors are bitmaps that mark rows as deleted without immediately rewriting the Parquet files. Readers skip those rows when reading the table. I would use them for frequent deletes or updates that change relatively few rows, because they can reduce file rewriting.

    • Support for DELETE, UPDATE, and MERGE with deletion vectors depends on the runtime and version.
    • They reduce write work but can add read overhead.
    • All clients accessing the table must support deletion vectors.
    • In Databricks, REORG TABLE APPLY (PURGE) rewrites affected files; VACUUM later removes old files, subject to retention.

    What the interviewer checks: The interviewer checks whether you understand logical deletion, its performance trade-offs, and when physical data is removed.

    Other ways it was asked
    • “What are Delta deletion vectors, and how do they work?”

  18. What does VACUUM remove in Delta Lake, when should you run it, and how does retention work?

    Asked in 2 of 74 interviews · last asked Aug 19 · Tooling · Mid-level

    Show answer

    VACUUM physically deletes old files that a Delta table no longer needs for its current version. Run it periodically to reclaim storage; the default seven-day retention does not schedule cleanup. For replaced or deleted data files, retention starts when they become obsolete. Keep retention longer than your longest running transaction or stream lag.

    • Set retention with RETAIN or delta.deletedFileRetentionDuration; safety checks normally block retention below seven days.
    • Deleted files are unavailable for time travel, so older table versions may stop working.
    • VACUUM does not delete transaction logs; separate log cleanup rules apply.
    • Databricks predictive optimization can run VACUUM automatically for eligible tables.

    What the interviewer checks: The interviewer checks whether you distinguish logical deletion from physical cleanup and understand retention safety.

    Other ways it was asked
    • “When should you run VACUUM on a Delta table, and what does it remove?”
    • “What does the VACUUM command do to a Delta table?”
    • “What is the default retention period for the vacuum command in Delta Lake?”

  19. What is the difference between partitioning and liquid clustering?

    Asked in 2 of 74 interviews · last asked Aug 20 · Concept · Senior

    Show answer

    Partitioning splits data into directories by column values. Liquid clustering groups similar values in files without fixed partition directories, and you can change its clustering keys. Prefer liquid clustering for high-cardinality columns or changing query filters. Partitioning can be better for stable, low-cardinality filters with large partitions, or compatibility with tools that do not support liquid clustering.

    • Partitioning skips whole partitions; liquid clustering helps skip files using file statistics.
    • Too many partition values can create many small partitions and files.
    • A table cannot use both partitioning and liquid clustering.
    • Changing clustering keys does not immediately reorganize existing data; OPTIMIZE performs clustering work.

    What the interviewer checks: The interviewer checks whether you can choose a data layout based on query filters, data size, and compatibility.

    Other ways it was asked
    • “How does liquid clustering differ from traditional partitioning?”
    • “If both partitioning and liquid clustering are available, when would partitioning be the better choice?”

  20. How would you optimize Delta tables, including techniques beyond partitioning, Z-ordering, and liquid clustering?

    Asked in 2 of 74 interviews · last asked Aug 20 · Concept · Senior

    Show answer

    I would compact small files with OPTIMIZE and tune file sizes for the workload. I would use optimized writes, auto compaction, and data-skipping statistics where supported. I would also reduce unnecessary scans and rewrites, then use query plans and runtime metrics to check the gains.

    • Use partition filters in MERGE when valid to avoid scanning unrelated data.
    • On supported Databricks configurations, predictive optimization automates maintenance for eligible Unity Catalog managed tables.
    • VACUUM removes old files to save storage; it limits time travel and usually does not speed up queries.
    • Liquid clustering cannot be combined with partitioning or Z-ordering on the same table.

    What the interviewer checks: The interviewer is checking whether you can improve reads, writes, and maintenance beyond choosing a table layout.

    Other ways it was asked
    • “What are the optimization techniques for Delta Lake?”
    • “What other techniques can optimize Delta Lake tables?”
    • “How would you optimize the Delta tables in your architecture, and what techniques could you use?”

Apache Spark fundamentals interview questions

Check your understanding of Spark architecture, execution, lazy evaluation, RDDs, DataFrames, resilience and how Spark compares with other processing tools. 26 questions.

  1. Is PySpark suitable for millisecond-latency work?

    Asked once · last asked May 25 · Concept · Senior

    Other ways it was asked
    • “Why is PySpark generally unsuitable for millisecond-latency workloads?”

  2. How does Apache Spark differ from AWS, and what problem does Spark solve?

    Asked once · last asked Sep 17 · Concept · Mid-level

    Other ways it was asked
    • “What problem does Apache Spark typically solve?”

  3. Can a large JSON file be read in parallel by multiple Spark executors?

    Asked once · last asked Apr 17 · Concept · Mid-level

  4. What would you process with Flink versus Spark? Can you differentiate the two and explain how you would use each?

    Asked once · last asked May 12 · Concept · Mid-level

  5. What is the difference between an RDD and a DataFrame?

    Asked once · last asked May 25 · Concept · Mid-level

  6. In PySpark, what are SparkContext and SparkSession?

    Asked once · last asked May 27 · Concept · Junior

  7. What is LakeSail, and how does its Rust-based engine execute Spark workloads?

    Asked once · last asked Jun 15 · Concept · Senior

    Other ways it was asked
    • “What is LakeSail, and how does its Rust framework execute Spark?”

  8. What is the difference between Python and PySpark?

    Asked once · last asked Aug 31 · Concept · Junior

  9. In a cloud environment, what Spark features make it preferable to one large machine for datasets with huge memory requirements?

    Asked once · last asked Sep 10 · Concept · Senior

  10. What are the trade-offs between using Spark and running queries directly in a database engine such as Snowflake, BigQuery, or Redshift?

    Asked once · last asked Sep 10 · Scenario · Senior

  11. What is the difference between batch and stream processing, and how do you read data with Spark?

    Asked once · last asked Sep 17 · Concept · Mid-level

  12. What did Apache Spark do differently from MapReduce, and what was its main contribution?

    Asked once · last asked Sep 17 · Concept · Mid-level

Show 9 more

  1. Can you explain RDDs and their types?

    Asked once · last asked Sep 18 · Concept · Mid-level

  2. Can you explain some common Apache Spark transformations?

    Asked once · last asked Jul 16 · Concept · Junior

  3. What happens when a Spark job runs?

    Asked once · last asked Aug 12 · Concept · Mid-level

  4. Can you explain what a shuffle is in Spark?

    Asked once · last asked Aug 14 · Concept · Junior

    Other ways it was asked
    • “Can you explain what a shuffle is?”

  5. How many Spark jobs will execute when reading a CSV file?

    Asked once · last asked Aug 20 · Concept · Mid-level

  6. What does resilience mean in PySpark?

    Asked once · last asked Aug 31 · Concept · Mid-level

  7. What does it mean for data to already be present on a Spark cluster?

    Asked once · last asked Sep 10 · Concept · Mid-level

  8. How does Apache Spark use in-memory computing?

    Asked once · last asked Sep 17 · Concept · Mid-level

  9. Can Apache Spark code be written in Java?

    Asked once · last asked Oct 20 · Concept · Junior

PySpark coding interview questions

Work through PySpark questions on reading and writing files, cleaning data, flattening JSON, joining DataFrames, aggregating values and using window functions. 31 questions.

  1. Can you write a PySpark script that reads a large nested JSON file, flattens the data, and writes the output to a Delta table in Azure Databricks, handling schema inference and optimized performance?

    Asked once · last asked Aug 11 · Write code · Mid-level

    Other ways it was asked
    • “Can you write a PySpark script that reads a large nested JSON file, flattens the data, and writes it to a Delta table in Azure Databricks with schema inference and improved performance?”

  2. Write a PySpark script to read CSV from ADLS, remove records with missing values in a specified column, group by another column to calculate an average, and write the results to a Delta Lake table in ADLS with schema enforcement and partitioning.

    Asked once · last asked Aug 11 · Write code · Mid-level

    Other ways it was asked
    • “Can you write a PySpark script that reads a CSV file from Azure Data Lake Storage, filters out records where a specific column has no values, groups the data by a given column to calculate the average of another column, and writes the transformed data back to a Delta Lake table in ADLS with schema enforcement and partitioning?”
    • “Can you write a PySpark script that reads a CSV file from Azure Data Lake Storage, filters out empty values, groups the data by a column to calculate an average, and writes the results to Delta Lake with schema enforcement and partitioning?”

  3. How would you join 500 million orders with 200 stores using a PySpark broadcast join?

    Asked once · last asked Sep 29 · Write code · Senior

    Other ways it was asked
    • “How would you use an explicit broadcast join in PySpark to join a 500-million-row orders DataFrame with a 200-row stores DataFrame?”
    • “How would you join a 500-million row orders dataset with a 200-row store dataset?”

  4. How would you use a PySpark broadcast join and aggregate total spend?

    Asked once · last asked Sep 29 · Write code · Mid-level

  5. Given a Spark DataFrame with columns for user ID, country, and amount, how would you filter rows where country equals US, group by user ID, and compute the total amount per user? Could you describe the code steps conceptually?

    Asked once · last asked May 18 · Write code · Junior

  6. Write PySpark code to fetch a JSON file from S3, flatten its data, and insert it into a table.

    Asked once · last asked May 27 · Write code · Mid-level

  7. How would you structure a PySpark AWS Glue job, starting with its first five lines, to read a raw JSON file from S3 containing a nested array field, deduplicate records on a composite key, and write the result?

    Asked once · last asked Jun 19 · Write code · Mid-level

    Other ways it was asked
    • “Walk me through the PySpark code structure for an AWS Glue job that reads a raw JSON file from S3 with a nested array field, deduplicates records on a composite key, and writes the result. What are the first five lines of the job?”

  8. What does the otherwise() method do in a PySpark conditional expression?

    Asked once · last asked Jul 16 · Concept · Junior

  9. Write a PySpark script to read a CSV from ADLS, calculate average sales per region, and save the result as Parquet, optimized for performance.

    Asked once · last asked Aug 10 · Write code · Mid-level

  10. Write a performance-optimized PySpark script to read large customer and transaction datasets in Parquet format from Azure Data Lake, inner join them on customer_id, calculate total transaction amounts per customer, and save the result as CSV in Azure Data Lake.

    Asked once · last asked Aug 10 · Write code · Senior

    Other ways it was asked
    • “Write a PySpark script to join two large datasets stored in Parquet format in Azure Data Lake. The first data set contains customer details and the second contains transaction details. Perform an inner join on the customer_id column and then calculate the total transaction amount for each customer. Save the result back to Azure Data Lake in CSV format, ensuring optimal performance.”

  11. How would you use PySpark to process millions of Azure Data Lake log entries with timestamp and event_type columns into daily event counts, pre-aggregate in memory despite memory limits, handle event_type skew during grouping and partitioning, and write date-partitioned Parquet back to Azure Data Lake?

    Asked once · last asked Aug 10 · Scenario · Senior

    Other ways it was asked
    • “Imagine you are tasked with creating a PySpark based solution to process a dataset containing millions of log entries stored in Azure Data Lake. The data set includes a timestamp column and an event_type column. Your goal is to generate a daily summary of event counts grouped by event_type and save the results as a partitioned parquet file, partitioned by date, back to Azure Data Lake. How would you modify your PySpark script to ensure that the event_type column is pre-aggregated in memory before writing to the parquet file, considering scenarios where the dataset could exceed the available memory? Additionally, how would you handle skewed data in the event_type column to avoid bottlenecks during grouping and partitioning?”

  12. Write a PySpark script to read a dataset containing event_type and timestamp columns from Azure Data Lake, filter for specific event_type values such as click and purchase, count occurrences of each event_type, and save the results as CSV in Azure Data Lake.

    Asked once · last asked Aug 10 · Write code · Mid-level

    Other ways it was asked
    • “Let's simplify this. Write a PySpark script to process a dataset stored in Azure Data Lake. The dataset includes event_type and timestamp columns. Your task is to filter only the rows with specific event_type values, e.g. click, purchase, and then count the occurrences of each event_type. Save the results as a CSV file back to Azure Data Lake.”
Show 18 more

  1. How would you use PySpark to find the names of employees who receive the second highest salary in each department?

    Asked once · last asked Aug 15 · Write code · Mid-level

  2. How would you add a `salary_class` column to a Spark DataFrame based on whether `salary` is greater than 50,000?

    Asked once · last asked Aug 19 · Write code · Junior

  3. How would you split a full name column into first name and last name using PySpark?

    Asked once · last asked Aug 20 · Write code · Junior

  4. Write a PySpark script to load a single JSON file from an Azure Data Lake storage, extract a specific field such as customer_id, and save the results as a partitioned Delta table by customer_id, handling missing or null customer_id values.

    Asked once · last asked Aug 22 · Write code · Mid-level

  5. Given a DataFrame with pizza toppings and their costs, how can you use PySpark to calculate the total cost of a pizza made with pepperoni, sausage, and chicken?

    Asked once · last asked Aug 25 · Write code · Junior

  6. Can you write PySpark code to keep only the latest record per employee ID?

    Asked once · last asked Sep 1 · Write code · Mid-level

    Other ways it was asked
    • “Can you write a PySpark code for a deduplication task where you keep only the latest record per employee ID?”

  7. Write a PySpark code snippet that joins two data frames and uses a window function with LAG.

    Asked once · last asked Sep 2 · Write code · Mid-level

  8. How would you flatten deeply nested JSON using Spark?

    Asked once · last asked Sep 7 · Tooling · Mid-level

  9. How would you remove duplicates from raw data? Write PySpark and Snowflake code to do it.

    Asked once · last asked Sep 21 · Write code · Mid-level

  10. Can you write an AWS Glue PySpark job to clean data coming from AWS Kinesis?

    Asked once · last asked Sep 21 · Write code · Mid-level

  11. How would you efficiently calculate the total successful transaction amount per customer in PySpark for 100 million transactions?

    Asked once · last asked Sep 23 · Write code · Senior

  12. How would you use PySpark with df_orders and df_customers to calculate each customer's total spend within their region and rank the results?

    Asked once · last asked Sep 29 · Write code · Mid-level

  13. How would you use PySpark to calculate total spend per customer and rank customers using dense rank after broadcasting the stores table?

    Asked once · last asked Sep 29 · Write code · Senior

  14. What is the difference between saveAsTable and save in PySpark, and when would you use each?

    Asked once · last asked Aug 15 · Concept · Mid-level

    Other ways it was asked
    • “What is the basic difference between the saveAsTable and save commands in PySpark?”
    • “What is the basic difference between saveAsTable and .save, and in which scenarios do we use each?”

  15. How do you implement data transformations in Databricks?

    Asked once · last asked Aug 19 · Tooling · Mid-level

  16. Write SQL or PySpark logic to swap values in a gender column.

    Asked once · last asked Aug 25 · Write code · Junior

  17. How do you add a new column to a PySpark DataFrame?

    Asked once · last asked Sep 18 · Tooling · Junior

  18. How do you rename a column in a PySpark DataFrame?

    Asked once · last asked Sep 18 · Tooling · Junior

Databricks and Spark deployment interview questions

Explore notebooks, reusable code, cluster choices, cloud storage, Spark deployment on Kubernetes and runtime troubleshooting. 18 questions.

  1. Why do you need Azure when you are already using Databricks to process and store data?

    Asked once · last asked Aug 26 · Concept · Mid-level

    Other ways it was asked
    • “Why do you need Azure storage if you can just store data in Databricks tables?”
    • “When you are already using Azure for the data lake and the middle layer architecture, why do you specifically need Databricks?”
    • “Why do you need Databricks if you are already using Azure Data Lake Storage for your middle layer?”

  2. How do you make a PySpark function available outside a particular notebook session?

    Asked once · last asked Sep 15 · Tooling · Mid-level

    Other ways it was asked
    • “How do you make a PySpark function available for use in other notebooks or sessions?”
    • “How do you make a reusable PySpark function available across Microsoft Fabric notebooks?”

  3. How do you tune Spark jobs running on Kubernetes or OpenShift?

    Asked once · last asked Jul 16 · Tooling · Senior

  4. If Spark executors or pods are not being scheduled, what would you check?

    Asked once · last asked Jul 16 · Scenario · Senior

  5. What storage considerations matter for Spark temporary files, and what roles do PostgreSQL, the Hive metastore, and Keycloak play?

    Asked once · last asked Jul 16 · Concept · Senior

  6. Spark writes succeed, but an Iceberg metadata commit fails. What object storage issue could be causing the failure?

    Asked once · last asked Jul 16 · Scenario · Senior

  7. What types of Databricks clusters are available, and which would you recommend for different environments?

    Asked once · last asked Aug 14 · Scenario · Mid-level

  8. How would you approach troubleshooting a failed Databricks workflow job?

    Asked once · last asked Aug 14 · Scenario · Mid-level

  9. Can you write data directly into Databricks tables without needing an external Azure storage layer like ADLS?

    Asked once · last asked Aug 26 · Concept · Mid-level

  10. In Databricks, why do we need a storage location for a table when an RDBMS like Oracle handles storage internally?

    Asked once · last asked Aug 26 · Concept · Mid-level

  11. How would you choose between Dataproc and other options for running PySpark jobs?

    Asked once · last asked Sep 10 · Concept · Mid-level

  12. Is Dataproc cluster-based or serverless, and why?

    Asked once · last asked Sep 16 · Concept · Mid-level

Show 3 more

  1. How would you load data from Azure Blob Storage into Databricks?

    Asked once · last asked Oct 7 · Tooling · Mid-level

  2. Why would you store data in Azure Data Lake Storage instead of keeping it inside Databricks?

    Asked once · last asked Aug 26 · Concept · Mid-level

    Other ways it was asked
    • “Where are you creating the data lake—in Azure or in Databricks?”
    • “Where are you creating the Data Lake, in Azure or in Databricks?”
    • “Why are you creating the data in Azure Data Lake Storage instead of just keeping it inside Databricks?”

  3. What is the difference between Azure Blob Storage and PySpark?

    Asked once · last asked Sep 14 · Concept · Junior

Delta Lake and lakehouse interview questions

Delta Lake questions cover table creation, transaction logs, schema changes, Parquet and Iceberg comparisons, partitioning, clustering, OPTIMIZE and VACUUM. 22 questions.

  1. How does Delta Lake differ from a data lake such as Azure Data Lake Storage?

    Asked in 2 of 74 interviews · last asked Oct 1 · Concept · Junior

    Other ways it was asked
    • “What is the difference between a Data Lake and a Delta Lake?”
    • “How is a Delta table different from data stored in Azure Data Lake Storage (ADLS)?”

  2. How does Delta Lake differ from plain Parquet, and what capabilities does it add?

    Asked in 2 of 74 interviews · last asked Aug 19 · Concept · Mid-level

    Other ways it was asked
    • “What is the difference between Parquet and Delta file formats?”

  3. How do we create a table in Databricks?

    Asked in 2 of 74 interviews · last asked Aug 26 · Tooling · Junior

    Other ways it was asked
    • “Which command do we use to create a managed table in Databricks?”
    • “How is data physically stored when you create a table in Databricks?”

  4. Can Z-ordering and liquid clustering be used together, and does liquid clustering automatically provide the benefits of Z-ordering?

    Asked once · last asked Apr 17 · Concept · Senior

    Other ways it was asked
    • “How do Z-ordering and liquid clustering optimize data, and what is the difference between them?”
    • “When would you use Z-ordering and liquid clustering together in Delta Lake?”

  5. When writing data to Iceberg with Spark, what is the difference between overwrite and dynamic partition overwrite?

    Asked once · last asked Aug 6 · Concept · Mid-level

    Other ways it was asked
    • “When writing to an Iceberg table with Spark using dynamic partition overwrite, what happens if the input data contains multiple partitions?”

  6. How are Delta logs stored?

    Asked once · last asked Aug 14 · Concept · Senior

  7. If you need to delete historical data, would you prefer partitioning or liquid clustering, and why?

    Asked once · last asked Aug 14 · Scenario · Senior

  8. What is Z-ordering in Delta Lake, and how does it improve query performance?

    Asked once · last asked Aug 19 · Concept · Senior

  9. Can you explain how lakehouse architecture and Apache Iceberg work together in Databricks?

    Asked once · last asked Aug 20 · Concept · Senior

  10. What happens if you have many small files and run the optimize command?

    Asked once · last asked Aug 20 · Concept · Senior

    Other ways it was asked
    • “Why do we need to run optimize when there are many small files?”

  11. When processing data in Databricks, what considerations would you account for with Delta Lake, partitioning, and performance optimization?

    Asked once · last asked Aug 22 · Design · Senior

  12. What is the difference between mergeSchema and overwriteSchema?

    Asked once · last asked Sep 1 · Concept · Mid-level

Show 5 more

  1. How do you rename or drop a column in a Delta Lake table?

    Asked once · last asked Sep 17 · Tooling · Mid-level

  2. What is the significance of “Delta” in Delta Lake?

    Asked once · last asked Aug 12 · Concept · Junior

  3. Can you specify a file path when creating a managed table in Databricks?

    Asked once · last asked Aug 15 · Concept · Mid-level

  4. What does the vacuum command do?

    Asked once · last asked Aug 19 · Concept · Junior

  5. Does Delta Lake support all table types?

    Asked once · last asked Sep 17 · Concept · Mid-level

Spark performance optimization interview questions

Test how you would handle join choices, shuffles, skew, caching, AQE, Photon, memory failures, slower jobs and costs. 21 questions.

  1. How do you cache or persist data in Spark, and when should you unpersist it?

    Asked in 2 of 74 interviews · last asked Aug 19 · Tooling · Mid-level

    Other ways it was asked
    • “How do you persist a data structure on a Spark worker and unpersist it after the operation?”
    • “How do you cache a DataFrame in Databricks, and how do you manage its lifecycle?”
    • “When should you unpersist cached data in Spark after a job completes?”

  2. What challenges or bottlenecks have you faced while building data pipelines, and what strategies did you use to resolve them and optimize your PySpark jobs in Databricks?

    Asked once · last asked Apr 14 · Behavioral · Senior

  3. What happens when you use a Spark UDF instead of Spark's native functions?

    Asked once · last asked Apr 15 · Concept · Senior

  4. How do you use Adaptive Query Execution (AQE) in Apache Spark, and what is the most complex Spark optimization you have implemented?

    Asked once · last asked Jul 16 · Design · Senior

  5. How would you optimize the migration of a large dataset using PySpark?

    Asked once · last asked Jul 16 · Scenario · Senior

  6. How would you efficiently join three large Parquet datasets stored in Azure Data Lake using Azure Databricks, minimizing data shuffle and optimizing resource utilization while handling skew in one dataset to prevent bottlenecks?

    Asked once · last asked Aug 10 · Scenario · Senior

    Other ways it was asked
    • “In a scenario where you are working with Azure Databricks and need to join three large datasets stored in Parquet format in Azure Data Lake, how would you efficiently join them, ensuring minimal data shuffle and optimal resource utilization? Additionally, how would you handle skewed data in one of the datasets to prevent performance bottlenecks?”

  7. How would you diagnose and resolve a sudden significant degradation in query performance for a Delta table containing 2 billion records?

    Asked once · last asked Aug 11 · Scenario · Senior

  8. In what cases does an out-of-memory error happen on the driver node versus an executor node?

    Asked once · last asked Aug 14 · Concept · Senior

  9. Can excessive shuffling lead to executor out-of-memory errors?

    Asked once · last asked Aug 14 · Concept · Mid-level

  10. What is the difference between static partition pruning and dynamic partition pruning?

    Asked once · last asked Aug 14 · Concept · Senior

  11. How do broadcast joins and Z-ordering work, and when would you use them?

    Asked once · last asked Sep 21 · Concept · Senior

  12. How would you optimize the cost of a PySpark job?

    Asked once · last asked Sep 21 · Scenario · Mid-level

Show 4 more

  1. If the stores table grew to around 100,000 rows, would a broadcast join still be applicable, and how would you handle it?

    Asked once · last asked Sep 29 · Scenario · Senior

  2. How does Photon relate to table optimization, and how does it differ from regular Spark execution?

    Asked once · last asked Apr 17 · Concept · Senior

    Other ways it was asked
    • “Photon is an execution engine, right, and is it unrelated to table optimization?”
    • “Do you think Photon optimizes a table differently from regular Spark?”

  3. How would you resolve a shuffle issue?

    Asked once · last asked Sep 1 · Scenario · Senior

  4. What factors guide your choice of join strategy in PySpark?

    Asked once · last asked Sep 29 · Concept · Mid-level

Spark Structured Streaming and incremental loading interview questions

Streaming and incremental loading questions cover Auto Loader, Kafka, Event Hubs, state, checkpoints, late data, exactly-once processing, CDC and Change Data Feed. 13 questions.

  1. How would you design a real-time stream processing solution for IoT device data using PySpark and Azure Databricks, including aggregations, fault tolerance, and schema evolution handling?

    Asked in 2 of 74 interviews · last asked Aug 11 · Design · Senior

    Other ways it was asked
    • “You are tasked with designing a real time data processing pipeline using Azure Databricks and PySpark. The data comes from an IoT device stream with thousands of records per second, containing information such as device_id, temperature, and timestamp. The requirements are to identify devices with abnormal temperature readings, e.g. above a threshold, and store these anomalies in a separate Azure SQL database table for further analysis. How would you design this pipeline to ensure low latency, scalability, and fault tolerance? Please include any PySpark transformations or optimizations you would use.”

  2. What does Delta Lake Change Data Feed provide, and when would you use it for incremental processing?

    Asked once · last asked Apr 17 · Concept · Mid-level

    Other ways it was asked
    • “What is Change Data Feed (CDF) in Databricks, and when would you use it?”
    • “How can you use Change Data Feed (CDF) to process data incrementally in a data architecture?”
    • “When would you use Change Data Feed (CDF) rather than filtering a table to identify changes?”

  3. What are the risks or limitations of using Change Data Feed (CDF) in production?

    Asked once · last asked Apr 17 · Scenario · Senior

  4. How would you read data from Azure Event Hubs using Spark Structured Streaming and Kafka?

    Asked once · last asked Aug 7 · Tooling · Mid-level

  5. Is the Spark Kafka connector available in Databricks, or only in open source Spark?

    Asked once · last asked Aug 7 · Concept · Mid-level

  6. How would you ensure exactly-once processing semantics in a streaming pipeline, particularly with checkpointing, late-arriving data, and output to Delta Lake?

    Asked once · last asked Aug 11 · Scenario · Senior

  7. How do you handle incremental data loading and change data capture, CDC, in Databricks?

    Asked once · last asked Aug 12 · Scenario · Mid-level

  8. When you work with Auto Loader, how does it detect new files?

    Asked once · last asked Aug 20 · Tooling · Mid-level

    Other ways it was asked
    • “How does Auto Loader detect new files and handle schema changes or breakages?”

  9. How would you write Databricks code to read a daily file and load only records not already in a target table?

    Asked once · last asked Aug 20 · Write code · Mid-level

  10. Why does Spark use micro-batch processing?

    Asked once · last asked Sep 10 · Concept · Mid-level

  11. Suppose you’re designing an end-to-end pipeline from the source through cloud storage, Spark, a database, and a dashboard, and the source sends late-arriving updates. How would you design incremental logic that remains correct throughout data loading and the pipeline?

    Asked once · last asked Dec 15 · Design · Senior

  12. What is Databricks Auto Loader?

    Asked once · last asked Aug 11 · Concept · Junior

    Other ways it was asked
    • “What is Auto Loader?”
Show 1 more

  1. How does state management work in Apache Spark?

    Asked once · last asked Sep 18 · Concept · Mid-level

Databricks Unity Catalog, security and data quality interview questions

Review Unity Catalog, access control, secrets, encryption, masking, auditing, data quality expectations and source-to-target validation. 9 questions.

  1. How does Unity Catalog affect how teams build and share data products?

    Asked once · last asked Jul 16 · Tooling · Senior

  2. How would you handle a case where the on-premise SQL Server contains sensitive data that must be encrypted during transfer and remain secure while being processed in Azure Databricks, including key management and access control?

    Asked once · last asked Aug 11 · Scenario · Senior

  3. How would you implement access control so only specific Azure Databricks jobs or users can access secrets securely, and how would you monitor and audit access to those secrets for compliance?

    Asked once · last asked Aug 11 · Scenario · Senior

    Other ways it was asked
    • “How would you ensure secure handoffs of sensitive data between teams working in the same Azure Databricks environment, and configure logging and monitoring to detect unauthorized access or changes to secrets?”

  4. How would you implement data masking to protect sensitive information during processing, and how would it work in a PySpark job running in Azure Databricks?

    Asked once · last asked Aug 11 · Scenario · Mid-level

  5. How is Unity Catalog different from a traditional Hive Metastore?

    Asked once · last asked Aug 14 · Concept · Mid-level

  6. How would you implement a scalable automated check in SQL, Spark, or Pandas to verify that every feature’s event time equals the label cutoff time for its entity and report violations?

    Asked once · last asked Aug 19 · Write code · Senior

  7. How can you use checksums or similar methods in Databricks to verify that source and target data are identical?

    Asked once · last asked Aug 20 · Tooling · Mid-level

  8. What are Delta Live Tables expectations, and how do you use them to enforce data quality?

    Asked once · last asked Sep 4 · Tooling · Mid-level

    Other ways it was asked
    • “How do you implement data quality rules inside Databricks using Delta Live Tables?”
    • “Could you please define expectations in Delta Live Tables?”

Databricks data pipeline and architecture interview questions

Think through production pipeline design, medallion layers, orchestration, migrations, platform choices, failure recovery and delivery plans with team training and human review. 28 questions.

  1. How would you design a data ingest and transformation pipeline for a retail company using Azure Data Factory and Azure Databricks to process data from multiple sources and load the results into a Delta Lake, maximizing performance and handling failures effectively?

    Asked in 2 of 74 interviews · last asked Aug 11 · Design · Senior

    Other ways it was asked
    • “You're tasked with designing an ETL pipeline using Azure Databricks to process incremental data loads from Azure Data Lake. The data contains multiple large Parquet files, and you need to perform transformations such as filtering, aggregations, and data enrichment before writing the output back to Azure SQL database. How would you ensure efficient data processing, optimize costs, and handle potential failures in the pipeline?”
    • “How would you design an Azure Data Factory pipeline for data ingestion from on-premise SQL Server, processing in Azure Databricks, and loading into Azure Synapse Analytics to ensure security, scalability, and performance?”

  2. How would you approach ingesting and processing a batch of JSON files from Azure Data Lake in Databricks, including Delta Lake, partitioning, and performance optimization?

    Asked once · last asked Aug 22 · Design · Senior

    Other ways it was asked
    • “How would you use Databricks to ingest and process a batch of JSON files stored in an Azure Data Lake, ensuring data quality and proper partitioning for efficient querying?”

  3. How would you trigger Spark on Kubernetes to process PostgreSQL staging tables populated by Flink without relying on scheduled orchestration?

    Asked once · last asked May 18 · Design · Senior

  4. How would you plan and deliver Spark training for a data engineering team, and estimate how long it would take to get the team up to speed?

    Asked once · last asked May 22 · Behavioral · Senior

    Other ways it was asked
    • “What role did you play in planning and delivering Spark training for a data engineering team, and how long did it take to get the team up to speed?”

  5. How would you design a Spark ingestion and curation framework in PySpark, and how would you define its target?

    Asked once · last asked Jun 15 · Design · Senior

  6. What target-state architecture would you propose for modernizing an on-premises Hadoop platform to Databricks on Azure?

    Asked once · last asked Jul 16 · Design · Senior

  7. How would you roll out a Spark-based data platform using Microsoft Fabric, Azure HDInsight, or Azure Synapse?

    Asked once · last asked Jul 28 · Scenario · Senior

  8. How would you use dbt to implement silver-to-gold transformations in a Databricks-orchestrated data platform?

    Asked once · last asked Jul 28 · Scenario · Senior

  9. Where do you include human review or intervention in PySpark code execution?

    Asked once · last asked Jul 29 · Scenario · Senior

  10. How do Databricks and Snowflake compare, and when would you choose each for analytics and BI?

    Asked once · last asked Jul 29 · Design · Senior

    Other ways it was asked
    • “How do you compare Databricks and Snowflake as data platforms, and where does Delta Lake fit?”
    • “Why might an organization move from a Databricks lakehouse to Snowflake for BI, and when might each platform fit different use cases?”

  11. How can Databricks and Airflow be integrated with Azure data services?

    Asked once · last asked Aug 7 · Design · Mid-level

  12. How would you design a solution to efficiently process and transform a large dataset with structured and semi-structured data using Azure Databricks and PySpark while ensuring scalability and cost optimization?

    Asked once · last asked Aug 11 · Design · Senior

Show 16 more

  1. How do you build ETL or ELT pipelines in Databricks, and which orchestration tools have you used?

    Asked once · last asked Aug 12 · Tooling · Mid-level

  2. Why might you use Azure Data Factory instead of implementing data processing in Databricks or data lake jobs?

    Asked once · last asked Aug 14 · Scenario · Mid-level

  3. For a star schema, would you store data in Delta tables or an external database such as PostgreSQL, and why?

    Asked once · last asked Aug 14 · Scenario · Mid-level

  4. How do Hadoop and Spark SQL fit into an ETL workflow?

    Asked once · last asked Aug 20 · Concept · Mid-level

  5. Why use Databricks for transformations when Azure has native tools to process data in the lake?

    Asked once · last asked Aug 26 · Concept · Mid-level

  6. Why use Databricks to create and manage tables instead of using native Azure services like Azure SQL?

    Asked once · last asked Aug 26 · Concept · Mid-level

  7. Why would you choose Databricks over Synapse for creating tables and managing the middle layer of your architecture?

    Asked once · last asked Aug 26 · Concept · Mid-level

  8. When would you use Azure Data Factory or Microsoft Fabric rather than Databricks or Spark for a large historical migration from Snowflake into Fabric Delta tables?

    Asked once · last asked Aug 31 · Tooling · Senior

  9. How would you migrate a data transformation from PySpark on Dataproc to BigQuery?

    Asked once · last asked Sep 10 · Scenario · Mid-level

  10. How would you choose file formats for a data platform using Kafka, object storage, and Spark?

    Asked once · last asked Sep 10 · Design · Senior

  11. How would you decide between OneLake and Databricks Lakehouse?

    Asked once · last asked Sep 16 · Scenario · Senior

  12. What preprocessing operations can be performed on data in the bronze layer of the Databricks medallion architecture?

    Asked once · last asked Sep 18 · Concept · Mid-level

  13. How would you build a production PySpark pipeline, from source data to final output, including transformations, joins, aggregations, and the PySpark APIs you would use?

    Asked once · last asked Sep 23 · Tooling · Senior

    Other ways it was asked
    • “Can you walk me through one production PySpark pipeline you personally developed, starting from the source data and explaining the transformations, joins, aggregations, final output, and PySpark APIs used?”

  14. If you had limited hands-on Databricks experience but were asked to deliver a supplier-risk analytical model on Databricks, what would you include in your first release, and what risks would you validate before promising a production date?

    Asked once · last asked Sep 29 · Scenario · Senior

  15. Which integration runtime would you use to copy data from Blob Storage to Databricks?

    Asked once · last asked Oct 7 · Scenario · Mid-level

  16. What are Databricks and Azure Data Factory, and what are their primary roles?

    Asked once · last asked Apr 14 · Concept · Junior

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Spark, PySpark and Databricks interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

74 interviews, 33 candidates, Dec 2025 to Oct 2026. Updated every month with new interviews.