All 61 questions
What is the difference between batch processing and stream processing?
Asked in 4 of 169 interviews · last asked Oct 19 · Concept · Mid-level
Show answer
Batch processing handles a fixed set of data in one job. Stream processing handles incoming data continuously, usually with lower delay. Use batch processing for daily reports or historical analysis, and stream processing for live alerts or fraud detection.
- Batch jobs process bounded data; streaming jobs usually process data with no defined end.
- Streaming may need to handle late events and often keeps state across events.
- Streaming can use small batches internally; it does not always process one event at a time.
- In Spark, use spark.read for batch input and spark.readStream for streaming input.
What the interviewer checks: The interviewer checks whether you understand data boundaries, processing delay, and when to use each approach.
Other ways it was asked
- “What are stream processing and batch processing, and when would you use each?”
- “What is the difference between batch processing and streaming?”
- “What is the difference between batch and stream processing, and how do you read data with Spark?”
How do you handle late or out-of-order events without applying stale updates or losing valid changes?
Asked in 4 of 169 interviews · last asked Oct 22 · Scenario · Senior
Show answer
I track event IDs and source versions for each payment or transfer. I apply updates and record processed events atomically, checking versions and allowed state transitions. I reject stale status updates, but preserve valid changes such as balance adjustments. If ordering is unclear, I buffer events or check the source system.
- Use event IDs for deduplication and source versions for ordering; arrival time does not show event order.
- Older full-state snapshots can be skipped; incremental changes may need buffering and replay in source order.
- After settlement, a stale “in progress” event must not move the transfer backward.
- Queue ordering by payment ID preserves arrival order, not necessarily the provider’s original event order.
What the interviewer checks: The interviewer checks whether you prevent stale state without confusing duplicate events with valid late changes.
Other ways it was asked
- “If a webhook provider can deliver events out of order, how would you design your processing so the final state is still correct and you don't drop legitimate updates?”
- “How would you handle a late-arriving event in a payment-processing workflow?”
- “An MQ consumer receives transfer events in this order: authorized, settled, then in progress. What should the final state be, and how would you handle the late event?”
How would you design a real-time stream processing solution for IoT device data using PySpark and Azure Databricks, including aggregations, fault tolerance, and schema evolution handling?
Asked in 2 of 169 interviews · last asked Aug 11 · Design · Senior
Show answer
I would use PySpark Structured Streaming on Azure Databricks to read from Kafka or Azure Event Hubs. I would validate records, filter temperature anomalies, and calculate per-device averages over event-time windows. Delta tables would store readings and aggregates, while retry-safe writes send anomalies to Azure SQL. Durable checkpoints would support recovery.
- Use watermarks to limit aggregation state; events beyond the lateness limit may be dropped.
- Partition input for parallel reads and tune micro-batch intervals to balance latency and processing capacity.
- Keep raw events, explicitly approve additive Delta schema changes, and quarantine records with incompatible types.
- Store checkpoints in ADLS; use stable event IDs and SQL upserts because foreachBatch retries can duplicate writes.
What the interviewer checks: The interviewer checks whether you can combine streaming transformations with low latency, scaling, safe recovery, and controlled schema changes.
Other ways it was asked
- “You are tasked with designing a real time data processing pipeline using Azure Databricks and PySpark. The data comes from an IoT device stream with thousands of records per second, containing information such as device_id, temperature, and timestamp. The requirements are to identify devices with abnormal temperature readings, e.g. above a threshold, and store these anomalies in a separate Azure SQL database table for further analysis. How would you design this pipeline to ensure low latency, scalability, and fault tolerance? Please include any PySpark transformations or optimizations you would use.”
Why is Schema Registry used with Kafka?
Asked once · last asked Feb 18 · Concept · Mid-level
Other ways it was asked
- “How do you use Schema Registry with Kafka?”
Why would you use DynamoDB Streams, and what benefits do they provide?
Asked once · last asked Jun 23 · Concept · Mid-level
Would you pass each event through the normalizer individually or process events in batches?
Asked once · last asked Jul 20 · Design · Mid-level
How would you design a simplified real-time data processing architecture using AWS services and Kafka Streams?
Asked once · last asked Sep 17 · Design · Senior
Other ways it was asked
- “What are the main components of a real-time data processing architecture using AWS and Kafka Streams, and how do they interact?”
What kinds of batch and real-time integrations can Apache NiFi support, and how would you implement them?
Asked once · last asked Feb 6 · Tooling · Mid-level
How do you create a Kafka Connect connector?
Asked once · last asked Feb 18 · Tooling · Mid-level
What audio chunk duration would you choose for speech analysis and translation, and which portion of the audio would you send to the model?
Asked once · last asked Feb 27 · Scenario · Senior
What is Kafka Streams used for, and how can you use it?
Asked once · last asked Mar 6 · Tooling · Mid-level
When migrating from Postgres to MySQL, how would you propagate CDC changes to keep the target system in sync?
Asked once · last asked Mar 20 · Design · Senior
What would you process with Flink versus Spark? Can you differentiate the two and explain how you would use each?
Asked once · last asked May 12 · Concept · Mid-level
Given Kafka data streaming into 40–50 PostgreSQL staging tables, how would you design near-real-time processing that transforms the data within 15 minutes and publishes it to downstream Kafka, without batch jobs or scheduled orchestration?
Asked once · last asked May 18 · Design · Senior
Other ways it was asked
- “Given a system that streams data from Kafka across 40–50 tables into PostgreSQL as a staging layer, how would you design near-real-time processing that transforms the data within 15 minutes and publishes it to a downstream Kafka layer, without relying on batch jobs or scheduled orchestration?”
What design considerations would you keep in mind when building a data integration pipeline that processes data from Kafka, applies business logic, and loads it into target tables or a database?
Asked once · last asked May 19 · Design · Senior
How would you roll out a schema change to the notification payload without breaking existing consumers?
Asked once · last asked May 23 · Design · Senior
How would you handle a consumer that cannot be updated quickly but must keep working during a notification payload schema migration?
Asked once · last asked May 23 · Scenario · Senior
What are the advantages and disadvantages of using Kafka Streams instead of Java Streams?
Asked once · last asked Jun 15 · Concept · Senior
What events does DynamoDB Streams emit when a table changes?
Asked once · last asked Jun 23 · Concept · Mid-level
How do you capture and process events from a DynamoDB Stream?
Asked once · last asked Jun 23 · Scenario · Mid-level
How would you validate that your running totals are correct and consistent when events can arrive late or out of order?
Asked once · last asked Jun 25 · Scenario · Senior
For event data, how would you define the metric and its time window, choose between event time and processing time, and handle duplicates?
Asked once · last asked Jun 25 · Scenario · Senior
How would you statistically detect and quantify whether events arriving after the watermark are biasing key KPIs, and what would you do if they are?
Asked once · last asked Jun 25 · Scenario · Senior
How would you design a CDC pipeline that ingests data from a streaming source such as Kafka or Flink and lands it in Delta Lake or Apache Iceberg?
Asked once · last asked Jul 16 · Design · Senior
How would you handle added, removed, or renamed columns when applying CDC events to a data lake?
Asked once · last asked Jul 16 · Scenario · Senior
Other ways it was asked
- “How do you handle schema changes when consuming CDC events into a lake destination?”
- “How would you handle added, removed, or renamed columns when applying CDC events to Delta Lake?”
How would you implement user-level deduplication and eligible-universe logic in a streaming pipeline, such as Kafka, Spark, Flink, BigQuery, or Snowflake, while handling late events and avoiding double counting?
Asked once · last asked Jun 30 · Design · Senior
Other ways it was asked
- “How would you implement the same user-level deduplication and eligible-universe logic in a streaming pipeline, such as Kafka, Spark, Flink, BigQuery, or Snowflake, while handling late events and avoiding double counting?”
How do you decide between real-time, batch, and other data-ingestion patterns for a use case?
Asked once · last asked Jul 16 · Scenario · Mid-level
Would you process records from a third-party platform individually or in batches, and what would you write to Kafka first?
Asked once · last asked Jul 20 · Design · Mid-level
How would you read data from Azure Event Hubs using Spark Structured Streaming and Kafka?
Asked once · last asked Aug 7 · Tooling · Mid-level
Is the Spark Kafka connector available in Databricks, or only in open source Spark?
Asked once · last asked Aug 7 · Concept · Mid-level
What is micro-batch processing, and when would you use it?
Asked once · last asked Aug 11 · Concept · Mid-level
How can an event grouper determine which user actions belong together, and where should it store the grouping records?
Asked once · last asked Aug 14 · Design · Senior
How would you approach the architecture to handle real-time processing and personalization?
Asked once · last asked Aug 17 · Design · Mid-level
How would you build a real-time fraud detection model, and which metrics would you use to evaluate its performance?
Asked once · last asked Aug 19 · Scenario · Mid-level
How do you engineer data ingestion to supply real-time feature context to ML models while handling data drift, schema changes, or high concurrency?
Asked once · last asked Aug 25 · Design · Senior
A company needs batch and streaming processing, including hourly and real-time workloads, in GCP. Which technology would you choose?
Asked once · last asked Sep 9 · Design · Senior
Why might Pub/Sub alone not be sufficient for a GCP solution that needs both batch and streaming processing?
Asked once · last asked Sep 9 · Concept · Senior
Why might Dataproc not be the best choice for a GCP solution that needs both batch and streaming processing?
Asked once · last asked Sep 9 · Concept · Senior
Other ways it was asked
- “Why might you choose not to use both Dataflow and Dataproc for live stream processing in GCP?”
Why does Spark use micro-batch processing?
Asked once · last asked Sep 10 · Concept · Mid-level
How would you choose file formats for a data platform using Kafka, object storage, and Spark?
Asked once · last asked Sep 10 · Design · Senior
How do you handle late-arriving data in Kafka?
Asked once · last asked Sep 10 · Scenario · Senior
How does watermarking work in stream processing?
Asked once · last asked Sep 10 · Concept · Senior
What are the key components a data platform needs to ingest batch, streaming, and diverse enterprise data sources?
Asked once · last asked Sep 14 · Design · Senior
What happens if streaming data is delayed or missing, and how should models compensate so reporting or forecasting is not impacted?
Asked once · last asked Sep 14 · Scenario · Senior
When would you use real-time APIs or event streams rather than batch-based data integration?
Asked once · last asked Sep 16 · Concept · Mid-level
How would you ingest an Avro payload from Kafka into Snowflake?
Asked once · last asked Sep 16 · Scenario · Senior
When would you choose a platform event over Change Data Capture to integrate with an external system, and what trade-offs and criteria would you consider? Give one concrete example.
Asked once · last asked Sep 17 · Design · Senior
When validating a raw event stream and defining the window duration for a real-time recommendation system, what key metric would you prioritize to ensure accurate recommendations?
Asked once · last asked Sep 17 · Scenario · Senior
How would you design Kafka topics, brokers, partitions, and Flink jobs to handle millions of user requests?
Asked once · last asked Sep 17 · Design · Senior
What role can Kafka play in a data pipeline, and what are some common use cases?
Asked once · last asked Sep 20 · Concept · Mid-level
What is the difference between Apache Airflow and Kafka?
Asked once · last asked Sep 20 · Concept · Mid-level
Why would you use Apache Flink with Kafka in a data pipeline?
Asked once · last asked Sep 20 · Scenario · Mid-level
Can you write an AWS Glue PySpark job to clean data coming from AWS Kinesis?
Asked once · last asked Sep 21 · Write code · Mid-level
How would you handle late-arriving data in a Kafka and Flink pipeline?
Asked once · last asked Sep 22 · Behavioral · Senior
Other ways it was asked
- “Tell me about a time you handled late-arriving data in a Kafka and Flink pipeline. What was the issue, and how did you resolve it?”
How would you monitor a late-data handling policy in Kafka and Flink to ensure it is not too aggressive and does not discard too much late-arriving data?
Asked once · last asked Sep 22 · Scenario · Senior
What is the integration strategy for large-scale data and event-driven updates?
Asked once · last asked Sep 30 · Design · Senior
How would you reduce the cost and optimize real-time transcript processing beyond optimizing the LLM itself?
Asked once · last asked Oct 7 · Design · Senior
How should a data pipeline handle late-arriving events?
Asked once · last asked Oct 18 · Scenario · Senior
How would you implement a real-time anomaly detection system, step by step?
Asked once · last asked Mar 27 · Design · Senior
Why use Kafka in a data ingestion pipeline?
Asked once · last asked Sep 16 · Concept · Mid-level
How does state management work in Apache Spark?
Asked once · last asked Sep 18 · Concept · Mid-level
More Kafka and Message Queues interview questions
- All Kafka and Message Queues interview questions
- Message Delivery Guarantees and Idempotency Interview Questions
How this list is made
Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.
Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.
The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Kafka and Message Queues interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.
169 interviews, 104 candidates, Jan 2026 to Oct 2026. Updated every month with new interviews.
Or add the Chrome extension