95 Incremental Loading, CDC, and Streaming Interview Questions

All 95 questions

How would you implement a Slowly Changing Dimension Type 2 in an ETL process?
Asked in 4 of 307 interviews · last asked Oct 7 · Scenario · Mid-level
Show answer
I would use SCD Type 2 to keep a new dimension row for each tracked change, preserving history. I would match incoming records to current rows by business key and compare tracked fields. For a change, I would close the old row and insert a new row with a new surrogate key, validity dates, and a current flag.
- Insert new business keys as current rows; leave unchanged records alone.
- Set the old row’s end time to the new row’s start time; treat end times as exclusive.
- Expire and insert rows in one transaction, and make retries safe.
- Link facts to the dimension version valid at the fact’s event time.
What the interviewer checks: The interviewer checks whether you can preserve history and link facts to the correct dimension version.
Other ways it was asked
- “How do you implement Slowly Changing Dimension Type 2 (SCD Type 2) in a data warehouse?”
- “What are slowly changing dimensions (SCDs) in data warehousing?”
- “What is a slowly changing dimension (SCD) in data warehousing?”
What is an idempotent data pipeline, and how does idempotency help?
Asked in 3 of 307 interviews · last asked Oct 18 · Concept · Mid-level
Show answer
An idempotent data pipeline leaves the output data in the same state when it processes the same input again. It does not create duplicates or apply the same change twice. This makes retries after failures and reprocessing safer.
- Use stable record keys to identify data already processed.
- Upsert by key or replace the affected partition instead of blindly appending.
- Repeated updates must not add the same amount to a total twice.
Code
```
Input: order_id=42, amount=100.
First run: insert the order using order_id as a unique key.
Retry: update that order to amount=100, rather than insert another row.
Result: one order row with amount=100 after either run.
```
What the interviewer checks: The interviewer checks whether you can make retries safe without duplicating data or changing results.
Other ways it was asked
- “What does idempotency mean in a data integration pipeline?”
- “What does it mean for a data pipeline to be idempotent? Give a concrete example.”
How would you design incremental loading to handle late-arriving updates across large time windows?
Asked in 2 of 307 interviews · last asked Jul 20 · Design · Senior
Other ways it was asked
- “How would you change a data pipeline to handle revisions across large time windows?”
- “How would you redesign the data flow to handle revisions across large time windows?”
- “Suppose you’re designing an end-to-end pipeline from the source through cloud storage, Spark, a database, and a dashboard, and the source sends late-arriving updates. How would you design incremental logic that remains correct throughout data loading and the pipeline?”
How would you safely recover a partially loaded API batch and prove no records are missing or duplicated?
Asked in 2 of 307 interviews · last asked Sep 29 · Scenario · Senior
Other ways it was asked
- “Suppose the API fails after five pages have been retrieved. How would you recover the batch without losing records or creating duplicates?”
- “If a new engineer has to support an incremental ADF-to-Snowflake pipeline while you are away, what would you document so they can detect a partial batch, rerun it safely, and prove the recovered data is complete?”
- “If a nightly load fails on 8% of an API batch after some rows have landed in raw and some may have reached the curated table, what recovery sequence would you follow from containment to checkpoint commit, and how would you prove the rerun did not skip or double-count records?”
A source record is corrected today, but it belongs to last month. The dashboard still shows the old value after its scheduled refresh. How would you investigate and fix it?
Asked in 2 of 307 interviews · last asked Sep 23 · Scenario · Senior
Other ways it was asked
- “A source record from last week is corrected today, but the daily Power BI dashboard still shows the old value. How would you investigate and fix it?”
How would you ensure reliable data synchronization between MongoDB and OpenSearch?
Asked once · last asked Sep 7 · Design · Senior
Other ways it was asked
- “How would you minimize synchronization delays between MongoDB and OpenSearch?”
- “How would you design reliable synchronization between MongoDB, OpenSearch, and APIs?”
If shipment records are being added or updated while your pipeline is moving through the API pages, how would you reduce the risk of missing records or creating duplicates?
Asked once · last asked Sep 22 · Scenario · Senior
Other ways it was asked
- “If shipment records are added or updated while a pipeline is paging through an API, how would you reduce the risk of missing records or creating duplicates?”
- “If shipment records are being added or updated while your pipeline is moving through the API pages, how would you reduce the risk of missing or duplicate records?”
How would you ingest changes from an on-premises SQL Server database, and why choose direct reads or transaction-log CDC?
Asked once · last asked Sep 16 · Scenario · Senior
Other ways it was asked
- “How would you design ingestion from an on-premises SQL Server database, choosing between direct reads and transaction-log CDC?”
What is the difference between an upsert and CDC?
Asked once · last asked Mar 20 · Concept · Mid-level
What SQL would you use to apply CDC events from a source such as Kafka to a target table?
Asked once · last asked Mar 20 · Write code · Senior
When applying CDC changes with a MERGE or upsert, how would you identify which target rows to update?
Asked once · last asked Mar 20 · Concept · Mid-level
What is an incremental load, and how can you implement it for a dataset?
Asked once · last asked Apr 14 · Concept · Mid-level
What is Change Data Feed (CDF) in Databricks, and when would you use it?
Asked once · last asked Apr 17 · Concept · Mid-level
How can you use Change Data Feed (CDF) to process data incrementally in a data architecture?
Asked once · last asked Apr 17 · Scenario · Senior
When would you use Change Data Feed (CDF) rather than filtering a table to identify changes?
Asked once · last asked Apr 17 · Concept · Senior
When you enable Change Data Feed (CDF) on a Delta table, what information does it provide?
Asked once · last asked Apr 17 · Concept · Mid-level
What are the risks or limitations of using Change Data Feed (CDF) in production?
Asked once · last asked Apr 17 · Scenario · Senior
How would you design a CDC pipeline using medallion architecture for nightly data from different transactional databases, including processing and best practices at the bronze, silver, and gold layers?
Asked once · last asked Apr 24 · Design · Senior
Other ways it was asked
- “How would you design a CDC data pipeline using medallion architecture for nightly data from different transactional databases, and what would you do at each layer—from the raw bronze layer through silver to gold? What best practices would you follow?”
When implementing CDC across multiple source systems, what should you keep in mind, including how to handle changes to previously loaded data?
Asked once · last asked Apr 24 · Scenario · Senior
How would you handle CDC for batch data when a new extract from a transactional database arrives each night?
Asked once · last asked Apr 24 · Scenario · Mid-level
In a typical ETL pipeline for training data, what's the difference between a full load and an incremental load, and when would you choose each?
Asked once · last asked Apr 25 · Concept · Mid-level
How would you design an incremental pipeline to handle late-arriving data and updates to historical records while keeping training datasets reproducible?
Asked once · last asked Apr 25 · Design · Senior
How do you implement idempotency and exactly-once behavior in your Python ETL when a delayed partition arrives and triggers a backfill?
Asked once · last asked Apr 25 · Scenario · Senior
In a Python batch ETL, how would you design pipeline retries so transient errors are retried safely but data quality failures don't cause endless reprocessing?
Asked once · last asked Apr 25 · Scenario · Senior
How do you ensure exactly-once processing and prevent duplicate writes in a production ETL pipeline when upstream systems retry or deliver late, out-of-order events?
Asked once · last asked May 3 · Design · Senior
How would you design a backfill for twelve months of data in a production pipeline while minimizing the impact on current daily loads and keeping results consistent with today's logic?
Asked once · last asked May 3 · Design · Senior
In a multistage pipeline—ingest, label, feature, and train—how do you decide where to enforce idempotency, and what mechanisms do you use to prevent duplicate or partial writes?
Asked once · last asked May 5 · Design · Senior
How do you handle incremental data loading in Azure Data Factory, and how do you ensure the pipeline is rerunnable if it fails halfway?
Asked once · last asked May 9 · Tooling · Senior
Assume the root cause was a Flink job restart that replayed Kafka messages, while the ClickHouse sync was not idempotent. What would you do, and how would you resolve it?
Asked once · last asked May 12 · Scenario · Senior
How should an AI ingestion pipeline handle document changes: full reindexing or incremental updates?
Asked once · last asked May 13 · Design · Senior
Given Kafka data streaming into 40–50 PostgreSQL staging tables, how would you design near-real-time processing that transforms the data within 15 minutes and publishes it to downstream Kafka, without batch jobs or scheduled orchestration?
Asked once · last asked May 18 · Design · Senior
Other ways it was asked
- “Given a system that streams data from Kafka across 40–50 tables into PostgreSQL as a staging layer, how would you design near-real-time processing that transforms the data within 15 minutes and publishes it to a downstream Kafka layer, without relying on batch jobs or scheduled orchestration?”
What design considerations would you keep in mind when building a data integration pipeline that processes data from Kafka, applies business logic, and loads it into target tables or a database?
Asked once · last asked May 19 · Design · Senior
How would you keep offer data in a database current when the offers change frequently?
Asked once · last asked May 22 · Scenario · Mid-level
When writing time-correlated queries across multiple data sources, how do you account for ingestion latency?
Asked once · last asked May 26 · Scenario · Senior
When designing an Azure Data Factory pipeline to ingest high-volume P&C claims data, which method offers the most efficient incremental loading strategy: using a watermark column, performing a full daily refresh, leveraging change data capture (CDC), or using a time-based windowing function?
Asked once · last asked Jun 10 · Scenario · Senior
How would you design a CDC pipeline that ingests data from a streaming source such as Kafka or Flink and lands it in Delta Lake or Apache Iceberg?
Asked once · last asked Jul 16 · Design · Senior
How would you implement user-level deduplication and eligible-universe logic in a streaming pipeline, such as Kafka, Spark, Flink, BigQuery, or Snowflake, while handling late events and avoiding double counting?
Asked once · last asked Jun 30 · Design · Senior
Other ways it was asked
- “How would you implement the same user-level deduplication and eligible-universe logic in a streaming pipeline, such as Kafka, Spark, Flink, BigQuery, or Snowflake, while handling late events and avoiding double counting?”
How would you control polling and relay frequency in a data pipeline while preventing duplicate data from being sent?
Asked once · last asked Jul 20 · Design · Senior
How would you ensure exactly-once processing semantics in a streaming pipeline, particularly with checkpointing, late-arriving data, and output to Delta Lake?
Asked once · last asked Aug 11 · Scenario · Senior
How do you handle incremental data loading and change data capture, CDC, in Databricks?
Asked once · last asked Aug 12 · Scenario · Mid-level
How do you handle versioning when a source PDF or spreadsheet is revised? Do you treat it as a new artifact, an in-place update, or a linked lineage, and how do you keep analyses reproducible?
Asked once · last asked Aug 17 · Scenario · Senior
Can you show how a target table is populated during an incremental load for employee ID and employee name, including the date columns?
Asked once · last asked Aug 18 · Scenario · Mid-level
In an SCD Type 2 pipeline, how would you set the start date, end date, and active flag using the current date?
Asked once · last asked Aug 18 · Concept · Mid-level
How do you implement incremental data loading in a data pipeline?
Asked once · last asked Aug 19 · Concept · Mid-level
When you work with Auto Loader, how does it detect new files?
Asked once · last asked Aug 20 · Tooling · Mid-level
Other ways it was asked
- “How does Auto Loader detect new files and handle schema changes or breakages?”
How would you write Databricks code to read a daily file and load only records not already in a target table?
Asked once · last asked Aug 20 · Write code · Mid-level
In a production ETL job, how would you make chunked aggregation idempotent so retries don't double-count results?
Asked once · last asked Aug 23 · Scenario · Senior
When loading incrementally, how do you prevent duplicate rows if events can arrive late or be replayed? What keys and constraints do you rely on?
Asked once · last asked Aug 23 · Scenario · Senior
How do you implement incremental load from bronze to silver layer?
Asked once · last asked Aug 25 · Scenario · Mid-level
What is a MERGE operation, and when would you use it to load data?
Asked once · last asked Aug 25 · Concept · Mid-level
How would you design a multi-source ingestion pipeline supporting full and incremental loads?
Asked once · last asked Aug 25 · Design · Senior
Other ways it was asked
- “How would you approach a requirement from a client to ingest data from multiple sources like SQL Server, Oracle, Salesforce, and flat files into a single platform?”
- “How would you design a data-driven pipeline to handle ingestion from multiple sources like SQL Server, Oracle, and Salesforce while managing both full and incremental loads?”
How do you prevent duplicates during incremental loads using control tables and watermarks?
Asked once · last asked Aug 25 · Scenario · Senior
Other ways it was asked
- “How do you prevent duplicate records during incremental loads when the update or increment column has already been updated?”
- “How do you ensure data deduplication when using a control table and watermark strategy for incremental loads?”
How will you ensure the watermark column in your control table is not updated if the pipeline fails midway?
Asked once · last asked Aug 25 · Scenario · Senior
How does Snowflake track which files have already been processed to prevent duplicate data ingestion?
Asked once · last asked Aug 26 · Concept · Senior
How would you implement delta processing or change data capture in an SAP data integration solution?
Asked once · last asked Aug 28 · Scenario · Senior
What conditions would lead you to requeue a source in a web-scraping pipeline?
Asked once · last asked Aug 29 · Scenario · Mid-level
How would you design a bulk process for 2,000,000 rows that fails after 800,000 so it can restart safely without reprocessing completed rows?
Asked once · last asked Sep 4 · Design · Senior
How would you structure a Python workflow to run daily with parameterized dates, log metrics, and remain idempotent after partial failure?
Asked once · last asked Sep 7 · Scenario · Senior
Other ways it was asked
- “Switching to automation, how would you structure a Python workflow to run daily parameterized dates, log metrics, and remain idempotent after partial failure?”
For an idempotent data workflow, what rules would you use to decide whether to skip, upsert, or fully rebuild a partition or date's data?
Asked once · last asked Sep 7 · Scenario · Senior
Why doesn't idempotency alone always handle late-arriving updates, and how would you process them correctly?
Asked once · last asked Dec 15 · Concept · Senior
How would you detect and capture only changed records and columns when tracking order updates?
Asked once · last asked Sep 18 · Scenario · Mid-level
How would you handle near real-time data replication between Salesforce and an external system?
Asked once · last asked Sep 18 · Design · Senior
How would you prevent multiple updates to the same shipment from creating duplicate fact records?
Asked once · last asked Sep 20 · Scenario · Senior
How would you design an end-to-end data pipeline using incremental loading, justify that choice, and make reruns safe?
Asked once · last asked Sep 20 · Design · Senior
Other ways it was asked
- “Walk me through an end-to-end data pipeline you built. Why did you use incremental loading, and how did you make reruns safe?”
What role can Kafka play in a data pipeline, and what are some common use cases?
Asked once · last asked Sep 20 · Concept · Mid-level
Why would you use Apache Flink with Kafka in a data pipeline?
Asked once · last asked Sep 20 · Scenario · Mid-level
How would you model several updates for the same operational event when its status or timestamp changes more than once?
Asked once · last asked Sep 21 · Scenario · Mid-level
How would you prevent records from being skipped or retrieved twice when records are added or updated during pagination?
Asked once · last asked Sep 21 · Scenario · Senior
How would you handle corrections or repeated updates without creating duplicate fact records?
Asked once · last asked Sep 21 · Scenario · Mid-level
Why would you use an updated-at timestamp together with a sequence value for a watermark instead of relying on the timestamp alone?
Asked once · last asked Sep 21 · Concept · Senior
Other ways it was asked
- “Why did you use updated-at together with a sequence value for the watermark instead of relying on the timestamp alone?”
What can happen if a data pipeline fails before its checkpoints are updated, and how would you handle recovery?
Asked once · last asked Sep 21 · Scenario · Senior
How do you manage change data capture in Snowflake?
Asked once · last asked Sep 22 · Tooling · Mid-level
How would you handle duplicate records in Snowpipe if an API request is replayed or a notification is delivered more than once?
Asked once · last asked Sep 22 · Scenario · Senior
How would you choose a business key for deduplication, and how would you handle late-arriving data?
Asked once · last asked Sep 22 · Scenario · Senior
If your PySpark job on Dataproc completes successfully but the BigQuery load fails halfway through, how would you recover without creating duplicate data?
Asked once · last asked Sep 23 · Scenario · Senior
How would you design an ADF and Snowflake pipeline so corrections made in the source system after records are first created appear in the target without creating duplicates?
Asked once · last asked Sep 23 · Design · Senior
What would you do if the source system did not provide a reliable update timestamp or change sequence?
Asked once · last asked Sep 23 · Scenario · Senior
How would you handle errors, retries, and duplicate records in an API integration using batch data processing?
Asked once · last asked Sep 28 · Behavioral · Mid-level
Other ways it was asked
- “How did you handle errors, retries, and duplicate records in an API integration using batch data processing?”
We receive an incremental API feed in pages, and a page has landed in Snowflake but needs to be rerun. How would you design the checkpoint and merge so the rerun neither skips remaining records nor duplicates records already loaded?
Asked once · last asked Sep 28 · Design · Senior
What would you change if the API had no reliable updated-at field and records could be corrected days after they were first created? How would you avoid missing those corrections, and what limitation would you state about completeness?
Asked once · last asked Sep 28 · Design · Senior
If the unit conversion changed from 12 to 10 units last month, how would you preserve historically correct quantities when rerunning and reporting on past periods?
Asked once · last asked Sep 28 · Scenario · Senior
If a source file used in a daily supply-risk report is overwritten after yesterday’s successful load, and today’s file has the same name but different rows, how would you detect the change, decide whether it is a correction or a new batch, and avoid double-counting?
Asked once · last asked Sep 29 · Scenario · Senior
If an API load fails after some pages have landed, how would you rerun the batch without missing transactions or creating duplicates, and when would you advance the incremental checkpoint?
Asked once · last asked Sep 29 · Scenario · Senior
If a business user overwrites yesterday’s SharePoint CSV with a corrected file under the same name after your pipeline has succeeded, how would you detect the change and prevent duplicate records or an unexplained change to a published report?
Asked once · last asked Sep 29 · Scenario · Senior
What is the integration strategy for large-scale data and event-driven updates?
Asked once · last asked Sep 30 · Design · Senior
What is the purpose of SQL MERGE logic in a data pipeline, and how would you improve it?
Asked once · last asked Oct 2 · Scenario · Mid-level
How would you implement incremental loading for a huge table that cannot be reloaded every day?
Asked once · last asked Oct 7 · Scenario · Mid-level
When would distributed transactions be appropriate in a file-ingestion pipeline that updates multiple database tables?
Asked once · last asked Oct 7 · Scenario · Senior
How would you design a data pipeline that isolates failures and prevents interruptions from causing duplicate data?
Asked once · last asked Oct 12 · Design · Senior
How should a data pipeline handle late-arriving events?
Asked once · last asked Oct 18 · Scenario · Senior
How would you handle incremental processing and deletions in a data pipeline?
Asked once · last asked Oct 19 · Scenario · Mid-level
A source database contains 100,000,000 records, and you need to load new and updated records every day. Would you reload everything?
Asked once · last asked Oct 23 · Scenario · Mid-level
If your pipeline runs twice because of a retry and the destination table contains duplicate records, how would you fix it?
Asked once · last asked Oct 23 · Scenario · Mid-level
Why use Kafka in a data ingestion pipeline?
Asked once · last asked Sep 16 · Concept · Mid-level
What is change data capture (CDC)?
Asked once · last asked Sep 18 · Concept · Junior

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every ETL and Data Pipelines interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

307 interviews, 169 candidates, Nov 2025 to Oct 2026. Updated every month with new interviews.

All 95 questions

How would you implement a Slowly Changing Dimension Type 2 in an ETL process?

What is an idempotent data pipeline, and how does idempotency help?

How would you design incremental loading to handle late-arriving updates across large time windows?

How would you safely recover a partially loaded API batch and prove no records are missing or duplicated?

A source record is corrected today, but it belongs to last month. The dashboard still shows the old value after its scheduled refresh. How would you investigate and fix it?

How would you ensure reliable data synchronization between MongoDB and OpenSearch?

If shipment records are being added or updated while your pipeline is moving through the API pages, how would you reduce the risk of missing records or creating duplicates?

How would you ingest changes from an on-premises SQL Server database, and why choose direct reads or transaction-log CDC?

What is the difference between an upsert and CDC?

What SQL would you use to apply CDC events from a source such as Kafka to a target table?

When applying CDC changes with a MERGE or upsert, how would you identify which target rows to update?

What is an incremental load, and how can you implement it for a dataset?

What is Change Data Feed (CDF) in Databricks, and when would you use it?

How can you use Change Data Feed (CDF) to process data incrementally in a data architecture?

When would you use Change Data Feed (CDF) rather than filtering a table to identify changes?

When you enable Change Data Feed (CDF) on a Delta table, what information does it provide?

What are the risks or limitations of using Change Data Feed (CDF) in production?

How would you design a CDC pipeline using medallion architecture for nightly data from different transactional databases, including processing and best practices at the bronze, silver, and gold layers?

When implementing CDC across multiple source systems, what should you keep in mind, including how to handle changes to previously loaded data?

How would you handle CDC for batch data when a new extract from a transactional database arrives each night?

In a typical ETL pipeline for training data, what's the difference between a full load and an incremental load, and when would you choose each?

How would you design an incremental pipeline to handle late-arriving data and updates to historical records while keeping training datasets reproducible?

How do you implement idempotency and exactly-once behavior in your Python ETL when a delayed partition arrives and triggers a backfill?

In a Python batch ETL, how would you design pipeline retries so transient errors are retried safely but data quality failures don't cause endless reprocessing?

How do you ensure exactly-once processing and prevent duplicate writes in a production ETL pipeline when upstream systems retry or deliver late, out-of-order events?

How would you design a backfill for twelve months of data in a production pipeline while minimizing the impact on current daily loads and keeping results consistent with today's logic?

In a multistage pipeline—ingest, label, feature, and train—how do you decide where to enforce idempotency, and what mechanisms do you use to prevent duplicate or partial writes?

How do you handle incremental data loading in Azure Data Factory, and how do you ensure the pipeline is rerunnable if it fails halfway?

Assume the root cause was a Flink job restart that replayed Kafka messages, while the ClickHouse sync was not idempotent. What would you do, and how would you resolve it?

How should an AI ingestion pipeline handle document changes: full reindexing or incremental updates?

Given Kafka data streaming into 40–50 PostgreSQL staging tables, how would you design near-real-time processing that transforms the data within 15 minutes and publishes it to downstream Kafka, without batch jobs or scheduled orchestration?

What design considerations would you keep in mind when building a data integration pipeline that processes data from Kafka, applies business logic, and loads it into target tables or a database?

How would you keep offer data in a database current when the offers change frequently?

When writing time-correlated queries across multiple data sources, how do you account for ingestion latency?

When designing an Azure Data Factory pipeline to ingest high-volume P&C claims data, which method offers the most efficient incremental loading strategy: using a watermark column, performing a full daily refresh, leveraging change data capture (CDC), or using a time-based windowing function?

How would you design a CDC pipeline that ingests data from a streaming source such as Kafka or Flink and lands it in Delta Lake or Apache Iceberg?

How would you implement user-level deduplication and eligible-universe logic in a streaming pipeline, such as Kafka, Spark, Flink, BigQuery, or Snowflake, while handling late events and avoiding double counting?

How would you control polling and relay frequency in a data pipeline while preventing duplicate data from being sent?

How would you ensure exactly-once processing semantics in a streaming pipeline, particularly with checkpointing, late-arriving data, and output to Delta Lake?

How do you handle incremental data loading and change data capture, CDC, in Databricks?

How do you handle versioning when a source PDF or spreadsheet is revised? Do you treat it as a new artifact, an in-place update, or a linked lineage, and how do you keep analyses reproducible?

Can you show how a target table is populated during an incremental load for employee ID and employee name, including the date columns?

In an SCD Type 2 pipeline, how would you set the start date, end date, and active flag using the current date?

How do you implement incremental data loading in a data pipeline?

When you work with Auto Loader, how does it detect new files?

How would you write Databricks code to read a daily file and load only records not already in a target table?

In a production ETL job, how would you make chunked aggregation idempotent so retries don't double-count results?

When loading incrementally, how do you prevent duplicate rows if events can arrive late or be replayed? What keys and constraints do you rely on?

How do you implement incremental load from bronze to silver layer?

What is a MERGE operation, and when would you use it to load data?

How would you design a multi-source ingestion pipeline supporting full and incremental loads?

How do you prevent duplicates during incremental loads using control tables and watermarks?

How will you ensure the watermark column in your control table is not updated if the pipeline fails midway?

How does Snowflake track which files have already been processed to prevent duplicate data ingestion?

How would you implement delta processing or change data capture in an SAP data integration solution?

What conditions would lead you to requeue a source in a web-scraping pipeline?

How would you design a bulk process for 2,000,000 rows that fails after 800,000 so it can restart safely without reprocessing completed rows?

How would you structure a Python workflow to run daily with parameterized dates, log metrics, and remain idempotent after partial failure?

For an idempotent data workflow, what rules would you use to decide whether to skip, upsert, or fully rebuild a partition or date's data?

Why doesn't idempotency alone always handle late-arriving updates, and how would you process them correctly?

How would you detect and capture only changed records and columns when tracking order updates?

How would you handle near real-time data replication between Salesforce and an external system?

How would you prevent multiple updates to the same shipment from creating duplicate fact records?

How would you design an end-to-end data pipeline using incremental loading, justify that choice, and make reruns safe?

What role can Kafka play in a data pipeline, and what are some common use cases?

Why would you use Apache Flink with Kafka in a data pipeline?

How would you model several updates for the same operational event when its status or timestamp changes more than once?

How would you prevent records from being skipped or retrieved twice when records are added or updated during pagination?

How would you handle corrections or repeated updates without creating duplicate fact records?

Why would you use an updated-at timestamp together with a sequence value for a watermark instead of relying on the timestamp alone?

What can happen if a data pipeline fails before its checkpoints are updated, and how would you handle recovery?

How do you manage change data capture in Snowflake?

How would you handle duplicate records in Snowpipe if an API request is replayed or a notification is delivered more than once?

How would you choose a business key for deduplication, and how would you handle late-arriving data?

If your PySpark job on Dataproc completes successfully but the BigQuery load fails halfway through, how would you recover without creating duplicate data?

How would you design an ADF and Snowflake pipeline so corrections made in the source system after records are first created appear in the target without creating duplicates?

What would you do if the source system did not provide a reliable update timestamp or change sequence?

How would you handle errors, retries, and duplicate records in an API integration using batch data processing?

We receive an incremental API feed in pages, and a page has landed in Snowflake but needs to be rerun. How would you design the checkpoint and merge so the rerun neither skips remaining records nor duplicates records already loaded?