All 53 questions

  1. How would you evaluate retrieval and answer quality in a RAG system before production?

    Asked in 12 of 191 interviews · last asked Oct 26 · Concept · Mid-level

    Show answer

    I would test retrieval and answers separately on realistic questions with labeled relevant sources and expected answers. For retrieval, I would measure recall@k and ranking quality. For answers, I would check correctness, support from sources, and citation accuracy, using human review to validate automated scores. I would also test missing evidence, access rules, latency, and cost before release.

    • Use held-out questions from each customer, including hard cases and questions the documents cannot answer.
    • Recall@k measures relevant sources found; ranking metrics check whether useful sources appear early.
    • Check answer correctness and source support separately; a supported answer can still be incomplete.
    • Set release thresholds and compare against a baseline after each pipeline change.

    What the interviewer checks: The interviewer wants to see whether you can separate retrieval failures from answer failures and define reliable release checks.

    Other ways it was asked
    • “How would you implement a RAG system step by step?”
    • “How would you evaluate a RAG pipeline for each customer?”
    • “How would you deploy and evaluate a RAG system, including its accuracy, latency, and performance?”
  2. How would you evaluate and compare retrieval and reranking approaches, and which metrics would you use?

    Asked in 4 of 191 interviews · last asked Oct 22 · Concept · Mid-level

    Show answer

    I would compare approaches on the same held-out set of real questions with labeled relevant passages. For retrieval, I would measure Recall@k; for reranking, nDCG@k and MRR. I would also compare latency, cost, and final answer quality. I would test rerankers on identical candidate sets, then compare complete pipelines.

    • Recall@k measures the fraction of known relevant passages that appear in the top k results.
    • nDCG@k rewards placing highly relevant passages near the top and supports graded relevance labels.
    • MRR measures how early the first relevant result appears; it does not measure coverage.
    • Check results across question types; reranking cannot recover relevant passages missing from its candidate set.

    What the interviewer checks: The interviewer is checking whether you can compare approaches fairly and separate retrieval coverage from ranking quality.

    Other ways it was asked
    • “How would you evaluate a retrieval pipeline after embedding and reranking, and which metrics would you use?”
    • “What's your evaluation strategy for retrieval?”
    • “How would you evaluate reranking and retrieval approaches as potential solutions for an enterprise AI platform?”
  3. What is Precision@k, how is it calculated, and what counts as a relevant result?

    Asked in 2 of 191 interviews · last asked Jul 20 · Concept · Mid-level

    Other ways it was asked
    • “What is Precision@k, and how is it used to evaluate a RAG system?”
    • “In retrieval evaluation with top-10 results, how do you calculate precision, and what counts as a true positive?”
  4. How would you test whether a change to chunking improves retrieval and answer quality?

    Asked in 2 of 191 interviews · last asked Oct 22 · Scenario · Mid-level

    Other ways it was asked
    • “How would you test a chatbot's data-chunking strategy to ensure it does not degrade answer quality?”
    • “How would you determine whether the new chunk-size configuration is actually better?”
  5. What would you do if a RAG system retrieves the right document but the LLM still generates an incorrect answer?

    Asked in 2 of 191 interviews · last asked Sep 30 · Scenario · Senior

    Other ways it was asked
    • “What would you do if a RAG system retrieved the wrong document?”
  6. How would you diagnose whether incorrect RAG answers come from retrieval, chunking, prompts, or the language model?

    Asked in 2 of 191 interviews · last asked Oct 22 · Scenario · Senior

    Other ways it was asked
    • “How would you determine whether poor RAG results come from retrieval or generation?”
    • “If an enterprise RAG system answers confidently but incorrectly even though the information is present in the document, how would you investigate the problem?”
    • “How would you determine whether the problem comes from retrieval, chunk size, the prompt, or the language model?”
  7. Which metrics do you use to evaluate retrieval quality in a RAG system?

    Asked in 2 of 191 interviews · last asked Nov 18 · Concept · Mid-level

    Other ways it was asked
    • “What metrics do you use to evaluate retrieval quality?”
  8. How would you evaluate a hybrid BM25 and k-nearest-neighbor vector-search approach, and which metrics would you use?

    Asked once · last asked Sep 7 · Concept · Senior

    Other ways it was asked
    • “How would you test whether adding k-nearest-neighbor vector search to BM25 improves performance compared with a BM25 baseline?”
    • “How would you evaluate a hybrid BM25 and vector-search approach, and how would you choose the weights for each retrieval method?”
  9. How would you evaluate whether an LLM's answer to a question about a document, such as a loan eligibility decision, is correct?

    Asked once · last asked May 12 · Scenario · Senior

  10. Which retrieval metrics can help assess whether retrieved chunks are complete and relevant?

    Asked once · last asked May 13 · Concept · Mid-level

  11. How would you evaluate a model's accuracy and retrieval quality using metrics such as precision, recall, or mean average precision?

    Asked once · last asked Jun 12 · Scenario · Senior

  12. Which metrics would you use to measure reductions in hallucinations and improvements in retrieval accuracy?

    Asked once · last asked Jun 29 · Concept · Mid-level

  13. How does the Ragas framework work, particularly for measuring faithfulness?

    Asked once · last asked Jun 29 · Concept · Mid-level

  14. Which metrics do you use to evaluate retrieval performance, especially when the classes are imbalanced?

    Asked once · last asked Jul 16 · Concept · Senior

  15. Besides recall and precision, what classical retrieval metrics would you use to evaluate retrieved chunks?

    Asked once · last asked Jul 20 · Concept · Mid-level

    Other ways it was asked
    • “Besides recall and precision, what classical retrieval metrics did you use to evaluate retrieved chunks?”
  16. How would you build an evaluation dataset for a RAG system that retrieves drug aliases?

    Asked once · last asked Jul 24 · Scenario · Senior

  17. How would you build an offline evaluation set and scoring rubric for a RAG question-answering feature that reliably detects regressions in groundedness and usefulness?

    Asked once · last asked Aug 24 · Design · Senior

  18. What metrics would you use to diagnose and improve a semantic video-search model and its architecture?

    Asked once · last asked Sep 7 · Concept · Senior

  19. How would you calculate precision and query accuracy for a search system?

    Asked once · last asked Sep 7 · Concept · Mid-level

  20. How would you evaluate the effectiveness of ranking videos using their content context rather than metadata alone?

    Asked once · last asked Sep 7 · Concept · Senior

  21. How would you establish the correct answers to use as ground truth when evaluating a RAG pipeline?

    Asked once · last asked Sep 10 · Scenario · Senior

  22. Your RAG system has 95% retrieval recall, but users still complain about incorrect answers. How would you investigate?

    Asked once · last asked Sep 27 · Scenario · Senior

  23. What are recall and mean reciprocal rank (MRR), and how are they used to evaluate retrieval?

    Asked once · last asked Oct 7 · Concept · Mid-level

  24. A RAG system performs well in development but starts giving poor answers in production after three months. What could have happened, and how would you investigate?

    Asked once · last asked Oct 7 · Scenario · Senior

  25. How would you evaluate the quality of answers produced by a RAG system?

    Asked once · last asked Oct 20 · Behavioral · Mid-level

    Other ways it was asked
    • “How did you evaluate the quality of answers produced by a RAG system?”
  26. How would you verify that cited passages actually support a RAG system's answer?

    Asked once · last asked Oct 22 · Scenario · Mid-level

    Other ways it was asked
    • “How do you ensure the particular cited page actually supports the answer?”
    • “How would you test whether the final answer is supported by citations?”
  27. How would you test questions that cannot be answered from the documents?

    Asked once · last asked Oct 22 · Scenario · Mid-level

  28. What metrics and test set would you use to evaluate a RAG feature?

    Asked once · last asked Oct 26 · Scenario · Mid-level

  29. What is the difference between recall and MRR?

    Asked once · last asked Oct 26 · Concept · Mid-level

  30. How would you prepare a golden set and choose metrics to evaluate quantization when migrating a search encoder from PyTorch to ONNX?

    Asked once · last asked Oct 28 · Scenario · Senior

  31. What criteria would you use to preserve retrieval quality, and under what conditions would you reject quantization?

    Asked once · last asked Oct 28 · Scenario · Senior

  32. What are MLOps and LLMOps, and how do you evaluate a RAG system?

    Asked once · last asked Nov 5 · Concept · Senior

  33. How do you measure semantic search speed and accuracy, and validate that the results are reliable?

    Asked once · last asked Nov 7 · Scenario · Senior

  34. How do you identify incorrect matches during semantic search?

    Asked once · last asked Nov 7 · Scenario · Mid-level

  35. How would you scale a process for correcting incorrect semantic-search matches?

    Asked once · last asked Nov 7 · Scenario · Senior

  36. How would you adapt a semantic-search correction process as new data is added or the product categories change?

    Asked once · last asked Nov 7 · Scenario · Senior

  37. If a deployed RAG-based advisory agent shows hallucinations and inconsistent performance, what's your first step in diagnosing retrieval quality issues in production?

    Asked once · last asked Nov 11 · Scenario · Senior

  38. Which trace metrics would you examine to diagnose retrieval degradation and hallucinations in a RAG system, and how would you adjust the pipeline to improve relevance and grounding in a financial compliance context?

    Asked once · last asked Nov 11 · Scenario · Senior

  39. How would you reliably validate that tuning chunk size, search settings, and prompts improves retrieval quality?

    Asked once · last asked Nov 11 · Behavioral · Senior

    Other ways it was asked
    • “How did you actually validate that the retrieval quality was improving when you tuned chunk size, search settings, and prompts? What did you do to confirm the results were reliable?”
  40. How would you troubleshoot a production RAG system that produces irrelevant answers?

    Asked once · last asked Nov 11 · Scenario · Senior

  41. You're building a RAG system over 10,000 internal HR policy documents, and the chatbot sometimes gives answers that sound right but are actually wrong. How would you diagnose and fix this issue?

    Asked once · last asked Nov 12 · Scenario · Senior

  42. When evaluating a RAG document intelligence system, what alternatives would you compare, and how would you determine which performs better?

    Asked once · last asked Nov 13 · Scenario · Mid-level

  43. How would you handle a RAG system that retrieves irrelevant passages or generates answers unsupported by its documents?

    Asked once · last asked Nov 13 · Scenario · Mid-level

  44. What specific metrics would you track to measure improvement in retrieval relevance?

    Asked once · last asked Nov 15 · Concept · Senior

  45. How would you decide whether reranking adjustments are effective over time?

    Asked once · last asked Nov 15 · Scenario · Senior

    Other ways it was asked
    • “How would you decide if the re ranking adjustments are effective over time?”
  46. What strategies do you use, and how do you guide your team, when a RAG application isn’t working as expected?

    Asked once · last asked Nov 17 · Scenario · Senior

  47. How would you evaluate whether an LLM response is grounded in retrieved context and avoids unsupported claims?

    Asked once · last asked Nov 18 · Scenario · Senior

  48. What does precision measure when evaluating a RAG pipeline?

    Asked once · last asked Sep 10 · Concept · Junior

    Other ways it was asked
    • “How do you calculate precision when evaluating a RAG pipeline?”
  49. What is a false positive in RAG evaluation?

    Asked once · last asked Sep 10 · Concept · Junior

    Other ways it was asked
    • “What is a false negative in RAG evaluation?”
  50. What are precision, recall, and R-precision?

    Asked once · last asked Aug 3 · Concept · Junior

  51. How can you verify RAG evaluation results against the source documents?

    Asked once · last asked Sep 10 · Scenario · Mid-level

  52. What deliverable would demonstrate that a RAG evaluation was successful?

    Asked once · last asked Oct 22 · Scenario · Mid-level

  53. How would you evaluate the strengths and weaknesses of an LLM response for a RAG task?

    Asked once · last asked Nov 18 · Scenario · Mid-level

More RAG interview questions

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every RAG interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

191 interviews, 122 candidates, Nov 2025 to Nov 2026. Updated every month with new interviews.