Questions 201–312 of 312

  1. How would you evaluate unsupported claims in an LLM-generated answer: using precision, recall, or another metric?

    Asked once · last asked Oct 27 · Concept · Mid-level

  2. How would you turn the rubric criteria for scoring AI-generated monthly variance commentary into a rater guide?

    Asked once · last asked Nov 1 · Scenario · Senior

    Other ways it was asked
    • “How would you turn the rubric criteria for scoring AI-generated monthly variance commentary into a rater guide with anchors, examples, thresholds, and borderline cases?”
  3. How would you score an AI-generated answer that is partly correct but contains one major hallucinated claim?

    Asked once · last asked Nov 10 · Scenario · Mid-level

    Other ways it was asked
    • “How would you score an AI-generated answer that is partly correct but contains one major hallucinated claim, and what feedback would help improve it without losing the correct parts?”
  4. How would you improve an evaluation process to catch missing conditions in AI-generated mathematical solutions?

    Asked once · last asked Nov 13 · Behavioral · Mid-level

    Other ways it was asked
    • “What did you change in your evaluation process to catch missing conditions in AI-generated mathematical solutions?”
  5. Describe a task where an AI tool performed particularly well. What made its output good?

    Asked once · last asked Nov 14 · Behavioral · Mid-level

  6. How would you assess whether an LLM's output met the intended requirements, and what prompt would you use to address any gaps?

    Asked once · last asked Nov 17 · Scenario · Mid-level

    Other ways it was asked
    • “How would you assess whether Claude Haiku's output met the intended requirements, and what prompt would you use to address any gaps?”
  7. How have you approached AI evaluation in a role or project?

    Asked once · last asked Oct 25 · Behavioral · Mid-level

  8. How would you determine whether each new version of an AI agent's script is better?

    Asked once · last asked Oct 26 · Scenario · Mid-level

  9. What rubric would you use to score AI support replies for food delivery cases, including criteria, one-to-five scale definitions, and a pass/fail threshold?

    Asked once · last asked Oct 26 · Scenario · Senior

  10. Can you tell me about a specific instance when you had to evaluate the readability and safety of AI-generated text? What was your approach?

    Asked once · last asked Oct 27 · Behavioral · Mid-level

  11. How would you evaluate AI search systems such as ChatGPT, Gemini, and Perplexity in practice, and what would you check for?

    Asked once · last asked Oct 27 · Concept · Mid-level

    Other ways it was asked
    • “Can you walk me through how you evaluated AI search systems such as ChatGPT, Gemini, and Perplexity in practice, including what you were checking for?”
  12. Can you give me a specific example of a time you found a meaningful error or flaw in an AI output? What was the claim, and how did you verify it was wrong?

    Asked once · last asked Oct 27 · Behavioral · Mid-level

  13. When evaluating an AI response, how do you decide whether a claim is unverified or actually wrong?

    Asked once · last asked Oct 27 · Concept · Mid-level

  14. When evaluating an AI agent's reasoning—not just its factual claims—how do you assess whether its reasoning steps are sound, and what do you look for?

    Asked once · last asked Oct 27 · Concept · Mid-level

  15. When an AI output is technically correct but still misleading or unhelpful to the user, how would you approach scoring it?

    Asked once · last asked Oct 27 · Scenario · Mid-level

  16. How would you validate test cases generated by an LLM?

    Asked once · last asked Oct 27 · Scenario · Mid-level

  17. How would you structure an evaluation to detect when an LLM-generated analytic summary overstates source confidence?

    Asked once · last asked Oct 27 · Scenario · Senior

  18. How would you design a rubric to score LLM outputs across source handling, reasoning quality, limitation disclosure, and practical utility, with clear scoring anchors and guidance for partially supported claims?

    Asked once · last asked Oct 27 · Scenario · Senior

  19. In an offline benchmark, how would you prevent training to the test and data contamination while iterating on prompts or models, especially if the same raters also help author test cases?

    Asked once · last asked Oct 27 · Scenario · Senior

  20. How would you test an image-description agent when correct descriptions can use different words?

    Asked once · last asked Oct 28 · Scenario · Senior

  21. What are the three to five most important acceptance criteria you would write for approving the model?

    Asked once · last asked Oct 29 · Scenario · Senior

  22. How would you compare a prompt and its response with source data, and verify factual claims in the response?

    Asked once · last asked Oct 29 · Scenario · Mid-level

  23. When evaluating an AI response, how do you determine whether it solves the user's problem rather than merely sounding confident?

    Asked once · last asked Oct 29 · Scenario · Mid-level

  24. How would you quality-control grammar, factual accuracy, and age-appropriate language in a model for children aged four to ten?

    Asked once · last asked Oct 31 · Scenario · Mid-level

  25. When an offline safety or utility benchmark and online user outcomes disagree after a model update, how do you design a root-cause experiment to determine whether the cause is distribution shift, metric misspecification, or model behavior drift?

    Asked once · last asked Nov 1 · Scenario · Senior

  26. Suppose you suspect your safety benchmark is contaminated by training data. What concrete tests would you run to detect contamination, and how would you redesign the benchmark to remain useful over time?

    Asked once · last asked Nov 1 · Scenario · Senior

  27. Imagine you have only partial visibility into training data and cannot scan the corpora directly. How would you estimate contamination risk and decide whether to retire a benchmark?

    Asked once · last asked Nov 1 · Scenario · Senior

  28. Suppose leadership wants a single model quality score combining safety, helpfulness, and latency, but stakeholders disagree on the weights. How would you choose weights and make the decision robust to that ambiguity?

    Asked once · last asked Nov 1 · Scenario · Senior

  29. What five to seven rubric criteria would you define to score an AI-generated monthly variance commentary for revenue, gross margin, and OpEx across three entities objectively, and how would you weight them?

    Asked once · last asked Nov 1 · Scenario · Senior

  30. If an LLM is used to evaluate an AI agent, how should its evaluations be validated?

    Asked once · last asked Nov 2 · Scenario · Senior

  31. How can you detect correlated errors when an LLM judge evaluates an AI agent?

    Asked once · last asked Nov 2 · Scenario · Senior

  32. What are the top three signals you'd use to judge an AI sales assistant's response to “Your price looks high, and procurement will need to review any agreement”?

    Asked once · last asked Nov 2 · Scenario · Mid-level

  33. When you're independently rating large volumes of similar AI-generated sales responses, how do you prevent rubric drift over time while keeping throughput high?

    Asked once · last asked Nov 2 · Scenario · Senior

  34. Two model answers are both factually correct. One is three times longer, restating the question and adding caveats. Which ranks higher?

    Asked once · last asked Nov 3 · Scenario · Mid-level

  35. How do we quickly test the quality of AI output?

    Asked once · last asked Nov 4 · Scenario · Mid-level

  36. What is the groundedness tool in Azure AI Foundry evaluations, and what does it do?

    Asked once · last asked Nov 4 · Tooling · Mid-level

  37. If a refund chatbot says a refund is allowed and both the policy and source document state a 30-day limit, is the answer grounded, accurate, or both?

    Asked once · last asked Nov 4 · Scenario · Mid-level

  38. If an agent evaluation framework sets a task-duration limit under one hour, how should it account for METR results showing agents completing tasks that take people most of a workday?

    Asked once · last asked Nov 4 · Scenario · Senior

  39. If another language model evaluates an AI system's outputs, how should you verify the reliability of that evaluator?

    Asked once · last asked Nov 4 · Scenario · Senior

  40. Research results show AI agents completing tasks that take people most of a workday. Does that invalidate a one-hour threshold for agent capability?

    Asked once · last asked Nov 4 · Scenario · Senior

    Other ways it was asked
    • “METR's latest results show AI agents completing tasks that take people most of a workday. Does that invalidate a one-hour threshold for agent capability?”
  41. When you design a multi-step reasoning question, how do you ensure it cannot be solved by a single shortcut lookup, and what specific techniques do you use to make later steps depend on earlier ones?

    Asked once · last asked Nov 4 · Scenario · Senior

  42. How would you design an evaluation task to test whether an AI agent can identify a revenue discrepancy across a CRM record, an order entry, and emails?

    Asked once · last asked Nov 5 · Scenario · Senior

  43. What output format would you require an AI agent to use to demonstrate that it correctly reconciled a revenue discrepancy?

    Asked once · last asked Nov 5 · Scenario · Mid-level

  44. What should a human rater check to validate an AI agent's revenue reconciliation?

    Asked once · last asked Nov 5 · Scenario · Mid-level

  45. How would you label missing or conflicting evidence in an AI evaluation task?

    Asked once · last asked Nov 5 · Scenario · Senior

  46. What edge case would you include to test an AI agent's revenue-reconciliation reasoning, and how would you weight it in the scoring rubric?

    Asked once · last asked Nov 5 · Scenario · Senior

  47. How would you guard against your own assumptions or bias when designing an evaluation task for raters?

    Asked once · last asked Nov 5 · Scenario · Senior

  48. When an AI agent's explanation is brief, how can you reliably assess whether it is trustworthy?

    Asked once · last asked Nov 5 · Scenario · Senior

  49. How would you evaluate an LLM pipeline at scale using three to five reliability metrics, including over-refusal and cross-domain leakage, and set alert thresholds for rollback or human routing?

    Asked once · last asked Nov 5 · Scenario · Senior

    Other ways it was asked
    • “How would you evaluate this pipeline at scale? Which three to five reliability metrics would you include, at least one for over-refusal and one for cross-domain leakage? What alert thresholds would trigger rollback or human routing?”
  50. How do you design and validate an evaluation rubric that separately scores source handling, reasoning quality, limitation awareness, and practical utility while keeping inter-rater reliability high across reviewers?

    Asked once · last asked Nov 5 · Scenario · Senior

  51. When kappa is acceptable but the rubric still fails to predict real-world outcomes, for example high-scoring answers still getting rejected by analysts, how would you redesign the evaluation to fix that?

    Asked once · last asked Nov 5 · Scenario · Senior

  52. How would you keep an evaluation failure-mode taxonomy stable over time while allowing new error codes without breaking longitudinal metrics?

    Asked once · last asked Nov 5 · Scenario · Senior

  53. How would you verify that an AI-generated output is accurate?

    Asked once · last asked Nov 6 · Scenario · Mid-level

  54. How would you validate an AI-generated document against an existing bug comment before adding it to a knowledge base?

    Asked once · last asked Nov 6 · Scenario · Senior

  55. What's one metric or signal you use to judge whether AI-assisted summaries are actually improving speed, reducing errors, or increasing stakeholder satisfaction?

    Asked once · last asked Nov 7 · Concept · Mid-level

  56. If the correction rate rises, what's the first change you'd make: better input notes, tighter prompts, or a different review step?

    Asked once · last asked Nov 7 · Scenario · Mid-level

  57. How would you design a lightweight evaluation rubric with three to five criteria to score AI responses for a customer-facing knowledge-base article rewrite?

    Asked once · last asked Nov 7 · Scenario · Senior

    Other ways it was asked
    • “How would you design a lightweight evaluation rubric with three to five criteria to score AI responses for a customer-facing knowledge-based article rewrite?”
  58. How would you measure and improve inter-rater reliability when two senior reviewers frequently disagree on clarity scores for the same AI output?

    Asked once · last asked Nov 7 · Scenario · Senior

  59. How would you ensure that generated protocols accurately follow a defined methodology?

    Asked once · last asked Nov 8 · Design · Mid-level

  60. How would you test the accuracy of methodology-based protocols at scale?

    Asked once · last asked Nov 8 · Scenario · Senior

  61. How do you design a small golden set to measure whether model revisions are actually improving over time: size, selection strategy, and update cadence?

    Asked once · last asked Nov 10 · Scenario · Senior

  62. How do you prevent overfitting to a golden evaluation set, including data leakage and reviewer bias, while keeping evaluations reproducible?

    Asked once · last asked Nov 10 · Scenario · Senior

    Other ways it was asked
    • “How do you ensure your golden set doesn't get overfit by the team or the model, for example, leakage, reviewer bias while staying reproducible?”
  63. What does it mean to test an AI agent?

    Asked once · last asked Nov 11 · Concept · Mid-level

  64. How would you use a rubric or red-team prompt to uncover an AI model failure?

    Asked once · last asked Nov 11 · Behavioral · Mid-level

    Other ways it was asked
    • “Tell me about a rubric or red-team prompt you used to uncover an AI model failure.”
  65. Walk me through how you would use an AI assistant to design an evaluation comparing GPT and Gemini on generation for the same dataset, including metrics and controls.

    Asked once · last asked Nov 12 · Scenario · Senior

  66. When you're refining instructions with an AI tool, what's your single most reliable method to verify it didn't change the meaning or add new requirements?

    Asked once · last asked Nov 12 · Scenario · Senior

  67. When a model's answer is unclear, how do you assess its clarity and what do you document?

    Asked once · last asked Nov 13 · Scenario · Mid-level

  68. How do you choose a metric for LLM evaluation, and why is that metric appropriate?

    Asked once · last asked Nov 13 · Concept · Mid-level

  69. When building LLM applications or agents, how do you test whether they’re working reliably and catch problems before or after release?

    Asked once · last asked Nov 13 · Scenario · Senior

  70. Tell me about a time you identified an unsound step in an AI-generated mathematical solution. What revealed the flaw?

    Asked once · last asked Nov 13 · Behavioral · Mid-level

  71. How would you balance speed and rigor when evaluating an AI-generated mathematical solution?

    Asked once · last asked Nov 13 · Behavioral · Mid-level

    Other ways it was asked
    • “What trade-off between speed and rigor did you make when evaluating an AI-generated mathematical solution?”
  72. You're auditing an AI that grades SDR calls. It flags missed discovery too often. What criteria and call evidence would you require before that label is considered correct?

    Asked once · last asked Nov 13 · Scenario · Senior

  73. When you're evaluating Malay-language responses and providing feedback for fine-tuning, walk me through what that process actually looks like.

    Asked once · last asked Nov 14 · Tooling · Mid-level

  74. How do you evaluate LLM responses: by using APIs or other tools, or through manual review outside those systems?

    Asked once · last asked Nov 14 · Tooling · Mid-level

  75. How do you organize and track discrepancies during an AI output review?

    Asked once · last asked Nov 14 · Scenario · Mid-level

  76. How do you set up a new AI output evaluation task to ensure it runs smoothly from beginning to end?

    Asked once · last asked Nov 14 · Scenario · Mid-level

  77. If you were given a question-solving output validation task, what would you tackle first?

    Asked once · last asked Nov 14 · Scenario · Mid-level

  78. When you're evaluating an AI-generated report, what are the top three error types you look for first, and how do you quickly detect each?

    Asked once · last asked Nov 14 · Scenario · Senior

  79. How do you label an unsupported claim in your review, and what exact revision do you request from the AI?

    Asked once · last asked Nov 14 · Scenario · Mid-level

  80. You're given an AI answer with 10 bullet points and citations, but two citations point to the wrong source section. What's your step-by-step process to find and fix those miscitations efficiently?

    Asked once · last asked Nov 14 · Scenario · Senior

  81. How do you score overall quality for a response using a rubric or weighted criteria, and what thresholds trigger a minor edit versus a fail?

    Asked once · last asked Nov 14 · Scenario · Senior

  82. What specific checks do you personally perform to verify AI-generated summaries before sending them to leadership?

    Asked once · last asked Nov 16 · Scenario · Mid-level

  83. How would you evaluate an AI agent's output, and how would you decide how much to trust it?

    Asked once · last asked Nov 17 · Scenario · Senior

  84. When an AI agent generates test evidence, where would you look for it and what would you verify?

    Asked once · last asked Nov 17 · Scenario · Senior

  85. When comparing outputs from two LLMs, how would you decide whether to give one model source documents if the other was evaluated without them?

    Asked once · last asked Nov 17 · Scenario · Mid-level

    Other ways it was asked
    • “When comparing outputs from Gemini and Claude, how would you decide whether to give Claude the source documents or evaluate it without them?”
  86. What evidence would justify concluding that an LLM failed to meet the intended outcome?

    Asked once · last asked Nov 17 · Scenario · Mid-level

    Other ways it was asked
    • “What evidence would justify concluding that Claude Haiku failed to meet the intended outcome?”
  87. How do you build a human-verified golden set, and how do you score agent-written tests against it?

    Asked once · last asked Nov 17 · Scenario · Senior

  88. If a product changes weekly, how do you keep a human-verified golden set up to date, and who should own it?

    Asked once · last asked Nov 17 · Scenario · Senior

  89. How would you evaluate whether an agent that classifies pipeline failures is accurate, and how should it handle a real bug mislabeled as flaky?

    Asked once · last asked Nov 17 · Scenario · Senior

  90. How can an application detect that an LLM has not answered a user's question adequately?

    Asked once · last asked Nov 17 · Scenario · Senior

  91. How would you evaluate whether an LLM response is grounded in retrieved context and avoids unsupported claims?

    Asked once · last asked Nov 18 · Scenario · Senior

  92. How do you typically review or audit the AI's output before finalizing your work?

    Asked once · last asked Nov 18 · Scenario · Mid-level

  93. What would a failing response look like compared to a strong one, and how would you score it?

    Asked once · last asked Nov 20 · Scenario · Mid-level

  94. How would you make sure the agent shows real triage reasoning?

    Asked once · last asked Nov 20 · Scenario · Mid-level

  95. How do you balance task complexity with clarity so the agent test stays unambiguous without oversimplifying evidence-based judgment?

    Asked once · last asked Nov 20 · Scenario · Mid-level

  96. How would you ensure an agent returns correct responses without manually validating every output?

    Asked once · last asked Nov 20 · Design · Senior

  97. How would you measure the accuracy of an AI agent's responses?

    Asked once · last asked Nov 20 · Scenario · Senior

  98. How do you ensure the accuracy and quality of data labeling for complex tasks such as LLM evaluations or code-generation assessments?

    Asked once · last asked Nov 21 · Behavioral · Mid-level

  99. How would you test an AI chatbot, and what tools could you use?

    Asked once · last asked Jul 30 · Tooling · Mid-level

  100. What final step would you take before deploying a ticket-handling agent to users?

    Asked once · last asked Aug 7 · Scenario · Mid-level

  101. What should an evaluator be able to assess from a well-designed AI support triage task?

    Asked once · last asked Sep 22 · Concept · Mid-level

  102. What is the hardest part of keeping a log-debugging task design stable over time?

    Asked once · last asked Oct 17 · Scenario · Senior

  103. What would you include in a task-validation harness?

    Asked once · last asked Nov 6 · Tooling · Senior

    Other ways it was asked
    • “What did your task-validation harness include?”
    • “How did your task-validation harness differ from Claude Code's default harness?”
  104. Could it be detected if my child used AI to develop the article?

    Asked once · last asked Oct 26 · Case

  105. How would you validate an LLM-based document intelligence system?

    Asked once · last asked Oct 27 · Scenario · Mid-level

  106. How do you recognize whether text was written by an AI agent or a human?

    Asked once · last asked Nov 6 · Concept · Mid-level

  107. What is one common, simple mistake you see in AI outputs, such as incorrect math, dates, or entities, and what specific change would you suggest to fix it?

    Asked once · last asked Nov 7 · Scenario · Mid-level

  108. What is the most important thing you check before completing an AI output evaluation task?

    Asked once · last asked Nov 14 · Scenario · Junior

  109. What is the latest development in investigating LLM failures that you have had to learn about?

    Asked once · last asked Nov 15 · Concept · Senior

  110. How would you decide whether to stop evaluating an LLM after two unsuccessful prompts or try a third?

    Asked once · last asked Nov 17 · Scenario · Mid-level

    Other ways it was asked
    • “How would you decide whether to stop evaluating Claude Haiku after two unsuccessful prompts or try a third?”
  111. How many unsuccessful prompt attempts should you allow before concluding that an LLM has failed the task?

    Asked once · last asked Nov 17 · Scenario · Mid-level

  112. How do you track human review rates and incorrect results when evaluating agent-generated tasks?

    Asked once · last asked Nov 17 · Scenario · Mid-level

Page 1 · Page 2

More Generative AI and LLM interview questions

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Generative AI and LLM interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

440 interviews, 243 candidates, Feb 2026 to Nov 2026. Updated every month with new interviews.