197 LLM Evaluation and Benchmarking Interview Questions

All 197 questions

How would you evaluate an AI agent's behavior and correctness, and which metrics would you use?
Asked in 7 of 291 interviews · last asked Oct 16 · Concept · Senior
Show answer
I would test the agent on realistic tasks with clear success rules, including difficult and unsafe requests. I would check both the final result and its actions, such as tool calls, using automated checks and human review. I would track task success, answer correctness, tool errors, safety violations, response time, and cost, and repeat tests to check reliability.
- Check whether the task was completed correctly, not just whether the answer sounds right.
- Review tool choices, arguments, and actions for correctness and permission limits.
- Repeat tests to measure consistency; include tool failures and unexpected inputs.
- Measure speed and cost alongside quality and safety.
What the interviewer checks: The interviewer wants to see whether you can evaluate both outcomes and actions with clear, repeatable tests.
Other ways it was asked
- “How do you evaluate an AI agent?”
- “Which metrics would you use to measure an AI agent's effectiveness?”
- “Which metrics would you use to evaluate an AI agent?”
How would you evaluate an LLM application?
Asked in 7 of 291 interviews · last asked Oct 6 · Scenario · Mid-level
Show answer
I would evaluate the application against clear task goals using a held-out test set of realistic inputs. I would combine automated checks with human review to measure correctness, relevance, safety, latency, and cost. Before release, I would compare results with a baseline, then monitor failures and user outcomes in production.
- Test the full application, including retrieval and tools, not just the model.
- Include common requests, edge cases, and attempts to bypass safety rules.
- Check factual claims against trusted sources; validate LLM-based judges against human ratings.
- Run regression tests after changes to prompts, models, tools, or data.
What the interviewer checks: The interviewer is checking whether you can turn product goals into reliable tests and track production quality.
Other ways it was asked
- “How would you evaluate an LLM system?”
- “What parameters or techniques would you use to evaluate an LLM application?”
- “How would you design and implement an LLM application?”
When evaluating AI-generated legal responses, how do you assess their accuracy and quality?
Asked in 2 of 291 interviews · last asked Oct 14 · Scenario · Senior
Other ways it was asked
- “What are some key factors you consider when assessing the quality of AI-generated responses?”
- “When you are evaluating those AI-generated legal responses, how do you assess their accuracy and quality?”
When designing a benchmark for an AI agent, what are the essential components of a good rubric?
Asked in 2 of 291 interviews · last asked Aug 29 · Scenario · Senior
Other ways it was asked
- “When you design an evaluation rubric for an AI agent, what are the minimum required rubric components to make scoring consistent and auditable?”
When you're offering a deliberately hard Q&A, how do you choose the one ambiguity to preserve to make it challenging while eliminating all other ambiguity that could make grading subjective?
Asked in 2 of 291 interviews · last asked Oct 23 · Scenario · Senior
Other ways it was asked
- “When you design a deliberately challenging question-and-answer pair, how do you preserve one intentional ambiguity while removing ambiguity that could make grading subjective?”
How do LLM evaluation frameworks calculate scores, and who or what assigns them?
Asked once · last asked Jul 30 · Concept · Mid-level
Other ways it was asked
- “What results does an LLM evaluation produce, and how are its scores calculated?”
- “What scores or score mappings can LLM evaluation frameworks produce?”
How would you test an LLM output when the acceptance criteria and expected behavior are known?
Asked once · last asked Jul 30 · Scenario · Mid-level
How would you compare two videos against a prompt about a golden retriever catching a red Frisbee on the beach at sunset and justify which video follows it more closely?
Asked once · last asked Oct 17 · Scenario · Junior
Other ways it was asked
- “How would you compare two videos against a prompt and justify which one follows it more closely?”
How would you assess whether an LLM is performing well, and what metrics or parameters would you use to validate it?
Asked once · last asked Feb 5 · Scenario · Mid-level
How would you evaluate an AI model’s response to a prompt?
Asked once · last asked Feb 13 · Scenario · Mid-level
How do you design evaluation rubrics or scoring mechanisms for model outputs, and which dimensions do you prioritize?
Asked once · last asked Mar 5 · Concept · Mid-level
Can you describe a challenging case you encountered while evaluating an AI model’s outputs, particularly when spotting hallucinations or checking logical consistency? What issue did you identify, and what steps did you take to address it?
Asked once · last asked Apr 22 · Behavioral · Mid-level
Can you share a specific example from your AI evaluation work where you found a significant issue or inconsistency? How did you pinpoint it, and what impact did it have on the broader project?
Asked once · last asked Apr 22 · Behavioral · Mid-level
When evaluating the logical consistency of an LLM’s output, how do you determine whether its reasoning holds up?
Asked once · last asked Apr 27 · Scenario · Senior
What common challenges do you face when designing evaluation frameworks for AI agents, and how do you address them?
Asked once · last asked Apr 27 · Scenario · Senior
If you had to choose between evaluating a model that produces highly fluent outputs but is sometimes factually inaccurate and one that is less fluent but consistently grounded in facts, which would you prioritize for further development, and why?
Asked once · last asked Apr 27 · Scenario · Senior
How would you balance fluency and factual accuracy in applications such as customer service or educational tools?
Asked once · last asked Apr 27 · Scenario · Senior
How would you evaluate whether an LLM's answer to a question about a document, such as a loan eligibility decision, is correct?
Asked once · last asked May 12 · Scenario · Senior
How would you address hallucinations in an LLM evaluation framework?
Asked once · last asked May 12 · Scenario · Senior
What should validation of an AI-generated advertiser report catch: hallucinated metrics, factually incorrect insights, or an inappropriate tone?
Asked once · last asked May 16 · Concept · Mid-level
How would you validate AI-generated reports and handle failures? Would you check required headers, use a parser, or ask the model to validate its output, and would you block a failed report, flag it, or fall back to a static template?
Asked once · last asked May 16 · Tooling · Senior
How do you evaluate and continuously monitor voice and policy compliance in production? What metrics, sampling strategy, and feedback loop do you use to catch drift over time?
Asked once · last asked May 28 · Scenario · Senior
When you roll out a new prompt or policy version, how do you design the evaluation to avoid regressions? What offline test set, acceptance criteria, and rollout strategy—for example, a canary or A/B test—do you use?
Asked once · last asked May 28 · Scenario · Senior
When you turn fragmented incident inputs—logs, chats, dashboards, and tickets—into an agent evaluation workflow, what core steps and artifacts do you require before judging the agent's performance?
Asked once · last asked Jun 15 · Scenario · Senior
How would you define dependencies, blockers, and unknowns in an agent evaluation workflow so the agent can proceed safely without guessing while still making progress?
Asked once · last asked Jun 15 · Scenario · Senior
How would you score and weight an agent's performance across timeline accuracy, dependency handling, escalation behavior, and recommendation quality without over-optimizing for speed?
Asked once · last asked Jun 15 · Scenario · Senior
How do you distinguish a nuanced evolution of an argument from a logical contradiction when analyzing a response?
Asked once · last asked Jun 22 · Concept · Senior
Suppose Goodhart’s law takes effect and agents start optimizing for rubric artifacts, such as formatting and verbosity, without real task success. How would you redesign the rubric and success criteria to resist gaming?
Asked once · last asked Jun 30 · Scenario · Senior
Given a ten-minute observation session of an AI agent, what exact template section headers would you use to document behavior and outcomes, and what one or two sentences would you include per section?
Asked once · last asked Jun 30 · Mid-level
If two observers disagree about whether an agent handled noise well, what reconciliation process and final write-up format would you use to resolve the discrepancy without losing minority evidence?
Asked once · last asked Jun 30 · Scenario · Senior
Other ways it was asked
- “If two observers have the right to disagree about whether an agent handled noise well, what reconciliation process and final write-up format would you use to resolve the discrepancy without losing minority evidence?”
Imagine you're given conflicting Slack threads, a meeting transcript, and emails. How would you write an evaluation prompt for an AI to synthesize them into a template, and what five rubric criteria would you score?
Asked once · last asked Jul 9 · Scenario · Senior
How would you design your rubric to distinguish a correct but incomplete response from one that is incorrect because of conflict resolution, especially when sources disagree on an owner or due date?
Asked once · last asked Jul 9 · Scenario · Senior
When AI gives you an answer, how do you decide whether to trust it or not?
Asked once · last asked Jul 16 · Scenario · Mid-level
How would you measure instruction-following quality for a prompt that asks the model to always respond in bullet points?
Asked once · last asked Jul 16 · Scenario · Mid-level
How would you build a test suite for a chatbot that answers questions about a company's product?
Asked once · last asked Jul 16 · Scenario · Mid-level
What is a feedback loop in an AI product, and how can production feedback be used to improve test cases?
Asked once · last asked Jul 16 · Concept · Mid-level
How do you measure an AI agent's error rate with human-in-the-loop review, and use the results to reduce hallucinations?
Asked once · last asked Jul 16 · Scenario · Senior
How do you automate testing or validate nondeterministic responses from an LLM?
Asked once · last asked Jul 16 · Scenario · Senior
When reviewing AI-generated responses, what criteria would you use to classify a response as acceptable, inaccurate, or unusable?
Asked once · last asked Jul 23 · Scenario · Mid-level
Other ways it was asked
- “When you reviewed AI-generated responses, what made a response acceptable, inaccurate, or unusable?”
How would you ensure that shipping AI agents improves the relevant success metrics?
Asked once · last asked Jul 25 · Scenario · Mid-level
Other ways it was asked
- “How do you balance the requirement that metrics improve with shipping AI agents?”
What constraints should you account for when testing an AI agent?
Asked once · last asked Jul 30 · Scenario · Senior
How would you test an AI chatbot using TestNG and TestRail?
Asked once · last asked Jul 30 · Tooling · Mid-level
How would you detect when a real, correctly tagged ad citation does not actually support the claim made about it?
Asked once · last asked Aug 4 · Scenario · Senior
How would you design a single-prompt reasoning stress test to reveal whether a model is overgeneralizing or following the guideline hierarchy?
Asked once · last asked Aug 4 · Scenario · Senior
How would you design a prompt for a reasoning model to evaluate an annotator's labeling decision and return a verdict with a falsifiable, auditor-ready rationale?
Asked once · last asked Aug 4 · Scenario · Senior
How would you define a quantitative rubric for evaluating a model's grounding, correctness, and calibration, including penalties for missing or incorrect citations and unwarranted confidence?
Asked once · last asked Aug 4 · Scenario · Senior
Other ways it was asked
- “How would you define a quantitative rubric for evaluating a model's grounding, correctness, and calibration, including penalties for missing or incorrect citations and overconfident uncertainty?”
How would you calibrate an LLM-as-a-judge compliance gate to avoid both passing noncompliant outputs and rejecting valid ones?
Asked once · last asked Aug 5 · Scenario · Senior
How would you benchmark and compare LLMs to identify the best fit for a particular use case?
Asked once · last asked Aug 5 · Scenario · Senior
What would you do if the success metric for a ticket-handling agent were to reduce the number of tickets in the human queue?
Asked once · last asked Aug 7 · Scenario · Senior
How could a metric focused on reducing tickets in the human queue be gamed?
Asked once · last asked Aug 7 · Scenario · Senior
How would you approach writing and maintaining tests for an AI agent?
Asked once · last asked Aug 7 · Tooling · Mid-level
When comparing two AI responses, what makes one clearly higher quality than the other in Bengali, especially in terms of fluency, fidelity, structure, safety, and tone?
Asked once · last asked Aug 10 · Concept · Senior
Could you share examples of common errors in AI-generated Bengali outputs, such as literal translation, wrong register, or awkward collocations, and explain how you correct them?
Asked once · last asked Aug 10 · Concept · Mid-level
When comparing two AI answers about mixing concentrated hydrochloric acid with an oxidizing reagent in a calcium carbonate reaction, what rubric criteria and scoring approach would you use?
Asked once · last asked Aug 17 · Scenario · Senior
How would you score an AI answer that is numerically correct but gives poor safety reasoning against a clear, safe answer that contains a stoichiometric error?
Asked once · last asked Aug 17 · Scenario · Senior
When you evaluate an AI response with a rubric, what core dimensions do you score, and how do you define clear pass/fail thresholds for each?
Asked once · last asked Aug 17 · Concept · Senior
Suppose the rubric's accuracy dimension yields low agreement for open-ended answers. How would you refine the metric or rubric to improve consistency without overpenalizing helpful responses?
Asked once · last asked Aug 17 · Scenario · Senior
When a model response includes five factual claims across finance and health care, how do you prioritize which claims to verify first under a tight time budget?
Asked once · last asked Aug 17 · Scenario · Senior
How do you distinguish a missing citation from an unverifiable claim, and how should each affect accuracy versus completeness ratings?
Asked once · last asked Aug 17 · Concept · Senior
How would you assess an agentic AI solution before and after deployment, including its evaluation harness, data integration challenges, metrics, and alignment with business goals?
Asked once · last asked Aug 19 · Design · Senior
What should the software testing life cycle for agentic AI systems look like, especially in regulated insurance environments?
Asked once · last asked Aug 19 · Scenario · Senior
How would you evaluate agents pragmatically when models, harnesses, and MCP tools are changing constantly?
Asked once · last asked Aug 20 · Scenario · Senior
How would you use an evaluation framework to compare model versions or configurations side by side in an agent harness?
Asked once · last asked Aug 20 · Scenario · Senior
When iterating on an LLM feature, how do you set up an experiment plan that separates prompt changes from model changes, and what metrics determine whether to promote a change?
Asked once · last asked Aug 24 · Scenario · Senior
Can you create an original senior-level adversarial question and gold-standard answer that targets a specific reasoning failure mode, such as base-rate neglect, Simpson's paradox, or scope ambiguity?
Asked once · last asked Aug 25 · Senior
How would you evaluate autonomous agents, given that execution is nondeterministic?
Asked once · last asked Aug 25 · Scenario · Senior
What specific metrics beyond the classic NLP metrics would you set up to measure multi-step planning accuracy?
Asked once · last asked Aug 25 · Scenario · Senior
What is the difference between model-level metrics and system-level agent evaluation?
Asked once · last asked Aug 25 · Concept · Mid-level
How do you determine the stronger response in side-by-side comparisons? Walk me through the specific criteria you use.
Asked once · last asked Aug 25 · Scenario · Mid-level
How would you build a programmatic evaluation harness using benchmark test suites or deterministic test fixtures to measure an agent's accuracy before deploying updates?
Asked once · last asked Aug 26 · Design · Senior
Other ways it was asked
- “How would you build a programmatic evaluation harness using benchmark test suites or deterministic text fixtures to measure an agent's accuracy before deploying updates?”
How would you convert correctness, completeness, and constraint-following into a weighted scoring scheme with clear thresholds while minimizing rubric gaming by the agent?
Asked once · last asked Aug 29 · Scenario · Senior
How would you design a self-contained evaluation task for an agent doing a practical computer workflow, such as cleaning a CSV and producing a summary table, including inputs, oracles or validators, and a mostly automatable rubric?
Asked once · last asked Aug 29 · Scenario · Senior
What inconsistencies would you flag first, and why, if a report claims a 2% accuracy improvement and fewer hallucinations, but the appendix shows different denominators, mixed seeds, and a metric definition change mid-run?
Asked once · last asked Aug 29 · Scenario · Senior
What are three non-obvious failure modes you would add to an evaluation rubric?
Asked once · last asked Aug 29 · Scenario · Senior
How would you design a prompt and rubric package to reliably test an LLM's multitask, step-by-step reasoning?
Asked once · last asked Sep 3 · Scenario · Senior
How do you design edge cases that distinguish a model truly following instructions from one pattern matching the schema without relying on hidden answers or shortcuts?
Asked once · last asked Sep 3 · Scenario · Senior
When authoring a suite of 30 reasoning tasks, how do you cover instruction hierarchy, implicit assumptions, and brittle reasoning while keeping the suite minimal and nonredundant?
Asked once · last asked Sep 3 · Scenario · Senior
Tell me about a time an AI-generated trading or financial explanation sounded plausible but was logically flawed. What was wrong with its reasoning, and how did you verify your assessment?
Asked once · last asked Sep 6 · Behavioral · Mid-level
How would you turn a flawed AI-generated financial response into evaluator feedback? Identify the issue, the criterion it failed, and the correction you would recommend.
Asked once · last asked Sep 6 · Scenario · Mid-level
What criteria would you use to compare two AI-generated responses, and how would you choose the better one and write a corrected answer in an SFT-style format?
Asked once · last asked Sep 6 · Scenario · Mid-level
An AI report recommends buying gold at $2,000, with a $2,100 target and a $1,900 stop loss. It claims a 5% expected return and 5% risk. What would you verify, and are both calculations correct?
Asked once · last asked Sep 6 · Case · Mid-level
If an AI report recommends buying gold based only on a 5% target return while ignoring the stop loss and market context, would you approve the recommendation? Why or why not?
Asked once · last asked Sep 6 · Scenario · Mid-level
How would you author a benchmark task that tests an agent's ability to produce a three-source reconciliation report, including the exact injected error cases and the required scoring rubric?
Asked once · last asked Sep 7 · Scenario · Senior
How would you design harder edge cases, like many-to-many joins, conflicting authoritative sources by field, and near-duplicate IDs, and specify deterministic resolution rules for grading?
Asked once · last asked Sep 7 · Scenario · Senior
How would you build an evaluation framework for an LLM that generates customer call summaries, and what metrics would you track?
Asked once · last asked Sep 9 · Design · Senior
In an LLM RLHF review program, how would you design a rubric to score helpfulness, factuality, and harmlessness with minimal overlap?
Asked once · last asked Sep 11 · Design · Senior
Other ways it was asked
- “In an RLHF review rubric, how would you resolve conflicts between helpfulness, factuality, and harmlessness scores?”
What is the most common failure mode you see in AI-generated code, and how do you catch it quickly?
Asked once · last asked Sep 15 · Scenario · Mid-level
Other ways it was asked
- “What is the most common failure mode you see in its coding outputs, and how do you catch it quickly?”
When ChatGPT and Claude disagree on an interpretation, how do you decide the final wording?
Asked once · last asked Sep 15 · Scenario · Mid-level
What does high-quality AI-generated output look like in your domain, and what three to five attributes must be true before you submit it?
Asked once · last asked Sep 15 · Concept · Mid-level
How would you document a specific error in an AI-generated response, including the sentence or claim that was incomplete or changed and the detail it missed or altered?
Asked once · last asked Sep 15 · Scenario · Mid-level
How would you characterize the underlying failure in an AI-generated Python response, and what written feedback would you give to help prevent the same mistake?
Asked once · last asked Sep 15 · Scenario · Mid-level
When two LLM drafts meet the rubric but use different framing, emphasis, or risk messaging, what tie-breaker process do you use?
Asked once · last asked Sep 16 · Scenario · Mid-level
Write an unambiguous benchmark prompt for a model to infer causality limits from a pre/post line chart with a dashed policy change marker. Include the chart description, required reasoning steps, and an objectively gradable answer rubric.
Asked once · last asked Sep 18 · Senior
Other ways it was asked
- “Revise a benchmark prompt for evaluating causal-inference limits from a pre/post line chart with a dashed policy-change marker. Specify at least six monthly values before and after the change, and define explicit pass/fail criteria for each two-point rubric bucket.”
If an answer were fluent but factually wrong, how would you evaluate it and write concise feedback explaining the error?
Asked once · last asked Sep 18 · Scenario · Mid-level
How do you design prompts or tasks to expose subtle errors or missing context in ChatGPT's outputs?
Asked once · last asked Sep 21 · Tooling · Mid-level
How would you evaluate whether an AI chatbot's answers and behavior are reliable enough for real users?
Asked once · last asked Sep 21 · Scenario · Senior
Other ways it was asked
- “How did you evaluate whether an AI chatbot's answers and behavior were reliable enough for real users?”
How would you validate a real estate chatbot before and after launch?
Asked once · last asked Sep 21 · Behavioral · Mid-level
Other ways it was asked
- “How did you validate a real estate chatbot before and after launch?”
How would you make sure the agent's reasoning is tested rather than just the output, especially when the facts are incomplete or ambiguous?
Asked once · last asked Sep 22 · Scenario · Senior
How do you balance task complexity with clarity so that it tests the AI yet the correct answer is unambiguous?
Asked once · last asked Sep 22 · Scenario · Senior
When designing a benchmark prompt for a highly skewed calibration plot, what numeric outputs would you require—such as per-bin counts, mean predicted probabilities, observed rates, and confidence-interval type—to make scoring objective?
Asked once · last asked Sep 23 · Scenario · Senior
Write a benchmark prompt for a reliability diagram with eight quantile bins and per-bin counts that requests ECE weighted by n and MCE, including the exact formulas and rounding requirements.
Asked once · last asked Sep 23 · Scenario · Senior
How would you define p_k as the mean predicted probability and o_k as the empirical event rate for bin k in an ECE/MCE benchmark prompt to avoid ambiguity?
Asked once · last asked Sep 23 · Scenario · Senior
How should a benchmark prompt specify when rounding occurs for input values, per-bin gaps, and final ECE and MCE values so the rule is applied consistently?
Asked once · last asked Sep 23 · Scenario · Senior
How do you create rubrics for evaluating AI outputs?
Asked once · last asked Sep 25 · Scenario · Senior
How would you identify errors in ChatGPT's variance analysis beyond checking whether the final numbers are wrong?
Asked once · last asked Sep 25 · Scenario · Mid-level
When comparing two AI-written finance explanations, what criteria do you use to pick the better one in terms of assumptions, math, clarity, and decision value?
Asked once · last asked Sep 25 · Case · Mid-level
How would you design a 35-plus item grading rubric for an AI agent validating a client CSV against a data dictionary, covering market judgment, compliance sensitivity, client communication, correct decisions, and failure modes?
Asked once · last asked Sep 26 · Scenario
How would you set up a single run to elicit both compliance handling and evidence-backed field validation, including the inputs, constraints, agent actions, and required artifacts?
Asked once · last asked Sep 26 · Scenario · Senior
Imagine your team introduces an AI assistant for financial-crime transaction batch-failure reconciliations. How would you validate its answers before engineers or business users rely on them?
Asked once · last asked Sep 29 · Scenario · Senior
If an AI assistant says a mismatch was caused by duplicate customer records with two customer IDs, how would you test whether that explanation is correct?
Asked once · last asked Sep 29 · Scenario · Senior
When evaluating AI responses and noticing a significant error, how do you document your findings and suggest improvements?
Asked once · last asked Oct 1 · Scenario · Mid-level
In an evaluation workflow, how would you design a lightweight, time-boxed verification protocol so annotators can consistently distinguish false from unverified claims without stalling throughput?
Asked once · last asked Oct 2 · Scenario · Senior
In a 5,000-item batch, how would you detect and annotate AI slop that is factually correct but low-value, generic, and content-free without over-penalizing legitimate executive writing?
Asked once · last asked Oct 2 · Scenario · Senior
When you use an AI tool like Hebbia, Rogo, or Claude for Financial Services to draft a variance memo after earnings, what specific validation steps do you run before you trust the numbers and citations?
Asked once · last asked Oct 3 · Tooling · Senior
In a rubric-based factual or semantic accuracy review, how do you handle content that’s plausible but not verifiable from the provided sources? What score and feedback would you give?
Asked once · last asked Oct 3 · Scenario · Senior
If a response is clear and well structured but contains one critical factual error, how do you weigh clarity versus accuracy in your rubric scoring and written feedback?
Asked once · last asked Oct 3 · Scenario · Senior
How do you decide whether an issue is critical versus a minor factual inaccuracy, especially when domain context is incomplete or sources conflict?
Asked once · last asked Oct 3 · Scenario · Senior
How would you design a short calibration exercise for evaluators to align on hallucination risk scoring, and what would you measure to confirm inter-rater reliability?
Asked once · last asked Oct 3 · Scenario · Senior
In an annotation tool, how would you write a single high-quality note for hallucination-like inaccuracy that is actionable without injecting your own external assumptions?
Asked once · last asked Oct 4 · Tooling · Senior
How do you calibrate reviewers to apply the unsupported versus false versus unverifiable distinction consistently across a large data set?
Asked once · last asked Oct 4 · Scenario · Senior
How would you structure the test matrix across agents, versions, settings, and seeds to make comparisons fair and reproducible?
Asked once · last asked Oct 5 · Scenario · Senior
If the same prompt produces different failures across five runs, how would you document those failures and assess the agent's reliability?
Asked once · last asked Oct 5 · Scenario · Senior
How would you distinguish model nondeterminism from test harness flakiness, network rate limits, or tool timeouts when failures occur intermittently across agents?
Asked once · last asked Oct 5 · Scenario · Senior
Imagine you're reviewing a golden prompt file and notice inconsistencies in spelling, whitespace, punctuation, and casing. What checklist would you apply before running the evaluation?
Asked once · last asked Oct 5 · Scenario · Mid-level
If a specification requires exactly three bullets under 80 words but the prompt also asks for two bullets of key risks, how would you resolve and document the conflict before running the evaluation?
Asked once · last asked Oct 5 · Scenario · Senior
When a style guide changes mid-project, how do you version and communicate the change and report it in evaluations to keep results comparable?
Asked once · last asked Oct 5 · Scenario · Senior
How do you assess data quality, compare results, and establish ground truth for an LLM application?
Asked once · last asked Oct 6 · Scenario · Mid-level
How does LLM-as-a-judge evaluation work, and when is it appropriate or inappropriate?
Asked once · last asked Oct 6 · Concept · Mid-level
Other ways it was asked
- “How does LLM-as-a-judge evaluation work, and when would you use it?”
- “Where is LLM as a judge a good fit, and where is it not a good fit?”
In a 200-page report, you're asked to provide written feedback on an AI-generated draft. What specific criteria do you review, and how do you present actionable fixes to the author?
Asked once · last asked Oct 9 · Scenario · Mid-level
In an LLM evaluation workflow, how would you define a rubric and scoring scale to compare two model outputs for the same prompt while ensuring inter-rater reliability?
Asked once · last asked Oct 9 · Scenario · Senior
When two raters materially disagree on a scoring dimension, such as assigning accuracy scores of two and five, what reconciliation process do you use and what artifacts do you record?
Asked once · last asked Oct 9 · Scenario · Senior
How would you design a golden-set evaluation prompt suite and reference expectations to detect regressions across model versions while minimizing overfitting to the test set?
Asked once · last asked Oct 9 · Scenario · Senior
How do you evaluate instruction-following versus helpfulness when they conflict, such as when a user asks for disallowed content, and how do you reflect that trade-off in your scoring?
Asked once · last asked Oct 9 · Scenario · Senior
In a prompt-driven workflow with multi-turn conversations, how would you define and measure conversation-level success beyond single-turn rubric scores?
Asked once · last asked Oct 9 · Concept · Senior
What concrete failures would you look for when using LLMs to generate test ideas?
Asked once · last asked Oct 9 · Behavioral · Senior
Other ways it was asked
- “When you tried using LLMs to generate test ideas, what concrete failures did you see most?”
How would you set up evaluations for AI agents to detect regressions when changing the model or prompts?
Asked once · last asked Oct 13 · Scenario · Senior
How do you evaluate a written AI response against a rubric, step by step?
Asked once · last asked Oct 14 · Behavioral · Mid-level
How would you validate a vision-language model's interpretation of engineering drawings, and when would you use a manually curated gold set?
Asked once · last asked Oct 14 · Scenario · Senior
Walk me through how you personally handle a language-focused content review, step by step, when you sit down to evaluate a piece of AI-generated content.
Asked once · last asked Oct 14 · Scenario · Mid-level
What specific jurisdictions or legal frameworks do you typically measure the model’s output against?
Asked once · last asked Oct 14 · Concept · Mid-level
When assessing how an LLM handles contract formation under Kenyan law, how do you evaluate its understanding of the distinction between an offer and an invitation to treat?
Asked once · last asked Oct 14 · Scenario · Senior
Could you walk me through a specific Kenyan case or statutory provision that you use to verify if a model is correctly identifying an invitation to treat versus a binding offer?
Asked once · last asked Oct 14 · Case · Senior
Since Boots is English authority, how do you evaluate whether a model correctly explains its status in Kenyan courts? What specific Kenyan statutory provision or case law governs how such persuasive authorities are received?
Asked once · last asked Oct 14 · Case · Senior
What specific Kenyan statute actually governs the reception of English common law and doctrines of equity in Kenya, and how do you test if a model applies it correctly?
Asked once · last asked Oct 14 · Case · Senior
How would you test whether a model can distinguish a director’s subjective good-faith duty to promote the company’s success from an objective standard when the decision has disastrous commercial consequences?
Asked once · last asked Oct 14 · Scenario · Senior
How do you test a model’s ability to identify when an asset sale triggers the requirement for a special resolution under Kenyan corporate law?
Asked once · last asked Oct 14 · Scenario · Senior
If a company sells its main undertaking, how do you test whether the model correctly determines if a special resolution is required at the parent company level?
Asked once · last asked Oct 14 · Scenario · Senior
How do you test a model’s ability to determine whether shareholders in a private company have pre-emption rights on a new share allotment under the Companies Act 2015?
Asked once · last asked Oct 14 · Scenario · Senior
Other ways it was asked
- “How do you test a model’s ability to determine if a private company has pre-emption rights on a new allotment under the Companies Act 2015?”
If a company’s articles are silent but its shareholders have a conflicting oral agreement, how do you test whether the model correctly determines which rule takes precedence for a new allotment?
Asked once · last asked Oct 14 · Scenario · Senior
How do you test a model’s ability to distinguish between unfair prejudice and a mere disagreement over commercial policy under the Companies Act 2015?
Asked once · last asked Oct 14 · Scenario · Senior
If majority shareholders offer to buy out a minority shareholder at a price set by an independent valuer, how do you test whether the model correctly assesses if that offer defeats an unfair prejudice claim?
Asked once · last asked Oct 14 · Scenario · Senior
How do you test a model’s ability to distinguish between a general partnership and a non-partnership joint venture under Kenyan law?
Asked once · last asked Oct 14 · Scenario · Senior
If parties sign an agreement stating they are not partners but share net profits in joint bank accounts, how do you test whether the model correctly identifies their legal status?
Asked once · last asked Oct 14 · Case · Senior
How do you test a model’s ability to identify the correct statutory priority of payments in a Kenyan insolvent liquidation?
Asked once · last asked Oct 14 · Scenario · Senior
How do you evaluate AI-generated responses for technical accuracy, practicality, and compliance with real-world requirements, and how do you refine them?
Asked once · last asked Oct 14 · Scenario · Senior
How would you identify gaps in an AI-generated quality control plan for a financial trading platform and account for real-time market volatility's impact on trade settlement accuracy?
Asked once · last asked Oct 14 · Scenario · Senior
How do you evaluate AI-generated code, including the tools or models and rubric steps you use?
Asked once · last asked Oct 15 · Scenario · Mid-level
Can you describe a comparison of two local AI models, naming both and explaining a specific task where one was more reliable than the other and what you observed?
Asked once · last asked Oct 15 · Behavioral · Mid-level
Other ways it was asked
- “Can you name the two local AI models you compared and describe one specific task where one was more reliable than the other, including what you observed?”
When comparing Claude, ChatGPT, and Gemini on the same task, what practical criteria do you use to decide which one to trust or proceed with?
Asked once · last asked Oct 17 · Concept · Mid-level
When evaluating a video against a prompt, what evidence would you cite to support a match?
Asked once · last asked Oct 17 · Scenario · Junior
What mismatches would you identify when assessing a video against a prompt, including mismatches in lighting?
Asked once · last asked Oct 17 · Scenario · Junior
Design a concrete log-debugging task for a production issue. What scenario and inputs would you provide, and what reasoning would be needed to solve it?
Asked once · last asked Oct 17 · Scenario · Senior
How would you distinguish a correct diagnosis from a superficial correlation in a log-debugging training task?
Asked once · last asked Oct 17 · Scenario · Senior
How would you keep a log-debugging training task challenging as agents improve, rather than allowing it to become trivially solvable?
Asked once · last asked Oct 17 · Scenario · Senior
Can you describe one anti-gaming tactic you would include to prevent keyword matching or memorization?
Asked once · last asked Oct 17 · Scenario · Senior
What would you do if an agent gives a confident explanation but misses a related symptom in the fix?
Asked once · last asked Oct 17 · Scenario · Senior
How would you keep the evaluation fair across different versions of the same training task?
Asked once · last asked Oct 17 · Scenario · Senior
How would you design a lightweight scoring guide with three to five levels to evaluate whether data-generated answers follow a policy while minimizing reviewer subjectivity?
Asked once · last asked Oct 20 · Scenario · Senior
What specific metrics or tools do you use to detect hallucinations and model drift?
Asked once · last asked Oct 20 · Tooling · Senior
How would you validate LLM outputs beyond simply checking whether the response looks correct?
Asked once · last asked Oct 21 · Behavioral · Mid-level
Other ways it was asked
- “How have you validated LLM outputs beyond simply checking whether the response looks correct?”
What metrics or frameworks would you use to measure prompt consistency, hallucination rate, and output quality?
Asked once · last asked Oct 21 · Scenario · Mid-level
How would you design a single question and answer pair that tests whether someone can detect and correct subtly ill-posed questions with ambiguous units, mismatched time windows, or confounded variables without letting them rewrite the question?
Asked once · last asked Oct 21 · Scenario · Senior
How would you construct an adversarial question and answer pair that probes causal reasoning under confounding, such as whether batteries improve PV availability, while requiring the candidate to separate correlation, mechanism, and evidence quality in a two to three minute verbal answer?
Asked once · last asked Oct 21 · Scenario · Senior
How would you make an evaluation prompt adversarial by embedding one subtle but realistic trap, such as selection bias, Simpson’s paradox, or label leakage, while keeping it fair and verbally answerable?
Asked once · last asked Oct 21 · Scenario · Senior
Other ways it was asked
- “How would you make the prompt adversarial by embedding one subtle but realistic trap, such as selection bias, Simpson’s paradox, or label leakage, while keeping it fair and verbally answerable?”
How would you score the model’s response with a four to six item rubric, pass or fail per item, that distinguishes good refusal, partial compliance, and policy theater without relying on hidden reference answers?
Asked once · last asked Oct 21 · Scenario · Senior
When you evaluate AI model outputs for coherence and contextual alignment, how do you ensure the quality and consistency of your assessments?
Asked once · last asked Oct 21 · Scenario · Mid-level
How do you make adjustments if the rubric you’re using doesn’t fit well with the model outputs you’re evaluating?
Asked once · last asked Oct 21 · Scenario · Mid-level
Imagine you're building an evaluation question to detect LLM shortcutting. What specific constraints would you add so the item requires multi-step reasoning with intermediate commitments, not pattern matching to a familiar template?
Asked once · last asked Oct 22 · Scenario · Senior
Which two or three quantitative signals would you use to identify an insufficiently challenging evaluation item and decide whether to revise it?
Asked once · last asked Oct 22 · Scenario · Senior
Other ways it was asked
- “How would you operationally measure when an item is not sufficiently challenging for an evaluation set? What two or three quantitative signals would trigger you to iterate an item?”
When reviewer feedback says an item is ambiguous, your diagnostics suggest it's challenging. What's your step-by-step process to adjudicate and decide whether to rewrite, rekey, or keep it?
Asked once · last asked Oct 22 · Scenario · Senior
How do you design an adversarial break test for a final item before shipping to ensure it can't be solved by superficial heuristics, prompt injection, or style instruction following?
Asked once · last asked Oct 22 · Scenario · Senior
How would you test an AI chatbot and build an evaluation dataset for it?
Asked once · last asked Oct 22 · Scenario · Senior
Which metrics would you use to evaluate whether an AI chatbot's answers are correct?
Asked once · last asked Oct 22 · Concept · Senior
What are the benefits and limitations of using golden answers to evaluate an AI chatbot?
Asked once · last asked Oct 22 · Concept · Senior
How would a change to an AI system prompt affect your tests and evaluation dataset?
Asked once · last asked Oct 22 · Scenario · Senior
Which metrics would you use to decide whether an AI chatbot is ready for production?
Asked once · last asked Oct 22 · Scenario · Senior
Write a fully specified, single-sentence Q&A prompt in a non-health domain that uses a multi-step reasoning trap without ambiguity or obscure facts, names two exact document titles and dates or versions, and requires a yes-or-no outcome.
Asked once · last asked Oct 23 · Senior
Other ways it was asked
- “Write a fully specified, single-sentence Q&A prompt in a non-health domain that names two exact documents with dates or versions, includes all necessary facts, requires a yes-or-no decision, and requires verbatim clause quotes.”
Give me one original unambiguous hard question you'd write to test an AI's ability to synthesize across two sources, and include the scoring rubric.
Asked once · last asked Oct 23 · Scenario · Senior
Write a complete question-and-answer pair to test AI synthesis across two sources, including the required source titles, publishers, dates, and a zero-to-four scoring rubric.
Asked once · last asked Oct 23 · Scenario · Senior
Describe one technique you use to make a Q&A genuinely hard for advanced users without relying on obscurity or ambiguous wording. What is the technique, and why does it work?
Asked once · last asked Oct 23 · Behavioral · Senior
Give one concrete non-finance domain and write a high-difficulty Q&A prompt using a technique that tests advanced users without relying on obscurity or ambiguous wording.
Asked once · last asked Oct 23 · Senior
Describe one concrete way you build a multi-step reasoning trap into a Q&A without ambiguity or obscure facts that reliably distinguishes a careful solver from a superficial one. What is the mechanism?
Asked once · last asked Oct 23 · Behavioral · Senior
If you were evaluating an AI model and coherence was inconsistent across datasets, which rubric criteria or analytical approach would you use first to diagnose the issue?
Asked once · last asked Oct 23 · Scenario · Senior
How would you test an AI chatbot, and what tools could you use?
Asked once · last asked Jul 30 · Tooling · Mid-level
What final step would you take before deploying a ticket-handling agent to users?
Asked once · last asked Aug 7 · Scenario · Mid-level
What should an evaluator be able to assess from a well-designed AI support triage task?
Asked once · last asked Sep 22 · Concept · Mid-level
What is the hardest part of keeping a log-debugging task design stable over time?
Asked once · last asked Oct 17 · Scenario · Senior

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Generative AI and LLM interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

291 interviews, 178 candidates, Feb 2026 to Oct 2026. Updated every month with new interviews.

All 197 questions

How would you evaluate an AI agent's behavior and correctness, and which metrics would you use?

How would you evaluate an LLM application?

When evaluating AI-generated legal responses, how do you assess their accuracy and quality?

When designing a benchmark for an AI agent, what are the essential components of a good rubric?

When you're offering a deliberately hard Q&A, how do you choose the one ambiguity to preserve to make it challenging while eliminating all other ambiguity that could make grading subjective?

How do LLM evaluation frameworks calculate scores, and who or what assigns them?

How would you test an LLM output when the acceptance criteria and expected behavior are known?

How would you compare two videos against a prompt about a golden retriever catching a red Frisbee on the beach at sunset and justify which video follows it more closely?

How would you assess whether an LLM is performing well, and what metrics or parameters would you use to validate it?

How would you evaluate an AI model’s response to a prompt?

How do you design evaluation rubrics or scoring mechanisms for model outputs, and which dimensions do you prioritize?

Can you describe a challenging case you encountered while evaluating an AI model’s outputs, particularly when spotting hallucinations or checking logical consistency? What issue did you identify, and what steps did you take to address it?

Can you share a specific example from your AI evaluation work where you found a significant issue or inconsistency? How did you pinpoint it, and what impact did it have on the broader project?

When evaluating the logical consistency of an LLM’s output, how do you determine whether its reasoning holds up?

What common challenges do you face when designing evaluation frameworks for AI agents, and how do you address them?

If you had to choose between evaluating a model that produces highly fluent outputs but is sometimes factually inaccurate and one that is less fluent but consistently grounded in facts, which would you prioritize for further development, and why?

How would you balance fluency and factual accuracy in applications such as customer service or educational tools?

How would you evaluate whether an LLM's answer to a question about a document, such as a loan eligibility decision, is correct?

How would you address hallucinations in an LLM evaluation framework?

What should validation of an AI-generated advertiser report catch: hallucinated metrics, factually incorrect insights, or an inappropriate tone?

How would you validate AI-generated reports and handle failures? Would you check required headers, use a parser, or ask the model to validate its output, and would you block a failed report, flag it, or fall back to a static template?

How do you evaluate and continuously monitor voice and policy compliance in production? What metrics, sampling strategy, and feedback loop do you use to catch drift over time?

When you roll out a new prompt or policy version, how do you design the evaluation to avoid regressions? What offline test set, acceptance criteria, and rollout strategy—for example, a canary or A/B test—do you use?

When you turn fragmented incident inputs—logs, chats, dashboards, and tickets—into an agent evaluation workflow, what core steps and artifacts do you require before judging the agent's performance?

How would you define dependencies, blockers, and unknowns in an agent evaluation workflow so the agent can proceed safely without guessing while still making progress?

How would you score and weight an agent's performance across timeline accuracy, dependency handling, escalation behavior, and recommendation quality without over-optimizing for speed?

How do you distinguish a nuanced evolution of an argument from a logical contradiction when analyzing a response?

Suppose Goodhart’s law takes effect and agents start optimizing for rubric artifacts, such as formatting and verbosity, without real task success. How would you redesign the rubric and success criteria to resist gaming?

Given a ten-minute observation session of an AI agent, what exact template section headers would you use to document behavior and outcomes, and what one or two sentences would you include per section?

If two observers disagree about whether an agent handled noise well, what reconciliation process and final write-up format would you use to resolve the discrepancy without losing minority evidence?

Imagine you're given conflicting Slack threads, a meeting transcript, and emails. How would you write an evaluation prompt for an AI to synthesize them into a template, and what five rubric criteria would you score?

How would you design your rubric to distinguish a correct but incomplete response from one that is incorrect because of conflict resolution, especially when sources disagree on an owner or due date?

When AI gives you an answer, how do you decide whether to trust it or not?

How would you measure instruction-following quality for a prompt that asks the model to always respond in bullet points?

How would you build a test suite for a chatbot that answers questions about a company's product?

What is a feedback loop in an AI product, and how can production feedback be used to improve test cases?

How do you measure an AI agent's error rate with human-in-the-loop review, and use the results to reduce hallucinations?

How do you automate testing or validate nondeterministic responses from an LLM?

When reviewing AI-generated responses, what criteria would you use to classify a response as acceptable, inaccurate, or unusable?

How would you ensure that shipping AI agents improves the relevant success metrics?

What constraints should you account for when testing an AI agent?

How would you test an AI chatbot using TestNG and TestRail?

How would you detect when a real, correctly tagged ad citation does not actually support the claim made about it?

How would you design a single-prompt reasoning stress test to reveal whether a model is overgeneralizing or following the guideline hierarchy?

How would you design a prompt for a reasoning model to evaluate an annotator's labeling decision and return a verdict with a falsifiable, auditor-ready rationale?

How would you define a quantitative rubric for evaluating a model's grounding, correctness, and calibration, including penalties for missing or incorrect citations and unwarranted confidence?

How would you calibrate an LLM-as-a-judge compliance gate to avoid both passing noncompliant outputs and rejecting valid ones?

How would you benchmark and compare LLMs to identify the best fit for a particular use case?

What would you do if the success metric for a ticket-handling agent were to reduce the number of tickets in the human queue?

How could a metric focused on reducing tickets in the human queue be gamed?

How would you approach writing and maintaining tests for an AI agent?

When comparing two AI responses, what makes one clearly higher quality than the other in Bengali, especially in terms of fluency, fidelity, structure, safety, and tone?

Could you share examples of common errors in AI-generated Bengali outputs, such as literal translation, wrong register, or awkward collocations, and explain how you correct them?

When comparing two AI answers about mixing concentrated hydrochloric acid with an oxidizing reagent in a calcium carbonate reaction, what rubric criteria and scoring approach would you use?

How would you score an AI answer that is numerically correct but gives poor safety reasoning against a clear, safe answer that contains a stoichiometric error?

When you evaluate an AI response with a rubric, what core dimensions do you score, and how do you define clear pass/fail thresholds for each?

Suppose the rubric's accuracy dimension yields low agreement for open-ended answers. How would you refine the metric or rubric to improve consistency without overpenalizing helpful responses?

When a model response includes five factual claims across finance and health care, how do you prioritize which claims to verify first under a tight time budget?

How do you distinguish a missing citation from an unverifiable claim, and how should each affect accuracy versus completeness ratings?

How would you assess an agentic AI solution before and after deployment, including its evaluation harness, data integration challenges, metrics, and alignment with business goals?

What should the software testing life cycle for agentic AI systems look like, especially in regulated insurance environments?

How would you evaluate agents pragmatically when models, harnesses, and MCP tools are changing constantly?

How would you use an evaluation framework to compare model versions or configurations side by side in an agent harness?

When iterating on an LLM feature, how do you set up an experiment plan that separates prompt changes from model changes, and what metrics determine whether to promote a change?

Can you create an original senior-level adversarial question and gold-standard answer that targets a specific reasoning failure mode, such as base-rate neglect, Simpson's paradox, or scope ambiguity?

How would you evaluate autonomous agents, given that execution is nondeterministic?

What specific metrics beyond the classic NLP metrics would you set up to measure multi-step planning accuracy?

What is the difference between model-level metrics and system-level agent evaluation?

How do you determine the stronger response in side-by-side comparisons? Walk me through the specific criteria you use.

How would you build a programmatic evaluation harness using benchmark test suites or deterministic test fixtures to measure an agent's accuracy before deploying updates?

How would you convert correctness, completeness, and constraint-following into a weighted scoring scheme with clear thresholds while minimizing rubric gaming by the agent?

How would you design a self-contained evaluation task for an agent doing a practical computer workflow, such as cleaning a CSV and producing a summary table, including inputs, oracles or validators, and a mostly automatable rubric?

What inconsistencies would you flag first, and why, if a report claims a 2% accuracy improvement and fewer hallucinations, but the appendix shows different denominators, mixed seeds, and a metric definition change mid-run?

What are three non-obvious failure modes you would add to an evaluation rubric?

How would you design a prompt and rubric package to reliably test an LLM's multitask, step-by-step reasoning?

How do you design edge cases that distinguish a model truly following instructions from one pattern matching the schema without relying on hidden answers or shortcuts?

When authoring a suite of 30 reasoning tasks, how do you cover instruction hierarchy, implicit assumptions, and brittle reasoning while keeping the suite minimal and nonredundant?

Tell me about a time an AI-generated trading or financial explanation sounded plausible but was logically flawed. What was wrong with its reasoning, and how did you verify your assessment?

How would you turn a flawed AI-generated financial response into evaluator feedback? Identify the issue, the criterion it failed, and the correction you would recommend.