161 Model Evaluation Metrics Interview Questions

All 161 questions

What are precision and recall, and how would you explain their differences using a business example?
Asked in 9 of 320 interviews · last asked Oct 16 · Concept · Junior
Show answer
Precision is the share of predicted positives that are truly positive. Recall is the share of actual positives the model finds. For customer attrition, precision measures how many flagged customers actually leave; recall measures how many leaving customers we flag. High precision limits wasted retention offers; high recall means fewer missed customers who leave.
- Precision = true positives / (true positives + false positives).
- Recall = true positives / (true positives + false negatives).
- Lowering the prediction threshold increases or maintains recall, but may reduce precision.
- Choose the threshold based on the costs of missed customers and unnecessary offers.
What the interviewer checks: The interviewer checks whether you can distinguish these metrics and connect them to business costs.
Other ways it was asked
- “How do precision and recall help evaluate a customer attrition model?”
- “What is the difference between precision and recall?”
- “What is precision and recall?”
How do you evaluate a machine learning model and choose appropriate metrics for the problem?
Asked in 8 of 320 interviews · last asked Oct 12 · Concept · Junior
Show answer
I evaluate a model on unseen data using metrics that match the task and the cost of mistakes. I use validation data or cross-validation to choose the model, then measure final performance on a separate test set. I compare it with a simple baseline and check whether it meets practical requirements.
- Choose data splits that prevent leakage; use time-based splits when predicting future events.
- For classification, use precision and recall when error costs differ; accuracy can mislead with imbalanced classes.
- For regression, use MAE for typical error size or RMSE to penalize large errors more.
- Check errors across important groups, prediction speed, and agreed performance targets before deployment.
What the interviewer checks: The interviewer wants to see whether you can test fairly and connect metric choices to real-world needs.
Other ways it was asked
- “How do you evaluate a model's performance?”
- “How do you determine whether a model is performing well or poorly, and what evaluation metrics do you look at?”
- “How do you evaluate a machine-learning model's accuracy?”
How would you evaluate a severely imbalanced classifier without being misled by accuracy?
Asked in 4 of 320 interviews · last asked Oct 22 · Concept · Mid-level
Show answer
I prioritize precision, recall, and the precision-recall curve for the minority class, rather than accuracy. I summarize that curve with average precision and inspect the confusion matrix at the chosen threshold. I choose the threshold on validation data using error costs. I report final results on untouched test data with realistic class proportions and no leakage.
- High accuracy can hide a model that never detects the minority class.
- Precision measures how often positive predictions are correct; recall measures how many actual positives are found.
- F1 balances precision and recall, but does not capture unequal error costs.
- ROC-AUC can look strong despite poor precision; precision-recall results depend on class proportions.
What the interviewer checks: The interviewer checks whether you can measure minority-class performance and choose a realistic, leakage-free evaluation setup.
Other ways it was asked
- “When evaluating a binary classifier on imbalanced data, what metrics do you prioritize, and why?”
- “When evaluating a binary classifier with strong class imbalance, which metrics would you rely on most and why?”
- “What evaluation setup and metrics do you use to avoid being misled by accuracy on a severely imbalanced dataset?”
Can you explain a confusion matrix?
Asked in 4 of 320 interviews · last asked Oct 9 · Concept · Junior
Show answer
A confusion matrix is a table that compares predicted classes with actual classes. For binary classification, it shows true positives, true negatives, false positives, and false negatives. It helps us see which mistakes a model makes, not just how often it is correct.
- True positives and true negatives are correct predictions of the positive and negative classes.
- False positives are false alarms; false negatives are missed positive cases.
- Check axis labels: rows and columns may represent actual or predicted classes.
- For multiple classes, each cell counts cases with one actual class and one predicted class.
Code
```
                    Predicted positive  Predicted negative
Actual positive     40 true positives   10 false negatives
Actual negative     5 false positives   45 true negatives
```
What the interviewer checks: The interviewer wants to know whether you can identify correct predictions and distinguish the types of classification errors.
Other ways it was asked
- “What is a confusion matrix?”
- “What do you understand by a confusion matrix?”
- “What are the components of a confusion matrix?”
Which metrics would you monitor during AI model training?
Asked in 3 of 320 interviews · last asked Oct 2 · Concept · Mid-level
Other ways it was asked
- “Which metrics do you monitor when benchmarking AI models?”
- “Which metrics would you check during machine learning model training?”
How would you measure whether fine-tuning improved the model?
Asked in 2 of 320 interviews · last asked Sep 27 · Scenario · Mid-level
Other ways it was asked
- “How would you determine whether fine-tuning actually improves a model?”
When would you prioritize precision over recall, and what are the limitations of confusion-matrix metrics?
Asked in 2 of 320 interviews · last asked Sep 9 · Concept · Mid-level
Other ways it was asked
- “When would you prefer precision over recall?”
- “When would you prefer recall over precision?”
- “Why might precision be preferred over recall, and what are the limitations of relying on confusion-matrix metrics?”
What is the difference between F1 and ROC-AUC?
Asked in 2 of 320 interviews · last asked Aug 12 · Concept · Mid-level
Other ways it was asked
- “What are the ROC curve and the AUC metric, and how do they differ?”
If your dataset is 99:1 imbalanced, which metrics would you choose for the model, and how would you set the decision threshold for deployment?
Asked in 2 of 320 interviews · last asked Jun 30 · Scenario · Mid-level
Other ways it was asked
- “When you evaluate a binary classification model for an imbalanced dataset, which metrics do you prioritize, and how do you choose the decision threshold for production?”
How would you verify a model evaluation answer involving metrics like accuracy, precision, recall, F1, ROC AUC, or RMSE?
Asked in 2 of 320 interviews · last asked Jun 26 · Scenario · Mid-level
Other ways it was asked
- “When evaluating model performance, how do you assess it beyond a single metric, and when might you favor precision, recall, F1, or ROC AUC?”
What evaluation data, tasks, and metrics would you use to measure a face recognition system's accuracy?
Asked in 2 of 320 interviews · last asked Sep 22 · Scenario · Mid-level
Other ways it was asked
- “How should you measure and interpret the accuracy of a face-recognition system such as DeepFace?”
- “What data or evaluation task did you use to measure a face recognition model's accuracy?”
What is an F1 score?
Asked in 2 of 320 interviews · last asked Sep 29 · Concept · Junior
Other ways it was asked
- “What is the F1 score?”
How would you evaluate the performance of an object-detection model?
Asked once · last asked Sep 7 · Scenario · Mid-level
Other ways it was asked
- “Which metrics would you use to evaluate an object-detection model?”
How would you evaluate AI or data science work using a structured rubric?
Asked once · last asked Oct 14 · Behavioral · Mid-level
Other ways it was asked
- “Tell me about a recent example of evaluating AI or data science work using a structured rubric.”
How can precision be calculated for supervised models, and how does evaluating precision differ for unsupervised models?
Asked once · last asked Mar 5 · Concept · Mid-level
How do you process data and evaluate a machine-learning model using accuracy, precision, recall, and a confusion matrix?
Asked once · last asked Mar 24 · Concept · Junior
What is the difference between R-squared and adjusted R-squared, and which is more reliable?
Asked once · last asked Apr 15 · Concept · Mid-level
How would you benchmark a model for extracting information from noisy, unstructured financial documents when no ground-truth labels or borrower repayment outcomes are available?
Asked once · last asked Apr 20 · Scenario · Senior
How do you evaluate the performance of an unsupervised learning model?
Asked once · last asked Apr 21 · Concept · Mid-level
Could you explain the silhouette score for clustering and how it is calculated?
Asked once · last asked Apr 21 · Concept · Junior
Your churn model has improved retention by 18%, but it performs 40% worse for users in lower-income regions, who consequently receive less outreach and churn faster. How would you handle this in practice? What would you do on Monday morning, and why?
Asked once · last asked Apr 26 · Scenario · Senior
In classical machine learning, how do metrics such as precision and F1 help identify how good a model is?
Asked once · last asked Apr 29 · Concept · Mid-level
When starting a new supervised machine learning project, how do you choose an evaluation metric and validation strategy that match the business goal and data structure?
Asked once · last asked May 3 · Scenario · Senior
In an online setting, how would you run an A/B test for a new model while controlling for seasonality and avoiding bias in traffic allocation or interference between variants?
Asked once · last asked May 3 · Scenario · Senior
How would you build a model and an evaluation framework to compare it with other models?
Asked once · last asked May 12 · Behavioral · Senior
Other ways it was asked
- “Can you describe a project where you built a model and an evaluation framework to compare different models?”
How would you verify that a machine learning model achieved 74% accuracy, and what would that accuracy be based on?
Asked once · last asked May 18 · Scenario · Mid-level
How would you validate that your redaction and tenant isolation controls do not meaningfully degrade the target audience's experience in terms of trust, relevance, and task completion across key segments?
Asked once · last asked May 28 · Scenario · Senior
How is precision different from accuracy?
Asked once · last asked May 30 · Concept · Junior
In an imbalanced setting, when would ROC AUC be misleading, and what would you use instead to align with business costs?
Asked once · last asked Jun 3 · Concept · Mid-level
For fraud or rare-event detection, how would you choose a decision threshold, and how would you validate that it holds up over time?
Asked once · last asked Jun 3 · Scenario · Senior
How do you design benchmarks for evaluating AI models?
Asked once · last asked Jun 17 · Concept · Senior
How do you incorporate uncertainty estimation or confidence intervals into your analyses or model outputs?
Asked once · last asked Jun 26 · Concept · Mid-level
How do you validate that your keyword, metadata, or ML triage isn't systematically missing a class of privileged content by measuring false negatives over time?
Asked once · last asked Jul 16 · Scenario · Senior
How would you validate that an OCR engine update didn't regress accuracy across document types? What would your evaluation dataset and acceptance criteria look like?
Asked once · last asked Jul 16 · Scenario · Senior
How would you calibrate predicted probabilities for risk scoring and verify that calibration remains stable over time in key subgroups?
Asked once · last asked Jun 30 · Scenario · Senior
How would you test an AI model to uncover failures and use those findings to retrain it?
Asked once · last asked Jul 16 · Scenario · Mid-level
Why compare model architectures using the same hyperparameters instead of tuning each architecture separately?
Asked once · last asked Jul 16 · Concept · Mid-level
What are the limitations of accuracy as a classification metric?
Asked once · last asked Jul 20 · Concept · Junior
When validation scores vary across runs, how would you track whether changes to the model or feature set caused an improvement?
Asked once · last asked Jul 21 · Scenario · Mid-level
How would you quantify the impact of a model or pipeline change compared with its previous performance or stability?
Asked once · last asked Jul 21 · Scenario · Mid-level
What metric would you use to measure the overlap between two bounding boxes?
Asked once · last asked Jul 21 · Concept · Junior
When adjusting an evaluation loop to improve sensitivity and detect regressions early, how would you check whether changes to the training data or method are causing unintended shifts in test results?
Asked once · last asked Jul 23 · Scenario · Senior
Other ways it was asked
- “When you're adjusting your evaluation loop to tackle sensitivity and capture early regressions, how do you ensure that changes to the training data or method aren't causing unintended shifts in your test results?”
When conducting sensitivity studies, how do you determine whether an evaluation issue is caused by a model flaw or by external factors such as data distribution changes?
Asked once · last asked Jul 23 · Scenario · Senior
Looking at the broader field of AI training and evaluation, what are some of the current discussions or trends when it comes to maintaining model robustness across varying data distributions?
Asked once · last asked Jul 23 · Senior
Imagine you're deciding whether to focus on enhancing the quality of your training data or refining your evaluation metrics to better capture edge cases. How would you approach making this decision, and what factors would influence your choice?
Asked once · last asked Jul 23 · Scenario · Senior
How do you determine whether synthetic images accurately represent real oncology findings?
Asked once · last asked Jul 27 · Scenario · Senior
How should an oncology specialist be involved in validating a skin cancer detection model?
Asked once · last asked Jul 27 · Scenario · Mid-level
How would you design an experiment to measure precision and recall?
Asked once · last asked Aug 3 · Scenario · Mid-level
When creating edge cases for model evaluation, how would you avoid superficial cues or keyword leakage, and what criteria would you use to certify an example as difficult?
Asked once · last asked Aug 4 · Scenario · Senior
What are the F1 score and sensitivity, and how do they differ?
Asked once · last asked Aug 11 · Concept · Junior
If accuracy measures the proportion of correct predictions, why do we also need metrics such as F1 score and sensitivity?
Asked once · last asked Aug 11 · Concept · Mid-level
You’re deploying a high-stakes binary classifier, and one protected subgroup has only about a 1% positive rate. How do you set thresholds and evaluate trade-offs without creating harmful disparate impact?
Asked once · last asked Aug 11 · Scenario · Senior
In a real-world harm scenario with extreme base rates, if equalized odds and calibration can’t both hold, which would you prioritize and why?
Asked once · last asked Aug 11 · Scenario · Senior
What sections would you include in a fairness evaluation memo for a high-stakes binary classifier, and what would each section cover?
Asked once · last asked Aug 11 · Scenario · Senior
How would you structure a one-page executive summary for a fairness evaluation memo so a technical compliance lead can approve or reject the classifier?
Asked once · last asked Aug 11 · Scenario · Senior
Design a single prompt that makes an AI produce a production-ready fairness evaluation memo for a high-stakes binary classifier. Specify the objective, context, constraints, required sections, evaluation criteria, and exact output format.
Asked once · last asked Aug 11 · Senior
How can adjusting a YOLO threshold or post-processing improve model performance, and how would you measure the before-and-after impact?
Asked once · last asked Aug 12 · Scenario · Senior
How would you design a computational ML problem that tests whether a model's precision improvement is statistically reliable?
Asked once · last asked Aug 12 · Scenario · Senior
In a deep learning model, what does it mean if the F1 score is low while accuracy is high?
Asked once · last asked Aug 13 · Scenario · Mid-level
How do you approach actuarial model validation during an audit or another validation engagement?
Asked once · last asked Aug 13 · Scenario · Senior
How would you design an evaluation plan for a highly imbalanced classification problem, including which metrics to report and how to choose decision thresholds?
Asked once · last asked Aug 13 · Case · Senior
How would you design a benchmark task to evaluate whether an AI model can correctly distinguish physics mis-specification versus measurement artifact from residual plots and metadata, including a quantitative scoring rubric?
Asked once · last asked Aug 17 · Scenario · Senior
When you analyze errors in model outputs, how do you structure a failure taxonomy, and what information should annotators capture for each error?
Asked once · last asked Aug 17 · Scenario · Senior
How do you handle borderline cases where an output is mostly correct but violates one critical constraint? Would you use hard gating, weighted scoring, or severity-based overrides, and how would you justify your choice?
Asked once · last asked Aug 17 · Scenario · Senior
Two evaluation tasks produced conflicting metrics because one used an older dataset snapshot. How would you diagnose the root cause and decide what to rerun versus invalidate?
Asked once · last asked Aug 17 · Scenario · Senior
Other ways it was asked
- “How would you explain to a nontechnical stakeholder, in three sentences, why two evaluation tasks produced conflicting metrics due to different dataset snapshots and what should be rerun or invalidated?”
How would you respond politely and persuasively if a colleague argued that rerunning an evaluation was sufficient and invalidating it was overkill?
Asked once · last asked Aug 17 · Scenario · Senior
What should an evaluation input bundle contain, and how do you verify that it has not changed between runs?
Asked once · last asked Aug 17 · Scenario · Senior
Explain precision, recall, F1 score, ROC, and AUC, and describe when you would use each.
Asked once · last asked Aug 22 · Concept · Mid-level
When you work on customer churn analysis, how do you validate your model's results to ensure they are actionable?
Asked once · last asked Aug 22 · Scenario · Mid-level
What is the difference between model-level metrics and system-level agent evaluation?
Asked once · last asked Aug 25 · Concept · Mid-level
If you needed to balance improving model accuracy with minimizing bias, how would you decide which to prioritize, and what factors would influence your decision?
Asked once · last asked Aug 25 · Scenario · Senior
When prioritizing bias reduction in sensitive applications, what specific methods or tools would you employ to effectively reduce bias in AI models?
Asked once · last asked Aug 25 · Scenario · Senior
How would you evaluate speech-to-text accuracy using separate validation and test sets?
Asked once · last asked Aug 27 · Scenario · Mid-level
Other ways it was asked
- “How would you evaluate a speech-to-text model and determine whether it achieves 90% accuracy?”
- “How would you evaluate speech-to-text output against a test set, and how would you use a separate validation split?”
If two models score similarly, but one is less consistent across runs, how would you design your benchmarking methodology to measure variance and report statistical confidence?
Asked once · last asked Aug 29 · Scenario · Senior
How would you design a benchmark to detect shortcut behavior, such as copying one source as truth, while still allowing legitimate tie-break decisions?
Asked once · last asked Sep 7 · Scenario · Senior
How would you analyze model behavior, choose datasets and benchmarks, and compare performance after changing the model architecture?
Asked once · last asked Sep 8 · Behavioral · Mid-level
Other ways it was asked
- “How did you analyze model behavior, choose datasets and benchmarks, and compare performance after changing the architecture?”
What do you see as the key differences between testing an AI-enabled application versus a traditional application that isn't leveraging AI?
Asked once · last asked Sep 10 · Concept · Senior
How would you select a decision threshold for a face-recognition system?
Asked once · last asked Sep 11 · Scenario · Senior
How would you use controlled comparisons when applying AI to process monitoring and predictive maintenance?
Asked once · last asked Sep 15 · Scenario · Senior
How would you evaluate the linear regression model to know if it is performing well?
Asked once · last asked Sep 18 · Concept · Mid-level
How would you calculate and display evaluation metrics for a linear regression model?
Asked once · last asked Sep 18 · Write code · Mid-level
If one region is consistently underperforming, what might that indicate, and what checks would you perform before concluding that the model is miscalibrated?
Asked once · last asked Sep 20 · Scenario · Senior
How do you compare the predictive performance of traditional machine learning and deep learning models?
Asked once · last asked Sep 22 · Concept · Senior
How would you set an accuracy target for a face recognition system?
Asked once · last asked Sep 22 · Scenario · Mid-level
How would you determine whether a dataset is large enough to benchmark a face recognition model reliably?
Asked once · last asked Sep 22 · Scenario · Mid-level
How would you measure whether the classifier is actually improving routing and reducing unnecessary tool calls?
Asked once · last asked Sep 22 · Scenario · Senior
How would you gather and use feedback from operators and point-cloud data users when evaluating a classification model?
Asked once · last asked Sep 23 · Scenario · Mid-level
How do you assess whether a classification model is reliable, including its errors and false positives?
Asked once · last asked Sep 23 · Concept · Mid-level
How would you handle a situation where a model is highly accurate overall but performs poorly for a particular patient group in healthcare data?
Asked once · last asked Sep 23 · Scenario · Senior
On a residual plot, what patterns indicate heteroscedasticity or model misspecification, and what would you change or test next?
Asked once · last asked Sep 23 · Scenario · Senior
For a binary classifier, if a calibration curve is below the diagonal for predicted probabilities from 0.7 to 0.9, with a 95% confidence interval that does not cross the diagonal, what does that imply about predicted versus observed risk in that range?
Asked once · last asked Sep 23 · Concept · Mid-level
If a calibration plot also shows a histogram of predicted probabilities, with 90% of predictions between 0.0 and 0.1 and only 1% between 0.7 and 0.9, does that affect how you interpret the confidence interval around the 0.7–0.9 calibration segment?
Asked once · last asked Sep 23 · Scenario · Senior
If a calibration curve is computed with 10 equal-width bins but the prediction distribution is highly skewed toward zero, what binning change would you make to reduce variance, and why?
Asked once · last asked Sep 23 · Scenario · Mid-level
When designing a benchmark prompt for a highly skewed calibration plot, what numeric outputs would you require—such as per-bin counts, mean predicted probabilities, observed rates, and confidence-interval type—to make scoring objective?
Asked once · last asked Sep 23 · Scenario · Senior
Write a benchmark prompt for a reliability diagram with eight quantile bins and per-bin counts that requests ECE weighted by n and MCE, including the exact formulas and rounding requirements.
Asked once · last asked Sep 23 · Scenario · Senior
How would you define p_k as the mean predicted probability and o_k as the empirical event rate for bin k in an ECE/MCE benchmark prompt to avoid ambiguity?
Asked once · last asked Sep 23 · Scenario · Senior
How should a benchmark prompt specify when rounding occurs for input values, per-bin gaps, and final ECE and MCE values so the rule is applied consistently?
Asked once · last asked Sep 23 · Scenario · Senior
If a stakeholder asks you to use standard ECE without further details, what exact clarifying questions would you ask before finalizing the benchmark specification?
Asked once · last asked Sep 23 · Scenario · Senior
How would you validate and evaluate regression and classification models on a toy dataset, compute metrics like accuracy, precision, recall, and MSE, produce a report, and handle model drift detection with retraining as labeled data grows?
Asked once · last asked Sep 24 · Scenario · Mid-level
How would you evaluate a customer attrition model, and which metrics would you use?
Asked once · last asked Sep 29 · Scenario · Mid-level
What techniques would you use to improve precision and recall after training?
Asked once · last asked Oct 1 · Scenario · Mid-level
What would you do if inter-annotator agreement looked high but the model performed poorly on rare labels?
Asked once · last asked Oct 5 · Scenario · Senior
How do you measure the extraction quality, and what metrics do you look for?
Asked once · last asked Oct 6 · Concept · Mid-level
When evaluating text and image attribute extraction, should you use exact-match or semantic-match metrics?
Asked once · last asked Oct 6 · Concept · Mid-level
How do you balance false positives and false negatives when designing an intrusion detection system?
Asked once · last asked Oct 7 · Scenario · Mid-level
When would you choose the F2 score over the F1 score?
Asked once · last asked Oct 9 · Concept · Mid-level
When comparing different machine learning models, should you use the same evaluation metrics for each?
Asked once · last asked Oct 9 · Concept · Mid-level
Other ways it was asked
- “When comparing different machine learning models, should you use the same evaluation parameters for each?”
When comparing logistic regression, Random Forest, and XGBoost, should you use the same evaluation metrics for each model?
Asked once · last asked Oct 9 · Concept · Mid-level
Other ways it was asked
- “When comparing logistic regression, Random Forest, and XGBoost, should you use the same evaluation parameters for each model?”
How do you evaluate model performance above and below a decision threshold?
Asked once · last asked Oct 9 · Concept · Mid-level
How can two classification models have the same AUC but different sensitivity?
Asked once · last asked Oct 9 · Concept · Mid-level
What could cause different AUC values to be reported for the same model?
Asked once · last asked Oct 9 · Scenario · Mid-level
How would you choose and evaluate a face recognition threshold across different age groups or demographic groups?
Asked once · last asked Oct 11 · Scenario · Senior
How would you validate a vision-language model's interpretation of engineering drawings, and when would you use a manually curated gold set?
Asked once · last asked Oct 14 · Scenario · Senior
For a multi-class taxonomy with strong class imbalance, which metrics do you trust most, and how do you interpret them alongside raw percent agreement?
Asked once · last asked Oct 16 · Concept · Senior
How do you set and recalibrate the confidence threshold over time so it catches real risk without creating constant false stops?
Asked once · last asked Oct 17 · Scenario · Senior
How would you evaluate the quality and usefulness of groups identified through unsupervised learning?
Asked once · last asked Oct 17 · Concept · Mid-level
If churn affects only 2% of the data, would you use accuracy as the model metric? What metric would you choose?
Asked once · last asked Oct 18 · Concept · Mid-level
Can you differentiate between PR AUC and ROC AUC, and which one would you use in this modeling exercise?
Asked once · last asked Oct 20 · Concept · Mid-level
How do you choose which metrics are best suited for evaluating your segmentation models?
Asked once · last asked Oct 20 · Concept · Mid-level
When an ensemble outperforms individual models, how do you determine whether the improvement is genuine rather than due to noise?
Asked once · last asked Oct 20 · Scenario · Senior
How would you make an evaluation prompt adversarial by embedding one subtle but realistic trap, such as selection bias, Simpson’s paradox, or label leakage, while keeping it fair and verbally answerable?
Asked once · last asked Oct 21 · Scenario · Senior
Other ways it was asked
- “How would you make the prompt adversarial by embedding one subtle but realistic trap, such as selection bias, Simpson’s paradox, or label leakage, while keeping it fair and verbally answerable?”
Given the growing complexity of AI models, how would you balance the trade-off between ensuring detailed, robust evaluations and maintaining efficiency in the evaluation process?
Asked once · last asked Oct 21 · Scenario · Senior
When deciding which AI model outputs warrant deeper evaluation, what factors or indicators do you consider most important?
Asked once · last asked Oct 21 · Scenario · Mid-level
How do you design an evaluation set and metrics to detect model regressions in a feature where failures are rare but highly severe, for example safety or compliance?
Asked once · last asked Oct 22 · Scenario · Senior
When human labeling is inconsistent on severe edge cases, how do you quantify and handle inter-annotator disagreement so your regression conclusions remain reliable?
Asked once · last asked Oct 22 · Scenario · Senior
Other ways it was asked
- “When human labeling is inconsistent on those severe edge cases, how do you quantify and handle inter-annotator disagreement so your regression conclusions remain reliable?”
Suppose a model’s overall metrics improve, but you observe a systematic error pattern in one user segment. How would you prove it is real, not sampling noise, and report it?
Asked once · last asked Oct 22 · Scenario · Senior
Suppose your model’s calibration looks good overall, but it is miscalibrated for a key subgroup. What steps would you take to diagnose and fix that without overfitting?
Asked once · last asked Oct 22 · Scenario · Senior
How would you decide between Platt scaling and isotonic regression for recalibrating probabilities?
Asked once · last asked Oct 22 · Concept · Senior
How would you choose between Platt scaling and isotonic regression when subgroup sample size is limited?
Asked once · last asked Oct 22 · Scenario · Senior
If you can manually review only 500 alerts per day, how would you choose and validate an operating threshold to maximize caught positives while keeping daily volume within 500?
Asked once · last asked Oct 22 · Scenario · Senior
When automating validation checks, which three to five KPIs would you track on a dashboard to measure whether model validation quality and speed are improving?
Asked once · last asked Oct 23 · Scenario · Senior
Other ways it was asked
- “When automating data validation checks, which three to five KPIs would you track on a dashboard to measure whether validation quality and speed are improving?”
When building a logistic regression model, how do you assess calibration and discrimination, and what would prompt you to recalibrate?
Asked once · last asked Oct 23 · Concept · Mid-level
When you use class weights or resampling, how do you adjust predicted probabilities to reflect the true base rate at deployment?
Asked once · last asked Oct 23 · Concept · Senior
If a model shows bias against certain demographic data, which data attributes or methods would you use to correct this bias efficiently?
Asked once · last asked Oct 23 · Scenario · Senior
What additional analyses would you use to diagnose regional variation in an AI model's performance?
Asked once · last asked Oct 23 · Scenario · Senior
What is the most critical factor in evaluating AI solutions for financial workflows?
Asked once · last asked Oct 23 · Case · Mid-level
How would you address a gap between an AI vendor's claims and the tool's real-world performance?
Asked once · last asked Oct 23 · Scenario · Senior
How do you evaluate whether a redaction model is over-redacting or under-redacting, and what metrics or thresholds do you use to decide it is acceptable for release?
Asked once · last asked Oct 23 · Scenario · Senior
How would you design a senior-level QC plan for TAR or active learning review to validate privilege and responsiveness decisions before production, including sampling method, confidence targets, and escalation triggers?
Asked once · last asked Oct 23 · Scenario · Senior
How would you explain the difference between precision and recall in simple terms using cancer detection as an example?
Asked once · last asked Oct 1 · Concept · Junior
Other ways it was asked
- “What is the difference between precision and recall, explained very simply with the cancer example?”
- “What is the difference between precision and recall, using a simple cancer or spam example?”
What factors can cause a classification model to have low sensitivity?
Asked once · last asked Oct 9 · Concept · Junior
What is model evaluation?
Asked once · last asked May 12 · Concept · Junior
What do you understand by recall?
Asked once · last asked May 30 · Concept · Junior
What metrics would you use to evaluate a classification model?
Asked once · last asked Jul 20 · Concept · Junior
Other ways it was asked
- “What metrics did you use for classification?”
What is precision in machine learning?
Asked once · last asked Jul 24 · Concept · Junior
What are precision, recall, and R-precision?
Asked once · last asked Aug 3 · Concept · Junior
What is accuracy?
Asked once · last asked Aug 11 · Concept · Junior
What evaluation metrics are used for deep learning models?
Asked once · last asked Aug 13 · Concept · Mid-level
What is benchmarking in terms of AI?
Asked once · last asked Aug 17 · Concept · Mid-level
What is IoU?
Asked once · last asked Sep 7 · Concept · Junior
Tell me about MSE (mean squared error).
Asked once · last asked Dec 16 · Concept · Junior
Other ways it was asked
- “What is mean squared error (MSE)?”
What does sensitivity mean when comparing error metrics that use higher powers?
Asked once · last asked Dec 16 · Concept · Junior
How would you validate classification results with business stakeholders?
Asked once · last asked Sep 23 · Scenario · Mid-level
What metrics would you use to validate a classification model?
Asked once · last asked Oct 1 · Concept · Mid-level
Can you explain true positives, false positives, and false negatives?
Asked once · last asked Oct 1 · Concept · Junior
How would you explain true positives, false positives, and false negatives to a nontechnical business stakeholder?
Asked once · last asked Oct 1 · Scenario · Mid-level
Explain the confusion matrix concepts clearly, rather than teaching them step by step.
Asked once · last asked Oct 1 · Concept · Junior
Explain confusion-matrix concepts without using the terms true positive or false positive.
Asked once · last asked Oct 1 · Concept · Junior
How do you evaluate open-source models against alternatives, and what improvements might they offer?
Asked once · last asked Oct 6 · Scenario · Mid-level
When evaluating whether a machine learning architecture fits a task, how do you determine what to measure?
Asked once · last asked Oct 19 · Concept · Mid-level
If you had more time, which aspect of an AI model evaluation would you investigate further?
Asked once · last asked Oct 23 · Scenario · Mid-level

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Machine Learning interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

320 interviews, 221 candidates, Nov 2025 to Oct 2026. Updated every month with new interviews.

All 161 questions

What are precision and recall, and how would you explain their differences using a business example?

How do you evaluate a machine learning model and choose appropriate metrics for the problem?

How would you evaluate a severely imbalanced classifier without being misled by accuracy?

Can you explain a confusion matrix?

Which metrics would you monitor during AI model training?

How would you measure whether fine-tuning improved the model?

When would you prioritize precision over recall, and what are the limitations of confusion-matrix metrics?

What is the difference between F1 and ROC-AUC?

If your dataset is 99:1 imbalanced, which metrics would you choose for the model, and how would you set the decision threshold for deployment?

How would you verify a model evaluation answer involving metrics like accuracy, precision, recall, F1, ROC AUC, or RMSE?

What evaluation data, tasks, and metrics would you use to measure a face recognition system's accuracy?

What is an F1 score?

How would you evaluate the performance of an object-detection model?

How would you evaluate AI or data science work using a structured rubric?

How can precision be calculated for supervised models, and how does evaluating precision differ for unsupervised models?

How do you process data and evaluate a machine-learning model using accuracy, precision, recall, and a confusion matrix?

What is the difference between R-squared and adjusted R-squared, and which is more reliable?

How would you benchmark a model for extracting information from noisy, unstructured financial documents when no ground-truth labels or borrower repayment outcomes are available?

How do you evaluate the performance of an unsupervised learning model?

Could you explain the silhouette score for clustering and how it is calculated?

Your churn model has improved retention by 18%, but it performs 40% worse for users in lower-income regions, who consequently receive less outreach and churn faster. How would you handle this in practice? What would you do on Monday morning, and why?

In classical machine learning, how do metrics such as precision and F1 help identify how good a model is?

When starting a new supervised machine learning project, how do you choose an evaluation metric and validation strategy that match the business goal and data structure?

In an online setting, how would you run an A/B test for a new model while controlling for seasonality and avoiding bias in traffic allocation or interference between variants?

How would you build a model and an evaluation framework to compare it with other models?

How would you verify that a machine learning model achieved 74% accuracy, and what would that accuracy be based on?

How would you validate that your redaction and tenant isolation controls do not meaningfully degrade the target audience's experience in terms of trust, relevance, and task completion across key segments?

How is precision different from accuracy?

In an imbalanced setting, when would ROC AUC be misleading, and what would you use instead to align with business costs?

For fraud or rare-event detection, how would you choose a decision threshold, and how would you validate that it holds up over time?

How do you design benchmarks for evaluating AI models?

How do you incorporate uncertainty estimation or confidence intervals into your analyses or model outputs?

How do you validate that your keyword, metadata, or ML triage isn't systematically missing a class of privileged content by measuring false negatives over time?

How would you validate that an OCR engine update didn't regress accuracy across document types? What would your evaluation dataset and acceptance criteria look like?

How would you calibrate predicted probabilities for risk scoring and verify that calibration remains stable over time in key subgroups?

How would you test an AI model to uncover failures and use those findings to retrain it?

Why compare model architectures using the same hyperparameters instead of tuning each architecture separately?

What are the limitations of accuracy as a classification metric?

When validation scores vary across runs, how would you track whether changes to the model or feature set caused an improvement?

How would you quantify the impact of a model or pipeline change compared with its previous performance or stability?

What metric would you use to measure the overlap between two bounding boxes?

When adjusting an evaluation loop to improve sensitivity and detect regressions early, how would you check whether changes to the training data or method are causing unintended shifts in test results?

When conducting sensitivity studies, how do you determine whether an evaluation issue is caused by a model flaw or by external factors such as data distribution changes?

Looking at the broader field of AI training and evaluation, what are some of the current discussions or trends when it comes to maintaining model robustness across varying data distributions?

Imagine you're deciding whether to focus on enhancing the quality of your training data or refining your evaluation metrics to better capture edge cases. How would you approach making this decision, and what factors would influence your choice?

How do you determine whether synthetic images accurately represent real oncology findings?

How should an oncology specialist be involved in validating a skin cancer detection model?

How would you design an experiment to measure precision and recall?

When creating edge cases for model evaluation, how would you avoid superficial cues or keyword leakage, and what criteria would you use to certify an example as difficult?

What are the F1 score and sensitivity, and how do they differ?

If accuracy measures the proportion of correct predictions, why do we also need metrics such as F1 score and sensitivity?

You’re deploying a high-stakes binary classifier, and one protected subgroup has only about a 1% positive rate. How do you set thresholds and evaluate trade-offs without creating harmful disparate impact?

In a real-world harm scenario with extreme base rates, if equalized odds and calibration can’t both hold, which would you prioritize and why?

What sections would you include in a fairness evaluation memo for a high-stakes binary classifier, and what would each section cover?

How would you structure a one-page executive summary for a fairness evaluation memo so a technical compliance lead can approve or reject the classifier?

Design a single prompt that makes an AI produce a production-ready fairness evaluation memo for a high-stakes binary classifier. Specify the objective, context, constraints, required sections, evaluation criteria, and exact output format.

How can adjusting a YOLO threshold or post-processing improve model performance, and how would you measure the before-and-after impact?

How would you design a computational ML problem that tests whether a model's precision improvement is statistically reliable?

In a deep learning model, what does it mean if the F1 score is low while accuracy is high?

How do you approach actuarial model validation during an audit or another validation engagement?

How would you design an evaluation plan for a highly imbalanced classification problem, including which metrics to report and how to choose decision thresholds?

How would you design a benchmark task to evaluate whether an AI model can correctly distinguish physics mis-specification versus measurement artifact from residual plots and metadata, including a quantitative scoring rubric?

When you analyze errors in model outputs, how do you structure a failure taxonomy, and what information should annotators capture for each error?

How do you handle borderline cases where an output is mostly correct but violates one critical constraint? Would you use hard gating, weighted scoring, or severity-based overrides, and how would you justify your choice?

Two evaluation tasks produced conflicting metrics because one used an older dataset snapshot. How would you diagnose the root cause and decide what to rerun versus invalidate?

How would you respond politely and persuasively if a colleague argued that rerunning an evaluation was sufficient and invalidating it was overkill?

What should an evaluation input bundle contain, and how do you verify that it has not changed between runs?

Explain precision, recall, F1 score, ROC, and AUC, and describe when you would use each.

When you work on customer churn analysis, how do you validate your model's results to ensure they are actionable?

What is the difference between model-level metrics and system-level agent evaluation?

If you needed to balance improving model accuracy with minimizing bias, how would you decide which to prioritize, and what factors would influence your decision?

When prioritizing bias reduction in sensitive applications, what specific methods or tools would you employ to effectively reduce bias in AI models?

How would you evaluate speech-to-text accuracy using separate validation and test sets?

If two models score similarly, but one is less consistent across runs, how would you design your benchmarking methodology to measure variance and report statistical confidence?

How would you design a benchmark to detect shortcut behavior, such as copying one source as truth, while still allowing legitimate tie-break decisions?

How would you analyze model behavior, choose datasets and benchmarks, and compare performance after changing the model architecture?

What do you see as the key differences between testing an AI-enabled application versus a traditional application that isn't leveraging AI?

How would you select a decision threshold for a face-recognition system?

How would you use controlled comparisons when applying AI to process monitoring and predictive maintenance?