All 51 questions

  1. What is KV caching, and how does it work during LLM inference?

    Asked in 5 of 440 interviews · last asked Nov 9 · Concept · Mid-level

    Show answer

    KV caching stores the attention key and value vectors for earlier tokens at each transformer layer. The prompt first fills the cache. During generation, each new token adds its keys and values, while its query attends to the cached and current keys and values. This avoids recomputing earlier tokens, making generation faster but using more memory.

    • The cache stores key and value vectors, not token IDs or attention scores.
    • Past queries are not needed for future tokens, so they are not cached.
    • Cache memory grows with sequence length and batch size.
    • Caching avoids repeated computation, but each new query still attends over the available context.

    What the interviewer checks: The interviewer checks whether you understand what attention reuses during generation and the speed versus memory trade-off.

    Other ways it was asked
    • “Can you explain the KV cache?”
    • “Where does the KV cache live?”
    • “What is a KV cache, and what does it store during transformer inference?”
  2. How would you reduce token usage in an LLM application?

    Asked in 3 of 440 interviews · last asked Sep 8 · Scenario · Mid-level

    Show answer

    I would measure input and output tokens, then remove repeated instructions and irrelevant context. I would retrieve only useful passages, trim or summarize chat history, and request concise answers. I would test these changes on real tasks to check quality, and cache repeated results to avoid extra model calls.

    • Keep instructions and context needed for a correct answer.
    • Summaries save tokens but may lose important details.
    • Set output limits carefully; low limits can cut answers short.
    • Prompt caching may lower cost without reducing token counts; support and pricing vary by provider.

    What the interviewer checks: The interviewer checks whether you can reduce token waste while measuring the effect on answer quality.

    Other ways it was asked
    • “What strategies do you use to reduce token usage in an LLM application?”
    • “How can you reduce LLM token costs without sacrificing quality?”
  3. What techniques can reduce LLM API costs, and how would you measure the savings?

    Asked in 2 of 440 interviews · last asked Oct 7 · Scenario · Senior

    Other ways it was asked
    • “What techniques can reduce latency and API costs in an LLM application?”
  4. How would you compare the token efficiency of a transformer-based next-word predictor with modern chatbot models?

    Asked once · last asked Apr 11 · Concept · Senior

  5. If a customer needs a RAG system for about 1,000–2,000 users, with 100 concurrent users, how would you determine the LLM's tokens-per-minute configuration?

    Asked once · last asked Apr 24 · Scenario · Senior

  6. How would you manage token limits when an AI application runs out of available tokens?

    Asked once · last asked Jun 25 · Scenario · Mid-level

  7. What techniques can you use to reduce latency and memory usage in models and AI agents?

    Asked once · last asked Jul 16 · Concept · Senior

  8. How do you decide when a task requires a more expensive AI model and when a cheaper model is sufficient?

    Asked once · last asked Aug 14 · Scenario · Mid-level

  9. The team is self-hosting an open-weight model and experiencing unacceptable time to first token and throughput bottlenecks during traffic spikes. What inference optimization strategies would you suggest?

    Asked once · last asked Aug 20 · Scenario · Senior

  10. What context caching techniques are available, and when would you use them?

    Asked once · last asked Aug 20 · Concept · Mid-level

  11. Can you tell me more about how model routing works within the system? Does it involve using several models for different tasks?

    Asked once · last asked Aug 22 · Design · Senior

  12. What pricing model does OpenAI use, and what cautions should you consider when using its services?

    Asked once · last asked Aug 25 · Concept · Mid-level

  13. When comparing OpenAI models for an AI chatbot, how do you account for concurrency, and what factors guide your choice?

    Asked once · last asked Aug 25 · Scenario · Senior

  14. When two users use an AI chatbot at the same time, how are token limits and usage allocated?

    Asked once · last asked Aug 25 · Concept · Mid-level

  15. How would you reduce token waste in an agent system when taking screenshots or observations, especially when delays are involved?

    Asked once · last asked Aug 25 · Scenario · Senior

  16. How do you avoid wasting tokens and manage costs when building AI agents, especially when users retry actions repeatedly?

    Asked once · last asked Aug 25 · Scenario · Senior

  17. Would you use the same LLM for every task in a pipeline or select different models for different tasks?

    Asked once · last asked Sep 2 · Design · Senior

  18. How would you compare LLM latency for an information extraction task and choose a model?

    Asked once · last asked Sep 4 · Scenario · Mid-level

  19. How would you change the architecture to reduce LLM costs for a chatbot handling 14,000 incident resolution requests per day?

    Asked once · last asked Sep 8 · Design · Senior

  20. How can resolved LLM query data be used to reduce future LLM costs and handle recurring questions internally?

    Asked once · last asked Sep 8 · Design · Senior

  21. LLM token costs have doubled. How would you diagnose the increase and optimize the system?

    Asked once · last asked Sep 8 · Scenario · Mid-level

  22. What types of AI workloads are vLLM and Ray Serve used for?

    Asked once · last asked Sep 24 · Concept · Mid-level

  23. How will you optimize LLM inference?

    Asked once · last asked Sep 25 · Scenario · Senior

  24. How do GGUF and quantization support running language models locally?

    Asked once · last asked Sep 29 · Concept · Mid-level

  25. How can changing the model and reasoning effort affect LLM output quality, cost, and latency?

    Asked once · last asked Oct 6 · Scenario · Mid-level

  26. How would you evaluate trade-offs among cost, latency, and answer quality in an LLM routing system, and what metrics would you track?

    Asked once · last asked Oct 6 · Scenario · Senior

  27. Why might an application use both Qwen and Gemini Flash instead of a single model?

    Asked once · last asked Oct 11 · Concept · Mid-level

  28. How can you use tokens economically when working with model APIs?

    Asked once · last asked Nov 18 · Scenario · Mid-level

    Other ways it was asked
    • “How can you optimize token use when invoking model APIs?”
  29. What phases make up an inference request?

    Asked once · last asked Oct 26 · Concept · Mid-level

  30. How would you decide which tasks to route to smaller models, and when might the complexity of building and governing the router outweigh the savings?

    Asked once · last asked Nov 2 · Scenario · Senior

  31. In your view, what are the key challenges and current trends in optimizing large language model inference on limited hardware, such as consumer GPUs?

    Asked once · last asked Nov 9 · Concept · Senior

  32. What different workloads do you see in general for LLMs, and how do they affect response time and pressure on the system?

    Asked once · last asked Nov 9 · Concept · Senior

  33. How do input and output lengths affect the model?

    Asked once · last asked Nov 9 · Concept · Mid-level

  34. How do you deploy the prefill and decode phases on hardware?

    Asked once · last asked Nov 9 · Design · Senior

  35. If you distribute prefill and decode to multiple GPUs, what is this called usually?

    Asked once · last asked Nov 9 · Concept · Senior

  36. What implications does it have if you suddenly have two racks in your data center, one doing prefill and one doing decode? What problems might you run into?

    Asked once · last asked Nov 9 · Scenario · Senior

  37. How would you design an LLM inference platform, and what happens between receiving a request and returning tokens?

    Asked once · last asked Nov 9 · Design · Senior

    Other ways it was asked
    • “Can you walk me through the inference platform architecture and what happens between a request and the moment tokens come back?”
  38. What does continuous batching actually do, and what did we have before it?

    Asked once · last asked Nov 9 · Concept · Mid-level

  39. What happens to the KV cache of the previous user when a new user starts?

    Asked once · last asked Nov 9 · Concept · Mid-level

  40. Do you know any techniques that address KV cache management?

    Asked once · last asked Nov 9 · Concept · Senior

  41. What is paged attention, and how does it help manage the KV cache?

    Asked once · last asked Nov 9 · Concept · Mid-level

  42. What do you mean by token budgeting, and when does it come into play?

    Asked once · last asked Nov 17 · Concept · Mid-level

  43. How would you truncate or otherwise limit LLM usage when a service reaches its token budget?

    Asked once · last asked Nov 17 · Scenario · Senior

  44. How can you estimate an LLM response's output token count before making a request?

    Asked once · last asked Nov 17 · Concept · Mid-level

  45. What are semantic caching and prompt caching, and how do they differ?

    Asked once · last asked Nov 18 · Concept · Mid-level

  46. Why is it called a key-value (KV) cache?

    Asked once · last asked Jul 16 · Concept · Junior

  47. Does an LLM token have a fixed price, such as two cents?

    Asked once · last asked Jul 16 · Concept · Junior

  48. How would you complete a coding task while using a limited number of tokens with an AI coding assistant?

    Asked once · last asked Sep 21 · Scenario · Senior

  49. How do you reduce token usage when working with an AI coding assistant such as Amazon Kiro?

    Asked once · last asked Sep 21 · Tooling · Senior

  50. What does “juice” refer to in LLM inference?

    Asked once · last asked Oct 6 · Concept · Mid-level

  51. What does cache mean in the LLM context?

    Asked once · last asked Oct 6 · Concept · Junior

More Generative AI and LLM interview questions

How this list is made

Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.

Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.

The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Generative AI and LLM interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.

440 interviews, 243 candidates, Feb 2026 to Nov 2026. Updated every month with new interviews.