All 51 questions
What is KV caching, and how does it work during LLM inference?
Asked in 5 of 440 interviews · last asked Nov 9 · Concept · Mid-level
Show answer
KV caching stores the attention key and value vectors for earlier tokens at each transformer layer. The prompt first fills the cache. During generation, each new token adds its keys and values, while its query attends to the cached and current keys and values. This avoids recomputing earlier tokens, making generation faster but using more memory.
- The cache stores key and value vectors, not token IDs or attention scores.
- Past queries are not needed for future tokens, so they are not cached.
- Cache memory grows with sequence length and batch size.
- Caching avoids repeated computation, but each new query still attends over the available context.
What the interviewer checks: The interviewer checks whether you understand what attention reuses during generation and the speed versus memory trade-off.
Other ways it was asked
- “Can you explain the KV cache?”
- “Where does the KV cache live?”
- “What is a KV cache, and what does it store during transformer inference?”
How would you reduce token usage in an LLM application?
Asked in 3 of 440 interviews · last asked Sep 8 · Scenario · Mid-level
Show answer
I would measure input and output tokens, then remove repeated instructions and irrelevant context. I would retrieve only useful passages, trim or summarize chat history, and request concise answers. I would test these changes on real tasks to check quality, and cache repeated results to avoid extra model calls.
- Keep instructions and context needed for a correct answer.
- Summaries save tokens but may lose important details.
- Set output limits carefully; low limits can cut answers short.
- Prompt caching may lower cost without reducing token counts; support and pricing vary by provider.
What the interviewer checks: The interviewer checks whether you can reduce token waste while measuring the effect on answer quality.
Other ways it was asked
- “What strategies do you use to reduce token usage in an LLM application?”
- “How can you reduce LLM token costs without sacrificing quality?”
What techniques can reduce LLM API costs, and how would you measure the savings?
Asked in 2 of 440 interviews · last asked Oct 7 · Scenario · Senior
Other ways it was asked
- “What techniques can reduce latency and API costs in an LLM application?”
How would you compare the token efficiency of a transformer-based next-word predictor with modern chatbot models?
Asked once · last asked Apr 11 · Concept · Senior
If a customer needs a RAG system for about 1,000–2,000 users, with 100 concurrent users, how would you determine the LLM's tokens-per-minute configuration?
Asked once · last asked Apr 24 · Scenario · Senior
How would you manage token limits when an AI application runs out of available tokens?
Asked once · last asked Jun 25 · Scenario · Mid-level
What techniques can you use to reduce latency and memory usage in models and AI agents?
Asked once · last asked Jul 16 · Concept · Senior
How do you decide when a task requires a more expensive AI model and when a cheaper model is sufficient?
Asked once · last asked Aug 14 · Scenario · Mid-level
The team is self-hosting an open-weight model and experiencing unacceptable time to first token and throughput bottlenecks during traffic spikes. What inference optimization strategies would you suggest?
Asked once · last asked Aug 20 · Scenario · Senior
What context caching techniques are available, and when would you use them?
Asked once · last asked Aug 20 · Concept · Mid-level
Can you tell me more about how model routing works within the system? Does it involve using several models for different tasks?
Asked once · last asked Aug 22 · Design · Senior
What pricing model does OpenAI use, and what cautions should you consider when using its services?
Asked once · last asked Aug 25 · Concept · Mid-level
When comparing OpenAI models for an AI chatbot, how do you account for concurrency, and what factors guide your choice?
Asked once · last asked Aug 25 · Scenario · Senior
When two users use an AI chatbot at the same time, how are token limits and usage allocated?
Asked once · last asked Aug 25 · Concept · Mid-level
How would you reduce token waste in an agent system when taking screenshots or observations, especially when delays are involved?
Asked once · last asked Aug 25 · Scenario · Senior
How do you avoid wasting tokens and manage costs when building AI agents, especially when users retry actions repeatedly?
Asked once · last asked Aug 25 · Scenario · Senior
Would you use the same LLM for every task in a pipeline or select different models for different tasks?
Asked once · last asked Sep 2 · Design · Senior
How would you compare LLM latency for an information extraction task and choose a model?
Asked once · last asked Sep 4 · Scenario · Mid-level
How would you change the architecture to reduce LLM costs for a chatbot handling 14,000 incident resolution requests per day?
Asked once · last asked Sep 8 · Design · Senior
How can resolved LLM query data be used to reduce future LLM costs and handle recurring questions internally?
Asked once · last asked Sep 8 · Design · Senior
LLM token costs have doubled. How would you diagnose the increase and optimize the system?
Asked once · last asked Sep 8 · Scenario · Mid-level
What types of AI workloads are vLLM and Ray Serve used for?
Asked once · last asked Sep 24 · Concept · Mid-level
How will you optimize LLM inference?
Asked once · last asked Sep 25 · Scenario · Senior
How do GGUF and quantization support running language models locally?
Asked once · last asked Sep 29 · Concept · Mid-level
How can changing the model and reasoning effort affect LLM output quality, cost, and latency?
Asked once · last asked Oct 6 · Scenario · Mid-level
How would you evaluate trade-offs among cost, latency, and answer quality in an LLM routing system, and what metrics would you track?
Asked once · last asked Oct 6 · Scenario · Senior
Why might an application use both Qwen and Gemini Flash instead of a single model?
Asked once · last asked Oct 11 · Concept · Mid-level
How can you use tokens economically when working with model APIs?
Asked once · last asked Nov 18 · Scenario · Mid-level
Other ways it was asked
- “How can you optimize token use when invoking model APIs?”
What phases make up an inference request?
Asked once · last asked Oct 26 · Concept · Mid-level
How would you decide which tasks to route to smaller models, and when might the complexity of building and governing the router outweigh the savings?
Asked once · last asked Nov 2 · Scenario · Senior
In your view, what are the key challenges and current trends in optimizing large language model inference on limited hardware, such as consumer GPUs?
Asked once · last asked Nov 9 · Concept · Senior
What different workloads do you see in general for LLMs, and how do they affect response time and pressure on the system?
Asked once · last asked Nov 9 · Concept · Senior
How do input and output lengths affect the model?
Asked once · last asked Nov 9 · Concept · Mid-level
How do you deploy the prefill and decode phases on hardware?
Asked once · last asked Nov 9 · Design · Senior
If you distribute prefill and decode to multiple GPUs, what is this called usually?
Asked once · last asked Nov 9 · Concept · Senior
What implications does it have if you suddenly have two racks in your data center, one doing prefill and one doing decode? What problems might you run into?
Asked once · last asked Nov 9 · Scenario · Senior
How would you design an LLM inference platform, and what happens between receiving a request and returning tokens?
Asked once · last asked Nov 9 · Design · Senior
Other ways it was asked
- “Can you walk me through the inference platform architecture and what happens between a request and the moment tokens come back?”
What does continuous batching actually do, and what did we have before it?
Asked once · last asked Nov 9 · Concept · Mid-level
What happens to the KV cache of the previous user when a new user starts?
Asked once · last asked Nov 9 · Concept · Mid-level
Do you know any techniques that address KV cache management?
Asked once · last asked Nov 9 · Concept · Senior
What is paged attention, and how does it help manage the KV cache?
Asked once · last asked Nov 9 · Concept · Mid-level
What do you mean by token budgeting, and when does it come into play?
Asked once · last asked Nov 17 · Concept · Mid-level
How would you truncate or otherwise limit LLM usage when a service reaches its token budget?
Asked once · last asked Nov 17 · Scenario · Senior
How can you estimate an LLM response's output token count before making a request?
Asked once · last asked Nov 17 · Concept · Mid-level
What are semantic caching and prompt caching, and how do they differ?
Asked once · last asked Nov 18 · Concept · Mid-level
Why is it called a key-value (KV) cache?
Asked once · last asked Jul 16 · Concept · Junior
Does an LLM token have a fixed price, such as two cents?
Asked once · last asked Jul 16 · Concept · Junior
How would you complete a coding task while using a limited number of tokens with an AI coding assistant?
Asked once · last asked Sep 21 · Scenario · Senior
How do you reduce token usage when working with an AI coding assistant such as Amazon Kiro?
Asked once · last asked Sep 21 · Tooling · Senior
What does “juice” refer to in LLM inference?
Asked once · last asked Oct 6 · Concept · Mid-level
What does cache mean in the LLM context?
Asked once · last asked Oct 6 · Concept · Junior
More Generative AI and LLM interview questions
- All Generative AI and LLM interview questions
- Generative AI and LLM Basics Interview Questions
- Prompt Engineering Interview Questions
- RAG Interview Questions
- LLM Fine-tuning and Training Interview Questions
- AI Agents and Context Management Interview Questions
- LLM Evaluation and Benchmarking Interview Questions
- LLM Hallucinations, Safety and Security Interview Questions
- Generative AI System Design Interview Questions
How this list is made
Every question here was asked by a real interviewer in a real job interview that a candidate ran MeetAssist in. After the call, the interviewer's questions are pulled from the transcript. Practice sessions, tests, exams and anything that was not a job interview are dropped.
Questions are rewritten so they make sense on their own: names of people, employers and clients are removed, and speech-to-text mistakes are fixed. Small talk, logistics and questions about one candidate's own résumé are left out. Rephrasings of the same question are merged, and the count is the number of different interviews it came up in.
The interviews come from candidates who use MeetAssist, so they lean towards software, data and AI roles: the shares describe these interviews, not every Generative AI and LLM interview. Answers to the most asked questions are written and checked by an editor; a "common mistake" is shown only when reviews of real answers in these interviews show it, with the count.
440 interviews, 243 candidates, Feb 2026 to Nov 2026. Updated every month with new interviews.
Or add the Chrome extension