Learn AI System Design Step by Step: From LLM Inference to Production Agents

What Is the AI System Design Project?

AI System Design is an open-source educational guide maintained by Amit Shekhar, founder of Outcome School. It explains how to design the complete system around a large language model rather than treating an AI product as a single model API call.

The guide connects familiar system-design components such as databases, caches, queues, load balancers, and gateways with AI-specific constraints such as GPUs, tokens, long-running requests, streamed output, non-deterministic behavior, and per-request model costs.

AI System Design is regular system design plus the new operational constraints introduced by AI models.

The material is intended for AI engineers, LLM engineers, backend engineers, machine learning engineers, platform engineers, architects, students, and anyone preparing for an AI system design interview.

Why AI System Design Matters

Calling an LLM API from a script may require only a few lines of code. Operating that feature for many users is a different problem. Production systems must control latency, throughput, reliability, privacy, output quality, and cost while handling long prompts and token-by-token responses.

For example, a production customer-support assistant may need to:

• Start responding quickly, even when traffic increases.
• Ground answers in company documents through retrieval.
• Prevent private information from leaking.
• Handle model failures, timeouts, and rate limits.
• Measure answer quality despite non-deterministic output.
• Stay within a defined token and infrastructure budget.

The repository organizes the building blocks required to solve these problems and links many subjects to deeper explanatory articles.

Key Topics and Features

Inference foundations

The guide begins with tokens, inference servers, inference engines, and AI hardware such as GPUs, TPUs, and LPUs. It explains that an LLM generates text autoregressively, predicting one token and then using that output while predicting the next token.

It also introduces the two major inference phases:

• Prefill: processing the input prompt and building the internal state needed for generation.
• Decode: generating the response one token at a time.

These phases affect time to first token, tokens per second, throughput, and infrastructure choices. The guide also covers chunked prefill and separating prefill and decode workloads.

Scaling, routing, and caching

The scaling sections cover vertical scaling, horizontal scaling, tensor parallelism, pipeline parallelism, autoscaling, GPU estimation, and cost estimation. LLM-specific load-balancing approaches include least outstanding tokens, prefix-aware routing, and sticky sessions.

The caching material distinguishes several layers:

• KV cache for reusing attention keys and values during generation.
• Prompt cache for repeated prompt prefixes.
• Semantic cache for queries that have similar meanings.
• Embedding cache for avoiding repeated embedding computation.

Model routing is also covered, including rule-based, classifier-based, embedding-based, LLM-based, and cascade approaches.

Retrieval and context management

The guide explains embeddings, vector databases, vector indexes, and retrieval-augmented generation, commonly called RAG. Its retrieval path includes document parsing, ingestion, chunking, hybrid search, HyDE query transformation, reranking, ColBERT, Agentic RAG, GraphRAG, and vectorless RAG.

It also addresses context-window limitations through truncation, summarization, sliding windows, compaction, and hierarchical memory. Related concepts include context rot, lost-in-the-middle behavior, RoPE decay, and context engineering.

Agents and multi-agent systems

The agent sections examine the core parts of an AI agent, end-to-end agent loops, tool calling, structured output, memory, and common failure modes. They also distinguish deterministic AI orchestration from more autonomous agent behavior.

Additional subjects include the Model Context Protocol, or MCP, Agent Skills, computer-use agents, browser agents, loop engineering, graph engineering, subagents, coordination patterns, and the Agent2Agent protocol.

Production operations

The repository extends beyond model calls into production concerns such as:

• Streaming through Server-Sent Events or WebSockets.
• Asynchronous processing and message queues for long tasks.
• Rate limits based on requests, tokens, concurrency, or cost.
• AI gateways and multi-tenant architectures.
• Guardrails, prompt-injection defenses, and AI red teaming.
• PII redaction, data residency, audit logs, and compliance clauses.
• Tracing, observability, evaluation, and LLM-as-a-judge workflows.
• Timeouts, retries with backoff, fallback models, and graceful degradation.

It also covers inference optimizations such as quantization, continuous batching, speculative decoding, Flash Attention, Mixture of Experts, model distillation, and Grouped Query Attention.

Installation and Setup

This repository is a learning guide rather than a software library, so it does not require package installation, an API key, or a runtime environment. You can read it directly on GitHub.

To keep a local copy, clone the repository with Git:

git clone https://github.com/amitshekhariitbhu/ai-system-design.git
cd ai-system-design

Open the README in a Markdown viewer or an editor that can render Markdown. Because the guide is continuously expanded, periodically update your local copy:

git pull

No build command is described in the README. The primary setup task is choosing a study path and using the table of contents to navigate between concepts.

How to Use the Guide

Option 1: Follow the complete learning path

If you are new to AI system design, read the guide in sequence. The recommended conceptual progression is:

1. Learn tokens, inference servers, and AI hardware.
2. Understand prefill, decode, latency, and throughput.
3. Study scaling, estimation, and load balancing.
4. Compare caching and model-routing strategies.
5. Learn embeddings, vector databases, and RAG.
6. Study context-window management.
7. Explore streaming, queues, rate limiting, and gateways.
8. Learn agents, tools, MCP, structured output, and memory.
9. Continue into multi-agent, multimodal, and voice systems.
10. Add guardrails, privacy, observability, and evaluation.
11. Study cost control, fine-tuning infrastructure, and inference optimization.
12. Apply the repository's system-design problem-solving framework.

Option 2: Prepare for an interview

For interview preparation, begin with the section about solving any AI system design problem. Then revisit the technical building blocks needed for each design decision. The framework follows eight stages:

1. Requirements: identify users, scale, latency goals, privacy needs, and constraints.
2. AI objective: define what the model must accomplish.
3. Data preparation: determine how source data is collected, parsed, cleaned, and indexed.
4. Architecture design: place gateways, services, models, retrieval, caches, queues, and storage.
5. Model selection and prompting: choose models and prompt strategies according to quality, latency, and cost.
6. Evaluation: define how output and system behavior will be measured.
7. Deployment and serving: plan inference, scaling, streaming, and resilience.
8. Monitoring: observe quality, latency, errors, token consumption, and cost.

Option 3: Use it as a reference

If you already build AI applications, use the table of contents as a reference. For example, consult the caching sections when reducing repeated computation, the RAG sections when retrieval quality is weak, or the observability and evaluation sections when a system works technically but its answer quality is unclear.

Basic Usage Example: Designing a Support Assistant

The repository is not an executable framework, so “usage” means applying its concepts to a design problem. Consider a customer-support assistant that must answer questions from private company documentation.

1. Define requirements

Write down the functional and operational requirements before selecting a model:

Use case: customer-support question answering
Knowledge source: private product and policy documents
Response mode: streamed text
Quality requirement: answers grounded in retrieved documents
Safety requirement: no private-data leakage
Operational goals: low latency, controlled cost, fault tolerance

Add expected traffic, prompt length, output length, peak concurrency, and acceptable latency when real measurements are available. These values drive token, GPU, throughput, and cost estimates.

2. Sketch the request path

A high-level design assembled from the guide's building blocks might be represented as:

Client
  -> AI Gateway
  -> Authentication and rate limiting
  -> Input guardrails and PII handling
  -> Retrieval pipeline
       -> Query embedding
       -> Vector or hybrid search
       -> Reranking
  -> Prompt construction
  -> Model router or inference server
  -> Output validation and guardrails
  -> Streamed response
  -> Tracing, evaluation, and cost monitoring

This is a design worksheet, not code supplied by the repository. Each component corresponds to a topic covered by the guide.

3. Design the retrieval pipeline

Parse and ingest the source documents, divide them into suitable chunks, create embeddings, and store the searchable representation. At query time, retrieve relevant chunks and consider hybrid search or reranking when vector similarity alone does not provide sufficient relevance.

Keep the final context within the model's context window. The guide's context-management strategies can help when retrieved material, conversation history, and instructions compete for space.

4. Plan latency and streaming

Measure time to first token separately from generation speed. Prefill affects how quickly generation can begin, while decode affects how quickly subsequent tokens arrive. Streaming through SSE or WebSockets can improve perceived responsiveness without reducing the underlying model computation.

5. Add reliability and safety

Use timeouts, retries with backoff, fallback models, graceful degradation, and output validation. Place input and output guardrails around the model, protect against prompt injection, redact PII where required, and retain suitable audit logs.

6. Evaluate and observe

Trace the complete request rather than monitoring only the model call. Useful dimensions from the guide include latency, throughput, token use, cost, retrieval behavior, model output quality, tool execution, and agent behavior. Because LLM output is non-deterministic, evaluation requires more than conventional exact-match unit tests.

Advanced Tips

Optimize the correct latency phase

Do not treat model latency as one number. Diagnose whether the bottleneck is prefill, decode, retrieval, queueing, or an external tool. Time to first token and tokens per second represent different user experiences and may require different optimizations.

Route requests instead of using one model for everything

The guide presents several model-routing strategies. A production design can classify requests and direct simpler tasks toward a cheaper model while reserving a more capable model for difficult cases. Cascade routing can escalate when an earlier response is not sufficient.

Apply caching at the right layer

KV, prompt, semantic, and embedding caches solve different problems. First identify what is repeated: generation state, prompt prefixes, semantically equivalent questions, or embedding computation. Then choose the matching cache and consider correctness, freshness, privacy, and tenant boundaries.

Treat retrieval quality as a pipeline

RAG quality does not depend only on the vector database. Parsing, chunking, embeddings, query transformation, hybrid search, reranking, and context assembly all influence the final answer. Evaluate intermediate retrieval results before blaming the language model.

Use agents only when the workflow needs them

Distinguish between predictable orchestration and an agent that dynamically selects tools or actions. The guide also notes that MCP can be useful for standardized tool and context integration but may be excessive for a simple workflow. Choose the least complex approach that satisfies the requirements.

Estimate tokens, GPUs, and cost early

Back-of-the-envelope estimates expose expensive assumptions before implementation. Estimate input and output tokens per request, request volume, concurrency, throughput requirements, and expected model cost. Remember that English text averages roughly four characters per token, although actual tokenization depends on the model's tokenizer.

Explain each topic in your own words

The README recommends explaining each completed section to someone else. A useful extension is to document the trade-off behind every architectural choice: why a queue is necessary, why a specific cache helps, why a fallback model is acceptable, and how the choice affects quality, latency, safety, or cost.

Suggested Practice Plan

1. Read one major section of the guide.
2. Summarize its purpose and trade-offs without looking at the text.
3. Add the component to a sample architecture.
4. Define at least one metric for that component.
5. Describe one failure mode and one mitigation.
6. Follow the linked deep-dive article when the overview is not sufficient.
7. Repeat the exercise with case studies such as coding assistants or real-time voice agents.

This approach turns the repository from a reference list into a practical design curriculum.

Conclusion

AI System Design provides a broad, beginner-friendly path from LLM inference fundamentals to production concerns such as RAG, agents, safety, evaluation, observability, fault tolerance, and cost optimization. It is not a package to install; it is a structured guide for understanding how AI components work together.

Start with the foundations if you are new, or begin with the eight-step design framework if you are preparing for interviews. In either case, apply every concept to a concrete system, measure its impact, and record the trade-offs behind your decisions.