Learn LLM Inference Engineering From First Principles

What Is Inference Engineering Zero to Hero?

Inference Engineering Zero to Hero is a hands-on course repository for understanding the runtime systems behind large language model serving. Instead of focusing on training or improving model quality, it examines what happens after a request reaches an LLM.

The project starts with the smallest useful inference operations and gradually works toward a compact inference runtime. Its intended progression is:

tokenization → prefill → decode → KV cache → sampling → request state → scheduling → continuous batching → memory management → serving-engine internals

The implementation-first approach emphasizes inspecting tensors, generated outputs, cache state, and memory use directly. This helps connect model operations with practical serving concerns such as latency, throughput, GPU utilization, and memory consumption.

What You Will Learn

The course is designed to develop a systems-oriented mental model of LLM inference. Its main topics include:

• Tokenization: how a prompt becomes the token IDs consumed by a model.
• Prefill: how the model processes the initial prompt and creates the state needed for generation.
• Decode: how subsequent tokens are generated one step at a time.
• KV caching: how attention state grows and avoids unnecessary recomputation.
• Sampling: how a token is selected from model output scores.
• Request state: how a runtime tracks prompts, generated tokens, and completion status.
• Scheduling: how an engine decides which requests to process.
• Continuous batching: how active requests can be combined efficiently as they enter and leave the system.
• Memory management: where GPU memory is used and how cache organization affects capacity.
• Serving-engine internals: how model execution is surrounded by systems that improve efficiency.

Course Status and Structure

The repository is divided into Beginner, Intermediate, and Advanced levels. At present, the Beginner track is in progress. Intermediate and Advanced material is not yet available.

Available Chapter

• Chapter 01: Manual LLM Inference From First Principles — available in beginner/01_manual_llm_inference/.

Planned Beginner Chapters

1. Sampling
2. Request State
3. Simple Scheduler
4. Continuous Batching
5. KV Cache Management
6. Paged or Block KV Cache
7. Prefix Caching
8. Benchmarking Inference

The roadmap is intentionally flexible. Chapters may be added, removed, split, or reordered as implementation work uncovers new problems.

Installation and Setup

The repository README does not provide a fixed dependency installation command or environment specification. Begin by downloading or cloning the repository, then inspect the files and instructions included with Chapter 1 before installing or running anything.

1. Obtain a local copy of the GitHub repository.
2. Open the repository directory in a terminal, editor, or notebook environment.
3. Navigate to beginner/01_manual_llm_inference/.
4. Read any chapter-specific instructions and identify the notebook in that directory.
5. Install only the dependencies requested by the chapter materials.
6. Open the notebook and execute its cells in order.

beginner/01_manual_llm_inference/

Because the README does not state exact package versions, hardware requirements, or installation commands, use the chapter notebook as the authoritative source for its runtime setup.

Basic Usage: Work Through Manual Inference

Chapter 1 is the recommended starting point. Its purpose is to examine manual LLM inference from first principles rather than hiding generation behind a complete serving API.

Step 1: Follow the Notebook Sequentially

Run the notebook from beginning to end. Avoid skipping setup or inspection cells, because later operations may depend on model objects, tokenized inputs, or cached state created earlier.

Step 2: Trace the Prompt

Observe how the input prompt changes as it passes through the inference process. A useful conceptual trace is:

prompt text
→ token IDs
→ prompt processing or prefill
→ next-token scores
→ selected token
→ updated sequence
→ repeated decoding

Record tensor shapes and token counts whenever the notebook exposes them. This makes it easier to see the distinction between processing an entire prompt and generating one additional token.

Step 3: Separate Prefill From Decode

Prefill processes the prompt and establishes the state needed for generation. Decode produces additional tokens incrementally. Treating these as separate stages is essential because they have different computational and memory characteristics.

While running the notebook, ask:

• How many input tokens are handled during prefill?
• What input is passed during each decode step?
• Which tensors change after a new token is generated?
• Which state can be reused instead of recomputed?

Step 4: Inspect Outputs Directly

Compare token IDs with decoded text and inspect intermediate outputs where the notebook permits it. The goal is not merely to produce text, but to understand the sequence of operations responsible for that text.

Understanding the Runtime Model

A useful way to approach the repository is to treat generation as a stateful loop rather than a single model call:

prepare request
process prompt
store reusable state
select next token
update request state
repeat until completion

This perspective prepares you for later chapters. Sampling determines which token is chosen, request state tracks each generation, scheduling selects work, and continuous batching combines work from multiple active requests.

Advanced Study Tips

Track Shapes, State, and Memory Together

Do not inspect tensors in isolation. For every important operation, note the input shape, output shape, sequence length, cache state, and expected memory growth. This connects mathematical model execution with systems behavior.

Compare Prompt Length and Decode Length

When the chapter allows you to change inputs, compare short and long prompts as well as short and long generations. Observe which costs occur during prefill and which accumulate during decoding. Do not assume benchmark conclusions beyond what your measurements demonstrate.

Build a Request-State Table

Before the dedicated request-state chapter becomes available, you can organize observations with fields such as:

• Input token IDs
• Generated token IDs
• Current sequence length
• Cache state
• Generation status

This is a study technique rather than a feature promised by the current repository. It helps reveal why inference engines need systems around the model itself.

Measure Before Optimizing

The roadmap includes a planned chapter on inference benchmarking. Until it is available, keep experiments controlled: change one input or generation condition at a time, record observations, and distinguish measured behavior from assumptions.

Revisit Earlier Chapters

As sampling, scheduling, batching, and cache-management chapters are added, revisit manual inference. Relate each new abstraction to the original token-generation loop so that optimizations remain grounded in actual model execution.

Current Limitations

Only the first Beginner chapter is currently listed as available. Sampling, schedulers, continuous batching, cache-management strategies, benchmarking, and the Intermediate and Advanced tracks remain planned. The repository should therefore be treated as an evolving course rather than a complete inference-engine framework.

Conclusion

Inference Engineering Zero to Hero offers a practical path into the runtime side of LLMs. Start with beginner/01_manual_llm_inference/, run the notebook in order, and focus on how prompts, tokens, tensors, and reusable state move through generation. This foundation will make the planned lessons on scheduling, batching, memory, and serving-engine internals easier to understand.