Build and Verify APEX, an Open RTL LLM Inference Tile

What APEX Does

APEX is a fully open, verification-first LLM inference tile. It implements one transformer decoder layer in real RTL, including attention, KV-cache compression, online softmax, RMSNorm, rotary position embeddings, SwiGLU, residual connections, and matrix operations.

The project targets a major edge-inference bottleneck: as a conversation grows, every new token must read an increasingly large key-value cache. APEX places the KV codec directly in the hardware datapath. Keys and values are compressed when produced, stored in compressed form, and decompressed while attention consumes them.

APEX is one inference tile, not a complete accelerator chip. DRAM controllers, PCIe, and a network-on-chip are outside the repository's stated scope.

The tile has been demonstrated with Qwen2.5-0.5B on FPGA hardware. Qwen2.5-7B tokens have passed through the software-verified golden pipeline, but the README does not claim that the 7B model has run through silicon.

Key Features

• Complete decoder-layer datapath: The RTL covers projections, attention, normalization, RoPE, feed-forward computation, residual operations, and KV-cache handling.
• In-datapath KV compression: Keys use per-channel INT4 quantization, values use per-token INT4 quantization, and difficult channels can use an fp16 outlier lane.
• Adaptive precision: The TIP unit tracks token importance and selects KVQ8, KVQ4, or KVQ4 with outliers by region.
• Shared matrix engine: The INT8 systolic MXE is time-multiplexed across the matrix jobs in a decoder layer.
• Online softmax: Scores stream through a numerically stable softmax implementation without requiring a context-length-sized score buffer.
• Bit-exact verification: RTL results are compared with executable NumPy reference models with zero tolerated mismatches.
• Mutation-tested testbenches: Verification is considered meaningful only when deliberate RTL defects cause tests to fail.
• Simulation-to-silicon replay: FPGA register-operation programs also run in a Verilator twin for bit-for-bit differential comparison.
• Explicit evidence labels: Results are classified as measured or projected rather than blended into a single performance claim.

Understand the Architecture

A decode step begins with an activation vector. RMSNorm prepares the input, and the MXE computes the query, key, and value projections. RoPE transforms queries and keys before the KVQ engine compresses keys and values for storage.

During attention, cached values are decompressed in the read path. The MXE calculates query-key products, the ASU applies online softmax, and another MXE job combines probabilities with cached values. Output projection, residual addition, another normalization stage, and the SwiGLU feed-forward path complete the layer.

Main RTL Blocks

• MXE: The INT8 systolic GEMM engine used for the layer's matrix operations.
• KVQ: The KV-cache compression and decompression engine.
• ASU: Online softmax, RMSNorm, SiLU, and SwiGLU.
• RoPE: Rotary position embedding for queries and keys.
• TIP: Token-importance tracking and precision-tier decisions.
• SEQ: The control and status register file, job decoder, and layer walker.
• SEAM: Activation framing, boundary quantization, and scale transport.
• XBR: Routing between engines without unnecessary host round trips.
• TOP: The apex_top composition layer and integration logic.

Read ARCHITECTURE.md for the detailed design, decision register, and module provenance.

Host Mode and Walked Mode

APEX supports two execution approaches. In host mode, the host constructs and submits each hardware job individually. This mode is proven on FPGA silicon, but every job incurs a host round trip.

In walked mode, the host starts a layer once. The on-tile layer walker fetches descriptors, sequences engines, routes intermediate tensors through the XBR fabric, and propagates quantization scales. The walked layer is bit-exact in simulation. The README states that walked attention on FPGA remains under active bring-up.

Do not treat host mode and walked mode as equivalent performance configurations. Host mode is useful for correctness and bring-up, while the walker is intended to remove the substantial per-job host overhead.

Installation and Local Setup

Prerequisites

For the documented local verification flow, install the following:

• Python 3.11
• NumPy
• Verilator 5.x
• GNU Make
• Icarus Verilog, optionally, for cross-checking

Run commands from the root of a checked-out APEX repository. Confirm the principal tools before starting:

python3 --version
verilator --version
make --version
python3 -c "import numpy; print(numpy.__version__)"

The README does not provide a package-install command or a pinned Python environment command. Use the repository's own files and subdirectory documentation when resolving additional dependencies for evaluation or FPGA workflows.

Inspect the Evidence First

Before running a long workflow, review the project's evidence and limitations:

• STATUS.md contains machine-generated verification counts and measured results.
• TRACEABILITY.md records known gaps and links claims to supporting evidence.
• The master table tracks implementation and bring-up status.
• The results index organizes committed evidence by campaign.

Run the Golden-Model Tests

Start with the executable NumPy reference models and frozen vectors:

make -C golden test

This command exercises the golden models and includes the pinned compression-accounting gate. Compression accounting is important because APEX includes tags, scales, padding, and outlier storage rather than reporting only the nominal INT4 payload.

Verify the RTL Pipeline

Run the Full Attention-Tile Smoke Test

make -C verif/top/smoke smoke

This is the documented end-to-end smoke target for the attention tile. Use it as an initial integration check after the golden-model suite passes.

Run Layer-3 Verification

make -C verif/top l3

The Layer-3 target compares the real composed tile against the golden model across the provided cases. APEX requires bit-identical results rather than an approximate numerical tolerance.

Verify the Layer Walker

make -C verif/seq_walker

This runs the walker suite and its mutation gates. Mutation testing deliberately breaks behavior and checks whether the verification environment detects the defect. A mutant that survives indicates a verification weakness rather than a successful design result.

Check the Performance Model

Run the analytic model's calibration checks with:

python3 perf/apex_perf_model.py --check

The performance model is explicitly projected, not measured silicon performance. Its assumptions and calibration anchors are documented in PERF_MODEL.md.

Projected 7B-class reading-speed and energy figures depend on unbuilt or incomplete system components, including the native W4 weight path, hardware layer walking, and wide LPDDR integration. They should not be presented as achieved hardware results.

Evaluate KV Compression on Models

The repository includes model-accuracy matrices for the verified KV codec. Run the 0.5B and 1.5B matrix with:

bash eval/kv_eval/run_matrix.sh

Run the 7B matrix with:

bash eval/kv_eval/run_matrix_7b.sh

Consult the evaluation READMEs beside the committed result files before running these commands. The published evaluation uses the full 10,042-document HellaSwag validation set. The README reports near-lossless behavior for the smaller models, no detectable full-set effect for KVQ8 at 7B, and a small measured cost for KVQ4 at 7B.

The associated evidence is available in the head-to-head results and the 7B evaluation results.

Run a Model Through the Golden Pipeline

The project documents prompt execution with run_tinynpu.py --prompt. This streams greedy Qwen tokens through the fixed-point golden pipeline and can produce hardware-shaped jobs for replay.

run_tinynpu.py --prompt

The README does not provide the complete model setup or every command-line argument in its top-level instructions, so follow the 7B token artifact documentation for the exact scope, artifacts, and reproduction details. Remember that this 7B flow is software-verified; it is not a claim of 7B execution on the FPGA tile.

Run the AWS F2 Demonstration

Requirements and Cost

The FPGA demonstration requires AWS CLI credentials with access to an f2.6xlarge instance in us-west-2. The README estimates approximately 30 minutes and about two US dollars for a full run.

The documented reference FPGA image is agfi-030a812cd224b409d. It uses the A2 recipe with a 15.625 MHz tile clock.

Run the Verification Battery

bash scripts/fpga/f2/run_walked_demo.sh agfi-030a812cd224b409d

The script boots the FPGA instance, loads the image, runs the documented verification battery and walked chains, prints the verdict, and terminates the instance.

Start the Interactive Prompt Demo

bash scripts/fpga/f2/run_chat_demo.sh

The interactive demo uses the walked pipeline. The README gives The capital of France is as an example prompt and reports the graded response Paris..

The DDR images derive from mlx-community/Qwen2.5-0.5B-Instruct-4bit. Team staging artifacts are configured in the scripts. External users should generate a weight image with:

python3 scripts/fpga/f2/make_weight_image.py

Configure the resulting image for a bucket you control rather than assuming access to the team's staging bucket.

Interpreting Hardware Results Correctly

The repository reports measured throughput of 0.56 tokens per second on an A0 image at 62.5 MHz and 0.25 tokens per second on the 15.625 MHz A2 reference image. It also reports a measured progression from a 0.004-token-per-second host-driven baseline.

These figures describe specific FPGA images, clocks, and execution paths. Check FIRST_WALKED_TOKENS.md before quoting them, and retain the associated configuration and measured label.

Advanced Tips

Use the Golden Model as the Arbiter

When changing an RTL block, update or validate the executable specification first, regenerate or confirm vectors as required by the repository, and compare RTL output bit for bit. Avoid introducing tolerance-based checks unless the project's documented arithmetic contract explicitly requires them.

Preserve Mutation Coverage

A passing regression is not enough under the APEX methodology. Run the relevant mutation gates after changing a testbench, scoreboard, sequencer, or arithmetic block. If a deliberately broken implementation passes, strengthen verification before trusting the original green result.

Replay the Same Job Across Environments

APEX operations use self-describing job descriptors that include shapes, routing, and quantization contracts. Preserve descriptors, scales, and register operations when diagnosing a mismatch. Replaying the same flight in the Verilator F2 twin and on hardware is the preferred route to separating RTL defects from synthesis or toolchain defects.

Track Scales with Tensor Data

Quantization scales are part of the hardware contract, not incidental metadata. When creating or inspecting jobs, retain the scale information that travels through the seams. A tensor without the corresponding scale cannot reproduce the intended fixed-point operation.

Distinguish Codec Payload from Total Storage

Do not calculate compression from INT4 payloads alone. Include scale banks, tags, padding, precision metadata, and outlier lanes. Use the repository's golden accounting gate and STATUS.md as the authoritative basis for published ratios.

Separate Model Sizes and Execution Claims

• Qwen2.5-0.5B: Demonstrated through the FPGA-backed pipeline described by the project.
• Qwen2.5-7B: Exercised through the software-verified golden pipeline.
• 7B-class architecture projections: Analytic projections for a larger system around the tile, not measured performance from the current FPGA implementation.

Study Weight Streaming Before Extending the Walker

Model weights do not fit entirely on the tile. The IB-FUEL design streams per-job weight records from DDR into an on-tile FIFO, while the W4 path transports four-bit groups and unpacks them for the INT8 matrix engine. Review IB_FUEL.md and W4_DATAPATH.md before modifying scheduling or feeder behavior.

Use the Repository Layout Efficiently

• rtl/ contains the hardware implementation.
• golden/ contains the bit-exact NumPy reference models.
• verif/ contains testbenches, assertions, scoreboards, mutation gates, and the F2 simulation twin.
• eval/ contains model-accuracy evaluation tools for the codec.
• perf/ contains the projected analytic performance model.
• scripts/fpga/ contains build, image, DDR-loading, flight, and token-loop scripts.
• docs/results/ contains committed campaign evidence.

Current Limitations and Roadmap

The active roadmap prioritizes walked attention validation on FPGA, faster timing closure, sustained weight streaming, end-to-end W4 support, additional measured token-rate work, and Sky130 signoff preparation. The live status is maintained in the master table.

Avoid assuming that roadmap items are complete. In particular, the README identifies full walked-attention hardware bring-up and several bandwidth-related dependencies as ongoing work.

Conclusion

APEX is both an RTL implementation of a transformer decoder tile and a case study in evidence-driven hardware verification. Its most distinctive feature is an integrated KV-cache codec that stores keys and values in compressed form while remaining part of the verified attention datapath.

Begin with the golden tests, then run tile and walker verification, inspect the measured evidence, and only proceed to model or FPGA workflows when their additional artifacts are available. Keep measured FPGA results separate from analytic projections, and use the project's traceability documents whenever presenting a claim.

APEX is released under the Apache License 2.0. See LICENSE for the applicable terms and notices.