Run Naive-N0.5-Flash Locally with Transformers

What Is Naive-N0.5-Flash?

Naive-N0.5-Flash is an open-weight Mixture-of-Experts model designed for coding and AI research and development. It contains 309 billion total parameters, with 15.5 billion active parameters, and supports a native context window of up to 1 million tokens.

The model weights and inference code are released under the MIT License. You can find the model on Hugging Face and the source project on GitHub.

Key Features

• Native 1M-token context: The model uses a hybrid of Sliding-Window Attention (SWA) and DeepSeek Sparse Attention (DSA), without full-attention layers.
• MoE efficiency: Although the model has 309B total parameters, only 15.5B are active for each token.
• Coding and AI R&D focus: The model was trained to improve software engineering, research, and systems optimization capabilities.
• FP8 inference: The released checkpoint supports mixed-precision inference on FP8-capable NVIDIA GPUs.
• High-throughput inference: The NaiveRT inference system combines mega-kernel fusion, Programmatic Dependent Launch, and speculative decoding. The README reports up to 50 tokens per second per user in Standard mode and up to 2,000 tokens per second in Ultrafast mode.

Understand the Hybrid Attention Design

Naive-N0.5-Flash uses 48 transformer layers: 39 SWA layers and 9 DSA layers. SWA limits local attention to a 128-token window, while DSA uses an indexer to score the full history and selects the top 2,048 tokens for backbone attention.

The model is organized into eight six-layer modules. A typical module contains five SWA layers followed by one DSA layer, while the first layer of the first module is also replaced with DSA. DSA uses grouped-query attention with four key-value groups, known as GQA4, and its indexer has 16 query heads.

This design keeps most attention computation local while allowing selected information from the long context to influence generation. The full key-value cache is retained, but the backbone attends only to the selected DSA tokens.

Hardware and Installation Requirements

Naive-N0.5-Flash requires FP8-capable NVIDIA GPUs. The model weights occupy approximately 315 GB, and you should allow additional GPU memory for inference.

The README provides a Transformers-based quick start. Install a recent version of Transformers with PyTorch and the required kernels:

pip install "transformers[torch,kernels]>=5.17.0"

Run the Model with Transformers

After installing the dependencies and ensuring that the required GPU resources are available, load the FP8 checkpoint from Hugging Face. The following example creates a tokenizer, loads the model, formats a user message, and generates a response:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NaiveAI/Naive-N0.5-Flash-FP8"
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
)

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}],
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=2048,
)
response = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(response)

How the Example Works

1. Select the checkpoint: NaiveAI/Naive-N0.5-Flash-FP8 identifies the FP8 model on Hugging Face.
2. Load the tokenizer: trust_remote_code=True enables the model repository's custom Transformers code.
3. Load the model: dtype="auto" lets Transformers use the checkpoint's appropriate data type, while device_map="auto" places model components across the available devices.
4. Format the conversation: apply_chat_template converts the user message into the format expected by the model and adds a generation prompt.
5. Generate text: max_new_tokens=2048 limits the number of newly generated tokens.
6. Decode only the response: The slice beginning after the input length removes the original prompt before printing the generated text.

Recommended Generation Settings

For general use, the project recommends setting temperature=1.0 and top_p=0.95. These values are not included in the minimal README example, but you can pass them to generate when you want to make the sampling behavior explicit:

output = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=1.0,
    top_p=0.95,
)

Use a suitable max_new_tokens value for your task. Larger limits can produce longer answers and require more generation time and memory.

Advanced Usage Considerations

Use the Long Context Deliberately

The 1M-token context window is supported natively through the hybrid SWA–DSA architecture. For long codebases, research material, or multi-file tasks, provide the relevant context in a structured way and keep instructions clear. The architecture is designed to preserve local detail through SWA while selecting important history through DSA.

Plan GPU Capacity Before Loading

The checkpoint occupies approximately 315 GB before accounting for additional inference memory. Verify that your FP8-capable NVIDIA GPU setup has sufficient capacity before starting the model. Automatic device placement through device_map="auto" can distribute the model across available devices, but it does not reduce the total memory required by the checkpoint and runtime.

Consider Optimized Inference

The README describes NaiveRT as the project's optimized inference system. Its techniques include mega-kernel fusion, Programmatic Dependent Launch, and speculative decoding. If you need high throughput beyond a basic Transformers deployment, consult the project's technical blog for the NaiveRT case study and implementation details.

Review Benchmark Context

The published evaluations cover coding, software engineering, agentic tasks, AI research, and systems optimization. Results were generally evaluated with Claude Code 2.1.207, a 1M-token context window, temperature 1.0, top-p 0.95, and basic file I/O and Bash tools unless otherwise noted. Treat benchmark comparisons as dependent on their stated harnesses and evaluation protocols.

License and Further Resources

Naive-N0.5-Flash is released under the MIT License. For model releases and related projects, visit the NaiveAI GitHub organization or the NaiveAI Hugging Face profile. Technical background on the architecture, training process, and inference optimizations is available on the project's technical blog.

Conclusion

Naive-N0.5-Flash combines a 309B-parameter MoE design with 15.5B active parameters, hybrid SWA–DSA attention, and a native 1M-token context window. To try it locally, install Transformers with the required kernels, use FP8-capable NVIDIA hardware with roughly 315 GB for the weights, load the Hugging Face checkpoint, and generate responses with the recommended sampling settings.