Build and Run a Brazilian Legislative Language Model with PyTorch

What Is LLM Legislativo Brasileiro?

LLM Legislativo Brasileiro is an end-to-end research and engineering project for building a causal language model specialized in Brazilian legislative documents. It organizes official material from the Chamber of Deputies and Federal Senate into auditable open datasets, then trains a Transformer decoder entirely from scratch in PyTorch.

The project does not use pretrained weights, closed APIs, or high-level training abstractions such as the Hugging Face Trainer. Its design emphasizes open science, reproducibility, SHA-256 integrity checks, and strict causal isolation between training, validation, and test data.

This model is a research and educational artifact. It should not be treated as a factual legal adviser or a tool for normative decisions, because causal language models can hallucinate laws, amendments, summaries, and legal provisions.

Key Features

• More than 360 GiB of official raw data, including Chamber and Senate collections.
• More than 930,000 documents and substantive versions, such as bills, constitutional amendments, reports, opinions, substitutes, final texts, and session records.
• Five public dataset releases on the Hugging Face Hub, covering raw archives, extracted text, and curated Apache Parquet data.
• An 8,192-token byte-level BPE tokenizer trained exclusively on the training split.
• An explicit causal Transformer decoder implemented in PyTorch, with six layers, six attention heads, a 384-dimensional hidden representation, and a 512-token context window.
• Weight tying between the token embedding and output projection layers.
• Exact gradient accumulation based on the total number of valid targets, which avoids padding-related weighting errors across heterogeneous sequences.
• Deterministic 80/10/10 data splits based on legislative family identifiers, preventing related documents from crossing training and test boundaries.
• Public model weights in safetensors format, alongside the tokenizer, configuration, metrics, and standalone inference code.

Understand the Data and Training Pipeline

The repository separates the project into specialized stages. Official Chamber and Senate inventories are collected and downloaded through resilient, idempotent processes. Raw files remain outside Git, while lightweight metadata, schemas, and configuration files are versioned.

On macOS, a native Swift extraction adapter uses PDFKit for documents with embedded text. Scanned pages or pages with insufficient text density are routed to Apple Vision OCR. The resulting text is then curated in R using exact SHA-256 deduplication, passage-containment detection, and a global legislative-family graph built with igraph.

Each proposition, amendment, report, opinion, and substitute connected to the same legislative process receives the same family_id. A deterministic hash of that identifier assigns the complete family to training, validation, or test. This prevents related versions from leaking across evaluation splits.

After partitioning, the Python tokenization stage trains a byte-level BPE vocabulary of 8,192 tokens using only training data. It supports Portuguese accents, legislative symbols such as §, º, ª, and Art., as well as numerals and other special characters. Tokenized data is written to binary shards with indexes for 512-token windows.

Set Up the Repository

1. Clone the project

The repository requires Python 3.9 or newer. Clone it and enter the project directory:

git clone https://github.com/BaruqueRodrigues/llm-legislativo.git
cd llm-legislativo

2. Create a virtual environment

Using a virtual environment keeps the project dependencies isolated from the rest of your system:

python3 -m venv .venv
source .venv/bin/activate

3. Install the package

Install the repository in editable mode so that local source changes are immediately available:

pip install -e .

The repository includes a Python package under src/legislm/, lockfiles for base and CPU environments, R reproducibility metadata, declarative configurations, notebooks, command-line scripts, and a test suite.

Validate the Installation

Before attempting model inference or large-scale training, run the automated tests locally on CPU:

pytest tests/ -q

The README reports more than 135 unit and contract tests covering model behavior, persistence, tokenization, and evaluation. A successful run checks that the core interfaces are working without requiring the full multi-hundred-gigabyte corpus or an A100 GPU.

Load the Released Model

The trained model is available from the BaruqRodrigues/llm-legislativo repository. The following example downloads the weights and tokenizer, reconstructs the architecture, and loads the safetensors state dictionary.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from legislm.model.gpt import GPTModel
from legislm.model.config import ModelConfig
from tokenizers import Tokenizer

# Download the model artifacts
weights_path = hf_hub_download(
    repo_id="BaruqRodrigues/llm-legislativo",
    filename="model.safetensors"
)
tokenizer_path = hf_hub_download(
    repo_id="BaruqRodrigues/llm-legislativo",
    filename="tokenizer.json"
)

# Recreate the tokenizer and model configuration
tokenizer = Tokenizer.from_file(tokenizer_path)
config = ModelConfig(
    vocab_size=8192,
    context_length=512,
    d_model=384,
    n_layers=6,
    n_heads=6,
    tie_embeddings=True
)
model = GPTModel(config)

# Load the trained parameters
state_dict = load_file(weights_path)
model.load_state_dict(state_dict)
model.eval()

Generate Legislative Text

Once the model is loaded, encode a Portuguese legislative prompt and call the model's generation method. The example uses a short bill opening and requests up to 64 new tokens.

prompt = "PROJETO DE LEI Nº , DE 2026\n\nDispõe sobre"
input_ids = torch.tensor(
    [tokenizer.encode(prompt).ids],
    dtype=torch.long
)

with torch.no_grad():
    output_ids = model.generate(
        input_ids,
        max_new_tokens=64,
        temperature=0.7,
        top_k=40,
        eos_token_id=2
    )

print(tokenizer.decode(output_ids[0].tolist()))

The model uses a maximum context length of 512 tokens. In this example, temperature=0.7 controls sampling variability and top_k=40 limits sampling to the 40 highest-probability candidates. Adjust these values when experimenting, while remembering that generated text is statistical output rather than verified legal content.

Explore the Public Datasets

The project publishes several stages of the corpus on the Hugging Face Hub:

• corpus-legislativo-pretraining: a unified curated release in Parquet format with train, validation, and test splits organized by family.
• camara-corpus-raw: raw Chamber of Deputies TAR archives and SHA-256 manifests.
• camara-corpus-text: structured Chamber text extractions with quality verification.
• senado-corpus-raw: Federal Senate inventories and primary data.
• senado-corpus-text: curated Senate documents and pages in columnar Parquet format.

These releases make it possible to inspect the project at different points in the pipeline, from source archives and extracted text to the final training matrices.

Reproduce or Extend Training

The training implementation is located in src/legislm/training/, while tokenization and shard generation are under src/legislm/tokenization/. Configurations are stored in configs/, and Google Colab workflows are provided in notebooks/ and src/legislm/colab/.

The published training run used an NVIDIA A100-SXM4-40GB environment and completed 173,209 optimization updates over approximately 20 hours. It processed about 1.25 billion tokens using AdamW, a cosine scheduler with a 2,000-step linear warmup, mixed FP16 precision, and checkpoint persistence through Google Drive.

For reproducible experiments, keep the tokenizer restricted to the training split, preserve the deterministic family-based split procedure, and retain the repository's SHA-256 manifests and checkpoint metadata. These details are essential for avoiding data leakage and for comparing results fairly.

Evaluate Results Correctly

The project defines a standardized byte_nll_v1 evaluation protocol on an independent test split containing 71,069 documents and approximately 482 million bytes. Reported results include 2.1065 average loss per valid target, 8.2197 token perplexity, and 65.6509 word perplexity.

The normalized byte-level result is 0.7297 nats per byte, or 1.0528 bits per byte. The reported values are 0.7049 nats per byte for the Chamber collection and 0.8817 for the Senate collection. When comparing future experiments, use the same split logic and evaluation definition rather than relying on token perplexity alone.

Repository Layout

The main directories clarify where to look when modifying or studying the system:

• R/: official data inventory, downloads, extraction integration, and curation.
• src/legislm/: tokenizer, model, training, Colab persistence, and evaluation code.
• schemas/: Arrow and JSON Schema data contracts.
• configs/: data, curation, training, and Colab configurations.
• notebooks/: pilot and full Google Colab workflows.
• scripts/: executable command-line entry points.
• tests/: unit and contract tests.
• docs/: normative documentation, contracts, and reports.

Advanced Practical Tips

• Start with the tests: CPU validation is a low-cost way to confirm the installation before downloading large datasets or requesting GPU resources.
• Preserve family boundaries: never create random document-level splits if you want results comparable to the project, because versions from one legislative process can be closely related.
• Train the tokenizer after splitting: fitting vocabulary statistics on validation or test data would compromise the project's leakage controls.
• Track valid targets: when using padded sequences, accumulate cross-entropy by the total number of valid targets across micro-batches.
• Keep artifacts outside Git: the repository intentionally versions metadata and schemas rather than raw binaries, model weights, and caches.
• Use the public releases for inspection: the raw, extracted, and curated Hub datasets let you investigate individual pipeline stages without recreating the entire collection process.
• Interpret generations cautiously: generated legislative language may look plausible while containing fabricated references or provisions.

Conclusion

LLM Legislativo Brasileiro demonstrates a complete, auditable route from official legislative archives to a specialized causal language model. After installing the package and running its tests, you can load the released tokenizer and PyTorch model, generate text, inspect the public datasets, and study the repository's leakage-resistant curation and training methods. Its MIT-licensed code and openly published artifacts provide a useful foundation for research into Portuguese legislative language modeling.