Yakha futhi Usebenzise Imodeli Yolimi Yomthetho wase-Brazil nge-PyTorch

Yini i-LLM Legislativo Brasileiro?

LLM Legislativo Brasileiro yiphrojekthi ephelele yocwaningo nobunjiniyela yokwakha imodeli yolimi eyimbangela ekhethekile kumadokhumenti omthetho wase-Brazil. Ihlela izinto ezisemthethweni ezivela e-Chamber of Deputies nase-Federal Senate zibe amasethi edatha avulekile, ahlolakalayo, bese iqeqesha i-Transformer decoder ngokuphelele kusukela ekuqaleni nge-PyTorch.

Le phrojekthi ayisebenzisi izisindo eziqeqeshwe kusengaphambili, ama-API avaliwe, noma izendlalelo zokuqeqesha ezisezingeni eliphezulu njenge-Hugging Face Trainer. Idizayini yayo igcizelela isayensi evulekile, ukuphindaphindeka kwemiphumela, ukuhlolwa kobuqotho nge-SHA-256, kanye nokuhlukaniswa okuqinile okuyimbangela phakathi kwedatha yokuqeqesha, yokuqinisekisa, neyokuhlola.

Le modeli iyinto yocwaningo neyokufundisa. Akufanele ithathwe njengomeluleki wezomthetho onikeza amaqiniso noma njengethuluzi lezinqumo ezijwayelekile, ngoba amamodeli olimi ayimbangela angakha imithetho, izichibiyelo, izifinyezo, kanye nezinhlinzeko zomthetho ezingelona iqiniso.

Izici Eziyinhloko

• Ngaphezu kuka-360 GiB wedatha eluhlaza esemthethweni, okuhlanganisa amaqoqo e-Chamber nase-Senate.
• Ngaphezu kwamadokhumenti angu-930,000 nezinguqulo ezibalulekile, njengezikweletu zemithetho, izichibiyelo zomthethosisekelo, imibiko, imibono, imibhalo eshintshile, imibhalo yokugcina, namarekhodi ezikhathi zeseshini.
• Ukukhishwa kwamasethi edatha omphakathi amahlanu ku-Hugging Face Hub, okuhlanganisa izingobo zomlando eziluhlaza, umbhalo okhishiwe, nedatha ye-Apache Parquet ekhethiwe.
• I-tokenizer ye-byte-level BPE enama-token angu-8,192, eqeqeshwe kuphela kusigaba sokuqeqesha.
• I-Transformer decoder eyimbangela ecacile eyenziwe nge-PyTorch, enezendlalelo eziyisithupha, amakhanda okunaka ayisithupha, ukumelwa okufihliwe okunobukhulu obungu-384, kanye newindi lomongo lama-token angu-512.
• Ukubopha izisindo phakathi kwezendlalelo zokushumeka ama-token kanye nezendlalelo zokuphrojetha okukhiphayo.
• Ukuqoqwa okunembile kwama-gradient okusekelwe enanini eliphelele lama-target avumelekile, okugwema amaphutha esisindo ahlobene nokugcwalisa phakathi kokulandelana okungafani.
• Ukuhlukaniswa kwedatha okunqunyiwe okungu-80/10/10 okusekelwe kuzihlonzi zemindeni yezomthetho, okuvimbela amadokhumenti ahlobene ukuthi eqe imingcele phakathi kwedatha yokuqeqesha neyokuhlola.
• Izisindo zemodeli zomphakathi ngefomethi ye-safetensors, kanye ne-tokenizer, ukumiswa, amamethrikhi, nekhodi ezimele yokwenza inference.

Qonda Idatha Nomjikelezo Wokuqeqesha

Indawo yokugcina ikhodi ihlukanisa iphrojekthi ibe yizigaba ezikhethekile. Uhlu lwezinto ezisemthethweni lwe-Chamber ne-Senate luyaqoqwa futhi ludawunilodwe ngezinqubo eziqinile nezingaphindi umsebenzi. Amafayela aluhlaza agcinwa ngaphandle kwe-Git, kuyilapho imethadatha elula, ama-schema, namafayela okumisa kugcinwa njengenguqulo.

Kuma-macOS, i-adapter yomdabu yokukhipha eyenziwe nge-Swift isebenzisa i-PDFKit kumadokhumenti anombhalo oshumekiwe. Amakhasi askeniwe noma amakhasi anobuningi bombhalo obunganele adluliselwa ku-Apple Vision OCR. Umbhalo ovelayo ube usukhethwa futhi ulungiswe ku-R kusetshenziswa ukususa okuphindaphindwayo okuqondile nge-SHA-256, ukutholwa kokufakwa kweziqephu komunye komunye, kanye negrafu yomhlaba wonke yemindeni yezomthetho eyakhiwe nge-igraph.

Isiphakamiso ngasinye, isichibiyelo, umbiko, umbono, nombhalo oshintshile oxhumene nenqubo efanayo yomthetho uthola i-family_id efanayo. I-hashi enqunyiwe yaleyo nhlonzi yabela wonke umndeni ekuqeqesheni, ekuqinisekiseni, noma ekuhlolweni. Lokhu kuvimbela izinguqulo ezihlobene ukuthi zivele ngokungahleliwe kuzo zonke izigaba zokuhlola.

Ngemva kokuhlukanisa, isigaba se-Python sokwenza ama-token siqeqesha ulwazimagama lwe-byte-level BPE olunama-token angu-8,192 sisebenzisa idatha yokuqeqesha kuphela. Sisekela ama-accents esiPutukezi, izimpawu zomthetho ezifana no-§, º, ª, kanye no-Art., kanye nezinombolo nezinye izinhlamvu ezikhethekile. Idatha enama-token ibhalwa kuma-shard kanambambili anama-index amawindi ama-token angu-512.

Lungisa I-Repository

1. Klona iphrojekthi

I-repository idinga i-Python 3.9 noma entsha. Yiklone bese ungene enkombeni yephrojekthi:

git clone https://github.com/BaruqueRodrigues/llm-legislativo.git
cd llm-legislativo

2. Dala indawo ebonakalayo

Ukusebenzisa indawo ebonakalayo kugcina iz依dingo zephrojekthi zihlukanisiwe nakho konke okunye ohlelweni lwakho:

python3 -m venv .venv
source .venv/bin/activate

3. Faka iphakheji

Faka i-repository ngemodi ehlelekayo ukuze izinguquko ezenziwe kukhodi yasendaweni zitholakale ngokushesha:

pip install -e .

I-repository iqukethe iphakheji ye-Python ngaphansi kwe-src/legislm/, ama-lockfile ezindawo eziyisisekelo ne-CPU, imethadatha yokuphindaphinda ye-R, ukumiswa okumemezelayo, ama-notebook, imibhalo yomugqa womyalo, kanye nesethi yokuhlola.

Qinisekisa Ukufakwa

Ngaphambi kokuzama ukwenza inference yemodeli noma ukuqeqesha ngezinga elikhulu, sebenzisa izivivinyo ezizenzakalelayo endaweni yakho ku-CPU:

pytest tests/ -q

I-README ibika izivivinyo ezingaphezu kuka-135 zamayunithi nezinkontileka ezimboza ukuziphatha kwemodeli, ukugcinwa, ukwenza ama-token, nokuhlola. Ukuqalisa ngempumelelo kuqinisekisa ukuthi izixhumi ezibalulekile ziyasebenza ngaphandle kokudinga i-corpus ephelele enamasayithi amaningi wamakhulu ama-gigabyte noma i-A100 GPU.

Layisha Imodeli Ekhishiwe

Imodeli eqeqeshiwe iyatholakala ku-repository ye-BaruqRodrigues/llm-legislativo. Isibonelo esilandelayo sidawuniloda izisindo ne-tokenizer, sakhe kabusha i-architecture, bese silayisha i-state dictionary ye-safetensors.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from legislm.model.gpt import GPTModel
from legislm.model.config import ModelConfig
from tokenizers import Tokenizer

# Download the model artifacts
weights_path = hf_hub_download(
    repo_id="BaruqRodrigues/llm-legislativo",
    filename="model.safetensors"
)
tokenizer_path = hf_hub_download(
    repo_id="BaruqRodrigues/llm-legislativo",
    filename="tokenizer.json"
)

# Recreate the tokenizer and model configuration
tokenizer = Tokenizer.from_file(tokenizer_path)
config = ModelConfig(
    vocab_size=8192,
    context_length=512,
    d_model=384,
    n_layers=6,
    n_heads=6,
    tie_embeddings=True
)
model = GPTModel(config)

# Load the trained parameters
state_dict = load_file(weights_path)
model.load_state_dict(state_dict)
model.eval()

Khiqiza Umbhalo Wezomthetho

Uma imodeli isilayishiwe, faka i-prompt yezomthetho yesiPutukezi bese ubiza indlela yokukhiqiza yemodeli. Isibonelo sisebenzisa isiqalo esifushane somthethosivivinywa futhi sicela amathokheni amasha afinyelela kwangu-64.

prompt = "PROJETO DE LEI Nº , DE 2026\n\nDispõe sobre"
input_ids = torch.tensor(
    [tokenizer.encode(prompt).ids],
    dtype=torch.long
)

with torch.no_grad():
    output_ids = model.generate(
        input_ids,
        max_new_tokens=64,
        temperature=0.7,
        top_k=40,
        eos_token_id=2
    )

print(tokenizer.decode(output_ids[0].tolist()))

Imodeli isebenzisa ubude obukhulu bomongo obungamathokheni angu-512. Kulesi sibonelo, temperature=0.7 ilawula ukuhluka kokusampula, kanti top_k=40 ikhawulela ukusampula kubalingiswa abangu-40 abanamathuba aphezulu kakhulu. Lungisa la manani lapho uzama, kodwa khumbula ukuthi umbhalo okhiqiziwe uwumphumela wezibalo, hhayi okuqukethwe kwezomthetho okuqinisekisiwe.

Hlola Amasethi Edatha Omphakathi

Iphrojekthi ishicilela izigaba eziningana zekhopasi ku-Hugging Face Hub:

• corpus-legislativo-pretraining: ukukhishwa okuhlanganisiwe nokukhethiwe ngefomethi ye-Parquet, okunamaqoqo okuqeqesha, okuqinisekisa, nokuhlola ahlelwe ngokomndeni.
• camara-corpus-raw: izingobo zomlando ze-TAR ezingakacutshungulwa zeChamber of Deputies kanye namamanifesti e-SHA-256.
• camara-corpus-text: ukukhishwa kwemibhalo ehlelekile yeChamber kanye nokuqinisekiswa kwekhwalithi.
• senado-corpus-raw: uhlu lwezinto zedatha ye-Federal Senate kanye nedatha eyisisekelo.
• senado-corpus-text: imibhalo namakhasi eSenate akhethiwe ngefomethi ye-Parquet esekelwe kumakholomu.

Lokhu kukhishwa kwenza kube nokwenzeka ukuhlola iphrojekthi ezigabeni ezihlukene zenqubo, kusukela ezingobozomlando zemithombo nasembhalweni okhishiwe kuya kumatrices okuqeqesha okugcina.

Phinda Ukhiqize Noma Wandise Ukuqeqeshwa

Ukusetshenziswa kokuqeqesha kutholakala ku-src/legislm/training/, kanti ukwenziwa kwamathokheni nokukhiqizwa kwezingcezu kutholakala ngaphansi kwe-src/legislm/tokenization/. Ukucushwa kugcinwe ku-configs/, futhi izinqubo ze-Google Colab zinikezwe ku-notebooks/ naku-src/legislm/colab/.

Ukuqeqeshwa okushicilelwe kwasebenzisa indawo ye-NVIDIA A100-SXM4-40GB futhi kwaqeda izibuyekezo zokwenza kahle ezingu-173,209 cishe emahoreni angu-20. Kwasetshenzwa cishe amathokheni ayizigidi eziyizinkulungwane ezingu-1.25 kusetshenziswa i-AdamW, isihleli se-cosine esine-linear warmup yezinyathelo ezingu-2,000, ukunemba okuxubile kwe-FP16, kanye nokugcinwa kwama-checkpoint nge-Google Drive.

Ukuze uthole izivivinyo ezingaphindwa ngendlela efanayo, gcina i-tokenizer ikhawulelwe engxenyeni yokuqeqesha, londoloza inqubo yokuhlukanisa enqunywe kusengaphambili esekelwe emindenini, futhi ugcine amamanifesti e-SHA-256 kanye nemethadatha yama-checkpoint e-repository. Le mininingwane ibalulekile ekugwemeni ukuvuza kwedatha nasekuqhathaniseni imiphumela ngendlela enobulungiswa.

Hlola Imiphumela Ngokufanele

Iphrojekthi ichaza umthetho olandelwayo ojwayelekile wokuhlola othi byte_nll_v1 engxenyeni yokuhlola ezimele enemibhalo engu-71,069 kanye namabhayithi acishe abe yizigidi ezingu-482. Imiphumela ebikiwe ihlanganisa ukulahleka okumaphakathi okungu-2.1065 ngethagethi evumelekile, i-token perplexity engu-8.2197, kanye ne-word perplexity engu-65.6509.

Umphumela ojwayelekile osezingeni lamabhayithi ungama-nats angu-0.7297 ngebhayithi, noma amabhithi angu-1.0528 ngebhayithi. Amanani abikiwe angama-nats angu-0.7049 ngebhayithi eqoqweni leChamber kanye no-0.8817 eqoqweni leSenate. Lapho uqhathanisa izivivinyo zesikhathi esizayo, sebenzisa umqondo ofanayo wokuhlukanisa kanye nencazelo yokuhlola, kunokuthembela kuphela ku-token perplexity.

Ukwakheka Kwe-repository

Izinkomba ezinkulu zicacisa ukuthi kufanele ubheke kuphi lapho ulungisa noma ufunda uhlelo:

• R/: uhlu olusemthethweni lwedatha, ukulanda, ukuhlanganiswa kokukhipha, kanye nokukhetha.
• src/legislm/: ikhodi ye-tokenizer, imodeli, ukuqeqesha, ukugcinwa kwe-Colab, kanye nokuhlola.
• schemas/: izinkontileka zedatha ze-Arrow ne-JSON Schema.
• configs/: ukucushwa kwedatha, ukukhetha, ukuqeqesha, kanye ne-Colab.
• notebooks/: izinqubo ze-Google Colab zokuhlola nezokugcwele.
• scripts/: izindawo zokungena ezisebenzisekayo zomugqa womyalo.
• tests/: ukuhlolwa kwamayunithi nezinkontileka.
• docs/: imibhalo evamile, izinkontileka, kanye nemibiko.

Amathiphu Athuthukile Asebenzayo

• Qala ngezivivinyo: Ukuqinisekisa i-CPU kuyindlela engabizi yokuqinisekisa ukufakwa ngaphambi kokulanda amasethi edatha amakhulu noma ukucela izinsiza ze-GPU.
• Gcina imingcele yemindeni: ungalokothi udale ukuhlukaniswa okungahleliwe ezingeni lamadokhumenti uma ufuna imiphumela eqhathaniseka nephrojekthi, ngoba izinguqulo zenqubo eyodwa yomthetho zingahlobana kakhulu.
• Qeqesha i-tokenizer ngemva kokuhlukanisa: ukubala izibalo zesilulumagama ngedatha yokuqinisekisa noma yokuhlola kungaphazamisa izilawuli zephrojekthi ezivimbela ukuvuza kwedatha.
• Landela okuhlosiwe okusemthethweni: uma usebenzisa ukulandelana okufakwe ama-padding, qoqela i-cross-entropy ngenani eliphelele lezinto okuhlosiwe ezisemthethweni kuwo wonke ama-micro-batch.
• Gcina ama-artifact ngaphandle kwe-Git: inqolobane igcina ngamabomu imethadatha nama-schema esikhundleni samabhanari aluhlaza, izisindo zamamodeli, nama-cache.
• Sebenzisa ukukhishwa komphakathi ukuze uhlolisise: amasethi edatha e-Hub aluhlaza, akhishiwe, nahleliwe akuvumela ukuthi uphenye izigaba ezithile zepayiphi ngaphandle kokwakha kabusha yonke inqubo yokuqoqwa.
• Humusha imiphumela ekhiqiziwe ngokuqaphela: ulimi lomthetho olukhiqiziwe lungabukeka luthembekile kodwa luqukethe izinkomba noma izinhlinzeko eziqanjwe ngamadlebe.

Isiphetho

LLM Legislativo Brasileiro ikhombisa indlela ephelele, ehlolisisekayo esuka ezinqolobaneni zomthetho ezisemthethweni iye kumodeli yolimi eyimbangela ekhethekile. Ngemva kokufaka iphakheji nokusebenzisa izivivinyo zayo, ungakwazi ukulayisha i-tokenizer ekhishiwe kanye nemodeli ye-PyTorch, ukhiqize umbhalo, uhlole amasethi edatha omphakathi, futhi ufunde izindlela zokuhlela nokuqeqesha zenqolobane ezimelana nokuvuza kwedatha. Ikhodi yayo enelayisensi ye-MIT nama-artifact ashicilelwe obala kunikeza isisekelo esiwusizo socwaningo lokumodela ulimi lomthetho lwesiPutukezi.