Isifundo Sokuqala se-NamiGPT: Bhala i-GPT-2 Kusukela Ekuqaleni futhi Ukhiqize Umbhalo

Yini i-NamiGPT

I-NamiGPT iwuchungechunge lwezifundo eziyisisekelo zamamodeli amakhulu, oluqondiswe kwabaqalayo abakhuluma isiShayina. Isebenzisa i-PyTorch ukwakha i-GPT-2 kusukela ekuqaleni, ngaphandle kokusebenzisa amamojuli e-Transformer akhona, futhi ingalayisha izisindo eziqeqeshwe kusengaphambili ezisemthethweni ze-OpenAI ukuze yenze inference nokukhiqiza umbhalo.

Le phrojekthi ayibeki ukusebenza kahle kokuqeqesha njengomgomo oyinhloko, kodwa igcizelela ukuqonda: imodeli eyinhloko iku-gpt2/model.py, futhi ikhodi ingaphansi kwemigqa engu-200; isifundo sisebenzisa isibonelo samagama amathathu esithi “猫吃鱼” njengomkhondo, sichaze kancane kancane i-tokenization, ama-vector, i-attention matrix kanye nenqubo yokukhiqiza yokugcina.

Igama elithi NamiGPT lihlonipha i-nanoGPT kaKarpathy. Ligxile kakhulu ekufundiseni abaqalayo ngokuphelele ngesiShayina, lichaze isinyathelo ngasinye se-GPT-2 lisebenzisa izibonelo ezincane.

Ozokufunda

• Indlela i-Tokenizer eguqula ngayo umbhalo ube ama-token ID.
• Indlela i-Embedding kanye ne-position encoding akha ngayo okokufaka kwemodeli.
• Indlela i-Self-Attention ne-Multi-Head Attention okubalwa ngayo.
• Imisebenzi ye-residual connection, i-LayerNorm, i-FFN kanye nama-activation function.
• Indlela ama-Transformer Block amaningi ahlanganiswa ngayo ukuze akhe i-GPT-2.
• Indlela yokulayisha izisindo ezisemthethweni ze-OpenAI kumodeli eyakhiwe ngesandla.
• Indlela yokusebenzisa amasu e-greedy, temperature ne-top-k ukukhiqiza umbhalo.
• Indlela yokusebenzisa izivivinyo ukuqinisekisa ukuthi ukwakhiwa ngesandla kuyahambisana nokwakhiwa kwe-HuggingFace.

Izici eziyinhloko zephrojekthi

Ukwakha izingxenye eziyinhloko kusukela ekuqaleni

I-NamiGPT ayibizi ngokuqondile amamojuli e-Transformer akhona. Umzimba wemodeli uqukethe i-LayerNorm, i-Attention, i-FFN, i-Block kanye ne-GPT, ngakho kulula ukufunda umugqa nomugqa ukuthi idatha nama-tensor kuhamba kanjani.

Imibhalo yezahluko ingasebenza ngokuzimela

Ingxenye ye-inference yakhiwe ngezihloko ezingu-15 ezikhuphuka kancane ngobunzima, ezihlanganisa i-tokenizer, i-Embedding, i-position encoding, i-self-attention, i-multi-head attention, i-residual connection, i-layer normalization, i-feedforward network, ama-activation function, i-Transformer Block, ukuhlanganisa, ukulayisha izisindo, ukukhiqiza umbhalo kanye nokuchazeka kwemodeli. Isihloko ngasinye sokufundisa sihambisana nombhalo osebenzayo.

Inikeza ukubonwa okusebenzisanayo

Le phrojekthi inikeza ukubonwa okuguqukayo kwesakhiwo se-Transformer kanye nokubonwa okuphelele kwama-matrix e-inference. Uma ikhodi noma amafomula kunzima ukuwaqonda, ungabuka indlela izakhi ze-matrix esibonelweni esithi “猫吃鱼” ezibalwa ngayo isikwele ngesikwele.

Kunconywa ukuthi uqale ubuke i-Transformer echazwe ngesithombe esisodwa, bese usebenzisa ukubonwa kwe-inference kwemodeli enkulu ukuze uqhathanise nekhodi yezahluko.

Ukuhambisana nemiphumela ye-HuggingFace

Indawo yokugcina ikhodi iqukethe ukuhlolwa kokuma kanye nezivivinyo zokuqhathanisa nokwakhiwa okusemthethweni kwe-HuggingFace. Ngaphandle kokuqhathanisa ngokushesha kusetshenziswa izisindo ezingahleliwe, ungadawuniloda nezisindo zangempela ze-GPT-2 ukuze wenze okunye ukuqinisekisa.

Okumele kulungiswe ngaphambi kokufakwa

Ngaphambi kokuqala, kudingeka ube nolwazi oluyisisekelo lokufunda i-Python futhi uqonde imiqondo eyisisekelo ye-linear algebra, njengokuphindaphinda kwama-matrix. Indawo yokusebenza ephawuliwe yephrojekthi yi-Python 3.9 noma ngaphezulu kanye ne-PyTorch 2.x.

Qala ngokuklona indawo yokugcina ikhodi, ungene enkombeni yephrojekthi bese ufaka okuncikile:

git clone https://github.com/aa1143/NamiGPT.git
cd NamiGPT
pip install -r requirements.txt

Uma inethiwekhi ikhawulela ukudawuniloda izisindo ezisemthethweni eziku-HuggingFace, ungabeka ikheli le-mirror kule terminal yamanje:

export HF_ENDPOINT=https://hf-mirror.com

Le mirror isetshenziselwa kuphela ukudawuniloda izisindo ezisemthethweni. Lapho ubiza from_pretrained kamuva, isakhiwo semodeli sisazoqhubeka sivela ekwakhiweni ngesandla kwe-NamiGPT.

Isinyathelo sokuqala: ukuqonda i-GPT-2 ngezahluko

Indlela efanele kakhulu kwabaqalayo akukhona ukufunda imodeli ephelele ngokushesha, kodwa ukufunda isahluko ngesahluko ngokulandela ukuhamba kwedatha ye-inference:

1. Isigaba sokufaka: qala ngokuqonda i-Tokenizer, i-Embedding kanye ne-position encoding.
2. Isigaba se-attention: qonda i-Self-Attention, bese udlulela ku-Multi-Head Attention.
3. Isigaba samamojuli: funda i-residual connection, i-LayerNorm, i-FFN kanye nama-activation function.
4. Isigaba semodeli: hlanganisa izingxenye zibe yi-Transformer Block, bese uwahlanganisa ukuze wakhe i-GPT-2 ephelele.
5. Isigaba sokusebenzisa: layisha izisindo ezisemthethweni, khiqiza umbhalo bese ubuka imiphumela echazekayo.

Umbhalo ngamunye ungasebenza ngokuzimela. Isibonelo, sebenzisa isahluko se-multi-head attention:

python chapters/01-inference/ep06_multi_head_attention.py

Okuqukethwe okuphelele okunemifanekiso nombhalo kungatholakala ku-isifundo esinemifanekiso nesombhalo se-NamiGPT, kanti umbhalo wokuqala osendaweni yokugcina ikhodi uku-docs/01-inference/.

Isinyathelo sesibili: layisha i-GPT-2 bese ukhiqiza umbhalo

Ngemva kokufaka, ungangenisa ngokuqondile izixhumanisi zokulayisha nokukhiqiza ezinikezwa yi-NamiGPT. Isibonelo esingezansi sisebenzisa i-GPT-2 tokenizer, sifake i-prompt yesiNgisi kuma-token ID, bese silayisha imodeli eyakhiwe ngesandla kanye nezisindo ezisemthethweni ezingu-124M:

import tiktoken, torch
from gpt2 import from_pretrained, generate

enc = tiktoken.get_encoding("gpt2")
model = from_pretrained("gpt2")

ids = torch.tensor([enc.encode("The cat sat on the")])
out = generate(
    model,
    ids,
    max_new_tokens=30,
    temperature=0.8,
    top_k=50,
)
print(enc.decode(out[0].tolist()))

Ukuhamba kokwenziwa kwekhodi

1. tiktoken.get_encoding("gpt2") ithola i-encoder esetshenziswa yi-GPT-2.
2. enc.encode(...) iguqula umbhalo we-prompt ube ama-token ID.
3. torch.tensor isonga ama-ID abe yi-batch tensor edingwa yimodeli.
4. from_pretrained("gpt2") idala i-GPT-2 eyakhiwe ngesandla bese ilayisha izisindo ezisemthethweni ezingu-124M.
5. generate ikhiqiza ama-token amasha afinyelela kwangu-30 ngemva kombhalo we-prompt.
6. enc.decode ibuyisela ama-token ID aphumayo abe umbhalo.

Qaphela: I-GPT-2 ayizange ilungiswe ngokukhethekile ezingxoxweni; inekhono lokuqhubeka nokubhala okuqukethwe okukhona, hhayi ukuphendula imibuzo njengemodeli yengxoxo. Idatha yayo yokuqeqesha igxile kakhulu esiNgisini, ngakho kunconywa ukuthi uqale usebenzise ama-prompt esiNgisi.

Isinyathelo sesithathu: ukuqonda amapharamitha okukhiqiza

Isibonelo sisebenzisa amapharamitha amabili abalulekile okusampula:

• temperature=0.8: isetshenziselwa ukulungisa ukusatshalaliswa kwamathuba ama-token aqokiwe. Imojuli yokukhiqiza yephrojekthi isekela isu le-temperature.
• top_k=50: ikhetha kuphela kuma-token aqokiwe angu-50 anamathuba aphezulu. Iphrojekthi iphinde isekele ukukhiqiza kwe-greedy nokukhiqiza kwe-top-k.

Ngesikhathi sokufunda, ungagcina umbhalo we-prompt ungashintshiwe, ushintshe kuphela i-temperature noma i-top-k, bese uqhathanisa imiphumela yokuqhubeka kombhalo. Lokhu kukubonisa ngokucacile ukuthi isu lokukhiqiza liwuthinta kanjani umphumela, kodwa amapharamitha athile kufanele akhethwe ngokwenhloso yokuhlola kwakho.

Isinyathelo sesine: qala idemo yewebhu ye-Gradio

Uma ufuna ukufaka ama-prompt esipheqululini bese ubuka imiphumela yokuqhubeka kombhalo namathuba amagama aqokiwe esinyathelweni ngasinye, faka i-Gradio bese uqala uhlelo lwedemo olusendaweni yokugcina ikhodi:

pip install gradio
python demo/app.py

Idemo yewebhu ilungele ukubuka izibikezelo zemodeli esinyathelweni ngasinye, kanti imibhalo yezahluko ilungele kakhulu ukutadisha inqubo yokubala yengxenye ngayinye. Uma kusetshenziswa kokubili, ungaxhumanisa umbhalo wokugcina nezinguquko zamathuba angaphakathi.

Isinyathelo sesihlanu: sebenzisa izivivinyo

Kunconywa ukuthi usebenzise izivivinyo ngemva kokushintsha i-attention, ukulayisha izisindo noma ingqondo yokukhiqiza. Ngokuzenzakalela, izivivinyo zisebenzisa izisindo ezingahleliwe ukuze ziqhathanise, azidingi ukudawuniloda imodeli eqeqeshwe kusengaphambili, futhi ngokuvamile ziphela ngokushesha:

pytest tests/ -v

Uma ufuna nokusebenzisa izisindo zangempela ze-GPT-2 ukuze wenze okunye ukuqondanisa, sebenzisa:

TEST_PRETRAINED=1 pytest tests/ -v

Umyalo wesibili uzodawuniloda izisindo zangempela ezengeziwe, ngakho okokuqala kungadingeka ulinde kuze kuqedwe ukudawuniloda.

Ukubuka ngokushesha isakhiwo sendawo yokugcina ikhodi

• gpt2/config.py: ukumiswa kwemodeli, okuhlanganisa osayizi abangu-124M, 355M, 774M kanye no-1.5B.
• gpt2/model.py: i-LayerNorm, i-Attention, i-FFN, i-Block kanye nomzimba wemodeli ye-GPT.
• gpt2/load_weights.py: ilayisha izisindo ezisemthethweni kumodeli eyakhiwe ngesandla.
• gpt2/generate.py: ingqondo yokukhiqiza kwe-greedy, temperature ne-top-k.
• chapters/01-inference/: imibhalo yedemo ehambisana nezifundo ezahlukene.
• docs/01-inference/: imibhalo yokuqala yezifundo.
• demo/app.py: idemo yewebhu ye-Gradio.
• tests/: ukuhlolwa kokuma kanye nokuhlolwa kokuqondanisa ne-HuggingFace.

Amasu athuthukile: imininingwane emithathu ebalulekile yokuqondanisa ne-HuggingFace

1. Sebenzisa kahle i-causal mask

I-GPT-2 iyimodeli yolimi ye-autoregressive. Igama ngalinye esilandeleni lingafunda kuphela lona ngokwalo namagama angaphambi kwalo, alikwazi ukubona ama-token alandelayo kusenesikhathi. Uma i-causal mask ishiywe, imiphumela yemodeli izoba nephutha noma ngabe izisindo ezisemthethweni zilayishwe ngempumelelo.

2. Phatha i-transpose lapho uthutha izisindo

Ukwakhiwa kwasekuqaleni kwe-GPT-2 kusebenzisa i-Conv1D, enokuma kwezisindo okungu-[in, out]; i-nn.Linear ye-PyTorch isebenzisa [out, in]. Ngakho-ke, lapho ulayisha amapharamitha asemthethweni kungqimba olwakhiwe ngesandla, kufanele wenze i-transpose.

Ngaphezu kwalokho, i-c_attn ihlanganisa ndawonye amapharamitha e-Query, Key kanye ne-Value. Lapho uqondanisa nokwakhiwa ngesandla, kufanele uqale uwahlukanise abe izingxenye ezintathu ezihambisanayo, bese uwabeka ezindaweni ezifanele.

3. Sebenzisa inguqulo ye-GELU esondezwa nge-tanh

I-GPT-2 isebenzisa inguqulo ye-GELU esondezwa nge-tanh, kanti i-nn.GELU() ye-PyTorch ngokuzenzakalela isebenzisa inguqulo enembile. Ukuze uqondanise into ngayinye, setha:

nn.GELU(approximate="tanh")

Le mininingwane emithathu ibonakala mincane, kodwa ithinta ngokuqondile ukuthi imodeli eyakhiwe ngesandla ingakwazi yini ukuphinda imiphumela yokwakhiwa okusemthethweni. Lapho uthola iphutha, hlola ngokulandelana i-mask, ukuma kwamapharamitha kanye nenguqulo ye-activation function.

Indlela enconywayo yokufunda nokulungisa amaphutha

1. Qala ngokuqonda ebudlelwaneni obubanzi ukuthi i-GPT-2 isebenzisa kuphela ingxenye ye-Transformer decoder, usebenzisa umdwebo oguqukayo wesakhiwo.
2. Sebenzisa umbhalo wesahluko samanje, ubhale phansi ukuma kokufaka, kokuphumayo kanye kwama-tensor aphakathi.
3. Uma i-attention matrix kunzima ukuyiqonda, sebenzisa ukubonwa kwe-matrix okuku-inthanethi ukuze uhlole isikwele ngesikwele.
4. Ngemva kokuqeda ingxenye, funda ukwakhiwa okuhambisanayo ku-gpt2/model.py.
5. Ngemva kokushintsha ukwakhiwa, sebenzisa kuqala izivivinyo zezisindo ezingahleliwe, bese usebenzisa izivivinyo zezisindo zangempela lapho kudingeka.
6. Ekugcineni, sebenzisa isibonelo sokukhiqiza umbhalo kanye nekhasi le-Gradio ukuze ubuke ukuziphatha kwemodeli ephelele.

Le ndlela ixhumanisa umdwebo wesakhiwo, ukubalwa kwama-matrix, ikhodi yezahluko, imodeli ephelele kanye nomphumela wokugcina, igweme ukuthi ekuqaleni ugajwe yisakhiwo esinezingqimba eziningi se-GPT-2 ephelele.

Isifinyezo

I-NamiGPT inikeza indlela yokufunda esuka ekubalweni kwamamatriksi okuyisisekelo iye ekusebenzeni okuphelele kwe-GPT-2. Ungaqala ngokusebenzisa imibhalo ezimele yezahluko ukuze uqonde ingxenye ngayinye, bese ulayisha izisindo ezisemthethweni ze-OpenAI ukuze uqedele umbhalo wesiNgisi, futhi ekugcineni uqinisekise ukuqonda kwakho nezinguquko zakho ngokusebenzisa ukubonwa kwedatha kanye nokuhlolwa kokuqondanisa ne-HuggingFace.

Ikhodi yephrojekthi isebenzisa ilayisensi ye-MIT; izindatshana ezikuhla ye-docs/ zisebenzisa ilayisensi ye-CC BY-NC 4.0, futhi uma ziphinde zishicilelwa kufanele kukhonjwe umthombo, kanti ukusetshenziswa kwezentengiso akuvunyelwe.