Yakha futhi Uhlole i-APEX, I-Tile Evulekile ye-RTL Yokusebenzisa ama-LLM

Okwenziwa yi-APEX

I-APEX iyithayela elivuleke ngokuphelele le-LLM inference elibeka ukuqinisekiswa kuqala. Isebenzisa ungqimba olulodwa lwe-transformer decoder nge-RTL yangempela, okuhlanganisa i-attention, ukucindezelwa kwe-KV-cache, i-online softmax, i-RMSNorm, ama-rotary position embeddings, i-SwiGLU, ukuxhumana kwe-residual, kanye nemisebenzi ye-matrix.

Le phrojekthi ibhekana nenkinga enkulu ye-edge inference: njengoba ingxoxo ikhula, ithokheni entsha ngayinye kufanele ifunde i-key-value cache eya ngokuya iba nkulu. I-APEX ibeka i-KV codec ngqo ku-hardware datapath. Ama-key nama-value ayacindezelwa lapho ekhiqizwa, agcinwe ecindezelwe, bese ecindezelwa kabusha ngesikhathi i-attention iwasebenzisa.

I-APEX iyithayela elilodwa le-inference, hhayi i-accelerator chip ephelele. Ama-DRAM controller, i-PCIe, kanye ne-network-on-chip kungaphandle komkhawulo ochazwe yi-repository.

Le tile iboniswe nge-Qwen2.5-0.5B ku-FPGA hardware. Amathokheni e-Qwen2.5-7B adlule ku-golden pipeline eqinisekiswe ngesofthiwe, kodwa i-README ayisho ukuthi imodeli ye-7B isike yasebenza ku-silicon.

Izici Eziyinhloko

• I-datapath ephelele ye-decoder layer: I-RTL ihlanganisa ama-projection, i-attention, i-normalization, i-RoPE, ukubalwa kwe-feed-forward, imisebenzi ye-residual, kanye nokuphathwa kwe-KV-cache.
• Ukucindezelwa kwe-KV ngaphakathi kwe-datapath: Ama-key asebenzisa i-per-channel INT4 quantization, ama-value asebenzisa i-per-token INT4 quantization, kanti ama-channel anzima angasebenzisa i-fp16 outlier lane.
• Precision eguquguqukayo: Iyunithi ye-TIP ilandelela ukubaluleka kwamathokheni bese ikhetha i-KVQ8, i-KVQ4, noma i-KVQ4 enama-outlier ngokwesifunda.
• Injini ye-matrix eyabelwanayo: I-INT8 systolic MXE isetshenziswa ngokushintshanisa kuyo yonke imisebenzi ye-matrix ku-decoder layer.
• I-online softmax: Ama-score adlula ekusetshenzisweni kwe-softmax okuzinzile ngokwezinombolo ngaphandle kwesidingo se-score buffer elingana nobude be-context.
• Ukuqinisekiswa okunembile nge-bit: Imiphumela ye-RTL iqhathaniswa namamodeli ayisithenjwa e-NumPy asebenzisekayo, kungavunyelwe ukungafani nhlobo.
• Ama-testbench ahlolwe nge-mutation: Ukuqinisekiswa kubhekwa njengokunenjongo kuphela lapho amaphutha e-RTL enziwa ngamabomu edala ukuthi izivivinyo zehluleke.
• Ukuphindaphinda kusuka ku-simulation kuya ku-silicon: Izinhlelo zemisebenzi yokubhalisa ze-FPGA ziphinde zisebenze ku-Verilator twin ukuze kuqhathaniswe umehluko we-bit-by-bit.
• Amalebula obufakazi acacile: Imiphumela ihlukaniswa ngokuthi ilinganisiwe noma ilinganiselwa phambili, kunokuba ihlanganiswe ibe isimangalo esisodwa sokusebenza.

Qonda I-Architecture

Isinyathelo se-decode siqala nge-activation vector. I-RMSNorm ilungisa okokufaka, bese i-MXE ibala ama-query, ama-key, nama-value projection. I-RoPE iguqula ama-query nama-key ngaphambi kokuba injini ye-KVQ icindezele ama-key nama-value ukuze kugcinwe.

Ngesikhathi se-attention, ama-value agcinwe ku-cache acindezelwa kabusha endleleni yokuwafunda. I-MXE ibala imikhiqizo ye-query-key, i-ASU isebenzisa i-online softmax, bese omunye umsebenzi we-MXE uhlanganisa ama-probability nama-value agcinwe ku-cache. I-output projection, ukungezwa kwe-residual, esinye isigaba se-normalization, kanye nendlela ye-SwiGLU feed-forward kuqedela i-layer.

Ama-RTL Block Ayinhloko

• MXE: Injini ye-INT8 systolic GEMM esetshenziselwa imisebenzi ye-matrix ye-layer.
• KVQ: Injini yokucindezela nokucindezela kabusha i-KV-cache.
• ASU: I-online softmax, i-RMSNorm, i-SiLU, kanye ne-SwiGLU.
• RoPE: I-rotary position embedding yama-query nama-key.
• TIP: Ukulandelela ukubaluleka kwamathokheni kanye nezinqumo zamazinga e-precision.
• SEQ: Ifayela lama-control and status register, i-job decoder, kanye ne-layer walker.
• SEAM: Ukwakhiwa kwe-activation frame, i-boundary quantization, kanye nokuthuthwa kwama-scale.
• XBR: Ukuqondisa phakathi kwezinjini ngaphandle kwama-host round trip angadingekile.
• TOP: Isendlalelo sokuhlanganisa se-apex_top kanye ne-integration logic.

Funda ARCHITECTURE.md ukuze uthole umklamo onemininingwane, i-decision register, kanye nemvelaphi yamamojula.

I-Host Mode ne-Walked Mode

I-APEX isekela izindlela ezimbili zokusebenzisa. Ku-host mode, i-host yakha futhi ithumele umsebenzi ngamunye we-hardware ngokwehlukana. Le mode iqinisekisiwe ku-FPGA silicon, kodwa umsebenzi ngamunye udinga i-host round trip.

Ku-walked mode, i-host iqala i-layer kanye kuphela. I-layer walker eku-tile ilanda ama-descriptor, ihlele izinjini, idlulise ama-intermediate tensor nge-XBR fabric, futhi isabalalise ama-quantization scale. I-layer ehanjisiwe inembe nge-bit ku-simulation. I-README ithi i-walked attention ku-FPGA isaqhubeka nokulungiselelwa.

Ungaphathi i-host mode ne-walked mode njengokucushwa kokusebenza okulinganayo. I-host mode iwusizo ekuqinisekiseni nasekuqaliseni, kanti i-walker ihloselwe ukususa i-host overhead enkulu yomsebenzi ngamunye.

Ukufakwa Nokulungiselelwa Kwendawo

Okudingekayo

Ukuze usebenzise indlela yokuqinisekisa yendawo echazwe emibhalweni, faka okulandelayo:

• Python 3.11
• NumPy
• Verilator 5.x
• GNU Make
• Icarus Verilog, ngokuzikhethela, ukuze kuqinisekiswe ngokuphambene

Sebenzisa imiyalo usuka empandeni ye-APEX repository ekhishiwe. Qinisekisa amathuluzi ayinhloko ngaphambi kokuqala:

python3 --version
verilator --version
make --version
python3 -c "import numpy; print(numpy.__version__)"

I-README ayinikezi umyalo wokufaka amaphakheji noma umyalo we-Python environment onezinguqulo ezinqunyiwe. Sebenzisa amafayela e-repository kanye nemibhalo yamafolda angaphansi lapho uxazulula okunye ukuncika okudingekayo ekuhloleni noma ezinhlelweni ze-FPGA.

Hlola Ubufakazi Kuqala

Ngaphambi kokuqalisa inqubo ende, buyekeza ubufakazi bephrojekthi kanye nemikhawulo yayo:

• STATUS.md iqukethe izibalo zokuqinisekisa ezikhiqizwe umshini kanye nemiphumela elinganisiwe.
• TRACEABILITY.md irekhoda izikhala ezaziwayo futhi ixhumanisa izimangalo nobufakazi obuzisekelayo.
• Ithebula eliyinhloko lilandelela isimo sokuqaliswa nokulungiselela ukusebenza.
• Inkomba yemiphumela ihlela ubufakazi obugcinwe ngokomkhankaso.

Qalisa Ukuhlolwa kwe-Golden Model

Qala ngamamodeli ereferensi e-NumPy asebenzisekayo namavekhtha afriziwe:

make -C golden test

Lo myalo usebenzisa ama-golden model futhi uhlanganisa isango eliqinile lokubalwa kokucindezelwa. Ukubalwa kokucindezelwa kubalulekile ngoba i-APEX ihlanganisa omaka, izikali, ukupheda, nokugcinwa kwama-outlier, esikhundleni sokubika kuphela i-INT4 payload evamile.

Qinisekisa i-RTL Pipeline

Qalisa Ukuhlolwa Okusheshayo Okugcwele kwe-Attention-Tile

make -C verif/top/smoke smoke

Lena yithagethi ebhaliwe yokuhlola kusukela ekuqaleni kuye ekugcineni ye-attention tile. Yisebenzise njengokuhlola kokuqala kokuhlanganiswa ngemva kokuphumelela kochungechunge lwama-golden model.

Qalisa Ukuqinisekiswa kwe-Layer-3

make -C verif/top l3

Ithagethi ye-Layer-3 iqhathanisa i-tile yangempela ehlanganisiwe ne-golden model kuzo zonke izimo ezinikeziwe. I-APEX idinga imiphumela efana ncamashi ezingeni lamabhithi, hhayi ukubekezelela okulinganiselwe kwezinombolo.

Qinisekisa i-Layer Walker

make -C verif/seq_walker

Lokhu kuqhuba uchungechunge lwezivivinyo ze-walker namasango ayo e-mutation. Ukuhlolwa kwe-mutation kuphula ukuziphatha ngamabomu bese kuhlola ukuthi indawo yokuqinisekisa iyakwazi yini ukubona iphutha. I-mutant esindayo ikhombisa ubuthakathaka bokuqinisekisa, hhayi umphumela ophumelelayo womklamo.

Hlola i-Performance Model

Qalisa ukuhlolwa kokulinganisa kwemodeli yokusebenza ngalo myalo:

python3 perf/apex_perf_model.py --check

Imodeli yokusebenza iwukuqagela okucacile, hhayi ukusebenza okulinganisiwe kwe-silicon. Imibono yayo eyisisekelo nezisekelo zokulinganisa kubhalwe ku-PERF_MODEL.md.

Izibalo eziqageliwe zejubane lokufunda nezamandla zesigaba se-7B zincike ezingxenyeni zesistimu ezingakakhiwa noma ezingakapheleli, okuhlanganisa i-native W4 weight path, ukuhamba kwezingqimba ngehadiwe, nokuhlanganiswa okubanzi kwe-LPDDR. Akufanele zethulwe njengemiphumela yehadiwe eseyizuziwe.

Hlola i-KV Compression Kumamodeli

Indawo yokugcina iqukethe amamatiriksi okunemba kwamamodeli e-KV codec eqinisekisiwe. Qalisa imatiriksi ye-0.5B ne-1.5B ngalo myalo:

bash eval/kv_eval/run_matrix.sh

Qalisa imatiriksi ye-7B ngalo myalo:

bash eval/kv_eval/run_matrix_7b.sh

Bheka ama-README okuhlola aseduze kwamafayela emiphumela agcinwe ngaphambi kokuqalisa le miyalo. Ukuhlola okushicilelwe kusebenzisa isethi ephelele yokuqinisekisa ye-HellaSwag enamadokhumenti angu-10,042. I-README ibika ukuziphatha okusondele ekulahlekelweni lutho kumamodeli amancane, umphumela ogcwele wesethi ongabonakali we-KVQ8 ku-7B, kanye nezindleko ezincane ezilinganisiwe ze-KVQ4 ku-7B.

Ubufakazi obuhambisana nalokhu butholakala ku-imiphumela yokuqhathanisa ngqo kanye nemiphumela yokuhlola ye-7B.

Qhuba Imodeli Nge-Golden Pipeline

Le phrojekthi ibhala ngokusetshenziswa kwe-prompt nge-run_tinynpu.py --prompt. Lokhu kusakaza amathokheni e-Qwen akhethwa ngobugovu nge-fixed-point golden pipeline futhi kungakhiqiza imisebenzi emise okwehadiwe ukuze idlalwe futhi.

run_tinynpu.py --prompt

I-README ayinikezi ukusethwa okuphelele kwemodeli noma zonke izimpikiswano zomugqa womyalo emiyalweni yayo esezingeni eliphezulu, ngakho landela imibhalo ye-artifact yamathokheni e-7B ukuze uthole ububanzi obuqondile, ama-artifact, nemininingwane yokuphinda ukhiqize. Khumbula ukuthi lokhu kuhamba kwe-7B kuqinisekiswe ngesoftware; akusona isimangalo sokusebenzisa i-7B ku-FPGA tile.

Qalisa Umboniso we-AWS F2

Izidingo Nezindleko

Umboniso we-FPGA udinga imininingwane ye-AWS CLI enokufinyelela kusimo se-f2.6xlarge ku-us-west-2. I-README ilinganisela cishe imizuzu engu-30 kanye namadola ase-US amabili ukuze kuqhutshwe ngokuphelele.

Isithombe sereferensi esibhaliwe se-FPGA yi-agfi-030a812cd224b409d. Sisebenzisa iresiphi ye-A2 newashi le-tile elingu-15.625 MHz.

Qalisa Iqoqo Lokuqinisekisa

bash scripts/fpga/f2/run_walked_demo.sh agfi-030a812cd224b409d

Iskripthi siqalisa isimo se-FPGA, silayisha isithombe, siqhuba iqoqo lokuqinisekisa elibhaliwe namaketanga ahanjisiwe, siphrinte isinqumo, bese sivala isimo.

Qala Umboniso We-Prompt Osebenzisanayo

bash scripts/fpga/f2/run_chat_demo.sh

Umboniso osebenzisanayo usebenzisa i-pipeline ehanjisiwe. I-README inikeza The capital of France is njengesibonelo se-prompt futhi ibika impendulo ehloliwe ethi Paris..

Izithombe ze-DDR zithathwe ku-mlx-community/Qwen2.5-0.5B-Instruct-4bit. Ama-artifact esiteji seqembu alungiselelwe ezikriptini. Abasebenzisi bangaphandle kufanele bakhiqize isithombe sesisindo ngalo myalo:

python3 scripts/fpga/f2/make_weight_image.py

Lungiselela isithombe esiwumphumela kubhakede olilawulayo, kunokuthatha ngokuthi unokufinyelela kubhakede lesiteji leqembu.

Ukuhumusha Kahle Imiphumela Yehadiwe

Indawo yokugcina ibika throughput elinganisiwe engu-0.56 wamathokheni ngomzuzwana esithombeni se-A0 ku-62.5 MHz, kanye no-0.25 wamathokheni ngomzuzwana esithombeni sereferensi se-A2 esingu-15.625 MHz. Iphinde ibike ukukhula okulinganisiwe kusukela kusisekelo esiqhutshwa yi-host esingu-0.004 wamathokheni ngomzuzwana.

Lezi zibalo zichaza izithombe ezithile ze-FPGA, amawashi, nezindlela zokusebenzisa. Hlola FIRST_WALKED_TOKENS.md ngaphambi kokuzicaphuna, futhi ugcine ukumiswa okuhambisanayo nelebula elinganisiwe.

Amathiphu Athuthukile

Sebenzisa I-Golden Model Njengomahluleli

Uma ushintsha ibhulokhi le-RTL, qala ngokubuyekeza noma uqinisekise isicaciso esisebenzisekayo, wakhe kabusha noma uqinisekise ama-vector njengoba kudingwa yirepository, bese uqhathanisa okukhiphayo kwe-RTL ibhithi nebhithi. Gwema ukwethula ukuhlolwa okusekelwe ekubekezeleleni ngaphandle kwalapho inkontileka yezibalo ebhalwe phansi yephrojekthi ikudinga ngokusobala.

Gcina Ukufakwa Kwe-Mutation Kugcwele

Ukudlula kokuhlolwa kokubuyela emuva akwanele ngaphansi kwendlela ye-APEX. Qalisa amasango afanele e-mutation ngemva kokushintsha i-testbench, i-scoreboard, i-sequencer, noma ibhulokhi yezibalo. Uma uhlelo oluphukile ngamabomu ludlula, qinisa ukuqinisekiswa ngaphambi kokuthemba umphumela wokuqala oluhlaza.

Phinda Usebenzise Umsebenzi Ofanayo Ezindaweni Ezihlukene

Imisebenzi ye-APEX isebenzisa izichazi zemisebenzi ezizichazayo, ezihlanganisa izimo, ukuhanjiswa, nezinkontileka ze-quantization. Gcina izichazi, izikali, nemisebenzi yama-register lapho uhlonza ukungafani. Ukuphinda usebenzise indiza efanayo ku-Verilator F2 twin naku-hardware kuyindlela ekhethwayo yokuhlukanisa amaphutha e-RTL kumaphutha e-synthesis noma e-toolchain.

Landela Izikali Kanye Nedatha Ye-Tensor

Izikali ze-quantization ziyingxenye yenkontileka ye-hardware, aziwona ama-metadata angeziwe nje. Lapho udala noma uhlola imisebenzi, gcina ulwazi lwezilinganiso oluhamba emikhawulweni yokuxhuma. I-tensor engenaso isikali esihambisanayo ayikwazi ukuphinda ikhiqize umsebenzi ohleliwe we-fixed-point.

Hlukanisa I-Codec Payload Kokugciniwe Sekukonke

Ungabali ukucindezelwa usebenzisa ama-payload e-INT4 kuphela. Faka namabhange ezikali, ama-tag, ama-padding, ama-metadata e-precision, nemizila yama-outlier. Sebenzisa isango lokubala eliyisisekelo le-repository kanye ne-STATUS.md njengesisekelo esigunyaziwe sezilinganiso ezishicilelwayo.

Hlukanisa Osayizi Bamamodeli Nezimangalo Zokusebenza

• Qwen2.5-0.5B: Ikhonjiswe nge-pipeline esekelwe ku-FPGA echazwe yiphrojekthi.
• Qwen2.5-7B: Ihlolwe nge-software-verified golden pipeline.
• Ukuqagela kwezakhiwo zohlobo lwe-7B: Ukuqagela kokuhlaziya kohlelo olukhulu oluzungeze i-tile, hhayi ukusebenza okulinganisiwe kokusetshenziswa kwamanje kwe-FPGA.

Funda Ngokugeleza Kwezisindo Ngaphambi Kokwandisa I-Walker

Izisindo zemodeli azingeni ngokuphelele ku-tile. Idizayini ye-IB-FUEL idlulisela amarekhodi ezisindo ngomsebenzi ngamunye esuka ku-DDR iye ku-FIFO engaphakathi kwe-tile, kuyilapho indlela ye-W4 ithutha amaqembu anamabhithi amane bese iwavula ukuze asetshenziswe injini ye-INT8 matrix. Buyekeza IB_FUEL.md kanye ne-W4_DATAPATH.md ngaphambi kokushintsha ukuhlela noma ukuziphatha kwe-feeder.

Sebenzisa Isakhiwo Se-Repository Ngempumelelo

• rtl/ iqukethe ukuqaliswa kwe-hardware.
• golden/ iqukethe amamodeli ereferensi e-NumPy anemba ncamashi ngamabhithi.
• verif/ iqukethe ama-testbench, ama-assertion, ama-scoreboard, amasango e-mutation, kanye ne-F2 simulation twin.
• eval/ iqukethe amathuluzi okuhlola ukunemba kwemodeli ye-codec.
• perf/ iqukethe imodeli yokusebenza yokuhlaziya eqagelwayo.
• scripts/fpga/ iqukethe izikripthi zokwakha, izithombe, ukulayisha i-DDR, izindiza, kanye ne-token loop.
• docs/results/ iqukethe ubufakazi bemikhankaso obuzibophezele kurepository.

Imikhawulo Yamanje Nemephu Yendlela

Imephu yendlela esebenzayo ibeka phambili ukuqinisekiswa kwe-walked attention ku-FPGA, ukuvala isikhathi ngokushesha, ukugeleza okuqhubekayo kwezisindo, ukwesekwa okuphelele kwe-W4, omunye umsebenzi olinganisiwe wezinga lama-token, kanye nokulungiselela ukusayinwa kwe-Sky130. Isimo esibukhoma sigcinwa kuthebula eliyinhloko.

Gwema ukucabanga ukuthi izinto ezikumephu yendlela seziqediwe. Ikakhulukazi, i-README ikhomba ukuthi ukulethwa okuphelele kwe-walked-attention hardware kanye nokuncika okuningana okuhlobene nomkhawulokudlulisa kusewumsebenzi oqhubekayo.

Isiphetho

I-APEX iwukuqaliswa kwe-RTL kwe-transformer decoder tile futhi iyisifundo sokuqinisekiswa kwe-hardware okuqhutshwa ubufakazi. Isici sayo esihluke kakhulu yi-KV-cache codec ehlanganisiwe, egcina okhiye namanani ngendlela ecindezelwe kuyilapho ihlala iyingxenye ye-attention datapath eqinisekisiwe.

Qala ngokuhlolwa kwe-golden, bese uqinisekisa i-tile ne-walker, uhlole ubufakazi obulinganisiwe, futhi uqhubekele kuphela kuma-workflow emodeli noma e-FPGA lapho ama-artifact awo engeziwe etholakala. Gcina imiphumela elinganisiwe ye-FPGA ihlukile ekuqageleni kokuhlaziya, futhi usebenzise imibhalo yokulandeleka yephrojekthi noma nini lapho wethula isimangalo.

I-APEX ikhishwa ngaphansi kwe-Apache License 2.0. Bheka LICENSE ngemigomo nezaziso ezisebenzayo.