Qalisa i-GLM-5.3 Flash EXL3 Ezinhlelweni Ezimbili ze-NVIDIA DGX Spark

Uhlolojikelele

I-GLM-5.3 Flash EXL3 ye-DGX Sparks ezingu-2x iyiresiphi yokusebenzisa kanye ne-vLLM overlay yokusebenzisa i-quantization engu-4 bpw EXL3/TR3 ye-GLM-5.3 Flash kumasistimu amabili e-NVIDIA GB10. Iveza i-API ehambisana ne-OpenAI ku-port 8888 futhi isebenzisa imodeli ngaphansi kwesihlonzi GLM-5.3-Flash-EXL3.

Iresiphi isebenzisa i-tensor parallelism enezinga elilodwa emshinini ngamunye. Ihlanganisa ama-routed expert e-EXL3 apakishiwe, ama-kernel omdabu e-sm_121a, i-fp8 sparse-MLA KV cache, ama-CUDA graph, kanye ne-DFlash2 speculative decoding enamathokheni ayisikhombisa ohlaka.

I-checkpoint esetshenziswa ngokuzenzakalelayo i-Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw, okuyisibuko somphakathi esifana ngamabhayithi ncamashi ne-snapshot kaBrandon Music okukhulunywe ngayo. Iqukethe cishe i-164 GiB kuma-shard angu-120. I-repository yemodeli inelayisensi yayo, kanti ama-script okusebenzisa kanye ne-overlay kunelayisensi ye-MIT.

Kungani i-overlay idingeka

I-image evamile ye-ARM64 GLM-5.3 vLLM ingalayisha le checkpoint, kodwa iyehluleka ku-forward pass yokuqala ngoba i-GLM-5.3 Flash isebenzisa i-NoPE MLA, kanti i-backend etholakalayo ye-SM12x sparse-MLA ilindele i-packed geometry ehlukile. I-overlay igcwalisa i-NoPE latent engu-512-dimensional ukuze ifinyelele ku-geometry edingwa yi-FLASHINFER_MLA_SPARSE_SM120 ngaphandle kokushintsha i-QK dot product.

Iphinde ibhalise indlela esebenzayo ye-EXL3 quantization. Lokhu kugcina ama-routed expert ku-trellis, SUH, SVH, kanye ne-MCG representation yawo epakishiwe esikhundleni sokuwandisa abe yi-BF16. Njengoba indlela ehlanganisiwe ivuliwe, ungqimba ngalunye lusebenzisa i-launch eyodwa ye-exllamav3_ext.exl3_moe.

I-overlay iphinde inikeze ukuhlanganiswa okuqondene ne-GLM kwe-DFlash2, okuhlanganisa ukuthwebula kwe-EAGLE3 auxiliary hidden-state, i-draft sliding-window attention eyabelana ngezikhala ze-target MLA KV, kanye ne-non-causal attention ngaphakathi kwe-speculative draft block.

Izici ezibalulekile

• Isevisi ehambisana ne-OpenAI: i-vLLM iveza i-/v1/chat/completions nama-endpoint ahlobene ku-head node.
• I-tensor parallelism yama-node amabili: deployment isebenzisa i---nnodes 2 kanye ne---tensor-parallel-size 2.
• Ama-weight e-EXL3 ahlangene: i-checkpoint engu-4 bpw icishe ibe ngu-164 GiB ukuyilanda futhi ibikwa njenge-176 GB kuphaneli yekhwalithi okukhulunywe ngayo.
• I-DFlash2 speculative decoding: imodeli yo-draft ezenzakalelayo yi-incoai/GLM-5.3-Flash-DFlash2 ene-k=7.
• Umongo omude: ubude obukhulu bemodeli obuzenzakalelayo amathokheni angu-900,000, busekelwa yi-fp8 MLA pool ebikwa inamathokheni angu-982,612 ngaphansi kokucushwa kwenkumbulo okubhaliwe.
• Ukusekela i-multimodal: okokufaka kwezithombe namavidiyo kuvuliwe, kunomkhawulo ozenzakalelayo wesicelo ngasinye wezithombe ezine nevidiyo eyodwa.
• Ukuhlaziya amathuluzi nokucabanga: i-launcher ivula ukukhethwa kwamathuluzi okuzenzakalelayo kanye nama-parser ahambisana ne-GLM we-tool-call nokucabanga.
• Ukugcina ama-prefix: ama-prefix aqondaniswe nama-block angasetshenziswa futhi lapho ama-hash afanele esekhona ku-cache.
• I-refusal-direction ablation ongayikhetha: i-ABLIT=1 isebenzisa ushintsho olusememori kuma-weight akhethiwe omdabu e-BF16 output-projection ngesikhathi sokulayisha.

Izidingo ze-hardware ne-software

Le recipe yakhelwe ngokuqondile amasistimu amabili e-NVIDIA GB10 asezingeni le-DGX Spark axhumeke nge-CX7 fabric ebhaliwe. Akuyona i-deployment evamile ye-GPU eyodwa.

• Amasistimu amabili e-NVIDIA GB10 anesitoreji esikhululekile esicishe sibe ngu-180 GiB node ngayinye sokusetha kokuqala.
• I-Docker etholakala ngaphandle kwe-sudo kuwo womabili amasistimu.
• I-SSH engenaphasiwedi esuka ku-head node iye ku-worker.
• I-hf noma huggingface-cli, curl, kanye ne-rsync efakwe ku-head.
• Ama-interface enethiwekhi e-CX7 namadivayisi e-RDMA alungile kuyo yomibili imishini.
• Indawo ye-Hugging Face cache ekwazi ukugcina i-checkpoint enama-shard angu-120 kanye nemodeli ye-DFlash2 ecishe ibe ngu-2.3 GiB.

Gcina izilungiselelo ze-CX7 interface ne-HCA ze-repository ngaphandle uma izintambo zakho ezibonakalayo zihlukile. I-README ixwayisa ngokuthi i-NCCL ayikwazi ukusebenzisa ama-loopback alias e-10.0.0.x futhi ingase ime ngesikhathi se-ncclCommInitRank uma ama-fabric pin engalungile.

Faka futhi ulungiselele i-deployment

1. Clone i-repository

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks

2. Dala ifayela le-environment

cp .env.example .env

Hlela i-.env uma imishini ingasebenzisi amakheli noma amagama ama-akhawunti azenzakalelayo eresiphi. Izilungiselelo ezibaluleke kakhulu yilezi:

• HEAD_IP=10.0.0.1 ye-head node ne-vLLM master.
• WORKER_IP=10.0.0.2 ye-Spark yesibili.
• WORKER_USER lapho i-akhawunti ye-SSH yehluka phakathi kwama-node.
• WORKER_HOME lapho i-home noma indawo ye-Hugging Face cache ye-worker yehluka.
• PORT=8888 yomsabalalisi we-API.

I-launcher idala i-.env isebenzisa i-.env.example ngokuzenzakalelayo uma ingekho. Amanani anikezwe ngqo ngaphambi komyalo anokubaluleka kunalawo asefayilini.

3. Faka ama-weight ngokuzikhethela

./download.sh

Lo myalo ulanda i-checkpoint ye-EXL3 kanye nemodeli ye-DFlash2 ku-Hugging Face cache ye-head node. Awusebenzisi i-Docker, awuxhumi ku-worker, futhi awuvumelanisi amafayela. Wusebenzise uma ufuna ukuqeda ukulanda kwemodeli ecishe ibe ngu-164 GiB ngaphambi kokuba i-worker ilunge.

Ukuvuselela ngokusobala amafayela aku-cache, sebenzisa:

REFRESH_WEIGHTS=1 ./download.sh

Gwema ukuvuselela ngokungenasidingo. Kokubili download.sh ne-start.sh kweqa ama-repository asevele ephelele.

4. Qalisa womabili ama-rank ne-API

./start.sh

Iskripthi sokuqalisa senza lokhu okulandelayo ngokulandelana:

1. Sihlola i-Docker, i-SSH, nesikhala sediski esitholakalayo kuwo womabili ama-node.
2. Sithatha i-image yomphakathi ethi ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 siyibeke ku-head.
3. Sidlulisela i-container image ku-worker lapho i-digest yayo isishintshile.
4. Silanda amafayela emodeli angekho siwabeke ku-Hugging Face cache ye-head.
5. Sivumelanisa i-cache ne-worker sisebenzisa i-rsync.
6. Siqalisa i-rank 1 ngemodi engena-head ku-worker.
7. Siqalisa i-rank 0 ne-API ehambisana ne-OpenAI ku-head.
8. Sihlola i-health endpoint kuze kuqedwe ukulayishwa kwemodeli ne-warmup.

Isikhathi esimisiwe sokulinda ukulungela singamasekhondi angu-3,600 ngoba ukulayisha ama-weight nokuthatha ama-CUDA graph kungathatha isikhathi eside.

5. Hlola isimo nama-log

./start.sh status
./start.sh logs
./start.sh logs worker

Misa womabili ama-container usebenzisa:

./start.sh stop

Ungasebenzisa futhi i-./stop.sh.

Thumela isicelo esiyisisekelo sengxoxo

Ku-node ye-head, i-base API URL ithi http://127.0.0.1:8888/v1. Ngamakheli e-LAN abhaliwe, eminye imishini ingasebenzisa http://10.0.0.1:8888/v1.

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [
      {"role": "user", "content": "Explain tensor parallelism briefly."}
    ],
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

Inani le-model esicelweni kufanele lihambisane negama elinikezwayo, elizenzakalelayo lingu-GLM-5.3-Flash-EXL3. Ukucabanga kuvuliwe ngokuzenzakalela. Ukuze ukuvale, beka i-chat_template_kwargs ezingeni eliphezulu lesicelo se-JSON njengoba kubonisiwe ngenhla.

Ungathumeli into ye-extra_body engokoqobo nesendlalelo ngaphakathi kwe-HTTP eluhlaza. Ku-OpenAI Python SDK, i-extra_body iyinketho ye-SDK okuqukethwe kwayo kuhlanganiswa nesicelo esisezingeni eliphezulu.

Ukucushwa kokukhiqiza kwe-checkpoint kungase kunikeze i-temperature=1.0 ne-top_p=0.95. Setha amanani okusampula esicelweni ngasinye lapho udinga ukuziphatha okuhlukile.

Qonda i-runtime ezenzakalelayo

Okuzenzakalelayo kokusatshalaliswa kuhlelwe ngamabomu ukuze kuhambisane. Ukushintsha ingxenye eyodwa kungabangela inhlanganisela yama-kernel engasebenzi noma kwehlise ukwamukelwa kokuqagela.

• Ama-weight: Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw, ngokubuyela ku-repository ka-Brandon Music lapho i-mirror ingatholakali noma ingaphelele.
• Ukufana: ama-node amabili nosayizi we-tensor-parallel ongu-2 kusetshenziswa i-multiprocessing executor.
• Quantization: exl3, enama-routed expert apakishiwe nama-weight omdabu e-dense, shared-expert, attention, embedding, kanye ne-output-head.
• Target KV cache: fp8, emelelwa ngaphakathi njenge-fp8_ds_mla epakishiwe.
• Ukuqagela: i-DFlash2 enamathokheni okuqagela ayisikhombisa, kanti i-draft tensor parallelism isethwe ku-1.
• Draft KV cache: ukukhetha okuzenzakalelayo, okuholela ku-BF16 yemodeli ye-dense draft esikhundleni se-MLA layout ye-target.
• Attention: i-sparse MLA ye-target ne-FLASH_ATTN ekhethwa ngokuzenzakalela ye-DFlash2 draft attention.
• Context: MAX_MODEL_LEN=900000.
• Concurrency: MAX_NUM_SEQS=4.
• Prefill: MAX_NUM_BATCHED_TOKENS=1024.
• Ukusetshenziswa kwememori: GPU_MEM_UTIL=0.87.

Shintsha izindlela ze-speculative decoding

I-DFlash2 iyona ezenzakalelayo futhi isebenzisa i-incoai/GLM-5.3-Flash-DFlash2. Ukuze ubuyele endleleni ye-MTP ye-checkpoint enamathokheni amabili okuqagela, qalisa kabusha usebenzisa:

SPEC_METHOD=mtp ./start.sh restart

Imvelo iphinde isekele i-SPEC_METHOD=none. I-draft tensor parallelism ezenzakalelayo ye-DFlash2 ingu-1, okugcina imodeli encane ye-draft ku-rank 0 futhi kugweme ukuxhumana kwe-CX7 esinyathelweni ngasinye se-draft.

Ungaphoqi i-TRITON_ATTN ku-DFlash2. Ngokwezilinganiso ze-README, leyo backend yenza i-attention ibe yi-causal ngaphakathi kwe-draft block futhi yabangela ukuba ukwamukelwa ezindaweni zokuqagela zakamuva kwehle kakhulu. Shiya i-draft attention ingasethiwe ukuze i-SM121 ikhethe i-FLASH_ATTN.

Ukucushwa kwe-KV cache ne-context ende

I-KV cache ye-target kufanele ihlale iyi-fp8. Kule architecture, i-sparse-MLA kernel yamukela i-fp8_ds_mla epakishiwe; i-BF16 ayinayo i-sparse kernel ehambisanayo, kanti ama-NVFP4 KV kernel asebenza ku-dense MHA kunase-sparse MLA.

Nge-DFlash2, ukwesekwa kombono, ukusetshenziswa kwememori okungu-0.87, nobude bemodeli obukhulu obungu-900,000, i-README ibika i-pool yamathokheni angu-982,612 esebenzisa cishe u-15.67 GiB ku-fp8 MLA. Lokhu kunikeza cishe izikhathi ezingu-1.09 zomthamo wesicelo esisodwa esigcwele samathokheni angu-900,000. Ukucushwa komdabu kwamathokheni ayisigidi kusalokhu kungabiwa ngaphansi kokusethwa okubhaliwe.

Gcina lezi zilungiselelo ngaphandle kwalapho uhlola ushintsho lwamabomu lwe-runtime:

KV_CACHE_DTYPE=fp8
GPU_MEM_UTIL=0.87
MAX_MODEL_LEN=900000
MAX_NUM_BATCHED_TOKENS=1024
SKIP_MM_PROFILING=1
LIMIT_MM={"image":4,"video":1}

SKIP_MM_PROFILING=1 ibalulekile ngoba ukwenza i-profiling ye-input eyisibonelo yesithombe nevidiyo enosayizi omkhulu kungaqeda imemori ehlanganisiwe. I-README iphinde ixwayise ngokuthi ukukhuphula umkhawulo wamathokheni ahlanganisiwe ube ngu-8,192 kulayisha ngokweqile indlela ye-GB10 indexer top-k ngesikhathi sokusebenza kwe-context ende.

Indlela i-prefix caching esebenza ngayo

Ukugcinwa kwe-prefix kunikwe amandla, kodwa i-OpenAI API isalokhu ingenazo izimo ezigcinwayo. Iklayenti kufanele liphinde lithumele wonke umlando wengxoxo kuzo zonke izikhathi zokuxhumana. I-vLLM ihasha i-prefix ethunyelwe futhi ingase iphinde isebenzise amabhulokhi agciniwe uma esatholakala.

• Amathokheni aqondaniswe namabhulokhi kuphela abalwa njengama-hit ngaphansi komphathi we-MLA cache ochaziwe.
• MAX_NUM_SEQS=4 ivumela izizukulwane ezine eziqhubekayo; ayigcini izikhathi ezine zengxoxo ezihlala zikhona.
• Izingxoxo ezingasebenzi akuqinisekiswa ukuthi zizohlala zigciniwe.
• Ama-activation engxoxo eyodwa awahlanganiswa nawengxoxo enye.
• Ingxoxo endala ingadinga i-prefill entsha ngemva kokuba omunye umthwalo ukhiphe i-prefix yayo.

Ukuhlolwa okubukhoma kwe-README kuphinde kwasebenzisa amathokheni angu-3,584 kokulandelayo okunamathokheni angu-7,760, okungaba amaphesenti angu-46 e-prompt. Phatha ukugcinwa kwe-prefix njengokusheshisa okungase kwenzeke, hhayi njengokugcinwa kweseshini okuhlala isikhathi eside.

Linganisa ukusebenza kwe-decode

Le repository iqukethe i-tests/bench_decode.py yokuhlola i-streaming decode nokuqinisekisa ukuhambisana. Qalisa ukuhlolwa kwesibalo esihleleke kanje:

python3 tests/bench_decode.py \
  --phase structured \
  --structured \
  --runs 5 \
  --max-tokens 400 \
  --skip-coherence \
  --out /tmp/glm53-structured.json

Qalisa ukuhlolwa kwencazelo ye-hash-map ebhalwe ngolimi olujwayelekile kanje:

python3 tests/bench_decode.py \
  --phase prose \
  --runs 5 \
  --max-tokens 400 \
  --skip-coherence \
  --out /tmp/glm53-prose.json

Ngaphansi kwephrothokholi eshisayo, engenayo i-KV, echazwe ku-README, umthwalo ohleleke wafinyelela kumathokheni angu-62.9 ngomzuzwana lapho kune-concurrency eyodwa, amathokheni angu-103.3 ngomzuzwana esewonke lapho kune-concurrency ezimbili, namathokheni angu-146.5 ngomzuzwana esewonke lapho kune-concurrency ezine. Ukusebenza kuncike kakhulu ekwamukelweni kwe-speculative: i-median yaselabhorethri ibingamathokheni angu-61.7 ngomzuzwana kokuphumayo okuhlelekile, kodwa yaba ngu-26.9 kokubhalwe ngolimi olujwayelekile. Ukuhlolwa okuxubile noma okune-context ende, okunamathokheni agciniwe acishe abe ngu-60,000 kuya ku-100,000, kwabikwa ku-24 kuya ku-27 wamathokheni ngomzuzwana.

Lezi zibalo zichaza ukusethwa okuqondile kwama-GB10 amabili, izinhlobo zama-prompt, ukulungiselelwa kwe-temperature-zero, ukucabanga okukhutshaziwe, i-DFlash2 k=7, kanye nezizukulwane zamathokheni angu-400 ku-README. Akufanele zihunyushwe njengeziqinisekiso zomthamo wokukhipha ulwazi ezisebenza yonke indawo.

Ukuhlolwa kokuzikhethela kokususwa kwesiqondiso sokwenqaba

Le repository ingasebenzisa ukususwa kwesiqondiso sokwenqaba ngesikhathi kulayishwa izisindo ngaphandle kokubhala kabusha noma ukwenza i-requantization ye-EXL3 checkpoint. Kwenze kusebenze kanje:

ABLIT=1 ./start.sh restart
./start.sh logs | grep ablit

Iresiphi ezenzakalelayo ihlela i-self_attn.o_proj ngayinye yomdabu ye-BF16 ezingqimbeni 15 kuya ku-45. Izingqimba 0 kuya ku-14 zihlala zingashintshiwe njengezisekelo zokuphepha. Isikali esizenzakalelayo ngu-ABLIT_ALPHA=3.0; ukusibeka ku-1.0 kusebenzisa i-projection evamile esusa ingxenye ehambisana nesiqondiso esinikeziwe.

Izilawuli eziwusizo zihlanganisa:

• ABLIT=0 noma inani elingasethiwe ukuze kushiywe izisindo ezijwayelekile zingashintshiwe.
• ABLIT_DIRECTION=dealign ukuze kusetshenziswe isiqondiso esizenzakalelayo esishicilelwe.
• ABLIT_LAYERS=15-45 ukuze kusetshenziswe ububanzi obuhleliwe obufaka imingcele yomibili.
• ABLIT_ALPHA=3.0 ukuze kusetshenziswe isikali se-projection.
• ABLIT_INCLUDE_MTP=1 ukuze kufakwe ibhulokhi ye-MTP ye-checkpoint uma kusetshenziswa i-MTP.

Imiphumela yekhwalithi ye-KLD ku-README yalinganiswa ngaphandle kokususwa kwesiqondiso. Ukukwenza kusebenze kushintsha indlela yokwenqaba futhi kungathinta ukuhambisana noma ukwamukelwa kwe-speculative, ngoba umdwebi we-DFlash2 uhlala engashintshiwe kuyilapho okuphumayo okuqondiwe kushintsha. Uma ukuhambisana kwehla, i-README iphakamisa ukuthi kwehliswe kuqala i-ABLIT_ALPHA.

Izindlela zokuqalisa ezithuthukile

Uma i-image ne-model cache sezikhona kuyo yomibili imishini, ungagwema ukudlulisa idatha okuphindaphindwayo:

SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh

Ukuze kugcinwe i-container image yasendaweni futhi kweqiwe konke ukudonsa, ukulanda, nokuvumelanisa:

SKIP_PULL=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh restart

Ukuze kwakhiwe kabusha i-overlay kusuka ku-repository yamanje esikhundleni sokusebenzisa i-image yomphakathi:

BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 ./start.sh restart

Ungakha futhi i-Dockerfile ngokuqondile:

docker build -t glm53-flash-sm121:local .

Iskripthi sokuqalisa ngokuvamile siqhuba ukuzihlola kwe-GPU overlay ngaphambi kokuthumela i-image ku-worker. Setha kuphela SKIP_OVERLAY_VERIFY=1 uma uhlose ngamabomu ukweqa lokho kuhlola.

Izixwayiso ezibucayi zokumisa

• Ungadlulisi --moe-backend marlin. Lokhu kusetshenziswa kusebenzisa izisindo ze-EXL3 nendlela yaso yokupakisha ochwepheshe.
• Ungasethi i-target KV cache ibe yi-NVFP4 noma i-BF16. Gcina KV_CACHE_DTYPE=fp8.
• Ungabopheleli ukunaka kwe-speculative draft ku-TRITON_ATTN.
• Ungayishintshi i-image ufake i-glm53-flash-sm121:v8 endala yokusethwa kwe-NVFP4/Ray.
• Ungashintshi i-tensor parallelism, inani lama-node, ama-CX7 pin, noma i-USE_HOST_NCCL ngaphandle kokucabangisisa. Kuyizingxenye ezibalulekile zeresiphi yama-node amabili.
• Ungasusi noma wenze i-requantization ye-Hugging Face checkpoint cache ngaphandle kokuthi uhlose ukuphinde ulande yonke imodeli.
• Ungakhubazi izivikelo zokuprofayila kwe-multimodal ngaphandle kokubala ukusetshenziswa kwe-unified memory.
• Ungakhombi i-ABLIT_DIRECTION ku-vector ye-checkpoint engahlobene ngaphandle kokuqinisekisa ubukhulu bayo obufihlekile.

Ukuxazulula izinkinga

Isevisi ayikaze ibe sesimweni esinempilo

Hlola womabili ama-log usebenzisa ./start.sh logs kanye ./start.sh logs worker. Hlola i-SSH engenaphasiwedi, isikhala sediski esitholakalayo, izimvume ze-Docker, ukuphelela kwe-cache, namagama ezixhumanisi ze-CX7. Ukulayishwa kokuqala kwezisindo kanye nokuthwebula igrafu kuthatha isikhathi, ngakho vumela i-timeout ezenzakalelayo yokulungela ukuthi iphele kuphela ngemva kokubuyekeza inqubekelaphambili kuma-log.

I-NCCL iyalenga ngesikhathi kuqaliswa ama-rank

Qinisekisa HEAD_CX7_IF, WORKER_CX7_IF, HEAD_CX7_IB, kanye ne-WORKER_CX7_IB. Izilungiselelo ezizenzakalelayo ezibhaliwe zihambisana nezimbobo ezibonakalayo zelabhorethri futhi zingadinga ukulungiswa uma izintambo zakho zihlukile.

I-worker ayikwazi ukufinyelela ku-GHCR

I-worker ayidingi ukufinyelela okuqondile ku-GHCR. I-head idonsa i-image yomphakathi bese iyidlulisela nge-SSH. Uma i-head ihlangabezana nemikhawulo yezinga yokusebenzisa ngaphandle kokungena, lungiselela GHCR_USER kanye ne-GHCR_TOKEN.

Ukukhishwa kwe-decoding okuqagelayo kuhamba kancane ngokungalindelekile

Qinisekisa ukuthi SPEC_METHOD=dflash, DFLASH_TOKENS=7, nokuthi ukunaka kwe-draft akuphoqelelwanga ku-TRITON_ATTN. Khumbula futhi ukuthi ukusebenza kwe-speculative kuyahlukahluka kuye ngohlobo lokuphumayo: ukulandelana okuhlelekile kakhulu kungaba nokwamukelwa okuphezulu kakhulu kunombhalo ongavinjelwe.

Isiphetho

Le phrojekthi iguqula amasistimu amabili e-NVIDIA GB10 abe yiseva ye-GLM-5.3 Flash ehambisana ne-OpenAI esebenzisa ama-weight e-EXL3 ahlangene, i-fp8 sparse-MLA caching, kanye ne-DFlash2 speculative decoding. Indlela eshesha kakhulu ukugcina izilungiselelo ezizenzakalelayo zenethiwekhi, ze-KV, zokunaka, nezokulinganisa ezinikeziwe, uqalise nge-./start.sh, bese uqinisekisa ukuthunyelwa nge-health endpoint, i-chat API, amalogi, kanye nama-decode benchmark afakiwe.