Sebenzisa i-DeepSeek-V4.1-Flash kuma-DGX Spark amabili nge-vLLM ne-Pi

Lokho ozokwakha

Le phrojekthi inikeza imibhalo yokusebenzisa DeepSeek-V4.1-Flash kuwo wonke amanodi amabili e-NVIDIA DGX Spark kusetshenziswa i-vLLM. I-Tensor parallelism isebenza kuwo wonke amanodi ngesixhumanisi sawo se-QSFP/RoCE, kuyilapho inodi eyinhloko iveza i-API ehambisana ne-OpenAI ku-port 8000.

Ukusetshenziswa kusebenzisa i-checkpoint ethi 0xSero/DeepSeek-V4.1-Flash-Spark. Ochwepheshe be-backbone abaqondiswayo bafakwe ikhodi njengedatha ye-EXL3 trellis enesilinganiso esibikiwe samabhithi angu-2.77 ngesisindo ngasinye. I-attention, ochwepheshe ababelwana ngabo, ama-router, amatafula e-Engram, izingqimba eziyidrafti, izingxenye zombono, ama-embedding, kanye ne-output head kusala ku-checkpoint yokuqala.

Ngemva kokusetshenziswa, ungasebenzisa imodeli ngokuqondile nge-API yayo noma uyixhume ku-Pi coding agent ukuze wenze ikhodi nemisebenzi esizwa ngamathuluzi.

Ukusebenza namakhono

I-README ibika izilinganiso ezithathwe kumanodi amabili e-DGX Spark GB10 kusetshenziswa i-tensor parallelism nge-RoCE, i-FP8 KV cache, i-DSpark speculative decoding enamathokheni ayisikhombisa, amatafula e-Engram agcinwe kudiski, kanye nezilandelana ezimbili ezisebenza ngesikhathi esisodwa.

• Ukugcwaliswa kwangaphambili kwe-prompt engu-8k: cishe amathokheni angu-2,050 ngomzuzwana.
• Ukugcwaliswa kwangaphambili kwe-prompt engu-32k: cishe amathokheni angu-2,061 ngomzuzwana.
• Ukudecoda kwekhodi: cishe amathokheni angu-39–41 ngomzuzwana kusakaza okukodwa.
• Ukudecoda kwekhodi ngemifudlana emibili: amathokheni angu-60 ngomzuzwana esewonke.
• Ukudecoda kombhalo ojwayelekile: cishe amathokheni angu-29 ngomzuzwana kusakaza okukodwa.
• Umongo omkhulu ngesicelo ngasinye: amathokheni angu-262,144.
• Ichibi le-KV cache: amathokheni angu-2,023,717.
• Okokufaka kombono, ukubizwa kwamathuluzi, nokucabanga: kuhlolwe ngempumelelo.
• Ukukhumbula umongo onamathokheni angu-175k: kudlule kuzo zonke izivivinyo eziyisithupha ezibikiwe zaselabhorethri.

Ukuqalisa kokuqala kuthatha cishe imizuzu engu-11–13 ngoba kulayishwa izisindo namabhange e-EXL3, kuhlanganiswa ama-kernel, futhi kuthathwa ama-CUDA graph.

Izidingo zangaphambili

Udinga amanodi amabili e-DGX Spark asebenzisa i-DGX OS. Khetha elilodwa libe yinhloko, elisebenzisa i-API, elinye libe isisebenzi.

• Ikhebula le-QSFP elixhuma amanodi, ne-RoCE inikwe amandla.
• Okungenani i-port eyodwa ye-RDMA esesimweni se-ACTIVE kunodi ngayinye, okuqinisekiswa nge-rdma link.
• Ikheli le-IPv4 kunethiwekhi ehambisanayo yokuxhumana kunodi ngayinye, njenge-10.10.10.13 kanye ne-10.10.10.11.
• I-SSH engenaphasiwedi esuka kunodi eyinhloko iya kweyomsebenzi kusetshenziswa ikheli layo lokuxhumana.
• I-Docker ene-NVIDIA runtime efakiwe kuwo womabili amanodi.
• Umsebenzisi wakho kumele abe seqenjini le-docker.
• Cishe u-700 GB wendawo yamahhala ye-NVMe kunodi eyinhloko no-470 GB kunodi yomsebenzi.
• python3, rsync, kanye ne-curl kuwo womabili amanodi.
• I-Node.js ne-npm emshinini lapho i-Pi izosebenza khona.

Clone-a i-repository bese uchaza umsebenzi

Qalisa imiyalo elandelayo kunodi eyinhloko. Faka ikheli lomsebenzi esikhundleni se-SSH target.

git clone https://github.com/0xSero/DeepSeek-V4.1-Flash-Two-Sparks
cd DeepSeek-V4.1-Flash-Two-Sparks
export WORKER=user@10.10.10.11

Imibhalo isebenzisa i-WORKER ukuxhuma ku-Spark yesibili. Qinisekisa ukuthi ukuqinisekiswa ngokhiye we-SSH kuyasebenza ngaphambi kokuqhubeka.

Landa bese usabalalisa i-checkpoint

Landa i-checkpoint ecishe ibe ngu-462 GB uye kumkhombandlela wemodeli ozenzakalelayo kunodi eyinhloko:

scripts/download.sh

Umbhalo wokulanda uphinde uhlanganise izingxenye ze-Engram. Njengoba i-Hugging Face ikhawulela amafayela ngamanye ku-50 GB, izingxenye ezimbili zomdabu zigcinwa njengamafayela e-.partNN. Umbhalo uyazihlanganisa, uqinisekise izingxenye zomdabu eziyisithupha ngokumelene ne-sha256-manifest.txt, bese ususa amafayela ezingxenye. Setha i-KEEP_PARTS=1 uma ufuna ukugcina lezo zingxenye.

Kopisha imodeli eqediwe uye kunodi yomsebenzi ngokusebenzisa inethiwekhi yokuxhumana:

scripts/copy-to-peer.sh

Njengenye indlela, sebenzisa futhi i-scripts/download.sh ngokuqondile kunodi yomsebenzi. Indlela yemodeli ezenzakalelayo yi-~/models/DeepSeek-V4.1-Flash-Spark. Uma inodi yomsebenzi isebenzisa enye indlela, lungiselela i-WORKER_MODEL_DIR ngaphambi kokuqalisa.

Donsa i-serving image

Le phrojekthi isebenzisa i-vLLM image equkethe indlela ye-EXL3 mixture-of-experts nama-kernel e-B12X Spark. Donsa i-image eboshelwe enguqulweni kuwo womabili amanodi:

docker pull ghcr.io/0xsero/deepseek-v4.1-flash-spark@sha256:5668e35e5ee021da4b9ce88a1964caf0e082e8ca01d81d1d64b6213a55c6add5

Umbhalo wokuqalisa udonsa i-image ngokuzenzakalelayo uma ingakatholakali, ngakho lesi sinyathelo singaqedwa nangesikhathi sokuqalisa.

Qalisa ama-rank womabili e-tensor-parallel

Qalisa iseva yamanodi amabili usuka kunodi eyinhloko:

scripts/launch.sh

Isiqalisi sithola izixhumanisi zenethiwekhi ezisebenzayo, sakhe ukhiye we-API ku-~/.dsv41-two-sparks/api_key, siqalise izivikeli zememori kuwo womabili amanodi, siqalise i-rank 1 kunodi yomsebenzi, bese siqalisa i-rank 0 kunodi eyinhloko. Silinda i-API itholakale futhi sivumela isikhathi esingafika emizuzwini engu-45 sokuqalisa.

Sebenzisa wona lo mbhalo ukuhlola noma ukuphatha ukusetshenziswa:

scripts/launch.sh status
scripts/launch.sh logs r0
scripts/launch.sh logs r1
scripts/launch.sh stop

Iseva ilalela kunodi eyinhloko ku-http://127.0.0.1:8000 endaweni. Amaklayenti angasebenzisa ikheli elifinyeleleka lenodi eyinhloko, kuyilapho ithrafikhi phakathi kwama-rank kumele isebenzise inethiwekhi ye-QSFP/RoCE.

Qalisa izivivinyo zokuqala

Qinisekisa ukukhiqizwa kombhalo, ukubizwa kwamathuluzi, nokokufaka kombono usebenzisa:

python3 scripts/smoke.py http://127.0.0.1:8000

Ukuhlola ngakunye kubika PASS noma FAIL. Ungaqalisa futhi ibhentshimakhi ongayikhetha:

python3 scripts/bench.py --out bench.json --conc 1 2

Ukuhlolwa kokusebenza kuhlanganisa isivinini se-prefill, decode, kanye nesimo somongo omude ngaphandle kokubeka imikhawulo kokuphumayo.

Sebenzisa isiphetho esihambisana ne-OpenAI

Iseva iveza imininingwane elandelayo ye-API:

• URL eyisisekelo: http://<head-ip>:8000/v1
• Igama lemodeli: deepseek-v4.1-flash
• Ukuqinisekisa: ithokheni ye-bearer evela ku-~/.dsv41-two-sparks/api_key

Isibonelo, thola ukhiye ku-head usebenzisa:

ssh <head> cat ~/.dsv41-two-sparks/api_key

Ngemva kwalokho ungasebenzisa isiphetho nanoma yiliphi iklayenti elihambisana ne-OpenAI ngokumisa i-URL yesisekelo, igama lemodeli, kanye nokhiye we-bearer.

Xhuma i-Pi kumodeli

Kumshini one-Node.js ne-npm, faka bese ubhalisa umhlinzeki we-Pi. Umyalo wamukela i-URL eyisisekelo ye-API ngaphandle kwendlela yokugcina yemodeli:

pi/install.sh http://<head-ip>:8000/v1

Setha ukhiye we-API ku-shell ezoqalisa i-Pi:

export DSV41_API_KEY=$(ssh <head> cat ~/.dsv41-two-sparks/api_key)
pi --model dsv41-sparks/deepseek-v4.1-flash

Isifaki sifaka i-Pi uma ingekho ku-PATH yakho, bese sengeza noma sishintsha kuphela umhlinzeki we-dsv41-sparks ekucushweni kwamamodeli kwe-Pi. Sakha ikhophi yasenqolobaneni enesitembu sesikhathi kuqala futhi sishiya abanye abahlinzeki bengathintekile.

Uma usebenzisa uhlu lwe-enabledModels, isifaki sengeza kuyo imodeli ehlinzekwayo. Setha SET_DEFAULT=1 ukuze wenze le modeli ibe imodeli yokuqalisa ye-Pi.

Isandiso esifakiwe sisusa umkhawulo wamathokheni aphumayo we-Pi kulo mhlinzeki, sivumele izimpendulo ziphele ngokwemvelo. Sifunda i-DSV41_API_KEY ngesikhathi sesicelo futhi asibhali ukhiye ekucushweni kwe-Pi.

Qonda ukucushwa okuzenzakalelayo

scripts/launch.sh siphinda sicushwe lokho okusetshenziswe ekulinganiseni. Izilungiselelo ezibalulekile zihlanganisa:

• I-FP8 KV cache ene-KV_BYTES=2700000000.
• Ubude bomongo obukhulu obungamathokheni angu-262144.
• Ukulandelana okungafika kokubili okusebenza ngesikhathi esisodwa.
• Amathokheni amaningi ahlanganiswayo asethelwe ku-2048.
• I-DSpark speculative decoding enamathokheni ayisikhombisa okuqagela nokuqinisekiswa okuguquguqukayo.
• Amathebula e-Engram agcinwe kudiski.
• Ama-kernel e-B12X okunaka, omugqa, kanye ne-MoE.
• Ochwepheshe abaqondiswe yi-EXL3 abane-ST_EXL3_PREFILL=st.
• Ama-CUDA graphs e-FULL_DECODE_ONLY.
• I-B12X RoCE one-shot all-reduce kanye ne-NCCL phezu kwe-RoCE kweminye imisebenzi esabalalisiwe.
• Isithombe esisodwa nge-prompt ngayinye ye-multimodal.
• Ama-parser e-deepseek_v41 okucabanga kanye nokubiza amathuluzi.

Izilungiselelo eziningi zingashintshwa ngokusebenzisa ama-environment variables. Uhlu lwemodeli lulawulwa yi-MODEL_DIR, uhlu lwemodeli lwe-worker lulawulwa yi-WORKER_MODEL_DIR, kanti idatha eqhubekayo yesikhathi sokusebenza ilawulwa yi-STATE_DIR. Uhlu lwesimo oluzenzakalelayo ngu-~/.dsv41-two-sparks.

I-launcher ifaka imodeli njenge-read-only ku-/model. Ikopisha uhlelo lokwabiwa lwe-EXL3 ohlwini lwesimo bese ibhala kabusha izindlela zamabhange ayo ukuze zikhombe ezinkombeni ze-EXL3 ezifakiwe.

Gcina ithrafikhi kunethiwekhi efanele

Izinqubo ze-vLLM, NCCL, ne-Gloo kufanele zibophezele kumakheli e-QSFP/RoCE kunokuba zibophezele ezindaweni ze-Wi-Fi, Ethernet, noma Tailscale. I-launcher ngokuvamile ithola ngokuzenzakalelayo izimbobo ze-RoCE ezisebenzayo bese isetha okuguquguqukayo okufana nalokhu:

• VLLM_HOST_IP
• NCCL_SOCKET_IFNAME
• GLOO_SOCKET_IFNAME
• NCCL_IB_HCA

Uma ukutholwa okuzenzakalelayo kukhetha i-interface engalungile, setha ngokusobala i-HEAD_IP, WORKER_IP, HEAD_IFNAME, WORKER_IFNAME, HEAD_HCAS, noma WORKER_HCAS. Sebenzisa ip -br a kanye ne-rdma link ukuze uthole amakheli namadivayisi e-RDMA afanele.

Phatha inkumbulo ngokuphepha

I-DGX Spark yabelana nge-pool yenkumbulo engu-128 GB phakathi kwe-CPU ne-GPU. Uma inkumbulo etholakalayo ifinyelela ku-zero, i-host ingase ingaphenduli esikhundleni sokuvala i-container ngendlela ehlanzekile. I-launcher iqala i-scripts/memguard.sh kuwo womabili ama-node ukuze imise ama-container e-dsv41-r* lapho inkumbulo etholakalayo yehla ngaphansi komkhawulo omisiwe.

Bheka ilogi ye-memory guard ku:

~/.dsv41-two-sparks/memguard.log

Ngokucushwa okuzenzakalelayo, inkumbulo etholakalayo ngesikhathi sokusebenza esizinzile ibikwa cishe ku-2–3 GiB. Uma i-guard iqala, yehlisa i-KV_BYTES noma i-GPU_UTIL ngaphambi kokwandisa i-MEMGUARD_GIB.

Xazulula izinkinga ezivamile

Ukuqalisa kuhamba kancane

Ukuqalisa kokuqala kungathatha cishe imizuzu engu-12 ngenkathi kulayishwa izisindo, kulungiswa amabhange e-EXL3, kuhlanganiswa ama-kernel e-B12X, futhi kuthathwa ama-CUDA graphs. Ama-kernel caches agcinwa ngaphansi kwe-STATE_DIR/cache, ngakho ukwethulwa okulandelayo kungashesha.

Ayikho amafayela e-Engram

Uma ukuqalisa kubika ifayela elilahlekile njenge-model-00005-of-00006.safetensors, cishe izingxenye ze-Engram azihlanganiswanga kabusha. Qalisa futhi i-scripts/download.sh. Iqhubeka nomsebenzi ongaphelele futhi yeqa izinyathelo eseziqediwe.

Ukutholwa kwe-fabric kukhetha i-interface engalungile

Hlola amakheli nge-ip -br a nesimo se-RDMA nge-rdma link. Bese unikeza okuguquguqukayo okucacile kwe-interface, IP, ne-HCA kwe-head ne-worker okuchazwe ngenhla.

Ukulanda kuhamba kancane

Uma i-Spark ifinyelela ku-inthanethi nge-Wi-Fi kuphela, landa i-checkpoint ku-node enokuxhumeka okuphezulu kakhulu bese usebenzisa i-scripts/copy-to-peer.sh ukuyidlulisela kunethiwekhi ye-fabric.

I-Pi ayikwazi ukubona imodeli

Qinisekisa ukuthi i-DSV41_API_KEY ithunyelwe ku-shell efanayo eqalisa i-Pi. Qalisa i-/model ngaphakathi kwe-Pi ukuze ulayishe kabusha uhlu lwamamodeli.

Imodi yokuphrinta ye-Pi ilinda unomphela

Uma okokufaka okujwayelekile kungesona isiphetho, i-pi -p nayo ifunda ku-stdin. Kumaskripthi, qondisa i-stdin isuke ku-/dev/null:

pi -p < /dev/null

Isiphetho

Le nqolobane ihlanganisa imodeli, isithombe sokusebenzisa, imibhalo yokuqalisa esabalalisiwe, ukuvikelwa kwememori, amathuluzi okuqinisekisa, nokuhlanganiswa kwe-Pi okudingekayo ukuze kusetshenziswe i-DeepSeek-V4.1-Flash kuwo womabili ama-DGX Sparks. Lungisa inethiwekhi ye-RoCE, landa futhi uqinisekise i-checkpoint kuwo womabili amanodi, qalisa ama-rank amabili, sebenzisa izivivinyo zokuqala, bese ukhomba amaklayenti ahambisanayo noma i-Pi endaweni yokuphela ehambisana ne-OpenAI yenodi eyinhloko.