Qalisa i-DeepSeek-V4.1-Flash-1017 kuma-DGX Spark amabili nge-TensorFold

Lokho okwenziwa yile phrojekthi

Le recipe yokusabalalisa isebenzisa i-DeepSeek-V4.1-Flash-1017 kuma-NVIDIA DGX Spark amabili, ngalinye line-128 GB yememori ehlanganisiwe. Isebenzisa injini ye-Zig yomdabu ye-TensorFold ene-tensor parallelism, enezinga elilodwa ku-Spark ngayinye, futhi inikeza i-API ehambisana ne-OpenAI ku-port 8888.

Lokhu kusetha kusebenzisa i-checkpoint yokuhlola engu-2.9-bit EXL3 Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw. Isekela iwindi elikhulu lomongo wemodeli, okokufaka kwezithombe, ukubiza amathuluzi, ukucabanga ongakukhetha, kanye ne-DSpark speculative decoding kusuka kumabhulokhi e-draft e-checkpoint.

Indawo yokugcina ikhodi yenza ngokuzenzakalelayo ukulungiselela, ukwakhiwa kwesithombe se-Docker, ukulandwa kwama-checkpoint, ukwakhiwa kwe-rank-cache, ukusethwa kwabasebenzi, ukuqaliswa kweseva, ukuhlolwa kwe-smoke, kanye nokuvala. I-Spark yokuqala isebenza njengekhanda: isebenzisa i-API, ixhumanisa ukusetha, futhi iqala i-rank 0. I-Spark yesibili isebenzisa i-rank 1.

Isakhiwo nemikhawulo ebalulekile

I-TensorFold isebenzisa umndeni we-BertholomusAI dsv41 ne-tensor parallelism ebekwe ku-2. Ama-rank amabili axhumana nge-direct ConnectX-7 link. Ikhanda lifunda ithrafikhi ye-API, kuyilapho womabili ama-Spark egcina cishe u-98 GiB wezisindo zemodeli kwimemori.

I-checkpoint icishe ibe ngu-196 GiB futhi ihlukaniswe yaba ama-shard angu-39. Ukusabalalisa kudinga namatafula amabili e-Engram anganqunywanga, athatha cishe u-203 GB. Amatafula e-Engram kufanele akopishwe kudiski yasendaweni kuwo womabili ama-Spark. Ukuwafunda nge-NFS kunganciphisa kakhulu ukusebenza kwe-prefill ne-decode ngoba i-decoding yenza ukufundwa okuningi okungahleliwe okungu-4 KiB.

Imodeli ingasebenzisa ngobuchwepheshe iwindi lamathokheni angu-1,048,576, kodwa imemori etholakalayo ekhanda iyona ebeka umkhawulo osebenzayo. Ekhanda elisebenzisa ezinye izinsiza zedeskithophu nezokuthuthukisa, i-README incoma ukugcina i-CONTEXT iku-524288 noma ngaphansi ngaphandle uma kukhululwa imemori eyengeziwe.

Izidingo zehadiwe nesofthiwe

• Ama-NVIDIA DGX Spark amabili noma amasistimu e-GB10 alinganayo ane-128 GB yememori ehlanganisiwe ngayinye.
• Uxhumano oluqondile lwe-QSFP phakathi kwama-port e-ConnectX-7.
• Ama-IPv4 address ku-direct link, anokuxhumana okusebenzayo kanye ne-RoCE v2 GID.
• Umkhawulo ofanayo wewashi le-GPU kuwo womabili amasistimu. I-README incoma ukuhlola amawashi nge-nvidia-smi -q -d CLOCK nokusebenzisa i-nvidia-smi -lgc 0,2200 uma kufanele.
• I-SSH esekelwe kuma-key kusuka ekhanda kuya kumsebenzi.
• I-Docker ene-NVIDIA container runtime kuwo womabili amasistimu, umsebenzisi wakho ekwiqembu le-docker.
• git, python3, kanye ne-Hugging Face CLI ekhanda.
• Isitoreji esanele se-checkpoint, amatafula e-Engram, amafayela ezisindo zama-rank, isithombe se-Docker, kanye nama-cache. Ikhanda lidinga cishe u-505 GB kanti umsebenzi udinga cishe u-310 GB ekuhlelweni okujwayelekile.

I-direct link ingavela njengamadivayisi enethiwekhi amabili namadivayisi amabili e-RoCE ku-Spark ngayinye. Lungiselela womabili ama-rail ngama-address akulinki futhi usebenzise i-MTU efanayo kuwo womabili amasistimu.

Lungisa inethiwekhi nokuxhumana kwe-SSH

Yabela izixhumi ze-ConnectX-7 ama-address avela ku-private subnet. Isibonelo, i-README isebenzisa ama-address afana no-192.0.2.1/24 no-192.0.2.2/24. Qinisekisa ukuthi amasistimu amabili ayakwazi ukwenza i-ping komunye nomunye nge-link ehlosiwe.

Ekhanda, faka i-SSH key yomsebenzi bese uqinisekisa ukufinyelela okungadingi ukuxhumana komsebenzisi:

ssh-copy-id user@<worker>
ssh -o BatchMode=yes user@<worker> true

Setha i-SSH target yomsebenzi kanye ne-fabric address kufayela lokumisa lendawo. Umsebenzi awudingi ikhophi yale ndawo yokugcina ikhodi.

Lungisa i-NFS noma isitoreji sasendaweni

Ngokuzenzakalelayo, umsebenzi ufaka i-cache ye-Hugging Face yekhanda ngendlela yokufunda kuphela nge-NFS. Ekhanda, faka iseva ye-NFS bese uthumela i-cache:

sudo apt install nfs-kernel-server
echo "$HOME/.cache/huggingface <worker-CX7-address>(ro,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -ra

Umsebenzi ungase udinge usizo lwe-NFS client lomsingathi, njenge-nfs-common ku-Ubuntu. Iskripthi sokulungiselela sakha ivolumu ye-Docker NFS yokufunda kuphela futhi sihlole ukuthi umsebenzi uyawabona yini amafayela adingekayo.

Ukuze kube nokuhlukaniswa okungcono noma lapho i-NFS ingafaneleki, beka ikhophi yendawo ye-cache edingekayo ku-Spark ngayinye. Setha i-HF_CACHE ibe yisihlahla sendawo, setha i-NFS_VOLUME ibe yindlela ehambisanayo yomsingathi, shiya i-HEAD_NFS_SERVER ingenalutho, bese usetha i-PREPARE=0. I-cache yendawo kufanele ibe namafayela amancane e-checkpoint, i-vision shard model-00039-of-00039, ifolda enikeziwe ye-model/, i-kit, i-token map, kanye nefayela lesisindo se-rank elifanele.

Clone futhi ulungise indawo yokugcina ikhodi

Clone indawo yokugcina ikhodi ekhanda bese udala ifayela lokumisa lendawo:

git clone https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks.git
cd DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks
cp scripts/local.sh.example scripts/local.sh

Hlela i-scripts/local.sh bese usetha okungenani i-WORKER ibe yi-SSH target yomsebenzi. Uma indlela ye-SSH yomsebenzi ihlukile endleleni ye-direct fabric, setha i-FABRIC_PEER ibe yi-ConnectX-7 address yomsebenzi.

Izilungiselelo eziningi zichazwa ku-scripts/config.sh. Amanani ajwayelekile ahlanganisa:

• MODEL_ID ne-MODEL_REVISION ze-checkpoint ye-EXL3 ephiniwe.
• CONTEXT, okuzenzakalelayo okuyi-1048576.
• PARALLEL, okuzenzakalelayo okuyizicelo ezine ezisebenza ngesikhathi esisodwa.
• HF_TOKEN, uma kudingeka ukuqinisekiswa kwe-Hugging Face.
• ABLIT, ekhetha imodi evamile noma e-abliterated.

Yenza ukuhlola kwe-dry-run ngaphambi kokuqala

Sebenzisa imodi ye-dry-run ukuze uhlole i-link etholakele nemiyalo ye-Docker ngaphandle kokwenza izinguquko:

DRY_RUN=1 ./start.sh

Indawo yokugcina futhi yenza ukuhlolwa kwesakhiwo se-shell futhi iqinisekise izimo ezilindelekile zenethiwekhi, i-Docker, i-GPU, i-SSH, kanye nenkumbulo ngesikhathi sokulungiselela. Ukuqalisa kuyenqaba ukuqhubeka ngaphandle kokuba inkumbulo encane etholakalayo ehleliwe, MIN_AVAIL_GIB, itholakale kuwo womabili ama-Spark.

Qalisa iseva enamarenki amabili

Qalisa konke ukufakwa kusuka ku-head:

./start.sh

Ekuqaliseni kokuqala, iskripthi sakha i-TensorFold image, silanda i-checkpoint namatafula e-Engram, silungise amatafula e-RoPE nedatha yokuhambisa izithombe, sakhe amafayela amathokheni namarank, silungise isitoreji se-worker, bese siqalisa womabili amarank. Ukwakha i-Zig image kuthatha cishe imizuzu eyisithupha ku-Spark futhi kukhiqiza i-image engaba ngu-24.9 GB.

Ngemva kokuqalisa, iskripthi silinda i-API bese sisebenzisa ukuhlolwa okusheshayo. Ukuqalisa okuyimpumelelo kubika ukuthi DeepSeek-V4.1-Flash-1017 iyasebenza kuphothi 8888.

Imiyalo ewusizo yokuphatha umjikelezo wokuphila ihlanganisa:

./start.sh restart
./stop.sh

Sebenzisa i-restart ngemva kokushintsha isilungiselelo sokuphakela. Iskripthi sokumisa sigcina amalogi acindezelwe, sigcine amalogi amasha kakhulu ngokuya nge-LOG_KEEP, ngaphambi kokususa ama-container.

Shayela i-API ehambisana ne-OpenAI

Amaklayenti kufanele asebenzise ikheli le-head elinomzila we-/v1 kanye nesihlonzi esiqondile semodeli DeepSeek-V4.1-Flash-1017. Isibonelo, iklayenti le-Python elihambisana ne-OpenAI lingalungiswa kanje:

from openai import OpenAI

client = OpenAI(
    base_url="http://<head-address>:8888/v1",
    api_key="unused"
)

response = client.chat.completions.create(
    model="DeepSeek-V4.1-Flash-1017",
    messages=[
        {"role": "user", "content": "Explain tensor parallelism briefly."}
    ],
    max_tokens=512
)

print(response.choices[0].message.content)

Imodeli iyacabanga ngaphambi kokuphendula ngaphandle uma ukucabanga kukhutshazwe yisicelo. Nikeza max_tokens eyanele yokucabanga nempendulo yokugcina, noma ulungise izilungiselelo ezifanele zokucabanga nokucubungula endaweni.

Qinisekisa umbhalo, amathuluzi, izithombe, nemiyalezo emide

Indawo yokugcina inamathuluzi okuqinisekisa okuhlola i-endpoint esebenzayo:

python3 tools/validate.py http://<head-address>:8888 --model DeepSeek-V4.1-Flash-1017
API_URL=http://<head-address>:8888 python3 tools/toolcheck.py

Iqoqo lokuqinisekisa lihlanganisa umbhalo, ukucabanga, amathuluzi, okokufaka kwezithombe, nemiyalezo emide. I-README iphawula ukuthi okokufaka kwezithombe nethuluzi lokuqinisekisa akuzange kuhlolwe ngokuphelele kuleli pheya lezingxenyekazi ezibikiwe, ngakho phatha lezi zivivinyo njengokuhlolwa kokufakwa kohlelo, hhayi njengeziqinisekiso.

Ukuze uthole olunye ulwazi lokuxilonga, hlola amalogi amarank:

docker logs -f dsv41-1017-exl3-tf
ssh <worker> docker logs -f dsv41-1017-exl3-tf

Sebenzisa imodi ye-abliterated

Ukucushwa okuzenzakalelayo kusebenzisa i-stock Mia EAQ 2.9-bit checkpoint. Ukusetha ABLIT=1 kugcina leyo checkpoint bese kusebenzisa i-Keys overlay ehambisanayo evela ku-drowzeys/keys-DeepSeek-V4.1-Flash-1017-Abliterated.

Lena yindawo yokugcina elawulwayo ku-Hugging Face. Ngaphambi kokulungiselela, yamukela imigomo yendawo yokugcina bese uqinisekisa nge-akhawunti enokufinyelela:

hf auth login
ABLIT=1 ./scripts/prepare.sh
ABLIT=1 ./start.sh restart

Ngokunye, nikeza ithokheni ngaphandle kokuyibeka emlandweni we-shell:

read -rsp 'Hugging Face read token: ' HF_TOKEN; echo
export HF_TOKEN
ABLIT=1 ./start.sh restart
unset HF_TOKEN

I-overlay iphinwe yi-ABLIT_REVISION. Ukulungiselela kwakha izindlela ezihlukile ze-cache namacache amarank ukuze idatha ye-stock neye-abliterated ingahlanganiswa ngephutha. Ama-shard abhalwe kabusha angadinga isikhala esicishe silingane nenye i-checkpoint, ngaphezu kwe-overlay namafayela amarank. Buyela kumodi ye-stock usebenzisa:

ABLIT=0 ./start.sh restart

I-README isho ngokusobala ukuthi imiphumela yokusebenza nokunemba ye-stock ayiqinisekisi imodi ye-abliterated kule engine. Futhi gcina isilungiselelo esifanayo se-ABLIT ngesikhathi sokulungiselela nokuqalisa.

Isiqondiso sokusebenza nokulungisa

Izilinganiso ezibikiwe ezinombhalo ojwayelekile nezicelo zekhodi zifinyelele cishe kumathokheni angu-74.9 ngomzuzwana kumfula owodwa wombhalo, kanye namathokheni angu-109.9 ngomzuzwana kumfula owodwa wekhodi. Ngemifula emine, ukuphuma okuphelele kufinyelele cishe kumathokheni angu-130.0 ngomzuzwana embhalweni kanye namathokheni angu-155.5 ngomzuzwana kukhodi. I-Prefill ilinganise cishe amathokheni angu-2,000 kuya ku-2,180 ngomzuzwana kuzo zonke izicelo ezisuka kumathokheni angu-8K kuya ku-256K.

Imiphumela incike kokuqukethwe kwesicelo, amawashi e-GPU, imithwalo esebenza ngesikhathi esisodwa, isitoreji, kanye nengcindezi yenkumbulo. Izivivinyo zamagama angahleliwe zibhala okusalungiswa okungasebenzi kahle kunombhalo ojwayelekile noma ikhodi, ngakho izibalo zazo eziphansi zokukhipha akufanele zithathwe njengomkhawulo ojwayelekile we-engine.

• Gcina amatafula e-Engram kudiski yendawo kuwo womabili ama-Spark.
• Sebenzisa imikhawulo efanayo yamawashi e-GPU ngoba amarank e-tensor-parallel avumelanisa kaningi.
• Gcina i-CONTEXT ngaphakathi komthamo wenkumbulo we-head, ikakhulukazi uma isebenzisa enye isofthiwe.
• Sebenzisa i-PARALLEL ngokuqapha lapho uphakela izicelo ezinde noma abasebenzisi abaningi ngesikhathi esisodwa.
• Sebenzisa i-DRY_RUN=1 ngaphambi kokushintsha izilungiselelo zenethiwekhi noma zokufakwa kohlelo.

Izinguquko nezichibiyelo ozikhethela zona ze-engine

Indawo yokugcina inamachibiyelo e-engine ngaphansi kwe-zig/patches/. Acinyiwe ngokuzenzakalelayo futhi adluliselwa kuwo womabili amarank nge-start.sh. I-README ibika ukuthi i-image ezenzakalelayo ihlolwe ukuze kutholakale ama-hashi amathokheni afanayo ezimeni ezihlukahlukene zezicelo nokusebenza ngesikhathi esisodwa.

TF_DS_ENGRAM_LANES=2 inika amandla indandatho yesibili ye-I/O yokufundwa kwe-Engram, kodwa izilinganiso zibonise ukuthi yayihamba kancane kusitoreji esihloliwe. TF_DS_IDX_SPLIT=1 ihlukanisa imigqa ye-prefill indexer phakathi kwama-rank, kodwa ayibonisanga nzuzo elinganisekayo kubhangqa elihloliwe. Uma ama-rank engavumelani ngalesi silungiselelo, ibhangqa lingabhajwa, ngakho sihlala sikhutshaziwe ngokuzenzakalelayo.

I-patch 0005 ithuthukisa ukushintshaniswa kwe-prompt-pass ukuze ama-stream akhona athole imijikelezo ye-decode ngesikhathi enye i-prompt ende isalungiselelwa kusengaphambili. Ukuziphatha kwayo kungalungiswa nge-PREFILL_INTERLEAVE_MS. I-patch 0006 igcina izimo zama-prompt zokufakwa kwezithombe uma TF_DS_IMAGE_CACHE=0 ingasetshenziselwa ukukhubaza lokho kuziphatha.

Gada imemori futhi ugweme osayizi bomongo abangaphephile

Imemori iyisithiyo esiyinhloko ekusebenzeni. I-README ibika ukuthi umongo ongu-512K wawusebenza kahle ku-head enesikhala esicishe sibe ngu-6 GiB esisetshenziswa enye isofthiwe, kanti ukucushwa okungu-655K kwakungehla ngaphansi komkhawulo wokuvikela imemori njengoba ingxoxo ikhula. Kufakwe i-watchdog ngoba i-Spark ingama lapho kufinyelelwa phansi kwememori ngesikhathi sama-prompt amade, anzima futhi asebenza ngesikhathi esisodwa.

Qala ngenani elizenzakalelayo noma eliqaphile le-CONTEXT, lihlole ngomthwalo wakho wangempela, futhi ugade imemori ekhululekile phakathi kwemilando emide. Imemori ekhululekile ingehla njengoba ingxoxo egcinwe kunqolobane ikhula, ngisho nalapho imodeli ilayishe ngempumelelo.

Isiphetho

Le ndawo yokugcina inikeza indlela ezenzakalelayo yokusabalalisa i-DeepSeek-V4.1-Flash-1017 kuma-DGX Spark amabili axhumene ngokuqondile. Uma i-hardware, inethiwekhi ye-RoCE, i-SSH, i-Docker, isitoreji, kanye nokufinyelela ku-Hugging Face sekulungile, i-start.sh iphatha ukusethwa kwezigaba eziningi futhi iveze i-endpoint ejwayelekile ehambisana ne-OpenAI.

Ukuze uthole ukusatshalaliswa okuthembekile, beka phambili isitoreji se-Engram sendawo, amawashi e-GPU afanayo, usayizi womongo oqaphile, kanye nokuqinisekiswa ngokucophelela kwemithwalo yezithombe, amathuluzi, kanye nemongo omude. I-overlay ye-abliterated kanye nokushintsha kwenjini kuyatholakala ukuze kuhlolwe, kodwa izilinganiso zokusebenza nokunemba ezibhaliwe zisebenza ikakhulukazi ku-checkpoint ejwayelekile kanye nokucushwa kwenjini okuzenzakalelayo.