Sebenzisa i-GLM-5.3-Flash Kuma-DGX Sparks Amabili nge-TensorFold

Okwenziwa Yile Projekthi

Le projekthi isebenzisa i-GLM-5.3-Flash ezinhlelweni ezimbili ze-NVIDIA DGX Spark isebenzisa i-TensorFold v0.5.0. I-Spark ngayinye isebenzisa i-rank eyodwa esabalalisiwe, kanti i-Spark yokuqala iphinde isingathe i-HTTP API ehambisana ne-OpenAI.

Ukucushwa okuzenzakalelayo kusekela izicelo ezine ngesikhathi esisodwa, iwindi lomongo eliphezulu elingu-1,048,576 tokens ngesicelo ngasinye, okokufaka kwezithombe namavidiyo, ukubizwa kwamathuluzi, okukhiphayo okuhlelekile, ama-endpoint e-tokenization, namamethrikhi e-Prometheus. Ukuxhumana phakathi kwalezi zinhlelo ezimbili kusebenzisa isixhumanisi esiqondile se-ConnectX-7 RoCE.

Ukusatshalaliswa kusebenzisa i-Mia-AiLab GLM-5.3-Flash EXL3 checkpoint. Ama-expert aqondisiwe agcinwa cishe ngamabhithi amane ngesisindo ngasinye, kanti ezinye izisindo ze-checkpoint ziyi-BF16. I-checkpoint ephelele icishe ibe ngu-176 GB.

Izici Eziyinhloko

• I-API ehambisana ne-OpenAI: Sebenzisa ama-endpoint okugcwalisa izingxoxo, okugcwalisa, izimpendulo, nokufakwa kuhlu kwamamodeli.
• Umongo omude: Kufika kuma-token angu-1,048,576 ezaziso nezimpendulo zesicelo ngasinye ngaphansi kokucushwa okuzenzakalelayo kwe-FP8 KV.
• Inference ngesikhathi esisodwa: Qopha izicelo ezifika kwezine ndawonye usebenzisa i-DFlash2 drafter ezenzakalelayo.
• Ichibi le-cache elabiwe: Izingxoxo ezenzeka ngesikhathi esisodwa zisebenzisa ichibi elilodwa le-FP8 KV elinama-token acishe abe yizigidi ezingu-2.1 kuya ku-2.7, kuye ngememori etholakalayo.
• Izicelo ezinemidiya eminingi: Thumela okokufaka kwe-JPEG, PNG, WebP, MP4, WebM, MOV, MKV, namanye amavidiyo anganqunywa nge-FFmpeg.
• Izilawuli zokucabanga: Khetha umzamo wokucabanga ophansi, ophezulu, noma omkhulu, noma ukhubaze ukucabanga kwesicelo.
• Izici zama-agent: Sebenzisa ukubizwa kwamathuluzi nokukhipha okuhlelekile okuboshwe ku-schema nge-xgrammar.
• Ama-endpoint okusebenza: Hlola impilo yesevisi, tokenize noma detokenize okokufaka, bese uqoqe amamethrikhi e-Prometheus.
• Ukusatshalaliswa okuzenzakalelayo: Umyalo owodwa ulungisa yomibili imishini, uqalise womabili ama-rank, wenze ukuhlolwa okusheshayo, bese ubika ikheli le-API.

Izidingo Zehadiwe Nezesoftware

Udinga ama-DGX Spark amabili, noma izinhlelo ezimbili ze-GB10 ezinememori ehlanganisiwe engu-128 GB ngayinye. Cishe u-110 GiB kufanele kube mahhala ku-GPU ngayinye lapho iseva iqala, ngakho qala umise eminye imithwalo emikhulu ye-GPU.

Izidingo zenethiwekhi

Xhuma izimbobo ze-ConnectX-7 ngentambo ye-QSFP bese unikeza isiphetho ngasinye ikheli le-IPv4 ku-subnet yangasese efanayo. Isibonelo, i-head ingasebenzisa i-192.0.2.1/24 kanti i-worker isebenzise i-192.0.2.2/24. Amakheli kufanele afinyeleleke nge-ping, futhi i-RoCE v2 GID kufanele itholakale.

Uma zombili izimbobo ze-CX7 zixhunyiwe futhi zinamakheli, iphrojekthi ingasebenzisa womabili ama-rail. Ngokwezilinganiso ze-README, i-all-gather yesiqephu se-prompt ishesha cishe ngokuphindwe ka-1.8 uma kusetshenziswa ama-rail amabili.

Izidingo ze-SSH nezohlelo

• Ukufinyelela kwe-SSH okusekelwe kukhiye kusuka ku-head Spark kuya ku-worker.
• I-Docker ene-NVIDIA container runtime kuzo zombili izinhlelo.
• I-akhawunti yakho ibe seqenjini le-docker kuzo zombili izinhlelo.
• rsync ifakwe kuzo zombili izinhlelo.
• Cishe u-205 GB wesikhala sediski ku-Spark ngayinye.
• Ngokuzikhethela, i-Hugging Face hf CLI kanye ne-token ye-Hugging Face ku-head.

Dala ukhiye we-SSH uma kudingeka, uwukopishe ku-worker, bese uqinisekisa ukufinyelela okungasebenzisi ukusebenzisana:

ssh-keygen -t ed25519
ssh-copy-id user@192.0.2.2
ssh -o BatchMode=yes user@192.0.2.2 true

I-DFlash2 drafter inelayisensi ye-CC BY-NC-ND 4.0 futhi ingeyokusetshenziswa okungekona okokuhweba kuphela. Khetha i-MTP drafter ye-checkpoint ngaphambi kokusetha kokuqala uma udinga ukugwema ukulanda noma ukusebenzisa i-DFlash2.

Faka Futhi Lungisa Iphrojekthi

1. Clone i-repository ku-head

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh

I-worker ayidingi i-checkout yayo. Imibhalo yokusetha idlulisela i-container image namafayela emodeli adingekayo ukusuka ku-head.

2. Setha ikheli le-worker

Hlela i-scripts/local.sh bese usetha i-WORKER ibe yi-SSH target ye-worker:

WORKER=user@192.0.2.2

Ama-rank asabalalisiwe axhumana ngomzila osetshenziselwa ukufinyelela ku-worker. Uma i-WORKER ibhekisela ekhelini elikunethiwekhi ehlukile, setha i-FABRIC_PEER ibe ikheli langasese le-CX7 le-worker:

WORKER=user@worker-hostname
FABRIC_PEER=192.0.2.2

3. Khetha i-drafter uma kudingeka

Okuzenzakalelayo yi-DRAFTER=dflash2, evumela izicelo ezifika kwezine ngesikhathi esisodwa futhi inikeze ukuthuthuka kokukhipha okulinganisiwe okungaba amaphesenti angu-5 kuya ku-10. Ukuze usebenzise i-MTP head ye-checkpoint, yicushwe ngaphambi kokuqala kokuqala:

DRAFTER=mtp

I-MTP igwema ilayisensi nokulanda kwe-DFlash2, kodwa ngokuzenzakalelayo isebenzisa isicelo esisodwa ngesikhathi.

4. Qalisa womabili ama-rank

./start.sh

Ekusebenzeni kokuqala, iskripthi sihlola zombili izinhlelo, sithola noma sakhe i-container image ecishe ibe ngu-25 GB, silande i-checkpoint ne-drafter ku-head, sikukopishe ku-worker, bese sihlanganisa ama-kernel e-CUDA ku-~/.cache/tensorfold-glm53.

Ukuqala okulandelayo ngokuvamile kuthatha imizuzu emibili kuya kweyisithupha ukulayisha izisindo ezingaba ngu-80 GiB ku-Spark ngayinye. I-launcher ilandela ilogi ye-rank 0, ibike ukuqhubeka kokulayisha, yenze ukuhlolwa okusheshayo okusabalalisiwe, bese iphrinta i-endpoint esebenzayo ku-port 8888.

Ukuze ulungise zombili izinhlelo ngaphandle kokuqala i-API, sebenzisa:

scripts/prepare.sh

Ukuze uphoqe ukwakhiwa kabusha okuhlanzekile kwe-image, sebenzisa:

scripts/prepare.sh --rebuild

Yenza Isicelo Sakho Sokuqala se-API

Isihlonzi esizenzakalelayo semodeli ngu-GLM-5.3-Flash-EXL3, kanti i-URL eyisisekelo ngu-http://<head-address>:8888/v1.

Faka ohlwini amamodeli atholakalayo

curl -s http://<head-address>:8888/v1/models

Thumela ukuqedelwa kwengxoxo

curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [
      {"role": "user", "content": "Bhala umsebenzi we-Python we-fibonacci."}
    ],
    "max_tokens": 2000
  }'

Imodeli icabanga ngaphambi kokuphendula ngokuzenzakalelayo. Ukucabanga kubuyiselwa ku-reasoning_content, kanti impendulo yokugcina ivela ku-content. Nikeza amathokheni okukhipha anele kuzo zombili lezi zigaba. Uma i-content ingenalutho, kungenzeka imodeli idle sonke isabelo sokukhipha ngesikhathi icabanga.

Lawula ukucabanga nokusampula

Setha i-reasoning_effort ibe ngu-low, high, noma max. Sebenzisa u-none ukucela impendulo eqondile ngaphandle kokucabanga:

curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [{"role": "user", "content": "Fingqa i-TCP ngemisho emibili."}],
    "reasoning_effort": "low",
    "temperature": 0,
    "max_tokens": 1000
  }'

I-API yamukela futhi temperature, top_p, top_k, min_p, kanye no-seed. Ngaphandle kwe-seed ecaciswe ngokusobala, i-sampler ithola ukhiye wayo ku-prompt, ngakho izicelo ezifanayo zikhiqiza impendulo efanayo.

Thumela Izithombe Namavidiyo

I-GLM vision tower isebenza ku-rank 0 ngokuzenzakalelayo. Izithombe namavidiyo zimelelwa njengezingxenye zokuqukethwe zesitayela se-OpenAI kumlayezo womsebenzisi.

Hlaziya isithombe sasendaweni

IMG=$(base64 -w0 photo.jpg)
curl -s http://<head-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-Flash-EXL3",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMG"'"}},
        {"type": "text", "text": "Yini ekulesi sithombe?"}
      ]
    }],
    "max_tokens": 2000
  }'

Ngokuzenzakalelayo, iseva yamukela imidiya ngama-data URL. Setha i-VISION_URLS=1 uma iseva kufanele iphinde ilande ama-URL omphakathi e-HTTPS.

Imikhawulo yemidiya esekelwayo

• Kufika ezithombeni ezingu-50 ngesicelo ngasinye, ngomkhawulo ongu-10 MB ngesithombe kanye no-64 MB wedatha yezithombe isiyonke.
• Kufika kumavidiyo angu-4 ngesicelo ngasinye, ngomkhawulo ongu-64 MB ngevidiyo kanye no-96 MB wedatha yamavidiyo isiyonke.
• Ividiyo ngayinye ingaba nokuqoshwa okufika ehoreni elilodwa.
• Umzimba wesicelo ungaba kufika ku-96 MiB, kushiye cishe u-70 MB wemidiya efakwe nge-base64.
• Isithombe sithola amathokheni angafika ku-2,048, kanti zonke izithombe esicelweni esisodwa zabelana ngesabelomali esizenzakalelayo samathokheni angu-16,384.
• Ukusampula kwevidiyo okuzenzakalelayo kungamafreyimu amabili ngomzuzwana futhi kufika kumafreyimu angu-128 kuyo yonke i-clip.

Kuvidiyo, shintsha ingxenye yokuqukethwe kwesithombe ibe ingxenye ye-video_url equkethe i-data URL efana no-data:video/mp4;base64,....

Uma udinga umbhalo kuphela, setha i-VISION=0. Lokhu kugwema ukulayisha i-vision tower futhi kushiye cishe u-1.8 GiB wememori eyengeziwe ku-rank 0 ukuze kusetshenziswe i-cache.

Sebenzisa Futhi Gada Iseva

Qalisa kabusha futhi ume

./start.sh restart
./stop.sh

Kudingeka uqalise kabusha ngemva kokushintsha izilungiselelo ezigcinwayo. Ukumisa isevisi kukhulula imemori ye-GPU kuwo womabili amakhompyutha.

Hlola amalogi

docker logs -f glm53-flash-tf
ssh <worker> docker logs -f glm53-flash-tf

Umyalo wokuqala ulandela i-rank 0 ku-head. Owesibili ulandela i-rank 1 futhi uwusizo ikakhulukazi lapho i-worker iphuma ngokungalindelekile.

Hlola impilo namamethrikhi

curl -s http://<head-address>:8888/health
curl -s http://<head-address>:8888/metrics

Impendulo yezempilo ibika isimo sokuba matasa, ama-stream asebenzayo, kanye nomthamo we-shared-pool, okuhlanganisa i-pool_tokens ne-pool_free_tokens. I-endpoint yamamethrikhi iveza izibalo zezicelo ze-Prometheus, ukubambezeleka, isikhathi kuze kufike ithokheni yokuqala, kanye nempilo.

Qonda Umongo Nememori Ye-KV Eyabiwe

Ukucushwa okuzenzakalelayo kusebenzisa i-FP8 KV cache futhi kunikeza isicelo ngasinye iwindi eliphezulu lamathokheni angu-1,048,576. Izicelo ezine zabelana nge-pool elinganiselwa kumathokheni angu-2,684,928 ekuqaleni okukodwa okubhalwe phansi, nakuba umthamo oqondile uncike kwimemori ekhululekile.

Isicelo esisodwa singasebenzisa iwindi eligcwele, kodwa zonke izingxoxo ezisebenzayo nezilondoloziwe kufanele zilingane ku-pool evamile. Ngokwesibonelo, izingxoxo ezimbili zingasondela kumathokheni ayisigidi ngasinye, kanti izingxoxo ezimfushane ezisebenza ngesikhathi esisodwa zingabelana ngomthamo osele. Isicelo esingakabekwa okwamanje siyalinda ezinye izicelo zize ziqede.

Sebenzisa i-CONTEXT=0 ukubala iwindi elikhulu kunawo wonke elilinganayo, kodwa qaphela ukuthi lokhu akushiyi memori yokugcina izimo zama-prompt zezinye izingxoxo. Ukushintshela kwisitoreji esiqondile se-BF16 KV kwehlisa kakhulu iwindi elitholakalayo.

Ukuphela kwememori ehlanganisiwe kungamisa i-Spark esikhundleni sokukhiqiza iphutha elicacile lokwabiwa. Gcina i-reserve ezenzakalelayo ngaphandle kokuthi uqonde ukuhwebelana kwememori, futhi unyuse i-MEMORY_RESERVE_GIB lapho eminye imisebenzi yabelana ngamakhompyutha.

Ukucushwa Okuthuthukile

Izilungiselelo zinganikezwa nge-environment, ku-scripts/local.sh, noma kufayela le-.env eliseduze ne-start.sh. Ukuhleleka kokubaluleka yi-environment, kulandele scripts/local.sh, bese kuba i-.env, futhi ekugcineni kube yizilungiselelo ezizenzakalelayo ze-repository.

Shintsha ukuhambisana kokusebenza kohlelo okukodwa

PARALLEL=2 ./start.sh restart

PARALLEL ingasethwa ukusuka kokukodwa kuye kwezine uma i-DFlash2 inikwe amandla. I-MTP ngokuzenzakalela icubungula isicelo esisodwa ngesikhathi.

Shintsha iwindi lomongo

CONTEXT=524288 ./start.sh restart

Umongo omncane unganciphisa ingcindezi yenqolobane lapho iwindi eligcwele lamathokheni ayisigidi lingadingeki.

Khulisa umkhawulo wokuphumayo ozenzakalelayo

./start.sh restart --max-tokens 16384

Izimpikiswano ezengeziwe zomugqa womyalo zidluliselwa ku-tensorfold serve ngemva kwezilungiselelo ezizenzakalelayo ezimisiwe, ngakho ziba nokubaluleka kuqala. Sebenzisa ./start.sh --help ukuze uhlole izimpikiswano ezisekelwayo.

Phoqa noma weqe ukulungiselela

PREPARE=1 ./start.sh restart
PREPARE=0 ./start.sh restart

Umyalo wokuqala uphoqa ukuhlolwa kokusetha kanye nenqubo yokulungiselela. Owesibili uyeqa le nqubo.

Sebenzisa kabusha imiyalelo ngempumelelo

Ukusetshenziswa kabusha kwesiqalo esabiwe kunikwe amandla ngokuzenzakalela nge-SHARED_PREFIX=1. Imiyalelo yesistimu ephindwayo kanye nemiyalelo egciniwe efanayo ingaqhubeka esimweni esigcinwe kunqolobane esikhundleni sokuthi ilandwe kusengaphambili futhi. Ezilinganisweni ezibhaliwe, umyalezo ophindwayo wamathokheni angu-64K wehle usuka kumasekhondi angu-34 waya ngaphansi kwamasekhondi angu-0.07, kanti ukusetshenziswa kabusha komyalezo wesistimu wamathokheni angu-7.9K kwehlise ukuqala kusuka kumasekhondi angu-4.24 kuya kumasekhondi angu-0.13.

Khetha ukunemba kwenqolobane nezisindo

• KV=fp8 kuyisilungiselelo esizenzakalelayo futhi kunika amandla iwindi lomongo elikhulu kakhulu.
• KV=bf16 isebenzisa ukugcinwa kwenqolobane okunembile kodwa inciphisa kakhulu iwindi elitholakalayo.
• DENSE=q4 kuyisilungiselelo esizenzakalelayo sezingxenye ze-checkpoint eziminyene.
• DENSE=fp8 kanye ne-DENSE=bf16 ezinye izindlela ezitholakalayo, ezinemithelela ehlukene enkumbulweni.

Amakhono Engeziwe e-API

Ngaphezu kokuqedelwa kwengxoxo, iseva iveza i-/v1/completions, i-/v1/responses, i-/v1/models, i-/tokenize, i-/detokenize, i-/health, ne-/metrics. Imizila yokuhlukanisa amathokheni iyatholakala futhi ngaphansi kwe-/v1/. Iseva ayinikezi isiphetho esihambisana ne-Anthropic esithi /v1/messages.

Ukubiza amathuluzi kusekela i-tools kanye nama-tool_calls asakazwayo, nezimpikiswano ezibhalwe ngokohlaka olunikeziwe. Ukukhiqiza okuhlelekile kusekela i-json_object, i-json_schema, i-JSON eqondisiwe, izinkulumo ezivamile, ukukhetha, ama-grammar, kanye nezinye izinkambu zokuphumayo okuhlelekile eziphoqelelwa nge-xgrammar ngemva kokuqedwa kokucabanga.

Uma umyalezo kanye ne-max_tokens kudlula iwindi elilungiselelwe, i-API ibuyisa i-HTTP 400 enekhodi yephutha yesitayela se-OpenAI ethi context_length_exceeded.

Ukuxazulula Izinkinga Ezivamile Zokuqalisa

• Inkumbulo enganele: Uma ukuqala kubika ukuthi kutholakala ama-GiB ambalwa kuphela, hlola docker ps bese umisa eminye imithwalo ye-GPU kuwo womabili ama-Spark.
• Ayikho idivayisi noma umzila we-RoCE: Qinisekisa ukuthi izixhumi ze-CX7 zinamakheli e-IPv4 nge-ip -4 addr. Uma i-SSH isebenzisa enye inethiwekhi, lungiselela i-FABRIC_PEER ngekheli le-CX7 lomsebenzi.
• Inqolobane yomsebenzi ayibhaleki: Lungisa ubunikazi benqolobane ye-Hugging Face yomsebenzi uma ikhonteyina yangaphambilini iyishiye iphethwe yi-root.
• Izinga elilodwa liyaphuma: Hlola amalogi kuwo womabili amakhompyutha. Isiqalisi sivamise ukumisa izinga elisele uma olunye uhlangothi luhluleka.
• Ukusetha kushintshile: Ngemva kokulungisa ama-patch noma ukushintsha i-TensorFold, isithombe esiyisisekelo, noma okokufaka kokwakha okuhlobene, sebenzisa scripts/prepare.sh --rebuild.

Okulindelekile Ngokusebenza

Ezilinganisweni ezizenzakalelayo zama-Spark amabili ku-README, ukubhala okuqhubekayo kusuka kumathokheni angu-60.4 ngomzuzwana ngesicelo esisodwa kuya kumathokheni ahlanganisiwe angu-108.8 ngomzuzwana ngezicelo ezine ezisebenza ngesikhathi esisodwa. Ukukhiqiza okuhlelekile kusuka ku-114.7 kuya ku-227.9 wamathokheni ahlanganisiwe ngomzuzwana. Ukugcwalisa kwangaphambili bekucishe kube amathokheni angu-1,952 ngomzuzwana kumyalezo wamathokheni angu-8,219, kanye namathokheni angu-1,015 ngomzuzwana kumyalezo we-needle-in-a-haystack wamathokheni angu-981,841, lapho inaliti yatholakala khona.

Lezi zibalo zikalwe kusetshenziswa imifudlana emine, inqolobane ye-KV ye-FP8, izisindo eziminyene zamabhithi amane, i-DFlash2 kanye nama-draft e-copy, umbono unikwe amandla, kanye namawashi e-GPU akhawulelwe ku-2,200 MHz. Zibheke njengemiphumela eyireferensi yalokho kulungiselelwa, hhayi iziqinisekiso zazo zonke izindawo.

Isiphetho

Lokhu kusatshalaliswa kuhlanganisa ukulungiselelwa okuyinkimbinkimbi kwe-TensorFold enamanodi amabili ngemuva kwenqubo yokuqalisa esebenzayo kanye ne-API ehambisana ne-OpenAI. Uma isixhumanisi se-CX7, ukufinyelela kwe-SSH, isikhathi sokusebenza se-Docker, isitoreji, nekheli lomsebenzi sekulungile, i-./start.sh iphatha ukulungiselela, ukuqaliswa okusabalalisiwe, ukuqinisekiswa, kanye nokunikezela. Kusukela lapho, ungasebenzisa amaklayenti e-API ajwayelekile kuyilapho ulungisa inani lezicelo ngesikhathi esisodwa, umongo, ukucabanga, imikhawulo yemidiya eminingi, nokuziphatha kwenkumbulo ukuze kuhambisane nomthwalo wakho womsebenzi.