Qalisa i-Qwen3.8 Flash Next 176B Kuma-DGX Spark Amabili

Uhlolojikelele

Le phrojekthi iyiresiphi yokuphakelwa elungele ukukhiqiza, yokuphakela i-RadixArk/Qwen3.8-Flash-Next-NVFP4 isebenzisa i-SGLang. Imodeli iyimodeli ye-mixture-of-experts enamapharamitha acishe abe yizigidi eziyizinkulungwane ezingu-176, engu-135 GB, futhi yenziwe i-NVFP4 quantization.

Isikripthi esinikeziwe se-start.sh sisabalalisa imodeli kuwo wonke amasistimu amabili e-NVIDIA DGX Spark sisebenzisa i-tensor parallelism eno-TP=2. Ama-node axhumana ngesixhumanisi esiqondile se-ConnectX-7 200 Gb RoCEv2. Uma isebenza, i-head node iveza i-API ehambisana ne-OpenAI ku-0.0.0.0:8888.

Le recipe ingaphezu nje kwesiqalisi. Ilanda futhi iqinisekise ama-weight, iwavumelanise ne-worker, yakha i-container image elungisiwe kuwo womabili ama-node, iqalise i-cluster, bese ilinda i-API ukuthi ilungele. Yakhelwe ukuba i-idempotent, ngakho ukuyisebenzisa futhi kungaqhubeka nokulanda okuphazamisekile, kusebenzise kabusha ama-cache, futhi kushintshe ama-container aphelelwe yisikhathi.

Okunikezwa Yi-Recipe

• Ukuphakelwa kwe-tensor parallel kwama-node amabili: I-DGX Spark eyodwa isebenzisa i-rank 0 ne-SGLang router, kanti eyesibili isebenzisa i-rank 1.
• Ama-endpoint ahambisana ne-OpenAI: I-API isekela i-chat, completions, okukhiphayo kokucabanga, ukuhlaziya ama-tool call, nokufaka umbhalo kanye nesithombe.
• Umongo omude: Ubude bomongo obuzenzakalelayo bungamathokheni angu-900000, kusetshenziswa i-YaRN scaling kusukela kumongo wemodeli wendabuko wamathokheni angu-262,144.
• I-NEXTN speculative decoding: Izilungiselelo ezizenzakalelayo ze-speculative zingu-3/1/4, ne-CUDA-graph decode kuwo womabili ama-node.
• Ukuhambisana ne-SM121: I-build isebenzisa i-Triton fallback ye-Qwen Sparse Attention path, ehluleka ukwakheka ku-DGX Spark.
• Imiyalo yokusebenza: Isikripthi sihlanganisa ukuhlolwa kwangaphambili, ukulanda, ukuqalisa, ukumisa, isimo, ama-log, nokuhlola ukusebenza okuyisisekelo.

I-README ibika amathokheni angu-64.4 ngomzuzwana ochungechungeni olulodwa lwe-decode, kanye namathokheni angu-116.8 ngomzuzwana esewonke ngemifudlana emibili ku-cluster enama-node amabili ehloliwe. Ngemifudlana emine, ukuphuma okuhlanganisiwe bekungamathokheni angu-114.1 ngomzuzwana.

Kungani I-SM121 Kernel Patch Idingeka

Isakhiwo se-Qwen4Exp sidlulisela ukunaka nge-backend ye-Qwen Sparse Attention. I-flash-attention resolver yayo ikhetha i-FA2 yakudala, kanti uma ingatholakali isebenzisa i-flash-attn-4 CuTe DSL interface. Esakhiweni se-SM121 se-DGX Spark, le fallback iyehluleka ngesikhathi sokuhlanganisa ngephutha le-MLIR layout-congruence.

Indawo yokugcina ixazulula le nkinga evimba ukuqalisa ngokwakha i-SGLang image ethathwe kwenye equkethe i-Triton attention fallback ehambisana ne-SM121.

Umongo wokwakha okhiqiziwe we-.patch/ uhlanganisa i-qsa_fa_fallback.py, okuyi-kernel yesitayela se-FlashDecoding enobude obuguqukayo, eklanyelwe inkontileka ye-QSA call yemodeli. Isekela i-grouped-query attention, ubukhulu bamakhanda obufika ku-256, ne-online softmax. Iphinde ifunde i-cu_seqlens kudivayisi, okugcina i-CUDA-graph replay isebenza lapho i-backend ibhala kabusha ithebula lokulandelana.

Ngesikhathi kwakhiwa i-Docker, i-recipe ilungisa i-qwen_sparse_attn_backend.py ukuze kukhethwe i-Triton fallback noma nini lapho i-is_sm100_supported() ingamanga. Ngakho-ke indlela ejwayelekile ihlala injalo kuma-GPU e-B100 ne-B200 asekelwayo ezikhungweni zedatha.

I-Hardware Ne-Software Edingekayo

I-hardware ye-cluster

• Amasistimu amabili e-NVIDIA DGX Spark anama-GPU e-GB10, isakhiwo se-SM121, kanye nememori ehlanganisiwe engu-128 GB ngayinye.
• Amaphothi e-ConnectX-7 axhunywe ngokuqondile phakathi kwemishini emibili ngaphandle kwe-switch.
• Ama-interface e-RoCE afanele asebenzayo kuwo womabili ama-node, njenge-rocep1s0f1/enp1s0f1np1 kwenye i-node kanye ne-rocep1s0f0/enp1s0f0np0 kwenye.
• Indawo yokugcina eyanele i-checkpoint ecishe ibe ngu-135 GB, ama-container image, ama-cache, nedatha yesikhashana yokwakha.

I-software ye-head node

• Docker.
• I-Hugging Face hf CLI.
• Ithokheni ye-Hugging Face ku-shell environment uma ukufinyelela ku-checkpoint kudinga ukuqinisekiswa.
• Ukufinyelela kwe-SSH okungenaphasiwedi kusukela ku-head node kuya ku-worker.

Faka i-Hugging Face CLI ku-head node ngalo myalo:

pip install hf

Ngokuzenzakalela, i-worker ifinyelelwa nge-SSH alias ebizwa ngokuthi spark2. Ukungena ku-worker ngokuzenzakalela kusebenzisa igama lomsebenzisi elifanayo nalelo elisetshenziswa ukuqhuba i-start.sh ku-head node. Le mibandela ingashintshwa ku-.env.

Lungiselela I-Cluster Enama-Node Amabili

Clone-a i-repository ku-head node, ngena ku-directory yayo, bese udala ifayela lakho lokumisa:

git clone <this-repo>
cd <this-repo>
cp .env.example .env

Hlela i-.env futhi uqinisekise okungenani i-head RoCE IP, i-worker RoCE IP, kanye nendawo ye-SSH. Okuzenzakalelayo kulindele amakheli alandelayo e-private fabric:

HEAD_CX7_IP=10.0.22.1
WORKER_CX7_IP=10.0.22.2
WORKER_HOST=spark2

Uma i-worker isebenzisa i-akhawunti ehlukile, setha i-WORKER_USER. Ukuze ulawule ngokuphelele indawo ye-SSH, setha i-WORKER_SSH enanini elifana ne-user@host. Ama-environment variable e-shell anokubaluleka kunamanani aku-.env.

I-topology elindelekile

Amaklayenti axhuma ku-API ye-head node kuphothi 8888. I-head isebenzisa i-rank 0, i-SGLang server, ne-router. I-rank 1 isebenza ku-worker. Ithrafikhi ye-NCCL iboshelwe kusixhumanisi esiqondile se-ConnectX-7 RoCEv2 phakathi kwamasistimu amabili.

I-recipe isebenzisa i-NCCL_NET=IB, inika amandla i-RoCE transport ehambisana ne-InfiniBand, futhi ikhubaza i-NVLS ne-CUMEM kulokhu kusetha. Ingaphinde ilayishe kusengaphambili i-host NCCL 2.30.7 build ebekwe endaweni elungiselelwe. Lokhu kuziphatha kunikwe amandla ngokuzenzakalela nge-USE_HOST_NCCL=1.

Qinisekisa I-environment

Qalisa i-preflight eyakhelwe ngaphakathi ngaphambi kokulanda imodeli:

./start.sh doctor

Umyalo we-doctor uhlola i-fabric, ama-GPU, i-RAM, umthamo wediski, ukuxhumeka kwe-SSH, ama-port, nemikhawulo yokusabalalisa. Lungisa noma yiziphi izinkinga zenethiwekhi, zokuqinisekisa ubuwena, zesitoreji, noma zezinsiza ezibikiwe ngaphambi kokuqala ukulanda okukhulu nokwakha i-image.

Landa futhi Uvumelanise Imodeli

Landa i-checkpoint ku-head node, yiqinisekise, bese uyivumelanisa ne-worker:

./start.sh download

I-DOWNLOAD_MODE=rsync ezenzakalelayo yenza i-head ilande amafayela bese iwakopisha ku-worker. Lo msebenzi ungaphinde uqhubeke uma uphazamisekile. Uma ukhetha ukuthi umshini ngamunye uzilandele ngokuzimela ku-Hugging Face, setha:

DOWNLOAD_MODE=direct

Ungayihloli le modeli usebenzisa inketho ye-SGLang ethi --load-format dummy ku-GB10. Ngokohlaziyo lokuphahlazeka olukurepository, i-dummy initializer iguqula okwesikhashana amapharamitha e-fp8 abe yi-fp16. Ithebula le-modeli le-51.2 GB fp8 PLE n-gram lingadala isidingo senkumbulo sesikhashana esingaphezu kuka-150 GB, ngaleyo ndlela limise ngokuphelele uhlelo olunama-DRAM ahlanganisiwe asebenzisekayo angaba ngu-121.7 GB.

Yakha I-image Epatchiwe bese Uqala Ukuseva

Qalisa ukusabalalisa kwama-node amabili usebenzisa:

./start.sh serve

Lo myalo wenza i-preflight, uqinisekisa ukuthi ama-weights nama-image ayatholakala, wakhe i-image elungiswe i-SM121 kuwo womabili ama-node, uqalise i-cluster ye-TP2, bese ulinda i-API ize ilungele ukusetshenziswa. I-image yasendaweni ekhiqiziwe ibizwa ngokuthi qwen38-flashnext-dspark:local.

Ukuqalisa kokuqala kuthatha cishe imizuzu eyishumi ngoba kuhlanganisa ukulayisha ama-weights angu-135 GB, ukuhlanganisa kwe-JIT, kanye nokuthwebula ama-CUDA graph. Iskripthi sivumela kufika ku-WAIT_TIMEOUT_MIN=90 ngokuzenzakalelayo.

Ukuze usebenzise ukuqalisa kokuqala okuzenzakalelayo ngokuphelele, sebenzisa:

./start.sh

Uma kungekho subcommand, iskripthi sisebenzisa i-doctor, i-download, ne-serve ngokulandelana.

Thumela Isicelo Esiyisisekelo Sengxoxo

Ngemva kokuba iseva isilungile, shayela i-chat endpoint yayo ehambisana ne-OpenAI:

curl http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next-NVFP4",
    "messages": [
      {
        "role": "user",
        "content": "Explain RoCEv2 in one paragraph."
      }
    ]
  }'

Ukucabanga kunikwe amandla ngokuzenzakalelayo, futhi okuqukethwe kokucabanga kusakazwa ngokuhlukile. Ukuvala ukucabanga esicelweni esisodwa, engeza i-chat_template_kwargs emzimbeni we-JSON:

{
  "model": "Qwen3.8-Flash-Next-NVFP4",
  "messages": [
    {"role": "user", "content": "Summarize tensor parallelism."}
  ],
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

Ukufaka izithombe kusekelwa nge-content part evamile ye-OpenAI-compatible ethi image_url. I-README iqinisekisa ukuthi ukufaka umbhalo nesithombe ndawonye kuyasebenza, nakuba inganikezi isibonelo esiphelele sesicelo sesithombe.

Xhuma Iklayenti Elihambisana ne-OpenAI

Izinhlelo zokusebenza ezisekela ama-endpoint e-OpenAI angokwezifiso zingasebenzisa le URL eyisisekelo:

http://127.0.0.1:8888/v1

Isikhombi semodeli sithi:

Qwen3.8-Flash-Next-NVFP4

Uma iklayenti lidinga inkambu yokhiye we-API noma i-endpoint yasendaweni ingaqinisekisi izicelo, isibonelo serepository sisebenzisa dummy futhi sikhubaza i-authentication header. Amakhono emodeli nezilungiselelo zokuhambisana ezifanele zihlanganisa:

• Ukucabanga kunikwe amandla.
• Ukufaka umbhalo nezithombe.
• Iwindi lomongo lamathokheni angu-900000.
• Kufika kumathokheni okukhipha angu-32768 ekucushweni kweklayenti okuyisibonelo.
• max_tokens njengenkambu yamathokheni okukhipha.
• Akukho ukusekelwa kwe-developer role ekucushweni kokuhambisana okuyisibonelo.
• Akukho ukusekelwa kwepharamitha ye-reasoning-effort.

Phatha Iseva

I-launcher eyodwa inikeza imiyalo yokuphatha eyinhloko:

• ./start.sh serve yenza ukuhlola kwe-preflight, iqinisekise ukuthi ama-image nama-weights akhona, iqalise i-TP2, bese ilinda ukulungela.
• ./start.sh download ilanda, iqinisekise, futhi ivumelanise ama-weights ngaphandle kokuqalisa iseva.
• ./start.sh stop isusa womabili ama-container futhi inqamule abalandeli bamalogi.
• ./start.sh status ibika isimo sama-container kuwo womabili ama-node.
• ./start.sh logs [N] ibonisa imigqa yokugcina engu-N yamalogi e-head nawe-worker.
• ./start.sh smoke ithumela i-greedy completion esheshayo ku-API esebenzayo.
• ./start.sh doctor iphinda yenze i-preflight ephelele yemvelo neresiphi.

Lapho uhlonza izinkinga zokuqalisa, hlola .serve.log, .sglang.log, kanye ne-.sglang-worker.log. Owokuqala urekhoda okukhipha i-launcher, kanti amabili okugcina aqukethe amalogi e-container ye-head nawe-worker.

Qonda Isabelomali Senkumbulo se-GB10

I-DGX Spark ngayinye iveza ichibi elilodwa lenkumbulo ephathekayo elabiwa ama-CUDA allocations, inkumbulo ye-host eboshiwe, i-Docker, kanye nesistimu yokusebenza. Ngakho-ke iresiphi isebenzisa ingxenye yenkumbulo emile nenokongiwa, kunokuphatha inkumbulo ye-host ne-GPU njengezinsiza ezizimele.

Isakhiwo esibikiwe se-node ngayinye sihlanganisa cishe u-62.5 GB wama-weights e-NVFP4 expert, dense, MTP, kanye ne-vision; cishe u-11 GB wenkumbulo ye-host eboshiwe yethebula le-fp8 PLE; i-KV cache engu-9 GB ye-bf16 ephethe cishe amathokheni angu-956,800; cishe u-3.6 GB wesimo se-Mamba/GDN; kanye no-8 GB wama-CUDA graph nama-workspace e-NCCL noma e-cuBLAS.

Ukwabiwa okuvela ngenxa yalokho, okubonakala ku-CUDA, cishe kungu-95.6 GB, kanti cishe u-17.5 GB uyatholakala emsebenzini owengeziwe. Ithebula le-PLE eliphiniwe alibonakali njengokwabiwa kwedivayisi kwenqubo ngayinye ku-nvidia-smi, kodwa lisasebenzisa yona leyo DRAM ebonakalayo. Bheka inani lelogi le-SGLang elithi available_gpu_mem noma ikholomu etholakalayo ku-free -h lapho uhlola isikhala sangempela esisele.

Gcina i-PLE offloading ivuliwe

Ungafaki --no-ple-offload-embedding ku-GB10. Umthetho ozenzakalelayo we-SGLang ugcina ithebula le-PLE embedding likhishelwe ku-CPU. Ukweqa lo mthetho kungabeka cishe u-26 GB kwiranki ngayinye yememori ye-host ephiniwe ohlangothini lwe-GPU futhi kudlule umthamo womshini.

Gcina isikhala sesikhashana se-prefill

I-QSA indexer idala indawo yokusebenza yama-logits e-fp32 usayizi wayo okhula ngokuya nge-chunk × history. Ngosayizi we-chunk ongu-4096 nomlando wamathokheni angu-300,000, le nto yesikhashana ingadinga cishe u-8 kuya ku-10 GB uma kubhekwa namakhophi ahlobene e-gather nawe-top-k.

Amanani afakiwe akhethwe ngokuqapha ngamabomu:

MEM_FRACTION_STATIC=0.82
CHUNKED_PREFILL_SIZE=1024
MAMBA_FULL_MEMORY_RATIO=0.3

Ungaphakamisi CHUNKED_PREFILL_SIZE ngaphezu kuka-1024 kumongo wamathokheni angu-900,000 ngaphandle kokuprofayela inkumbulo yesikhashana ephezulu uma iqhathaniswa nesabelomali sememori ehlanganisiwe esitholakalayo. I-repository ibika ukuthi usayizi we-chunk ongu-4096 ohlanganiswe no-MEM_FRACTION_STATIC=0.85 wawanele ukubamba imishini.

Gwema ukuhlola i-allocator

Iresiphi yeluleka ngokusobala ngokumelene nokusetha PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Lokhu kulungiselelwa akukakaqinisekiswa kulokhu kuthunyelwa futhi kwakungekho kumarecipe e-DGX Spark aziwa ukuthi asebenza kahle asetshenziswe ngababhali.

Lungisa ama-tunable abalulekile

Setha ama-tunable ku-.env noma uwathumele ku-shell. Amanani e-shell anqoba. Izilungiselelo okungenzeka kakhulu zidinge ukulungiswa yilezi:

• PORT: Okuzenzakalelayo ngu-8888 futhi kubopha i-API kuzo zonke izixhumi.
• CONTEXT_LENGTH: Okuzenzakalelayo ngu-900000.
• MAX_RUNNING_REQUESTS: Okuzenzakalelayo ngu-16, nakuba ukulungiselelwa kwe-Mamba kukhawulela ukuhambisana okusebenzayo kuma-slot angu-14 ngesilinganiso esizenzakalelayo.
• SPEC_STEPS, SPEC_TOPK, kanye ne-SPEC_DRAFT: Okuzenzakalelayo kungu-3, 1, no-4 ngokulandelana ku-NEXTN speculative decoding.
• KERNEL_PATCH: Okuzenzakalelayo ngu-1 ukuze kwakhiwe futhi kusetshenziswe isithombe se-SM121 fallback.
• IMAGE: Isebenzisa isithombe esishiwo ngokuqondile futhi yeqa i-patching.
• CPUSET: Okuzenzakalelayo ngu-5-9,15-19 kuma-CPU cores amakhulu e-GB10. Yisete ibe yinani elingenalutho ukuze ucime i-pinning.
• EXTRA_ARGS: Yengeza izimpikiswano ezengeziwe ze-SGLang ekugcineni, okuvumela ukweqa lapho inani lokugcina linqoba.

Phatha ama-loop okucabanga nawe-tool-call

Inkinga eyaziwayo ye-SGLang ingathinta amasession ahlanganisa ukucabanga okuzenzakalelayo namathuluzi e-OpenAI kanye ne-qwen3_coder tool-call parser. Iseva ingase ikhiphe ngokuphindaphindiwe i-token ID 0, le tokenizer eyibonisa njengo-!, kuze kufinyelelwe ku-max_tokens.

Indlela yokusebenza enconyiwe ukukhubaza ukucabanga kuphela ezicelweni ezithumela amathuluzi:

"chat_template_kwargs": {"enable_thinking": false}

Ukuze ukhubaze ukucabanga kuyo yonke iseva ngenkathi usebenzisa i-tool parser, misa ukuthunyelwa bese ukuqala kabusha nge-override:

./start.sh stop
EXTRA_ARGS='--tool-call-parser qwen3_coder --default-chat-template-kwargs {"enable_thinking":false}' ./start.sh serve

Uma i-EXTRA_ARGS isivele ilungiselelwe ku-.env, hlanganisa izinketho kulokho okulungiselelwe okukhona. Ungaphinde uvule ukucabanga esicelweni esifanayo esihlinzeka ngamathuluzi. Ngaphandle kwe-parser, ukucabanga kungase kusebenze, kodwa ama-tool call avela njenge-<tool_call> XML kokuqukethwe okujwayelekile esikhundleni se-message.tool_calls ehlelekile. I-README iphinde yeluleke ukuthi ugcine izinga lokushisa le-agent liku-0.7 noma ngaphansi lapho usebenzisa le ndlela yokusebenza.

Amathiphu athuthukile okuthembeka nokusebenza

Sebenzisa i-speculative decoding ukuze ukhuphule i-throughput

I-NEXTN speculative decoding iyona eqhuba kakhulu i-throughput ekusethweni okukaliwe. Ngoxhaxha oluzenzakalelayo luka-3/1/4, isinyathelo ngasinye se-decode siqinisekisa amathokheni amane ayidrafti nge-forward pass eyodwa. I-cluster ehloliwe yafinyelela cishe kuma-token angu-117 ngomzuzwana isihlangene ngemifudlana emibili, yabe isima kancane njengoba imifudlana eminingi incintisana nememori esele.

Khetha i-determinism lapho kudingeka

I-Greedy decoding ayikhiqizi imiphumela efanayo ncamashi kuwo wonke amabhithi ekulungiselelweni okuzenzakalelayo. Amathokheni acishe alingane ngezikhathi ezithile angashintsha phakathi kokusebenza, cishe ngenxa yokwehluka kokuncishiswa kwe-floating-point kuma-NVFP4 MoE GEMM ku-SM121. Ukuze uthole ukuziphatha okunensayo kodwa okunqunyekayo, qala kabusha nge:

EXTRA_ARGS="--moe-runner-backend triton" ./start.sh serve

Humusha kahle isexwayiso se-TileLang

Ngesikhathi sokuhlanganiswa kwe-JIT, i-TileLang ingabika ukuthi Logits(bx, position) ibhalwa imicu eminingi. I-repository ikuchaza lokhu njenge-positive engamanga ngoba izikhundla ezibaliwe azihlangani, nakuba isihloli singakwazi ukufakazela leyo mpahla lapho group iyinani lesikhathi sokusebenza. Kungathuliswa nge-PassKey.TL_DISABLE_DATA_RACE_CHECK uma kudingeka.

Vikela ukufinyelela kwangaphandle

I-API ibopha ku-0.0.0.0, okwenza ifinyeleleke nganoma yisiphi isixhumi esivunyelwe. I-README ichaza ukufinyelela okuvela kumakhasimende e-LAN noma e-Tailscale kodwa ayifaki ungqimba lokuqinisekisa. Sebenzisa izilawuli ezifanele ze-firewall ye-host, zenethiwekhi eyimfihlo, noma ze-proxy lapho i-endpoint kungafanele ifinyeleleke kuwo wonke umuntu.

Isiphetho

Le ndawo yokugcina ikhodi iguqula ukuthunyelwa okunama-node amabili okunzima kube ukuhamba komsebenzi okuphindaphindekayo. Umnikelo wayo oyinhloko ukuhlanganiswa kwe-SM121-compatible Qwen Sparse Attention fallback, izilungiselelo zememori ehlanganisiwe ze-GB10 ezibekelwe imingcele ngokucophelela, i-tensor parallelism eqondile ye-RoCEv2, kanye neskripthi esisodwa sokusebenza esine-idempotency.

Qala ngezimiso ezizenzakalelayo ezinikeziwe, sebenzisa ./start.sh doctor ngaphambi kokuthunyelwa, futhi ugweme izisindo mbumbulu, i-PLE offloading ekhubaziwe, izingcezu ze-prefill ezinkulu ngokweqile, kanye nezilungiselelo zokuhlola ze-allocator. Uma i-cluster isilungile, izinhlelo zokusebenza zingasebenzisa iseva yasendaweni ngokusebenzisa izixhumi ezijwayelekile ze-chat ne-completion ezihambisana ne-OpenAI.