Qalisa i-Qwen3.8 Flash Next ku-DGX Spark eyodwa nge-TensorFold

Uhlolojikelele

Le phrojekthi isebenzisa i-Qwen3.8 Flash Next ku-NVIDIA DGX Spark eyodwa nge-API ehambisana ne-OpenAI. Isebenzisa i-TensorFold v0.3.6.3, i-checkpoint ye-MLX engu-4-bit enekhanda le-MTP draft, kanye neqoqo lama-patch aklanyelwe ukusheshisa ukucutshungulwa kwama-prompt ngaphandle kokushintsha umbhalo okhiqizwayo.

Ukucushwa okuzenzakalelayo kusekela izicelo ezinhlanu ezisebenza ngesikhathi esisodwa, ngasinye sinewindi eligcwele lomongo wemodeli elinama-token angu-262,144. Iphinde ilayishe i-vision tower yemodeli ku-CUDA, okwenza imilayezo yengxoxo ibe nezithombe namavidiyo.

• I-ID yemodeli ye-API: Qwen3.8-Flash-Next
• Indawo yokugcina ezenzakalelayo: http://<spark-address>:8888/v1
• Imifudlana ehambisanayo: 5
• Umongo ngemifudlana: ama-token angu-262,144
• I-KV pool iyonke: ama-token angu-1,310,720
• I-cache ye-KV: int8 ngokuzenzakalelayo
• Izindlela zokufaka: umbhalo, izithombe, namavidiyo

Izici Eziyinhloko

Inference ehambisanayo enomongo omude

ITensorFold inika umfula ngamunye i-cache yawo egcwele yomongo. Ngokucushwa okuzenzakalelayo, imifudlana emihlanu ithola iwindi lama-prompt nezimpendulo elinama-token angu-262,144 ngayinye. I-KV pool ye-int8 ewumphumela idla cishe u-23.4 GiB.

Ukuseva okuhambisana ne-OpenAI

Iseva iveza izindawo zokugcina ezifana ne-/v1/models kanye ne-/v1/chat/completions. Amaklayenti ahambisana ne-OpenAI angaxhuma ngokushintsha i-base URL yawo nokukhetha i-ID yemodeli ethi Qwen3.8-Flash-Next.

Ama-prompt anezindlela eziningi zemidiya

Imilayezo yabasebenzisi ingaba nezingxenye zokuqukethwe ze-image_url ne-video_url zesitayela se-OpenAI. Amafomethi ezithombe asekelwayo afaka i-JPEG, i-PNG, ne-WebP. Amavidiyo angasebenzisa i-MP4, i-WebM, i-MOV, i-MKV, noma enye ifomethi i-FFmpeg engayifunda.

Ukucutshungulwa kwama-prompt okunama-patch

Ama-patch e-TensorFold afakiwe engeza ukufundwa kusengaphambili ku-SSD, ukufundwa kwemicu yomdabu, ukukhethwa okuthuthukisiwe kwamabhulokhi e-sparse attention, izingxenye zama-prompt ezilungisekayo, ama-copy draft, izibali zama-token ezibukhoma, kanye nokusekelwa kombono kwe-Flash Next. Ngokwezilinganiso zekhosombe, ukucutshungulwa kwama-prompt kushesha cishe izikhathi ezingu-1.7 kunenguqulo eqhathanisiwe engenawo ama-patch, kuyilapho izimpendulo zombhalo ezihloliwe zihlala zifana byte-by-byte.

Izidingo Zesistimu

Lungisa i-host ngaphambi kokuklona ikhosombe:

• I-NVIDIA DGX Spark noma olunye uhlelo lwe-GB10 olunememori ehlanganisiwe engu-128 GB.
• Cishe u-115 GiB wememori ekhululekile lapho kuqalwa ukucushwa okuzenzakalelayo.
• I-Docker ene-NVIDIA container runtime.
• I-akhawunti yomsebenzisi eyingxenye yeqembu le-docker.
• Cishe u-160 GB wesikhala sediski esikhululekile ohlelweni olusha.
• Ongakukhetha: i-Hugging Face hf CLI ukuze kulandwe kabusha ngokushesha.
• Ongakukhetha: i-token ye-Hugging Face ku-~/.cache/huggingface/token noma ku-variable yemvelo ethi HF_TOKEN.

Ukusetshenziswa okuzenzakalelayo kuhloselwe i-DGX Spark engasebenzi eminye imisebenzi. Ezinhlelweni ezinememori ehlanganisiwe, ukuphela kwememori kungamisa umshini esikhundleni sokukhiqiza iphutha elijwayelekile lokwabiwa kwememori.

Ukufakwa Nokuqalisa Kokuqala

1. Klona ikhosombe

git clone https://github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold.git
cd Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold

2. Qalisa iseva

./start.sh

Ukuqalisa kokuqala kwenza ukusethwa okuphelele. Idonsa i-container image eyakhiwe ngaphambilini uma ikhona, ilanda i-checkpoint engaba ngu-106 GiB, iyayiqinisekisa, bese ihlanganisa ama-kernel e-CUDA e-GB10. I-image eyakhiwe ngaphambilini icishe ibe ngu-11 GB, kanti i-checkpoint ithatha cishe u-114 GB ku-cache ye-Hugging Face.

Ngesikhathi sokuqalisa, iskripthi sibonisa inqubekelaphambili yokusetha, amalogi eseva, inqubekelaphambili yokulayisha izisindo, nezilinganiso zememori. Bese senza ukuhlolwa okusheshayo. Ukuqalisa okuyimpumelelo kugcina ngomyalezo othi Qwen3.8-Flash-Next iyasebenza ku-port ethi 8888.

Ukuqalisa okulandelayo kweqa umsebenzi wokusetha osuphelile futhi kuthatha cishe imizuzu engu-2.5 ukulayisha izisindo, ngokusho kwe-README.

3. Lungiselela ngaphandle kokuqalisa

Uma ufuna ukulanda nokwakha konke kusengaphambili, sebenzisa:

scripts/prepare.sh

Ukuze wakhe kabusha i-container image kusukela ekuqaleni:

scripts/prepare.sh --rebuild

Iskripthi sokulungiselela sihlola i-Docker, i-NVIDIA runtime, nesikhala sediski. Bese sithola noma sakhe i-TensorFold image, silande i-checkpoint ku-cache ye-Hugging Face, futhi siqinisekise imodeli nge-tensorfold info.

Ukusetshenziswa Okuyisisekelo kwe-API

Hlola imodeli etholakalayo

curl -s http://<spark-address>:8888/v1/models

Faka esikhundleni se-<spark-address> igama le-host noma ikheli le-IP le-DGX Spark. Uma ubiza usebenzisa i-Spark uqobo, ungasebenzisa i-127.0.0.1.

Thumela isicelo sokuqedela ingxoxo

curl -s http://<spark-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "messages": [
      {
        "role": "user",
        "content": "Bhala umsebenzi we-fibonacci we-Python."
      }
    ],
    "max_tokens": 1000
  }'

Imodeli icabanga ngaphambi kokuphendula ngokuzenzakalelayo, futhi ukucabanga kwayo kusebenzisa isabelo sama-token empendulo. Setha i-max_tokens ibe phezulu ngokwanele ukuze ifake kokubili ukucabanga nempendulo yokugcina. Ukucabanga kubuyiselwa ku-reasoning_content, kanti impendulo yokugcina ivela ku-content.

Xhuma iklayenti elihambisana ne-OpenAI

Lungiselela iklayenti ngamanani alandelayo:

• I-Base URL: http://<spark-address>:8888/v1
• Imodeli: Qwen3.8-Flash-Next

I-API isekela ukusakaza, izingcingo zamathuluzi ezinohlobo, okuqukethwe kokucabanga, okokufaka kwezithombe, nokokufaka kwamavidiyo. Amapharamitha amathuluzi afana nama-array abuyiswa kusetshenziswa izinhlobo zawo ezihambisanayo ze-JSON.

Ukuthumela Izithombe

Ngokuzenzakalela, imidiya kufanele inikezwe njengama-URL edatha. Isibonelo esilandelayo se-shell sifaka i-JPEG ngekhodi bese sicela imodeli ukuthi iyichaze:

IMG=$(base64 -w0 photo.jpg)
curl -s http://<spark-address>:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next",
    "messages": [{
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,'"$IMG"'"
          }
        },
        {
          "type": "text",
          "text": "Yini ekulesi sithombe?"
        }
      ]
    }],
    "max_tokens": 2000
  }'

Isicelo singaqukatha izithombe ezingafika kwezine. Isithombe ngasinye singaba ngu-10 MB, kanti umkhawulo ohlanganisiwe ngu-20 MB. Zonke izithombe zihlangene zingasebenzisa amathokheni afinyelela ku-4,096. Ukusetha "detail": "low" kukhawulela isithombe kumathokheni angu-256.

Ukuthumela Amavidiyo

Amavidiyo asebenzisa ingxenye yokuqukethwe ethi video_url:

{
  "type": "video_url",
  "video_url": {
    "url": "data:video/mp4;base64,..."
  }
}

Isicelo ngasinye singaqukatha amavidiyo afinyelela kwamabili. Ividiyo ingaba ngu-64 MB, kanti umkhawulo wesicelo esihlanganisiwe ngu-96 MB futhi ungafaka okuqukethwe okufika ehoreni elilodwa. Ngokuzenzakalela, iseva ithatha amafreyimu amabili ngomzuzwana, ifike kumafreyimu angu-256 asatshalaliswe kuyo yonke ividiyo, futhi ivumela amathokheni evidiyo afinyelela ku-16,384 ngesicelo ngasinye.

Imiyalelo yezithombe namavidiyo ayigcinelwa ukusetshenziswa kabusha kwesiqalo. Izicelo zombhalo ezijwayelekile azithinteki uma kuvunyelwa ukubona.

Vumela ama-URL emidiya asesidlangalaleni e-HTTPS

Ama-URL edatha asetshenziswa ngokuzenzakalela. Ukuvumela iseva ukuthi ilande ama-URL asesidlangalaleni e-HTTPS ezithombe namavidiyo, yiqale kabusha usebenzisa:

VISION_URLS=1 ./start.sh restart

Vumela ukulanda okukude kuphela uma kufanelekile kunqubomgomo yokuphepha nenethiwekhi yokuphakelwa kwakho.

Sebenzisa imodi yombhalo kuphela

Khubaza i-vision tower ukuze unikeze umbhalo kuphela:

VISION=0 ./start.sh restart

Imodi yombhalo kuphela ayilayishi i-vision tower engu-0.84 GiB futhi isebenzisa izingcezu zemiyalelo ezinemigqa engu-4,096 esikhundleni semigqa engu-2,048 esetshenziswa ngokuzenzakalela lapho ukubona kuvuliwe.

Ukuphatha Iseva

Sebenzisa imibhalo yokugoqa ekwindlalifa ukuze wenze imisebenzi evamile:

./start.sh restart
./stop.sh
docker logs -f qwen38-flash-next-tf
curl -s http://<spark-address>:8888/health

• ./start.sh restart iqinisekisa ukusethwa kanye nama-agumenti ngaphambi kokufaka iseva esebenzayo esikhundleni.
• ./stop.sh imisa i-container futhi ikhulule inkumbulo ye-GPU. Izicelo ezisebenzayo ziyaphazamiseka.
• docker logs -f ilandela ilogi yeseva ye-TensorFold.
• /health ibika isimo sokuba matasa kanye nesamba samathokheni esibukhoma.

Uma iseva isivele isebenza, i-./start.sh evamile iyishiya injalo. Kumphathi wezinqubo ofana ne-systemd, sebenzisa imodi yangaphambili:

FOREGROUND=1 ./start.sh

Lungiselela Umongo, Ukuncintisana, kanye Nokunemba kwe-KV

Amanani azenzakalelayo agcinwe ku-scripts/config.sh. Wabhale ngaphezulu usebenzisa okuguquguqukayo kwemvelo noma udlulise amafulegi e-TensorFold ngemva kwe-start.sh.

Yehlisa ukuncintisana ukuze uthole isikhala esengeziwe

PARALLEL=4 ./start.sh restart

Imifudlana emine ye-int8 inikeza i-KV pool yamathokheni angu-1,048,576 futhi inesilinganiso sokuqala esibikiwe esicishe sibe ngu-97.7 GiB.

Sebenzisa iwindi lomongo elifushane

./start.sh restart --context 131072

Amanye ama-agumenti omugqa womyalo adluliselwa ku-tensorfold serve ngemva kwamanani azenzakalelayo, ngakho anokubaluleka kuqala. Sebenzisa ./start.sh --help ukuze uhlole izinketho ezitholakalayo.

Khetha olunye uhlobo lwe-KV cache

KV_DTYPE yamukela bf16, int8, noma int4:

PARALLEL=6 KV_DTYPE=int4 ./start.sh restart

I-README iphawula ukuthi i-int4 ishintsha imiphumela kancane futhi ayinikezi isilinganiso sekhwalithi. Sebenzisa i-int8 uma ufuna ibhalansi ezenzakalelayo yephrojekthi. Sebenzisa i-bf16 uma ukugcinwa kwe-KV ngokunemba okuphelele kubaluleke kakhulu kunenani lemifudlana.

Qonda ukuvikelwa kokuqalisa

I-TensorFold ibala isabelomali sayo ngokusekelwe kwimemori etholakalayo kususwe inqolobane ye-host elingana nengxenye yeshumi ye-RAM. Uma ukucushwa okuceliwe kungalingani, ukuqala kuyenqatshwa ngaphambi kokulayishwa kwezisindo, futhi iphutha likhomba iwindi lomongo okufanele lilingane.

Lawula Ukucabanga kanye Nokuthatha Amasampula

Amanani azenzakalelayo okuthatha amasampula kwimodi yokucabanga yi-temperature 1.0, top-p 0.95, kanye ne-top-k 20. Izicelo zingabhala ngaphezulu kwe-temperature, top_p, top_k, kanye ne-seed. I-temperature engu-0 ivumela i-greedy decoding.

Khubaza ukucabanga kwesicelo esisodwa

{
  "model": "Qwen3.8-Flash-Next",
  "messages": [
    {"role": "user", "content": "Phendula ngomusho owodwa."}
  ],
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 200
}

Setha izinga lomzamo wokucabanga

{
  "chat_template_kwargs": {
    "reasoning_effort": "low"
  }
}

I-template futhi yamukela i-"xhigh". Uma izinga lomzamo wokucabanga linganikezwanga, i-template isebenzisa inani layo elizenzakalelayo eliphakathi. Inkambu esezingeni eliphezulu yesitayela se-OpenAI ethi reasoning_effort iyazitshwa, ngakho yibeke ngaphakathi kwe-chat_template_kwargs.

Amathiphu Okusebenza Athuthukile

Gcina amathebula e-n-gram ku-SSD

I-PLE_ON_SSD=1 ezenzakalelayo ishiya u-29.8 GiB wamathebula e-n-gram ku-SSD esikhundleni sokuwabeka kwimemori ehlanganisiwe. Lokhu kugcinela i-KV cache inkumbulo. Ngakho-ke, isitoreji sasendaweni esisheshayo sibalulekile ekucushweni okuzenzakalelayo.

Lungisa usayizi wesiqephu somyalo ngokucophelela

Uma umbono uvuliwe, i-TENSORFOLD_PREFILL_ROWS iba ngu-2048 ngokuzenzakalelayo ukuze kushiywe indawo ye-vision tower. Imodi yombhalo kuphela iba ngu-4096 ngokuzenzakalelayo. I-README ibika ukuthi imigqa engu-4,096 ingathuthukisa ukucutshungulwa kwe-prompt cishe ngo-2–5% kuma-prompt anama-token acishe abe ngu-3,000, kodwa idinga inkumbulo eyengeziwe engaba ngu-0.94 GiB.

TENSORFOLD_PREFILL_ROWS=4096 ./start.sh restart

Yandisa leli nani kuphela uma kunesikhala esanele senkumbulo.

Lawula i-MTP kanye nama-draft akopishiwe

Ukucushwa okuzenzakalelayo kwe-MTP kukhiqiza ama-draft afinyelela kwayisithupha ngomjikelezo futhi kumisa uchungechunge ngaphambi kokuba i-draft yehle ngaphansi kokuzethemba okungu-60%. Ukulungiselelwa kwe-copy-draft kwenziwa kusebenze ngu-TENSORFOLD_MTP_COPY=1 futhi kudinga okungenani ama-stream amabili asebenza ngokuhambisana. Kuhloselwe ukuthuthukisa izimpendulo ezicaphuna noma ezihlela umbhalo wangaphambilini we-prompt.

Cela ukucaciswa kwereferensi okulandelanayo

Ukuze weqe ukwakhiwa kwama-draft esicelweni esisodwa futhi usebenzise indlela yereferensi ye-TensorFold esebenza nge-token eyodwa ngesikhathi, engeza:

{
  "draft": false
}

Lokhu kuyasiza uma uqhathanisa ukucaciswa kokuqagela nereferensi elandelanayo.

Phoqa noma weqe ukulungiselela

PREPARE=1 ./start.sh restart
PREPARE=0 ./start.sh restart
PULL=0 scripts/prepare.sh

• PREPARE=1 kuphoqa ukuhlolwa kokulungiselela ngaphambi kokuqala kabusha.
• PREPARE=0 kweqa ukulungiselela.
• PULL=0 kwakha i-image endaweni esikhundleni sokuzama i-image eyakhiwe kusengaphambili ekurejista.

Ngemva kokushintsha amafayela aku-patches/, yakha kabusha i-image ukuze lawo ma-patch afakwe ku-TensorFold.

Ukuqinisekisa Nokulinganisa Ukusebenza

Le repository iqukethe amathuluzi axhumana neseva esebenzayo. I-URL ye-API ezenzakalelayo ithi http://127.0.0.1:8888.

tools/bench.py local-test
tools/needle.py
tools/toolcheck.py
tools/visioncheck.py

• tools/bench.py ikala osayizi abahlukahlukene bama-prompt futhi yenza ukuhlolwa okufushane kokucacisa.
• tools/needle.py ifihla i-passphrase ku-prompt enama-token acishe abe ngu-195,000 futhi iqinisekise ukutholwa kwayo.
• tools/toolcheck.py ihlola ukuthi ipharamitha yethuluzi enenani eliyi-array ibuyiswa njenge-JSON array.
• tools/visioncheck.py ithumela i-image ekhiqiziwe enombuthano obomvu nesikwele esiluhlaza okwesibhakabhaka, bese ihlola ukuthi kokubili kuyabonwa yini.

Ukuhlola komunye umshini, setha i-API_URL:

API_URL=http://<spark-address>:8888 tools/bench.py remote-test

Ukusebenza Okulindelekile

Le repository ibika imiphumela elandelayo yokucacisa ku-DGX Spark eyodwa ene-int8 KV caching, amatafula e-n-gram asekelwa yi-SSD, kanye nokwakhiwa kwama-draft nge-MTP:

• Isicelo esisodwa: ama-token angu-62.4 ngomzuzwana, ngesikhathi esingu-152 ms kuze kufike i-token yokuqala.
• Izicelo ezimbili: ama-token angu-90.5 ngomzuzwana esewonke.
• Izicelo ezine: ama-token angu-106.7 ngomzuzwana esewonke.
• Izicelo ezinhlanu: ama-token angu-119.3 ngomzuzwana esewonke, noma ama-token angu-27.0 ngomzuzwana ngesicelo ngasinye.

Isivinini esibikiwe se-prefill sisukela cishe kuma-token angu-2,503 ngomzuzwana ku-prompt enama-token angu-8,229 siye kuma-token angu-2,200 ngomzuzwana ku-prompt enama-token angu-131,110. Lezi zibalo ziyizilinganiso zephrojekthi, hhayi iziqinisekiso, futhi zingahluka ngokuya ngokucushwa, ukwakheka kwe-prompt, okokufaka kwemidiya, isitoreji, kanye nenkumbulo etholakalayo.

Isiphetho

Le repository ihlanganisa imodeli, i-TensorFold runtime, ukusethwa okuqondene ne-CUDA, ama-patch okusebenza kahle, kanye nemibhalo yokusebenza edingekayo ukuze kusetshenziswe i-Qwen3.8 Flash Next ku-DGX Spark eyodwa. Qala ngokucushwa okuzenzakalelayo kwe-int8 okunama-stream amahlanu, uqinisekise isevisi nge-API ehambisana ne-OpenAI, bese ulungisa ukuhambisana kwezicelo, usayizi womongo, umbono, kanye nokunemba kwe-KV ngokuvumelana nomthwalo wakho womsebenzi nesikhala senkumbulo esitholakalayo.