Sebenzisa i-Qwen3.8-Flash-Next ku-Colab A100 nge-API Ehambisana ne-OpenAI

Okwenziwa yi-collabosm

collabosm isebenzisa i-Qwen3.8-Flash-Next ku-Colab A100 80 GB High-RAM eyodwa. Le modeli iyi-mixture-of-experts engu-125B-A6B enesakhiwo esihlanganisa i-Gated-DeltaNet ne-full-attention, kanye nobude bomongo bomdabu obungamathokheni angu-262,144.

Iphrojekthi ihlinzeka noma ibuyisela i-Colab VM, ifake i-ExLlamaV3 runtime ephiniwe, ilande ama-model weights aphiniwe, iqalise iseva ye-HTTP ehambisana ne-OpenAI, bese iyiveza nge-Cloudflare tunnel. Ama-client axhuma ku-URL ye-tunnel ngaphansi kwe-/v1 esebenzisa ukhiye we-bearer okhiqiziwe.

I-ExLlamaV3 iyona kuphela injini ye-inference esetshenziswa yile repository. Iseva ye-API ngokwayo yenziwe yi-collabosm ngoba i-ExLlamaV3 ayifaki iseva ye-HTTP.

Izici ezibalulekile

• Ukusetshenziswa kwe-GPU eyodwa: isebenzisa cishe u-63.6 GiB wama-weight ahlala ku-VRAM ku-A100 80 GB High-RAM runtime eyodwa.
• Izixhumi ezihambisana ne-OpenAI: isekela kokubili /v1/chat/completions kanye /v1/responses, kuhlanganise ne-incremental streaming.
• Ukubuyiselwa kwe-Colab okuphephile: iphinda ixhume ama-assignment ashiywe dengwane esikhundleni sokuthemba amarekhodi eseshini e-CLI agcinwa endaweni futhi alahlwayo, noma idale ama-VM aphindwe kabili akhokhiswayo.
• Ukuqinisekisa i-hardware: iyenqaba futhi imise i-A100 engu-40 GB engahambisani ngaphambi kokulayisha imodeli.
• Ukugcina ama-prompt cache unomphela: igcina i-ExLlamaV3 Generator eyodwa esebenza isikhathi eside ukuze i-page table ne-pinned-RAM cache kuqhubeke nokusebenza phakathi kwezicelo.
• Idizayini ye-cache enamazinga amabili: ihlanganisa umthamo we-GPU KV ne-pinned host-RAM cache ongakhethwa ukuze ivuselele ngokushesha izingxoxo ezingasebenzi.
• Ukuhlola i-protocol endaweni: inikeza injini mbumbulu esebenzisa i-HTTP handler yangempela ngaphandle kwe-GPU noma ama-model weights.
• Umbono ongakhethwa: ingalayisha ingxenye ye-vision yemodeli futhi yamukele ama-input ezithombe alawulwayo.
• I-client ne-UI yasendaweni: kufaka i-terminal chat, i-browser UI, amakhasi esimo, amamethrikhi okusebenza atholakala ku-stream, kanye ne-proxy egcina ukhiye we-upstream ungabonakali ku-browser.

Izilinganiso ezibikwe yiphrojekthi zifaka kufika kumathokheni angu-3,882 ngomzuzwana ku-prefill ye-prompt engu-30K ene-GCS=8192, amathokheni angu-97.4 ngomzuzwana ku-decode ene-MTP depth engu-4 kumongo ongu-30K, kanye namathokheni angu-90.1 ngomzuzwana kumongo ongu-114K. Bheka docs/MEASURED.md ukuze uthole umsuka walezi zilinganiso kanye nemikhawulo yazo; ungazithathi njengeziqinisekiso.

Izidingo nokuhlela umthamo

Ngaphambi kokuqala, qinisekisa ukuthi indawo yakho ihlangabezana nazo zonke lezi zidingo:

• Uhlelo lwe-Colab olukwazi ukwaba i-A100.
• Uhlobo lomshini lwe-HIGH_RAM. I-A100 ejwayelekile engu-40 GB ayikwazi ukulayisha le modeli.
• Cishe u-110 GiB wesikhala sediski ye-Colab sama-weight.
• Ithuluzi le-Google Colab lomugqa womyalo, elifakwe nge-uv tool install google-colab-cli.
• Ngokuzikhethela, i-GitHub CLI uma uhlela ukwenza i-fork noma i-push ye-repository.

uv tool install google-colab-cli

Imfuneko ye-High-RAM iyimpoqo. Ungakhombi ikhadi elifanele ngokusesha okukhiphayo kwe-hardware uthi “80,” ngoba i-A100 ingabika i-compute capability sm_80 noma ine-VRAM engu-40 GB kuphela. I-collabosm iqhathanisa inani elibikiwe le-vram_GiB esikhundleni salokho.

Iphrojekthi yathuthukiswa ku-Colab Pro tier enamayunithi e-compute acishe abe ngu-200 ngenyanga. Izindleko zayo ezilinganisiwe ze-A100 High-RAM zazingu-7.52 CU ngehora, cishe u-$0.75 ngehora endaweni ebhaliwe. Amanani nokutholakala kokwabiwa kungashintsha ngokuzimele kwi-repository.

Qalisa iseva

1. Vula ifolda ye-repository

Landa noma wenze i-clone ye-collabosm repository, bese usebenzisa imiyalo elandelayo usuka kufolda yayo eyimpande. Ama-script alindele izindlela ezihlobene ne-repository ezifana ne-scripts/up.sh, scripts/bootstrap.sh, kanye ne-scripts/serve.sh.

2. Hlinzeka, lungisa, bese usebenzela

bash scripts/up.sh

Lo myalo owodwa ubuyisela i-assignment ekhona uma kungenzeka noma udale i-High-RAM assignment entsha, ulayishe iphrojekthi, ufake i-runtime ephiniwe, ulande ama-weight aphiniwe, uqalise i-API, uqalise i-Cloudflare tunnel, bese ulinda ukuhlolwa kwezempilo okuphumelelayo.

Ukusetshenziswa okuphumelelayo kuphuma ngesimo 0 kuphela ngemva kokuba kokubili i-API ne-URL yomphakathi ye-tunnel sekulungile. Umyalo uphrinta i-URL ye-tunnel nokhiye we-API. Ukhiye ubuye ugcinwe ku-VM ku-/content/api-key.txt.

I-runtime ikhethwa ku-ExLlamaV3 wheel eyakhiwe ngaphambilini futhi ephiniwe, ehambisana ne-Colab Python, PyTorch, kanye ne-CUDA environment. Ama-weight alandwa e-Hugging Face ku-revision ephiniwe. Uma indlela yokulanda ye-Xet ima, i-README ikhomba i-HF_HUB_DISABLE_XET=1 njengendlela yokwenza okunye.

3. Rekhoda izilungiselelo zokuxhuma

base_url = https://<host>.trycloudflare.com/v1
api_key  = <printed API key>
model    = qwen3.8-flash-next-exl3

Igama le-quick tunnel lingashintsha phakathi kokuqaliswa. Ukuze uthole igama elizinzile, lungisa i-Cloudflare tunnel eqanjwe nge-TUNNEL_TOKEN bese uyihlanganisa ne-PUBLIC_URL yekheli eliboniswa ekuphumeni kwesimo.

4. Hlola ukucushwa okusebenzayo

Cela GET /v1/status ukuze ubone amapharamitha inqubo esebenzayo eyawathola ngempela, kuhlanganise nezilungiselelo ze-cache, amapharamitha okuqalisa, isikhathi esisukela ekuqalisweni, ukutholakala kwe-vision, kanye nenqubomgomo yama-input ezithombe.

I-repository iphawula ukuthi akuzona zonke izilungiselelo zayo ezizenzakalelayo eziqinisekisiwe ndawonye. Hlola i-status endpoint esebenzayo bese ubheka docs/MEASURED.md ngaphambi kokucabanga ukuthi inhlanganisela ethile iqinisekisiwe.

Lungisa ama-cache nokukhiqiza

Zonke izilawuli eziyinhloko zokuqalisa ziyiziguquguquko zemvelo ezidluliselwa ku-scripts/up.sh. Isibonelo:

SESSION=mybox CACHE_SIZE=524288 CPU_CACHE_GB=8 bash scripts/up.sh

• SESSION: igama leseshini yasendaweni; okuzenzakalelayo ngu-collabosm.
• CACHE_SIZE: inani eliphelele lamathokheni e-KV kuyo yonke imisebenzi; okuzenzakalelayo ngu-500224 futhi kufanele libe ukuphindaphindwa kuka-256.
• CACHE_QUANT: ububanzi bamabhithi bekheshi ye-KV; okuzenzakalelayo ngu-4, futhi kuvunyelwe amanani asuka ku-2 kuya ku-8.
• CPU_CACHE_GB: ikheshi yamakhasi e-KV esigaba sesibili esisebenzisa i-RAM ephiniwe; okuzenzakalelayo ngu-32 GiB, kanti u-0 uyayikhubaza.
• RECURRENT_CACHE_GB: isitoreji se-RAM yomsingathi samaphoyinti okuhlola e-Gated-DeltaNet; okuzenzakalelayo ngu-24 GiB.
• GCS: usayizi wengxenye ye-generator kanye nesilawuli esibalulekile sokusebenza kwe-prefill; okuzenzakalelayo ngu-8192.
• NDT: ukujula kokusalungiswa kokubikezela kwamathokheni amaningi; okuzenzakalelayo ngu-4.
• RUNTIME: wheel wesikhathi sokusebenza esakhiwe ngaphambilini noma source sokusethwa okusekelwe kukhodi yomthombo.
• TUNNEL_TOKEN: ithokheni yokuzikhethela yomhubhe oqanjwe igama.

Amakheshi e-RAM yomsingathi awawona umthamo wamahhala. Ikheshi ye-CPU ezenzakalelayo yabelwa ngokuphelele njengememori ephiniwe, futhi ikheshi ephindaphindekayo nayo isebenzisa i-RAM eningi. Zihlala kanye nethebula le-n-gram elingu-36.4 GiB elibhalwe endaweni yokugcina.

Isigaba se-RAM ephiniwe sithuthukisa isikhathi sokuqhubeka kwezingxoxo ezingasebenzi amakhasi azo angabhekiselwanga kuwo akhishwe. Asikhulisi inani lezimo eziphilayo ngesikhathi esisodwa, ngoba amakhasi aphilayo awakwazi ukukhishwa sengathi i-RAM yomsingathi iyi-swap evamile.

Shayela i-API ehambisana ne-OpenAI

Faka kuhlu imodeli

curl https://<host>.trycloudflare.com/v1/models \
  -H "Authorization: Bearer <api-key>"

Isiphetho sibuyisa isihlonzi semodeli esisodwa esithi qwen3.8-flash-next-exl3. Ukuqinisekisa kuyadingeka kulo mzila we-API, kanti i-/health ayidingi ukuqinisekiswa futhi ibuyisa u-ok osobala.

Dala ukuqedelwa kwengxoxo

curl https://<host>.trycloudflare.com/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-flash-next-exl3",
    "messages": [
      {"role": "user", "content": "Reply with exactly: pong"}
    ],
    "max_tokens": 16
  }'

Isixhumi sengxoxo sibuyisa okuqukethwe ngaphansi kwe-choices[].message.content, isizathu sokuqeda esithi stop noma length, kanye nolwazi lokusetshenziswa olungafaka amathokheni omyalo agcinwe ekheshi.

Sakaza ukuqedelwa

Setha i-stream ibe yi-true ukuze uthole i-Server-Sent Events ngesikhathi imodeli ihlaziya. Ekuqedweni kwezingxoxo, ukusakaza kugcina ngo-[DONE]. Ukusetha stream_options.include_usage kucela idatha yokusetshenziswa empendulweni esakazwayo.

{
  "model": "qwen3.8-flash-next-exl3",
  "messages": [{"role": "user", "content": "Explain prompt caching briefly."}],
  "stream": true,
  "stream_options": {"include_usage": true},
  "max_completion_tokens": 200
}

Isiphetho se-/v1/responses naso sisekela ukusakaza. Ukulandelana kwemicimbi yaso kufaka imicimbi yokudala, inqubekelaphambili, into yokukhipha, ingxenye yokuqukethwe, umehluko wombhalo, ukuqedelwa, kanye nemicimbi yempendulo yokugcina. Wonke umcimbi osakazwayo uphethe i-sequence_number ekhuphuka ngokulandelana, kanti umehluko uphethe izinkomba zento nokuqukethwe ezidingwa amaklayenti ukuze azibophe ngendlela efanele.

Sebenzisa imodi yokucabanga

Ukucabanga kukhutshaziwe ngokuzenzakalelayo ukuze kuhambisane namaklayenti ezingxoxo ajwayelekile. Kwenze kusebenze nge-enable_thinking: true noma ngenani elisekelwayo le-reasoning_effort elithi xhigh, medium, noma low. Esibonakalayo sengxoxo, umkhondo wokucabanga ubuyiswa ngokuhlukile ku-message.reasoning_content futhi awungeni endaweni yokuqukethwe evamile.

temperature kanye ne-top_p ziyamukelwa kodwa okwamanje azinakwa ngoba iseva yakha imisebenzi nge-sampler=None. Izilawuli zokucabanga, max_tokens, kanye nokulandelana kokuma ziyasebenza.

Sebenzisa iklayenti elifakiwe

Ikhasimende lendawo elithi collabosm.py lisebenzisa kuphela umtapo ojwayelekile we-Python futhi alidingi ukufakwa okuhlukile. Qalisa ukumiswa kwalo ngephoyinti lokugcina lomhubhe nokhiye okhiqiziwe:

python collabosm.py config --init \
  --endpoint https://<host>.trycloudflare.com/v1 \
  --api-key <key>

Ngemva kwalokho ungaxoxa kutheminali noma uvule isixhumi esibonakalayo sesiphequluli sendawo:

python collabosm.py chat "hello"
python collabosm.py ui
python collabosm.py start

ui isebenzisa ikhasi kanye ne-proxy ku-http://127.0.0.1:8790. start ivula amawindi engxoxo nesimo. Ikhasimende aliyiqali noma liyimise i-Colab VM, ngakho ukulivula ngokwalo akukwazi ukusebenzisa amayunithi ekhompuyutha.

Nikeza iklayenti elikhona elihambisana ne-OpenAI i-proxy yasendaweni esikhundleni sephoyinti lokugcina le-VM uma ufuna i-proxy ifake ukhiye wangempela we-bearer:

base URL : http://127.0.0.1:8790/v1
API key  : anything
model    : qwen3.8-flash-next-exl3

Lo mklamo ugcina ukhiye wangempela ngaphakathi kwenqubo ye-proxy yasendaweni, ugwema inqubomgomo ebanzi ye-CORS, futhi uvumela isixhumi esibonakalayo sesiphequluli ukuthi sibuyekezwe ngokuzimele ku-VM.

Hlola iphrothokholi ngaphandle kwe-GPU

Sebenzisa i-stub yokuthuthukisa ukuze uqinisekise ukuziphatha kwezicelo nokusakaza ngaphandle kokukhokhela i-A100 noma ukulanda izisindo:

python scripts/dev_stub.py --port 8099 --chunk 24 --delay 0.01
python scripts/check_surface.py --base http://127.0.0.1:8099/v1

I-stub ingenisa i-handler yangempela yokuthumela futhi ithathela indawo kuphela umsebenzi wayo wengxenye ye-engine. I-surface checker yenza ukuhlola okungu-13, okuhlanganisa ama-delta okuqala, imicimbi yokugcina, ukuhambisana kwe-chat nokuphumayo kwe-Responses, ama-index ento adingekayo, kanye nezinombolo zokulandelana ezikhuphuka ngokulandelana. Iphuma ngesimo 1 uma ukuhlola kwehluleka.

Faka --think emyalweni we-stub ukuze uhlole umjikelezo wokuphila kwe-reasoning item. Lokhu kuwusizo kakhulu ngaphambi kokushintsha izimo zemicimbi ye-stream noma ukuxhuma amaklayenti aqinile afana ne-Codex CLI.

Nika amandla futhi uhlole i-vision ngokucophelela

I-model pack iyakwazi ukusebenza ngezinhlobo eziningi zemidiya, kodwa i-vision ivaliwe ngokuzenzakalela ngoba izindleko ezengeziwe ze-VRAM ze-vision tower azikalinganiswa ekhadini eselihlala kakhulu. Nika amandla ingxenye ye-vision ehlukile ngokucacile:

VISION=1 bash scripts/up.sh

Izilawuli zezithombe ozikhethela zona zihlanganisa:

IMAGE_URLS=1
MAX_IMAGE_BYTES=12582912
MAX_IMAGES=8

Zombili izilimi ze-API ezisekelwayo zamukela izingxenye zezithombe ezijwayelekile ze-OpenAI. Ama-Data URL ayamukelwa uma i-vision inikwe amandla. Ama-URL ezithombe akude e-HTTP noma e-HTTPS ahlala evaliwe ngaphandle kokuthi IMAGE_URLS=1.

{
  "role": "user",
  "content": [
    {"type": "text", "text": "What colour is this image?"},
    {
      "type": "image_url",
      "image_url": {"url": "data:image/png;base64,..."}
    }
  ]
}

Uma ama-URL akude enikwe amandla, iseva ixazulula igama le-host futhi idinga ukuthi ikheli ngalinye lifinyeleleke emhlabeni wonke. Yenqaba amakheli e-loopback, ayimfihlo, e-link-local, nawemetadata futhi ayilandeli ama-redirect. Izindlela zesistimu yamafayela azamukelwa nhlobo. Uma isithombe singakwazi ukufakwa, iseva ibuyisa iphutha le-JSON 400 elinohlobo vision_unavailable esikhundleni sokungasinaki isithombe buthule.

Hlola GET /v1/status ukuze uthole ukuthi i-vision iyasebenza yini, bese uqinisekisa ukuziphatha okunikwe amandla noma okunqatshwe ngamabomu usebenzisa:

python scripts/check_vision.py \
  --base https://<host>.trycloudflare.com/v1 \
  --key <key>

Amathiphu e-concurrency nokusebenza

I-API isebenzisa i-generator eyodwa esebenza isikhathi eside futhi evikelwe yi-lock. Ngakho izicelo zicutshungulwa ngokulandelana: ama-stream amaningi afakwa kulayini esikhundleni sokudecodwa ngesikhathi esisodwa. Usayizi we-batch ongaphezu kokukodwa awukahlolwa, futhi izilinganiso ezishicilelwe zisebenzise i-slot eyodwa.

Umthamo olinganisiwe wama-live session uncike kakhulu kubude be-context:

• Amathokheni angu-500K nge-stream: i-live session engu-1.
• Amathokheni angu-262K nge-stream: ama-live session angu-2.
• Amathokheni angu-131K nge-stream: ama-live session angu-5.
• Amathokheni angu-32K nge-stream: ama-live session angu-11.
• Amathokheni angu-16K nge-stream: ama-live session angu-14.

I-live slot ngayinye isebenzisa cishe u-546 MiB wesimo se-Gated-DeltaNet ngaphambi kokugcinwa kwe-KV. Kuma-context amafushane, isimo esiphindaphindwayo siba yisithiyo esikhulu; ububanzi obusebenzayo obubhaliwe cishe bungama-live slot angu-8 kuya ku-14, nakuba izicelo zisacutshungulwa ngokulandelana yi-API lock yamanje.

Ekusebenzeni kwe-prefill, i-GCS ibe yisici esinamandla kakhulu esitholwe yiphrojekthi. Nokho, izilungiselelo ezinkulu noma ezishintshiwe ze-cache kufanele zihlolwe ngokumelene nokusetshenziswa kwe-VRAM ne-host-RAM kunokuba zikopishwe ngaphandle kokuhlolwa.

Gwema ukwakha i-Generator entsha kuzo zonke izicelo. Ithebula lamakhasi kanye ne-CPU page cache kwakhiwa ngaphakathi kwe-Generator.__init__; ukuyakha kabusha kungalahla buthule ukusetshenziswa kabusha kwe-prompt-cache futhi kuphoqe ama-prefill aphindaphindwayo. Iseva efakiwe isivele igcina i-instance esebenza isikhathi eside ngendlela efanele.

Ukuxazulula izinkinga

I-CLI ithi awekho ama-session asebenzayo

Irekhodi le-session lendawo linganyamalala lapho ithokheni ye-runtime proxy iphelelwa yisikhathi, nakuba i-VM isasebenza futhi ikhokhiseka. Sebenzisa inqubo yokubuyisela yekhosombe esikhundleni sokwakha enye i-VM ngesandla. I-scripts/restore.py ibuza ama-assignment asohlangothini lweseva bese ixhuma kabusha i-instance eyasala ingenamnikazi.

I-model ayingeni

Qinisekisa ukuthi i-runtime ine-VRAM ecishe ibe ngu-80 GB futhi iwuhlobo lwe-High-RAM. I-restore script yakhelwe ukwenqaba nokumisa i-assignment engu-40 GB ngaphambi kokulayishwa kwe-model.

I-API inempilo kodwa akuveli i-URL yomphakathi

up.sh ihlukanisa i-VM enempilo engenayo i-URL ye-tunnel eshicilelwe ekulungeleni okuphelele. Isimo sokuphuma 9 sikhombisa leli cala. Hlola ama-log esevisi nawe-tunnel esikhundleni sokucabanga ukuthi i-endpoint ifinyeleleka emphakathini.

Izicelo zibonakala zingazinaki izilawuli zesampling

Lokhu kulindelekile ku-temperature naku-top_p kuseva yamanje. Ziyahlukaniswa kodwa azidluliselwa ku-sampler.

Ukuhambisana kwe-stream kuyaphuka

Phinda ukhiqize inkinga usebenzisa scripts/dev_stub.py, bese usebenzisa scripts/check_surface.py. I-handler yangempela iyahlolwa, ngakho lokhu kubamba ukuhlehla kwephrothokholi ngaphandle kokulungiselela i-model.

Misa i-VM futhi ulawule izindleko

Uma usuqedile, misa i-runtime ekhokhiswayo ngokushesha:

bash scripts/down.sh

Lona umyalo obaluleke kakhulu wokulawula izindleko wale phrojekthi. Ukuvala isiphequluli, ukulahlekelwa irekhodi le-CLI lendawo, noma ukunqamuka ku-Colab akufakazeli ukuthi ukukhokhisa kumile.

I-control plane yangaphambili ongayikhetha ingeza ukuqinisekiswa ngaphambi kokulungiselela, i-CU ledger yendawo, ukumisa okuzenzakalelayo lapho ingasebenzi, ubude beseshini obukhulu, ukukhansela, nokuhlanza lapho ukulungiselela kwehluleka ngemva kwe-assignment. Ungazijwayeza leyo nqubo ngaphandle kwekhadi noma ama-compute unit:

python frontend/server.py --fake-provision
python frontend/server.py --mock
python scripts/dev_stub.py --port 8099 &
python frontend/server.py --mock --backend http://127.0.0.1:8099

Isiphetho

i-collabosm ihlanganisa izingxenye ezinzima zokusebenzisa i-Qwen3.8-Flash-Next ku-Colab: ukucela ukucushwa okulungile kwe-A100, ukubuyisa izabelo ezingenabanikazi, ukufaka i-runtime ehambisanayo ye-ExLlamaV3, ukulayisha ama-weight aqondiswe ezinguqulweni ezithile, ukugcina ama-cache, ukuveza i-tunnel evikelekile, nokuhlinzeka ngezindlela ezimbili ze-API ezihambisana ne-OpenAI.

Qala ngo-bash scripts/up.sh, uqinisekise ukucushwa okusebenzayo nge-/v1/status, xhuma nge-endpoint ye-/v1 ephrintiwe, futhi uhlale uqeda ngo-bash scripts/down.sh. Ukuze uzame ngendlela efana nokukhiqiza, naka kakhulu izicelo ezicutshungulwa ngokulandelana, izilawuli ze-sampling ezingakaqaliswa, izindleko zememori zama-cache, ukusetshenziswa kwe-VRAM okungokuzikhethela kombono, kanye nomehluko phakathi kokuqalisa kabusha i-cache engasebenzi nomthamo wangempela weseshini ebukhoma.