Qalisa i-GLM-5.3-Flash ku-RTX 3090 eyodwa engu-24 GB usebenzisa i-Host-RAM Offloading

Uhlolojikelele

glm53-flash-offload isebenzisa i-GLM-5.3-Flash engu-125.3 GB, eyakhiwe nge-EXL3 3.05-bpw, kwi-GPU eyodwa ye-NVIDIA engu-24 GB kanye nememori ye-DDR4 yohlelo. Iveza i-HTTP API ehambisana ne-OpenAI, ngesikhathi igcina izisindo ze-mixture-of-experts eqondiswe yimodeli ikakhulukazi ku-RAM yesistimu eboshiwe.

Le phrojekthi iyisethi encane yama-monkeypatches phezu kwe-exllamav3 1.5.1 ejwayelekile, ehambisana nama-kernel amancane ambalwa enziwe ngokwezifiso. Ayidingi ukuhlela ikhodi yomthombo ye-exllamav3. Ukumiswa okubhaliwe kwalinganiswa ku-RTX 3090 exhunywe nge-PCIe 4.0 x16 ohlelweni olune-AMD EPYC 7443P.

Umqondo oyinhloko uwukugcina ama-expert asetshenziswa njalo kunqolobane ye-GPU enwebekayo. Ama-expert angatholakali kunqolobane angafundwa ngokuqondile kwimemori yesingathi eboshiwe nge-PCIe. Kumodi esheshayo, ama-miss abandayo kakhulu ngesikhathi se-decode angacutshungulwa isisebenzi se-CPU se-AVX2.

Izici ezibalulekile

• Ukusetshenziswa kwe-GPU eyodwa: isebenzisa i-GLM-5.3-Flash nge-RTX 3090-class GPU eyodwa engu-24 GB.
• I-API ehambisana ne-OpenAI: isekela ukuqedelwa kwengxoxo, ukuqedelwa kombhalo, ukutholwa kwamamodeli, ukusakaza, okuqukethwe kokucabanga, kanye nezingcingo zamathuluzi zesitayela se-OpenAI.
• Isitoreji esiboshiwe esingenakho ukukopishwa: wonke ama-expert aqondisiwe agcinwa ku-RAM yesingathi eboshiwe futhi angafinyelelwa nge-CUDA unified virtual addressing.
• Inqolobane enwebekayo yama-expert ku-GPU: ama-expert asetshenziswa kakhulu athatha i-VRAM etholakalayo, kuyilapho indawo yenqolobane ingabuyiselwa okwesikhashana ukuze kusetshenziswe ama-activation e-prefill ende namabhafa esiteji.
• Ukuhlelwa kwesiteji se-prefill: ama-expert angekho kunqolobane akopishwa ungqimba olulodwa ngaphambili ukuze ukudluliswa kwe-PCIe kudlule ngesikhathi esifanayo nokubala.
• Isigaba se-CPU esingakhethwa: imodi esheshayo inikeza ama-miss abandayo akhethiwe ngesikhathi se-decode ku-kernel ye-CPU ye-AVX2.
• I-decode ehlanganisiwe: imifudlana efinyelela kwemine ingenza i-decode ndawonye ngokumiswa okubhaliwe kwe--ambs 4.
• Ukumiswa okuphindaphindekayo: inguqulo yemodeli, i-container digest, inguqulo ye-exllamav3, ukukhethwa kwe-Triton, izivivinyo zokusebenza, kanye nephaneli yekhwalithi eyisethenjwa kugcinwe kungaguquki.

Khetha phakathi kwemodi esheshayo nemodi enembile

Iseva inikeza izindlela ezimbili zokusebenza nge-environment variable ethi GLM53_MODE.

Imodi esheshayo

fast iyona emisiwe. Ihlanganisa inqolobane yama-expert ku-GPU, ama-miss enqolobane angenakho ukukopishwa, ukucutshungulwa kwe-CPU ye-AVX2 kwama-miss abandayo akhethiwe ngesikhathi se-decode, i-decode ehlanganisiwe, kanye nama-kernel e-decode ahlanganisiwe. Kusingathi okulinganisiwe, isebenzise cishe 218 GiB, noma 234 GB, ye-RAM yesingathi.

Umkhankaso oshicilelwe walinganisa cishe amathokheni angu-28.15 ngomzuzwana kumfudlana owodwa we-decode, amathokheni angu-31.57 ngomzuzwana esewonke emifudlaneni emibili, kanye namathokheni angu-33.98 ngomzuzwana esewonke emifudlaneni emine. Ukuhlolwa kwesithombe kwe-digest yamanje kwalinganisa amathokheni angu-28.88, 31.27, kanye no-35.00 ngomzuzwana esewonke ngokulandelana.

Isexwayiso sekhwalithi: i-prefill inembile, kodwa i-decode yemodi esheshayo ayilingani ncamashi ne-exllamav3 ejwayelekile. Ukuhlolwa okuqhathanisa ama-decode kubike i-KL emaphakathi engu-0.0053 kanye nokuvumelana kwe-top-1 okungu-0.977. Sebenzisa imodi enembile lapho kudingeka khona ukufana kokuphumayo.

Imodi enembile

exact ikhubaza isigaba se-CPU nama-kernel e-decode akhethekile emodini esheshayo. Igcina inqolobane yama-expert ku-GPU kanye nama-miss ememori yesingathi angenakho ukukopishwa. Iphaneli eyisethenjwa yalinganisa ukuvumelana kwe-top-1 okungu-1.0000 kanye ne-KL engu-0, kanti ukusetshenziswa kwememori yesingathi kwakucishe kube 111 GiB, noma 119 GB.

Imodi enembile ihamba kancane kodwa isebenzisa i-RAM encane kakhulu. Izinga le-decode elilinganisiwe lomfudlana owodwa lalingamathokheni angu-12.57 ngomzuzwana kumongo omfushane owahlolwa, uma liqhathaniswa cishe namathokheni angu-7.5 kuya ku-8.4 ngomzuzwana lapho i-exllamav3 ejwayelekile ibeka ama-expert angangeni kusisebenzi sayo se-CPU.

Izidingo zehadiwe nesofthiwe

I-GPU nomshayeli

• I-GPU eyodwa ye-NVIDIA ene-VRAM engu-24 GB.
• I-GPU ye-sm_86 efana ne-RTX 3090 noma i-A5000 iyona ehlosiwe eyalinganiswa.
• Amabhinari e-sm_86 afakiwe asebenza naku-sm_89, kuhlanganise ne-RTX 4090, kodwa lokho kumiswa akuzange kulinganiswe.
• Kudingeka umshayeli we-NVIDIA okwazi i-CUDA 13.0 ngoba isithombe sisebenzisa i-CUDA 13.0.3.
• Kuyanconywa i-PCIe 4.0 x16. Ama-miss enqolobane asebenzisa cishe u-25 GB/s phezu kwesixhumanisi esilinganisiwe.

Imemori yesingathi

• Imodi esheshayo: imemori ephelele engu-256 GiB iyona encane kakhulu uma kusebenza ezinye izinto ezimbalwa; kunconywa u-320 GB noma ngaphezulu. Uma kusetshenziswa imikhawulo yememori ye-Docker, vumela okungenani 230g.
• Imodi enembile: u-128 GiB uyisidingo esincane, kanti kunconywa u-160 GB noma ngaphezulu.

I-CPU nesitoreji

• Imodi esheshayo idinga i-CPU ye-x86-64 ene-AVX2, FMA, ne-F16C.
• Kunconywa cishe ama-core aphathekayo angu-24 kanye ne-DDR4 eneziteshi eziyisishiyagalombili noma engcono ukuze kuzuzwe ukusebenza kwemodi esheshayo okulinganisiwe.
• I-checkpoint ithatha cishe u-117 GiB, noma u-125.3 GB, kudiski.
• Kunconywa isitoreji se-NVMe ukuze kuncishiswe isikhathi sokulayisha imodeli.
• Kudingeka i-Docker, i-NVIDIA Container Toolkit, kanye ne-Hugging Face CLI ukuze kusethwe njengoba kubhaliwe.

Faka bese uqala iseva nge-Docker

Isinyathelo 1: Landa i-checkpoint

Landa i-checkpoint ye-3.05-bpw egcinwe inguqulo uyifake kufolda yendawo:

hf download turboderp/GLM-5.3-Flash-exl3 \
  --revision 3.05bpw \
  --local-dir /data/GLM-5.3-Flash-exl3-3.05bpw

Kunalokho ungafaka ifolda engenalutho ku-/models. Uma i-config.json ingekho, i-container ilanda i-checkpoint emisiwe iyifake kuleyo folda.

Isinyathelo 2: Qalisa imodi esheshayo

Umyalo olandelayo uqala iseva ku-GPU 0 futhi uyishicilele ku-port 30000. Usebenzisa i-image digest equkethe ukulungiswa kokulethwa kokusakaza.

docker run -d \
  --name glm53 \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d

Isilungiselelo --ulimit memlock=-1 siyanconywa ngoba iseva igcina inani elikhulu lememori ye-host liboshiwe. Isithombe esishicilelwe siphinde salayisha ngaphandle kwaso kumshayeli olinganisiwe, kodwa lokho akufanele kuthathwe njengokuqinisekile kuwo wonke ama-host.

Isinyathelo 3: Buka ukuqala

docker logs -f glm53

Linda umlayezo oqukethe loaded in ... s. Ukuqalisa kuthatha cishe imizuzu emibili kuya kwemithathu ohlelweni olubhaliwe, kuye nge-cache nezimo ze-host.

Isinyathelo 4: Qinisekisa impilo

curl -s http://localhost:30000/health

Qalisa imodi eqondile

Ukuze ubeke phambili ukufana komphumela nokunciphisa ukusetshenziswa kwememori ye-host, engeza -e GLM53_MODE=exact:

docker run -d \
  --name glm53-exact \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -e GLM53_MODE=exact \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d

Ungaqalisi izibonelo ezisheshayo neziqondile ngesikhathi esisodwa echwebeni elifanayo. Misa futhi ususe i-container yokuqala ngaphambi kokushintsha amamodi:

docker rm -f glm53

Ukusetshenziswa okuyisisekelo kwe-API

Thumela ukuqedelwa kwengxoxo

curl -s http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Explain zero-copy model offloading."}],"max_tokens":512}'

Igama lemodeli elihlinzekwa ngokuzenzakalelayo ngu-glm-5.3-flash. Ngaphandle kokuthi ilungiselelwe ngokusobala, max_tokens ngokuzenzakalelayo isebenzisa ingxenye esele ye-context esikhundleni sokubekelwa inani elincane elingaguquki.

Vala ukucabanga

Ukucabanga kwe-GLM kubuyiselwa ku-reasoning_content. Ukuvala ukucabanga esicelweni, dlulisa inketho ye-chat-template:

curl -s http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Reply with one sentence."}],"chat_template_kwargs":{"enable_thinking":false},"max_tokens":128}'

Sakaza impendulo

curl -N http://localhost:30000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"glm-5.3-flash","messages":[{"role":"user","content":"Write a short explanation of mixture-of-experts models."}],"stream":true,"max_tokens":512}'

Sebenzisa i-digest ephiniwe bb633b0b... eboniswe ngenhla. Isithombe esidala 1159044a... sakhiqiza ngesivinini esigcwele kodwa siletha imicimbi yokusakaza cishe ngesigamu sejubane ngaphambi kokuthulula i-backlog ekugcineni.

Bhala amamodeli atholakalayo

curl -s http://localhost:30000/v1/models

Hlola izibalo zesikhathi sokusebenza

curl -s http://localhost:30000/stats

I-endpoint yezibalo ihlanganisa ulwazi lokushaywa kwe-expert-cache nezibali ze-CPU-tier. Amanye ama-endpoint afaka /v1/completions, /server_info, kanye nama-route anesimo se-SGLang athi /generate no-/tokenize.

Ukusetshenziswa kusekelwe embhalweni kuphela. I-vision tower ye-checkpoint ayilayishwa.

Sebenzisa amathuluzi esitayela se-OpenAI

I-chat endpoint yamukela inkambu ethi tools. Izinkulumo zamathuluzi ezikhiqizwa ngefomethi yangaphakathi ye-GLM ethi <tool_call> ziguqulwa zibe tool_calls ezihambisana ne-OpenAI empendulweni. Ngakho-ke izinhlelo zamakhasimende zingasebenzisa umjikelezo ojwayelekile wokubiza amathuluzi: thumela izincazelo zamathuluzi, hlola ukubizwa kwethuluzi komsizi, sebenzisa umsebenzi, bese uthumela umphumela wethuluzi emuva njengomunye umlayezo.

Lungisa i-context nezimpikiswano zeseva

Izimpikiswano ezibekwe ngemva kwegama lesithombe zengezwa emyalweni weseva, futhi inani lokugcina yilona elisetshenziswayo. Ngokwesibonelo, sebenzisa i-KV cache enamathokheni angu-65,536:

docker run -d \
  --name glm53-64k \
  --gpus '"device=0"' \
  --ulimit memlock=-1 \
  --shm-size 16g \
  -p 30000:30000 \
  -v /data/GLM-5.3-Flash-exl3-3.05bpw:/models \
  ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d \
  -cs 65536

I-KV cache encane ingashiya isikhala sezikhala eziningi ze-expert-cache. Ukucushwa kwebhentshimakhi okushicilelwe kusebenzise i-KV cache yamathokheni angu-131,072, usayizi omkhulu we-batch oyisishiyagalombili, nosayizi we-chunk ongu-8,192 wamathokheni.

Faka ngaphandle kwe-Docker

Ukufakwa kwendabuko kudinga i-exllamav3 1.5.1 eyakhelwe i-GPU eqondiwe. Ku-clone yephrojekthi, sebenzisa isifaki se-extension bese uqala i-entrypoint enikeziwe, izindlela ze-repository nemodeli zisethiwe:

scripts/install.sh
GLM53_ROOT=$PWD \
GLM53_MODEL_DIR=/path/to/model \
docker/entrypoint.sh

Isifaki sakha ama-extension angokwezifiso ephrojekthi. I-Docker iseyindlela ekhiqiza imiphumela ephindaphindekayo kakhulu ngoba iphina i-CUDA, i-PyTorch, i-Triton, i-SGLang, i-exllamav3, i-commit yephrojekthi, kanye nokucushwa kweseva okuhloliwe.

Qinisekisa ikhwalithi nokusebenza

Thola amaphuzu ephaneli yereferensi

Kwi-clone ye-repository, sebenzisa iklayenti lebhentshimakhi elifakiwe elisebenzisa i-standard library kuphela:

python3 bench/score_ref_panel.py \
  --url http://127.0.0.1:30000 \
  --panel reference/glm-5.3-flash-exl3-ref-panel.json \
  --out score.json

Iphaneli elisebenzisa i-teacher-forcing kufanele libike ukuvumelana kwe-top-1 okungu-1.0000 ne-KL engu-0 kuwo womabili amamodi asheshayo naqondile. Leli phaneli lihlola indlela ye-prefill noma ye-teacher-forced futhi alibonisi ukuthi i-autoregressive decode yemodi esheshayo inembile.

Qalisa ukuhlolwa kwejubane

python3 bench/sweep.py \
  --url http://127.0.0.1:30000 \
  --card rtx-3090-24gb/glm-5.3-flash-offload \
  --template glm \
  --config "bb633b0b fast" \
  --prefill 8192 32768 \
  --conc 1 2 4 \
  --reps 3 \
  --dec-reps 2 \
  --no-early-exit \
  --out sweep.json

Hlola ukulethwa kokusakaza

python3 bench/stream_rate.py --url http://127.0.0.1:30000

Umphumela ophumelele kufanele ubonise umbhalo ufika ngokulinganayo, kunokuba cishe ingxenye yempendulo ivele ngesikhathi esisodwa ekugcineni.

Ukulungisa okuthuthukile

Lungisa usayizi wesigaba se-CPU

Ngokuzenzakalelayo, GLM53_CT_CPUS=auto ikhetha i-CPU eyodwa enengqondo kukhori ngayinye ebonakalayo, bese ishiya amakhori amabili okuqala etholakala ohlelweni lokusebenza nentambo ye-Python eshayela i-GPU. Kusingathi esilinganisiwe esinamakhori angu-24, lokhu kwaholela kuma-CPU 2 kuya ku-23 nemicu yokusebenza engu-22.

Ukuze unikeze ukucaciswa okuqondile, sebenzisa amanani afana nalawa:

-e GLM53_CT_CPUS=2-23 \
-e GLM53_CT_THREADS=22

Imodeli yezindleko ze-CPU isebenzisa GLM53_CT_B, elinganiswa ngamamilisekhondi angu-0.104 ngochwepheshe ngamunye ngemicu engu-22. Emicu yokusebenza engu-N, i-README incoma ukuyilinganisela njenge-0.104 x 22 / N. Isibonelo, usingathi onamakhori angu-16 oshiya amakhori amabili ekhululekile uzosebenzisa imicu yokusebenza engu-14 nenani eliseduze no-0.16.

Ngamakhori aphathekayo angaphansi kwangaba ngu-12, kulindeleke ukuthi iningi lokuphuthelwa libuyele endleleni ye-PCIe, okwenza ukusebenza kusondele kumodi eqondile. Kuleso simo, imodi eqondile ingaba ukukhetha okungcono.

Lawula ukwakheka kwememori yesigaba se-CPU

GLM53_CT_SWZ=1 yenza isigaba se-CPU sisebenzise ikhophi ehlukile, ehleleke ngamabhulokhi, yochwepheshe. Lokhu ukucushwa okusheshayo okulinganisiwe futhi kwengeza cishe u-114.2 GB wokusetshenziswa kwe-RAM. Ukuyibeka ku-0 kwenza isigaba se-CPU sifunde ngokuqondile ikhophi yasekhaya ephiniwe, kunciphise kakhulu ukusetshenziswa kwememori, kodwa iphrojekthi ibika ukuthi ibhentshimakhi yekheneli yochwepheshe ye-CPU ihamba kancane cishe ngo-40%. Ukusebenza kokuphela kuya kokuphela kwalesi silungiselelo akuzange kulinganiswe.

Lungisa izinqolobane zememori

GLM53_EC_RESERVE_GB ishiya i-VRAM ikhululekile ngemva kokwaba inqolobane yochwepheshe. Okuzenzakalelayo ngu-1.5 GB kwimodi esheshayo no-1.0 GB kwimodi eqondile. Gcina indawo egodliwe yemodi esheshayo iku-1.5 GB ngaphandle kwalapho ama-prefill amade ephelelwa imemori.

GLM53_EC_ELASTIC_GB okuzenzakalelayo ngu-10 GB. Leli nani lememori yenqolobane lingakhululwa ukuze kusetshenziswe ama-activation e-prefill kanye nokubeka idatha, bese liphinde litholakale ngesikhathi se-decode. GLM53_EC_STAGE_GB okuzenzakalelayo ngu-2.6 GB ngebhafa yokubeka idatha, futhi kusetshenziswa amabhafa amabili ukuze umsebenzi weqe ngesikhathi esifanayo.

Sebenzisa ama-GPU amakhulu

Inqolobane yochwepheshe izilinganisela ngokusekelwe ku-VRAM etholakalayo ngemva kokulayishwa kwemodeli kanye nendawo egodliwe emisiwe. Ngakho ikhadi elingu-32 GB noma elingu-48 GB lithola izikhala eziningi zochwepheshe, okufanele kwenyuse izinga lokushaya kwenqolobane futhi kuthuthukise ukusebenza kwe-decode. Le miphumela yamakhadi amakhulu ayinikezwanga ku-README.

I-RTX 4090 ingasebenzisa i-sm_86 build efakiwe, nakuba ingazange ilinganiswe. I-Blackwell idinga ukwakhiwa kabusha nge:

TORCH_CUDA_ARCH_LIST="8.6;12.0"

Kungase kudingeke futhi i-Triton pin entsha kwesinye isakhiwo. Uma udala eyodwa, qinisekisa ukuthi iphaneli yereferensi isalokhu iku-top-1 1.0000 ne-KL 0.

Vikela ukukhethwa okuqondile kwe-Triton

Isithombe sibopha ukukhethwa kwamakheneli e-KDA linear-attention prefill nge-data/triton_pin_exact.json. Ukuvumela i-Triton ikhethe ukucushwa okuhlukile kungashintsha ukuhlanganiswa kwezinombolo ze-floating-point. Gcina GLM53_TRITON_PIN kunani layo elizenzakalelayo lapho ukuziphatha kwe-prefill okulingana nereferensi kubalulekile.

Indlela ipayiphu yokudlulisela ngaphandle esebenza ngayo

1. Layisha ochwepheshe abaqondiswe ememori yesingathi ephiniwe. Isendlalelo ngasinye sithola indawo ebekelwe yona ephiniwe enosayizi oqondile, futhi izikhombisi zochwepheshe ze-exllamav3 zikhomba kuma-alias e-CUDA UVA.
2. Gcwalisa inqolobane ye-CLOCK enwebekayo ku-VRAM. Ukukhethwa komzila kuyasuswa okuphindiwe, okutholakala enqolobane kuyamakwa, bese kukhethwa izisulu zokuphuthelwa okwamukelwe.
3. Funda ochwepheshe abangekho enqolobaneni nge-PCIe. Amakheneli e-exllamav3 angafinyelela ukumelwa kwesingathi okuphiniwe ngaphandle kwekhophi evumelanayo eseceleni kwesingathi.
4. Beka ama-prefill amade esigabeni sokudlulisa. Ekubalweni okukhulu ngokwanele kokukhetha kwesendlalelo ngasinye, ochwepheshe abangekho bayakopishwa ebhafeni lokudlulisa kusendlalelo esisodwa ngaphambili ngenkathi isendlalelo esandulele sibala.
5. Hlukanisa ukuphuthelwa okukhethiwe kwe-decode kwimodi esheshayo. Imodeli yezindleko ithumela ukuphuthelwa okufudumele ekusebenzeni kwe-GPU kwe-zero-copy, bese ukuphuthelwa okubandayo okukhethiwe esigabeni se-CPU.
6. Hlanganisa imiphumela ye-CPU ne-GPU. I-CPU ishicilela umsebenzi wayo ngememori ephiniwe nemapped, bese i-GPU ingeza ingxenye ye-FP32 etholakele ngemva kokulinda ifulegi eliseceleni kwedivayisi, kunokuvumelanisa nesingathi.

Izixwayiso zokusebenza

• Sebenzisa GLM53_MODE=exact lapho okukhiphayo kwe-decode kufanele kufane ne-stock exllamav3.
• Gwema isithombe esidala esithi sha256:1159044a... uma ukubambezeleka kokusakaza kubalulekile. Sebenzisa i-digest ebhaliwe ethi sha256:bb633b0b....
• Imodi esheshayo okwamanje igcina amakhophi amabili ochwepheshe ku-RAM yesingathi. Qinisekisa ukuthi kunememori eyanele ngaphambi kokuqala i-container.
• Ungacabangi ukuthi ukuphuma okulinganisiwe kwe-RTX 3090 kuzofana ezixhumanisini ze-PCIe ezihamba kancane. Isivinini se-decode sizwela kuthrafikhi yokuphuthelwa kwenqolobane.
• Qapha /stats, imemori yesistimu, nememori ye-GPU lapho ushintsha usayizi womongo, indawo egodliwe yenqolobane, concurrency, noma izilungiselelo zokudlulisa.
• Isiphakeli siphatha umbhalo kuphela; ukufakwa kwezithombe akusekelwa yilesi sakhiwo sokusetshenziswa.

Isiphetho

i-glm53-flash-offload yenza i-checkpoint enkulu ngokungajwayelekile ye-mixture-of-experts isebenze ku-GPU eyodwa engu-24 GB ngokuhlanganisa isitoreji sesingathi esiphiniwe, ukufundwa kwe-PCIe kwe-zero-copy, inqolobane ye-VRAM enwebekayo, ukudluliswa kwe-prefill okuhleliwe, kanye nesigaba se-CPU esisebenzisa i-AVX2 ngokuzikhethela. Qala ngesithombe se-Docker esiphiniwe, khetha imodi eqondile ukuze ugcine ukunemba kokukhiphayo noma imodi esheshayo ukuze uthole throughput, bese uqinisekisa ukufakwa ngokuhlola ikhwalithi, isivinini, nokusakaza okufakiwe ngaphambi kokulungisa izilungiselelo zenqolobane noma ze-CPU.