Qalisa i-DeepSeek V4.1 Flash Endaweni Kuma-DGX Spark Amabili nge-EXL3 ne-vLLM

Uhlolojikelele

Le phrojekthi isebenzisa i-DeepSeek V4.1 Flash endaweni kumasistimu amabili e-NVIDIA GB10 DGX Spark. Isebenzisa i-checkpoint ye-EXL3 engu-2.9 bits-per-weight, i-tensor parallelism kuwo womabili ama-node, ama-kernel omdabu e-sm_121a, kanye ne-vLLM API ehambisana ne-OpenAI ethekwini elithi 8888.

Isihlonzi semodeli esinikezwayo sithi DeepSeek-v4.1-Flash-EXL3. I-deployment ingeyombhalo kuphela futhi yakhelwe ngokukhethekile ikhithi enama-node amabili e-GB10 exhunywe nge-CX7.

I-container yengeza ukwesekwa kwe-EXL3 phezu kwe-vllm/vllm-openai:deepseekv41-flash-0909, okuyi-ARM64 vLLM image equkethe i-architecture edingekayo ye-DeepseekV41.

Okunikezwa Iphrojekthi

• Ukunikezwa kwasendaweni okuhambisana ne-OpenAI: Izinhlelo zokusebenza zingabiza i-endpoint ejwayelekile ethi /v1/chat/completions.
• I-tensor parallelism yama-node amabili: Imodeli isebenza nge-tensor-parallel size engu-2 phezu koxhumano lwe-CX7.
• I-EXL3 quantization: I-checkpoint inesilinganiso esingu-2.9 bpw futhi isebenzisa i-codebook ye-mul1 nezilungiselelo ze-per-tensor quantization.
• I-DSpark speculation yomdabu: Ama-draft expert asevele agcinwe ngaphansi kwe-mtp.* ku-checkpoint, ngakho akudingeki imodeli ye-draft ehlukile.
• Ukwesekwa kwe-long-context: Ukucushwa okuthunyelwe kusetha MAX_MODEL_LEN=600000 futhi kusebenzisa i-KV cache pool engu-2.5 GiB.
• Amathebula e-Engram asekelwe kumafayela: Idatha ye-Engram enganquantized ifundwa kuma-shard emodeli yoqobo 47 no-48 kunokuba igcinwe ngokuqondile kwimemori ye-host.
• Ukuqalisa okuzenzakalelayo: I-start.sh ilanda ama-weight, ithola noma yakhe i-image, yabelana ngamafayela ne-worker, yenza ukuhlolwa kwe-GPU, bese iqala i-API.
• Ukuvikelwa ngesikhathi sokusebenza: I-launcher yenza ukuhlolwa kwe-boot margin, ikhulula ama-allocator phakathi kwezigaba ze-prefill, isusa i-page cache ngemva kokulayisha, futhi ilungisa ama-OOM score ama-container.

Izidingo Ze-hardware Ne-storage

Le repository ihloselwe ama-node amabili e-NVIDIA GB10 DGX Spark kuphela. Ikheli elizenzakalelayo le-head node ngu-10.0.0.1, futhi ama-node axhumana nge-CX7. I-README ikhomba ama-interface ahlolwe njengokuthi enp1s0f1np1/rocep1s0f1 ku-node yokuqala kanye ne-enp1s0f0np0/rocep1s0f0 kweyesibili.

Hlela cishe u-387 GiB wedatha yemodeli:

• Cishe u-197 GiB wama-shard angu-39 e-EXL3 avela ku-Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-2.9bpw.
• Cishe u-190 GiB wama-shard oqobo e-DeepSeek 47 no-48 kanye ne-index. Lokhu kuqukethe amathebula e-Engram omdabu.
• Cishe u-9 GiB we-container image eshicilelwe lapho kungadonswa khona i-image ehambisanayo esikhundleni sokuyakha.

Uma usuvele unohla lwemibhalo oluphelele lwe-DeepSeek-V4.1-Flash, khomba ENGRAM_DIR kulo ukuze ugweme ukuphinda ulande umthombo we-Engram. Kule njongo kufundwa kuphela ama-shard 47 no-48 kanye ne-index.

Ku-DGX Spark, i-RAM ye-host nayo iyimemori ye-GPU. Yonke i-CUDA allocation idla ngokushesha imemori eyabiwe, ngakho izilungiselelo ze-storage ne-memory kufanele zishintshwe ngokuzimela futhi zihlolwe nge-prompt ende.

Lungisa I-head Node

1. Clone i-repository bese udala ifayela le-environment

Qalisa i-deployment usuka kufolda ye-repository ku-head Spark:

cp .env.example .env

I-launcher iphinde idale i-.env ekuqaleni kokuqala uma ingekho. Buyekeza ifayela bese ulungisa amanani afana ne-WORKER_USER ne-GID lapho kudingeka khona ezinhlelweni zakho.

Amanani aqanjwe ngokuqondile ku-.env anokubaluleka ngaphezu kwama-command-line environment prefix ngoba i-start.sh ilayisha ifayela phezu kwe-environment eyizuzile. Isibonelo, shintsha i-WEIGHT_SYNC ngaphakathi kwe-.env esikhundleni sokusebenzisa WEIGHT_SYNC=zfs ./start.sh.

2. Faka amathuluzi okulanda e-Hugging Face

Ukulanda imodeli ngokuzenzakalelayo kudinga i-Hugging Face CLI enokwesekwa kwe-transfer:

pip install -U 'huggingface_hub[hf_transfer]'

Ukulanda kungaqhubeka kusukela lapho kuphazamiseke khona. Uma ukuqalisa okuphazamisekile kushiya imodeli ingaphelele, sebenzisa i-launcher noma i-downloader futhi.

3. Ngokuzithandela landa ngaphandle kokuqala iseva

Ukuze ulungise amafayela e-EXL3 ne-Engram ngaphambi kokuqalisa i-vLLM, sebenzisa:

./download.sh

Ukulanda okuzenzakalelayo kuvuliwe ngokuzenzakalelayo. Setha AUTO_DOWNLOAD=0 ekucushweni uma ufuna ukuphatha amafayela ngokwakho.

Qalisa Iseva

Ku-repository ekwi-head node, sebenzisa:

./start.sh

Ekuqaleni kokuqala, i-launcher yenza le misebenzi elandelayo esezingeni eliphezulu:

1. Ihlola i-memory margin etholakalayo kanye nokucushwa kwama-node.
2. Ilanda noma yimaphi amafayela e-EXL3 ne-Engram ashodayo.
3. Idonsa i-image yomphakathi ethi ghcr.io/miaai-lab/deepseek-v4.1-flash-exl3-2x-dgx-sparks:2.9bpw.
4. Yakha endaweni kuphela uma i-recipe stamp ishintshile noma i-registry ingatholakali.
5. Yenza ama-weight atholakale ku-worker.
6. Iqhuba ukuhlolwa kwe-EXL3 GEMM kanye ne-Engram dequantization GPU.
7. Yethula ama-rank amabili e-tensor-parallel futhi iveze i-API ethekwini elithi 8888.

I-checkout ehlanzekile kufanele ihambisane ne-recipe stamp eku-image eshicilelwe, ngakho ngokuvamile akudingeki ukuhlanganisa.

Lawula ukwakhiwa kwe-image nokudluliswa

• BUILD=1 ./start.sh ihlala yakha isithombe endaweni.
• SKIP_BUILD=1 ./start.sh igcina isithombe esilandiwe ngisho noma uphawu lweresiphi luhlukile.
• SKIP_PULL=1 ./start.sh igwema ukuxhumana nerejista yeziqukathi.
• PULL=1 ./start.sh iphinda ilande isithombe ngisho noma uphawu lwendawo lufana.
• SKIP_SHIP=1 ./start.sh ivimbela i-head ekukopisheni isithombe iye ku-worker.

Ngokuzenzakalelayo, ukuthumela isithombe kusebenzisa ifayela le-tar elilungiselelwe, elidluliswa nge-rsync engaqhubeka lapho kugcine khona. Setha IMAGE_SHIP=stream ukuze uxhume okukhipha kwe-docker save ngokuqondile kokuthi docker load ku-worker. I-streaming ayidingi isikhala sesikhashana, kodwa ukudlulisa okuphazamisekile kuqala phansi. Imodi ye-auto ikhetha ukudlulisa okuqhubekayo futhi ishintshele ku-streaming lapho isikhala sediski ku-worker silinganiselwe.

Hlola futhi Uphathe Ukusatshalaliswa

Sebenzisa imiyalo yokuphatha yesiqalisi esikhundleni sokusebenzisa lezi ziqukathi ezimbili ngokwehlukana:

./start.sh status
./start.sh logs
./start.sh logs worker
./start.sh stop
./start.sh restart

Uma i-worker isivele inezisindo ezivumelanisiwe, qala kabusha ngaphandle kokuphinda isinyathelo sokuvumelanisa:

SKIP_SYNC=1 ./start.sh restart

Ngesikhathi sokuqalisa, umtshina wokuthula kwamalogi wemizuzwana engu-420 uqoqa ama-stack trace e-Python kuwo womabili ama-rank. Amafayela okuxilonga abhalwa ngaphansi kwe-logs/.

Thumela Isicelo Sakho Sokuqala Se-API

Uma iseva isesimweni esihle, thumela isicelo sokuqedela ingxoxo ku-head node. Lesi sicelo sokuhlola sivala ukucabanga ukuze impendulo eqondile kube lula ukuyihlola:

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-v4.1-Flash-EXL3","messages":[{"role":"user","content":"What is 17*19? Reply with the integer only."}],"max_tokens":32,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'

Impendulo elindelekile ngu-323. Noma yiliphi iklayenti elihambisana ne-OpenAI lingasebenzisa i-base URL kanye nesikhombi semodeli okufanayo.

Sebenzisa izilungiselelo zokuthatha amasampula ezinconyiwe

Ekusetshenzisweni okuvamile kwemodeli, izilungiselelo ezisemthethweni zokuthatha amasampula yilezi:

• temperature=1.0
• top_p=0.95
• Ukucabanga kunikwe amandla

I-chat template ngokuzenzakalelayo isebenzisa umzamo wokucabanga ophezulu, omelelwa ngu-75. Iphinde yamukele low ku-50, max ku-100, noma inombolo ephelele esukela ku-1 kuye ku-100.

Isicelo esisebenzisa ukucushwa kokuthatha amasampula okunconyiwe singabukeka kanje:

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-v4.1-Flash-EXL3","messages":[{"role":"user","content":"Explain how tensor parallel inference works."}],"max_tokens":512,"temperature":1.0,"top_p":0.95,"reasoning_effort":"high"}'

Qonda i-DSpark Speculative Decoding

Lokhu kusatshalaliswa kusebenzisa i-DSpark speculative decoding enochwepheshe abasalungiselelwe abashumekwe ku-checkpoint. Akukho drafter ehlukile okumele ilandwe, ilayishwe, noma inikezwe.

Ukucushwa okuyisisekelo kwe-vLLM kulingana nalokhu:

--speculative-config '{"method":"dspark","num_speculative_tokens":3}'

I-checkpoint inomkhawulo we-draft block-size ongu-5, kodwa isilungiselelo esithunyelwe sisebenzisa ama-speculative token angu-3 ngoba silinganise ngokushesha embhalweni ovamile. Osayizi bokuthwebula bahlanganisa 1 2 3 4 6 8 12 18 24, kuhlanganise nosayizi 6 wokulandelana okubili okukhiqiza ama-speculative token angu-3 ngakunye.

I-DSpark ithuthukisa i-single-stream decoding ezilinganisweni ezibikiwe. Kuma-batch abanzi, yivale ngo-SPEC_METHOD=none. Lokhu futhi kukhulula cishe u-3.5 GiB. I-README ibika cishe ama-token angu-31.6 ngomzuzwana ku-speculative stream eyodwa, kuyilapho ukunikezwa okungenayo i-speculation kufinyelele ku-throughput ehlanganisiwe ephezulu kuma-stream angu-4.

Phatha Imemori Ngokuphepha

Ukwabiwa okulinganiselwe kwe-node ngayinye kuhlanganisa izisindo ze-EXL3 ezingu-99.5 GiB ku-rank ngayinye, i-KV pool engu-2.5 GiB, cishe u-5 kuya ku-7 GiB wama-context, i-NCCL, ama-workspace, nama-CUDA graph, kanye cishe no-9 GiB wezinqubo ze-vLLM, i-Docker, uhlelo lokusebenza, nedeskithophu.

Imikhawulo ethunyelwe yile:

• MAX_MODEL_LEN=600000
• MAX_NUM_SEQS=2
• MAX_NUM_BATCHED_TOKENS=1024
• I-KV cache pool engu-2.5 GiB

Isakhiwo se-KV sikhethwa ngokuzenzakalelayo njenge-fp8_ds_mla ye-DeepSeek. Ungadlulisi i---kv-cache-dtype. I-pool elinganisiwe ibiphethe ama-token angu-774,400 emkhawulweni we-context ongu-614,400, ngezindleko ezicishe zibe ngu-3.4 KiB nge-token ngayinye.

Ungalinganisi i-quantization yale checkpoint ngokuguqula u-2.9 ube inombolo ephelele engu-2. Ipharamitha yayo ye-quantization ikhethwa nge-tensor ngayinye, futhi ochwepheshe abahlukahlukene abaqondiswayo, ochwepheshe ababelwana ngabo, ama-matrix okunaka, ama-tensor e-Engram, ama-tensor e-MTP, kanye nekhanda le-language model basebenzisa amanani ahlukene.

Ngemva kokufudumala, i-head igcine cishe u-4.07 kuya ku-4.21 GiB wememori etholakalayo ezivivinyweni ezibikiwe. I-prefill yama-token angu-601,000 yehlise leso sikhala saba cishe u-2.1 GiB. Ngakho-ke, qinisekisa izinguquko ngemva kwe-prefill ende kunokuba wenze lokho ngokushesha ngemva kokuqalisa.

Khulisa isilungiselelo esisodwa kuphela esihlobene nememori ngesikhathi. I-README ixwayisa ngokuthi ukwengeza u-0.5 GiB ku-KV pool kungadla cishe u-1 GiB we-prefill margin. I-pool engu-3 GiB yaphumelela ukuhlolwa okufushane, kodwa yehluleka cishe kuma-token angu-470,000 ku-prompt yama-token angu-600,000.

Sebenzisa i-memory guard ozikhethela yona ngokucophelela

scripts/memguard.sh ihlola /proc/meminfo kanye ngomzuzwana futhi ibulale isiqukathi sokunikeza esisendaweni ngemva kwamasampula amabili alandelanayo angaphansi kwe-DSV41_MEM_GUARD_GIB, umkhawulo waso wokuzenzakalelayo kungu-1.5 GiB.

Isivikelo sivaliwe ngokuzenzakalela ngoba asikwazi ukunquma ukuthi iyiphi inqubo edle inkumbulo. Sihlala sibulala i-container yayo ye-vLLM, ngisho noma ingcindezi idalwe umsebenzi ongahlobene osebenza ku-host. Ukuze usivule ngokusobala, setha:

DSV41_MEM_GUARD=1

Emisebenzini esindayo ese-host, i-README incoma ukukhawulela umsebenzi odala inkinga esikhundleni salokho:

systemd-run --scope -p MemoryMax=2G <job>

Ukusatshalaliswa kusaqhubeka nokusebenzisa --oom-score-adj 1000 kuwo womabili ama-container, kanye ne-boot-margin preflight, ukukhululwa kwe-allocator ngemva kwe-prefill ngayinye, nokwehliswa kwe-page cache ngemva kokulayisha.

Khetha Isu Lokuvumelanisa Ama-Weight

NFS: okuzenzakalelayo

I-NFS iyilungiselelo elihambisana ne-kit ehloliwe. Iphinda isebenzise i-exporter ethi vllm-fn-nfs, iveze ama-directory e-EXL3 ne-slim Engram, bese iwafaka kuma-container abasebenzi njenge-/model kanye ne-/engram-src.

./start.sh share
./start.sh

start.sh yenza isinyathelo sokwabelana ngokuzenzakalela, ngakho umyalo ocacile we-share uwusizo kakhulu lapho kulungiselelwa noma kuxazululwa inkinga ye-export.

ZFS: ama-replica asendaweni ye-node

Setha WEIGHT_SYNC=zfs ku-.env ukuze kugcinwe i-dataset yasendaweni ku-node ngayinye. I-ZFS ingathuthukisa ukuqina futhi ithumele kuphela ama-block ashintshiwe ngemva kwe-replica yokuqala. Iphinde ivumele ukubuyela emuva kusetshenziswa ama-snapshot.

Izindleko zikhulu: i-node ngayinye idinga cishe u-385 GiB wesikhala samahhala ku-pool. I-README iphawula ukuthi i-Spark yesibili ehloliwe yayinesikhala samahhala esingaba u-41 GiB kuphela, ngakho i-ZFS ayikwazanga ukusetshenziswa lapho.

Ukusethwa okujwayelekile ku-node ngayinye kungaba:

sudo apt-get install -y zfsutils-linux
sudo zpool create -f models <vdev>
sudo zfs create -o recordsize=1M -o atime=off -o compression=lz4 \
  -o mountpoint=/path/to/model models/dsv41-exl3
sudo zfs create -o recordsize=1M -o atime=off -o compression=lz4 \
  -o mountpoint=/path/to/engram models/dsv41-engram
sudo zfs allow -u "$WORKER_USER" create,destroy,mount,receive,snapshot,rollback models

Gcwalisa i-Engram dataset ngama-shard 47 no-48 kanye ne-index kuphela. Ungakopishi wonke umuthi wemodeli ye-native ongu-476 GiB. Uma i-pool noma i-zfs recv yomsebenzi ingatholakali, i-launcher ibuyela ekusakazeni.

Rsync: ikhophi yasendaweni elula

WEIGHT_SYNC=rsync idala ikhophi evamile yasendaweni ku-node ngaphandle kokudinga i-ZFS pool. Ayinikezi izibuyekezo ezikhulayo ezisezingeni lama-block ezihlinzekwa yi-ZFS.

Paka Idatha ye-Engram Ukuze I-Prefill Isheshe

Amatafula e-Engram ahlala esefomethini ye-native FP8 futhi awayona ingxenye yomuthi we-EXL3 owenziwe quantize. Ngokuzenzakalela, uhlelo lwe-embedding olusekelwe kumafayela luthola ama-row ngaphandle kokufaka cishe u-47 GiB ngesendlalelo ngasinye enkumbulweni.

Ukuhambisa amafayela e-Engram row aqondene ne-rank ku-NVMe yasendaweni ye-node ngayinye, sebenzisa:

./start.sh pack

Ukupaka kubhala cishe u-94 GiB ku-node ngayinye lapho usayizi we-tensor parallel uwubili. Kuyazikhethela futhi akudingeki ukuze kuqalwe. Ezivivinyweni ezibikiwe, amafayela e-Engram apakiwe endaweni athuthukisa ukusebenza kwe-prefill cishe ngamaphesenti angu-25 kuya kwangu-50 ngoba ama-row miss asebenzisa i-direct I/O yasendaweni esikhundleni se-NFS.

Ama-row e-Engram kufanele aqhubeke ethayiphiwe njenge-fp8_e4m3fn lapho enikezwa i-dequantization kernel. Ukuphathwa kwawo njenge-uint8 kungakhiqiza okukhiphayo okubukeka kunembile kodwa okungalungile.

Qalisa Ukuhlolwa Kokuqinisekisa

Indawo yokugcina ikhodi ihlanganisa ukuhlolwa kwezinga lomthombo okungadingi i-PyTorch noma i-vLLM:

python3 tests/test_numeric_config.py
python3 tests/test_engram_src.py
python3 tests/test_engram_secondary.py
python3 tests/test_k_map.py
python3 tests/test_memory_log.py
python3 tests/test_exl3_lm_head.py
python3 tests/test_sm120_block64.py
python3 tests/test_h2d_stage.py
python3 scripts/weight_budget.py --tp 2

Ukuhlolwa kokufana kwe-chat-template kudinga transformers kanye ne-encoding/encoding.py yemodeli yokuqala:

python3 tests/test_chat_template.py --src /path/to/DeepSeek-V4.1-Flash

Ngemva kokuba i-API isebenza, qalisa ukuhlolwa kwe-smoke kusukela ekuqaleni kuze kube sekugcineni:

bash tests/test_smoke.sh

Ukuhlolwa kwe-smoke kucela imodeli ukuthi ibale 17*19 futhi kulindele 323. Ngaphambi kokuqalisa isevisi, start.sh iphinde iqhube ukuhlolwa kokufana kwe-GPU kanye nokuhlolwa kwe-Engram dequantization ngaphakathi kwe-container yesikhashana. Imiphumela ibhalwa ku-logs/overlay-verify.log.

Amasu Okusebenza Athuthukile

• Khetha i-DSpark emifudlaneni eyodwa noma emibili: Izinga elibikiwe lokuqopha kabusha kumfudlana owodwa lalingamathokheni angu-31.6 ngomzuzwana, kanti inani elihlanganisiwe lalingamathokheni angu-42.5 ngomzuzwana emifudlaneni emibili.
• Vala ukuqagela kuma-batch abanzi: Nge-SPEC_METHOD=none, ukusebenza komfudlana owodwa kwehla, kodwa ukusebenza okuhlanganisiwe kwemifudlana emine kwafinyelela kumathokheni angu-53.7 ngomzuzwana esivivinyweni esibikiwe.
• Paka i-Engram kumaprompt amade aphindaphindwayo: Amafayela emigqa e-NVMe yasendaweni anciphisa ukuncika ku-NFS export ye-head node futhi angathuthukisa kakhulu isivinini se-prefill.
• Gcina i-deployment igxile embhalweni kuphela: I-SM120 kernel envelope ayinayo i-sparse-MLA kernel edingekayo ngobubanzi bama-vision-token bemodeli.
• Ungawavuli kabusha kalula ama-EXL3 auxiliary streams: I-deployment yenza ama-EXL3 streams abe serial ngoba ama-kernel amabili kuma-CUDA streams ahlukene angabambeka ne-runtime ngenxa ye-single lock buffer eyodwa kudivayisi ngayinye.
• Bheka ama-network interface afanele: I-NCCL ayikwazi ukusebenzisa ama-loopback alias e-10.0.0.x. Ukukhethwa okungalungile kwe-GID nge-NIC ngayinye kungahluleka ngemva cishe komzuzu nge-ibv_modify_qp errno 61.
• Hlola inkumbulo ye-prefill ende: Impumelelo yokuqalisa kanye nezivivinyo ezimfushane ze-smoke akufakazeli ukuthi i-KV pool enkulu noma i-context setting iphephile.
• Sebenzisa i-repository launcher: Ihlanganisa ukuvumelanisa kwe-NFS noma kwasendaweni, ukuhlolwa kwe-preflight, ukuqinisekiswa kwesithombe, ukuqaliswa kwama-rank amabili, kanye nokuxilonga.

Ukusebenza Okulindelekile

Izilinganiso ezishicilelwe zibika amathokheni angu-31.6 ngomzuzwana kumfudlana owodwa we-decode kanye namathokheni angu-42.5 ngomzuzwana ngokuhlanganisiwe emifudlaneni emibili. I-prefill yamaprompt amafushane ibicishe ibe amathokheni angu-965 kuya kwangu-1,041 ngomzuzwana ngedatha ye-Engram epakiwe nama-kernel ochwepheshe ahlanganisiwe.

I-prefill yamaprompt amade iba kancane ngokwemvelo. Izibonelo ezibikiwe zifaka cishe amathokheni angu-979 ngomzuzwana kuprompt enamathokheni angu-100,000, amathokheni angu-802 ngomzuzwana kumathokheni angu-455,000, kanye namathokheni angu-810 ngomzuzwana kumathokheni angu-601,000. Imiphumela yangempela incike ekubunjweni kweprompt, ekucushweni kwesitoreji, ekuqageleni, ebubanzini be-batch, kanye nosayizi we-prefill chunk olungiselelwe.

Isiphetho

Le phrojekthi ihlanganisa ama-kernel akhethekile, i-EXL3 loader, ukuqaliswa kwe-Engram okusekelwe kumafayela, ukucushwa kwenethiwekhi, izilawuli zenkumbulo, kanye ne-launcher okudingekayo ukuze kusetshenziswe i-DeepSeek V4.1 Flash kuma-GB10 DGX Sparks amabili. Qala ngamasethingi anikeziwe, qinisekisa i-API usebenzisa i-smoke test, bese ulungisa isethingi eyodwa ngesikhathi. Ukuze usebenze ngokuphepha kakhulu, hlola kokubili ukusebenza kanye nenkumbulo etholakalayo ngesikhathi semithwalo yangempela ye-prefill ende.