Qalisa Amamodeli e-Code-as-World Okucabanga Ngomzimba Ngokobuningi

Yini i-Code-as-World?

I-Code-as-World iphrojekthi yocwaningo emele umhlaba ophathekayo ngekhodi esebenzisekayo. Umqondo wayo oyinhloko uthi amaphikseli anikeza ubufakazi ngesigcawu, kodwa awazichazi ngokuqondile izinto, izimo, izakhiwo, ukuguquguquka, noma izindlela ezikhiqize lokho okwabonwayo.

Iphrojekthi isebenzisa inqubo esebenza njengomenzeli, esekelwe ekulingiseni nasekuqinisekiseni okuphindaphindiwe, ukuze ithole izethulo zomhlaba ezisebenzisekayo kulokho okubonwayo. Lezi zethulo ziguqula ubufakazi obubonakalayo bube idatha ephathekayo engasetshenziswa kabusha futhi zisekele ukucabanga ngobunjalo bomzimba ngobuningi.

Lokhu kukhishwa kunikeza ukusekelwa kokucubungula nokuhlola endaweni kwamamodeli amabili okubona nolimi: Code-as-World-VL-4B kanye Code-as-World-VL-9B. Kuhlanganisa nokusebenza okuhambisana ne-OpenAI nge-vLLM, kanye nesibonelo sokukhipha isifinyezo somzimba esiqhutshwa yividiyo, esakhiwe nge-MuJoCo.

Amakhono abalulekile

• Izethulo zomhlaba ezisebenzisekayo: Imodeli izimo, ukuguquguquka, nezindlela zomzimba, kunokuba iphathe amaphikseli njenge-ontoloji ephelele yesigcawu.
• Amamodeli amabili akhishiwe: Khetha phakathi kwamamodeli e-Code-as-World-VL e-4B ne-9B.
• Ukuhlola kwe-QuantiPhy: Yenza ukuhlola kokucabanga ngobunjalo bomzimba ngobuningi kumavidiyo okuqinisekisa futhi ukhiqize amafayela ezibikezelo ahambisana nomhloli.
• Ukusebenza okuhambisana ne-OpenAI: Veza noma iyiphi imodeli usebenzisa iseva evamile ye-vLLM API.
• Isibonelo sokulingisa: Qalisa icala lebhola likanobhutshuzwayo elihamba ngendlela ye-ballistic elihlanganisiwe bese ugcina ividiyo e-renderiwe nomzila wokuhamba.

Okudingekayo ngaphambi kokuqala

Sebenzisa i-Python 3.10 noma 3.11 kukhompuyutha enikwe amandla yi-CUDA. Uzodinga futhi i-Git kanye ne-Hugging Face CLI esetshenziswa imiyalo ye-hf download. Izidingo zesitoreji samamodeli nememori ye-GPU zincike ekutheni usebenzisa imodeli ye-4B, eye-9B, noma zombili.

Landela imiyalo yenqolobane usendaweni ye-CUDA. I-README ayinikezi iresiphi yokucubungula esebenza ku-CPU kuphela.

Faka i-Code-as-World

1. Klona inqolobane

git clone https://github.com/MirroS-Lab/Code-as-World.git
cd Code-as-World

2. Dala futhi uvule indawo ebonakalayo

python -m venv .venv
source .venv/bin/activate

Umyalo wokuvula ongenhla uhloselwe amagobolondo afana ne-Unix. Uma isivuliwe, faka okuncikile ekucubunguleni okunikezwe iphrojekthi:

pip install -r requirements/inference.txt

3. Dawuniloda amamodeli

hf download MirroS-Lab/Code-as-World-VL-4B --local-dir weights/4b
hf download MirroS-Lab/Code-as-World-VL-9B --local-dir weights/9b

Le miyalo ibeka amamodeli e-4B ne-9B ku-weights/4b naku-weights/9b. Uma uhlose ukusebenzisa imodeli eyodwa kuphela, dawuniloda imodeli ehambisana nayo kuphela.

Yenza ukuhlola kwe-QuantiPhy

I-QuantiPhy iyisethi yokuhlola ngobuningi esetshenziswa yilokhu kukhishwa. Ngaphambi kokwenza ukuhlola kwe-Code-as-World, klona i-QuantiPhy bese udawuniloda amavidiyo ayo okuqinisekisa.

1. Lungisa idatha yokuhlola

git clone https://github.com/Paulineli/QuantiPhy.git /path/to/QuantiPhy
hf download PaulineLi/QuantiPhy-validation \
  --repo-type dataset \
  --local-dir /path/to/QuantiPhy-validation

Faka izindawo zangempela ohlelweni lwakho esikhundleni se-/path/to/QuantiPhy ne-/path/to/QuantiPhy-validation.

2. Hlola imodeli ye-4B

Usuka enqolobaneni ye-Code-as-World, sebenzisa:

python -m code_as_world.evaluation 4b \
  --input-csv /path/to/QuantiPhy/quantiphy_validation.csv \
  --video-dir /path/to/QuantiPhy-validation/validation_videos

3. Hlola imodeli ye-9B

python -m code_as_world.evaluation 9b \
  --input-csv /path/to/QuantiPhy/quantiphy_validation.csv \
  --video-dir /path/to/QuantiPhy-validation/validation_videos

Ukuqalisa ngakunye kubhala izinhlobo ezintathu zezinto ku-outputs/quantiphy/: i-CSV yezibikezelo ehambisana nomhloli, isifinyezo samamethrikhi sokuqalisa ngakunye, kanye nemiphumela eluhlaza ekhiqizwe imodeli. Ukugcina imiphumela eluhlaza kuyasiza lapho kuhlolwa izimpendulo ezingemuva kwamaphuzu ngamunye.

4. Qalisa umhloli osemthethweni

Ngemva kokukhiqiza amafayela e-CSV ezibikezelo, faka i-pandas bese ubiza umhloli osemthethweni we-QuantiPhy:

pip install pandas
python /path/to/QuantiPhy/evaluator.py \
  outputs/quantiphy \
  outputs/quantiphy_metrics \
  --gt_file /path/to/QuantiPhy/quantiphy_validation.csv

Lo myalo uhlola amafayela ezibikezelo atholakala ku-outputs/quantiphy ngokuqhathanisa nawo i-quantiphy_validation.csv, bese ubhala amamethrikhi avelayo ku-outputs/quantiphy_metrics.

Sebenzisa imodeli nge-vLLM

Amamodeli angavezwa nge-endpoint ye-vLLM ehambisana ne-OpenAI. Umyalo olandelayo usebenzisa imodeli ye-4B ku-GPU 0:

CUDA_VISIBLE_DEVICES=0 vllm serve weights/4b \
  --served-model-name code-as-world-4b \
  --chat-template code_as_world/templates/qwen3_5_no_think.jinja \
  --chat-template-content-format openai \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --max-model-len 4608 \
  --gpu-memory-utilization 0.90 \
  --media-io-kwargs '{"video":{"num_frames":16,"fps":-1,"video_backend":"openpangu"}}' \
  --mm-processor-kwargs '{"do_sample_frames":false}' \
  --mm-processor-cache-gb 0 \
  --generation-config vllm

Iseva ibhalisa imodeli ngaphansi kwegama elithi code-as-world-4b. Isebenzisa i-chat template yephrojekthi engacabangi, isetha ubude obukhulu bemodeli bube ngu-4608, futhi ilungiselela okokufaka kwevidiyo kube amafreyimu angu-16 kusetshenziswa i-backend ye-openpangu.

Ukuze usebenzise i-checkpoint enkulu, shintsha weights/4b ufake weights/9b, bese ushintsha code-as-world-4b ufake code-as-world-9b.

Qalisa isibonelo sokwenza i-abstraction esiqhutshwa yividiyo

Indawo yokugcina ikhodi iqukethe isibonelo sebhola likanobhutshuzwayo le-ballistic esisekelwe ku-MuJoCo. Qala ngokufaka izidingo ezihlukile zokulingisa:

pip install -r requirements/simulation.txt

Bese uqala ukulingisa:

python -m code_as_world.simulation

Ukulingisa kubhala ividiyo ekhiqiziwe kanye ne-trajectory ku-outputs/simulations/.

Khiqiza i-trajectory kuphela

Uma ungadingi ividiyo ekhiqiziwe, vala ukukhiqizwa kwayo:

python -m code_as_world.simulation --no-render

Le modi iyasiza lapho kudingeka idatha ye-trajectory kuphela noma lapho ukukhiqiza ividiyo kungangeza umthwalo ongadingekile.

Amathiphu okusetshenziswa okuthuthukile

• Khetha imodeli ngenhloso: Sebenzisa impikiswano yokuhlola engu-4b noma engu-9b ngokuhambisana ne-checkpoint oyilandile.
• Landa okudingayo kuphela: Gwema ukugcina ama-checkpoint womabili uma ukuhamba kwakho komsebenzi kusebenzisa usayizi owodwa wemodeli kuphela.
• Gcina ama-generation angahluziwe: Hlola amafayela aku-outputs/quantiphy/ lapho uphenya ukuthi kungani isibikezelo sathola amaphuzu athile.
• Qalisa womabili amazinga okuhlola: Umyalo we-Code-as-World udala isifinyezo sokuqalisa okukodwa, kanti i-evaluator esemthethweni ye-QuantiPhy ingahlola amafayela e-CSV ezibikezelo ezikhiqiziwe ndawonye.
• Gcina i-chat template enikeziwe: Iresiphi yokusebenzisa iseva isebenzisa ngokuqondile code_as_world/templates/qwen3_5_no_think.jinja futhi ivala ukucabanga. Gcina lezi zilungiselelo lapho uphinda ukucushwa okubhalwe phansi.
• Lungisa ukukhethwa kwe-GPU ngokucophelela: Isibonelo sokusebenzisa iseva sisebenzisa CUDA_VISIBLE_DEVICES=0 kanye nenani lokusetshenziswa kwememori ye-GPU elingu-0.90. Khetha idivayisi ebonakalayo ngokuvumelana ne-CUDA host yakho, kodwa uphathe ezinye izinguquko njengokuphambuka eresiphi ekhishiwe.
• Yeqa ukukhiqizwa kwevidiyo lapho kufanele: Faka --no-render emyalweni wokulingisa lapho i-trajectory kuwukuphela komphumela odingekayo.

Okunye okufundwayo

Ngolwazi lwangemuva locwaningo nendlela yokusebenza, bheka umbiko wezobuchwepheshe, ikhasi lephrojekthi, kanye nokuthunyelwe kwebhulogi ye-MirroS.

Isiphetho

ICode-as-World inikeza ukukhishwa okusebenzisekayo kokuhlola izethulo ezisebenzisekayo zezigcawu ezibonakalayo. Ngemva kokufaka indawo yokusebenza ye-inference nokulanda i-checkpoint, ungaphinda uhlole i-QuantiPhy, usebenzise imodeli nge-vLLM, noma uqale ukulingisa kwe-MuJoCo okufakiwe. Ngokuhlangene, lezi zinqubo zibonisa ukuthi ukubonwa okubonakalayo kungasekela kanjani ukucabanga komzimba okusobala nokuqinisekisekayo.