EchoWM Uyasebenza

I-EchoWM iyimodeli yomhlaba esebenzisanayo yomsindo nezithombe, ekhishwe njenge-open source yi-JD Exploration Research Institute. Imodeli iqhubeka namandla omdabu okwenza umsindo nevidiyo e-JoyAI-Echo, ivumela abasebenzisi ukuthi bahambe futhi bahlole ngesikhathi sangempela emhlabeni owenziwe yi-AI ngokubuka komuntu wokuqala noma wesithathu. Izithombe, ubudlelwano bendawo, umsindo wemvelo nenkulumo kuhlala kuvumelana futhi kungaphambuki isikhathi eside. Imodeli iqeqeshwe kusetshenziswa isixhumi esibonakalayo esihlanganisiwe senhloso yekhamera kanye nenjini yedatha yomhlaba; isilinganiso sayo esingu-81.7 ekuhlolweni kokuzulazula kwe-WBench siyibeke endaweni yokuqala.

Ukwakhiwa komhlaba ongahlolwa ngesikhathi sangempela: Abasebenzisi bangasebenzisa okhiye be-WASD ukuze bahambe ngokukhululeka, bajike futhi babuyele emuva ezimweni ezakhiwe yi-AI; umhlaba uyavuleka ngesikhathi sangempela ngokulandela izenzo zabo.
Ukwakhiwa komsindo nevidiyo okudidiyelwe ngokomdabu: Ngaphakathi kohlaka olulodwa, kwakhiwa ngesikhathi esisodwa ividiyo engu-720p nomsindo wemvelo, umculo nenkulumo yabalingiswa; umsindo uyaqhubeka ngokwemvelo njengoba izithombe zishintsha.
Ukulawula ukubuka komuntu wokuqala nowesithathu: Ekubukeni komuntu wokuqala, ikhamera ihamba njengombukeli; ekubukeni komuntu wesithathu, ikhamera ilandela ngokuzenzakalelayo ukunyakaza kwabalingiswa, izimoto nezinye izinto eziyinhloko.
Ukuqhubeka kwemijikelezo eminingi yesikhathi eside: Umsindo nevidiyo ekupheleni komjikelezo odlule kusetshenziswa njengomongo womjikelezo olandelayo; ngemva kwezenzo eziningi, ukuhleleka kwendawo, ukubukeka kwezinto eziyinhloko nomsindo kuhlala kuhambisana.
Indawo yokuqala ekuhlolweni kokuzulazula kwe-WBench: Ezibonelweni ezingu-158, kutholwe isilinganiso esingu-81.7; izinkomba zokuhambisana nokusebenzisana zombili zibe phakathi kweziphezulu.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxhumana lamaphrojekthi e-AI open source
Isixhumi esibonakalayo esihlanganisiwe senhloso yekhamera: Imiyalo yekhibhodi ehlukene iguqulwa ibe umzila oqhubekayo we-6-DoF ohlobene nesimo sokuqala, kusetshenziswa isixhumi esibonakalayo esisodwa ukuchaza ukuthi ikhamera ihamba kanjani. Ekubukeni komuntu wokuqala, lokhu kumelela ngokuqondile ukunyakaza kombukeli; ekubukeni komuntu wesithathu, imodeli ifunda kudatha ubudlelwano obuhlanganisiwe “bokunyakaza kwento eyinhloko nokulandela kwekhamera”, ngaphandle kwesidingo semizila eyengeziwe yabalingiswa noma amapharamitha ekhamera asethwe kusengaphambili. Umzila ufakwa kuphela egatsheni levidiyo, kuyilapho umsindo ulandela izithombe.
Injini yedatha yomhlaba: Ihlanganisa izinhlobo ezine zedatha ezihambisanayo, okuhlanganisa gameplay eqoqwe ngaphakathi, amavidiyo abantu bedlala khona, ukulingisa kwe-UE namavidiyo ajwayelekile e-inthanethi. Bese isebenzisa i-percentile engu-90 yobukhulu bokuhumusha bomzila njenge-“rula” yomhlaba wonke ukuze yenze i-normalization efanayo, igcine umehluko wokuhamba ohlobene phakathi kwemizila ehlukene futhi igweme ukushintsha okungazelelwe kwesivinini okubangelwa ukukala ingxenye ngayinye ngokwehlukana.
Ukuqeqeshwa kwezigaba ezine okuqhubekayo: Esigabeni sokuqala, kusetshenziswa idatha ecebile ngomsindo nevidiyo ukwenza i-pre-training eqhubekayo yomsindo nevidiyo. Esigabeni sesibili, i-backbone iyamiswa, kuqeqeshwe kuphela igatsha lomzila elilula ukuze kuqiniswe ukusabela kokulawula. Esigabeni sesithathu, kwenziwa i-fine-tuning ehlangene ngezinga eliphansi lokufunda kusetshenziswa idatha enokwethenjelwa kokulawula nekhwalithi yomsindo nezithombe. Esigabeni sesine, kwenziwa i-autoregressive post-training, lapho i-Teacher Forcing iguqulelwa ekukhiqizeni okuyimbangela ngamabhulokhi, kuhlanganiswe ne-Self-Gradient Forcing yesikhathi esifushane ukuze imodeli ijwayele umlando ewukhiqizayo, bese kusetshenziswa i-distillation yokufanisa ukusatshalaliswa ukuze kuncishiswe ukukhiqiza kube amasampuli amane. Indlela yokugcina i-sink-plus-FIFO isekela ukusebenzisana okusakazwayo kwesikhathi eside ngesikhathi ilawula izindleko ze-control KV.

Clone indawo yokugcina: sebenzisa i-git clone ukulanda i-github.com/jd-opensource/JoyAI-Echo, bese ungene ku-directory ye-echo_wm/ (ezimele kuphrojekthi yamavidiyo amade i-echo_longvideo/).
Dala imvelo: sebenzisa i-conda create -n echo-wm python=3.11 ukuze wakhe imvelo ezimele bese uyivula.
Faka okuncikile: qala ufake i-PyTorch 2.9.1 (cu128), bese usebenzisa i-pip install -r requirements.txt; qinisekisa ukutholakala kwe-GPU nge-torch.cuda.is_available().
Landa izisindo zemodeli: sebenzisa i-hf download Echo-Team/Echo-WM ukulanda i-checkpoint yemodeli; futhi kudingeka ulande i-Gemma 3 text encoder (indawo yokugcina gated, ngakho qala wamukele ilayisensi ekhasini le-HF bese usebenzisa i-hf auth login).
Qalisa isibonelo esisemthethweni ukuze uqinisekise: sebenzisa scripts/run_wm_case.py --case examples/wm_cases/0010 nezinye izibonelo eziphelele ezakhelwe ngaphakathi ukuze uqinisekise ukuthi imvelo ilungiswe kahle.
Ukwakha ngokwezifiso: sebenzisa i-inference_wm.py, unikeze isithombe sohlaka lokuqala + prompt enezinkambu eziyisithupha + uchungechunge lwezenzo (Action DSL), bese uthola ividiyo enomsindo.
Ukubhala imiyalo yezenzo: Chaza ukunyakaza kwekhamera nge-Action DSL, ngefomethi ethi <ukhiye>-<inani lohlaka>, kuhlukaniswe ngamakhoma, isibonelo w-60,a-60 sisho ukuya phambili amafreyimu angu-60 bese uya kwesokunxele amafreyimu angu-60; kusekelwa inhlanganisela ye-w/s/a/d/i/j/k.

Ukuhlanganiswa komsindo nezithombe: Ngaphakathi kohlaka olulodwa, kwakhiwa ngokomdabu ividiyo engu-720p nomsindo wemvelo, umculo nenkulumo; umsindo ushintsha ngokuvumelana nendawo nezenzakalo ezibonakalayo, kuxazululwe inkinga yamamodeli amaningi omhlaba “akhiqiza izithombe kuphela ngaphandle komsindo”.
Isixhumi esibonakalayo sokulawula esihlanganisiwe: Umzila owodwa we-6-DoF ohlobene ushayela kokubili ukunyakaza kombukeli kumuntu wokuqala nokulandela kwento eyinhloko ngekhamera kumuntu wesithathu, ngaphandle kokwakha izilawuli ezihlukene zokubuka ngakunye.
Ukuhambisana kwesikhathi eside: Nge-autoregressive post-training nendlela yokugcina i-sink-plus-FIFO, abasebenzisi bangaqhubeka nokuhlola imijikelezo eminingi besebenzisa umsindo nevidiyo ekupheleni komjikelezo odlule njengomongo; indawo, izinto eziyinhloko nomsindo kuhlala kungaphambuki isikhathi eside.
Ukuhola ekuhlolweni: Isilinganiso esingu-81.7 ekuhlolweni kwe-WBench Navigation siyibeke endaweni yokuqala, futhi ucwaningo lwabasebenzisi lubonise ukuthandwa okuphelele okuphezulu kakhulu kune-LingBot-World-v2 ne-HappyOyster.

Iwebhusayithi yephrojekthi: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/
Indawo yokugcina ye-GitHub: https://github.com/jd-opensource/JoyAI-Echo
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2608.23189

Ukuthuthukiswa kwama-prototype emidlalo: Abadali bangena endaweni ekhiqiziwe ukuze bayidlale, baqinisekise ukuhleleka kwendawo, ukuklanywa kwemizila nolwazi lwekhamera ngaphambi kokunquma indlela yokukhiqiza.
Ukuxoxa izindaba okusebenzisanayo: Indaba iyaqhubeka futhi ikhiqizwa ngokulandela amagatsha akhethwe umsebenzisi, ithathe indawo yezinto ezilungiselelwe kusengaphambili futhi isetshenziswe emidlalweni yedrama nasezincwadini ezisebenzisanayo.
Okuqukethwe okugxilisa umsebenzisi: Okuqukethwe kwamafilimu nezikhangiso kuguquka ekubeni “ukubuka” kube “ukungena”, lapho ababukeli bengahlola khona ngokukhululeka umhlaba osezithombeni.
Ukuvakashela okungokoqobo: Ukuhamba ngedijithali ngezindleko eziphansi kumamnyuziyamu, ezindaweni eziheha izivakashi, ezindaweni zezindlu nezinye izimo, ngaphandle kokwakha imodeli yohlaka ngalunye.
Ukudlala kwabantu bedijithali: Inkulumo nokunyakaza kwabalingiswa, imvelo eguqukayo ezungezile, ikhamera nomsindo kukhiqizwa ngokubambisana, kulungele abasakazi ababonakalayo nabasebenzi bedijithali.