WorldSonus Uyasebenza

I-WorldSonus iwuhlaka lokudubha amavidiyo olusakazwayo olukhishwe ngokubambisana yi-HKUST, i-Noiz AI, i-MetaX kanye ne-Shanghai Jiao Tong University. Ikhiqiza umsindo ngesikhathi ividiyo isakazwa, ikhiphe umsindo we-stereo ongu-48kHz ngamabhulokhi angu-100ms. I-WorldSonus isebenzisa i-causal AR diffusion architecture kanye ne-5-second Ring-KV cache, futhi ivumela ukushintshwa okungenazihibe kwemiyalelo yemisindo ngesikhathi sokukhiqiza. Iziteshi zesokunxele nesokudla zilandela indawo yomthombo womsindo nokunyakaza kwekhamera. Kumasethi okuhlola amahlanu, ikhwalithi yomsindo idlula ngokuphelele amamodeli angaxhunyiwe e-Offline afana ne-AudioX ne-ThinkSound.

Ukudubha okusakazwayo ngesikhathi sangempela: umsindo ukhiqizwa futhi ukhishwe ngesikhathi ividiyo isakazwa, ngebhulokhi engu-100ms ngayinye; ukubala ibhulokhi eyodwa kuthatha u-41.2ms kuphela, ukusetshenziswa kwememori ye-GPU kuhlala kungashintshi, futhi ukusakaza kungaqhubeka isikhathi esingenamkhawulo.
Ukushintsha umsindo phakathi nokukhiqiza: imiyalelo ebhaliwe yemisindo ingashintshwa noma nini ngesikhathi sokukhiqiza, ishintshwe ngaphandle kwezihibe emngceleni webhulokhi yakamuva, ngaphandle kokulahlekelwa isimo seseshini noma ukubala kabusha amafreyimu adlule.
Umsindo we-spatial stereo: ikhipha umsindo weziteshi ezimbili ongu-48kHz, lapho indawo yomthombo womsindo nokunyakaza kwekhamera kubonakala ngesikhathi sangempela eziteshini zesokunxele nesokudla.
Imodular futhi ixhunywa kalula: izimele ku-visual backbone, yamukela kuphela “amafreyimu evidiyo asevele akhiqiziwe”, futhi ingaxhunywa kunoma iyiphi i-world model noma injini yomdlalo.
Isebenza ezizindeni ezivulekile: ngaphandle kwezigcawu ezisebenzisanayo, ikhwalithi yomsindo kumavidiyo ajwayelekile afana ne-VGGSound isadlula amamodeli angaxhunyiwe e-Offline.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lezingxoxo zamaphrojekthi e-AI avulekile
Ukukhiqiza okuyimbangela okusakazwayo ngamabhulokhi (AR-Diffusion): umsindo ufakwa endaweni ye-continuous latent ye-causal stereo VAE, lapho u-100ms ngamunye uhambisana namafreyimu angu-3 e-latent; i-decoder-only Transformer ifunda kuphela ama-token amabili ahlanganisiwe, “ibhulokhi yomsindo edlule + ibhulokhi levidiyo lamanje”, esinyathelweni ngasinye. I-Ring-KV circular cache enamabhulokhi angu-W=50 inikeza inkumbulo enomkhawulo, bese i-flow head elula isusa umsindo ebhulokhini lamanje isebenzisa umgomo we-rectified-flow.
Izimo ezibonakalayo ezinezilinganiso ezimbili zesikhathi: i-DINOv3 emisiwe ifaka ifreyimu ngayinye njengegridi yama-patch futhi ihlanganise izithombe zomehluko wamafreyimu ukuze ibambe ukunyakaza; ifreyimu ngayinye ihlanganiswa ibe yi-token engu-1. Ama-token amafreyimu angu-3 aphinde ahlanganiswe abe yi-token engu-1 yezinga lebhulokhi, ethunyelwa ku-AR backbone ukuze ilawule ukuhambisana kwesikhathi eside. Ama-token angu-3 aqondaniswe namafreyimu adlula i-backbone ngqo aye ku-flow head ukuze alawule ukunikezwa okuningiliziwe kokuvumelanisa umsindo nesithombe, kuhlukaniswe ukuhlela kwesikhathi eside nokunikezwa kwasendaweni.
Ukufundisa kokuqondanisa kwe-ShiftNCE: i-Synchformer emisiwe isetshenziswa njengothisha ngesikhathi sokuqeqesha, kanti ukufunda ngokuqhathanisa kudlulisela ikhono lokuvumelanisa esimweni sangaphakathi se-AR. Izibonelo ezinhle ngamawindi aqondaniswe nesikhathi; ezimbi ngamawindi evidiyo efanayo asuswe ngamabhulokhi ambalwa. Ama-negative false anokuphakama kokufana ayahlungwa. Ngesikhathi sokucabanga akukho zindleko ezengeziwe; ngemva kokuwususa, i-DeSync iba yimbi ukusuka ku-0.831 iye ku-1.067.
Ukuhlelwa kwezinkomba okushintsha ngesikhathi nokulawulwa kwezinkomba endaweni: izinkomba zicindezelwa yi-T5Gemma 2 zibe iqoqo lama-token futhi zigcinwe ku-cross-attention. Lapho kushintshwa amagama phakathi nokukhiqiza, le cache ishintshwa endaweni emngceleni webhulokhi yakamuva kuphela; i-Ring-KV kanye nesimo somsindo nesithombe akushintshi. Ngesikhathi sokuqeqesha kufakwa ngamabomu izinhlelo ezine ezishintsha ngesikhathi: ukushintsha phakathi, ukuhoxisa umyalo, ukwengeza umyalo, nokushintsha phakathi kokunye. Ngaleyo ndlela imodeli ifunda ukushintsha izigcawu ngaphandle kwezihibe.

Ukulungiselela imvelo: lungisa i-Python 3.10+, ikhadi le-CUDA GPU, i-FFmpeg/ffprobe, futhi uqinisekise ukuthi izishayeli ze-GPU zifakiwe.
Thola ikhodi: sebenzisa i-git clone https://github.com/NoizAI/WorldSonus.git bese ungena kumkhombandlela wephrojekthi.
Faka okuncikile kukho: sebenzisa bash scripts/setup.sh .venv ukuze ufake ngokuzenzakalelayo i-PyTorch ye-CUDA 12.4, bese usebenzisa source .venv/bin/activate kanye ne-pip install -e '.[features]'.
Thola izisindo zemodeli enkulu: sebenzisa python scripts/download_model.py --output assets ukuze ulande izisindo ze-WorldSonus.
Thola ama-encoder: sebenzisa python scripts/download_encoders.py --output assets ukuze uthole i-DINOv3 ne-T5Gemma 2.
Ukudubha ingxenye yonke ungaxhunyiwe: sebenzisa python -m worldsonus.infer --video input.mp4 --prompt "umbhalo ochaza imisindo" --output output.wav. Lokhu kuqedela ngokuzenzakalelayo ukukhishwa kwezici, ukukhiqiza nokuhlela kabusha, bese kukhipha i-WAV ye-stereo engu-48kHz.
Izinketho ozikhethela zona: sebenzisa --start 5 --seconds 10 ukuze usike iwindi levidiyo, noma uyeke --prompt ukuze ukhiqize usebenzisa izimo zevidiyo kuphela.
Ukukhiqiza okusakazwayo ngesikhathi sangempela: engeza i- --stream ukuze ukhiqize umsindo ngokwandayo ngamabhulokhi.
Ukudlala ngesikhathi sokukhiqiza: engeza --stream --pcm-stdout ukuze uthumele i-PCM eluhlaza ngepayipi uye ku-ffplay -f f32le -ar 48000 -ac 2 -i - -nodisp -autoexit futhi uzwe umsindo ngokushesha.

Ukwakheka kwe-causal okusakazwayo: kukhiqiza umsindo ngesikhathi sangempela ngebhulokhi engu-100ms, kubalwa u-41.2ms kuphela ibhulokhi ngayinye; ukusetshenziswa kwememori ye-GPU kuhlala kungashintshi futhi ukusakaza kungaqhubeka isikhathi esingenamkhawulo.
Ukushintsha amagama phakathi nokukhiqiza: imiyalelo yemisindo ingashintshwa ngaphandle kwezihibe noma nini ngesikhathi sokukhiqiza, ngaphandle kokulahlekelwa isimo futhi ukubambezeleka kuhlala kungaphakathi kwesikhathi sangempela.
Umsindo we-spatial stereo: okukhiphayo kweziteshi ezimbili okungu-48kHz, okuqondanisa indawo yomthombo womsindo nokunyakaza kwekhamera ngesikhathi sangempela.
Kulula ukuhlanganisa: yamukela kuphela amafreyimu evidiyo asevele akhiqiziwe futhi ingaxhunywa kunoma iyiphi i-world model noma injini yomdlalo.
Ikhwalithi yomsindo idlula amamodeli angaxhunyiwe e-Offline: kumasethi okuhlola amahlanu, izinkomba ezifana ne-FAD ziphakeme ngokuphelele kunezisekelo ezinezinhlangothi ezimbili ezibona ividiyo ephelele.
Ukuzinza kwesikhathi eside: ngaphansi kwe-5-second Ring-KV cache, ikhiqiza ngokuqhubekayo imizuzwana engu-30 ngaphandle kokuphambuka noma ukuwa.

Iwebhusayithi yephrojekthi: https://noizai.github.io/WorldSonus/
Indawo yokugcina ye-GitHub: https://github.com/NoizAI/WorldSonus
Indawo yamamodeli e-HuggingFace: https://huggingface.co/FF2416/WorldSonus
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/abs/2610.08760

Ukudubha ngesikhathi sangempela kwama-world model nezinjini zemidlalo: ikhiqiza umsindo wendawo ngesikhathi sangempela ngokuhambisana nokusakazwa kwezithombe kwama-world model asebenzisanayo afana ne-GameGen-X ne-LingBot-World 2.0, kanye nezinjini zemidlalo. Lapho isithombe siya khona, umsindo uyalandela.
Ukusakaza okubonakalayo nabantu bedijithali: umsindo uvumelaniswa ngesikhathi isithombe sokusakaza sikhiqizwa; umsakazi angashintsha imisindo ngokushesha ngemiyalelo yezwi noma yombhalo phakathi nokusakaza, ngaphandle kokumisa noma ukuqopha kabusha.
Umculo wamavidiyo amade okhiqizwa ngesikhathi sokusakaza: ezimweni ezifana nama-trailer amafilimu namavidiyo amade akhiqizwe yi-AI, umculo we-stereo ongu-48kHz ukhiqizwa ngamabhulokhi ngaphandle kokulinda ukunikezwa kwengxenye yonke, kuncishiswe kakhulu isikhathi sokuhlela ngemva kokukhiqiza.
I-VR/AR nezindawo zokuqeqesha zokulingisa: ezindaweni ezifana nokulingisa okubonakalayo, ukuqeqesha ukushayela okuzenzakalelayo nokulingisa ezempi, ikhiqiza umsindo we-spatial oqondaniswe nekhamera ngokusakazwa kwezithombe ezinezindawo zokubuka eziguqukayo, ithuthukise ukucwiliswa nokuba ngokoqobo kokuqeqesha.
Ukubona kwama-agent namarobhothi: inikeza ama-agent ahlakaniphile anomzimba amasignali avumelanisiwe okubona nokuzwa ngesikhathi sangempela, ukuze athole impendulo yendawo ye-stereo ngesikhathi esifanayo nokufakwa okubukwayo, isize ukuzulazula nezinqumo zokusebenzisana.