HelixWorld Uyasebenza

I-HelixWorld iyimodeli yomhlaba ye-audio nevidiyo esebenzisanayo ngesikhathi sangempela, ekhishwe njengomthombo ovulekile ngokubambisana yi-Noiz AI ne-HKUST. Nikeza imodeli isithombe sokuqala nencazelo yombhalo, bese usebenzisa okhiye be-WASD nemicibisholo ukuhamba nokuphendula ikhanda emhlabeni okhiqizwe yi-AI. Izithombe nemisindo ye-stereo yendawo kukhiqizwa ngesikhathi sangempela futhi kuvumelaniswe, kuyilapho indawo yomsindo ishintsha ngokwemvelo ngokubuka. I-HelixWorld yakhiwe phezu kwe-LTX-2.3 futhi isebenzisa ukuqeqeshwa kwe-“bidirectional teacher + causal distillation”. Ikhadi elilodwa le-H800 lingakhiqiza ngesikhathi sangempela ngokulungiswa kwe-768×512 nange-24 FPS. Kukhishwe kanye ne-HelixBench, ibhentshimakhi eklanyelwe ukuhlola ukuvumelana kwe-acoustics yendawo.

Ukukhiqizwa komhlaba osebenzisanayo ngesikhathi sangempela: faka isithombe sokuqala nencazelo yombhalo ukuze uhlole ngokukhululeka umhlaba okhiqizwe yi-AI.
Ukulawula ngekhibhodi/izenzo: i-WASD ilawula ukuhamba, kanti okhiye bemicibisholo balungisa ukubuka; isithombe siphendula ngokushesha ezenzweni.
Ukukhiqizwa okuhlangene komsindo nevidiyo ngokwemvelo: ividiyo nomsindo we-stereo kukhiqizwa ngesikhathi esisodwa, hhayi ukwengezwa komsindo ngemva kwalokho.
Ukuvumelana kwe-acoustics yendawo: indawo yomsindo ishintsha ngokwemvelo lapho ikhamera ijikiswa; isibonelo, umsindo ovela kwesokunxele ushintsha uhlangothi uma uphenduka ubhekise ngemuva komthombo.
Imiyalelo ehlukaniswe ngamathrekhi amathathu: sebenzisa izincazelo ezihlukene zevidiyo, umsindo, kanye ne-audio nevidiyo okuhlangene ukuze ulawule okuqukethwe okukhiqizwayo.
Okuhlangenwe nakho okushelelayo ngesikhathi sangempela: ikhadi elilodwa le-H800 lifinyelela ku-768×512 naku-24 FPS ekukhiqizeni kwesikhathi sangempela.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxoxa lamaphrojekthi e-AI omthombo ovulekile
Ukuqeqeshwa kwangaphambili kwemodeli kathisha enezinhlangothi ezimbili: kuqala kuqeqeshwa imodeli kathisha yokusabalalisa i-audio nevidiyo enezinhlangothi ezimbili. I-trajectory yekhamera eqhubekayo ye-6-DoF ifakwa ku-visual attention nge-PRoPE, izenzo ezihlukene ezingu-81 zilungisa izinyathelo zesikhathi sokusabalalisa nge-AdaLN-Zero, kanti i-cross-modal attention idlulisela ukunyakaza kwekhamera egatsheni lomsindo ukuze indawo ye-stereo ishintshe ngokwemvelo.
I-causal distillation yokufeza ukusebenza kwesikhathi sangempela: imodeli kathisha idistilwa ibe imodeli yomfundi ye-causal enezinyathelo ezimbalwa. Ama-video latent nama-audio token acutshungulwa ngamabhulokhi; kusetshenziswa i-bidirectional attention ngaphakathi kwebhulokhi, ukuxhumana kwe-causal phakathi kwamabhulokhi nge-KV cache, kanye nokuqalisa kwe-ODE, ukuze umfundi aqede ukususwa komsindo ngezinyathelo ezi-4 futhi ahlangabezane nezidingo ze-inference yesikhathi sangempela.
I-online trajectory distillation yokunciphisa i-drift: ukubhekana nokuqoqeka kwamaphutha ekukhiqizweni kwe-autoregressive okude, ngesikhathi sokuqeqesha isiphetho se-Probability Flow ODE sikathisha sisetshenziswa njengokuqondisa kwenkundla yejubane yomfundi. Lokhu kushintshaniswa ngokungahleliwe ne-distribution matching loss (DMD), kugcinwe kokubili ukuhlukahluka kokukhiqizwa nokuzinza kwesikhathi eside.
Ukulungisa kahle ukusakazwa kwesikhathi eside: kusetshenziswa i-sliding-window KV cache ukuze kunwetshwe uchungechunge olude kancane kancane ngesikhathi sokuqeqesha, kugcinwe ama-sink frame anchors futhi kunqunywe ama-gradient omlando. Kanye nokulungisa kahle uchungechunge olude oluthatha cishe umzuzu, lokhu kunciphisa ngokuqhubekayo i-drift yesithombe nokonakala komsindo ekukhiqizweni okude.

Indlela yokuqala: ukusebenzisa ku-inthanethi Vula iwebhusayithi esemthethweni https://helixworld.org, bese ukhetha isithombe sesibonelo noma ulayishe esakho.
Gcwalisa incazelo yombhalo yesigcawu.
Ngemva kokungena emhlabeni, sebenzisa i-WASD ukuhamba nokhiye bemicibisholo ukuphendula ukubuka, uhlole okuqukethwe okukhiqizwa ngesikhathi sangempela.
Ngesikhathi kuqhutshwa, ungashintsha imiyalelo noma nini ukuze ulungise izigcawu ezizokhiqizwa kamuva.
Indlela yesibili: ukufaka nokusebenzisa endaweni yangakini Ukulungiselela imvelo: isistimu ye-Linux, i-Python 3.11, i-CUDA 12.x, ikhadi lemifanekiso le-NVIDIA (kunconywa imemori yevidiyo engu-80GB), i-ffmpeg ne-ffprobe.
Landa iphrojekthi ufake okudingekayo: git clone https://github.com/NoizAI/HelixWorld.git, bese udala imvelo ye-conda futhi usebenzise i-pip install -r requirements.txt.
Qalisa i-python download_models.py ukuze ulande izisindo zemodeli.
Lungisa isithombe sokuqala, bese usebenzisa i-./run.sh ukucacisa isithombe, imiyalelo yesithombe/yomsindo/okuhlangene, ukulandelana kwezenzo neminye imingcele. I-MP4 izokhiqizwa kufolda ye-outputs.

Imodeli yokuqala yomhlaba ye-audio nevidiyo esebenzisanayo ngesikhathi sangempela yomthombo ovulekile: isithombe nomsindo we-stereo yendawo kukhiqizwa ndawonye ngesikhathi sangempela ngaphansi kokulawulwa ngumsebenzisi, futhi ikhodi nezisindo kuvulekile ngokuphelele.
“Ukuphendula ikhanda kushintsha umsindo”: indawo yomsindo ilandela ngokuqinile ukujika kokubuka, kanti amaphuzu okuvumelana kwendawo aphindwe kaningi kunalawo emikhiqizo efanayo.
Isikhathi sangempela ngekhadi elilodwa: i-H800 eyodwa ifinyelela ku-24 FPS ne-RTF 0.77 ekukhiqizweni kwesikhathi sangempela, ijubane elingaphezu kokuphindwe ka-2.5 kunemikhiqizo efanayo.
Ukukhiqizwa okude ngaphandle kwe-drift: i-online trajectory distillation nokulungisa kahle isikhathi eside kugcina isithombe nomsindo kuzinzile ekukhiqizweni okuqhubekayo okungaphezu kwemizuzwana engu-30, kuyilapho ukuhlukahluka kunganciphi kodwa kwanda.
Ikhwalithi yomsindo ehamba phambili ngokuphelele: i-HelixBench ihamba phambili kuzo zonke izinkomba eziyinhloko, kanti ekuhlolweni okungaziwa ngabasebenzisi ithole izinga lokuthandwa elingu-67.9% ezigcawini ezinomsindo, yahlula kakhulu izimbangi.

Iwebhusayithi esemthethweni yephrojekthi: https://helixworld.org/
Indawo yokugcina ye-GitHub: https://github.com/NoizAI/HelixWorld
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2609.38123

Amamodeli emidlalo egxilisa umsebenzisi: ngaphandle kokumodela noma injini yomdlalo, faka isithombe ukuze ukhiqize umhlaba ovulekile ongahanjwa futhi uhlolwe, unciphise kakhulu isikhathi sokwakha i-demo yomdlalo.
Izikhala ezibonakalayo ze-VR/AR: ukuhlanganiswa kokubuka komuntu wokuqala nomsindo wendawo kuhambisana ngokwemvelo namadivayisi e-headset, futhi kungakha ngokushesha amahholo ombukiso abonakalayo nezindawo zama-digital twin.
Ukuzijwayeza kwangaphambili kwefilimu nokukhangisa: umqondisi angalayisha isithombe somqondo ukuze “angene” esigcawini futhi ezwe abone umphumela womsindo nesithombe, aqinisekise ngentengo ephansi ulimi lwekhamera nokwakheka komoya ngaphambi kokuqoshwa kwangempela.
Isizinda sezingcingo zevidiyo ze-AI/abasakazi ababonakalayo: khiqiza ngesikhathi sangempela isizinda esishintshayo nemisindo ehambisana nezwi nokuxhumana, kuthathelwe indawo isikrini esiluhlaza nezinto ezingashintshi.
Ukuqeqeshwa nokulingisa kobuhlakani obunezitho: njengesilingisi somhlaba esisebenzisanayo, sinikeza amarobhothi nokushayela okuzenzakalelayo indawo yokuqeqesha ebiza kancane enempendulo yomsindo nokukhanya, kusize ukufunda imithetho yokuxhumana ngokomzimba.