WorldCast Uyasebenza

I-WorldCast iyimodeli yokukhiqiza amavidiyo ekhishwe yi-Wuhan University ngokubambisana ne-Xiaomi. Ngokufaka isithombe esisodwa sereferensi yomlingiswa, umzila wekhamera ohlosiwe kanye nezithombe ezingu-1–4 zezikhonkwane ezivela kunoma yimaphi ama-engeli, ingakhiqiza ividiyo lapho ikhamera ijikeleza umlingiswa ngo-360 degrees ngaphandle kokulahlekelwa ubuwena bakhe. I-WorldCast ikufeza lokhu ngezindlela ezimbili: enye ilawula umzila wekhamera, iguqule isithombe sereferensi sibe yi-point cloud bese inikeza ukuqondisa kwe-frame ngayinye endleleni; enye ilawula ukuvumelana kwama-engeli ahlukahlukene, igcine izithombe zezikhonkwane njengememori engaseshwa futhi ibuyise ulwazi lwe-engeli efanele lapho ikhamera ifika kuyo, ngaleyo ndlela igweme ukushintsha kobuso lapho umlingiswa efulathele ikhamera.

Ukulawula ikhamera nge-engeli enkulu: Faka umzila wekhamera ohlosiwe ukuze ikhamera ijikeleze, isondele noma ihlehle, futhi izungeze umlingiswa, ngokusekela ukunyakaza okuphelele okufika ku-360 degrees.
Ukuvumelana kobuwena kuma-engeli ahlukahlukene: Ubuso, izinwele nezingubo zomlingiswa kuhlala kuwubuwena obufanayo kunoma iyiphi i-engeli kuvidiyo ekhiqiziwe.
Ukugcwaliswa ngezithombe zezikhonkwane zama-engeli amaningi: Yamukela izithombe zezikhonkwane ezingu-1–4 ezivela kunoma yimaphi ama-engeli njengolwazi olwengeziwe, ikakhulukazi ukugcwalisa uhlangothi olungabonakali esithombeni esisodwa sereferensi.
Ukukhiqiza okunezimo zombhalo nezithombe: Kuhlanganisa umyalo wombhalo nesithombe sokuqala somlingiswa ukuze kukhiqizwe ama-frame alandelayo alawulekayo endleleni, okulungele ukuboniswa kwabalingiswa kanye nokuhlolwa kwangaphambili kokuthwebula okubonakalayo.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lezingxoxo lamaphrojekthi e-AI omthombo ovulekile
Ukuqondisa kwejiyomethri okuhambisana nomzila: Kusetshenziswa i-depth estimator eqeqeshwe kusengaphambili ukulinganisa ukujula kwesithombe sokuqala sereferensi, bese isithombe sibuyiselwa emuva sibe yi-point cloud ye-3D enemibala. Ngokuqinisekisa kutholwa isikali esenza ukunyakaza okunikeziwe kuhambisane kakhulu nomzila ohlosiwe; kusetshenziswa i-point splatting kanye ne-depth buffer ukukhiqiza i-RGB guide image ne-cover mask ye-frame ngayinye endleleni. Ngasikhathi sinye, i-Plücker ray encoding imele ulwazi lwejiyomethri lwekhamera ohlosiwe, bese ihlanganiswa nama-noise latent kanye nesimo se-frame yokuqala ngaphambi kwe-patch embedding. I-cover mask imaka amaphikseli angamboziwe yi-point cloud, itshele i-generator ukuthi yiziphi izindawo okufanele izigcwalise ngokukhululeka.
I-Geometric Alignment Memory Attention (GAMA): Isithombe sokuqala nezithombe zezikhonkwane zihlukaniswa phakathi kwe-foreground futhi kulinganiselwa ijiyomethri yazo; kusetshenziswa i-Umeyama alignment ukubeka amakhamera ezikhonkwane ohlelweni olufanayo lwezixhumanisi njengomzila ohlosiwe. Ukubuka ngakunye kufakwa ikhodi nge-VAE futhi kwenezelwa ukumelwa kwemisebe ye-Plücker, bese kuhlanganiswa kube “yimemori yomlingiswa ehambisana nejiyomethri”. Ngemva kwamabhulokhi 0, 2, …, 38 e-backbone kufakwa amabhulokhi angu-20 e-memory attention, ukuze ama-video token nama-memory token enze i-joint self-attention futhi kubuyiswe ubufakazi bejiyomethri obuhambisanayo ngokwe-engeli yamanje. Igatsha le-residual lisebenzisa i-zero initialization, ngakho ekuqaleni kokuqeqeshwa aliphazamisi i-backbone eqeqeshwe kusengaphambili. Ukuhlolwa kwe-ablation kubonisa ukuthi ngemva kokususa i-GAMA, ukufana kobuwena kwehla kusuka ku-0.555 kuya ku-0.370.
Ukuqeqeshwa ngezigaba ezimbili nokwakhiwa kwedatha: Esigabeni sokuqala kuqeqeshwa amandla okulawula ikhamera kusetshenziswa ukulandelana kokunyakaza kwekhamera okungu-50,000 okwenziwe nge-UE5. I-ablation ibonisa ukuthi ngaphandle kwale datha, iphutha lomzila i-ATE liyawohloka lisuka ku-0.100 liye ku-0.359, futhi isimo se-Plücker sibalulekile ngokufanayo. Esigabeni sesibili kuqeqeshwa i-memory attention ngedatha engu-30,000 yokunikezwa kwama-anchor asebenzisa ama-engeli amaningi ku-Blender, ehlanganisa abalingiswa abangu-99 nezindawo ezingu-33, okwenza amasampula angu-80,000 esewonke. Isampula ngayinye inikezwa ngevidiyo yereferensi, i-RGB-D yokuqala, amapharamitha angaphakathi nangaphandle ekhamera, i-rendering guide kanye ne-cover mask.

Clone indawo yokugcina: git clone https://github.com/PixCtrol/WorldCast bese ungene enkombeni yayo.
Lungisa imvelo: Lungisa i-Python ≥3.10 kanye ne-CUDA GPU esekela i-bf16, bese usebenzisa pip install -r requirements.txt.
Landa ama-weight: Landa i-Wan2.1-I2V-14B-720P base ku-HuggingFace uyibeke ku-pretrained/Wan2.1-I2V-14B-720P/, bese ubeka i-base_dit.safetensors kanye ne-worldcast_bank.safetensors ku-checkpoints/.
Lungisa okokufaka: Faka isithombe esisodwa sereferensi yomlingiswa kanye nezithombe zezikhonkwane ezingu-1–4, bese ucacisa izindlela zezithombe, umyalo wombhalo kanye nomzila wekhamera ku-manifest (ifayela lohlu).
Qalisa inference: Sebenzisa python demo.py --manifest demo_case_traj5.json --model_root pretrained/Wan2.1-I2V-14B-720P --ckpt checkpoints/worldcast_bank.safetensors --init_dit_from checkpoints/base_dit.safetensors ukuze ukhiqize ividiyo.
Ukucubungula ngobuningi nokwenza ngokwezifiso: Bheka scripts/infer/infer_worldcast.py kanye ne-launch/infer/run_anchor_inference_example.sh ukuze uthole i-batch inference; imizila ingakhiqizwa nge-scripts/data/trajectories.py.

Ukuvumelana kobuwena kuma-engeli ahlukahlukene kunamandla kakhulu: Ku-CLACT benchmark eyakhiwe ngaphakathi, ukufana kobuwena kwe-ArcFace kufinyelela ku-0.555, kudlule ngokusobala ama-baseline afana ne-Uni3C, futhi ubuso abushintshi ngisho nalapho ikhamera izungeza iye ngemuva komlingiswa.
Ukunemba kokulawula ikhamera kuphezulu kakhulu: I-ATE engu-0.065, i-RTE engu-0.020 kanye ne-RRE engu-0.896 yizona ezihamba phambili phakathi kwezindlela eziqhathanisiwe, ngokusekela ukuzungeza okuphelele okufika ku-360 degrees.
Idizayini ekhethekile yezimo zama-engeli amakhulu: Ngokusesha imemori yezithombe zezikhonkwane, igcwalisa ngokusobala ulwazi “lwangemuva” olungabonakali esithombeni esisodwa, iklanyelwe ngokukhethekile ukuxazulula ukuwohloka kobuwena emizileni enama-engeli amakhulu.
Umnikelo wendlela ungaqinisekiswa: I-ablation ibonisa ukuthi ngemva kokususa i-GAMA memory attention, i-ArcFace yehla kakhulu isuka ku-0.555 iye ku-0.370; umthombo wenzuzo yokuvumelana kwama-engeli ahlukahlukene ucacile futhi awuncikile ekucabangeleni okungaqinisekisiwe.
Ukuvuleka okuphelele: Iphepha, ikhodi ye-inference nama-weight emodeli konke kuyatholakala ngaphansi kwelayisensi ye-Apache-2.0, okuyinto engavamile emsebenzini ovuleke ngokuphelele emkhakheni wokulawula ikhamera.

Iwebhusayithi yephrojekthi: https://pixctrol.github.io/WorldCast/
Indawo yokugcina ye-GitHub: https://github.com/PixCtrol/WorldCast
Indawo yamamodeli ye-HuggingFace: https://huggingface.co/ly56/worldcast
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2610.33135

Ukuboniswa komlingiswa ozungezwe yikhamera: Vumela ikhamera ijikeleze umlingiswa obonakalayo ngo-360 degrees, kuyilapho imininingwane yezingubo, ama-figurine kanye nesithixo esibonakalayo ihlala ivumelana.
Ukuhlolwa kwangaphambili kokuthwebula okubonakalayo: Ngaphambi kokuthwebula kwangempela, sebenzisa omunye umlingiswa ukuze uhlole kusengaphambili ukunyakaza kwekhamera emizileni ehleliwe, uqinisekise umphumela we-storyboard futhi unciphise izindleko zokuzama nokwenza amaphutha.
Ividiyo yabantu bedijithali enokuvumelana kwama-engeli amaningi: Khiqiza izinto ezivela kuma-engeli amaningi, njengokuphenduka, ukubuka eceleni kanye nokubuka ngemuva, ugcine “ubuso obufanayo” ukuze kusetshenziswe kubasakazi ababonakalayo nakumavidiyo okukhangisa abantu bedijithali.
Ukugqwayiza kwabalingiswa bemidlalo: Khiqiza ukugqwayiza kokuzungeza kwekhamera kwezigcawu eziphakathi noma amakhasi okubonisa abalingiswa emidlalweni, ugweme izindleko eziphindaphindwayo zokumodela nokunikezela ngama-engeli amaningi ngendlela yendabuko.
Ukugqwayiza kwemikhiqizo ye-e-commerce: Khiqiza amavidiyo okubonisa umkhiqizo ojikelezayo ukuze asetshenziswe emakhasini emininingwane nasezintweni zokukhangisa.