PixelUMM Uyasebenza

I-PixelUMM iyimodeli ehlanganisiwe ye-multimodal engenawo ama-encoder, eyethulwe ngokubambisana yi-NVIDIA neNyuvesi yase-Waterloo. Impumelelo yayo eyinhloko ukususa ngokuphelele i-VAE ne-visual encoder, bese isebenzisa i-Decoder-only Transformer eyodwa ukufunda nokubhala izithombe namavidiyo ngqo endaweni yama-pixel aluhlaza. Ukuqonda nokukhiqiza kwabelana nge-decoder efanayo ne-self-attention efanayo. Izithombe zihlukaniswa zibe ama-patch angu-16×16, kanti amavidiyo ahlukaniswa abe ama-tubelet anozimele abangu-4; ikhodi isikhishiwe njengomthombo ovulekile.

Umbhalo uye kuvidiyo (T2V): Isekela ukukhiqiza amavidiyo afinyelela ku-832×464, anozimele abangu-96, nangu-24fps, anobude bemizuzwana engu-4, ikhwalithi yawo iqhathaniswa namamodeli amakhulu e-video diffusion.
Umbhalo uye esithombeni (T2I): Ikhiqiza izithombe ngqo ngaphakathi kwemodeli efanayo, ngaphandle kwegatsha lesithombe noma i-decoder ehlukile.
Ukuqonda izithombe: Ingenza imisebenzi engu-21 yokuqonda izithombe, efaka i-VQA, i-OCR, amashadi namadokhumenti, futhi ifinyelela ezingeni elijwayelekile le-VLM enama-parameter angu-8B.
Ukuqonda amavidiyo: Isekela izilinganiso zemibuzo nezimpendulo zamavidiyo ezifana ne-MVBench ne-Video-MME, inikeze izindlela ezimbili zokufaka: i-dense ne-sparse.
Isithombe siye kuvidiyo (I2V): Ngokufaka isithombe esihlanzekile sohlaka lokuqala njengombandela ku-expert yokuqonda, ingakhiqiza amavidiyo alandelayo ngaphandle kokufaka ikhodi yohlaka lombandela nge-VAE.
Ukuhlela amavidiyo: Isebenzisa ividiyo efakiwe njengombandela obonakalayo, futhi yenza imisebenzi yokuhlela ngomshini wemibandela yomongo we-multimodal.

Ukwakhiwa okungenawo ama-encoder: I-PixelUMM isusa ngokuphelele i-VAE ne-visual encoder kumamodeli ajwayelekile ahlanganisiwe. Izithombe zihlukaniswa zibe ama-patch angu-16×16, amavidiyo ahlukaniswa abe ama-tubelet anozimele abangu-4, bese kufakwa ama-pixel e-RGB aluhlaza ngqo. Okuphumayo nakho kubuyiselwa ngqo kuma-pixel yi-linear head, ngaphandle kokulahleka kokufaka nokukhipha ikhodi endaweni ye-latent, ngaleyo ndlela kufezwe ukufunda nokubhala ama-pixel ngqo kusukela ekuqaleni kuya ekugcineni.
Isakhiwo se-MoT sokwabelana nge-attention: Imodeli isebenzisa i-Qwen3-8B njengesisekelo sokwakha i-Mixture-of-Transformers. I-expert yokuqonda ne-expert yokukhiqiza ngayinye ine-FFN yayo ezimele, kodwa yabelana ngesethi efanayo ye-self-attention. Lokhu kuvumela umbhalo, ama-pixel ahlanzekile nama-pixel anomsindo ukuthi axhumane endaweni efanayo ye-attention, kuhlanganiswe ukuqonda nokukhiqiza ngaphakathi kwemodeli eyodwa.
Ukuqeqeshwa kwe-diffusion endaweni yama-pixel: Imisebenzi yokukhiqiza iqeqeshwa ngqo endaweni yama-pixel ngendlela efana ne-flow matching. Ukuhlolwa kokususa izingxenye kubonisa ukuthi ubukhulu be-gradient endaweni yama-pixel nasendaweni ye-VAE bucishe bufane, kodwa ukuqeqeshwa endaweni yama-pixel kuzinze kakhulu. Ukuhlolwa kwezinga lokucindezela kuqinisekisa ukuthi ama-patch angu-16×16 nama-tubelet anozimele abangu-4 kuyibhalansi engcono kakhulu phakathi kobunzima bokufunda nokusebenza kahle.
Indlela yokufaka imibandela ye-multimodal: Imibandela “ehlanzekile” efana nombandela wombhalo, isithombe sereferensi nevidiyo efakiwe idlula ku-expert yokuqonda, kanti ama-token anomsindo azokhiqizwa adlula ku-expert yokukhiqiza. Kokubili kuhlanganiswa ku-attention eyabiwe. Lokhu kwandisa indlela ye-SenseNova-U1 yokubeka imibandela engenayo i-encoder iye ekukhiqizeni ividiyo ngesithombe nasekuhleleni amavidiyo. Ukuhlolwa kokususa izingxenye kubonisa ukuthi lo mklamo awunciphisi ikhono lokuqonda, futhi uthuthukisa izilinganiso ezine zamavidiyo ngamaphuzu angu-0.89–3.49.
Ukuhlolwa kwe-linear output head ne-convolution head: I-linear output head elula ezenzakalelayo idala ama-artifact egridi anobukhulu obungu-16 pixels. Ukuqhathanisa ne-Wan-style upsampling convolution head ne-PixelShuffle head kubonisa ukuthi i-convolution head ingasusa lawa ma-artifact. I-PixelShuffle ehamba kuqala ngesikhathi iyibhalansi engcono kakhulu phakathi kokulahleka kokuqeqesha nokunciphisa ama-artifact, kodwa ukushintsha i-head kudinga ukuqeqeshwa kabusha.

Clone indawo yokugcina: git clone https://github.com/nv-tlabs/PixelUMM, bese ulungisa indawo ye-CUDA ngokwe-ENVIRONMENT.md.
Landa ama-weight: Ku-Hugging Face landa i-checkpoint ezenzakalelayo S8-F22-R05 ku-nvidia/PixelUMM, bese ngokwe-CHECKPOINT.md ulanda amafayela okumisa ne-tokenizer e-Qwen3-8B.
Setha izindlela: Lungisa iziguquguquko zemvelo ezintathu, i-PIXELUMM_CKPT, i-PIXELUMM_QWEN_DIR ne-PIXELUMM_OUTPUT, ukuze zikhombe ezindleleni eziphelele zasendaweni.
Hlola i-checkpoint: Qala usebenzise i-check_checkpoint.py ukuhlola ukuphelela kwama-weight nokuhambisana kwama-tensor.
Faka isicelo semvume yokuhlolwa kokuphepha: Ku-Hugging Face, faka isicelo semvume yokufinyelela ku-nvidia/Cosmos-1.0-Guardrail bese ungena ngemvume.
Qalisa i-inference: Ku-inference.py khetha ngokwehlukana i---task t2i / t2v / image-vlm / video-vlm, bese usebenzisa i-prompt, ukulungiswa kwesithombe, inani lozimele nezinye iz参数 ukuze ukhiqize izithombe noma amavidiyo, noma ukhiphe imiphumela yokuqonda.
Ukukhiqiza ngobuningi (ongakukhetha): Sebenzisa i-inference_batch.py kanye nefayela le-YAML lama-prompt ukuze usebenzise imisebenzi ngobuningi.
Ukuqeqeshwa kokuhlola (ongakukhetha): Ngokwe-TRAIN.md landa i-toy dataset, bese usebenzisa i-train_toy.py ukuqhuba isibonelo sokuqeqesha imisebenzi emine phezu kwe-F22 checkpoint.

Ukwakhiwa okungenawo nhlobo ama-encoder: Kususa i-VAE ne-visual encoder, kufundwe futhi kubhalwe ngqo endaweni yama-pixel aluhlaza, kuqedwe ukulahleka kolwazi nama-artifact okubangelwa ukufaka nokukhipha ikhodi endaweni ye-latent.
Ukuqonda nokukhiqiza okuhlanganisiwe: I-Decoder-only Transformer eyodwa yenza imisebenzi emine ngesikhathi esisodwa: umbhalo uye esithombeni, umbhalo uye kuvidiyo, ukuqonda izithombe nokuqonda amavidiyo. I-expert yokuqonda ne-expert yokukhiqiza yabelana nge-self-attention efanayo.
Isakhiwo esilula, inference eqondile: Okuphumayo kubuyiselwa ngqo kuma-pixel yi-linear head elula, ngaphandle kwe-decoder eyengeziwe, okwenza uchungechunge lokukhiqiza lube mfushane.
Ukufaka imibandela okuguquguqukayo nokusebenza kahle: Izimo ezihlanzekile ezifana nezithombe zereferensi namavidiyo afakiwe zidlula ku-expert yokuqonda kuphela, okwenza ukukhiqiza ividiyo ngesithombe nokuhlela amavidiyo kwenzeke. Ukuhlolwa kokususa izingxenye kufakazela ukuthi lo mklamo awunciphisi ikhono lokuqonda.
Imithetho yokuqeqesha ecacile futhi enwebekayo: Imodeli engu-8B idinga cishe ingxenye yesithathu yezinyathelo zokuqeqesha ezidingwa imodeli engu-1.7B ukuze ifinyelele ekulahlekeni okufanayo, okubonisa inzuzo ecacile yokukhulisa isikali.

Iwebhusayithi yephrojekthi: https://nv-tlabs.github.io/PixelUMM/
Indawo yokugcina ye-GitHub: https://github.com/nv-tlabs/PixelUMM
Indawo yamamodeli ye-Hugging Face: https://huggingface.co/nvidia/PixelUMM
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2609.38597

Ukudala okuqukethwe kwamavidiyo amafushane: Faka umusho owodwa bese ukhiqiza ividiyo yemizuzwana engu-4 enama-24fps, efanele ukukhiqiza ngokushesha izinto zokuxhumana ezinkundleni zokuxhumana.
Ukukhiqiza ividiyo ngesithombe nokuhlela amavidiyo: Sebenzisa uhlaka lokuqala noma ividiyo yoqobo njengombandela ukuze ukhiqize noma uguqule okuqukethwe, okufanele ukubonisa imikhiqizo ye-e-commerce nokulungiselela amafilimu.
Ukuqonda namaphendulo kumavidiyo amade: Chaza, finyeza futhi wenze ukucabanga kwesikhathi ngevidiyo efakiwe, okulungele ukuhlolwa kokuqukethwe, ukusesha izinto kanye nezifinyezo zemihlangano.
Ukuhlaziya amadokhumenti namashadi: Imiphumela ye-OCRBench engu-78.0 ne-DocVQA engu-90.4 ingasekela ukubonwa kwamarisidi, ukuphendula imibuzo ngemibiko nezinye izimo zobuhlakani bamadokhumenti ezinkampanini.