DuoMatching Uyasebenza

I-DuoMatching iwuhlaka lokucindezela ukukhiqizwa kwamavidiyo ngezinyathelo ezimbalwa, olwakhiwe ngokubambisana yi-MMLab yase-Chinese University of Hong Kong kanye ne-ByteDance. Ngesikhathi i-DuoMatching igcina ukuhambisana kwesikhathi ngosizo lothisha bevidiyo, yethula imodeli yesithombe emisiwe ukuze iqondise ikhwalithi yesithombe esisodwa nokuhambisana kwencazelo. Isebenzisa i-LatentBridge ukuxhuma izikhala ze-latent, futhi isebenzisa i-LVS ukukhetha amafreyimu abalulekile okuqondisa, ngaleyo ndlela ithuthukise imininingwane yesithombe nokulandela imiyalelo. I-DuoMatching ayikhulisi izindleko zokwenza inference; amaphuzu aphelele e-VBench ekukhiqizweni ngezinyathelo ezi-2 afinyelela ku-83.53.

Ukuthuthukisa ikhwalithi yesithombe esisodwa: Ithuthukisa izinkinga zokufiphala noma ukulahleka kwemininingwane efana noboya, ukwakheka kanye nezinto zokwakha ekukhiqizweni kwamavidiyo ngezinyathelo ezimbalwa.
Ukuqinisa ukuhambisana kwencazelo: Ithuthukisa ikhono lemodeli lokulandela imiyalelo, inciphise izimo zokulahleka kwezinto noma amaphutha ombala.
Ukugcina ukusebenza kwesikhathi: Ngesikhathi ithuthukisa ikhwalithi yesithombe esisodwa, igcina ngokuyisisekelo ukuhambisana phakathi kwamafreyimu namandla okunyakaza kwevidiyo.
Ukusekela ukukhiqizwa ngezinyathelo ezimbalwa: Ingasetshenziswa ekukhiqizweni kwamavidiyo ngezinyathelo ezi-2, ezi-4 nezinye, ihlangabezane nezidingo zezinhlelo zesikhathi sangempela noma ezisebenzisanayo.
Akukhulisi izindleko ze-inference: Uthisha wesithombe ubamba iqhaza ekuqeqesheni kuphela; imodeli yomfundi ngemva kokusetshenziswa ayidingi ukusebenzisa imodeli yesithombe eyengeziwe.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxhumana lamaphrojekthi e-AI omthombo ovulekile
Ukufaniswa kokusabalalisa okuhlangene nokuseceleni: Ngaphezu kwe-DMD ehlangene yendabuko, i-DuoMatching yengeza i-marginal DMD: uthisha wevidiyo unesibopho sokufanisa ukusabalalisa okuhlangene kwevidiyo yonke, agcine ikhwalithi yesikhathi nokunyakaza; uthisha wesithombe omisiwe unesibopho sokufanisa ukusabalalisa okuseceleni kwamafreyimu asampuliwe, athuthukise ikhwalithi yesithombe esisodwa nokuhambisana kwencazelo. Lezi zilahleko ezimbili ziqeqesha imodeli yomfundi ndawonye, kanti isisindo esizenzakalelayo sokulahleka okuseceleni singu-0.4.
I-LatentBridge ixhuma izikhala ze-latent zevidiyo nezithombe: Njengoba izakhiwo ze-latent ze-video VAE ne-image VAE zingahambisani, i-DuoMatching isebenzisa i-LatentBridge enamapharamitha acishe abe ngu-10.75 million ukuze imephu ngokuqondile izingcezu ze-video latent ziye esikhaleni se-latent sikathisha wesithombe. Le module iqeqeshwa kusengaphambili kusetshenziswa ukulahleka kokwakha kabusha kwe-L1, igweme izindleko eziphezulu zememori ye-GPU ezidalwa ukukhipha ku-RGB bese kufakwa ikhodi futhi, futhi yenze uthisha wesithombe akwazi ukuqondisa ngempumelelo umfundi wevidiyo.
I-LVS ithuthukisa ukusampula kokuqondisa ezingeni lamafreyimu: I-Latent Variation Sampling ihlukanisa izingxenye zevidiyo ngokwamandla oshintsho phakathi kwezingcezu ze-latent eziseduze, bese ikhetha amafreyimu okuqondisa engxenyeni ngayinye. Ngokuzenzakalelayo, kusampulwa izingcezu ze-latent ezi-4 kwividiyo ngayinye. Uma kuqhathaniswa nokusampula okungahleliwe, lokhu kunciphisa ukuqoqeka kokuqondisa ezithombeni ezimile futhi kwabela isabelomali esilinganiselwe sokuqondisa kwamafreyimu kokuqukethwe okunoshintsho olukhulu.

Lungisa indawo yokusebenza: Lungisa ikhadi lezithombe le-NVIDIA elisekela i-CUDA, bese ulungisa indawo ye-Python 3.10.
Landa futhi ufake iphrojekthi: Klona i-repository ye-DuoMatching, ufake i-PyTorch, i-FlashAttention kanye nokuncika okudingwa yiphrojekthi.
Landa amafayela emodeli: Thola izingxenye eziyisisekelo ze-Wan2.1-T2V-1.3B kanye nezisindo ezisemthethweni ze-DuoMatching, uzibeke kumafolda azenzakalelayo e-repository.
Lungisa imiyalelo: Dala ifayela lombhalo le-UTF-8 bese ubhala umyalo owodwa emgqeni ngamunye; ungasebenzisa ngokuqondile i-prompts/demos.txt esemthethweni.
Qalisa ukukhiqizwa kwevidiyo: Sebenzisa i-inference.py, ucacise ekumy command ukumiswa kwe-inference, izisindo zemodeli, ifayela lemiyalo kanye nefolda yokukhipha.
Hlola imiphumela ekhiqiziwe: Thola amavidiyo kufolda yokukhipha; isibonelo esisemthethweni sikhiqiza amafreyimu angu-21 ngokuzenzakalelayo, futhi inani lamafreyimu nembewu engahleliwe kungasethwa.

Ukuthuthuka okubonakalayo kwekhwalithi yesithombe esisodwa: Ngokuqondisa okuzimele ezingeni lamafreyimu, ithuthukisa ukulahleka kwemininingwane efana noboya, ukwakheka nezinto zokwakha ekukhiqizweni ngezinyathelo ezimbalwa.
Ukulandela imiyalelo okunembe kakhudlwana: Ngosizo lolwazi lwangaphambilini lwemodeli yesithombe, inciphisa ukulahleka kwezinto, amaphutha ombala nokuphambuka kokwakheka kwesithombe.
Akukhulisi izindleko ze-inference: Uthisha wesithombe ubamba iqhaza ekuqeqesheni kuphela; imodeli yomfundi esetshenzisiwe ayidingi izinyathelo ezengeziwe ze-inference noma ukubalwa okwengeziwe.
Ibhalansisa ikhwalithi namandla okunyakaza: Ngesikhathi ithuthukisa ikhwalithi ebonakalayo nokuhambisana kwencazelo, igcina ngokuyisisekelo ukuhambisana kwesikhathi nokusebenza kokunyakaza kwevidiyo.
Ukusebenzisa intuthuko yamamodeli ezithombe ngokuqhubekayo: Ukushintsha uthisha wesithombe onamandla kakhulu kungase kuthuthukise imodeli yomfundi, kodwa ngokuvamile kudinga ukuqeqeshwa kabusha kwe-LatentBridge ehambisanayo.

Iwebhusayithi yephrojekthi: https://johnzhan2023.github.io/DuoMatching/
I-repository ye-GitHub: https://github.com/JohnZhan2023/DuoMatching
I-repository yamamodeli e-HuggingFace: https://huggingface.co/JohnZhan/DuoMatching
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2610.03543

Ukusebenzisana kwesikhathi sangempela nokukhiqizwa kokusakaza bukhoma: Sebenzisa izici zezinyathelo ezimbalwa kanye ne-latency ephansi ukuze ukhiqize ngokushesha amavidiyo okunyakaza nezinkulumo zabantu bedijithali, abasakazi ababonakalayo kanye nokusakaza bukhoma okusebenzisanayo.
Ukukhiqizwa kokuqukethwe kokukhangisa ne-e-commerce: Thuthukisa ukunemba kokukhiqizwa kwezinto, imibala nezinto ezicacisiwe zomkhiqizo, unciphise ukungahambisani phakathi kwesithombe sesikhangiso nemiyalelo yokudala.
Ukuhlola kuqala kwemidlalo emifushane nopopayi: Khiqiza ngokushesha ama-storyboard namasampuli anyakazayo, usize abadali baqinisekise ngokwezindleko eziphansi ukunyakaza kwabalingiswa, ukwakheka kwamashothi nemiphumela ebonakalayo.
Ukukhiqizwa kokuqukethwe kwemidlalo nemihlaba ebonakalayo: Nikeza ama-prototype amavidiyo asheshayo emidlalo ephakathi, ukunyakaza kwabalingiswa nezindawo ezibonakalayo, uthuthukise ukusebenza kahle kokukhiqizwa kokuqukethwe okusebenzisanayo.
Izinsizakalo zokukhiqiza amavidiyo anokucubungula okuningi ngesikhathi esisodwa: Ngaphandle kokwengeza izindleko ze-inference, thuthukisa ikhwalithi yamamodeli anezinyathelo ezimbalwa, okulungele amapulatifomu adinga ukulinganisa isivinini sokukhiqiza, izindleko zesevisi kanye nekhwalithi yomphumela.