InstructAV2AV – Imodeli yokuhlela okuhlangene kwevidiyo nomsindo evulwe yi-Zhiyuan ngokubambisana ne-Peking University Uyasebenza

I-InstructAV2AV iyimodeli yokuhlela okuhlangene kwevidiyo nomsindo eyethulwe ngokubambisana yi-Beijing Academy of Artificial Intelligence ne-Peking University. Umsebenzisi udinga kuphela umyalo owodwa wolimi lwemvelo ukuze ahlele ngesikhathi esisodwa izithombe zevidiyo nomsindo ohambisana nazo ngesikhathi sokukhiqiza kusukela ekuqaleni kuye ekugcineni, ngaphandle kokusebenzisa imaski ngesandla noma ukucubungula ngezinyathelo. Imodeli isekela ukuguqulwa kwemigqa yenkulumo, ukushintshwa kwabantu, ukufakwa nokususwa kwezinto, kuyilapho igcina umsindo wangemuva nowemvelo ungashintshi futhi igcina ukuvumelana kwesikhathi phakathi kwesithombe nomsindo.

Ukuguqulwa kwenkulumo kugcinwa ubuwena: kugcinwa ukubukeka komuntu, kushintshwe kuphela okuqukethwe akushoyo.
Ukushintshwa kwezibonelo zevidiyo nomsindo: kushintshwa umuntu othile osesithombeni kanye nomsindo nemigqa yakhe yenkulumo.
Ukufakwa kwesibonelo: kwengezwa into entsha esithombeni futhi kukhiqizwe ngokuzenzakalelayo umsindo ohambisana nayo.
Ukususwa kwesibonelo: kususwa into ethile kanye nomsindo wayo ohambisana nayo kuthrekhi yomsindo.
Ukuhlela okuhlanganisa izici: ukubukeka komuntu, okuqukethwe kwenkulumo nombala wezwi kungashintshwa ngokuzimela futhi kuhlanganiswe ngokukhululekile.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lezingxoxo lamaphrojekthi e-AI avulekile
Ukuhlanganiswa kweziguquguquko ezifihlekile zomthombo: kuhlanganiswa iziguquguquko ezifihlekile zevidiyo nomsindo womthombo nomsindo ongahleliwe ngobukhulu beziteshi, ukuze okuqukethwe komthombo kube yisimo sesakhiwo kuyo yonke inqubo yokususa umsindo. Lokhu kuphoqa imodeli ukuthi igcine ingemuva, izinto ezingahlosiwe nomsindo wemvelo, kugwenywe “ukuguqula into eyodwa bese kudwetshwa kabusha wonke umhlaba”.
Ukunaka okulawulwa yi-SIGA: I-Source-Instruction Gated Attention yenza ithokheni ngayinye ixhumane ngesikhathi esisodwa nezici zomthombo nezici zomyalo. Ngamasango athambile afundekayo, inquma endaweni ngayinye ukuthi yini okufanele iguqulwe nokuthi yini okufanele igcinwe, ngaleyo ndlela ilinganise ngokushintshashintshayo ukugcinwa kokuqukethwe nokusetshenziswa komyalo.
Isakhiwo sokuxhumana kwemifudlana emibili: kusetshenziswa i-Ovi symmetric dual-stream diffusion Transformer. Amagatsha evidiyo nawomsindo aqala akhe amamodeli okuzinaka ngokwahlukana, bese eshintshana ngezici ngokunaka okunqamula izindlela ezimbili, ukuze ukunyakaza okubonakalayo nezenzakalo zomsindo kuphoqane futhi kuhlale kuvumelana.
Ukuqeqeshwa ngezigaba ezimbili: kuvalwa ukunaka okunqamula izindlela ukuze kuqeqeshwe ngokwahlukana ikhono lokuhlela lendlela ngayinye, bese kubuyiselwa isakhiwo esiphelele ukuze kwenziwe ukulungiswa okuhlangene, kufundwe ubudlelwano obunemininingwane phakathi kwesithombe nomsindo.
Ulayini wedatha ozenzakalelayo: isethi yedatha i-InsAVE-80K yakhiwa ngezigaba ezintathu, okuwukuhlunga izinto, ukuhlanganisa okuhlosiwe kokuhlela, kanye nokuhlola okuzenzakalelayo ngezindlela ezinhlanu nokubuyekezwa komuntu. Lokhu kuxazulula ukushoda kwedatha yokuqeqesha ehambisanayo.

Klonisha ikhodi: klonisha inqolobane yekhodi ye-InstructAV2AV ku-GitHub uyibeke kukhompyutha yangakini.
Lungisa indawo: landela iziqondiso zenqolobane ukuze ulungise indawo ye-Python futhi ufake okuncikile okudingekayo.
Landa izisindo: landa izisindo zemodeli ye-InstructAV2AV ku-Hugging Face.
Lungisa izinto: lungisa ividiyo enomsindo ezosetshenziswa njengokufakwayo.
Faka umyalo: faka umyalo owodwa wokuhlela ngolimi lwemvelo ochaza ofuna ukukuguqula.
Qalisa ukuqagela: qalisa iskripthi sokuqagela bese ulinda imodeli ikhiqize ividiyo nomsindo ohleliwe.
Gcina umphumela: buka umphumela wokuphumayo bese ugcina ifayela levidiyo nomsindo elihleliwe.

Iwebhusayithi yephrojekthi: https://hjzheng.net/projects/InstructAV2AV/
Inqolobane ye-GitHub: https://github.com/suimuc/InstructAV2AV
Inqolobane yemodeli ku-Hugging Face: https://huggingface.co/suimu/InstructAV2AV
Iphepha lobuchwepheshe ku-arXiv: https://arxiv.org/pdf/2605.18467

Ukuhlelwa kwamafilimu nethelevishini: ochwepheshe bokuhlela basebenzisa yona ukushintsha imigqa yabalingisi nokuvumelanisa ukunyakaza kwezindebe, behlise izindleko zokuphinda kuthwetshulwe.
Ukukhiqizwa kwamavidiyo amafushane: abadali bamavidiyo amafushane basebenzisa umyalo owodwa ukuguqula abantu nomsindo ezintweni zabo, bakhulise ukusebenza kahle kokudala.
Abantu ababonakalayo: amaqembu aphatha abantu ababonakalayo ayisebenzisa ukulungisa ukubukeka, umbala wezwi nokuqukethwe kwenkulumo yabantu bedijithali.
Ukuklama izikhangiso: ochwepheshe bezikhangiso bayisebenzisa ukukhiqiza ngobuningi amavidiyo okukhangisa anezinguqulo ezihlukene zabantu nemigqa yenkulumo.
Ukukhiqizwa kokuqukethwe okusebenzisanayo: abasebenzi bemidlalo nezemfundo bayisebenzisa ukukhiqiza ngamandla izinto zemultimedia ezihambisanayo ngesithombe nomsindo.