EvaOmni Uyasebenza

I-EvaOmni iyibhentshimakhi yokuhlola amamodeli amakhulu anezindlela zonke, ekhishwe njenge-open source yi-Shanghai Jiao Tong University, Fudan University, Tencent YouTu nabanye. Iqukethe amavidiyo angempela angu-392, imibuzo nezimpendulo engu-1,027 enelebuli ngabantu, ihlanganise amakhono angu-15 namazinga okucabanga angu-3. I-EvaOmni idinga ukuthi kunikezwe impendulo kanye nezikhawu zesikhathi zobufakazi bomsindo nevidiyo, bese ikala ngemethrikhi ezintathu: ukunemba kwempendulo, ukumbozwa kobufakazi, kanye nokutholakala kwezindlela ezimbili. Ekuhlolweni kwamamodeli angu-24, i-Gemini3.7-Flash ibe namandla kakhulu ngokunemba okungu-79.4%, kodwa ukutholakala kwezindlela ezimbili kube ngu-35.1% kuphela, umehluko wamaphesenti angu-59.2 uma kuqhathaniswa nochwepheshe babantu abangu-94.3%. Lokhu kuveza ubuthakathaka bokuphendula kahle ngaphandle kokukwazi ukusho isisekelo sempendulo.

Ukuhlola izimpendulo: Kuhlolwa ukuthi impendulo yemodeli emibuzweni yokukhetha esekelwe ekuboneni nasekuzweni ilungile yini, kunikezwe ukunemba i-Acc.
Ukulebula nokubuyisa ubufakazi: Umbuzo ngamunye unezikhawu zomsindo nevidiyo ezisekela impendulo ezilebulwe ngabantu, kanti imodeli kufanele ibuyise izikhawu zobufakazi bezindlela ezimbili kanye nempendulo.
Isikolo sokumbozwa kobufakazi (C.G.): Kubalwa ingxenye yezigaba zobufakazi ezilebulwe ngabantu ezishaywa yizikhawu ezibikezelwe yimodeli, kukalwe ukuthi imodeli ithole izinkomba ezisekelayo ezingaki.
Ukutholakala kwebhodlela lezindlela ezimbili (B.G.): Kubalwa i-temporal IoU yomsindo nevidiyo ngokwehlukana, bese kusetshenziswa indlela ebuthaka kakhulu ukunquma isikolo, kudingeke ukuthi zombili izindlela zitholakale ngokunembile.
Ukuhlukaniswa kwamakhono ngamazinga amathathu: Imisebenzi ihlukaniswa yaba ukuthola nokuthola indawo, ukuqonda okuyisisekelo, nokucabanga okuyinkimbinkimbi, kuhlolwe ububanzi bemodeli kusukela ekuqondaniseni iphuzu elilodwa kuya ekucabangeni ngezinyathelo eziningi.
Ukuhlunga izindlela ezinqamulelayo: Imibuzo ihlolwa kusetshenziswa okokufaka kombhalo, umsindo, nokubonwayo ngokwehlukana, kususwe amasampula angaphendulwa ngokuthembela endleleni eyodwa noma olwazini oluvamile lombhalo kuphela.
Ukuhlola amamodeli ngokuvundlile: Kunikezwa ibhentshimakhi efanayo kanye nebhodi lamamodeli anezindlela zonke angu-24, okusekela ukuqhathaniswa nokukhethwa kwamamodeli anesisindo esivulekile namamodeli obunikazi.

Landela i-WeChat uphendule ngokuthi “开源” ukuze ujoyine iqembu lamaphrojekthi e-AI e-open source
Isakhiwo sedatha esinobufakazi: I-EvaOmni ilebula ngesikhathi esisodwa izikhawu zobufakazi bomsindo nevidiyo zombuzo ngamunye, futhi inikeze iqoqo lezinketho eziphazamisayo eziklanywe ngokusekelwe kokuqukethwe kwevidiyo. Ngakho okukhipha imodeli kuchazwa njenge-tuple enamathathu: inketho, iqoqo lezitembu zesikhathi zomsindo, neqoqo lezitembu zesikhathi zevidiyo. Uhlelo lokuhlola lungabheka ngokwehlukana ukunemba kwempendulo nekhwalithi yokutholakala kobufakazi.
Uchungechunge lokwakha idatha ngezigaba ezine: Ukukhiqizwa kwedatha kudlula ezigabeni ezine: ukukhetha ividiyo, ukulebula ngabantu, ukubuyekeza ngokuphambana, nokuhlunga izindlela ezinqamulelayo. Kuqala kususwe amavidiyo anemibhalo engezwe ngokuqinile noma agcwele umbhalo, ukuze izinkomba zokubona nokuzwa zibe zangempela futhi zisebenze. Ochwepheshe babhala imibuzo, bakhe izinketho eziphazamisayo ezixhumene nokuqukethwe kwevidiyo, bese belebula izikhawu zobufakazi bezindlela zombili. Abanye abalebuli babuyekeza ngokuphambana ukuhlukaniswa kwekhono, ikhwalithi yobufakazi, nezinkinga zezindlela ezinqamulelayo; amasampula aphikisanayo abuyekezwa ngokuzimela ochwepheshe ababili abengeziwe. Imodeli iphendula ngokuhlukana ngokokufaka kombhalo kuphela, umsindo kuphela, nokubonwayo kuphela; yonke imibuzo engaphendulwa kahle ngendlela eyodwa ibuyiselwa ukuze ilungiswe futhi ilebulwe kabusha, kuvinjelwe ezingeni ledatha indlela “yokudlula ukucabanga okuhlangene kokubona nokuzwa”.
Uhlelo lokukala oluhlanganisa ukumbozwa nokutholakala: Isikolo sokumbozwa kobufakazi i-C.G. sibala ukuthi isigaba ngasinye sobufakazi esilebulwe ngabantu sinokudlulela kwesikhathi esihle yini nanoma yisiphi isikhawu esibikezelwe, sikale ukuthi ubufakazi butholakele yini. Isikolo sokutholakala kwebhodlela i-B.G. sihlanganisa zonke izikhawu ezibikezelwe nezilebulwe ngaphakathi kwendlela ngayinye, sibale ukugqagqana nge-temporal IoU, bese sithatha i-IoU ebuthaka phakathi komsindo nevidiyo njengesikolo sokugcina. Zombili izikolo ziphindaphindwa ngesikolo sokunemba kwempendulo; impendulo engalungile ithola uziro. Lo mklamo wenza ukuthi ukubikezela okubanzi kuthole kuphela isikolo sokumbozwa, kuyilapho imingcele enembile inikela esikolweni sokutholakala; futhi usebenzisa indlela ebuthaka kakhulu ukunquma isikolo, uvimbele indlela enamandla ukuthi ifihle ubuthakathaka yenye.
Ukuhlelwa kwamakhono ngamazinga amathathu: Idathasethi ihlelwe yaba amazinga amathathu ngokobunzima bokucabanga nobobufakazi. I-Level 1, ukuthola nokuthola indawo, ihlola ukuqondanisa kwesikhathi, komcimbi, nokwebhizinisi phakathi komsindo nevidiyo. I-Level 2, ukuqonda okuyisisekelo, idinga ukuhlanganisa izinkomba ezihambisanayo zokubona nokuzwa ukuze kuqondwe imicimbi, imizwa, nezinhloso. I-Level 3, ukucabanga okuyinkimbinkimbi, idinga ukuhlanganisa ubufakazi obusabalale ezikhathini ezahlukene ukuze kuqedwe isinqumo esinezinyathelo eziningi. Njengoba izinga likhuphuka, inani lobufakazi nesikhathi sokumbozwa kombuzo ngamunye kwanda kakhulu, ngaleyo ndlela kutholakale izindawo ezibuthaka zemodeli ekutholeni, ekuqondeni, nasekucabangeni izinyathelo eziningi zesikhathi eside.

Faka iklayenti le-ModelScope: Sebenzisa i-pip install -U modelscope ukuze ufake noma uthuthukise i-ModelScope SDK.
Landa idathasethi: Sebenzisa i-dataset_snapshot_download futhi ucacise i-ID yedathasethi ethi VisionXLab/EvaOmni-Dataset ukuze ulande idatha kufolda yasendaweni.
Lungisa okokufaka kwemodeli: Lungisa ividiyo ngokwephrothokholi yokuhlola futhi ulungiselele nokuqukethwe komsindo okuhambisanayo.
Xhuma imodeli ezohlolwa: Landela imiyalelo esendaweni yokugcina ye-GitHub ukuze uxhume imodeli yakho enezindlela zonke, usebenzise umyalo ofanayo ukuze imodeli ikhiphe i-tuple enezinketho, izitembu zesikhathi zomsindo, nezitembu zesikhathi zevidiyo.
Qalisa ukuhlola: Biza uhlaka lokuhlola ukuze kwenziwe ukucabanga ngobuningi kuyo yonke imibuzo nezimpendulo engu-1,027, uhlelo lube seluzihlola ngokuzenzakalelayo izimpendulo nokufana kwezikhawu.
Buka imiphumela: Funda amamethrikhi amathathu i-Acc., C.G., ne-B.G. kanye namaphuzu okuhlukaniswa kwamakhono ngamazinga amathathu, uqhathanise nebhodi ukuze uthole ubuthakathaka bemodeli.

Ukutholakala kobufakazi kufakwa esikolweni: Imodeli kufanele ibuyise izikhawu zesikhathi zobufakazi bomsindo nevidiyo, kuhlukaniswe ukuqagela impendulo nokuba nesisekelo esiqinisekisekayo.
Ikhwalithi yobufakazi ikalwa ngamamethrikhi amabili: I-C.G. ikala ukuthi izinkomba zitholwe zonke yini, kanti i-B.G. isebenzisa i-temporal IoU yendlela ebuthaka kakhulu ukunquma isikolo, ivimbele indlela enamandla ukuthi ifihle ubuthakathaka yenye.
Ukuhlukaniswa kwamakhono ngamazinga amathathu kunemininingwane: Kusukela ekutholeni nasekutholeni indawo, ekuqondeni okuyisisekelo, kuya ekucabangeni okuyinkimbinkimbi ngezinyathelo eziningi, kutholakala ubuthakathaka obuthile bemodeli ekujuleni kokucabanga okuhlukene.
Ikhwalithi ephezulu yokulebula ngabantu: Imibuzo ibhalwa ochwepheshe, ibuyekezwe ngokuphambana, futhi amasampula aphikisanayo abuyekezwe ochwepheshe ababili, okwenza idatha ithembeke kakhulu kunamabhentshimakhi afanayo akhiwe ngokuzenzakalelayo.
Indlela eqinile yokuhlunga izindlela ezinqamulelayo: Imibuzo ihlolwa ngokokufaka kombhalo, umsindo, nokubonwayo ngokwehlukana, kususwe amasampula angaphendulwa ngokudlula ukucabanga okuhlangene kokubona nokuzwa, kuqinisekiswe ubulungiswa bokuhlola.
Uveza igebe langempela lamakhono: Ukuhlolwa kubonisa ukuthi umehluko wokunemba kwempendulo phakathi kwemodeli enamandla kakhulu nabantu ungamaphesenti angu-17 kuphela, kodwa umehluko wokutholakala kwezindlela ezimbili ungamaphesenti angu-59.2, okuveza ubuthakathaka bangempela bamamodeli akhona.

Iwebhusayithi esemthethweni: https://xiaoxing2001.github.io/EvaOmni.github.io/
Indawo yokugcina ye-GitHub: https://github.com/VisionXLab/EvaOmni
Ilabhulali yamamodeli ye-HuggingFace: https://huggingface.co/datasets/VisionXLab/EvaOmni-Dataset
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2610.28444

Ukuthuthukiswa nokuhlolwa kwamamodeli anezindlela zonke: Amaqembu amamodeli angasebenzisa i-Acc./C.G./B.G. lapho ephindaphinda i-Omni LLM ukuze athole igebe lamakhono lapho “impendulo ithuthuka kodwa ukutholakala kobufakazi kungalandeli”, aqondise izindlela zokuthuthukisa ukuqeqeshwa.
Ukukhetha nokwamukela i-AI yezinga lebhizinisi: Ezindaweni ezifana nezezimali nezinsizakalo zikahulumeni, lapho imodeli kufanele “inikeze isisekelo esingaqinisekiswa”, abathengi bangasebenzisa isikolo se-B.G. sokutholakala kwebhodlela ukukhetha amamodeli athembekile ngezindlela zombili.
Ukuhlolwa nokuveza umsuka kokuqukethwe kwevidiyo: Izinhlelo zokuhlola okuqukethwe zidinga ukuthi imodeli iveze ukuthi okuqukethwe okwephula imithetho kuvela “kukuphi esikrinini noma kuyiphi ingxenye yomsindo”; okukhipha izikhawu zesikhathi kwe-EvaOmni namamethrikhi okutholakala kungasetshenziswa ngokuqondile njengezindinganiso zokwamukela.
Imikhiqizo yemibuzo nezimpendulo kwezemfundo nakumafilimu: Abasizi bokufunda abanamavidiyo nemikhiqizo echaza amafilimu kudingeka bacaphune izingxenye zokuqala ukuze baphendule, futhi le bhentshimakhi ingasetshenziswa ukuhlola ukuthi imodeli ingazithola ngokunembile yini izigcawu nemigqa efanele.
Ucwaningo lwezemfundo nokwakhiwa kwamabhodi: Abacwaningi bangasebenzisa amavidiyo angu-392 aneleyibheli lobufakazi, akhishwe njenge-open source, ukuze benze olunye ucwaningo ekucabangeni kokubona nokuzwa, ekutholeni indawo ngesikhathi, nasekucabangeni ngezinyathelo eziningi, noma bawafake ohlelweni lokuhlola oluhlanganisa amabhentshimakhi amaningi.