HOMIE Uyasebenza

I-HOMIE iwuhlaka oluvulekile lokukhiqiza amavidiyo abantu bedijithali olukhishwe yi-Hong Kong University of Science and Technology, olwakhelwe kunethiwekhi eyinhloko i-Wan2.1-T2V-14B futhi oluhlanganisa imodeli enkulu yezindlela eziningi i-Qwen3-VL. Uhlaka luphatha ngokuhlangene izici ezine: umuntu wedijithali + umkhiqizo + i-Logo + umbhalo, luxazulule ngokunembile izinselele ezine ezinkulu: ukusebenzisana komuntu nento, ukufakwa kophawu lomkhiqizo, ukwethembeka kombhalo we-OCR, nokuvumelana kwama-engeli amaningi. Ukuqeqeshwa kwemodeli kudinga izinyathelo ezingu-5.5K kuphela / cishe amahora angu-10,000 e-A100; ukunemba kwe-OCR kukhuphuke ngo-38.7% uma kuqhathaniswa nemodeli yomthombo ovulekile ehamba phambili, futhi kusekelwa ukucutshungulwa kusuka ku-480P ngekhadi elilodwa kuya ku-720P ngamakhadi amaningi, kunciphisa kakhulu umgoqo wokukhiqiza amavidiyo ezimo ezifana nokuthengisa bukhoma kwe-e-commerce nabasakazi abangokoqobo.

Ukwenza amavidiyo okusebenzisana komuntu nento abe ngawomuntu siqu: kusekelwa abantu abaningi nezinto eziningi ngesikhathi esisodwa kuvidiyo eyodwa, kugcinwe ukubukeka okungaguquki kwento ngayinye, futhi kwenziwe ukusebenzisana kwemvelo nokunengqondo, njengokubamba, ukukhombisa nokudlulisa umkhiqizo.
Ukunamathisela okunembile kwe-Logo nemiqondo engaqondakali: kusetshenziswa ikhono lokucabanga ngolimi lwemodeli enkulu yezindlela eziningi (MLLM), ama-Logo emikhiqizo neminye imiqondo engaqondakali kunamathiselwa ngokuzenzakalelayo ezintweni ezihambisana nayo ngokwencazelo, ngaphandle kokuchaza ngokusobala ubudlelwano bezindawo emiyalweni.
Ukukhiqizwa kombhalo we-OCR onokwethembeka okuphezulu: kuhlanganiswa nesithombe esiyireferensi se-OCR ukuze kuqinisekiswe ukuthi umbhalo osepaketheni lomkhiqizo noma kulebula uyafundeka kahle kuvidiyo ekhiqiziwe, awufiphali futhi awuhlangani, kuxazululwe inkinga yombhalo wevidiyo ye-AI ogcina usuyinqwaba engafundeki.
Ukukhiqizwa okuvumelanayo kwama-engeli amaningi: uma kunikezwa izithombe eziyireferensi zama-engeli amaningi zento efanayo, lapho leyo nto ijikeleza noma inyakaza kuvidiyo, ukubukeka nemininingwane yayo kuhlala kuvumelana. Lokhu kulungele izimo zokubonisa izithombe ezincane, amamodeli e-3D, onodoli nezinye izinto.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxhumana lamaphrojekthi e-AI omthombo ovulekile.
Isakhiwo sonke: i-DiT ye-Ali Wan2.1-T2V-14B isetshenziswa njengesisekelo, kwengezwe i-Qwen3-VL-2B-Thinking njengemojuli yokuqonda okubukwayo, kanye nesifaki sombhalo i-UmT5 nesifaki esibonakalayo i-VAE, kwakheke isakhiwo sokukhiqiza esinemizila emithathu ehambisanayo: umbhalo + incazelo yezindlela eziningi + okubukwayo.
Isu lokuhlanganisa i-MLLM: ngaphandle kokuphazamisa ubudlelwano bokuqala bokuhambisana phakathi kombhalo nevidiyo noma ukudala izindleko ezinkulu zokuphinda kuqondaniswe, ikhono lokucabanga ngokwencazelo lemodeli enkulu yezindlela eziningi lifakwa ku-DiT, ukuze imodeli ikwazi ukuqonda ngokuzenzakalelayo ubudlelwano obufihlekile phakathi kwezithombe eziyireferensi.
Ukuqondisa ukunaka kwakho komhlaba wonke kwezindlela eziningi (GMG): ezingqimbeni zokuzinaka ze-DiT, izici zencazelo ezikhishwe i-MLLM zihambisana emhlabeni wonke nama-token okubukwayo e-VAE. Ngokusebenzisa ukuqagela, ukuhlanganisa nokuguqulwa kwe-affine, i-token ngayinye yevidiyo ngesikhathi sokukhiqiza iyakwazi “ukubona” ulwazi lwencazelo yomhlaba wonke lwesithombe esiyireferensi.
Ukushumeka kwemodi nereferensi (MRE): ama-token ezici ze-MLLM nama-token okubukwayo e-VAE anikezwa ukushumeka kwemodi okuhlukile, izithombe eziyireferensi ezahlukene zinikezwe ukushumeka kwereferensi okuzimele, futhi ama-token ezithombe eziyireferensi zama-engeli amaningi/OCR zento efanayo ayaxhunyaniswa, kugwenywe ukudideka kolwazi.
Ukuqeqeshwa okuqhubekayo ngezigaba ezintathu: 480P yento eyodwa → 480P yezinto eziningi → 720P enencazelo ephezulu; ukuqeqeshwa kuphela ngezinyathelo ezingu-5.5K kuphela, okungaphansi kakhulu kwezinyathelo ezingamashumi ezinkulungwane ezidingwa izindlela ezifanayo.

Ukulungiswa kwemvelo: klona ikhosombe lekhodi ye-HOMIE ku-GitHub ulifake endaweni yakho, bese uqedela ukufaka konke okudingekayo kwemvelo.
Ukulanda izisindo: landa izisindo ezisemthethweni eziqeqeshwe kusengaphambili ku-HuggingFace uzibeke kufolda yemodeli ebekiwe.
Ukulungisa izinto: lungisa izithombe eziyireferensi, ezihlanganisa izithombe zabantu, izithombe zemikhiqizo, izithombe zama-Logo noma izithombe zombhalo we-OCR nezinye izinto eziningi.
Ukubhala imiyalelo: bhala imiyalelo yombhalo echaza ngokunembile izenzo zokusebenzisana phakathi kwabantu nezinto kanye nesimo esikuvidiyo okuhlosiwe.
Ukuqalisa ukucubungula: sebenzisa iskripthi sokucubungula; i-HOMIE izohlanganisa ngokuzenzakalelayo ulwazi lwezindlela eziningi oluyireferensi futhi ikhiqize ividiyo eyenzelwe wena.
Ukukhetha isinqumo: i-GPU enekhadi elilodwa ingakhiqiza ngokuqondile ividiyo enesinqumo esingu-832×480, kanti i-FSDP enamakhadi amaningi inganweba okukhiphayo okucacile kufinyelele ku-720P.

Uhlaka oluhlanganisiwe: uhlaka olulodwa luphatha imisebenzi emine emikhulu ngesikhathi esisodwa: ukusebenzisana komuntu nento, ukufakwa kwe-Logo, ukwethembeka kombhalo we-OCR, nokuvumelana kwama-engeli amaningi, ngaphandle kokushintsha amamodeli ezimweni ezahlukene.
Ukuqeqeshwa okuphumelelayo: ukuqeqeshwa kuphela ngezinyathelo ezingu-5.5K / cishe amahora angu-10,000 e-A100, izindleko ziphansi kakhulu kunezinyathelo ezingu-30–40K ezivame ukudingwa izindlela ezifanayo.
Ukuhlanganiswa kwe-MLLM ngaphandle kokulahlekelwa: ikhono lokucabanga ngokwencazelo lemodeli enkulu yezindlela eziningi lifakwa epayipini lokukhiqiza amavidiyo ngaphandle kokudela ukulawuleka kwesifaki sombhalo noma ukudala izindleko ezinkulu zokuphinda kuqondaniswe.
Ikhwalithi ehamba phambili: ukunemba kwe-OCR kukhuphuke ngo-38.7%, ukuvumelana kwama-engeli amaningi kwakhuphuka ngo-8.2%, futhi ocwaningweni lwabasebenzisi abangu-40, u-64.7% bavotele ikhwalithi iyonke njengeyokuqala.
Ukucubungula okuguquguqukayo: kusekelwa ukucubungula kusuka ku-480P ngekhadi elilodwa kuya ku-720P ngamakhadi amaningi, futhi kungakhipha amavidiyo aqondile angu-1280×720, ahambisana ngqo nokuthengisa bukhoma nokukhiqizwa kwamavidiyo amafushane.

Iwebhusayithi yephrojekthi: https://yiyangcai.github.io/homie-page.github.io
Ikhosombe le-GitHub: https://github.com/YIYANGCAI/HOMIE
Ikhosombe lemodeli ku-HuggingFace: https://huggingface.co/yychai/homie-r2v-wan2.1
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2607.18217

Ukuthengisa bukhoma kwe-e-commerce: khiqiza amavidiyo okuthengisa lapho umuntu wedijithali othile ephethe umkhiqizo othile futhi enza isenzo esithile, kusekelwe okukhiphayo kwevidiyo eqondile engu-1280×720, okuhambisana ngqo nezinkundla zamavidiyo amafushane.
Ukukhiqizwa kwezikhangiso zemikhiqizo: namathisela ngokuzenzakalelayo i-Logo yomkhiqizo emkhiqizweni ohambisana nayo ngokwencazelo, ngaphandle kokucacisa ngokusobala ubudlelwano bezindawo emiyalweni.
Abasakazi abangokoqobo / abantu bedijithali: gcina ubuwena obungaguquki bomuntu wedijithali ngesikhathi esebenzisana ngokwemvelo nemikhiqizo eminingi.
Ukuboniswa komkhiqizo ngama-engeli amaningi: uma kunikezwa izithombe eziyireferensi zama-engeli amaningi zesithombe esincane, imodeli ye-3D noma unodoli, khiqiza ividiyo yokuwujikelezisa ukuze ukubukeka kuhlale kufana kuwo wonke ama-engeli.
Ukwethembeka kombhalo osepaketheni: kuhlanganiswe nesithombe esiyireferensi se-OCR ukuze kuqinisekiswe ukuthi umbhalo osepaketheni lomkhiqizo noma kulebula uyafundeka kahle kuvidiyo, kulungele izimo zemikhiqizo yokudla, izimonyo, izithasiselo zezempilo neminye imikhiqizo edinga ukubonisa izithako noma imiyalelo.