JoyAI-Video-Edit Uyasebenza

I-JoyAI-Video-Edit iyimodeli ka-JD.com eyakhiwe futhi yavulwa njengomthombo ovulekile yokuhlela amavidiyo ngokusakaza ngesikhathi sangempela. Isekelwe ekwakhiweni kwe-autoregressive diffusion enamapharamitha angu-16B, ifinyelela isivinini sokucubungula esingu-30FPS ngokulungiswa kwe-720P futhi isekele ukuhlela okusakazwayo okuzinzile kwanoma ibuphi ubude bevidiyo. Abasebenzisi bangasebenzisa imiyalelo yolimi lwemvelo ukuze baguqule abantu, izigcawu, izitayela nezinto ngesikhathi ividiyo idlala, ngaphandle kokulinda ukuthi ividiyo ephelele ikhiqizwe. Ekuhlolweni kwe-OpenVE-Bench, imodeli idlula zonke izindlela zokuhlela ezisakazwayo futhi ihola kuzo zonke izinkomba, kanti iphinde inikeze indlela yokukhiqiza idatha ngezinga elikhulu yobuhlakani obugxilile emzimbeni.

Ukuhlela okusakazwayo ngesikhathi sangempela: hlela ngokushesha lapho amafreyimu evidiyo efika, ngaphandle kokwazi kusengaphambili ukulandelana okuphelele.
Ukulawulwa ngemiyalelo evulekile: sebenzisa ukuguqulwa kwesitayela somhlaba wonke, ukwengeza, ukususa noma ukuguqula izinto ezithile, ukushintsha ingemuva, ukulungisa ukunyakaza nokuhlela okuqondiswa yisithombe esiyireferensi.
Ukucubungula kwanoma ibuphi ubude: yeqa imikhawulo yezinga lamasekhondi noma amaminithi, futhi uqhubeke nokuhlela ngokuzinzile ngenkathi ividiyo idlala.
Ukusatshalaliswa okunomthamo omkhulu ku-720P: i-pipeline ephelele ifinyelela ku-30.19 FPS ngokulungiswa okungu-720×1280.
Amakhono okuhlela anezinhlangothi eziningi: hlanganisa ukushintsha izingubo zabantu, ukuguqula izigcawu, ukuguqula izitayela, ukushintsha izinto nokuhlela imibhalo engezansi.

Landela i-WeChat bese uphendula ngokuthi “umthombo ovulekile” ukuze ujoyine iqembu lezingxoxo lamaphrojekthi e-AI omthombo ovulekile.
Isifaki khodi esinemibandela se-MLLM: sebenzisa imodeli yolimi enkulu enemodi eziningi ukuze kufakwe imiyalelo yolimi lwemvelo njengemibandela yokuhlela, ngaleyo ndlela kunikezwe ukulawulwa kwesemantiki okuvulekile.
I-Causal video VAE: sebenzisa i-variational autoencoder enesakhiwo sesikhathi esiyimbangela ukuze ucindezele futhi wakhe kabusha amafreyimu evidiyo, uqinisekise ukuvumelana kwesikhathi ezimweni zokusakaza.
I-backbone ye-MMDiT enamapharamitha angu-16B: sebenzisa i-multimodal diffusion Transformer enamapharamitha angu-16B njengengqikithi, uhlanganise izici zombhalo nezokubona ukuze kukhiqizwe i-autoregressive diffusion.
I-autoregressive distribution matching distillation: sheshisa ukucubungula usebenzisa isu le-distillation eliqondanisa ukusatshalaliswa kwe-autoregressive, unciphise kakhulu umehluko wokusatshalaliswa phakathi kokuqeqesha nokucubungula.
Ukuthuthukiswa kwesikhathi eside nokucubungula nge-KV eboshiwe: faka ukuthuthukiswa kokuzinza kwesikhathi eside kanye nendlela ye-KV cache eboshiwe ukuze kuncishiswe ukuchezuka kwesikhathi okunqwabelanayo futhi kugcinwe umthamo omkhulu wokucubungula.

Ukulungiselela indawo: dala indawo ye-Conda nge-python=3.10 bese ufaka okuncike kukho okuku-requirements.txt.
Ukulanda izisindo: thola amafayela emodeli ku-Hugging Face bese ubeka izindawo zokuhlola ngokwesakhiwo semibhalo esishiwo.
Ukuqalisa isevisi: ngena kumkhombandlela we-deploy bese usebenzisa i-bash run_server.sh ukuze uqale iseva yasendaweni.
Ukuvula isixhumi esibonakalayo: vula i-http://localhost:8080 esipheqululini ukuze ungene esixhumi esibonakalayo sokuhlela esisebenzisanayo.
Ukuhlela ngesikhathi sangempela: layisha ividiyo noma uxhume ukusakaza kwekhamera, bese ufaka imiyalelo yolimi lwemvelo ukuze uhlele ngenkathi ividiyo idlala.

Ukuhlela okusakazwayo ngesikhathi sangempela: cubungula amafreyimu evidiyo lapho efika, ngaphandle kokulinda ukulandelana okuphelele, ukuze uthole ukuzizwisa kwangempela kokuhlela ngenkathi isakaza.
Isivinini nekhwalithi ephezulu: ukucubungula okuphelele kufinyelela ku-30.19 FPS ngokulungiswa kwe-720P, futhi imodeli idlula zonke izindlela zokuhlela ezisakazwayo nezingasakazwa ekuhlolweni kwe-OpenVE-Bench.
Okukhiphayo okuzinzile kwanoma ibuphi ubude: yeqa umkhawulo wamamodeli endabuko asebenza kuphela ngeziqeshana zamasekhondi noma amaminithi, futhi iqhubeke nokuhlela ngokuzinzile ngaphandle kokusuka kwesithombe ngenkathi ividiyo idlala.
Ukulawulwa kwesemantiki okuvulekile: ngokusekelwe kusifaki khodi esinemibandela se-MLLM, isekela ukuhlela okuhlukahlukene okuqondiswa yimiyalelo yolimi lwemvelo, okuhlanganisa isitayela somhlaba wonke, izinto ezithile, ukushintshwa kwengemuva kanye nokuqondiswa yisithombe esiyireferensi.
Ukwakhiwa kokucubungula okusebenza kahle: i-autoregressive diffusion Transformer enamapharamitha angu-16B, ihlanganiswe ne-distribution matching distillation kanye ne-KV cache eboshiwe, inciphisa kakhulu umehluko phakathi kokuqeqesha nokucubungula futhi ivimbele amaphutha esikhathi anqwabelanayo.

Indawo yokugcina ye-GitHub：https://github.com/jd-opensource/JoyAI-Video-Edit
Indawo yamamodeli ye-Hugging Face：https://huggingface.co/jdopensource/JoyAI-Video-Edit
Iphepha lobuchwepheshe le-arXiv：https://arxiv.org/pdf/2608.03974

Ukukhiqizwa kwamavidiyo amafushane: abadali bangashintsha izingubo zabantu futhi balungise isitayela sesigcawu ngesikhathi ividiyo idlala, bathuthukise ukusebenza kahle kokukhiqizwa kokuqukethwe.
Imiphumela yesikhathi sangempela ekusakazeni bukhoma: abasakazi bangashintsha ingemuva noma bengeze izinto ezingokoqobo ngenkathi besakaza, ukuze kuthuthukiswe ukubukeka kokusakaza ngendlela esebenzisanayo.
Umklamo wezindlu nezangaphakathi: abasebenzisi bangashintsha ifenisha, izinto zodonga nemiphumela yokukhanya ngesikhathi bebuka ividiyo yegumbi, basizwe ekwenzeni izinqumo zokulungisa indlu.
Ukuhlolwa kwangaphambili kokukhiqizwa kwamafilimu nethelevishini: abaqondisi bangazama ngokushesha izitayela ezahlukene zokubuka nokushintshwa kwezigcawu ngesikhathi sokuhlela, banciphise izindleko zokuzama.
Ukukhiqizwa kwedatha yobuhlakani obugxilile emzimbeni: guqula amavidiyo okusebenza kwezandla zabantu abe yizinto zokusebenza kwezingalo zamarobhothi, wandise ngezindleko eziphansi inani ledatha yokuqeqesha amarobhothi.