TalkLikeYou Uyasebenza

I-TalkLikeYou iwuhlaka lwe-avatar yedijithali oluvulekile nolwesikhathi sangempela olukhishwe yi-Institute of Automation ye-Chinese Academy of Sciences. Ngokusebenzisa isithombe esisodwa nomsindo owodwa, yenza umuntu osesithombeni akhulume, kanti ukunyakaza kwezindebe kungalingisa imikhuba yokukhuluma yomuntu wangempela okhethiwe. I-TalkLikeYou isebenzisa ukumodela indawo yokunyakaza nokusampula kwesinyathelo esisodwa ukuze ikhiqize ngesikhathi sangempela. Isekela ukwenza ngokwezifiso imikhuba kusetshenziswa inombolo echazwe kusengaphambili noma ividiyo eyisithenjwa, iphakamisa amamethrikhi akhethekile okukala ukufana kokulingisa, futhi yonke ikhodi nezisindo zayo kuvulekile.

Isithombe esikhulumayo: Layisha isithombe sobuso bomuntu nomsindo, bese ukhiqiza ividiyo yomuntu ovula umlomo akhulume.
Ukulingisa umkhuba: Lingisa umkhuba wokukhuluma wezindebe womuntu othile, owuchaza ngenombolo echazwe kusengaphambili noma ngevidiyo eyisithenjwa yemizuzwana engu-4.
Ukukhiqiza ngesikhathi sangempela: Isebenzisa ubuchwepheshe bokusampula kwesinyathelo esisodwa, ifinyelele ku-32.43 FPS futhi isekele okukhiphayo okusakazwayo ngesikhathi sangempela.
Ukudabha ividiyo: Shintsha ukunyakaza kwezindebe zomuntu osevidiyweni, ugcine ukunyakaza kwekhanda nokulandelana kwesikhathi kwasekuqaleni, okulungele ukudabha ngezilimi eziningi.
I-avatar yengxoxo: Ngemva kokuxhuma imodeli enkulu yolimi nokubonwa nokuhlanganiswa kwenkulumo, ingakha idijithali ephindaphinda umuntu ukuze ixoxe ngesikhathi sangempela.
Ukushayela ngezitayela: Isekela izithombe ezingezona ezabantu, njengezikeshi, imidwebo kawoyela, ama-anime nabalingiswa be-3D.

Landela i-WeChat uphendule ngokuthi “开源” ukuze ujoyine iqembu lamaphrojekthi e-AI avulekile
Ukumodela indawo yokunyakaza: Ngokungafani nokukhiqiza izithombe ngokuqondile endaweni yamaphikseli, i-TalkLikeYou iqala ngokuhlaziya isithombe sibe ngamapharamitha okunyakaza, bese ifunda futhi ikhiqize ukunyakaza kokukhuluma ngalezi zinombolo ezilula. Irenderer ibuyisela lokhu kuvidiyo, ihlanganise isivinini nokuzinza kobuwena.
Ukusampula kwe-Flow Matching kwesinyathelo esisodwa: Isebenzisa i-Flow Matching ukufunda indlela eqondile esuka emsindweni iye kumgomo, bese ibikezela ngokuqondile ukulandelana kokunyakaza okuhlanzekile ngokusebenzisa i-reparameterization. Ukusampula okukodwa kufinyelela ikhwalithi yokuvumelanisa izindebe esondela ezinyathelweni ezingu-50, kufinyelele ku-32.43 FPS ngesikhathi sangempela.
Indawo yemikhuba ehlanganisiwe: Yakha indawo yez vectors zemikhuba enobukhulu obuphansi engahlanganiswa. Izibalo zokunyakaza kwezindebe zabantu abasethini ledatha zicindezelwa zibe ikhodi ye-one-hot, kanti ividiyo eyisithenjwa ifakwa endaweni efanayo nge-extractor yomkhuba. Lezi zindlela ezimbili zingashintshaniswa noma zihlanganiswe ngokufaka phakathi.
Isu lokuqeqesha ngezigaba ezimbili: Okokuqala, sebenzisa i-HDTF enabantu abangu-380 ufunde ukuthi “uyini umkhuba” ngekhodi ye-one-hot; bese usebenzisa amavidiyo ayisithenjwa ku-CelebV-HQ enkulu ukuze ufunde amandla okwenza jikelele. Idatha ye-one-hot ibuyekezwa ngezikhathi ezithile ukuze kuvinjelwe ukukhohlwa, okuphakamisa ukufana komkhuba kusuka ku-0.33 kuya ku-0.64.
Imethrikhi yokuhlola ye-PLAD: Iphrojetha ukunyakaza kwezindebe kuma-axis amathathu okuphimisa: ububanzi bokuvula umlomo, ukuhlanganiswa kwezindebe nokweluleka kwezindebe. Iqhathanisa ama-histogram okusabalalisa nomuntu oyisithenjwa ukuze ibale ukufana, inikeze okokuqala indinganiso elinganisekayo yokuthi “umkhuba wokukhuluma uyafana kangakanani”.

Ukulungiselela imvelo: Sebenzisa isistimu ye-Linux, ufake i-Python 3.10 + CUDA 12.1 + FFmpeg, futhi uqinisekise ukuthi ikhadi lemifanekiso le-NVIDIA linenkumbulo okungenani engu-12GB.
Ukulanda ikhodi: Clona inqolobane eyinhloko ye-TalkLikeYou ku-GitHub.
Ukufaka okuncikile: Qalisa i-conda env create -f environment.yaml ukuze wakhe imvelo bese uyisebenzisa, noma ufake i-PyTorch yenguqulo ye-CUDA ngokwakho bese usebenzisa i-pip install -r requirements.txt.
Ukulanda izisindo: Sebenzisa umyalo we-Hugging Face hf download doubi-killer/TalkLikeYou ukuze ulande izisindo zemodeli uzibeke empandeni yenqolobane.
Ukulungisa izinto: Lungisa isithombe sobuso bomuntu (noma ividiyo) nomsindo, futhi ividiyo yokufaka kufanele ibe ngu-25 FPS.
Ukuqalisa inference: Qalisa i-python -m talklikeyou --source isithombe --audio umsindo --person-id 192 --sampling-steps 1 ukuze ukhiqize ividiyo; shintsha i---person-id ukuze ushintshe umkhuba wokukhuluma.
Ukwenza umkhuba ngokwezifiso: Faka esikhundleni se---person-id ngokuthi --habit-reference bese ukhomba ividiyo eyisithenjwa okungenani enama-frame angu-100, ukuze ulingise umkhuba wokukhuluma womuntu okuleyo vidiyo.
Ukudabha ividiyo: Khomba i---source kufayela levidiyo bese wengeza i---pose-mode source ukuze ushintshe ukunyakaza kwezindebe kuphela, ugcine ukunyakaza kwekhanda kwasekuqaleni.

Ukulingisa okukhethekile komkhuba wokukhuluma: Iwuhlaka lokuqala oluvulekile embonini oluphatha umkhuba womuntu ngamunye wokukhuluma ngezindebe njengento eyisisekelo, futhi lungawucacisa ngenombolo noma ngevidiyo, lususe indlela yokuthi bonke abantu babe nokunyakaza okufanayo kwezindebe.
Isivinini sesikhathi sangempela: Ukusampula kwe-Flow Matching kwesinyathelo esisodwa kufinyelela ku-32.43 FPS, cishe ngokushesha izikhathi eziyi-100 kune-Sonic esebenzisa indlela yokusabalalisa izithombe, kanti ukusetshenziswa kwenkumbulo yekhadi le-RTX 3090 elilodwa kungaphansi kuka-4GB.
Ikhwalithi nokuvumelanisa okuhle: I-FID engu-7.71 ingaphansi kwengxenye yesithathu yomkhiqizo osendaweni yesibili, kanti i-Sync-C engu-8.73 iphezulu kakhulu, kuhlanganiswe ukuvumelanisa kwezindebe nekhwalithi yesithombe.
Ukufana komkhuba okuhamba phambili: Ukufana komkhuba ngemethrikhi ye-PLAD kungu-0.62, kuphakeme kakhulu kune-MimicTalk (0.48), PC-Talk (0.39) nezinye izimbangi.
Ukwenza ngokwezifiso okuguquguqukayo ngezindlela ezimbili: Isekela ukushintsha izitayela mahhala ngezinombolo ezichazwe kusengaphambili ezisemthethweni, futhi ifunde ngokushesha kunoma iyiphi ividiyo eyisithenjwa, kuhlanganise nokwenza imikhuba emisha ngokufaka phakathi.

Iwebhusayithi yephrojekthi: https://bq-wang0511.github.io/TalkLikeYou/
Inqolobane ye-GitHub: https://github.com/BQ-Wang0511/TalkLikeYou
Inqolobane yamamodeli e-HuggingFace: https://huggingface.co/doubi-killer/TalkLikeYou
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2610.06658

Abasakazi ababonakalayo/ukuthengisa bukhoma: Dala umsakazi obonakalayo onezindleko eziphansi onomkhuba wokukhuluma womuntu ngamunye. Ungasebenza ngesikhathi sangempela endaweni yangakini usebenzisa inkumbulo engu-4GB, ngaphandle kokuthi umuntu wangempela avele esikrinini.
Ukudabha amavidiyo ngezilimi eziningi: Shintsha ukunyakaza kwezindebe zomuntu osevidiyweni ugcine ukunyakaza kwekhanda nokulandelana kwesikhathi kwasekuqaleni, ukuze ukuma kwezindebe kwamafilimu ahunyushiwe noma okuqukethwe okuqondiswe kwamanye amazwe kuhambisane ngokwemvelo nolimi oluqondiwe.
Umuntu wedijithali/isevisi yamakhasimende ebonakalayo: Hlanganisa imodeli enkulu yolimi nokuhlanganiswa kwenkulumo ukuze wakhe umuntu wedijithali oxoxa ngesikhathi sangempela, ozosetshenziswa kusevisi yamakhasimende aku-inthanethi, ekuqondiseni okuhlakaniphile nakwezinye izimo ezidinga ukuxhumana “ubuso nobuso”.
Imfundo nokuqeqeshwa komuntu ngamunye: Dala isithombe sedijithali esikhethekile sikathisha noma somqeqeshi, uphindaphinde isitayela sakhe esiyingqayizivele sokukhuluma ukuze urekhode izifundo noma wenze ukufundisa ukude.