IndexTTS-2.5 Uyasebenza

I-IndexTTS-2.5 iyimodeli yokukopisha izwi esezingeni lezimboni ye-zero-shot, ekhishwe yi-Bilibili njengomthombo ovulekile. Inamapharamitha angu-0.8B kuphela, futhi isekela ukudluliselwa kwezwi phakathi kwezilimi ezinhlanu: isiShayina, isiNgisi, isiJapane, iSpanishi nesi-Arabhu. Imodeli iphinde yakha kabusha ukwakheka kokuqagela, yandisa isivinini sokuqagela izikhathi ezingu-2.28, futhi isekela ukulungiswa kwejubane lokukhuluma kusuka ku-0.5x kuya ku-2.0x, kanye nokulawula okunemininingwane kokubizwa kwamagama nemizwa. Ikhodi ephelele, izisindo zemodeli kanye nephepha locwaningo sekukhishiwe, ngaphansi kwesivumelwano se-Bilibili Model License.

Ukukopisha izwi kwe-zero-shot: Kudingeka kuphela umsindo oyisibonelo wemizuzwana emi-3 kuya kweyi-10 ukuze kuphindwe ngokunembile izici zethoni yanoma yimuphi umuntu okhulumayo, ngaphandle kokuqeqeshwa okwengeziwe okuqondiswe kulowo muntu.
Ukusekelwa kwezilimi ezinhlanu: Isekela isiShayina, isiNgisi, isiJapane, iSpanishi nesi-Arabhu.
Ukulawulwa kwemizwa okunemininingwane: Ihlinzeka ngezindlela eziningi zokulungisa imizwa: ukudlulisa imizwa ngomsindo ozimele oyisibonelo somuzwa, ukucacisa ngokunembile amandla omuzwa usebenzisa i-vector yemizwa enezilinganiso eziyisishiyagalombili, ukuvula ukuqagela okuzenzakalelayo komuzwa okuqhutshwa umbhalo, kanye nokulungisa ukujula komuzwa ngepharamitha ye-emo_alpha.
Ukuhlukanisa ithoni nemizwa: Umsindo oyisibonelo sethoni kanye nomsindo oyisibonelo somuzwa kungacaciswa ngokuzimela ngokuphelele. Umsebenzisi angakwazi ukulawula ngokwehlukana ukuthi “ubani okhulumayo” nokuthi “ukhuluma kanjani”.
Ukulungiswa kwejubane lokukhuluma ngokungenamkhawulo: Ipharamitha ye-duration_factor ivumela ukulungiswa kwejubane lokukhiqiza nganoma yiliphi inani eliphakathi kuka-0.5 no-2.0, ukuze kuhlangatshezwane nezidingo zesigqi ezimweni ezahlukene.
Ukungenelela okunembile ekubizweni kwamagama: Isekela amazinga amathathu okulawula ukubizwa kwamagama: i-pinyin yesiShayina, amafonimu e-CMU esiNgisi, kanye ne-kana yesiJapane. Lokhu kusiza ukuxazulula izinkinga zamagama anezindlela eziningi zokubizwa, amagama anokufundwa okuhlukile kanye nokubizwa okuncike kumongo.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxhumana lamaphrojekthi e-AI anomthombo ovulekile
Uchungechunge lokukhiqiza olunezigaba ezintathu: Imodeli isebenzisa ukwakheka kwezigaba ezintathu okuthi “umbhalo → i-semantic Token → i-mel spectrogram → i-waveform”. Imodeli yolimi ye-Transformer eya kumbhalo iye ku-semantic (T2S) ikhiqiza ama-semantic Token, bese imojuli ye-semantic iye ku-mel spectrogram (S2M) esebenzisa i-flow matching engeyona i-autoregressive ikhiqiza i-mel spectrogram. Ekugcineni, i-neural vocoder ibuyisela umsindo wokugcina.
Ukucindezelwa kwe-semantic codec: Izinga lamafreyimu lama-semantic Token lehliswa lisuka ku-50Hz liye ku-25Hz, okwenza ubude bokulandelana bunciphe ngesigamu ngqo. Lokhu kwehlisa kakhulu inani lokubala nokusetshenziswa kwememori kwemojuli ye-T2S ngesikhathi sokuqeqesha nokuqagela, futhi kungenye yezinto ezibalulekile ezisheshisa ukuqagela.
Ukuthuthukiswa kokwakheka kwe-Zipformer: Inethiwekhi eyisisekelo yemojuli ye-S2M ishintshwe isuka ku-U-DiT yayiswa ekwakhekeni kwe-Zipformer esebenza kahle kakhulu. I-Zipformer ifinyelela ikhono eliqinile lokumodela ukuncika kwesikhathi eside ngamapharamitha ambalwa, okwenza ukukhiqizwa kwe-mel spectrogram kusheshe futhi kuzinze, kuyilapho kugcinwa ikhwalithi yomsindo owenziwe.
Isu lokumodela izilimi eziningi: Ukuze kuxazululwe ukudideka okubangelwa ukugqagqana kwezinhlamvu ezimweni zezilimi eziningi, ithimba liphakamise amasu amathathu: ukuqondanisa okuqaphela imingcele, okufaka izimpawu zolimi ngaphambi nangemva kombhalo njenge-; ukuhlanganisa ezingeni le-Token, okuhlanganisa i-embedding eqondene nolimi ngalunye kuyo yonke i-Token yokufaka; kanye nokukhiqiza okuqondiswa imiyalelo, okufaka isiqalo somyalelo wolimi lwemvelo ngaphambi kombhalo njenge-“sicela ufunde ngesiNgisi”.
Ukuthuthukiswa ngokufunda okuqinisa kwe-GRPO: Esigabeni sokuqeqeshwa kwangemva kokuqeqeshwa semojuli ye-T2S, kwethulwa i-GRPO (Group Relative Policy Optimization). Izinga lamaphutha lamagama (WER) lemodeli ye-ASR evaliwe lisetshenziswa njengesignali yomvuzo. Kukhiqizwa amasampula amane angaba khona ngaso sonke isikhathi, bese ngokulungisa amathuba ahlobene amasampula anomvuzo ophezulu, kunwetshwa ngokuqhubekayo ukunemba kokubizwa kwamagama kanye nemvelo yezwi.

Ukulungiselela indawo: Umsebenzisi kufanele aqale afake i-Git kanye nomphathi wamaphakheji we-uv, bese esebenzisa i-git clone https://github.com/index-tts/index-tts.git ukulanda ikhodi esemthethweni endaweni.
Ukufaka okuncikile: Ngemva kokungena kufolda yephrojekthi, sebenzisa umyalo othi uv sync --all-extras ukuze kwakhiwe ngokuzenzakalelayo indawo ebonakalayo futhi kufakwe konke okuncikile okudingekayo kwe-Python.
Ukulanda imodeli: Sebenzisa ithuluzi le-huggingface-cli noma le-modelscope ukulanda izisindo zemodeli endaweni esemthethweni ethi IndexTeam/IndexTTS-2.5, uzibeke kufolda yendawo ethi checkpoints.
Ukuqalisa isixhumi esibonakalayo seWeb: Sebenzisa uv run webui.py ukuze uqalise isixhumi esibonakalayo esisebenzisanayo se-Gradio, bese uvakashela ku-http://127.0.0.1:7860 esipheqululini ukuze wenze imisebenzi yokukhiqiza izwi ngokubonakalayo.
Ukulayisha imodeli: Kuskripthi se-Python, ngenisa ikilasi le-IndexTTS2 kusuka ku-indextts.infer_v2_5, bese udlulisa indlela yefayela lokumisa kanye nefolda yemodeli ukuze kwakhiwe isibonelo semodeli.
Ukukopisha izwi okuyisisekelo: Shayela indlela ye-infer yesibonelo, udlulise indlela yomsindo oyisibonelo, umbhalo okuqondiswe kuwo kanye nekhodi yolimi, ukuze kukhiqizwe futhi kugcinwe ifayela lezwi elinethoni ekopishiwe.
Ukulawula imizwa: Endleleni ye-infer, dlulisa i-emo_audio_prompt eyengeziwe ukuze ucacise umsindo oyisibonelo somuzwa, noma usebenzise i-emo_vector ukudlulisa i-vector yemizwa enezilinganiso eziyisishiyagalombili ukuze ulawule ukubonakaliswa komuzwa ezwini elikhiqizwayo.
Ukulungisa ijubane lokukhuluma: Ngesikhathi sokuqagela, setha ipharamitha ye-duration_factor enanini eliphakathi kuka-0.5 no-2.0 ukuze usheshise noma wehlise ijubane lokukhuluma ngokwesilinganiso.
Ukungenelela ekubizweni kwamagama: Faka omaka abafana no-<行|XING2>, noma <上手|じょうず> embhalweni wokufaka ukuze ulawule ngokunembile ukubizwa kwamagama anezindlela eziningi zokubizwa noma amagama anokufundwa okuhlukile.

Amapharamitha ambalwa, ukusebenza okuphezulu: Imodeli inamapharamitha angu-0.8B kuphela. Ngokucindezelwa kwe-semantic codec kanye nokuthuthukiswa kokwakheka kwe-Zipformer, ifinyelela ekusheshisweni kokuqagela izikhathi ezingu-2.28. Nge-BF16, i-real-time factor (RTF) yehla yaya ku-0.20, kuhlanganisa ukukhanya nokusebenza okuphezulu.
Ukudluliselwa kwezwi phakathi kwezilimi ezinhlanu: Isekela isiShayina, isiNgisi, isiJapane, iSpanishi nesi-Arabhu. Umsebenzisi angasebenzisa umsindo oyisibonelo wanoma yiluphi ulimi ukukopisha izwi lolunye ulimi, futhi angakwazi ukudlulisa imizwa phakathi kwezilimi ngaphandle kwedatha yokuqeqesha imizwa yolimi okuqondiswe kulo.
Ukulawula okunemininingwane nobukhulu obuningi: Imodeli isekela ukulungiswa kwejubane lokukhuluma kusuka ku-0.5x kuya ku-2.0x ngokungenamkhawulo, ukulawulwa kwe-vector yemizwa enezilinganiso eziyisishiyagalombili, ukuqagela kwemizwa okuqhutshwa umbhalo, kanye nokungenelela ekubizweni kwamagama ngamazinga amathathu: i-pinyin, amafonimu e-CMU kanye ne-kana yesiJapane. Lokhu kuhlangabezana nezidingo zokulungisa ngokunembile ukukhiqizwa kwezwi ekudalweni kokuqukethwe.
Ukuhlukaniswa okuphelele kwethoni nemizwa: Umsebenzisi angacacisa ngokuzimela umsindo oyisibonelo sethoni kanye nomsindo oyisibonelo somuzwa, futhi lokhu kungavela nasezilimini ezahlukene. Ngale ndlela, kugcinwa ithoni yomuntu okuqondiswe kuye, kuyilapho kufakwa ngokuguquguqukayo isitayela somuzwa esidingekayo.
Ukusekelwa komthombo ovulekile kwesi-Arabhu: Phakathi kwamamodeli e-TTS anomthombo ovulekile ajwayelekile, i-IndexTTS-2.5 ingenye yamamodeli ambalwa asekela ngokugcwele ukukopishwa kwezwi kwesi-Arabhu kanye nokudluliselwa phakathi kwezilimi, igcwalisa igebe lolimi emkhakheni wokukhiqiza izwi okunomthombo ovulekile.

Indawo yokugcina ye-GitHub: https://github.com/index-tts/index-tts
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2601.03888

Ukudubula kwe-AI nokukhiqizwa kokuqukethwe okulalelwayo: Abadali bokuqukethwe badinga kuphela ukunikeza umsindo oyisibonelo womsakazi noma womlingiswa ukuze bakhiqize ngokushesha izincwadi ezilalelwayo, imidlalo yomsakazo kanye nokuqukethwe kwama-podcast ngezilimi nezitayela zemizwa eziningi, ngaphandle kokuqopha kabusha.
Ukwenziwa kwasendaweni kwamavidiyo ngezilimi eziningi: Amaqembu efilimu namavidiyo amafushane angasebenzisa umsindo oyisibonelo wesiShayina ukukhiqiza ngokuqondile ukudubula kwesiNgisi, isiJapane, iSpanishi noma isi-Arabhu, agcine ithoni yomlingisi wokuqala futhi ehlise kakhulu izindleko zokusabalalisa kwamanye amazwe.
Izwi lemidlalo nabalingiswa abangokoqobo: Abathuthukisi bemidlalo bangakopisha ithoni ethile ye-NPC noma yesithombe esibonakalayo, basebenzise ama-vector emizwa ukushintsha ngesikhathi sangempela phakathi kwenjabulo, intukuthelo, usizi nezinye izimo, ukuze bakhiqize izwi lendaba eliguqukayo kanye nokusebenzisana kokusakaza bukhoma.
Ukusebenzisana kwezwi ngesikhathi sangempela nabantu bedijithali: Ngenxa ye-real-time factor ephansi kakhulu engu-0.20, i-IndexTTS-2.5 ingasetshenziswa ezinsizeni zamakhasimende ezihlakaniphile, ekusakazeni bukhoma kwabantu bedijithali nasekuhumusheni ngesikhathi sangempela, isekele izimpendulo zezwi ezenziwe ngezifiso ezine-latency ephansi.
Imfundo nokufunda izilimi: Izikhungo zemfundo zingasebenzisa umsebenzi wokulawula ukubizwa kwamagama ukukhiqiza umsindo oyisibonelo we-pinyin ejwayelekile, amafonimu e-CMU noma i-kana yesiJapane, futhi zikopishe ithoni kathisha ukuze zenze izifundo ezivumelanisiwe ngezilimi eziningi.