dots.tts Uyasebenza

I-dots.tts iyimodeli eyisisekelo yokwenziwa kwenkulumo ye-autoregressive eqhubekayo ngokuphelele, enamapharamitha ayizigidigidi ezi-2, ekhishwe ngomthombo ovulekile ngokubambisana yiqembu le-Xiaohongshu dots kanye ne-X-LANCE Laboratory yase-Shanghai Jiao Tong University. Imodeli ikhiqiza umsindo we-48kHz ngamaqoqo ngokuqondile endaweni efihlekile eqhubekayo. Ifinyelela ezingeni eliphezulu kakhulu le-SOTA ekufaneni kwezwi nokunemba kokuqukethwe kumabhentshimakhi afana ne-Seed-TTS-Eval, futhi isekela ukukopisha izwi ngaphandle kwezibonelo zangaphambilini, okukhipha okusakazwayo kanye nezingxoxo ze-duplex ezinokubambezeleka okuphansi. Ikhodi ephelele yephrojekthi kanye nezisindo zayo kukhishwe ngomthombo ovulekile, kwengezwa namandla okuhlela izwi ngokunembile nge-dots.tts.edit.

Ukukopisha izwi ngaphandle kwezibonelo zangaphambilini: Kudingeka kuphela umsindo oyisibonelo wemizuzwana emi-3–10 ukuze kuphindwe lelo zwi lifunde noma yimuphi umbhalo omusha. Isekela ukukopisha izwi phakathi kwezilimi, futhi ifinyelela ekusebenzeni kwe-SOTA ekunembeni kokuqukethwe nasekufaneni kwesikhulumi ku-Seed-TTS-Eval.
Ukwenziwa kwenkulumo kusuka embhalweni: Inikeza ukukhiqizwa kwenkulumo kwekhwalithi ephezulu, kusukela ekukhetheni izwi elingahleliwe kuya ekukhiqizeni ngezwi lesikhulumi esithile, isebenzisa imodeli ye-autoregressive eqhubekayo enamapharamitha angu-2B ukukhipha umsindo we-48 kHz ngokuqondile.
Ukukhiqizwa okusakazwayo nokusebenzisana okunokubambezeleka okuphansi: Ngokwemvelo isekela okukhiphayo okusakazwayo; ngemva kokufakwa kwesiqalo sombhalo, umsindo ukhiqizwa ngezigaba. Kumodi yokusakaza okubili ye-1T1A, uhlangothi lwenkulumo luphendula ngokushesha lapho i-LLM ekhiqiza i-Token yombhalo eyodwa, futhi ukubambezeleka kwephakheji yokuqala yomsindo kwehla kufike kuma-millisecond angu-54.4, okulungele ama-agent ezwi esikhathi sangempela nezingxoxo ze-duplex.
Ukuhlela izwi ngokunembile: Isekela ukushintshwa kombhalo, ukulungiswa kwemizwa, ukuguqulwa kwephimbo nejubane lokukhuluma, kanye nokufakwa noma ukususwa kokuma ekurekhodweni okukhona. Imisebenzi eminingi ingahlanganiswa ibe umyalo owodwa, kuyilapho izingxenye ezingahlelwanga zihlala zinjalo.
Ukwesekwa kwezilimi eziningi: Ihlanganisa izilimi ezingu-24. Ekuhlolweni kwezilimi eziningi kwe-MiniMax, ukufana kwesikhulumi kufinyelela ku-83.9 ngokwesilinganiso, kanti izilimi ezingu-19 zithole indawo yokuqala emiphumeleni yazo ngayinye.
Izivinini eziningi zokucabanga: Inikeza izindawo zokuhlola eziningi, okuhlanganisa i-Base, i-SOAR, i-MeanFlow (izinyathelo ezi-4), i-sCM yezinyathelo ezimbili kanye ne-DMD yesinyathelo esisodwa, ukuze abasebenzisi bakhethe phakathi kwekhwalithi yomsindo nesivinini sokucabanga ngokwezidingo zabo.

Landela i-WeChat, uphendule ngokuthi “开源”, bese ujoyina iqembu lezingxoxo zamaphrojekthi e-AI omthombo ovulekile
Isakhiwo se-autoregressive esiqhubekayo ngokuphelele: I-dots.tts iyayeka ukusebenzisa indlela evamile yokuguqula inkulumo ibe ama-Token ahlukene. Lonke uchungechunge lokukhiqiza lusebenza endaweni efihlekile eqhubekayo: imodeli ibikezela izingcezu ze-acoustic nge-autoregressive, bese idekhoda yesitayela se-BigVGAN ibuyisela ifomu legagasi elingu-48 kHz. Lokhu kugwema ngokuyisisekelo umkhawulo wolwazi nokulahleka kwemininingwane yezwi okubangelwa yi-discrete quantization.
Ukumelwa okuqhubekayo okunencazelo: Imodeli isebenzisa i-AudioVAE esekelwe ekwakhiweni kwe-HoliTok ukucindezela ifomu legagasi langempela libe okuguquguqukayo okufihlekile okuqhubekayo okungu-25 FPS. I-encoder, ngokuqeqeshwa okusekelwe encazelweni, ilinganisa ukwethembeka kokwakhiwa kabusha nokuhlelwa kwendawo efihlekile, okwenza ukufana kwesikhulumi ngemva kokwakhiwa kabusha kufinyelele ku-0.969 futhi kugcine imininingwane ecebile yezwi ne-acoustic ukuze kusetshenziswe ekukhiqizeni kwe-autoregressive.
Indlela yokulawula amaphutha: Ukuze kubhekwane nenkinga yokuthi ukuchezuka okuncane ezinyathelweni zangaphambilini kunganqwabelana futhi kusabalale ngokuhamba kwesikhathi ku-autoregressive eqhubekayo, i-dots.tts yethula i-causal semantic encoder. Le encoder icindezela ngesikhathi sangempela ama-patch e-VAE asevele ekhiqiziwe abe umlando wencazelo ohlanganisiwe bese iwubuyisela ku-large language model. Ngokuhlukanisa izinguquko ze-acoustic zendawo ezinokuhlukahluka okuphezulu nokugcina kuphela isifinyezo sencazelo yesikhathi eside, kunciphisa ngempumelelo ukusabalala kwamaphutha, umsindo wokuhayiza nokushintsha kombala wezwi ekukhiqizeni okulandelanayo okude.
Inqubo yokukhiqiza: I-large language model iyisikhungo senqubo yokukhiqiza. Iqaliswa nge-Qwen2.5-1.5B, yamukela ngokuqondile ama-Token ombhalo e-BPE futhi ikhiphe izimo ezifihlekile zesinyathelo ngasinye. Ngemuva kwalokho, inhloko ye-autoregressive flow matching, eyi-DiT enezingqimba ezingu-18, ikhiqiza i-acoustic patch elandelayo ngokususa umsindo ngaphansi kwezimo zesimo esifihlekile se-LLM nesiqalo se-autoregressive. Ngesikhathi esifanayo, i-CAM++ speaker encoder emisiwe inikeza i-x-vector yomhlaba wonke ukuze kugcinwe ukuvumelana kwephimbo.
Ukusheshisa ngemva kokuqeqeshwa: Iqembu lisebenzisa ukuqondanisa kwe-SOAR okuzilungisayo ukuze imodeli ilingise izimo ezichezuka enqubweni yangempela yokucabanga futhi ifunde ukuzilungisa, okuthuthukisa ukuzinza ngaphandle kwemodeli yomvuzo. I-MeanFlow distillation icindezela izindlela eziningi zikathisha zibe yijubane eliphakathi, yenze ukukhiqizwa kwekhwalithi ephezulu ngezinyathelo ezi-4. Imodeli ye-sCM eyenziwe yaba lula icindezela lokhu kube yizinyathelo ezi-2. I-distillation ye-reward-aware distribution matching (DMD), ihlanganiswe nesu lokubuyekeza lezilinganiso zesikhathi ezimbili, ikhiqiza imodeli yesinyathelo esisodwa ecacile nekhwalithi ephezulu.

Ukufaka: Sebenzisa i-pip install dots.tts ukuze ufake, noma u-clone ikhodi yomthombo ku-GitHub bese usebenzisa i-pip install -e . ukuze uyifakele ukuthuthukiswa kwasendaweni. Ezimo ezinokuxhumana okuningi, ungafaka i-SGLang Omni ukuze uthole ukusheshiswa kwe-CUDA Graph nokusekelwa kwe-continuous batching.
Ukwenziwa nge-command line: Sebenzisa i-dots.tts entry point bese ucacisa i---model-name-or-path, i---text, kanye ne---prompt-audio ne---prompt-text okuqondiwe ukuze ukopishe izwi ngaphandle kwezibonelo zangaphambilini. Ukushiya i---prompt-text kwenza uhlelo lusebenzise ukukopisha nge-x-vector, kanti ukushiya i---prompt-audio kwenza ukukhethwa kwezwi elingahleliwe.
Ukuhlela nge-command line: Sebenzisa i-dots.tts.edit entry point, unikeze i---source-audio kanye ne---instruction yesitayela se-XML, ukuze uhlele ngokunembile umbhalo, imizwa noma iprosodi ekurekhodweni okukhona.
Ukusetshenziswa okuyisisekelo kwe-Python API: Ngemva kokulayisha imodeli nge-DotsTtsRuntime.from_pretrained(), shayela i-generate(), udlulise umbhalo nomsindo oyisibonelo ongakhethwa ukuze uthole i-audio tensor eyenziwe futhi uyigcine njengefayela.
Okukhiphayo okusakazwayo nge-Python API: Shayela i-generate_stream() ukuze uthole ama-audio tensor ngezigaba, uwathumele ngesikhathi sangempela kumdlali noma ku-WebSocket, wenze kube nokudlala ngesikhathi esifanayo nokukhiqizwa okunokubambezeleka okuphansi.
Ingxoxo yokusakaza okubili nge-Python API: Sebenzisa i-DotsTtsRuntimeDoubleStreaming ukuqala iseshini, bese uthumela umbhalo i-Token ngayinye. Uhlangothi lwenkulumo luqala ukukhiqiza ngokushesha, futhi ukubambezeleka kwephakheji yokuqala yomsindo kwehla kufike kuma-millisecond angu-54.4, okulungele ama-agent ezwi esikhathi sangempela.
Ukulungisa kahle: Sebenzisa i-accelerate launch scripts/train_dots_tts.py futhi ulungise indlela yedatha yakho ukuze wenze ukulungisa kahle kwephimbo noma kwesizinda ngesisekelo sezindawo zokuhlola ezivulekile.
Ukusheshisa nge-distillation: Sebenzisa i-scripts/train_dots_tts_meanflow.py, usebenzise indawo yokuhlola ye-SOAR njengothisha, ukuze uthole imodeli yomfundi yekhwalithi ephezulu yezinyathelo ezi-4, ezi-2 noma esisodwa, ehambisana nezidingo ezihlukene zokubambezeleka nokuxhumana okuningi.
Isixhumi esibonakalayo sewebhu: Sebenzisa i-python apps/gradio/app.py ukuqalisa isixhumi esibonakalayo sokwenziwa esibonakalayo, noma usebenzise i-apps/edit_playground/app.py ukuqalisa i-Edit Playground, lapho ungalayisha khona umsindo, umake izikhawu zokuhlela futhi ubuke umphumela ngesikhathi sangempela esipheqululini.

Ukumodela endaweni efihlekile eqhubekayo ngokuphelele: Iyeka ukusebenzisa ama-acoustic Token ahlukene, ikhiqize ngokuqondile nge-autoregressive endaweni efihlekile eqhubekayo, igcine imininingwane ecebile yezwi efana nokuthungwa kwe-spectrum nomsindo wokuphefumula. Ukufana kwesikhulumi ngemva kokwakhiwa kabusha kufinyelela ku-0.969.
Ikhwalithi yokwenziwa ye-SOTA: Ifinyelela ku-SOTA kokubili ekunembeni kokuqukethwe nasekufaneni kwephimbo ku-Seed-TTS-Eval. Ekuhlolweni kwe-MiniMax kwezilimi ezingu-24, i-SIM emaphakathi ingu-83.9, idlule izixazululo ezijwayelekile ekusebenzeni okuphelele.
Ukusakaza komdabu nokubambezeleka okuphansi: Kumodi yokusakaza okubili ye-1T1A, iphakheji yokuqala yomsindo itholakala ngaphakathi kwama-millisecond angu-54.4. Ngokuhlanganisa ne-SGLang Omni, ikhadi elilodwa le-H100 lifinyelela ku-16 concurrent streams kanye ne-4.76 req/s, okuhlangabezana nezidingo zama-agent ezwi esikhathi sangempela.
Ikhono lokuhlela elihlanganisiwe: I-dots.tts.edit isebenzisa isisekelo esifanayo futhi isekela imiyalo ehlelekile ye-XML yokuhlela ngokunembile umbhalo, imizwa, iprosodi nokuma. Ekuhlolweni kwe-doteBench, ihamba phambili ngokujwayelekile kunezinye izinhlelo zomthombo ovulekile.

Indawo yokugcina ye-GitHub: https://github.com/studio-dots-ai/dots.tts
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2608.02673

I-agent yezwi esikhathi sangempela: Kumodi yokusakaza okubili ye-1T1A, ukubambezeleka kwephakheji yokuqala kwehla kufike kuma-millisecond angu-54.4, okulungele izingxoxo ze-duplex zesikhathi sangempela nabasizi bezwi abaqhutshwa yi-LLM.
Ukukopisha izwi ngaphandle kwezibonelo zangaphambilini: Kudingeka imizuzwana embalwa kuphela yomsindo oyisibonelo ukuze kuphindwe noma yiliphi izwi. Isekela ukukopisha phakathi kwezilimi, futhi ilungele izincwadi ezilalelwayo, ukudabha nokudala okuqukethwe komuntu siqu.
Ukukhiqiza okuqukethwe ngezilimi eziningi: Ihlanganisa izilimi ezingu-24 futhi igcine ukufana kwesikhulumi okuphezulu, okulungele ukusakazwa kwenkulumo ngezilimi eziningi nezinsizakalo zokwenziwa kwasendaweni zemikhiqizo yomhlaba wonke.
Ukuhlela izwi ngokunembile: Sebenzisa i-dots.tts.edit ukushintsha umbhalo, ukulungisa imizwa, ukuguqula iprosodi nokulungisa izikhawu ekurekhodweni okukhona, okulungele ukuhlelwa kwangemuva kwamaphodikasti nokucwengwa kokuqukethwe komsindo.
Ukusakaza bukhoma nokusakaza kwesikhathi sangempela: Ngokuhlanganisa nokukhipha okusakazwayo kwe-LLM, umsindo uyakhiqizwa futhi udlalwe ngesikhathi esifanayo, okulungele ukusakazwa kwezindaba, ukuthengisa bukhoma nokusakazwa kolwazi lwesikhathi sangempela.