Xiaomi-CocktailASR-1 Uyasebenza

I-Xiaomi-CocktailASR-1 iyimodeli enkulu ye-Xiaomi evulekile yokubona inkulumo yomuntu okhethiwe, esekelwe esakhiweni se-LLM esiphelele, isebenzisa inkulumo eyireferensi njengenkomba yezwi futhi ibhala ngokunembile inkulumo yomuntu okhethiwe evela enkulumweni exubene yabantu abaningi ngaphandle kokuhlukanisa umsindo. Imodeli ifinyelela ezingeni le-SOTA ezivivinyweni zabakhulumi abaningi, iphinde ihambisane nezimo zomkhulumeli oyedwa, inamandla okwenqaba amasampula angemahle kanye nokucabanga okuchazekayo nge-chain of thought, futhi ikhishwa ngaphansi kwelayisensi ye-Apache 2.0.

Ukubona inkulumo yomuntu okhethiwe: Uma unenkulumo eyireferensi nomsindo oxubene wabantu abaningi, ibhala ngokuqondile okuqukethwe okushiwo umuntu okhethiwe ngaphandle kokuhlukanisa umsindo.
Ukuhambisana nomkhulumeli oyedwa: Imodeli efanayo icubungula izimo zomuntu oyedwa, ngokusebenza okulingana ne-ASR evamile, ngaphandle kokushintsha amamodeli.
Ukwenqaba amasampula angemahle: Uma umuntu okhethiwe engekho emsindweni, ikhipha umbhalo ongenalutho, yehlise ngempumelelo ukuqaliswa okungamanga.
Ukucabanga nge-chain of thought: Imodi ye-CoT ikhipha inani labakhulumi, ubulili, ukufana kwamazwi nezinye izinyathelo zokucabanga, ihlanganise ukuchazeka nokunemba kokubona.

Isakhiwo esihlanganisiwe esiphelele: Imodeli yakhiwe izingxenye ezintathu: i-encoder yomsindo engu-0.6B ethuthukiswe kusuka ku-Data2Vec2, i-Adapter elula yomugqa, kanye nomgogodla wemodeli yolimi enkulu i-Qwen3-8B. Okokufaka kuhlanganiswa ngokulandelana kokuthi「inkulumo eyireferensi + ukuthula kwesekhondi elingu-1 + inkulumo exubene」. Ngemva kokuba i-encoder ikhiphe izici zezinga lohlaka, i-Adapter iziqondanisa nendawo efihlekile ye-LLM, bese izithumela ku-LLM kanye ne-Prompt yombhalo ukuze kukhiqizwe umbhalo womuntu okhethiwe.
Inkulumo eyireferensi njengenkomba enemibandela: I-encoder isebenzisa indlela ye-Data2Vec2 yokubikezela okuzenzakalelayo ngamamaski, ihlanganise ulwazi lwencazelo nolomkhulumeli kunethiwekhi eyodwa, ngaphandle kwe-encoder ezimele yezwi. Inkulumo eyireferensi iphathwa njenge-conditional Prompt, okwenza i-encoder igxile ngokuzenzakalelayo kumuntu okhethiwe ngesikhathi sokukhipha izici futhi icindezele inkulumo ephazamisayo, igweme kusenesikhathi ukunqwabelana kwamaphutha kwezinhlelo zendabuko ezixhumanisa「ukuhlukanisa + ukubona」.
Ukuqeqeshwa kwezigaba eziningi okulinganisa amakhono amaningi: Ngokusebenzisa ukusatshalaliswa kwedatha okungamahora acishe abe yisigidi, kuqeqeshwa amakhono amane ngendlela ehlanganisiwe: cishe amahora angu-400,000 edatha yabakhulumi abaningi ukuze kuqeqeshwe ukukhipha okuqondiwe, cishe amahora angu-600,000 okubhangqa inkulumo eyireferensi neyomuntu ofanayo ukuze kugwenywe ukucindezelwa ngokweqile kwezimo zomuntu oyedwa, cishe amahora angu-10,000 amasampula angemahle anezinkomba ezingahambisani ukuze kufundwe ukwenqaba ngaphandle komkhawulo ngesikhathi sokucabanga, kanye nedatha ye-CoT efundisa imodeli ukuthi iqale ikhiphe uchungechunge lokucabanga bese inikeza impendulo.
I-Prompt yezindlela ezimbili nendlela yokwenqaba: Imodi ejwayelekile isebenzisa i-Prompt eyodwa ukumboza imisebenzi emithathu: ukubona umkhulumeli oyedwa, ukubona umuntu okhethiwe phakathi kwabakhulumi abaningi, nokwenqaba amasampula angemahle. Uma umuntu okhethiwe engekho, imodeli ikhipha umbhalo ongenalutho ngokwemvelo. Imodi ye-CoT isebenzisa i-Prompt ezimele ukuze ivuse amalebula okucabanga, ikhiphe inani labakhulumi, ubulili, ukufana kwamazwi nezinye izinyathelo eziphakathi, bese inikeza umbhalo wokugcina, ithuthukise ukuchazeka futhi yehlise kancane i-WER.

Ukufaka okudingekayo: Sebenzisa i-pip install torch torchaudio transformers soundfile ukufaka indawo edingekayo.
Ukulanda imodeli: Landa amafayela ezisindo zemodeli ku-HuggingFace（Ease3/Xiaomi-CocktailASR-1）.
Ukulayisha imodeli: Layisha imodeli nge-AutoModel.from_pretrained("Ease3/Xiaomi-CocktailASR-1", trust_remote_code=True, torch_dtype="bfloat16").cuda().eval().
Ukulungisa umsindo: Lungisa inkulumo eyireferensi engu-16k mono equkethe izwi lomuntu okhethiwe, kanye nomsindo womuntu oyedwa noma wabantu abaningi ozobonwa. Umsindo ongeyona i-16k uzoguqulwa ngokuzenzakalelayo.
Ukucabanga okukodwa: Shayela model("target.wav", "ref_speaker.wav") ukuze ubuyise umbhalo womuntu okhethiwe. Kumasampula angemahle, kukhishwa umbhalo ongenalutho ngokuzenzakalelayo ngaphandle kwamapharamitha angeziwe.
Ukucabanga nge-chain of thought: Faka ipharamitha cot=True; imodeli izokhipha inqubo yokucabanga ku- futhi umphumela wokugcina ku-.
Ukucabanga ngobuningi: Hlela idatha ibe yi-TSV enamakholomu angu-5 (utt_id, wav, text, ref_wav, ref_id), bese usebenzisa tools/test_batch_scp.py ucacise indlela yemodeli, i-TSV yokufaka nefayela lokukhipha ukuze ubhale okuningi ngesikhathi esisodwa.

Ukusebenza kwe-SOTA kubakhulumi abaningi: Ifinyelela ezingeni eliphezulu lokubona ezivivinyweni eziningi ezijwayelekile zabakhulumi abaningi, okuhlanganisa i-AliMeeting, i-AMI ne-LibriMix, njenge-AliMeeting Far WER engu-20.63%, uma kuqhathaniswa ne-SOTA yangaphambilini engu-27.5%.
Ukuhlanganisa izimo zomuntu oyedwa nabantu abaningi: Imodeli efanayo isebenza ngokulingana ne-ASR evamile esimweni somkhulumeli oyedwa, ngaphandle kokushintsha imodeli ngokwenani labakhulumi.
Ikhono lokwenqaba amasampula angemahle: Uma umuntu okhethiwe engekho, kukhishwa umbhalo ongenalutho; izinga lokwenqaba lifinyelela ku-68%-80%, kanti amamodeli afana ne-Qwen3-ASR ne-StepAudio anezinga elingu-0.
Ukucabanga okuchazekayo nge-chain of thought: Imodi ye-CoT ikhipha inani labakhulumi, ubulili, ukufana kwamazwi nenqubo yokucabanga, ithuthukise ukuchazeka futhi yehlise kancane i-WER.
Isakhiwo esilula esiphelele: Inkulumo eyireferensi isetshenziswa ngokuqondile njenge-Prompt yezwi, kususwe imojuli yendabuko yokuhlukanisa inkulumo futhi kugwenywe ukunqwabelana kwamaphutha kwezinhlelo ezixhumanayo.
Ukusetshenziswa okulula: Ishayelwa ngomugqa owodwa wekhodi, isekela ukuguqulwa kwesampula okuzenzakalelayo nokucabanga ngobuningi, futhi inikeza ama-Demo amasampula amahle namabi, ilungele ukusetshenziswa ngokushesha.

Ikhosombe le-GitHub：https://github.com/xiaomi-research/xiaomi-cocktailasr-1
Ikhosombe lemodeli ku-HuggingFace：https://huggingface.co/Ease3/Xiaomi-CocktailASR-1
Iphepha lobuchwepheshe le-arXiv：https://arxiv.org/pdf/2609.11274

Ukubhalwa okuqondisiwe kwemihlangano ehlakaniphile: Emihlanganweni yabantu abaningi, ilalela futhi ibhale kuphela okushiwo umkhulumeli okhethiwe, ihlunge ngokuzenzakalelayo ukuphazamiseka nezingxoxo zabanye ababambiqhaza.
Ukubona imiyalo eyinhloko kumsizi wezwi: Ezimweni zefoni, zesipikha noma zemoto, iphendula kuphela ezwini lomnikazi, ukuze amazwi angemuva angaqalisi imiyalo ngephutha.
Ukuhlaziya izingcingo zesevisi yamakhasimende nokuhlolwa kwekhwalithi: Ikhipha ngokunembile yonke inkulumo yomuntu okhethiwe ekurekhodweni kwezingcingo zabantu abaningi, ukuze kusetshenziswe ekuhloleni ukuhambisana nemithetho nezinga lesevisi.
Ukulawula ikhaya elihlakaniphile ngezwi: Endaweni yasekhaya enomsindo wethelevishini nezingxoxo zabantu abaningi, ibona ngokunembile imiyalo yezwi yomuntu ophethe isilawuli kude, igweme ukusebenza okungahlosiwe.
Imishini yokusiza ukuzwa nokurekhoda engenazithiyo: Ikhipha ngokuqondisiwe okuqukethwe kwezwi lomuntu othile futhi ikuguqulele embhalweni ngesikhathi sangempela kubantu abangezwa kahle noma abarekhoda, ukuze ezindaweni zomphakathi ezinomsindo kugxilwe「ekuzweni ngokucacile」umuntu ofuna ukumuzwa.