MiDashengLM-Spatial Uyasebenza

I-MiDashengLM-Spatial iyimodeli enkulu yokuqonda umsindo wendawo evulelwe umphakathi yithimba lakwa-Xiaomi Dasheng, futhi iyimodeli yokuqala embonini ehlanganisa ukuqonda umsindo okuvamile nokuqonda indawo ngaphakathi kwesakhiwo esisodwa, kusukela ekuqaleni kuze kube sekugcineni. I-MiDashengLM-Spatial isekelwe ku-MiDashengLM-7B, yengeza i-Spatial-Dasheng encoder kanye nemojuli yezimo ezisuka ezincazelweni ezisezingeni eliphezulu ziye endaweni, ekwazi ukubona indawo yomthombo womsindo, indlela yokuhamba kanye nobudlelwano bendawo nesikhathi phakathi kwemithombo eminingi yomsindo, futhi isekele ukutholwa komthombo ogqagqene ifreyimu nefreyimu. Iphakheji ihlanganisa izigcawu zendawo eziyizigidi ezingu-1 kanye nedatha yokuqeqesha engu-6M ye-QA, ikhishwe ngaphansi kwelayisensi ye-Apache 2.0, futhi izosetshenziswa ezindlini ezihlakaniphile nasemakhabetheni ezimoto.

Ukuthola indawo yomthombo womsindo: Ukunquma ukuthi umsindo uvela kwesokunxele, kwesokudla noma ngqo ngaphambili.
Ukuqonda indlela yokuhamba: Ukubona indlela nendlela umthombo womsindo ohamba ngayo endaweni.
Ukucabanga ngobudlelwano beqembu bendawo nesikhathi: Ukucabanga ngobudlelwano bendawo obuhambisanayo phakathi kwemithombo eminingi yomsindo, njengokuthi ubani oseduze kakhulu kwesokunxele noma ukuthi basohlangothini olulodwa yini.
Ukuqonda okuqukethwe okwenziwe kube kwendawo: Ukuhlanganisa ukuthi “kuzwakale ini” nokuthi “umsindo uvela kuphi” ukuze kuphendulwe imibuzo, njengokunquma ukuthi yimuphi okhulumayo o-ode kuqala.
Incazelo yomsindo wendawo: Ukukhiqiza incazelo ephelele evulekile yendawo, ukuhamba nobudlelwano besikhathi bayo yonke imithombo yomsindo esigcawini.
Ukuguqula inkulumo yendawo ibe umbhalo: Ukubhala okuqukethwe ngokulandelana kokukhuluma nokumaka indawo yesikhulumi ngasinye.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lezingxoxo zamaphrojekthi e-AI avulelekile
Isakhiwo esihlanganisiwe esinamagatsha amabili: Imodeli isekelwe ku-MiDashengLM-7B futhi yandiswe nge-Spatial-Dasheng spatial audio encoder, kwakheka amagatsha amabili ahambisanayo, elencazelo nelendawo, abhekele ngokuhlukana ukuqonda okuqukethwe nokubona indawo. Lokhu kwenza ukuqonda umsindo okuvamile nokuqonda indawo kwenzeke ngesikhathi esisodwa ngaphakathi kwemodeli eyodwa.
I-Spatial-Dasheng encoder: Isebenzisa okokufaka kwe-binaural ukuze ithole futhi ibeke imithombo yomsindo egqagqene ifreyimu nefreyimu, ibambe izinkomba ze-binaural ezifana ne-ITD/ILD, futhi yakhe imodeli yendawo enemininingwane yendawo yomthombo womsindo nendlela yawo yokuhamba.
Izimo ezisuka ezincazelweni ezisezingeni eliphezulu ziye endaweni (HSSC): Lena yindlela yezimo edlulisa kancane kancane izethulo ezivela ezingqimbeni ezimaphakathi zegatsha lezincazelo iye ezingqimbeni ezihambisanayo zegatsha lendawo, ukuze ukwakhiwa kwendawo kusebenzise umongo wencazelo, kuyilapho kugcinwa ukuzimela kokusebenza kwamagatsha womabili.
Ipayipi ledatha lezigcawu zendawo elinwebekayo: Ngokulingisa kwakhiwa izigcawu ze-acoustics zendawo eziyizigidi ezingu-1, ezihlanganisa imisindo yemvelo, inkulumo nomculo, bese kwenziwa ngokuzenzakalelayo izincazelo zendawo ezisezingeni lesigcawu kanye namapheya e-QA ayizigidi ezingu-6 okuqinisekisa amaqiniso, okusekela ukuqeqeshwa kwamakhono endawo ngezinga elikhulu.
Ukwenza kube jikelele ukusuka ku-Sim kuya ku-Real: Imodeli eqeqeshwe ngedatha yokwenziwa ibonisa ikhono eliqinile lokuqonda indawo ekurekhodweni kwangempela kwe-binaural, futhi ayilahlekelwa ukusebenza kokuqonda umsindo okuvamile kwesisekelo ngesikhathi ithola amakhono endawo.

Klona indawo yokugcina ikhodi: git clone https://github.com/xiaomi-research/midashenglm-spatial.git.
Dala indawo yokwenza inference bese ufaka okuncikile: conda create -n midashengspatial python=3.13 -y && conda activate midashengspatial, bese usebenzisa pip install -r requirements.txt.
Thola izisindo zemodeli: Akudingeki ukuzilanda ngokwehlukana; ikhodi izozilanda ngokuzenzakalelayo ku-HuggingFace mispeech/midashenglm-spatial.
Lungisa umsindo we-binaural: Okokufaka kube umsindo wangempela we-binaural oneziteshi ezimbili onobukhulu obungu-(2, T).
Layisha imodeli ne-processor: Sebenzisa i-AutoModelForCausalLM ne-AutoProcessor ukuze ulayishe imodeli nge-trust_remote_code=True.
Yakha imilayezo yengxoxo: Hlela i-messages ngokulandelana kokuqeqesha kokuthi “umsindo kuqala, umbhalo kamuva”, isibonelo umsindo + “Write a general audio caption describing the sound source and its spatial information.”
Yenza inference: Ngemva kokucubungula okokufaka nge-processor.apply_chat_template, shayela model.generate bese usebenzisa i-batch_decode ukuze ukhiphe imiphumela.
Qalisa isibonelo esiyinkomba: I-example/inference.py esendaweni yokugcina inikeza isibonelo esiphelele esingaqhutshwa ngokuqondile.

Eyokuqala embonini: Imodeli yokuqala evulelekile yokuphela-kuya-ekupheleni ehlanganisa umsindo nolimi, esekela ukuqonda umsindo okuvamile nokuqonda indawo ngaphakathi kwesakhiwo esisodwa.
Ikhono elihamba phambili lokuqonda indawo: Kuqoqo elincane lokucabanga kwendawo le-STAR-Bench, ukunemba kufinyelela ku-70.3%, kusondele ezingeni labantu elingu-73.7%, futhi kuphakeme kakhulu kune-47.0% ye-Gemini-3.6-Flash.
Ukuqonda okuthembekile kwangempela: I-Channel-Swap ACR ifinyelela ku-72.6%, okufakazela ukuthi ukwahlulela indawo yesokunxele nesokudla kusekelwe ezinkombeni zangempela zokuzwa.
Akukho ukulahleka kwamakhono: Emabhentshimakhi avamile e-QA yomsindo we-mono, ukuqaphela inkulumo kanye nencazelo yomsindo, imiphumela icishe ifane neka-MiDashengLM-7B eyisisekelo; amakhono endawo awanciphisi ukusebenziseka okuvamile.
Ukuhlukanisa imithombo yomsindo egqagqene: I-Spatial-Dasheng encoder isekela ukutholwa nokubekwa kwemithombo yomsindo egqagqene ifreyimu nefreyimu, futhi ing处理 izimo eziyinkimbinkimbi njengabantu abaningi abakhuluma ngesikhathi esisodwa.

Iwebhusayithi esemthethweni yephrojekthi: https://jinbo-hu.github.io/midashenglm-spatial-demo/
Indawo yokugcina ye-GitHub: https://github.com/xiaomi-research/midashenglm-spatial
Indawo yemodeli ye-HuggingFace: https://huggingface.co/mispeech/midashenglm-spatial
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2610.11156

Ukuxhumana kwekhabhinethi ehlakaniphile: Ukunquma ukuthi yimuphi umgibeli esihlalweni esithile okhulumayo ngaphakathi emotweni, ukuze kusebenze ukuqaphela inkulumo nokusebenzisa imiyalo ngokwezindawo.
Ukuqonda umsindo ezindlini ezihlakaniphile: Ukusebenzisa izipikha noma amadivayisi asekhaya ukuze kubonwe indawo nokuhamba kwemisindo ekhaya.
Umsindo wendawo we-AR/VR: Ukuhlinzeka umhlaba obonakalayo ngokuqonda kwendawo ezingeni lokuzwa, okuvumela ukubekwa kwemithombo yomsindo, ukwenziwa kwezincazelo zendawo kanye nokuxhumana okugxilisa umuntu.
Amadivayisi okusiza ukuzwa nokufinyeleleka: Ukusiza abantu abanenkinga yokuzwa ukuthi babone ukuthi umsindo uvela ngakuphi, kuthuthukiswe ukuqonda imvelo nokuphepha lapho behamba.