SALMONN-2 – imodeli yolimi olukhulu yomsindo ejwayelekile ekhishwe yi-Tsinghua nabanye Uyasebenza

I-SALMONN-2 iyimodeli yolimi olukhulu yomsindo ejwayelekile ekhishwe ngokuhlanganyela yi-Tsinghua University, i-Shanghai Artificial Intelligence Laboratory kanye ne-University of Cambridge. Yakhiwe nge-SPEAR unified self-supervised audio encoder yakwa-ByteDance kanye ne-multi-layer feature fusion adapter, ne-Qwen3 njengesisekelo sombhalo. Isebenzisa cishe amahora angu-18,000 edatha egadiwe kuphela ukuze izuze ukusebenza okuhamba phambili phakathi kwamamodeli omthombo ovulekile anosayizi ofanayo kuma-benchmark amathathu aphelele i-MMAU-Pro, i-MMAR kanye ne-MMSU. Ngokokuqala ngqa ku-ALLM ejwayelekile, ihlanganisa ngokuhlelekile amakhono athuthukile okuhlonza izehlakalo zomsindo, ukuthola umsindo okhohlisiwe, ukuhlola ikhwalithi yenkulumo kanye nokuqaphela inkulumo okunomongo we-multimodal.

Ukuqonda umsindo okujwayelekile: Isekela imisebenzi ehlukahlukene efana nokuqaphela inkulumo, ukuchaza umsindo, ukuqonda umculo, ukuqaphela imizwa nokuqinisekisa isikhulumi.
Ukuhlonza izehlakalo zomsindo (SED): Ingathola futhi ikhiphe izikhathi zokuqala nezokuphela zezehlakalo zemisindo yasendaweni, njengokukhonkotha kwenja noma izinyathelo.
Ukuthola umsindo okhohliwe ngokujulile: Inquma ukuthi inkulumo ingeyangempela yini, futhi ithole izikhathi zezingxenye ezisolwa ngokuthi zenziwe noma zahlelwa.
Ukuhlola ikhwalithi yenkulumo (SQA): Ihlola izici zomsindo ezisezingeni eliphansi, njengomsindo ophazamisayo, ukuhlanekezela nokucaca, bese inikeza isilinganiso sekhwalithi nencazelo.
Ukuqaphela inkulumo okunomongo we-multimodal (MICL): Ihlanganisa isipelingi sombhalo nomsindo oyisibonelo wokuphimisa ukuze ithuthukise ukunemba kokuqaphela amagama angavamile namagama abantu angajwayelekile.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lezingxoxo lamaphrojekthi e-AI omthombo ovulekile
I-unified self-supervised audio encoder: I-SALMONN-2 isebenzisa i-SPEAR njenge-audio front end eyodwa. I-encoder iqeqeshwa ngokufunda okuzilawulayo kudatha enkulu yomsindo engenamalebula, futhi ikwazi ukubamba ngesikhathi esisodwa ulwazi lwencazelo, lokuphimisa, lombala wezwi, lwesikhulumi kanye nolwendawo yomsindo. Lokhu kuthatha indawo yohlelo lwama-encoder amabili i-Whisper+BEATs, kunciphisa ubunkimbinkimbi besakhiwo kuyilapho kugcinwa ikhono elilinganiselayo ezizindeni ezahlukene.
I-multi-layer feature fusion (MLF) adapter: Izendlalelo ezahlukene ze-SPEAR ziphethe amazinga ahlukene olwazi—izendlalelo ezingajulile zigcina imininingwane yomsindo nokuphimisa, eziphakathi ziphethe ulwazi lombala wezwi nesikhulumi, kanti ezijulile ziqongelela imiqondo yencazelo. I-MLF adapter iqala ngokwenza i-layer normalization nokuhlanganisa ama-hidden states azo zonke izendlalelo, bese isebenzisa i-down-projection efundekayo kanye nokucindezela ngokuhlanganisa ama-frame. Ekugcineni, imephu imelela ulwazi oluhlanganisiwe lwamazinga ahlukene iye endaweni yama-embeddings yemodeli yolimi, ukuze imodeli ikwazi ukubiza ngokuguquguqukayo izinkomba zomsindo zamazinga ahlukene kuye ngezidingo zomsebenzi.
Indlela yokufaka ama-timestamp: Imodeli ifaka ama-timestamp ngohlobo lombhalo wolimi lwemvelo, njengokuthi <2.0 seconds>, ngqo ochungechungeni lomsindo. Lokhu kufakwa kushintshana nama-audio embeddings lapho kungena kwimodeli yolimi, okwenza imodeli ikwazi ukwenza imisebenzi yokuthola isikhathi ngaphakathi kohlaka olulodwa lokukhiqiza ngaphandle kwesendlalelo esengeziwe esikhethekile sama-timestamp.
Ukuqeqeshwa kokufunda okunomongo we-multimodal (MICL): Emisebenzini yokuqaphela inkulumo okunomongo, imodeli ayitholi nje kuphela uhlu lwamagama oluthonyayo lombhalo, kodwa ithola ngesikhathi esifanayo nomsindo oyisibonelo wokuphimisa kwalawo magama njengomongo we-multimodal. Ngesikhathi sokuqeqeshwa kusetshenziswa isu lokulahla ngokungahleliwe amagama aphazamisayo namagama okuqondiwe, ukuze kugwenywe ukuncika ngokweqile emathonyeni futhi kufundiswe imodeli ukusebenzisa izinkomba zomongo ngokufanele kuphela lapho ubufakazi bomsindo buwasekela.

Vakashela i-repository bese u-clone ikhodi: Vakashela i-GitHub repository https://github.com/bytedance/SALMONN/tree/salmonn2, bese usebenzisa i-git clone ukulanda ikhodi endaweni yangakini.
Dala indawo yokusebenzisa: Qalisa i-conda create -n salmonn2 python=3.10 ukuze wakhe futhi uvule indawo ebonakalayo, bese uthuthukisa i-pip, i-setuptools ne-wheel.
Faka ama-dependencies nephrojekthi: Ku-root directory ye-repository, sebenzisa i-pip install -r requirements.txt kanye ne-pip install -e . --no-deps ukuze uqedele ukufakwa kwe-SALMONN-2 nama-runtime dependencies ayo.
Landa ama-pretrained weights: Sebenzisa i-HuggingFace CLI ukulanda i-model checkpoint: hf download marcoyang/SALMONN-2-8B --repo-type model --local-dir /path/to/salmonn-2-hf.
Layisha imodeli bese wenza inference: Sebenzisa i-AutoProcessor ne-AutoModelForCausalLM ukulayisha ama-local weights, udlulise ifayela lomsindo nombhalo wemiyalelo, bese ubiza i-model.generate() ukuze uthole imiphumela yokuqonda umsindo.

Ukusebenza kahle kwedatha: Isebenzisa cishe amahora angu-18,000 edatha egadiwe, okungaphansi kakhulu kwamakhulu ezinkulungwane kuya ezigidini zamahora asetshenziswa izimbangi ezinosayizi ofanayo, ngaleyo ndlela yehlise kakhulu izindleko zokulebula.
I-front end ehlanganisiwe elinganiselayo: I-SPEAR self-supervised encoder eyodwa ithatha indawo yama-encoder amabili, ithole ukusebenza okulinganiselayo enkulumweni, emisindweni yasendaweni, emculweni nasemisebenzini ye-paralinguistic.
Ukusetshenziswa okuphelele kolwazi lwamazinga ahlukene: I-MLF adapter ihlanganisa izici zazo zonke izendlalelo ze-encoder, igweme ukulahleka kwemininingwane ebalulekile yomsindo nombala wezwi obekungalahleka ngokusebenzisa isendlalelo sokugcina kuphela.
Amakhono anwetshiwe okuhlaziya umsindo: Ngokokuqala ngqa ku-ALLM ejwayelekile, isekela ngokuhlelekile imisebenzi yokuhlaziya umsindo efana ne-SED, ukuthola umsindo okhohliwe kanye ne-SQA, ebikade inganakwa.
Iyakhula ngesikali: Ngemva kokwandisa isisekelo sombhalo sisuka ku-8B saya ku-30B-A3B, amaphuzu kuwo womathathu ama-benchmark aphelele aqhubeke nokukhuphuka, okuqinisekisa ukuthi lesi sakhiwo sinamandla amahle okukhula ngesikali.

I-GitHub repository: https://github.com/bytedance/SALMONN/tree/salmonn2
I-HuggingFace model repository: https://huggingface.co/marcoyang/SALMONN-2-8B
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2607.17079

Umsizi womhlangano ohlakaniphile: Ubhala phansi okuqukethwe komhlangano ngesikhathi sangempela, futhi usebenzisa izibonelo zokuphimisa zabahlanganyeli ukuze abone ngokunembile amagama abantu bangaphandle namagama obuchwepheshe.
Ukuhlolwa kokuqukethwe komsindo: Uthola ngokuzenzakalelayo izehlakalo zomsindo ezingavamile ekusakazeni bukhoma noma kuma-podcast, futhi abone inkulumo eyenziwe ngobuqili ukuze kuvinjelwe ukukhwabanisa.
Ukuqapha ikhwalithi yenkulumo: Unikeza ngokuzenzakalelayo amaphuzu ekhwalithi ezingxoxweni zesevisi yamakhasimende nasemibhalweni yesitudiyo, futhi athole izingxenye ezinomsindo ophazamisayo noma ukuhlanekezela.
Ukusesha ezinqolobaneni zemultimedia: Wakha izincazelo zezehlakalo ezinama-timestamp zamarekhodi omsindo omlando nezinhlelo zomsakazo, okwenza kube nokusesha okunembile okusekelwe kokuqukethwe.
Imishini esiza abantu abangezwa kahle: Yazisa abasebenzisi abangezwa kahle ngesikhathi sangempela ngezehlakalo zemisindo yasendaweni, njengensimbi yomnyango nama-alamu, ngombhalo nama-timestamp.