VoxMem Uyasebenza

I-VoxMem iyibhentshimakhi yememori yamamodeli amakhulu omsindo eyethulwe amaqembu avela eNyuvesi yaseMelbourne naseNyuvesi yaseNew South Wales. Iqala ukusebenzisa izinhlobo ezi-4 zobufakazi be-acoustic: incazelo, isikhulumi, ithoni, nomsindo wendawo × imisebenzi emi-4 yememori: ukukhipha, ukucabanga phakathi kwamaseshini, ukulandelela ngokulandelana kwesikhathi, nokwenqaba ukuphendula, ohlelweni lokuhlola olunama-eksisi amabili oluhlanganisa umlando wamaseshini amaningi osukela kuma-token angu-8K kuya kwangu-64K. Ukuhlolwa kwamamodeli angu-15 kubonisa ukuthi ngaphansi kuka-32K akukho modeli edlula ukunemba okuphelele okungu-40%; amamodeli ayakwazi ukukhumbula okushiwo, kodwa akuthola kunzima ukukhumbula ukuthi ubani okushilo nokuthi kushiwoni kanjani. Ukwelula kuphela umongo womsindo akwanele ukufeza ukuxhumana kwezwi okuhlala isikhathi eside.

Ukuhlolwa kwememori yezwi kumaseshini amaningi: ngaphansi komlando wezwi onamaseshini amaningi nobude obuhlukahlukene, kuhlolwa ukuthi amamodeli amakhulu olimi lomsindo angakwazi yini ukukhumbula nokubuyisa okuqukethwe kwezingxoxo zangaphambilini.
Ukumbozwa kwezinhlobo ezine zobufakazi be-acoustic: kuhlolwa incazelo yezwi, ubunikazi besikhulumi, izinkomba ze-paralinguistic, nomsindo wendawo; izinhlobo ezintathu zokugcina zingolwazi “oluzalwa ngomsindo” olungenakubuyiselwa ngokubhala phansi kuphela.
Ukuhlukaniswa kwemisebenzi emine yememori: kuhlanganisa ukukhipha ulwazi, ukucabanga kumaseshini amaningi, ukulandelela izinguquko zesimo ngokuhamba kwesikhathi, nokwenqaba ukuphendula lapho ubufakazi bunganele. Ukuhlanganiswa kwazo ngababili kwakha amayunithi okuhlola asebenzayo angu-15.
Ukulawulwa kobude bomlando: umbuzo ofanayo ugcinwa unobufakazi obufanayo ngokuphelele kubude bomlando obungu-8K/16K/32K/64K token, kwengezwa kuphela amaseshini aphazamisayo, ukuze umthelela wobude kwimemori uhlukaniswe ngokuzimela.
Uhlelo lokuhlola oluvulekile: ikhodi nedatha kuvulekile ngokuphelele, kunikezwa amathuluzi ajwayelekile okuqalisa, okubala amaphuzu kanye ne-Leaderboard, futhi kusekelwa amamodeli e-API namamodeli asebenza endaweni.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lamaphrojekthi e-AI avulekile
Uhlaka lokuhlukanisa olunama-eksisi amabili: umklamo oyinhloko we-VoxMem yithebula lokuhlukanisa elihlanganisa ubufakazi be-acoustic nemisebenzi yememori. Ubufakazi be-acoustic buphendula ukuthi yini okufanele ikhunjulwe, buhlukanise ulwazi lwezwi lube incazelo, isikhulumi, i-paralinguistic, nomsindo wendawo; imisebenzi yememori iphendula ukuthi lusetshenziswa kanjani, okuhlanganisa ukubuyisa ulwazi eseshini eyodwa (IE), ukuhlanganisa ubufakazi kumaseshini amaningi (MSR), ukulandelela izinguquko zesimo ngokulandelana kwesikhathi (TET), nokwenqaba ukuphendula lapho ubufakazi bunganele (AR). Inhlanganisela ye-incazelo×IE ikhishiwe ngenhloso ngoba igcina isifana nokusesha umbhalo okumsulwa. Amayunithi angu-15 asele akha indawo ephelele yokuhlola, kuqinisekiswe ukuthi ikhono lememori ye-acoustic lihlolwa ngendlela ehlelekile kunokuba kuhlolwe izingxenye ezingaxhumene.
Ukwakhiwa kwedatha ngezigaba ezintathu: idatha ikhiqizwa ngochungechunge oluthi “hlela—bhala ingxoxo—hlanganisa izwi”: kuqala kukhiqizwa uhlelo oluhlelekile lombuzo ngamunye, oluchaza umbuzo, impendulo, ukusatshalaliswa kobufakazi kanye nemikhawulo yokuba semthethweni; bese kukhiqizwa ngokuhlukana imibhalo yengxoxo yohlangothi lomsebenzisi neyomsizi, kungakhulunywa ngobunikazi besikhulumi, ithoni noma umsindo wangemuva, ukuze impendulo ingavuzi embhalweni; ekugcineni kusetshenziswa i-TTS ukuhlanganisa izwi, umsebenzisi ngamunye anikezwe umbala wezwi ongaguquki, izinkomba zethoni zifakwe ngokulawula isitayela, umsindo wendawo uhlanganiswe ngesilinganiso se-signal-to-noise esingu-10dB, futhi kugcinwe inguqulo ehambisanayo engenazo izinkomba emjikelezweni ngamunye onezinkomba ukuze kuhlolwe ikhwalithi.
AmaseXhini aphazamisayo nokulawulwa kobude: umlando ngamunye wakhiwa yiseshini yobufakazi, amaseshini aphazamisayo namaseshini okugcwalisa. Amaseshini aphazamisayo anezihloko eziseduze nezobufakazi kodwa amanani ahlukile. Lokhu kuqinisekiswa umhlukanisi obheka umbhalo kuphela onokunemba okungu-51–56%, okuqinisekisa ukuthi imodeli ayikwazi ukuthola impendulo ngokufanisa izihloko zangaphandle. Ngobude, amazinga amane omlando akhiwe ngendlela ehlanganisiwe; umlando welulwa kuphela ngamaseshini okugcwalisa angahlobene, ukuze umbuzo, impendulo nobufakazi kuhlale kufana kuwo wonke amazinga amane. Ngakho izinguquko zokunemba zingabangelwa kuphela ukwelulwa komlando, ngaphandle kokuxubana nobunzima bombuzo.

Clone indawo yokugcina: donsela ikhodi ku-GitHub usebenzisa i-git clone https://github.com/swagshaw/voxmem bese ungena kufolda yephrojekthi.
Faka okudingekayo: sebenzisa pip install -r requirements.txt; ngokuzithandela faka pip install "datasets" ukuze umsindo uguqulwe ngokuzenzakalelayo ube ama-array.
Lungisa okhiye be-API: uma uhlola imodeli ye-API, faka i-SDK efanele bese usetha ukhiye, isibonelo pip install openai bese export OPENAI_API_KEY=.
Qalisa ukuhlola okusheshayo: qala usebenzise python run_voxmembench.py --config 8k_speaker_information --model abstain --allow-abstention --limit 25 --out predictions_smoke.jsonl ukuze uqinisekise ukuhamba kwenqubo, bese usebenzisa score_voxmembench.py --judge exact-match ukubala amaphuzu. Qinisekisa ukuthi ungqimba lokwenqaba luthola u-1.0 emibuzweni okufanele yenqatshwe no-0.0 emibuzweni engaphenduleka.
Ukuhlola okusemthethweni: khetha ukucushwa, njengamaqoqo aphelele e-32k noma i-subset engu-32k_speaker_information, kanye nemodeli okuhloswe ngayo, isibonelo python run_voxmembench.py --config 32k --model openai:gpt-4o-audio-preview --allow-abstention --out predictions_32k.jsonl; amamodeli asebenza endaweni adinga --adapter ukuze kuchazwe umsebenzi wokuxhuma wangokwezifiso kanye ne- --model-dir ekhomba kufolda yama-weight.
Ukubalwa ngamajaji e-LLM: sebenzisa python score_voxmembench.py --predictions predictions_32k.jsonl --judge openai:gpt-4o-mini --out metrics_32k.json. Ijaji libheka umbuzo, impendulo ejwayelekile kanye nempendulo kuphela, bese linquma ukufana kwezimpendulo ezivulekile.

Eyokuqala ukumboza ngokuhlelekile imemori ezalwa ngomsindo: idlula umkhawulo wamabhentshimakhi akhona agxila kokuqukethwe kwamagama kuphela, okokuqala ifake ukuthi ubani okushilo, ukuthi kushiwoni kanjani, nokuthi kuzwakale ini njengolwazi olungelona umbhalo ohlelweni olulodwa lokuhlola.
Umklamo wokuhlukanisa ngama-eksisi amabili unokuqina: usebenzisa amayunithi angu-15 asebenzayo avela ezinhlotsheni ezi-4 zobufakazi be-acoustic × imisebenzi emi-4 yememori esikhundleni sokukhetha izihloko ngokungahlelekile, ukuze ikhono lememori yezwi lihlukaniswe futhi lihlolwe ngokugcwele nangokuzimela.
Ukusethwa kwamaseshini amaningi kusondelene nezimo zangempela: kulingisa izingxoxo eziningi zokuxhumana komuntu nomshini ngezwi ezenzeka ngokuhamba kwesikhathi, hhayi i-monologue eyodwa ende, kuhlolwe imemori yesikhathi eside yemodeli kubakhulumi abaphindayo nasezibuyekezweni zesimo.
Umthelela wobude uhlukaniswe ngokuphelele: umbuzo ofanayo unobufakazi obuhlanganisiwe nobungaguquki kubude bomlando obungu-8K kuya ku-64K, okokuqala kuhlukaniswe ngokucacile umthelela “wokwelulwa komlando” ebunzimeni bombuzo nasohlotsheni lobufakazi.
Ukwakhiwa okuvimbela ukuvuza nokuhlolwa kwekhwalithi okuphindwe kabili: izinhlangothi zombili zengxoxo azikhulumi ngezinkomba ze-acoustic, izinguqulo ezihambisanayo ziyaqhathaniswa, futhi ukuhlolwa kwekhwalithi kwamazinga amabili kususa imibuzo engaphendulwa ngokuthembela ekubhalweni kuphela, kuqinisekiswe ukuthi umbuzo udinga “ukulalela” kunokuthi “ufunde”.
Idizayini yeziphazamiso yenza ubuqili bube nzima: amaseshini aphazamisayo anezihloko eziseduze kodwa amanani ahlukile, aqinisekiswe umhlukanisi wombhalo onokunemba okungu-51–56%, kuvalwe indlela emfushane yokuthola impendulo ngokufanisa amagama ayisihluthulelo.

Iwebhusayithi yephrojekthi: https://swagshaw.github.io/voxmem/
Indawo yokugcina ye-GitHub: https://github.com/swagshaw/voxmem
Indawo yamamodeli e-HuggingFace: https://huggingface.co/datasets/AudioMemory/voxmembench
Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2609.32607

Ukukhetha ikhono lememori yesikhathi eside kubasizi bezwi nezikhulumi ezihlakaniphile: abakhiqizi bangasebenzisa i-VoxMem ukuhlola ngokufanayo ukuthi amamodeli akhethiwe angakwazi yini ukukhumbula izintandokazi zomsebenzisi, amazwi amalungu omndeni nezithembiso zangaphambilini phakathi kwamaseshini, kugwenywe ukuthi “lokho okwashiwo ngesonto eledlule kulibaleke ngokushesha”.
Ukuhlolwa kokuxhumana ngezwi ezimotweni nakumadivayisi agqokekayo: umsindo wangaphakathi emotweni uyaphazamisa futhi ithoni yomsebenzisi iyashintsha; ubukhulu bomsindo wendawo nezinkomba ze-paralinguistic ku-VoxMem bungahlola ukwethembeka kwememori yemodeli ezimweni zangempela ezinomsindo.
Ukuhlolwa kwekhwalithi ezinhlelweni zamasevisi amakhasimende nasemihlanganweni enezikhulumi eziningi: izikhungo zesevisi yamakhasimende nemininingwane yemihlangano kudinga ukuhlanganisa kahle ukuthi “yimuphi umusho owashiwo ubani”; amaphutha okuhlanganisa isikhulumi ku-VoxMem angasetshenziswa ngokuqondile ukuhlonza izinkinga zokudida izikhulumi zalezi zinhlelo.
Ukuqinisekisa imemori yemizwa kumarobhothi ahambisana nabantu nasemikhiqizweni yokunakekela asebekhulile: into umuntu osekhulile ayisho ngokububula ngesonto eledlule kanye nokukhathala okubonakala ethonini yakhe kungathwalwa kuphela yizinkomba ze-paralinguistic; ubukhulu bokulandelela ngokulandelana kwesikhathi ku-VoxMem bungahlola ukuthi umkhiqizo uyakwazi yini ukubona izinguquko zesimo ngokuhamba kwesikhathi.