R2T2 Uyasebenza

I-R2T2 iyimodeli ye-automatic speech recognition esakazwe yi-NetEase Youdao enokubambezeleka okuphansi nokusakaza kwangempela, igama layo eligcwele lingu-Real Real-Time Transcription, futhi yakhelwe phezu kwe-Qwen3-ASR. I-R2T2 isebenzisa indlela ye-append-only, ngakho umbhalo osuvele ubonisiwe awuguqulwa, isekela ukuhlukaniswa okulungisekayo okungu-80ms—2s, ithuthukiselwe isiShayina nesiNgisi kanye nomongo wamagama abalulekile, inokubambezeleka okumaphakathi okungaba ngu-200—600ms, futhi inikeza ama-backend e-vLLM, Transformers, llama.cpp/GGUF nokunye. Ilungele imibhalo engezansi yesikhathi sangempela, ama-voice Agent kanye ne-frontend yokuhumusha ngesikhathi esisodwa.

Ukubona inkulumo ngokusakaza kwangempela: yamukela umsindo ngama-chunk alungisekayo angu-80ms–2s, ikhiphe umbhalo ngokuqhubekayo.
Okuthunyelwe akubuyekezwa: umbhalo osuthunyelwe awuguqulwa, kugwenywe ukuqhwanyaza kwemibhalo engezansi nokuguqulwa kwezenzo ze-Agent.
Ukubambezeleka okuphansi nokunemba okuphezulu: ngokusho komkhiqizi, ukubambezeleka okumaphakathi kungaba ngu-200–600ms, kanti ukunemba kokusakaza kusondelene nokwe-recognition engaxhunyiwe ku-inthanethi.
Amagama abalulekile nomongo: isekela ama-prompt, amagama omkhakha namagama akhethekile ukuze kwandiswe izinga lokutholwa kwamagama angajwayelekile.
Izilimi eziningi: ithuthukiselwe isiShayina nesiNgisi, kuyilapho igcina amandla ezilimi eziningi okuhlanganisa isiFulentshi, isiJalimane, isiNtaliyane, isiJapane, isiKorea, isiRashiya nesi-Arabhu.
Ukusetshenziswa kwama-backend amaningi: isekela i-vLLM, Transformers, llama.cpp/GGUF, futhi ingaqalisa isevisi ye-WebSocket.

Landela i-WeChat bese uphendula ngokuthi “开源” ukuze ujoyine iqembu lokuxhumana lamaphrojekthi e-AI avulekile
Isisekelo semodeli: i-R2T2 yakhelwe phezu kwe-Qwen3-ASR, izuza amakhono ayo okubhala ikhodi yomsindo nokumodela ukulandelana, bese iqeqeshelwa ngokukhethekile ukuphuma kokusakaza.
Ukufunda isiqalo esizinzile: ngesikhathi sokuqeqesha kusetshenziswa i-stable-prefix, ukuqondanisa kwesikhathi okuphoqelelwe, ukuhlukaniswa komsindo ezingeni le-token nezinye izindlela zokwakha idatha, ukuze imodeli ifunde ukunquma ukuthi “yiziphi iziqalo esezizinzile ngokwanele ukuthi zithunyelwe”.
Ukukhipha nge-Commit/Wait: lapho kufika ingxenye encane yomsindo, imodeli iqala ihlole isiqalo esizinzile samanje; uma sizinzile ngokwanele, isenza i-commit sibe umbhalo ongaguquleki, kungenjalo ilinda futhi igcine omunye umongo.
Ukukhiqiza okuncike kumongo: kusetshenziswa kuphela isiqalo esizinzile njengesimo sokubikezela okulandelayo, okunikeza imingcele eyanele yencazelo futhi kuqinisekise ukuthi okukhiphwe ngaphambilini akuguqulwa umsindo olandelayo.
Ukulungisa ukubambezeleka nokunemba: usayizi we-chunk, ubude bomsila/iwindi lokubuyela emuva namanye amapharamitha kulawula umthamo wokucubungula, ukubambezeleka nokubekezelela amaphutha. Ama-chunk amancane ayashesha kodwa ayingozi kakhulu; amakhulu azinza kakhulu kodwa ayanensa.

Clone indawo yokugcina: git clone https://github.com/netease-youdao/Confucius4-R2T2.git.
Dala indawo ehlukanisiwe: sebenzisa i-Conda noma i-uv ukuze wakhe indawo ye-Python 3.12 bese usebenzisa pip install -e ., noma usebenzise ngokuqondile i-Docker esemthethweni ye-qwenllm/qwen3-asr.
Landa izisindo: khetha i-Confucius4-R2T2 noma i-Confucius4-R2T2-GGUF ku-HF/ModelScope kuye nge-backend.
Qala ngesibonelo se-offline/streaming: ./run_example.sh audio.wav --model_path ... --infer_mode stream_vllm --language Chinese --chunk_size_ms 160.
Bhala ikhodi yokusakaza: layisha i-R2T2ASRModel.LLM, ngemva kwe-init_streaming_state() shayela i-streaming_transcribe() ngokuphindaphindiwe ngama-chunk e-16kHz, bese ekugcineni ushayele i-finish_streaming_transcribe().
Ukuhambisa ku-CPU/llama.cpp: sebenzisa imodi ye-stream_llama_hybrid ye-r2t2_llama noma imodi ye-stream_llama/onetime_llama.
Qalisa isevisi yesikhathi sangempela: sebenzisa ./run_start_server.sh start --model_path ... --vad_model_path checkpoints/vad/Stream-VAD --port 8272 --gpu 0.
Xhuma iklayenti: thumela i-16kHz mono int16 PCM ku-ws://localhost:8272/asr_stream_api_v1, futhi ekugcineni uthumele i-YOUDAO_ONETIME_ASR_STREAM_EOS.
Lungisa amapharamitha olwazi lomsebenzisi: sebenzisa i-language, i-chunk_size_ms, i-context/hotword kanye ne-unfixed_token_num ukuze ulinganise ukubambezeleka, ukunemba nokutholwa kwamagama omkhakha.

Ukusakaza kwangempela ngaphandle kokubuyekeza: umbhalo osukhishiwe “awubuyekezwa”, kugwenywe ukuqhwanyaza kwemibhalo engezansi nokwenziwa kwezenzo ezingalungile yi-Agent.
Ukubambezeleka okuphansi: ngokusho komkhiqizi, isilinganiso singaba ngu-200–600ms, okulungele imibhalo engezansi yesikhathi sangempela, ukuhumusha ngesikhathi esisodwa nama-voice Agent.
Ukubambezeleka nokunemba okulungisekayo: isekela ama-chunk angu-80ms–2s, okuvumela ukushintsha phakathi kwejubane nokuzinza kuye ngesimo.
Ikhono le-offline eliqinile: imingcele yokusakaza ayinciphisi kakhulu ukunemba kwe-offline, ngakho imodeli eyodwa ingasebenza kuzo zombili izinhlobo zemisebenzi.
Isisekelo ne-ecosystem enamandla: yakhelwe phezu kwe-Qwen3-ASR, futhi ngemva kokukhishwa kwayo kwavela ngokushesha ukusekelwa kwe-GGUF, llama.cpp, Core ML namathuluzi e-offline.
Ilungele ukuhlanganiswa kwezinhlelo: inikeza ama-backend e-vLLM, Transformers, llama.cpp kanye nesevisi ye-WebSocket, okwenza kube lula ukuyihlanganisa ekukhiqizeni.
Umongo olawulwayo wokufaka: isekela ama-prompt omongo namagama abalulekile, ithuthukise ukubonwa kwamagama abantu, amagama emikhiqizo namagama obuchwepheshe.
Ukusekelwa kwezilimi eziningi: ithuthukiselwe isiShayina nesiNgisi, kuyilapho igcina amandla okusakaza ngezilimi eziningi.

Iwebhusayithi esemthethweni yephrojekthi: https://r2t2.ai/
Indawo yokugcina ye-GitHub: https://github.com/netease-youdao/Confucius4-R2T2
Indawo yamamodeli ye-HuggingFace: https://huggingface.co/netease-youdao/Confucius4-R2T2

Imibhalo engezansi yemihlangano yesikhathi sangempela: imibhalo engezansi yokugcina ivela ngesikhathi umuntu ekhuluma, kuncishiswe ukuphazamiseka kokufunda okubangelwa ukulungiswa kombhalo.
Indawo yokufaka imiyalo ye-Voice Agent: ibona ngokuzinza imisho yokuzilungisa efana nokuthi “dlulisa ama-yuan angu-300, shintsha kube ngu-3,000”, kuncishiswe ingozi yokuthi i-Agent yenze isenzo esingalungile.
Ukusebenzisana ngezwi ezimotweni nakumadivayisi ahlakaniphile: ikhipha umbhalo wemiyalo ongabuyiselwa emuva ngaphansi kwemfuneko yokubambezeleka okuphansi, okulungele ukuzulazula, izingcingo nokulawula imoto.
Usizo lwesikhathi sangempela ezikhungweni zezingcingo: ibhala inkulumo ngesikhathi esisodwa bese iyithumela kumeluleki ukuze athole iziphakamiso, kugcinwe ibhalansi phakathi kokubambezeleka, amagama abalulekile omkhakha nokuhambisana nemithetho yokusetshenziswa kwangasese.
Ukubhala ngesikhathi sangempela emakilasini nasekufinyeleleni kwabantu abakhubazekile: kunikeza abantu abanenkinga yokuzwa noma abasebenzisa amanothi ukusakazwa kwamagama okuzinzile, kuncishiswe ukuqhwanyaza nezindleko zokulungisa kamuva.