Ukwakha Igumbi Lokusakaza Bukhoma le-AI Kusukela Ekuqaleni: Umuntu Wedijithali, Ukuhlela Imiyalezo Yengxoxo kanye Nokudlulisa nge-OBS

Isingeniso sephrojekthi

Igumbi lokusakaza bukhoma le-AI yiphrojekthi yomthombo ovulekile evumela umuntu wedijithali we-AI ukuthi asakaze bukhoma ngokuzenzakalelayo. Uma ababukeli bethumela amazwana, uhlelo luqala ngokuhlunga okuqukethwe, bese luwabeka kulayini ngokwesimo samanje sezwi nokunyakaza, ukuze umyalo olandelayo ungafakwa ngaphambi kokuba umusho wokuqala uqedwe.

Iphrojekthi isebenzisa i-Vidu S2 real-time digital human ukukhiqiza isithombe, umsindo nezimpendulo. Le phrojekthi iphethe ukuhlunga nokuhlela amazwana, imininingwane yomlingiswa, ikhabethe lezingubo, izendlalelo zegumbi lokusakaza, indawo yokusebenza yasendaweni, kanye nokukhiphela ku-OBS noma ukusakaza bukhoma ngokuqondile.

Ikhodi inelayisensi ye-MIT. Indawo yokugcina ayifaki umfanekiso wombhali, ubuntu, ithoni yezwi noma iqoqo lezinto zakhe. Ngakho-ke, ngemva kokuyifaka, kufanele ulungiselele izimpahla zomlingiswa wakho futhi uzithathele wena umthwalo wamalungelo okuqukethwe nokuhambisana nemithetho.

Imisebenzi nemikhawulo

Amakhono asekelwayo njengamanje

• Ukuphendula izingxoxo ngokobuntu obuchazwe ngumsebenzisi, kanye nokulungisa amazwi okuvula.
• Ukushintsha isethi ephelele yezingubo ekhabetheni elilungiselelwe kusengaphambili; ukushintsha izingubo kuthatha cishe imizuzwana engu-6 ngesikhathi.
• Ukusebenzisa amagama angukhiye ukwenza ukunyakaza okuncane njengokwenza uphawu lwenhliziyo ngezandla, ukuny waving nokucwayiza.
• Uma kungekho okushiwo ababukeli, ukuqalisa ingxoxo ngokuzenzakalelayo ngezihloko ezisohlwini lomlingiswa, kugwenywe ukuphinda ngaphambi kokuphela komjikelezo wesihloko.
• Ukuhlunga ngokuzenzakalelayo amazwana angcolile noma aqondisa abantu kwenye indawo; izipho azidluli emgqeni.
• Ukukhetha ukuvula ukuqala kabusha kokusakaza ngokuzenzakalelayo ngemva kokunqanyulwa phakathi nendawo yi-Vidu.

Imikhawulo okufanele iqashelwe

• Ngokuzenzakalelayo, ukushintsha ingemuva, ukudla nokuphuza, kanye nokuphatha izinto akuvunyelwe, ngoba ukudwetshwa kabusha ngesikhathi sangempela kungaphazamisa izingubo zomlingiswa.
• Okwamanje le phrojekthi ayifanele ukucula, ukudlala umculo, ukuma, ukunyakaza komzimba wonke, ukudansa kakhulu noma ukuthengisa imikhiqizo.
• Ngeke ikhumbule ababukeli abadala phakathi kwezikhathi zokusakaza ezihlukene.
• Amazwana e-Douyin awanawo umsebenzi owakhelwe ngaphakathi wokuwathatha. Sebenzisa isixhumi esisemthethweni esivulekile senkundla; ungathathi okuqukethwe ekhasini legumbi lokusakaza.

Imisebenzi yokushintsha ingemuva nezinto ivaliwe ngokuzenzakalelayo. Uma uhlola ngasese kuphela, ungasetha {"safety":{"live_visual_edits":true}} ku-config.json. Akunconyiwe ukuyivula ekusakazeni komphakathi.

Ukulungisa indawo yokusebenza

Ngaphambi kokuqala, lungisa lezi zindawo nama-akhawunti:

• i-macOS noma i-Linux.
• i-Python 3.11 noma inguqulo yakamuva; umbhali usebenzisa i-Python 3.13.
• i-ffmpeg ene-libx264 ne-aac.
• I-API Key yeplatifomu evulekile ye-Vidu, eqala ngo-vda_, futhi uqinisekise ukuthi i-akhawunti inamaphuzu atholakalayo.
• I-API Key ye-Evolink ongayikhetha, yokukhiqiza izithombe zomlingiswa, izingubo, izinto nezizinda, futhi ingasetshenziswa ukulayisha umsindo lapho kuhlanganiswa izwi.
• I-OBS 30 noma inguqulo yakamuva ongayikhetha, yekhamera, izendlalelo nokusakaza.
• Ikhodi yokusakaza ye-YouTube ongayikhetha; iyadingeka kuphela uma usebenzisa ukusakaza okwakhelwe ngaphakathi.

Umuntu wedijithali wesikhathi sangempela we-Vidu ukhokhiswa ngobude besikhathi. Intengo esemthethweni eshiwo ku-README ingamaphuzu angu-1.5 ngomzuzwana, iphuzu elingu-1 lilingana no-0.03125 yuan, kanti ihora elilodwa libiza cishe ama-yuan angu-169. Kunconywa ukuthi uqale ngemodi yokulingisa kanye nesevisi yasendaweni yokuhlola ukugeleza komsebenzi.

Ukugcina okhiye ngokuphepha

Okhiye kufanele bagcinwe kuphela kumafayela akule khompyutha futhi bafundwe yiseva; ngeke bathunyelwe esipheqululini. Qala ngokwakha izinkomba ezizenzakalelayo:

mkdir -p ~/.deepsea/vidu ~/.deepsea/evolink && chmod 700 ~/.deepsea/vidu ~/.deepsea/evolink

Ku-macOS, ungagcina i-Vidu Key esebhodini lokunamathisela endleleni ezenzakalelayo:

pbpaste > ~/.deepsea/vidu/api.key && chmod 600 ~/.deepsea/vidu/api.key

I-Key ye-Evolink ibekwa ku-~/.deepsea/evolink/api.key. Uma usebenzisa ukusakaza kwe-YouTube okwakhelwe ngaphakathi, faka ikhodi yokusakaza ku-~/.deepsea/youtube/stream.key; iphasiwedi ye-OBS WebSocket ibekwa ku-~/.deepsea/obs/ws.password. Izindlela ezizenzakalelayo zingashintshwa naku-config.json.

Ukufaka iphrojekthi

Clone indawo yokugcina ku-GitHub bese ungena enkombeni yephrojekthi:

git clone https://github.com/garycaogan/ai-live-avatar.git
cd ai-live-avatar

Dala indawo ebonakalayo ye-Python bese ufaka okuncikile:

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

Ikhasi lesiteji lincike ku-Alibaba Cloud RTC Web SDK. Iphrojekthi ayisabalalisi le SDK kabusha; kunalokho, iskripthi siyilanda ku-CDN esemthethweni bese siyayiqinisekisa:

./tools/fetch_vendor.sh

Ekugcineni kopisha ifayela lokumisa eliyisibonelo:

cp config.example.json config.json

Ukuqalisa indawo yokusebenza

Qalisa isevisi:

.venv/bin/python server.py

Isevisi ilalela kule khompyutha kuphela ngokuzenzakalelayo. Ngemva kokuqala, vula http://127.0.0.1:28900/workbench esipheqululini ukuze ungene endaweni yokusebenza.

Uma okwamanje ungafuni ukusebenzisa amaphuzu e-Vidu, ungasebenzisa ukuqala ukusakaza kokulingisa ekhasini legumbi lokusakaza. Le modi ayibizi i-Vidu; ibonisa kuphela ukuhlunga, ukufaka emgqeni nokusebenzisa izendlalelo zamazwana. Iphrojekthi iphinde inikeze nge-tests/dev_mock_stack.py, exhuma ku-Vidu elingisiwe lendawo ukuze ikusize uhlole ngokuphelele ukugeleza kokulawula.

Ukwakha umlingiswa wakho wedijithali

1. Lungisa isithombe somlingiswa

Layisha isithombe somlingiswa ekhasini lezimpahla, noma usikhiqize usebenzisa i-Evolink. Isithombe esiyisisekelo sithinta ubuso bomlingiswa, izingubo, igumbi nesimo sobuso esihlala sikhona. I-README incoma isithombe esiseduze somuntu oyedwa, izandla zombili phezu kwetafula, nesimo sobuso esingokwemvelo.

Ukwakheka okuvundlile kuzofakwa ebhokisini elincane elingaphezulu kwekhasi lokusakaza elime mpo elifana ne-Douyin futhi kunqunywe ngesilinganiso esingu-1.2:1, ngakho-ke isithombe esiyisisekelo kufanele sihlelwe kusengaphambili ngokuvumelana nalokho.

2. Dala imininingwane yomlingiswa

Ekhasini labalingiswa, dala umlingiswa omusha, khetha isithombe somlingiswa njengesithombe esiyisisekelo, bese ugcwalisa igama, ubuntu namazwi okuvula. Ubuntu kufanele buchaze kakhulu isitayela sokukhuluma; ungabhali imisho eyisibonelo ephelele, ngoba imodeli ingase iyifunde ngqo.

3. Engeza ikhabethe

Engeza amasethi ambalwa ezingubo ekhasini lekhabethe. Isethi ngayinye kufanele ibe isithombe esiphelele, futhi igumbi, ukukhanya nokwakheka kufanele kufane ngangokunokwenzeka nesithombe esiyisisekelo; ngaphandle kwalokho kungase kube nokugxuma okubonakalayo lapho kushintshwa izingubo.

4. Lungisa izwi

Ekhasini lezwi, khetha ithoni yezwi yesistimu noma ulayishe imizuzwana engu-10 kuya ku-20 yezwi elihlanzekile ukuze uhlanganise izwi. Ukuhlanganisa izwi nge-Vidu kuyakhokhelwa; ngaphambi kokulayisha impahla, qinisekisa ukuthi unemvume efanele yokusebenzisa lelo zwi.

5. Lungisa ukunyakaza

Ekhasini lokunyakaza ungahlela ukunyakaza okuhambisana namagama angukhiye amazwana. Kuzilungiselelo zomlingiswa ungahlela futhi ukunyakaza okuncane kwesikhathi sokulinda, ukuze isithombe sibonakale singokwemvelo uma kungekho mazwana amasha.

Imininingwane yomlingiswa igcinwa ku-assets/avatars/profiles.json, kanti ubuntu bugcinwa ku-personas/. Lawa mafayela akhishwe ku-.gitignore, ngakho ngeke afakwe endaweni yokugcina.

Ukuqonda ukugeleza kokucubungula amazwana

Ikhasi legumbi lokusakaza lendawo yokusebenza lisekela ukuthumela amazwana okulingisa ngesandla. Amazwana angempela angathunyelwa ngesixhumi esilandelayo:

curl -s -X POST http://127.0.0.1:28900/api/danmaku \
  -H 'Content-Type: application/json' \
  -d '{"user":"观众A","text":"换件衣服","gift":0}'

Ngemva kokungena ohlelweni, amazwana aqala ahlolwe ngamagama okuhlunga aku-moderation.json, bese i-live/router.py iwahlukanisa ngokwemithetho njengengxoxo, ukushintsha izingubo, into, ingemuva, ukunyakaza noma ukubuyisela esimweni sokuqala. Le nqubo ayibizi imodeli enkulu, ngakho ayengezi ukubambezeleka noma izindleko.

Ngokuzenzakalelayo, izicelo zezinto nezizinda ziphathwa njengezingxoxo ezivamile. Ukuhlonza amazwana kusebenza kangcono uma inkulumo ilula futhi icacile, njengokubuza ngento encane eseduze nomsakazi, ukusebenzisa igama lengubo esekhabetheni ukushintsha izingubo, noma ukucela ukunyakaza okukodwa ngesikhathi.

Lokhu okulandelayo kuvame ukuthinta umphumela: umusho onemisebenzi eminingi, ukubuza ngezindaba zesikhathi sangempela noma isimo sezulu, nokucela ukucula noma ukudlala umculo. Kunconywa ukwehlukanisa izicelo eziyinkimbinkimbi zibe amazwana amaningana alula.

Ukuxhuma ku-OBS

Ekhasini lokusakaza lendawo yokusebenza, shintsha indlela yokukhipha ibe yi-OBS, bese wengeza umthombo wesiphequluli ku-OBS. Khetha ikheli elifanele:

• Ukubonisa umuntu wedijithali kuphela: http://127.0.0.1:28900/stage?obs=1&role=obs&layer=avatar
• Ukubonisa isendlalelo esisobala kuphela: http://127.0.0.1:28900/stage?overlay=1
• Ukubonisa umuntu wedijithali nesendlalelo: http://127.0.0.1:28900/stage?obs=1&role=obs&layer=all

Isithombe esiphelele sinconywa sibe nobubanzi obungu-1080 nokuphakama okungu-1920. Emthonjeni wesiphequluli, khetha “Lawula umsindo nge-OBS”. Njengoba isevisi ilalela kule khompyutha kuphela, i-OBS nesevisi yephrojekthi kufanele kusebenze kukhompyutha eyodwa.

Uma ufuna ukudala izigcawu ze-OBS ngokuzenzakalelayo, ungasebenzisa i-tools/obs_control.py. Isendlalelo sizohlala sibonisa uphawu oluthi “AI real-time generated content”; lokhu kuyimfuneko yesikhulumi ekusakazweni kwabantu bedijithali be-AI futhi kufanele kugcinwe.

Ukulungisa isithombe sokusakaza ku-Douyin

Amagumbi okusakaza abantu bedijithali e-Douyin ngokuvamile asebenzisa isithombe esivundlile. Efonini, isithombe sibekwa endaweni ephezulu, kanti amazwana aboniswa ngezansi. Ungasebenzisa umyalo olandelayo ukuze udale isigcawu esithi “Douyin live room”:

.venv/bin/python tools/obs_control.py setup douyin --camera FaceTime

Lo myalo uzosetha ikhanvasi ye-OBS ibe ngu-1296×1080, okusho isilinganiso esingu-1.2:1, bese ulungisa umuntu wedijithali, ikhadi lencazelo, imibhalo engezansi nekhamera yomuntu wangempela. Ngemva kwe---camera, faka ingxenye yegama lekhamera; uma uyishiya, kusetshenziswa ikhamera yangempela yokuqala etholakele.

Ekusebenzeni kokuqala, i-macOS ingase icele i-OBS imvume yokusebenzisa ikhamera. Vumela lokho bese usebenzisa umyalo futhi ukuze ukunqunywa kwekhamera kusebenze kahle. Izigcawu ezivundlile nezime mpo zingashintshwa noma nini; zombili zisebenzisa isendlalelo esifanayo somuntu wedijithali futhi azisebenzisi izikhala ezengeziwe zokungena.

Kunezindlela ezimbili zokuxhuma ku-Douyin Live Companion: vula ikhamera ebonakalayo ye-OBS bese ukhetha i-OBS Virtual Camera ku-Live Companion; noma wengeze umthombo wewindi ku-Live Companion bese uthwebula iwindi lokubuka kuqala le-OBS. Ikhamera ebonakalayo idlulisa isithombe kuphela, hhayi umsindo, ngakho kufanele uqinisekise ukuthi i-Live Companion iyakwazi ukuthatha umsindo wekhompyutha wesigcawu esithi “Douyin live room”.

Uma i-akhawunti inemvume yokusakaza yenkampani yangaphandle, ungafaka ngokuqondile ikheli lokusakaza elinikezwe yinkundla ku-OBS. Amazwana e-Douyin ngeke angene ngokuzenzakalelayo kule phrojekthi; kufanele ufake isicelo semvume ngesixhumi semidlalo yokusakaza seplatifomu evulekile ye-Douyin, bese uthumela idatha ku-/api/danmaku.

Ukuhlela ikhadi lencazelo legumbi lokusakaza

Amakhadi encazelo asezingxenyeni ezingaphezulu kwesokunxele nesokudla zekhasi lesiteji asetshenziselwa ukubonisa okungenziwa kulokhu kusakaza, okungakenziwa okwamanje, kanye nendlela yokuthumela amazwana asebenza kangcono. Okuqukethwe kuvela ku-assets/stage_panel.json; uma lelo fayela lingekho, kusetshenziswa i-assets/stage_panel.example.json.

Ngemva kokukopisha ifayela lesibonelo, ungalihlela libe okuqukethwe kwakho. Ikhasi lesiteji lizivuselela ngokuzenzakalelayo kungakapheli umzuzu, kanti isendlalelo sokusakaza esakhelwe ngaphakathi sizosebenzisa okuqukethwe okusha ngokushesha. I-outfit_hint ingasetshenziswa ukusetha izeluleko zokushintsha izingubo ezihlukile kumlingiswa ngamunye.

Ukuvula ukuqala kabusha kokusakaza ngokuzenzakalelayo

I-Vidu ingase inqamule ngezikhathi ezithile ngesikhathi sokusakaza. Uma ukubuyisela okuzenzakalelayo kuvuliwe, iphrojekthi izolinda imizuzwana engu-3 bese iqala ukusakaza futhi, izame izikhathi ezingafika ku-6 ngehora. Kodwa ukunqanyulwa okubangelwa amaphuzu anganele noma ukuhlolwa kokuqukethwe ngeke kuqalwe kabusha ngokuzenzakalelayo.

Vula nge-API:

curl -s -X POST http://127.0.0.1:28900/api/settings \
  -H 'Content-Type: application/json' \
  -d '{"auto_recover":true}'

Ungakubeka futhi ku-config.json:

{"schedule":{"auto_recover":true}}

Ngaphambi kokusakaza isikhathi eside, kunconywa ukuthi uqinisekise ibhalansi ye-akhawunti, izimpahla zomlingiswa nesimo senethiwekhi, bese usebenzisa imodi yokulingisa noma isevisi yasendaweni yokulingisa ukuze uhlole inqubo yokubuyisela.

Ukuhlola nesakhiwo sephrojekthi

Ukuhlolwa kwephrojekthi kusebenzisa i-Vidu elingisiwe lendawo futhi akudli mali. Ungasebenzisa:

python -m unittest tests.test_logic tests.test_workbench tests.test_e2e

Ukusetshenziswa kwezinkomba namafayela amakhulu kungokulandelayo:

• server.py: indawo yokuqala yesevisi kanye nokucushwa okuzenzakalelayo.
• live/: ukuhlela ukusakaza bukhoma, i-interface yesikhathi sangempela ye-Vidu, ukuqaphela imiyalezo yababukeli, ikhabethe, ukukhiqizwa kwezinsiza, ukusakaza okwakhelwe ngaphakathi, nama-interface ebhodi lomsebenzi.
• static/: ibhodi lomsebenzi, ikhasi lesiteji, kanye nekhonsoli yokulawula yenguqulo endala.
• WORKBENCH_API.md: wonke ama-interface asetshenziswa ibhodi lomsebenzi.
• tests/: ikhodi yokuhlola kanye nezinsizakalo zokulingisa zasendaweni.

Isiphetho

Le phrojekthi ixhumanisa umuntu wedijithali wesikhathi sangempela we-Vidu, imithetho yemiyalezo yababukeli, izimpahla zabalingiswa, kanye nokukhishwa kokusakaza bukhoma kube ukuhamba komsebenzi kwasendaweni okukodwa. Kunconywa ukuthi uqale ngemodi yokuhlola ukuze ujwayelane nokuhlunga, ukubekwa emgqeni, ikhabethe, nezendlalelo, bese ulungisa isithombe, izwi, nezigcawu zakho ze-OBS, futhi ekugcineni uxhumane nenkundla yangempela.

Lapho usakaza bukhoma esidlangalaleni, kufanele uvale ukuhlela kwangempela kwangemuva nezinsiza okungabangela ukuhlanekezela kwesithombe, usebenzise ama-interface asemthethweni okuxhumana enkundla, futhi ugcine uphawu lokuthi okuqukethwe kukhiqizwe yi-AI. Ngaphezu kwalokho, imvume yezwi lomuntu, isithombe somlingiswa, umculo, nezinye izinsiza kufanele iqinisekiswe kusenesikhathi.