Isifundo Esiphelele se-SFighterAI: Qeqesha i-ejenti ekwazi ukuqeda i-Street Fighter 2 nge-PPO

Isingeniso Sephrojekthi

Ultimate SFighterAI yiphrojekthi yokufunda okujulile ngokuqinisa ebhekiswe ku-《Street Fighter 2: Champion Edition》. Isebenzisa u-Ryu njengomlingiswa womdlali, iqeqesha imodeli ye-NatureCNN nge-algorithm ye-PPO, futhi yandisa ukuqeqeshwa kuwo wonke amazinga angu-12 e-Street Fighter.

Okokufaka kwezinqumo zemodeli kuvela ngokuphelele kumaphikseli e-RGB esithombe somdlalo; ayifundi ngokuqondile indawo yomlingiswa, isimo sesenzo noma impilo. Indawo yokuqeqesha isebenzisa kuphela okuguquguqukayo kwenkumbulo agent_hp kanye enemy_hp ukubala imiklomelo nokunquma umphumela.

Iphrojekthi isivele inesisindo semodeli eqeqeshiwe namarekhodi okuqeqesha, ngakho ungabuka izimpi ze-AI ngokuqondile, uphinde wenze ukuhlola, ulungise inani lezindawo ezisebenza ngokufanayo, noma uqhubeke nokuqeqesha usebenzisa imodeli ekhona.

Imisebenzi Eyinhloko Nezici Zokusetshenziswa

• Imboza wonke amazinga angu-12: Izinga ngalinye lingasebenza kusifanisi esizimele, futhi wonke amasifanisi aqeqesha imodeli efanayo ye-NatureCNN.
• Izinqumo ezisekelwe embonweni kuphela: Imodeli ikhetha ukunyakaza, ukuhlasela namasu akhethekile ngokusekelwe esithombeni somdlalo esicutshunguliwe.
• Izenzo ezihlukene ezingu-27: Kuhlanganisa ukuma, ukunyakaza, ukuhlasela okuvamile, ukuhlasela ugobile, kanye ne-Hadouken, i-Tatsumaki Senpu-kyaku ne-Shoryuken eziqondiswe kwesokunxele noma kwesokudla.
• Ukufunda kokuqinisa kwe-PPO: Iphrojekthi iqeqeshwa nge-algorithm ye-PPO ye-Stable-Baselines3.
• Izindlela eziningi zokuhlola: Isekela ukuhlola ngokulayisha izilondolozi zezinga ngalinye, ukunqoba imidlalo emibili kwemithathu, ukuhlola kwe-KO eyodwa, kanye nokuqhubeka kusukela ezingeni lokuqala kuze kuphele umdlalo.
• Imodeli eqeqeshwe kusengaphambili: Isisindo esimisiwe sitholakala ku-logs/trained_models/ppo_ryu.zip.
• Ukubonisa ukuqeqeshwa: Ungasebenzisa i-TensorBoard ukubuka izinguquko zemiklomelo namarekhodi okubuyekezwa kwemodeli.
• Ukusekelwa kwamadivayisi amaningi: Amapharamitha okuqeqesha nawokusebenzisa imodeli asekela auto, cpu, cuda kanye mps.

Ukucutshungulwa Kwesithombe

Isithombe sokuqala somdlalo sinqunywa phakathi, kukhishwe ingaphambili elinyakazayo, bese sincishiswa, ukuze kwakheke okokufaka kwemodeli okungu-84×84×3. Ukukhishwa kwengaphambili elinyakazayo kusebenzisa umehluko ompunga wezikhathi ezintathu, u-t-16, u-t-8t, kuphindaphinde ukukhanya kwangemuva ngo-0.35, bese kuhlanganisa iziteshi ezintathu u-R, u-G no-B. Lokhu kusiza imodeli ukubona ukunyakaza nokuhlasela kwabalingiswa.

Indlela Yemiklomelo

Umklomelo wakhiwa ikakhulu izinguquko empilweni yabaphikisi bobabili, umphumela wokuwisa, kanye nezindleko zesikhathi. Umklomelo womonakalo ubalwa kanje:

0.01 × (3 × impilo elahlekile yomphikisi − impilo elahlekile eyakho)

Umklomelo wokuwisa umphikisi uthi:

0.01 × (176 + impilo yakho esele)

Isijeziso sokuwiswa nguwe sithi:

0.01 × (176 + impilo yomphikisi esele)

Ngaphezu kwalokho, isenzo ngasinye esenziwa ngempela sithola isijeziso sesikhathi, esizenzakalelayo esingu-0.0005.

Incazelo Yesakhiwo Sephrojekthi

• data/: Igcina izilondolozi zokuqala zamazinga angu-12, izincazelo zokuguquguquka kwenkumbulo yomdlalo kanye nokucushwa kwezigcawu.
• main/train.py: Iphethe ukuqeqeshwa okuhambisanayo kwawo wonke amazinga nokuqhubeka nokuqeqesha.
• main/test.py: Iphethe ukuhlolwa kwezinga ngalinye nokuhlola ukuqhubeka kwamazinga.
• main/env_utils.py: Iphatha ukucushwa kwe-ROM nokwakhiwa kwendawo.
• main/custom_wrappers.py: Isebenzisa ukucutshungulwa kwesithombe, ukumappingwa kwezenzo nokubalwa kwemiklomelo.
• logs/trained_models/ppo_ryu.zip: Imodeli eqeqeshiwe enikezwe yiphrojekthi.
• logs/: Iqukethe amarekhodi e-TensorBoard enqubo yokuqeqesha.
• requirements.txt: Uhlu lokuncika kwe-Python.

Ukufakwa Nokucushwa Kwendawo

Isinyathelo Sokuqala: Lungisa Indawo Ye-Python

Iphrojekthi incoma ukusebenzisa i-Python 3.10. Indawo eqinisekisiwe ebhalwe ku-README yi-macOS, i-Apple Silicon ne-Python 3.10.20, futhi kwaqinisekiswa izindlela ze-CPU ne-Apple MPS. Kwamanye amapulatifomu, bheka imiyalelo esemthethweni ye-Stable-Retro ne-PyTorch.

Dala futhi uvule indawo ebonakalayo empandeni yephrojekthi, bese ufaka okuncikile:

python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Abasebenzisi be-Windows bangasebenzisa lo myalo ukuvula indawo ebonakalayo:

.venv\Scripts\activate

Ungasebenzisa futhi i-Anaconda ukudala indawo ye-Python 3.10, bese ufaka okuncikile okuku-requirements.txt. Iphrojekthi incike kakhulu ku-Gymnasium, ku-Stable-Retro naku-Stable-Baselines3.

Isinyathelo Sesibili: Lungisa I-ROM Yomdlalo

I-Stable-Retro ayinikezi ama-ROM omdlalo, futhi le phrojekthi ayifaki i-ROM. Sicela uthole ngokusemthethweni i-ROM yase-USA esekelwayo ye-《Street Fighter 2: Champion Edition》.

Ngemva kokuthola i-ROM, ungayinikeza usebenzisa okuguquguqukayo kwendawo. Ku-macOS noma ku-Linux sebenzisa:

export SF2_ROM_PATH="/absolute/path/to/your/rom.md"

Ku-Windows PowerShell sebenzisa:

$env:SF2_ROM_PATH = "C:\path\to\your\rom.md"

Enye indlela ukubeka i-ROM endaweni emisiwe ngaphakathi kwephrojekthi:

data/street-fighter-ii-special-champion-edition-usa.md

Izinhlelo zokuqeqesha nezokuhlola zizongenisa i-ROM ngokuzenzakalelayo. Uma i-Stable-Retro isivele ingenise le ROM endaweni yamanje ye-Python, ungayisebenzisa ngokuqondile.

Isinyathelo Sesithathu: Qinisekisa Idatha Yamazinga

Amafayela angu-12 e-.state akumkhombandlela we-data/ agcina izimo zokuqala zamazinga 1 kuya ku-12 ngokulandelana. I-data.json kanye ne-scenario_full_frame.json zichaza okuguquguqukayo kwenkumbulo yomdlalo nokucushwa kwezigcawu. Uhlelo luzowafunda ngokuqondile lawa mafayela; akudingekile ukuwakopisha kumkhombandlela wokufakwa kwe-Stable-Retro.

Ukusebenzisa Imodeli Eqeqeshwe Kusengaphambili

Hlola Wonke Amazinga angu-12 Ngezinga Ngezinga

Ngemva kokuqedela ukucushwa kwendawo, sebenzisa lo myalo empandeni yephrojekthi:

python -m main.test --episodes 1

Uhlelo luzolayisha ngokulandelana izilondolozi zokuqala zamazinga 1 kuya ku-12, futhi izinga ngalinye ngokuzenzakalelayo lisebenzisa ukunqoba imidlalo emibili kwemithathu. --episodes 1 kusho ukuhlola okuphelele okukodwa; uma le pharamitha ingafakiwe, ngokuzenzakalelayo kuhlolwa imijikelezo engu-10.

Uma izinga elithile lihluleka, uhlelo luzoqhubeka nokuhlola amazinga alandelayo. Itheminali izobonisa ukunqoba nokuhlulwa kwezinga ngalinye, imiklomelo, izinga lokunqoba lezinga ngalinye, kanye nengxenye yemijikelezo yokuhlola lapho wonke amazinga angu-12 enqotshiwe.

Vala Isithombe Ukuze Ugxile Emiphumeleni Yokuhlola

Uma ungadingi ukubuka umdlalo, ungavala ukuboniswa futhi usebenzise ngokucacile i-CPU ekusebenziseni imodeli:

python -m main.test --episodes 1 --no-render --device cpu

Ukuvala ukuboniswa kulungele ukuhlola ngobuningi futhi kunciphisa izindleko ezengeziwe zokubonisa.

Qhubeka Kusukela Ezingeni Lokuqala

Ukuze ubuke i-AI iphonsela inselelo umphikisi olandelayo ngokulandela ukugeleza komdlalo, sebenzisa:

python -m main.test --episodes 1 --no-load-level-saves

Le modi ilayisha kuphela isilondolozi sezinga lokuqala ekuqaleni, bese umdlalo uqhubeka ngokujwayelekile. Uma ihluleka, lowo mzamo uyaphela. Uhlelo luzobonisa izinga lokunqoba lamazinga asekuhloliwe kanye nezinga lokuqeda wonke amazinga.

“Ukunqoba konke” ekuhlolweni kwezinga ngalinye kusho ukuthi imodeli iphumelele ukuhlolwa kuzo zonke izilondolozi zamazinga angu-12 ezizimele; “izinga lokuqeda wonke amazinga” lidinga ukuthi imodeli iqede wonke amazinga angu-12 ekugelezeni okukodwa okuqhubekayo. Le miqondo emibili ayifani.

Amapharamitha Okuhlola Asetshenziswa Kaningi

• --model-path PATH: Icacisa imodeli ye-PPO, ngokuzenzakalelayo logs/trained_models/ppo_ryu.zip.
• --episodes N: Isetha inani lemijikelezo yokuhlola kwezinga ngalinye noma imizamo yokuqhubeka kwamazinga, ngokuzenzakalelayo 10.
• --no-render: Ivala isithombe somdlalo.
• --no-load-level-saves: Ayilayishi izilondolozi zezinga ngalinye, kodwa iqala kusukela ezingeni lokuqala iqhubeke.
• --reset-round: Ekuhlolweni kwezinga ngalinye, isebenzisa i-KO eyodwa njengeyunithi yokuhlola esikhundleni sokunqoba imidlalo emibili kwemithathu okumisiwe.
• --device auto/cpu/cuda/mps: Ikhetha idivayisi yokusebenzisa imodeli, ngokuzenzakalelayo auto.
• --speed N: Isheshisa ukuboniswa ngokunciphisa inani lamafreyimu aboniswayo, ngokuzenzakalelayo 4.

Qeqesha Imodeli Yakho

Qala Ngokucushwa Okuncane

Izidingo zezinsiza zokucushwa okuzenzakalelayo ziphezulu kakhulu. Ekuqaliseni kokuqala, kunconywa ukuthi uqale ngesifanisi esisodwa ezingeni ngalinye futhi usebenzise iqoqo elincane:

python -m main.train --envs-per-level 1 --batch-size 256 --log-dir runs/ryu

Ngenxa yokuthi iphrojekthi inamazinga angu-12, --envs-per-level 1 kusho ukuthi kuzosebenza amasifanisi angu-12 ahambisanayo. Qinisekisa kuqala ukuthi i-ROM, ukwakhiwa kwendawo, idivayisi nokukhishwa kwamalogi kusebenza kahle, bese ukhulisa kancane inani lezindawo ezisebenza ngokufanayo.

Qonda Ubukhulu Obumisiwe Bokuqeqesha

Uma usebenzisa ngokuqondile python -m main.train, ngokuzenzakalelayo kuqalwa amasifanisi angu-64 ezingeni ngalinye, okwenza kube nezindawo ezihambisanayo ezingu-768 sezizonke. Ngokuhambisana ne-n-steps=128 emisiwe, ukubuyekezwa okukodwa kwe-PPO kuqoqa amasampuli angu-98,304, kanti inhloso ephelele yokuqeqesha izinyathelo eziyizigidi ezingu-700.

Ukucushwa okumisiwe kudinga i-CPU nenkumbulo eyanele, futhi akufanele kusetshenziswe njengokuhlola kokuqala kukhompyutha evamile. Yehlisa --envs-per-level, --batch-size noma inani eliphelele lezinyathelo zokuqeqesha ngokuvumelana nehadiwe ekhona.

Amapharamitha Okuqeqesha Asetshenziswa Kaningi

• --envs-per-level N: Inani lezindawo ezifanayo ezisebenzayo ezingeni ngalinye; uma kungacaciswanga, izindawo ezingu-64 ezingeni ngalinye.
• --total-timesteps N: Inani eliphelele lezinyathelo zokuqeqesha kulokhu kuqeqeshwa; lapho kuqhubekwa khona ukuqeqeshwa, limelela izinyathelo ezengeziwe, inani elizenzakalelayo lingu-700000000.
• --n-steps N: Inani lezinyathelo eziqoqwa yisimulator ngayinye ngaphambi kokubuyekeza, inani elizenzakalelayo lingu-128.
• --batch-size N: Usayizi wenqwaba yokubuyekeza ye-PPO, inani elizenzakalelayo lingu-8192.
• --n-epochs N: Inani lemijikelezo yokuqeqesha idatha eqoqwe enqwabeni ngayinye, inani elizenzakalelayo lingu-3.
• --learning-rate N: Izinga lokufunda, inani elizenzakalelayo lingu-0.0002.
• --device auto/cpu/cuda/mps: Khetha idivayisi yokuqeqesha, inani elizenzakalelayo lingu-mps.
• --log-dir PATH: Uhla lwemibhalo lokukhipha imodeli namagrafu okuqeqesha, okuzenzakalelayo ngu-logs/ALL_MOTION_SIMULTANEOUS.

Ukuqhubeka nokuqeqesha ngesisindo esiqeqeshwe ngaphambilini

Le phrojekthi isekela ukuqhubeka nokuqeqesha kusukela esisindweni se-PPO esinikeziwe. Umyalo ongezansi wengeza izinyathelo zokuqeqesha eziyisigidi futhi ubhale imiphumela kuhla olusha lwemibhalo:

python -m main.train \
  --resume-model logs/trained_models/ppo_ryu.zip \
  --envs-per-level 1 --batch-size 256 \
  --total-timesteps 1000000 --log-dir runs/ryu-resume

Lapho kuqhubekwa khona ukuqeqeshwa, --total-timesteps limelela inani lezinyathelo zokuqeqesha ezengeziwe ezenziwa kulokhu kuqhutshwa, hhayi umgomo oqoqekayo ohlanganisa inani lokuqeqeshwa kwemodeli endala.

Ukubuka amagrafu okuqeqesha nge-TensorBoard

Le phrojekthi iqukethe amarekhodi okuqeqesha. Qalisa i-TensorBoard:

tensorboard --logdir=logs

Bese uvula http://localhost:6006/ esipheqululini ukuze ubuke izinguquko zemiklomelo namagrafu ahlobene nokubuyekezwa kwemodeli.

Uma ufuna ukubuka amalogi akhiqizwe ukuqeqeshwa kwakho, vele ushintshe uhla lwemibhalo:

tensorboard --logdir=runs/ryu

Iziphakamiso zokusebenzisa ezithuthukile

1. Qinisekisa kuqala ngaphambi kokwandisa: Sebenzisa indawo eyodwa ezingeni ngalinye ekuqaleni ukuze uqeqeshe isikhathi esifushane, uqinisekise ukuthi ukungeniswa kwe-ROM, ukufundwa kwesilondolozo nokubhalwa kwamalogi kusebenza kahle, bese wandisa inani lama-simulator asebenza ngokufanayo.
2. Cacisa idivayisi ngokweplatifomu: Ku-Apple Silicon ungazama mps; ezindaweni ezine-NVIDIA GPU ehambisanayo ungakhetha cuda; ngaphandle kwalokho sebenzisa cpu noma auto. I-README iqinisekiswe endaweni ye-macOS Apple Silicon; kwamanye amapulatifomu, lungiselela ngokwezidingo ze-PyTorch ne-Stable-Retro.
3. Hlukanisa izinhlobo ezimbili zokuhlola: Ukuhlola izilondolozo ngokwezinga kulungele ukuhlaziya ukusebenza kwabaphikisi ngabanye, kuyilapho ukudlula amazinga ngokulandelana kusondela kakhulu enkambisweni ephelele yomdlalo. Lapho ubika imiphumela, kufanele ucacise imodi esetshenzisiwe.
4. Sebenzisa ukuhlola kwenqwaba ngaphandle kokunikezwa kwezithombe: Lapho wenza izibalo zemijikelezo eminingi, engeza --no-render; uma ubuka ukusebenza kwecebo, vula ukuboniswa kwezithombe futhi.
5. Gcina ukuhlolwa kuhla oluzimele lwemibhalo: Sebenzisa --log-dir ukwakhela izinhlanganisela zamapharamitha ezahlukene amahla azo, ugweme ukuthi amarekhodi okuqeqesha abhalelane futhi wenze kube lula ukuziqhathanisa ku-TensorBoard.
6. Buka izinketho eziphelele zomyalo: Imibhalo yokuqeqesha neyokuhlola yomibili inikeza ulwazi losizo.

python -m main.train --help
python -m main.test --help

Ngaphandle kwendlela yamamojuli, le phrojekthi iphinde isekele ukusebenzisa imibhalo ngokuqondile:

python main/train.py
python main/test.py

Isiphetho

Ultimate SFighterAI ikhombisa indlela yokuhlanganisa okokufaka kwamaphikseli, ukumakwa kwezenzo, ukwakheka kwemiklomelo, i-PPO nezindawo ezifanayo ezinamazinga amaningi ukuze kwakhiwe i-ejenti ephelele yomdlalo wokulwa. Indlela ethembeke kakhulu yokuqala iwukumisa i-ROM bese usebenzisa imodeli eqeqeshwe ngaphambilini, uphinde wenze ukuqeqeshwa ngenani eliphansi lezindawo ezifanayo, bese wandisa kancane kancane ububanzi bokuhlolwa ngokuqhubeka nokuqeqesha, ukukhetha idivayisi nokuhlaziya nge-TensorBoard.

Le phrojekthi ikhishwa ngaphansi kwe-Apache License 2.0 futhi isebenzisa izingxenye zomthombo ovulekile ezifana ne-Gymnasium, i-Stable-Retro ne-Stable-Baselines3.