Qeqesha Amamodeli Olimi Asebenza Ngobuhlakani nge-FlashREINFORCE

Yini i-FlashREINFORCE?

FlashREINFORCE iwuhlaka lwe-reinforcement learning olungenaye umgxeki, olwenzelwe ukuqeqesha ngokungavumelanisi amamodeli olimi anama-agent. Luhloselwe imisebenzi ethatha isikhathi eside, lapho ama-trajectory ephela ngezikhathi ezingafani ngenxa yokucabanga okunobude obuguqukayo, izingcingo zamathuluzi, noma ukuxhumana nemvelo.

Esikhundleni sokukhiqiza ama-rollout amaningi ahlobene nge-prompt ngayinye, i-FlashREINFORCE isebenzisa i-rollout eyodwa nge-prompt ngayinye. Ngakho-ke, i-batch equkethe ama-trajectory angu-B ingamboza ama-prompt angu-B ahlukene, futhi ama-trajectory aqediwe angafinyelela kumfundi ngaphandle kokulinda izimpendulo eziningi ze-prompt efanayo.

Indawo yokugcina ikhodi inikeza i-reference loss ezimele ye-PyTorch, isibonelo sokubuyekeza i-optimizer ku-CPU, izivivinyo, ama-launcher e-Molt e-R1 ne-Qwen2.5-Math, kanye nezilungiselelo zokuhlola zokucabanga, amathuluzi e-Python, amamodeli e-mixture-of-experts, ne-ALFWorld. Ama-launcher agcwele okuqeqesha ngokungavumelanisi asebenzisa i-Molt.

I-Reinforcement Learning Kufanele Yenze i-REINFORCE.

Kungani kusetshenziswa i-rollout eyodwa?

Izindlela eziqhathanisa amaqembu zincike kuma-rollout amaningi avela ku-prompt efanayo. Emisebenzini yama-agent engavumelanisi, le mfuneko ingadala izithiyo zokuvumelanisa futhi inciphise inani lama-prompt ahlukene ambozwa yisabelomali esinqunyiwe sama-rollout.

Ukufunda nge-rollout eyodwa kususa imfuneko yama-rollout ahlobene, kodwa kudala izinkinga ezintathu zokuzinza:

• Ayikho iqembu lama-rollout ahlobene okwakhiwa kulo i-baseline yezinga le-prompt.
• Ama-trajectory aqoqwe ngokungavumelanisi kungenzeka akhiqizwe yi-behavior policy endala.
• Ama-rollout amade ahlulekile angabusa ukubuyekeza uma wonke ama-token ethola isisindo esilinganayo ezingeni le-batch.

I-FlashREINFORCE ibhekana nalezi zinkinga nge-One-Batch REINFORCE, i-Sequence Trust Region, kanye ne-Sample-Mean Optimization.

Izingxenye eziyinhloko

1. I-One-Batch REINFORCE

Kuyo yonke i-batch entsha, umfundi ubeka imivuzo yesikali phakathi kwama-prompt azimele. Uma u-R_i kuwumvuzo we-trajectory i, i-advantage yawo ingumvuzo we-trajectory kususwe isilinganiso se-batch:

mean_reward = sum(R_j for j in batch) / B
A_i = R_i - mean_reward

Ama-trajectory angaphezu kwesilinganiso athola ama-advantage amahle, kanti angaphansi kwesilinganiso athola ama-advantage amabi. Lokhu kunikeza impendulo enezimpawu ngaphandle kokuqeqesha umgxeki ofundiwe. Umfundi wenza ukubuyekeza okukodwa kwe-optimizer ku-batch bese eyilahla.

2. I-Sequence Trust Region

I-trajectory ngayinye kufanele igcine amathuba e-behavior policy akhiqize ngempela i-token ngayinye eyisampuliwe. Umfundi ubala kabusha amathuba e-current policy bese enza i-token importance ratio:

rho_i_t = exp(current_log_probability_i_t
              - behavior_log_probability_i_t)

Le ratio ilungisa ukusatshalaliswa kwezenzo ezisampuliwe emlandweni ogciniwe. Ayilungisi iqiniso lokuthi umlando uqobo uvela ku-behavior policy endala, ngakho i-FlashREINFORCE iphinde ihlole ukuchezuka okuqoqiwe kuyo yonke i-trajectory.

Ngesenzo ngasinye esisampuliwe, ibala i-Bernoulli KL proxy kusetshenziswa amathuba e-behavior u-p nawe-current u-q:

d_i_t = p * log(p / q)
        + (1 - p) * log((1 - p) / (1 - q))

Amanani ama-token enziwa isilinganiso kuyo yonke i-trajectory. I-trajectory yamukelwa kuphela uma i-proxy yayo emaphakathi ingeqi umkhawulo wokwethembana omisiwe:

mean_drift_i = sum(d_i_t for t in trajectory_i) / T_i
mask_i = 1 if mean_drift_i <= delta else 0

I-trajectory enqatshiwe ayiniki ama-token loss. Ngakho isinqumo sokwethembana sisebenza kuwo wonke umlando ogciniwe, hhayi kuma-token ahlukanisiwe.

3. I-Sample-Mean Optimization

Isilinganiso se-batch sezinga le-token sinikeza ngokuzenzakalelayo ama-trajectory amade ithonya elikhulu. Lokhu kungaba yingozi uma i-rollout ende ehlulekile iqukethe izinyathelo eziphakathi eziwusizo kodwa iphetha ngomvuzo ongemuhle.

I-FlashREINFORCE iqala ngokwenza isilinganiso seminikelo yama-token ngaphakathi kwe-trajectory ngayinye, bese yenza isilinganiso kuwo wonke ama-trajectory:

trajectory_term_i = (
    mask_i * advantage_i / trajectory_length_i
    * sum(importance_ratio_i_t for each token t)
)
objective = sum(trajectory_term_i for i in batch) / B

Ngenxa yalokho, i-trajectory ngayinye ithola isisindo esingu-1/B kungakhathaliseki ubude bempendulo. I-loss esetshenziswayo isebenzisa ama-importance ratio ahlukanisiwe. Indlela eyisisekelo ayisebenzisi umgxeki ofundiwe, ukusikwa kwe-ratio, noma i-forward pass yemodeli eyireferensi.

Ukufaka nokusetha endaweni

Clone noma dawuniloda i-repository ye-FlashREINFORCE, ngena kufolda yayo eyinhloko, bese uyifaka ngemodi ehlelekayo kanye nezincika zezivivinyo:

pip install -e '.[test]'

Okulandelayo sebenzisa isibonelo esifakiwe se-CPU:

python examples/toy_update.py

Lesi sibonelo sihlola ukuthi i-reference objective ingakwazi yini ukwenza ukubuyekezwa kwe-optimizer ku-CPU. Siyisibonelo sokuqinisekisa ukusebenza, hhayi ukuphindaphinda kwemiphumela yezivivinyo zebhentshimakhi yephepha.

Ekugcineni, sebenzisa iqoqo lezivivinyo:

python -m pytest -q

Isibonelo esiphumelele kanye nokusebenza kwezivivinyo kuqinisekisa ukuthi i-reference implementation yendawo nezincika zayo ziyatholakala.

Hlola i-reference implementation

I-reference objective emaphakathi itholakala ku-flashreinforce/loss.py. Ihlanganisa imisebenzi echazwe ephepheni naku-README:

• Ukubeka imivuzo phakathi nendawo kuyo yonke i-batch.
• Importance sampling yezinga le-token.
• Ukwamukelwa kokwethembana ezingeni le-sequence.
• Sample-mean optimization.
• Ukuhlunga ongakukhetha kwama-token okuhluleka okusekelwe ku-entropy.

Qala nge-examples/toy_update.py ukuze ubone ukuthi i-reference loss ibamba kanjani iqhaza ekubuyekezweni kwe-optimizer. Bese uhlola i-flashreinforce/loss.py ukuze uxhumanise ama-tensor nama-mask esibonelo nezigaba ezithile ze-objective.

Isibonelo se-CPU siyindlela ephephe kunazo zonke yokusetshenziswa okuyisisekelo ngoba asidingi ingqalasizinda ephelele yokusabalalisa ukwethulwa.

Qonda isibuyekezo esingavumelani nesikhathi esidlula kanye

Isibuyekezo se-FlashREINFORCE esisetshenziswa ekukhiqizeni sidlula ezigabeni eziyisithupha:

1. Qoqani: Abasebenzi bokwethula bathumela ngokuzimela i-trajectory eyodwa eqediwe nge-prompt ngayinye. I-trajectory ngayinye ihlanganisa umvuzo wayo namathuba angempela enqubomgomo yokuziphatha asetshenziswe ukukhetha amathokheni ayo.
2. Hlanganisani: Umfundi uthatha ama-trajectory B alandelayo aqediwe bese ebeka imivuzo yawo maphakathi ukuze athole ama-advantage anezimpawu.
3. Lungisani: Umfundi ubala kabusha amathuba e-log enqubomgomo yamanje bese akha izilinganiso zokubaluleka kwamathokheni.
4. Thembani: Ubala i-KL proxy emaphakathi yesenzo esikhethiwe nge-trajectory ngayinye futhi wamukela i-trajectory ephelele kuphela uma inani lingaphakathi komkhawulo omisiwe.
5. Yenzani ukulungiselela: Ulinganisa ngaphakathi kwe-trajectory ngayinye eyamukelwe, alinganise kuwo wonke ama-trajectory e-batch, bese enza isinyathelo esisodwa se-optimizer.
6. Lahlani: Akaphindi asebenzise i-batch eqoqiwe ukuze enze ezinye izibuyekezo zomfundi.

Isinyathelo sokugcina sokulahla siyingxenye ebalulekile yale ndlela. I-FlashREINFORCE isebenzisa isibuyekezo esisodwa se-batch ephelele nge-batch ngayinye esanda kuqoqwa, kunokusebenzisa izibuyekezo eziningana ezilandelanayo zama-minibatch kusukela kusithombe esifanayo senqubomgomo yokuziphatha.

Buka kuqala ukumiswa kokuqeqeshwa kwe-Molt

Indawo yokugcina ikhodi ihlanganisa scripts/train_molt.py ukuze kuqalwe ukuqeqeshwa kusetshenziswa ukuqaliswa kwe-Molt okungavumelani nesikhathi. Ngaphambi kokwabela ukuqeqeshwa okuphelele, sebenzisa imodi yokuhlola ngaphandle kokuqalisa ukuze ubuke kuqala amafulegi okuqeqesha:

python scripts/train_molt.py --dry-run

Bheka docs/training.md ukuze uthole ukusethwa kwe-GPU, ukulungiswa kwedatha, imiyalo yokuqeqesha, imephu esuka ephepheni iye kukhodi, kanye nemikhawulo eyaziwayo yokuphindaphinda imiphumela. Umhlahlandlela examples/README.md uqukethe izilungiselelo zemisebenzi, ama-ablation, kanye nesiqalisi somqeqeshi wendabuko esinokuhlolwa kwamakhono.

Iziqalisi ze-R1 ne-Qwen2.5-Math ziboshelwe ezinguqulweni ezithile endaweni yokugcina ikhodi. Izilungiselelo zokusebenzisa amathuluzi ne-ALFWorld zidinga futhi ama-agent nezindawo ezihambisanayo, ngakho hlola izidingo zazo zesikhathi sokusebenza ngaphambi kokuqala lezo zivivinyo.

Idatha okufanele i-trajectory ngayinye iyigcine

Isibuyekezo sincike olwazini oluqoqwe ngesikhathi sokukhiqiza. Okungenani, ukugeleza kokuqeqesha komqondo kudinga ukulandelana kwamathokheni okuqediwe, umvuzo wako oyinani elilodwa, kanye namathuba okuziphatha ahambisana nesenzo ngasinye esikhethiwe. Umfundi ube esethola amathuba ahambisanayo enqubomgomo yamanje ngokuhlola izenzo nemilando egciniwe.

Ungawashintshi amathuba okwenza isampula aqoshiwe ngamathuba akhiwe kabusha kamuva. Ukuhlolwa kwemodeli okwenziwa kamuva akuqinisekisi ukuphinda kukhiqizwe ithuba injini ye-inference eyalisebenzisa ngempela ngesikhathi sokwenza isampula.

Amathiphu athuthukile okusimama

Gcina ukwesekwa kokwenza isampula

Ukulungiswa okunembile kokubaluleka kucabanga ukuthi inqubomgomo eqondisiwe ayinikezi amathuba ezintweni ezingaphandle kokwesekwa kwenqubomgomo yokuziphatha. Ukunciphisa okunamandla kwe-top-k noma kwe-top-p kungaphula lo mcabango. Ngakho izilungiselelo zokwenza isampula kufanele zikhethwe kucatshangelwa imfuneko yokwesekwa.

Ungaphathi ukwenza isampula kokubaluleka kwamathokheni njengokulungiswa kwe-trajectory ephelele

Izilinganiso zokubaluleka kwamathokheni zilungisa izenzo ezikhethiwe ezinemilando egciniwe. Azilungisi ukusatshalaliswa kwaleyo milando. Sebenzisa i-Sequence Trust Region ukuhlunga ama-trajectory lapho ukuchezuka okuqoqiwe phakathi kokuziphatha nomfundi kukhulu kakhulu.

Gcina izibuyekezo zintsha

Yenza isibuyekezo esisodwa se-optimizer nge-batch ngayinye eqoqiwe bese uyilahla. Ukusebenzisa kabusha i-batch endala ezibuyekezweni eziningana zomfundi kuphambuka endleleni ebhaliwe yokudlula kanye futhi kungandisa ukungafani kwezinqubomgomo.

Gcina ukukala kwezinga le-trajectory

Linganisela iminikelo yamathokheni ngaphakathi kwe-trajectory ngayinye ngaphambi kokulinganisa ama-trajectory kuwo wonke ama-batch. Ukubeka wonke amathokheni avumelekile ku-mean eyodwa ye-batch kungabuyisa isiphindaphindi sobude bempendulo futhi kuvumele ama-trajectory amade athole isisindo esingalingani.

Humusha kahle imaski yokwethembana

I-Sequence Trust Region yenza isinqumo esisodwa sokwamukela i-trajectory ephelele. Uma i-trajectory yeqa umkhawulo, konke ukulahleka kwamathokheni ayo kuyasuswa kuleso sibuyekezo.

Hlukanisa ukuqinisekisa nokuphindaphinda imiphumela

Isibonelo se-CPU siqinisekisa isibuyekezo se-optimizer kuphela. Ukuphindaphinda imiphumela yocwaningo kudinga ukulungiselelwa kwe-GPU, idatha, imodeli, i-agent, indawo, kanye nesikhathi sokusebenza esingavumelani nesikhathi okubhalwe ngakho emihlahlandleleni yokuqeqesha neyezibonelo.

Izindawo zokusetshenziswa ezibikiwe

Izivivinyo ezibikiwe zihlanganisa ukucabanga kwezibalo, ukusetshenziswa kwamathuluzi e-Python okunemijikelezo eminingi, ukuqeqeshwa kwe-mixture-of-experts, kanye nokwenza izinqumo okuhlanganyelwayo kwe-ALFWorld. I-README ibika ukuqeqeshwa okuzinzile kwe-long-chain-of-thought kuze kufike ezibuyekezweni ezingu-6,000 kwe-DeepSeek-R1-Distill-Qwen-1.5B, imiphumela yokucabanga kwezibalo ye-Qwen2.5-Math-1.5B, izivivinyo zamathuluzi e-Python nge-Qwen2.5-7B-Instruct ne-Qwen3-30B-A3B, kanye nezivivinyo ze-ALFWorld nge-Qwen2.5-7B-Instruct.

Le miphumela ikhombisa ububanzi obuhlosiwe bohlaka, kodwa akufanele ilindelwe esibonelweni esincane sesibuyekezo se-CPU.

Izinsiza eziwusizo

• Indawo yokugcina ye-FlashREINFORCE
• Iwebhusayithi yephrojekthi
• Ikhodi ye-NVIDIA NeMo Molt
• Umhlahlandlela wokuqeqesha
• Izilungiselelo nezibonelo zokuhlola

Isiphetho

I-FlashREINFORCE ihlanganisa ukumbozwa kwe-prompthi nge-rollout eyodwa kanye nesibuyekezo esihlangene esingenaye umgxeki. Imivuzo egxile ku-batch inikeza ama-advantage anophawu, i-token importance sampling ibhekana nezenzo eziphelelwe isikhathi, i-Sequence Trust Region ilawula ukuchezuka okuqoqekayo kwe-trajectory, kanti ukuthuthukisa okusekelwe esilinganisweni samasampula kuvimbela ubude bempendulo ekunqumeni isisindo se-trajectory.

Qala ngokufakwa okuhlelekayo, isibonelo sesibuyekezo se-CPU, kanye nezivivinyo. Uma lokho sekusebenza, hlola kuqala i-Molt launcher usebenzisa --dry-run, bese ulandela umhlahlandlela wokuqeqesha wokulungiselela i-GPU nedatha okuqondene nemodeli. Iphrojekthi itholakala ngaphansi kwe-Apache License 2.0.