Ukufunda ngokuqinisa sekuyindlela evamile yokuqeqesha amamodeli amakhulu, okusheshisa ukusetshenziswa kobuchwepheshe bamamodeli ezimweni ezibalulekile zezimboni. Phakathi kwalokhu, ukuvumelana phakathi kokuqeqesha nokwenza inference kubaluleke kakhulu, ngoba lezi zinqubo ziyahambisana, kanti umehluko wengqalasizinda ungakhulisa ukungaqiniseki.
Umnyango osemthethweni weHuawei Computing umemezele ngo-Agasti 5 ukuthi i-Ascend inikeza isethi ephelele yama-operator okuvumelanisa ukuqeqesha ne-inference isebenzisa ulimi lokuhlela i-Ascend C. Ukuhlolwa kwemodeli ye-Qwen3-30B MoE kuthole i-Logdiff engu-0, kwathi ngokwenza ngcono ama-operator e-FA ahambisana ne-Ascend, kwatholakala inzuzo yokusebenza engu-20%-60% kumodi ye-Eager, okuhlinzeka abathuthukisi ngohlelo olusebenzayo noluthembekile lokuqeqesha ngokufunda ngokuqinisa.
Isizathu esiyisisekelo sokungavumelani phakathi kwe-Rollout nokuqeqesha ku-RL post-training ukuthi ukuhlanganiswa ngesikhathi sokubala ezingeni eliphansi akulandeli ukuhleleka okuqinile. Ukuze kuqinisekiswe kusukela kuhlaka kuye phansi ukuthi ukuhleleka kokuhlanganisa kwenjini yokuqeqesha neyokwenza inference kuyafana ngaso sonke isikhathi, uhlaka nama-operator kumele kusebenze ndawonye. Njengoba amapharamitha emodeli ekhula, inkinga iba nkulu: amasu okuhlukanisa afana ne-tensor parallelism ne-expert parallelism, kanye nezindlela zokuxhumana eziqoqiwe, angaphazamisa ukuvumelana uma kukhona ingxenye eyodwa engahambisani. Lokhu kwenza kube nzima ukufinyelela ku-true-on-policy ekugcineni. Ama-operator nawo kumele asekele izimo eziningi zokusetshenziswa, abeke imingcele ezinyathelweni ezibalulekile zokubala, futhi alawule ukulahleka kokusebenza kube sezingeni elisebenzisekayo. Lena yinselele yobunjiniyela besistimu yonke.
Ukungavumelani phakathi kokuqeqesha ne-inference kusho ukungafani kwezinombolo phakathi kwenjini ye-Rollout (inference) nenjini yokuqeqesha. Ngisho noma zombili zisebenzisa izisindo zemodeli ezifanayo ngokuphelele, ama-log probability (logprob) abalwa ngokulandelana okufanayo kwama-Token angahluka kancane; lo mehluko uvame ukukalwa nge-logdiff.
I-Ascend ibeke imingcele futhi yaqondanisa ama-operator abalulekile ochungechungeni lokuqeqesha ne-inference isebenzisa ulimi lokuhlela i-Ascend C. Umqondo oyinhloko uwukwenza ukuthi ukuqeqesha ne-inference kulandele indlela efanayo yezinombolo “ezindaweni lapho kufanele kuhlanganiswe khona”. Izinguquko ezi-4 eziyinhloko yilezi:
Ukuhlanganisa incazelo ye-attention: Ukuqondanisa ububanzi bokubala ezindaweni ezisebenzayo phakathi kwe-Softmax yemaski yokuqeqesha ne-step-by-step attention ye-inference, kususwe umehluko wezinombolo obangelwa amasethi ama-Token abonakalayo angafani.
Ukukhiya ukuhleleka kokuhlanganisa kwe-reduce: Ukulawula ukuhlukaniswa nokuncishiswa kobukhulu be-reduction ku-FA yokuqeqesha kanye ne-PFA / PagedAttention ye-inference, kugwenywe ukuhlanganisa okungalandeli ukuhleleka okubangelwa “ukuhlukaniswa okungafani kwesifanekiso esifanayo”.
Ukuqondanisa isu lokuhlukanisa le-Softmax eku-inthanethi: Ukuhlanganisa usayizi wamabhulokhi nesigqi sokunciphisa, kugwenywe ukungavumelani phakathi kwe-Softmax ehlanganisa amabhulokhi amaningi ohlangothini lwe-decoder ne-Softmax ehlukaniswe ngamabhulokhi ohlangothini lokuqeqesha.
Ukuqondanisa ukunemba kwendlela ebalulekile: Ukuhlanganisa isu lokuguqula ukunemba ezinyathelweni ezizwelayo njengokuhlanganisa kwe-Softmax, kuncishiswe amaphutha amancane edijithi alethwa umehluko wokusetshenziswa kokunemba okuxubile.
Ngalokhu, kanye nokwenza ngcono ukusatshalaliswa nokuhlelwa kwe-FA, ukuvumelana phakathi kokuqeqesha ne-inference akusadingi ukubuyela emuva okukhulu kokusebenza. Ngakho ekuhlolweni kutholwe kokubili i-logdiff=0 nokusheshiswa kokusebenza kusukela ekuqaleni kuya ekugcineni.
IHuawei imemezele ukuthi ingqalasizinda efanele isivulelwe umphakathi, futhi izokwethula i-“Skill yokuhlonza izinkinga zokuvumelana phakathi kokuqeqesha ne-inference”, esekela ukuhlola i-logdiff esele nokuthola ukuthi inkinga ibangelwa indlela ye-operator noma umehluko ekusetshenzisweni kohlaka.
Ikheli lekhodi evulekile:
https://github.com/verl-project/verl-ascend-recipe/tree/main/true_on_policy