WebCraftBench Uyasebenza

I-WebCraftBench iyisisekelo sokuhlola ukukhiqizwa kwamakhasi ewebhu nge-AI, esethulwe yi-Tencent Hunyuan ngokubambisana ne-Tsinghua University ne-Peking University. I-WebCraftBench iqala ngokufaka ngokuzenzakalelayo i-instrumentation ezinhlelweni ezikhiqiziwe, bese ivumela ama-agent ukuthi ahlole imisebenzi yewebhu ngokuyisebenzisa ngempela nge-Playwright, isebenzise i-code coverage ukuqondisa ukutholwa kwezikhala; bese iguqula amarekhodi okusebenzisana abe yigrafu yezimo, ikhiphe amaphuzu ngokwehlukana ngobuhle, ukusebenziseka kalula, nokuhambisana nezidingo. Isisekelo siqukethe izidingo zangempela ezingu-369, izindinganiso zokwamukelwa ezingu-5,088 eziqinisekiswe abantu, sihlanganisa izinhlelo ezingu-6,273 zamamodeli aphambili angu-17, futhi izinga lokuvumelana nokuthandwa abantu lifinyelela ku-85.3%.

Isethi yokuhlola yezidingo zangempela: Iqukethe izidingo zabasebenzisi ezingu-369 ezivela ekusetshenzisweni kwangempela, ezihambisana nezindinganiso zokwamukelwa ezingu-5,088 eziqinisekiswe abantu, ezihlanganisa izidingo zokusebenza, okuqukethwe, nokubukeka.
Ukuhlola okuzenzakalelayo okusebenzisanayo: Ama-agent asebenzisa i-Playwright MCP njengabahloli, achofoze, afake umbhalo, ashintshe amakhasi, athole imisebenzi ngokuqondile futhi ahlole izimo ezisemkhawulweni.
Ukuhlola okuqondiswa yi-code coverage: Ngokusekelwe ku-Istanbul instrumentation ezenzakalelayo, uhlelo luqapha i-coverage ngesikhathi sangempela; lapho kutholakala ukuthi ukuhlola akwanele, imodeli esizayo iphakamisa izindawo zekhodi ezingakathintwa.
Ukufingqa nokuhlela izimo: Amarekhodi amade okusebenza ayasuswa okuphindaphindwayo futhi ajwayezwe, ahlanganiswe abe yigrafu yokushintsha izimo, bese kukhishwa ubufakazi obubalulekile bokusebenza obusetshenziselwa ukukala.
Ukukala okuzimele ngezindlela ezintathu: Ubuhle, ukusebenziseka kalula, nokuhambisana nezidingo kuhlolwa ngokwehlukana, bese kuhlanganiswa kube amaphuzu ajwayelekile aphelele.
Ukuqinisekisa ngokuthandwa abantu: Kusethi yokuqinisekisa ezimele, imiphumela iqhathaniswa namavoti abantu abuyekeziwe; izinga lokuvumelana kwezinqumo ezibhanqiwe lifinyelela ku-85.3%, okuqinisekisa ukwethembeka kokuhlola.

Ukufaka i-instrumentation ngokuzenzakalelayo ukuze kuqoqwe ubufakazi bokusebenza: Uhlelo lufaka izibali kukhodi yohlelo futhi lusekela i-static HTML, Vite, Next, CRA, Astro namanye ama-framework. Ngemva kokuqala kohlelo, konke ukusebenzisana kubuyisela i-code coverage kanye nokwanda kwayo, okwenza kubonakale ukuthi “yikuphi okukhodi okuke kwasetshenziswa”.
Ukuhlola kwama-agent okuqondiswa yi-coverage: I-agent yokuhlola ayibheki ikhodi yomthombo futhi ayibheki uhlu lokwamukelwa; isebenzisa ikhasi lewebhu ngokulandela inqubo yokuhlola i-GUI. Lapho ukusebenzisana kathathu okulandelanayo kungalethi i-coverage entsha, uhlelo lucela enye imodeli ukuthi ihlaziye ikhodi engakaze isetshenziswe, bese luguqula izincomo zibe ulimi lwemvelo olunikezwa i-agent yokuhlola ukuze iqhubeke nokuzama.
Ukufingqa kwezimo kunciphisa umzila wokusebenzisana: Isakhiwo sekhasi sijwayezwa, amakhasi afanayo ahlanganiswe abe yisimo esisodwa, kanti izenzo zixhunywe zibe yigrafu yokushintsha izimo. Lokhu kugwema ukuthi izithombe-skrini eziphindaphindwayo zithathe umongo wemodeli yokukala, futhi kwenza izindlela zamaphutha zibonakale ngokushesha.
Ukuhlola ngezindlela eziningi okuhlukanisa ukuhlola nokukala: Ngesikhathi sokuhlola ubufakazi buqoqwa ngaphandle kokubheka uhlu lokwamukelwa; uhlu lusetshenziswa kuphela ngesikhathi sokukala ukuze ubufakazi buqhathaniswe nezindinganiso ngakunye. Ubuhle nokusebenziseka kalula kuhlolwa ngemijikelezo emithathu ethi “thola okuhle—khomba amaphutha—umahluleli anikeze amaphuzu” (; ukuhambisana nezidingo kudinga ukuthi isinqumo ngasinye sicaphune ubufakazi bokusebenza, bese amaphuzu anikezwa ngokwesilinganiso sokwaneliseka. Ubukhulu bento ngayinye kwezintathu buyingxenye eyodwa kwezintathu yamaphuzu aphelele.

Lungisa uhlelo olukhiqiziwe: Cela imodeli ehlolwayo ukuthi ikhiqize uhlelo lwewebhu ngokwezidingo, bese iluthumela esimweni esisebenzayo.
I-instrumentation ezenzakalelayo: Sebenzisa ithuluzi le-instrumentation le-WebCraftBench ukufaka izibali ze-coverage kukhodi yohlelo, ukuze konke ukusebenzisana kubuyisele ulwazi lokusebenza kwekhodi.
Ukuhlola okusebenzisanayo: Qalisa i-agent yokuhlola, uyivumele isebenzise ikhasi nge-Playwright MCP, yenze izenzo ezinjengokuchofoza, ukufaka umbhalo nokushintsha amakhasi, ithole imisebenzi futhi ihlole izimo ezisemkhawulweni.
Ukutholwa kwezikhala okuqondiswa yi-coverage: Lapho ukusebenzisana kathathu okulandelanayo kungangezi i-coverage, imodeli esizayo ihlaziya ikhodi engakaze isetshenziswe futhi ikhiqize izincomo zolimi lwemvelo ezibuyiselwa ku-agent yokuhlola ukuze iqhubeke nokuhlola (izinyathelo ezifika ku-100).
Hlela ubufakazi bokusebenza: Uhlelo luzihlela ngokuzenzakalelayo izimo zesakhiwo sekhasi, luhlanganise amarekhodi okusebenza ngemva kokususa okuphindaphindwayo lube yigrafu yokushintsha izimo, futhi lugcine izithombe-skrini kanye nomzila wokusebenzisana.
Ukukala ngezindlela ezintathu: Ama-agent okukala ahlola ngokwehlukana ubuhle nokusebenziseka kalula, bese ehlola ukuhambisana nezidingo ngokubheka izindinganiso zokwamukelwa ngakunye; isinqumo ngasinye kufanele sicaphune ubufakazi bokusebenza.
Okuphumayo: Kutholakala amaphuzu ajwayelekile ezindlela ezintathu kanye namaphuzu aphelele, angafaniswa nezinga leqoqo lamamodeli; uma umehluko wamaphuzu umkhulu, umphumela uba nokwethembeka okukhulu.

Iqondiswe ekuhlolweni kwangempela: Ivumela ama-agent ukuthi asebenzise ikhasi lewebhu ngempela ukuze anikeze amaphuzu, kunokubheka ikhodi noma izithombe-skrini kuphela, ngaleyo ndlela ithole izinkinga eziphuthelwa ukuhlola okumile, njengokuthi “inkinobho yokungena iphukile, ikhasi lasekhaya alivuleki”.
Ukuhlola okuqondiswa yi-coverage: Isebenzisa i-code coverage “njengomkhondo wokuthola izikhala”; lapho i-agent yokuhlola ingawutholi umsebenzi, imodeli yangemuva iphakamisa indlela, kuncishiswe ukuthi ukwehluleka kokuhlola kuthathwe ngephutha njengokushiyeka komkhiqizo.
Ukuhlola nokukala kuhlukene: Uhlu lokwamukelwa lusetshenziswa kuphela ngesikhathi sokukala, kugcinwe izithiyo zezidingo ngaphandle kokuvimba ukubonwa kokusebenzisana okungaphandle kohlu, ngaleyo ndlela kwandiswe ububanzi bokuhlola.
Ukukala okuzimele ngezindlela eziningi: Ubuhle, ukusebenziseka kalula, nokuhambisana nezidingo kunikezwa amaphuzu ngokwehlukana, kugwenywe ukuthi imodeli “ebukeka kahle kodwa iphahlazeke uma isetshenziswa” ifihle ubuthakathaka bayo ngaphansi kwephuzu elilodwa eliphelele.
Ukuvumelana okuphezulu nokuthandwa abantu: Eziqoqweni ezingu-197 zokuqhathanisa kwabantu eziqinisekisiwe, izinga lokuvumelana kwezinqumo lifinyelele ku-85.3%; lapho umehluko wamaphuzu mkhulu (≥0.75), izinga lokuvumelana lenyuka laya ku-98%, okwenza imiphumela ithembeke kakhulu.
Ukuzinza okuphezulu kokuhlola: Ngemva kokushintsha imodeli yomahluleli wokukala, i-coefficient yokuhambisana kwamazinga isalokhu ingu-0.980, futhi ukuhleleka kwabane abaphezulu akushintshi, okubonisa ukuthi iziphetho azincikile kumahluleli oyedwa.

Iphepha lobuchwepheshe le-arXiv: https://arxiv.org/pdf/2609.15387

Ukukhetha amamodeli amathuluzi ohlelo lwe-AI: Lapho iqembu likhetha imodeli enkulu elizoyisebenzisa ukukhiqiza izinhlelo zewebhu, lisebenzisa amaphuzu amathathu e-WebCraftBench ukwenza ukuqhathanisa okuqondile, ligweme ukwenza izinqumo ngokubheka kuphela ubuhle be-demo.
Ukuhlolwa kokwamukelwa kwemikhiqizo ekhiqiza amakhasi ewebhu nge-AI: Hlanganisa “inqubo yokuhlola kwangempela” ye-WebCraftBench esigabeni sokulethwa komkhiqizo, esikhundleni sokuhlola ikhasi ngalinye ngesandla, ukuze kutholwe ngokuzenzakalelayo izinkinga ezinjengokungasebenzi kwendawo yokungena noma imisebenzi engafinyeleleki.
Ukuhlola kokubuyela emuva kwamamodeli akhiqiza ikhodi: Ngemva kokuthuthukiswa kwemodeli ngakunye, sebenzisa isisekelo futhi, uqhathanise izinguquko ebuhleni, ekusebenziseni kalula, nasekuhambisaneni nezidingo, ukuze kukalwe ukuthi inguqulo entsha ithuthukile noma yehlile.
Ukuxilonga okuhlukanisa ikhwalithi yokukhiqiza: Sebenzisa ukuhlukaniswa kwezindlela kanye nemikhondo ye-coverage ephansi ukuthola inkinga, kungakhathaliseki ukuthi idizayini ayilungile, ukusebenzisana kune-bug, noma ukuqonda izidingo kuphambukile, bese uqondisa ukuthuthukiswa okukhethekile.