Hlola Ama-Agent Okungenela Izifundo Zeziqu nge-Graduate Admissions Agent Benchmark

Okwenziwa yi-Graduate Admissions Agent Benchmark

I-Graduate Admissions Agent Benchmark ihlola ukucushwa kwama-agent okweluleka ngokwamukelwa ezifundweni ze-postgraduate. Ukucushwa kwe-agent kuhlanganisa uhlelo lokuqalisa imisebenzi nemodeli yolimi, bese kuhlolwa ezincomweni zezicelo, ekuhleleni, ekuthatheni izinqumo, nasekubuyekezeni izeluleko lapho kutholakala ulwazi olusha lomfundi.

Njengamanje i-benchmark inemisebenzi engu-70 yolimi lwesiShayina, amacala angu-23, nezigaba zezimpendulo ezingu-83. Umsebenzi ngamunye unama-rubric okuhlola nezinhlelo eziyizinkomba. I-dataset yomphakathi iyatholakala ngaphandle kokugunyazwa kokungena, kodwa izinto zokukala kufanele zihlale zingatholakali kuma-agent ahlolwayo ukuze imiphumela ingangcoliswa.

Indawo yokugcina ikhodi iphinde ishicilele amarekhodi okuhlola okucushwa okuyisishiyagalombili. Konke ukucushwa kuqede imisebenzi engu-70 yamanje, nakuba iphrojekthi iphawula ukuthi ama-run asebenzise izinguqulo ezahlukene zokufaka nezabelomali, ngakho imiphumela ye-leaderboard akufanele ithathwe njengokuhlolwa okufanayo ngokuphelele.

Qonda Isakhiwo Sokuhlola

Iqoqo lemisebenzi lihlukaniswe laba yizigaba ezintathu:

• Imisebenzi yezincomo engu-21: ihlola ukuthi izinhlelo eziphakanyisiwe zihambisana kangakanani nesizinda nemikhawulo yomfundi.
• Imisebenzi yokuhlela nokuthatha izinqumo engu-36: ihlola izinhlelo zokulungiselela, ukuhlolwa kobungozi, nezeluleko ezingasetshenziswa.
• Imisebenzi enezigaba ezimbili engu-13: inikeza ulwazi olusha kuseshini efanayo futhi ihlole ukuthi i-agent iyazibuyekeza yini izeluleko zayo zangaphambilini.

Umsebenzi ngamunye unamaphuzu angu-100. Imisebenzi yezincomo inika amaphuzu angu-70 ukufaniswa nezinhlelo eziyizinkomba, kuthi angu-30 aye kwikhwalithi yezincomo. Umhloli ohlukile ukala impendulo ephelele, kuyilapho ama-rubric ehlala engaphandle kwenqubo yokuphendula.

Imisebenzi enezigaba ezimbili iqala ngomongo wokuqala, obizwa nge-S₀, bese inikeza ulwazi olusha, olubizwa nge-S₁. I-agent ikhiqiza impendulo yokuqala, u-A₀, kanye nempendulo ebuyekeziwe, u-A₁, kuseshini eyodwa. Zombili izimpendulo zikalwa ngokuhlanganyela. Isibonelo somphakathi se-O03-04 sibonisa isimo sikaJanuwari kuya kuMashi lapho umfundi eshintsha indlela ebheke ezicelweni eziqondile ze-PhD.

Lungisa Indawo Yokusebenza

Qalisa imiyalo usuka empandeni yendawo yokugcina ikhodi. Ukuqalisa okuzenzakalelayo kwendabuko kudinga i-macOS Seatbelt ne-Python 3.10 noma entsha. Ukucushwa kokubhaliselwe kwe-Kimi kudinga i-Python 3.11 noma entsha.

Uhlaka lusebenzisa kuphela umtapo ojwayelekile we-Python, ngakho indawo yokugcina ikhodi ayidingi isinyathelo se-pip install. Nokho, kufanele ufake iklayenti elisemthethweni lokucushwa ofuna ukukuqalisa futhi ulungise ukungena kokubhaliselwe noma ukhiye we-API.

Amaklayenti atholakalayo ahlanganisa i-Codex, i-Claude Code, i-Antigravity, i-DeepSeek Harness, i-Qwen Code, ne-Kimi Code. Ukuqinisekisa ubuwena kuyahluka ngeklayenti: okunye ukucushwa kusebenzisa okubhaliselwe noma i-OAuth, kuyilapho i-DeepSeek ne-Qwen zisebenzisa ukuqinisekiswa okusekelwe ku-API njengoba kuchazwe emibhalweni yephrojekthi.

Khetha Ukucushwa Kwe-agent

Amakhodi e-agent avela kurejista yamanje yendawo yokugcina ikhodi ku-config/agents-cli.example.json. Lawa makhodi ayizihlonzi zokucushwa kwe-benchmark, hhayi izimpikiswano eziqondile zemodeli.

• gpt-6-astra ye-Codex ene-GPT Astra 6
• gpt-6-1-sol ye-Codex ene-GPT Sol 6.1
• claude-opus-5-5 ye-Claude Code ene-Claude Opus
• claude-sonnet-5-5 ye-Claude Code ene-Claude Sonnet
• antigravity-gemini-3-8-flash-high ye-Antigravity ene-Gemini Flash
• deepseek-v4-1-flash ye-DeepSeek Harness ene-DeepSeek Flash
• qwen3-8-max-0902 ye-Qwen Code ene-Qwen Max
• kimi-k3 ye-Kimi Code ene-Kimi K3

Qalisa Inqubo Yokuqala Ngokushesha

I-benchmark isebenzisa inqubo efanayo kukho konke ukucushwa. Okokuqala nikeza i-ID ye-agent ku-variable ye-shell AGENT:

AGENT=gpt-6-1-sol

Okulandelayo, gcina ukucushwa kwasendaweni:

python3 -m benchmark setup --agent "$AGENT"

Hlola ukuthi iklayenti elikhethiwe nokuqinisekisa ubuwena sekukulungele yini:

python3 -m benchmark cli-check --agent "$AGENT"

Lapho ukuhlola kubuyisa i-ready_for_trial, qalisa iqoqo usebenzisa ukukhethwa kwemisebenzi kwamanje:

python3 -m benchmark cli-batch \
  --selection config/selections/current.json \
  --agent "$AGENT" --out "./runs/current-$AGENT"

Ngokuzenzakalela, ukuqaliswa kweqoqo kubuka kuqala imisebenzi engu-70 nezigaba ezingu-83 ngaphandle kokwenza amakholi kumodeli. Faka i---execute lapho usulungele ukusebenzisa isabelo se-akhawunti yakho:

python3 -m benchmark cli-batch \
  --selection config/selections/current.json \
  --agent "$AGENT" --out "./runs/current-$AGENT" \
  --execute

Qala ngomsebenzi owodwa wokuzama ngemva kokuphumelela kwe-cli-check. Lokhu kusiza ukuqinisekisa ukuthi iklayenti, indlela yokuqinisekisa ubuwena, i-sandbox, nendawo yokukhipha kusebenza kahle ngaphambi kokuqalisa iqoqo eligcwele.

Qalisa Ukucushwa Okuningi

Lungiselela futhi uhlole iklayenti ngalinye ngokwehlukana, bese udlulisela izinketho eziningi ze---agent kumyalo weqoqo. Sebenzisa ifolda entsha yokukhipha ukuze imiphumela yokucushwa okuhlukene ingabhali phezu komunye nomunye.

python3 -m benchmark cli-batch \
  --selection config/selections/current.json \
  --agent gpt-6-1-sol \
  --agent claude-opus-5-5 \
  --out "./runs/comparison" \
  --execute

Gcina indlela yokuqinisekisa yokucushwa ngakunye ihambisana neklayenti elisohlwini lwerejista. Ukucushwa okusekelwe ekubhaliseni kufanele kusebenzise ukungena ngemvume kokubhalisa okusekelwayo, kanti ukucushwa kwe-API kufanele kusebenzise imininingwane edingekayo ye-API.

Hlola Imisebenzi Nemiphumela

Inkomba yemisebenzi iyatholakala ku-config/selections/current.json. Ukuze uthole imibuzo, izindlela, nezincazelo zokulinganisa, bheka ku-Imibuzo nokulinganisa, umhlahlandlela wefomethi yedatha, kanye nemithetho yokulinganisa.

Imiphumela eshicilelwe iyatholakala ngezindlela eziningana:

• Isifinyezo samaphuzu semiphumela ehlanganisiwe.
• Imiphumela yomsebenzi ngamunye yamarekhodi anemininingwane ezingeni lomsebenzi.
• Izimpendulo namaphuzu ngokucushwa kombhalo wezimpendulo zomphakathi kanye nolwazi lokuhlola.

Ibhodi lamaphuzu elishicilelwe libika amaphuzu amaphakathi namazinga okuqedwa phakathi kokucushwa okuyisishiyagalombili okuhloliwe. Inqolobane iphinde igcine amalebula obunzima emisebenzi yamanje: 14 elula, 34 ephakathi, kanye nama-22 enzima. Lawa malebula aqinisekiswe abanikazi ngokulinganiswa kothisha, futhi awethulwanga njengokubuyekezwa okuzimele okuphindwe kabili.

Amathiphu Okulinganisa Nokubuyekeza

Ukulinganisa kungase kusebenzise iseshini ehlukile yokubhalisa ye-Codex ukuze kubuyekezwe izinto ezikhishiwe. Ukuhlola okukhethwayo nge-API kudinga ukucushwa okwengeziwe. Gcina inqubo yokuphendula ihlukene nezinto zokulinganisa, ngoba ukuveza amarubhrikhi noma izinhlelo eziyizithenjwa ku-agent ehlolwayo kungachitha ubuqiniso bokuqhathanisa.

Lapho uqhathanisa ukugijima kwakho, rekhoda i-ID ye-agent, inguqulo yeklayenti, inguqulo yokufaka, ibhajethi yokusebenza, ifolda yokukhipha, kanye nanoma yimaphi ama-hashi wokuqukethwe afanele. Iphrojekthi ixwayisa ngokuthi ukugijima kwebhodi lamaphuzu lomphakathi akuhlanganyeli zonke izimo zokusebenza ezifanayo, ngakho qhathanisa imiphumela kanye nendlela esetshenzisiwe namarekhodi okusebenza.

Ukuze wenze izivivinyo eziphindaphindekayo, landela umhlahlandlela wokuphindaphinda, ohlanganisa okokufaka okuqandisiwe, ukuthunyelwa, nokuqinisekiswa kwamaphuzu. Imibhalo yendlela yokusebenza ichaza izilungiselelo zokuhlola nemikhawulo, kanti umhlahlandlela we-native CLI unikeza imininingwane eyengeziwe ngokufakwa, ukuqinisekiswa, nezigaba eziningi.

Isiphetho

I-Graduate Admissions Agent Benchmark inikeza indlela ehlelekile yokuhlola ukuthi ama-agent okwamukelwa anikeza yini izincomo ezinembile, izeluleko eziwusizo zokuhlela, kanye nezibuyekezo ezifanele lapho izimo zomfundi zishintsha. Lungiselela iklayenti elisekelwayo, khetha i-ID ye-agent kurejista, yiqinisekise nge-cli-check, hlola kuqala iqoqo, bese usebenzisa izivivinyo noma ukuhlola okuphelele. Sebenzisa izincazelo zemisebenzi, imiphumela, nemibhalo yendlela yokusebenza eshicilelwe ukuze uhumushe amaphuzu ngokuzibophezela futhi uphinde ukhiqize ukuqhathanisa.