Yakha Amapayipi Ezinqumo Asheshayo Aqondene Nomsebenzi nge-JevHarness

Lokho okwenziwa yi-JevHarness

I-JevHarness isiza i-LLM yokubhala ukuthi yakhe i-decision harness eqondene nomsebenzi ye-Jev. Le harness iguqula ukubonwa okuvunyelwe kube izici eziwusizo, yakha imibuzo nemibandela ehlelekile ye-Jev, bese ihlanganisa izimpendulo zibe izenzo zomsebenzi ezisemthethweni.

Umbono oyinhloko ukuhlukanisa ukuthuthukiswa kwesu okubizayo ekwenzeni okuvamile:

Cabanga ngokujulile ngesikhathi sokuthuthukisa. Qandisa isu. Vumela i-Jev yenze izinqumo ezisheshayo nezingacacile.

Ngesikhathi sokuthuthukisa, i-LLM ingabuyekeza ikhodi ye-harness, imibuzo, igrafu, isimo, imibandela, nememori. Ngemva kokukhetha nokumisa i-harness, izinqumo zesikhathi sokusebenza zisebenzisa leyo logic engaguquki kanye nezingcingo zayo ze-Jev. I-LLM yokubhala ayidingi ukuphinde icabange ngaso sonke isenzo.

Lokhu kwenza i-JevHarness ifanele imisebenzi lapho ikhodi ingabala amaqiniso athembekile kodwa ezinye izinqumo zisalokhu zincike kumongo noma zingacaci. Isibonelo, ikhodi inganquma izenzo ezisemthethweni futhi ibale izici zezinombolo, kuyilapho i-Jev ihlola ukuthi yisiphi isenzo esitholakalayo esifeza kangcono imibandela eqondene nomsebenzi.

Ukwakhiwa kanye Nemingcele Yezibopho

I-JevHarness ihlukanisa i-adaptha yomsebenzi ethembekile ku-decision harness. Lo mngcele uvimbela i-harness ekushintsheni umvuzo wayo noma ekufinyeleleni esimweni somsebenzi esifihliwe.

• I-adaptha yomsebenzi: Iphethe okubonwayo, izenzo ezisemthethweni, imiphumela eseceleni, kanye nokukala.
• I-harness: Iphethe ukwakhiwa kwezici, izahlulelo ze-Jev, kanye ne-logic yezinqumo.
• Imvelo: Isebenzisa izenzo futhi ibuyise imiphumela noma imivuzo ethembekile.
• I-LLM yokubhala: Yakha noma ibuyekeze i-harness ngesikhathi sokuthuthukisa nasekuzindleni okukhethwayo.

I-PipelineSpec eqinisekisiwe isetshenziswa nge-PipelineRuntime. Uma isikhethiwe, i-pipeline ingamiswa kanye ne-specification yayo, i-runtime, i-evaluator, kanye nezinsiza zomsebenzi ezimenyezelwe.

Izici Eziyinhloko

Izahlulelo ze-Jev ezihlelekile

I-Jev isekela izinhlobo zezimpendulo ze-choice, score, kanye ne-noul. Isicelo singaqukatha imibuzo eminingi enamagama, futhi ama-node azimele egrafini angasebenza ngesikhathi esisodwa.

Ukumelwa kwesu okusobala

Isu ligcinwa kukhodi, ezicini ezibaliwe, esimweni, emiyalweni, emibandeleni, kwimemori, nasekuhambeni kokulawula. Lokhu kwenza inqubo yesinqumo ihlolwe futhi iphinde isetshenziswe, esikhundleni sokuyishiya ngaphakathi kwengxoxo yokubhala.

Amarekhodi aphelele okwenziwa

Ukuzindla kungathola amarekhodi aphelele eziqephu ezikhethiwe, okuhlanganisa okubonwayo, izinqumo, okokufaka nokuphumayo kwama-node, imibuzo nezimpendulo ze-Jev, imemori, kanye nokwehluleka. Ukususa ukuphindaphindwa ngaphandle kokulahlekelwa imininingwane kunciphisa ukuphindaphinda. Uma okokufaka kokuzindla kudlula umkhawulo wamabhayithi omisiwe, kugcinwa kungobo yomlando futhi kwenqatshwe esikhundleni sokunqunywa buthule.

Ukuthuthukiswa okusekelwe emvuzweni ngokuzithandela

Ukuhlanganiswa okukhethwayo kwe-GEPA kukhetha abazali ku-instance frontier, kuqhathanisa umzali ngamunye nesiphakamiso eqoqweni lokuqeqesha elifanayo, bese kuhlola iziphakamiso ezamukelwe ku-Eval. Kurekhoda imvelaphi yangempela, okuhlanganisa neziphakamiso ezinqatshiwe.

Izinto ezimisiwe ezingahlolwa

Ukumisa kugcina i-harness ekhethiwe kanye nokuncika kwayo okumemezelwe. I-harness emisiwe equkethe ama-node e-Jev isadinga i-Jev ngesikhathi sokusebenza. Futhi qaphela ukuthi i-alias yemodeli esingathwe ayiqinisekisi ukuziphatha komhlinzeki okuzayo okungashintshi, ngakho izimpendulo ezigciniwe nezingcingo ezintsha zinikeza amazinga ahlukene okuphindaphindeka.

Izidingo Zokuqala

Khetha izidingo ezihambisana nalokho ofuna ukukusebenzisa:

• Iwebhusayithi egcinwe kungobo yomlando: I-Node.js nesiphequluli sesimanje. Ayidingi iziqinisekiso zemodeli, injini yomdlalo, noma i-Python sandbox.
• Iphrojekthi ye-Python: I-Python 3.11 noma eyakamuva.
• Ama-node e-Python asebenzayo: I-sandbox yomdabu ye-macOS esekelwayo. Uma ukuhlukaniswa kungatholakali, ukwenziwa kuyama ngendlela ephephile.
• Izisho ze-Version 2 nokugeleza kwe-Jev: Lokhu akuqalisi abasebenzi be-Python abasebenzayo.
• Izingcingo ezibukhoma ze-Jev: I-akhawunti yomhlinzeki esekelwayo kanye nokhiye we-API ohambisanayo.
• Ukubhala nokuzindla okukhethwayo: I-Proposer emisiwe esebenzisa i-Claude CLI yasendaweni, i-OpenAI, i-Azure, noma i-Anthropic endpoint.

Faka i-Skill ku-Claude Code

Indlela esheshayo yokumisa i-Claude Code ukwengeza i-repository njenge-plugin marketplace, ufake i-plugin, bese ulayisha kabusha ama-plugin.

/plugin marketplace add https://github.com/TianyuCodings/JevHarness.git
/plugin install jev-harness@jevharness
/reload-plugins

Ukufaka i-plugin kudawuniloda i-skill kanye nezikhombo zayo, kodwa akufaki izincika ze-Python noma kulungise iziqinisekiso zemodeli. Uma i-repository iyimfihlo, ukuqinisekiswa kwe-GitHub kumele kube sekusebenza futhi i-akhawunti yakho kumele ibe nemvume yokufunda.

Faka i-Skill ku-Codex

Kumsingathi we-Codex ohlinzeka nge-skill-installer eyakhelwe ngaphakathi, cela ukufakwa ngokuqondile ku-repository:

$skill-installer Install the jev-harness skill from https://github.com/TianyuCodings/JevHarness/tree/main/skills/jev-harness

Sebenzisa i-skill emzuliswaneni olandelayo. Uma i-Codex ingayitholi ngokushesha, qala kabusha iseshini.

Ukufaka Ngokuzenzela

Ungafaka i-skill ngokuzenzela ku-Claude Code, ku-Codex, noma kukho kokubili. Faka esikhundleni se-/path/to/your-project ngohlu lwemibhalo lwephrojekthi olukhona.

git clone https://github.com/TianyuCodings/JevHarness.git
cd JevHarness
python3 scripts/install-skill.py --target both --scope project --project /path/to/your-project

Lokhu kwakha ukufakwa kwezinga lephrojekthi ku-.agents/skills/jev-harness/ kwe-Codex naku-.claude/skills/jev-harness/ kwe-Claude Code. Sebenzisa i---target codex noma i---target claude ukuze ufake eyodwa kuphela.

Ukuze ufake ezingeni lomsebenzisi futhi kutholakale kuwo wonke amaphrojekthi, sebenzisa:

python3 scripts/install-skill.py --target both --scope user

Izindawo zokufaka zomuntu siqu zithi ~/.agents/skills/jev-harness/ kanye ne-~/.claude/skills/jev-harness/. Isifaki siyenqaba ukubhala phezu kokufakwa okukhona okuhlukile. Qala kabusha iseshini ye-ejenti uma ikhono lingatholakali.

Hlola Isibonelo Se-Pokémon Esigciniwe

Isibonelo se-Pokémon esifakiwe sihlinzeka ngokuqalisa okugcinwe kungobo yomlando, iharness ekhethiwe, isihlahla sokuziphendukela kwemvelo esiphequlekayo, amarekhodi ezinqumo aphelele, iziphakamiso ezenqatshiwe, kanye namaphuzu avelele okuphinda adlalwe avumelaniswa nezimpendulo ze-Jev. Ukuphequlula lesi sigcini akwenzi izicelo kumamodeli futhi akuqali imidlalo.

1. Yenza i-clone yendawo yokugcina.
2. Yakha iwebhusayithi.
3. Qalisa iseva yokubuka kuqala yasendaweni.
4. Vula isayithi esipheqululini sesimanje.

git clone https://github.com/TianyuCodings/JevHarness.git
cd JevHarness
node website/build.mjs
node website/preview.mjs --port 8768

Vula localhost:8768. Opopayi bezimpi balanda izimpahla kumhlinzeki osemthethweni wokubonisa we-Pokémon Showdown. Ilogi yempi egciniwe ihlala esipheqululini futhi ayilayishwa kuseva yokuphinda kudlalwe.

Umphumela obikiwe we-Pokémon wathuthukisa izinga lokuwina le-Eval lisuka kumaphesenti angu-25, noma imidlalo engu-3 kwengu-12, laya kumaphesenti angu-75, noma imidlalo engu-9 kwengu-12, ngemva kwemijikelezo emihlanu yokuzindla. Ukusesha kugcine ikhandidethi lomjikelezo wesithathu njengeharness yalo ehamba phambili. Lokhu kukalwe kusethi ye-Eval yokukhetha futhi akufanele kuthathwe njengesilinganiso esizimele sokusebenza emidlalweni engakaze ibonwe.

Qonda Isinqumo Esiyisisekelo

I-harness ibhala isicelo se-Jev esiqondene nomsebenzi esiqukethe i-state, ama-questions aqanjwe amagama, i-type yempendulo, kanye ne-instructions ne-criteria ezisetshenziselwa ukwahlulela izenzo ezitholakalayo.

Esibonelweni se-Pokémon esirekhodiwe, i-harness ibala ukuthi i-Slowbro kungenzeka ihlulwe emncintiswaneni womonakalo ibhekene ne-Gastrodon. Iveza izinketho ezine ezisemthethweni, okuhlanganisa iminyakazo emithathu nokushintshela ku-Scizor. Umbuzo wesenzo usebenzisa uhlobo lwe-choice futhi udinga ukuthi i-Jev ibuyise i-ID yesenzo esisodwa kuphela esohlwini.

{
  "questions": {
    "action": {
      "type": "choice",
      "instructions": "Khetha isenzo esisodwa esinamathuba amaningi okunqoba yonke impi, hhayi leli thuba kuphela.",
      "criteria": {
        "switch:2": "shintshela ku-Scizor; ukushintsha kunikela ngokuhlasela okukodwa kwamahhala",
        "move:2": "Psychic; cishe amagalelo angu-3 ukukhipha okuqondiwe",
        "move:3": "Ice Beam; cishe amagalelo angu-4 ukukhipha okuqondiwe",
        "move:4": "Slack Off; umonakalo obikezelwe ozayo udlula i-HP yamanje"
      }
    }
  }
}

I-Jev ikhetha i-switch:2 ngamathuba okukhetha isenzo angu-0.72, futhi i-harness iyakwamukela lokho kukhetha. Lelo nani liqhathanisa izenzo ezisohlwini; alibikezeli ukuthi umdlali unethuba elingamaphesenti angu-72 lokuwina impi.

Isicelo esirekhodiwe esiphelele, impendulo, isikhathi, kanye nemvelaphi kuyatholakala kwirekhodi le-JSON le-Pokémon turn-12.

Yakha Eyakho I-Harness

Awudingi ukubhala ngesandla yonke imiyalelo nemibandela ye-Jev. Cela ikhono elifakiwe bese uchaza umsebenzi, bese uvumela i-ejenti yokubhala ikhodi iqoqe imininingwane edingekayo yenkontileka.

Ukucela i-plugin ye-Claude Code

/jev-harness:jev-harness Yakha i-harness ehambisa amathikithi osizo eqenjini elifanele.
Qala ngokucacisa okokufaka kwami, izenzo ezisemthethweni, izibonelo, imibandela yempumelelo, kanye nesabelomali.
Uma sinemivuzo ethembekile, engeza ukuhlola nokwenza ngcono ngokuzindla.

Ukucela i-Claude Code ezimele

/jev-harness Yakha i-harness ehambisa amathikithi osizo eqenjini elifanele.
Qala ngokucacisa okokufaka kwami, izenzo ezisemthethweni, izibonelo, imibandela yempumelelo, kanye nesabelomali.
Uma sinemivuzo ethembekile, engeza ukuhlola nokwenza ngcono ngokuzindla.

Ukucela i-Codex

$jev-harness Yakha i-harness ehambisa amathikithi osizo eqenjini elifanele.
Qala ngokucacisa okokufaka kwami, izenzo ezisemthethweni, izibonelo, imibandela yempumelelo, kanye nesabelomali.
Uma sinemivuzo ethembekile, engeza ukuhlola nokwenza ngcono ngokuzindla.

Ikhono kufanele liqale licacise umsebenzi esikhundleni sokukhiqiza ipayiphi ngokushesha. Lindela ukuthi liqoqe ulwazi mayelana nalokhu:

• Okubonwa ngumsebenzi okuvunyelwe kanye nemingcele yesimo esifihlekile.
• Izenzo ezisemthethweni kanye nendlela izenzo eziqinisekiswa ngayo.
• Izibonelo ezimele kanye nedatha etholakalayo.
• Imibandela yempumelelo, imivuzo, noma izinqubo zokuhlola.
• Izidingo zesikhathi sokusebenza nokuhlukaniswa.
• Izifakazelo kanye nokutholakala kwabahlinzeki.
• Isabelomali sokuhlola kanye nezinsiza zokwenza ngcono.

Uma inkontileka isicacile, i-ejenti ingakha futhi iqinisekise i-harness. Uma impendulo ethembekile ikhona futhi ucela ukwenza ngcono, ingasebenzisa ama-trajectory nemivuzo ukuze izindle ngaphambi kokumisa umphumela okhethiwe.

Lungiselela Izifakazelo ze-Jev

I-JevHarness isekela ukulungiselelwa okubili okubhaliwe kwezokuthutha ze-Jev:

• Vercel AI Gateway: Dala i-JevClient(transport="vercel") bese usetha i-AI_GATEWAY_API_KEY.
• TypeSafe ngokuqondile: Dala i-JevClient(transport="typesafe") bese usetha i-TYPESAFE_API_KEY.

Ama-adapter abahlinzeki awaqinisekisi ukuthi wonke amamodeli noma ama-endpoint ayatholakala ku-akhawunti yakho. Gcina izifakazelo eziguquguqukweni zemvelo noma kufayela lendawo elithi .env elinganakiwe. Ungalokothi ufake izifakazelo ekubukweni kwemisebenzi, kuma-trace, noma kuma-artifact afriziwe.

Hlola Futhi Uthuthukise I-Harness

Ukuzindla kuyazikhethela. Kusebenzise kuphela uma unendlela ethembekile yomvuzo noma yokuhlola. Inqubo esebenzayo yile:

1. Chaza umngcele womsebenzi. Gcina isimo esifihlekile, ukubalwa kwamaphuzu, ukuqinisekiswa kwezenzo ezisemthethweni, nemiphumela engemihle ku-adaptha ethembekile.
2. Dala i-harness eyisisekelo. Bala izici ezithembekile ngekhodi, bese ugcine i-Jev yezinqumo ezincike kumongo.
3. Qinisekisa i-pipeline. Qinisekisa ukuthi i-PipelineSpec ne-runtime kuyahlangabezana nenkontileka yomsebenzi.
4. Qoqa ama-trace aphelele. Gcina okuboniwe, okokufaka kwama-node, okuphumayo, imibuzo, izimpendulo, inkumbulo, amaphutha, nemivuzo.
5. Phakamisa izinguquko. Vumela i-LLM ebhalayo ihlole ubufakazi bese ibuyekeza ikhodi, ama-prompt, isakhiwo segrafu, noma inkumbulo.
6. Qhathanisa ngobulungisa. Qhathanisa i-parent nesiphakamiso kwinqwaba efanayo yokuqeqesha.
7. Hlola iziphakamiso ezamukelwe. Qalisa ukuhlolwa okuphelele kwe-Eval futhi ugcine uhlu lwangempela lozalo lwabazokhethwa, kuhlanganise neziphakamiso ezenqatshiwe.
8. Vala i-harness ekhethiwe. Xhumanisa imininingwane ewine ekugcineni ne-runtime yayo, umhloli wayo, nezinsiza zomsebenzi ezimenyezelwe.

Gcina ubufakazi bokuqeqesha nemiphumela yokukhetha kwe-Eval kuhlukaniswe ngokusobala. Uma i-Eval isetshenziselwa ukukhetha umuntu ozokhethwa, bika lokho njengokusebenza kokukhetha, hhayi njengesilinganiso esingenakho ukwenzelela sokusebenza emisebenzini engakabonwa.

Ukubambezeleka Nokusebenza Ngokuhambisana

Ku-run ye-Pokémon Eval egcinwe kungobo yomlando, i-harness ekhethiwe yaqeda isinqumo esiphelele ngesikhathi esimaphakathi esingu-568 milliseconds. Izicelo ze-Jev ngazinye zibe nesikhathi esimaphakathi esingu-269 milliseconds. I-harness yokuqala ibe nesikhathi esimaphakathi sesinqumo esiphelele esingu-678 milliseconds.

• Isinqumo esiphelele se-harness yokuqala: 678 ms median, 1,495 ms P95, izinqumo ezingu-238.
• Isinqumo esiphelele se-harness ekhethiwe: 568 ms median, 657 ms P95, izinqumo ezingu-113.
• Isicelo esisodwa se-Jev se-harness ekhethiwe: 269 ms median, 348 ms P95, izicelo ezingu-226.

Isikhathi sesinqumo esiphelele sihlanganisa ukubalwa kwezici, izingcingo ze-Jev ezisebenza ngokuhambisana, nokukhethwa kwesenzo sokugcina. Isikhathi sesicelo ngasinye sihlanganisa izindleko zeklayenti nezokuxhumana kwenethiwekhi. Izikhathi zezingcingo ezisebenza ngokuhambisana ziyedlulana, ngakho akufanele uzihlanganise. Ukulingiswa kwempi nesikhathi sokubhala noma sokuzindla kungaphandle kwalezi zilinganiso zezinqumo.

Lokhu kubonwa okugcinwe kungobo yomlando akuzange kufake ama-hit e-cache yezimpendulo zasendaweni, kodwa bekungeyona i-benchmark elawulwayo eqhathanisa ne-LLM eyenza zonke izinqumo ze-runtime.

Amathiphu Athuthukile

Bala amaqiniso ngaphambi kokucela isahlulelo

Sebenzisa ikhodi noma izisho ezinqumayo kumaqiniso afana nezenzo ezisemthethweni, izilinganiso zezinombolo, imingcele, nesimo esithathwe kokunye. Nikeza i-Jev isimo esigxilile nemibandela ecacile yokukhetha okungacacile okusele.

Yenza imibandela yesenzo ibe sobala

Chaza isenzo ngasinye esitholakalayo usebenzisa ubufakazi obuhambisana nomsebenzi. Ku-harness ye-Pokémon, imibandela ihlanganisa impilo esele, izilinganiso zomonakalo, isivinini, ama-turn adingekayo ukuze kukhishwe umphikisi, nezindleko zokuvumela ukuhlasela kwamahhala lapho ushintsha.

Sebenzisa ama-node ahambisanayo lapho ukuzimela kukhona ngempela

Ama-node azimele egrafu angasebenza ngesikhathi esisodwa, futhi imibuzo eminingi ingabelana ngesicelo esisodwa. Hlela igrafu ukuze usebenzise lokhu ngaphandle kokwethula ukuncika okungadingwa umsebenzi.

Hlola amathuba ngendlela efanele

Amathuba okukhetha ahlela izinketho ezinikeziwe ngaphansi kwesicelo. Ungawahumushi kabusha njengamathuba alinganisiwe empumelelo yomsebenzi wonke ngaphandle kokuthi umhloli wakho asungule ngokwehlukile leyo ncazelo.

Gcina amaphutha aphelele

Amaphutha angaveza imicabango engalungile, okuboniwe okushodayo, nemibandela entekenteke. Wagcine kuma-trace kanye nezinqumo eziphumelele ukuze ukuzindla kuthole ubufakazi obuphelele.

Gwema ukunqunywa buthule

Lungiselela umkhawulo wangempela wama-byte wokuzindla. I-JevHarness igcina kungobo yomlando futhi yenqabe okokufaka kokuzindla okudlula usayizi ovunyelwe esikhundleni sokukunquma, okuvikela ubuqotho bobufakazi kodwa kudinga ukuthi uhlele usayizi wezivivinyo ngendlela efanele.

Hlukanisa phakathi komqondo ovaliwe nokuziphatha komhlinzeki okuvalwe

Ukuvala i-harness kulungisa i-pipeline oyikhethile nezinsiza ezimenyezelwe, kodwa i-alias yemodeli ephethwe ingashintsha ngokuhamba kwesikhathi. Gcina izimpendulo ezirekhodiwe lapho ukuhlolwa komlando okunembile kubalulekile, futhi uphathe izingcingo ezintsha zomhlinzeki njengesimo esihlukile sokuphindaphindeka.

Sebenzisa imephu ye-repository

• auto_jev/ iqukethe ukuqinisekiswa kwemininingwane, i-runtime ehambisanayo, ezokuthutha ze-Jev, ukuzindla, i-GEPA, ukugcinwa, nokuvala.
• examples/pokemon/ iqukethe i-adaptha yempi ethembekile, ibhuloho lenjini yasendaweni eline-seed, ama-harness, nekhodi yokwethula.
• examples/pokemon/sample/ iqukethe i-harness ekhethiwe kanye nengobo yomlando ekhethiwe yewebhusayithi.
• docs/ iqukethe izinto zokubhala imisebenzi, ucingo lwe-Jev olurekhodiwe, amavidiyo, nezithombe-skrini.
• skills/jev-harness/ iqukethe imiyalelo yama-coding agent abhala ama-harness aqondene nomsebenzi.
• website/ iqukethe idemo yokufunda kuphela kanye ne-adaptha yokuthunyelwa.

Isiphetho

I-JevHarness iguqula isu elithuthukiswe yi-LLM libe umqondo we-runtime ocacile futhi ohlolisisekayo. Qala ngokuchaza umngcele oqinile womsebenzi, vumela ikhono lakhe lakhe futhi liqinisekise i-pipeline, bese usebenzisa ama-trace aphelele kanye nemivuzo ethembekile uma ufuna ukuthuthukiswa okusekelwe ekuzindleni. Uma umuntu ozokhethwa esekhethiwe, yivale ukuze izinqumo ezijwayelekile zisebenzise ikhodi yayo nezahlulelo ze-Jev ezigxilile ngaphandle kokubiza i-LLM ebhalayo kuzo zonke izenzo.