Hlola Ukubizwa Kwamathuluzi Kwe-ejenti ye-AI nge-ToolReplay

Lokho okwenziwa yi-ToolReplay

I-ToolReplay ihlola izingcingo zamathuluzi zama-ejenti e-AI ezirekhodiwe ngaphandle kokusebenzisa amathuluzi angempela. Ihlaziya umbhalo wengxoxo ukuze ibone izinkinga ezintathu: izimpendulo ezinganqunywa kusengaphambili, izingcingo eziphindaphindwayo, nezingcingo ezingaphandle kwezimvume ezimenyezelwe yi-ejenti. Ingaphinde ifake umbhalo wengxoxo ochungechungeni lwezihlanganisi ze-SHA-256 futhi iqinisekise ukuthi ifayela elivaliwe alihlelwanga.

Ngenxa yokuthi i-ToolReplay iphinda idlale izimpendulo ezirekhodiwe kunokwenza izingcingo ezibukhoma, ukuhlolwa kuphephile, akunamiphumela eseceleni, kuyaphindaphindeka, futhi kuyasebenziseka noma amathuluzi okuqala engatholakali.

Faka futhi usebenzise i-ToolReplay

I-ToolReplay idinga i-Python 3.11 noma entsha futhi ayinakho ukuncika kwamanye amaphakheji esikhathi sokusebenza. Ayifinyeleli kunethiwekhi.

Kusuka ekukopishweni kwekhodi yomthombo, sebenzisa imojuli ngokuqondile:

python -m toolreplay version

Ungafaka futhi iskripthi sekhonsoli:

pip install .
toolreplay version

Zombili lezi zindlela ziphrinta inguqulo efakiwe, isibonelo:

toolreplay 0.6.0

Qonda ifomethi yombhalo wengxoxo

Umbhalo wengxoxo uyifayela le-JSON Lines. Umugqa ngamunye ongewona ongenalutho umele ukubizwa kwethuluzi elilodwa futhi kufanele ube nezinkambu ezine ngqo:

• index: inombolo ephelele eqala ku-0 futhi ikhuphuka ngo-1 ngqo.
• tool: igama lethuluzi elibiziwe.
• args: into equkethe izimpikiswano zethuluzi.
• response: into equkethe impendulo yethuluzi erekhodiwe.

Rekhodi elivumelekile libukeka kanje:

{"index": 4, "tool": "write_file", "args": {"path": "docs/intro.md", "text": "edited"}, "response": {"ok": true}}

Ukuhlaziya kuqinile ngenhloso. Izinkambu ezingaziwa, izinkambu ezingekho, izinhlobo zezinto ezingavumelekile, ama-index angewona amanombolo aphelele, nama-index angahleleki kahle kubangela iphutha elikhulu. Amanani e-Boolean awamukelwa njengamanombolo aphelele nakuba i-Python iphatha i-bool njengohlobo oluncane lwe-int.

I-ToolReplay iqhathanisa izingcingo isebenzisa i-JSON ejwayelekile: igama lethuluzi nezimpikiswano kufakwa ikhodi kusetshenziswa okhiye abahlelwe futhi kungabikho izikhala ezingadingekile. Ngakho-ke, izinto zezimpikiswano ezinokuhleleka kokhiye okuhlukile zisamele ucingo olufanayo. Izimpendulo zisebenzisa ukufakwa kwekhodi okufanayo okujwayelekile lapho i-ToolReplay ihlola ukunqunywa kusengaphambili.

Yenza ukuhlolwa kokudlala futhi

Umyalo we-replay uhlola umbhalo wengxoxo ngokumelene nezimpendulo zawo ezirekhodiwe. Zama isampula engcolile efakwe endaweni yokugcina ikhodi:

python -m toolreplay replay samples/session-dirty.jsonl

Umyalo ukhiqiza umbiko onqunywe kusengaphambili, ohlelwe ngemigqa:

calls: 6
divergence: index 5
findings: 2
index 2: redundant-call: tool 'read_file' repeats the identical call at index 1 with no state change between them
index 5: non-determinism: tool 'search' returned a different response than the identical call at index 3

Isihloko sifundwa ukusuka phezulu kuye phansi. calls: 6 kusho ukuthi kuhlaziywe amarekhodi ayisithupha avumelekile. divergence: index 5 ikhomba indawo yokuqala lapho ukuphinda kwenziwe okunqunywe kusengaphambili bekungaphikisana nokurekhodiwe. findings: 2 ibala okutholakele okulandelayo.

Okutholakele kuhlelwa nge-index bese kuba ngohlobo, ngakho ukuhlolwa okuphindwayo kokokufaka okufanayo kukhiqiza okukhiphayo okufana ngebhayithi.

Ukunganqunywa kusengaphambili

I-ToolReplay ikhumbula impendulo yokuqala yocingo ngalunye olujwayelekile. Uma ucingo olufanayo kamuva lunempendulo erekhodiwe ehlukile, ibika ukunganqunywa kusengaphambili. Esampuleni, ucingo lwe-search lwe-install luvela kuma-index 3 no-5, kodwa izimpendulo zibika inani lemiphumela elihlukile. I-index 5 iba yindawo yokuphambuka.

Lokhu kubalulekile kuma-ejenti ngoba isinqumo esisekelwe empendulweni yokuqala singashintsha ukuba kubuyiswe impendulo yesibili. Okutholakele kubonisa ukuthi iseshini incike olwazini olungazinzile ngaphakathi kokurekhodiwe.

Izingcingo eziphindaphindwayo

Ucingo oluphindwayo luyaphindaphindeka kuphela lapho kungekho mcimbi orekhodiwe phakathi kwezingcingo ezimbili obungashintsha isimo. Abaguquli abazenzakalelayo yi-write_file, delete_file, create_file, move_file, kanye ne-run_command. Noma yiluphi ucingo oluhlukile phakathi kwezingcingo ezifanayo nalo luphathwa ngokuqapha njengoshintsho lwesimo olungenzeka.

Esampuleni, i-read_file ifunda i-docs/intro.md kuma-index 1 no-2 kungekho shintsho lwesimo phakathi kwazo, ngakho i-index 2 imakwa. Ukufunda okwenzeka kamuva ngemva kocingo lwe-write_file bekungeke kubhekwe njengokuphindaphindwayo.

Hlola izimvume ze-ejenti

Sebenzisa i-scope uma unesimemezelo sezimvume ze-ejenti:

python -m toolreplay scope samples/session-dirty.jsonl samples/scope.json

Isampula likhiqiza:

calls: 6
findings: 1
index 4: permission-overreach: tool 'write_file' is not in the declared scope for agent 'docs-reader'

I-replay ne-scope yimiyalo ehlukene ngenhloso. I-replay ihlola iseshini ngokumelene nayo futhi idinga umbhalo wengxoxo kuphela. I-scope iqhathanisa ucingo ngalunye nefayela elihlukile lezimvume.

Ifayela le-scope liyinto eyodwa ye-JSON equkethe igama le-ejenti nohlu oluqondile lwamathuluzi avunyelwe:

{"agent": "docs-reader", "allowed_tools": ["read_file", "list_dir", "search"]}

Amagama amathuluzi afaniswa ngokunembile futhi ngokuzwela osonhlamvukazi nabancane. Izinkambu ezingekho, izinhlobo ezingalungile, noma okufakiwe okungewona amayunithi ezinhlamvu kuyaliwa. Ukuhlola i-scope kusekelwe egameni; akuhloli izimpikiswano futhi akunqumi ukuthi ithuluzi elivunyelwe libhekisa endleleni evunyelwe yini.

Vala umbhalo wengxoxo

Umyalo we-seal uguqula umbhalo wengxoxo ube ifayela le-JSONL elixhunywe ngama-hash:

python -m toolreplay seal samples/session-dirty.jsonl > sealed.jsonl

Isixhumanisi ngasinye esivaliwe siqukethe izinkambu zerekhodi lokuqala kanye ne-prev ne-digest. Isixhumanisi sokuqala sisebenzisa izinhlamvu ze-hexadecimal ezingu-64 ezinguziro njenge-digest yaso yangaphambilini. I-digest ngayinye elandelayo ye-SHA-256 ihlanganisa i-digest yangaphambilini namabhayithi e-canonical erekhodi lamanje, okuhlanganisa inkomba yalo, ithuluzi, izimpikiswano, nempendulo.

Ngenxa yokuthi isixhumanisi ngasinye sincike kuleso esandulele, ukushintsha irekhodi langaphambili kushintsha i-digest yalelo rekhodi nawo wonke ama-digest alandelayo. Ngakho-ke uchungechunge lwenza ukuhlelwa kubonakale futhi lukhomba lapho kuvela khona isixhumanisi sokuqala esiphukile.

Qinisekisa Ifayela Elivaliwe

Qinisekisa uchungechunge olungaguquki usebenzisa:

python -m toolreplay verify sealed.jsonl

Ifayela elingashintshiwe libika:

chain: intact

Uma impendulo erekhodiwe noma enye inkambu exhunywe ochungechungeni ishintshiwe, ukuqinisekisa kubika isixhumanisi sokuqala esiphukile kanye nama-digest alindelekile natholakele:

chain: broken
first broken link: index 3
expected: 34d19a9dfb112baa07afa37994ddcbbebe2973674385f3942340653d08d205bd
found: 40aadf3b8156217f0b5b1d95010b6b79f82a26ccf5e780aed0f5f311f865028a

Ukuvala kuthola ukuhlelwa ngemva kokuvalwa, okuhlanganisa izinguquko ekuhlelekeni, ezinkombeni, kumathuluzi, ezimpikiswaneni, nasezimpendulweni. Akusona isiginesha yedijithali: noma ubani ongakwazi ukuhlela ifayela angakwazi nokulivala kabusha. Ukuqinisekisa umsuka kungadinga ukusayinda i-digest yokugcina ngokhiye ophethwe ngaphandle kwe-ToolReplay.

Sebenzisa i-ToolReplay ku-CI

Amakhodi okuphuma e-ToolReplay ayenza ifanele izivimbamnyango zokwakha:

• 0: ukuhlolwa kuhlanzekile noma uchungechunge aluphukile.
• 1: kukhona okutholakele noma uchungechunge luphukile.
• 2: kwenzeke iphutha lokusebenzisa, njengokufakwa okungavumelekile noma ifayela elingekho.

Qalisa ukudlala kabusha nokuhlolwa kobubanzi ngokuqondile emsebenzini we-CI:

python -m toolreplay replay session.jsonl
python -m toolreplay scope session.jsonl scope.json

Ikhodi yokuphuma engu-1 isho ukuthi ithuluzi liqedile ukuhlola kwalo futhi lithole okuthile elizokubika. Ikhodi yokuphuma engu-2 isho ukuthi ukuhlola akukwazanga ukuqalwa, ngakho okuzenzakalelayo kungaphatha ukufakwa okungavumelekile ngokuhlukile kokutholakele kwangempela.

Ukuze kuqhathaniswe imiphumela ngendlela ephindaphindekayo, vala ukusebenza ngakunye bese uqhathanisa imiphumela:

python -m toolreplay seal run-a.jsonl > run-a.sealed.jsonl
python -m toolreplay seal run-b.jsonl > run-b.sealed.jsonl
git --no-pager diff --no-index run-a.sealed.jsonl run-b.sealed.jsonl

Ngenxa yokuthi imibiko ayinazo izitembu zesikhathi noma amanani angahleliwe, okufakiwe okufanayo kukhiqiza umphumela ofanayo. Umugqa wokuqala ovaliwe ohlukile ukhomba lapho ukusebenza okubili kuyeke ukuvumelana khona, kuyilapho i-digest eshintshiwe ikhombisa ukuthi izixhumanisi ezilandelayo zincike kulowo mehluko.

Ukusetshenziswa Okuthuthukile Nemikhawulo Ebonakalayo

• Gcina imibhalo ilungele ukwengezwa. I-JSON Lines ivumela umkhiqizi ukuthi engeze irekhodi elilodwa ocingweni ngalunye futhi yenza izinguquko kube lula ukuzibuyekeza ku-version control.
• Gcina izimpendulo ezirekhodiwe. Ukudlala kabusha akubizi amathuluzi angempela; kungathola kuphela izinkinga ezimelelwe embhalweni.
• Humusha ukuphindaphindeka ngokucophelela. Uhlu oluzenzakalelayo lwamathuluzi aguqulayo lumisiwe, futhi amathuluzi enziwe ngokwezifiso aguqulayo angekho kulolo hlu aphathwa njengamathuluzi okufunda kuphela ekuhlaziyweni kokuphindaphindeka.
• Khumbula umngcele wokuqhathanisa. Ukung deterministiki kutholwa kuphela lapho ucingo olufanayo lwe-canonical luphindwa. Izingcingo ezihlukile ezinezimpendulo ezihlobene ngokwencazelo azihlolwa ngokulingana.
• Hlukanisa igunya ekuqondiseni. Ukuhlolwa kobubanzi kuhlola amagama amathuluzi kuphela. Ngeke kubambe ithuluzi elivunyelwe lisetshenziswa esisetshenziswa esingavunyelwe.
• Sebenzisa ukuvala ubuqotho, hhayi ubuwena. Uchungechunge lubonisa ukuthi ifayela lishintshile yini ngemva kokuvalwa, kodwa alufakazeli ukuthi ubani owadala uchungechunge.

Qalisa Isethi Yokuhlola

Le phrojekthi isebenzisa uhlaka lwe-unittest olusemthaleni ojwayelekile we-Python futhi ayinakho ukuncika kokuhlola okuvela eceleni. Usuka empandeni yenqolobane, qalisa:

python -m unittest discover -s tests

Ukuhlola kuhlanganisa ukuhlukaniswa okuqinile kombhalo, ukufakwa kwe-canonical, ukudalwa kochungechunge lwama-hash nokutholwa kokuphazamisa, okutholakele ekudlaleni kabusha, ukuphoqelelwa kobubanzi, ukuziphatha kwe-CLI, namakhodi okuphuma.

Isiphetho

I-ToolReplay inikeza ukuhamba komsebenzi okugxile ekuhloleni amaseshini ama-agent e-AI. Qala nge-replay ukuze uthole ukuziphatha okungahambisani noma okumosha, sebenzisa i-scope ukuhlola izimvume zamathuluzi ezimenyezelwe, bese usebenzisa i-seal kanye ne-verify ukuthola izinguquko ngemva kokuqoshwa. Ukuqinisekiswa kwayo okuqinile kokufakwayo nemibiko enqunywa ngendlela efanayo kuyenza ibe usizo ekulungiseni amaphutha, ekubuyekezeni, ekulawuleni izinguqulo, nasekuphoqeleleni i-CI.