Audit AI Agent Tool Calls with ToolReplay

What ToolReplay Does

ToolReplay audits recorded AI-agent tool calls without invoking the real tools. It examines a transcript to identify three problems: non-deterministic responses, redundant calls, and calls outside an agent's declared permissions. It can also seal a transcript into a SHA-256 hash chain and verify that the sealed file has not been edited.

Because ToolReplay replays recorded responses rather than making live calls, audits are safe, side-effect free, reproducible, and usable even when the original tools are unavailable.

Install and Run ToolReplay

ToolReplay requires Python 3.11 or newer and has no third-party runtime dependencies. It does not access the network.

From a source checkout, run the module directly:

python -m toolreplay version

You can also install the console script:

pip install .
toolreplay version

Both approaches print the installed version, such as:

toolreplay 0.6.0

Understand the Transcript Format

A transcript is a JSON Lines file. Every non-blank line represents one tool invocation and must contain exactly four fields:

• index: an integer starting at 0 and increasing by exactly 1.
• tool: the name of the called tool.
• args: an object containing the tool arguments.
• response: an object containing the recorded tool response.

A valid record looks like this:

{"index": 4, "tool": "write_file", "args": {"path": "docs/intro.md", "text": "edited"}, "response": {"ok": true}}

Parsing is intentionally strict. Unknown fields, missing fields, invalid object types, non-integer indexes, and out-of-order indexes cause a hard error. Boolean values are not accepted as integers even though Python treats bool as an int subclass.

ToolReplay compares calls using canonical JSON: the tool name and arguments are encoded with sorted keys and no unnecessary whitespace. Therefore, argument objects with different key ordering still represent the same call. Responses use the same canonical encoding when ToolReplay checks determinism.

Run a Replay Audit

The replay command checks a transcript against its own recorded responses. Try the dirty sample included in the repository:

python -m toolreplay replay samples/session-dirty.jsonl

The command produces a deterministic, line-oriented report:

calls: 6
divergence: index 5
findings: 2
index 2: redundant-call: tool 'read_file' repeats the identical call at index 1 with no state change between them
index 5: non-determinism: tool 'search' returned a different response than the identical call at index 3

The header is read from top to bottom. calls: 6 means six valid records were parsed. divergence: index 5 identifies the first point where a deterministic rerun would disagree with the recording. findings: 2 counts the findings that follow.

Findings are sorted by index and then by kind, so repeated audits of the same input produce byte-identical output.

Non-determinism

ToolReplay remembers the first response for each canonical call. If an identical call later has a different recorded response, it reports non-determinism. In the sample, the search call for install appears at indexes 3 and 5, but the responses report different hit counts. Index 5 becomes the divergence point.

This matters for agents because a decision based on the first answer might change if the second answer had been returned. The finding indicates that the session depends on information that is not stable within the recording.

Redundant Calls

A repeated call is redundant only when no recorded event between the two calls could have changed state. The default mutators are write_file, delete_file, create_file, move_file, and run_command. Any different call between identical calls is also treated conservatively as a possible state change.

In the sample, read_file reads docs/intro.md at indexes 1 and 2 with no intervening state change, so index 2 is flagged. A later read after a write_file call would not be considered redundant.

Check Agent Permissions

Use scope when you have a permission declaration for the agent:

python -m toolreplay scope samples/session-dirty.jsonl samples/scope.json

The sample produces:

calls: 6
findings: 1
index 4: permission-overreach: tool 'write_file' is not in the declared scope for agent 'docs-reader'

Replay and scope are separate commands by design. Replay evaluates a session against itself and needs only the transcript. Scope compares each call with a separate permission file.

A scope file is one JSON object containing the agent name and an exact list of permitted tools:

{"agent": "docs-reader", "allowed_tools": ["read_file", "list_dir", "search"]}

Tool names are matched exactly and case-sensitively. Missing fields, incorrect types, or non-string entries are rejected. Scope checking is name-based; it does not inspect arguments or determine whether a permitted tool targets an allowed path.

Seal a Transcript

The seal command converts a transcript into a hash-chained JSONL file:

python -m toolreplay seal samples/session-dirty.jsonl > sealed.jsonl

Each sealed link contains the original record fields plus prev and digest. The first link uses 64 zero hexadecimal characters as its previous digest. Every later SHA-256 digest includes the previous digest and the canonical bytes of the current record, including its index, tool, arguments, and response.

Because every link depends on the preceding link, changing an earlier record changes that record's digest and all later digests. The chain therefore makes edits visible and identifies where the first broken link occurs.

Verify a Sealed File

Verify an intact chain with:

python -m toolreplay verify sealed.jsonl

An unchanged file reports:

chain: intact

If a recorded response or any other chained field is modified, verification reports the first broken link and the expected and found digests:

chain: broken
first broken link: index 3
expected: 34d19a9dfb112baa07afa37994ddcbbebe2973674385f3942340653d08d205bd
found: 40aadf3b8156217f0b5b1d95010b6b79f82a26ccf5e780aed0f5f311f865028a

Sealing detects edits after sealing, including changes to ordering, indexes, tools, arguments, and responses. It is not a digital signature: anyone who can edit the file can also reseal it. Provenance would require signing the final digest with a key managed outside ToolReplay.

Use ToolReplay in CI

ToolReplay's exit codes make it suitable for build gates:

• 0: the audit is clean or the chain is intact.
• 1: findings are present or the chain is broken.
• 2: a usage error occurred, such as a malformed input or missing file.

Run replay and scope checks directly in a CI job:

python -m toolreplay replay session.jsonl
python -m toolreplay scope session.jsonl scope.json

Exit code 1 means the tool completed its audit and found something to report. Exit code 2 means the audit could not run, so automation can treat malformed input differently from an actual finding.

For reproducible comparisons, seal each run and diff the results:

python -m toolreplay seal run-a.jsonl > run-a.sealed.jsonl
python -m toolreplay seal run-b.jsonl > run-b.sealed.jsonl
git --no-pager diff --no-index run-a.sealed.jsonl run-b.sealed.jsonl

Since reports contain no timestamps or random values, identical inputs produce identical output. The first differing sealed line identifies where two runs stopped agreeing, while the changed digest shows that subsequent links depend on the difference.

Advanced Usage and Practical Limits

• Keep transcripts append-friendly. JSON Lines lets a producer append one record per call and makes changes easy to review in version control.
• Preserve recorded responses. Replay does not call real tools; it can only detect problems represented in the transcript.
• Interpret redundancy conservatively. The default mutator list is fixed, and custom mutating tools not in that list are treated as read-only for redundancy analysis.
• Remember the comparison boundary. Non-determinism is detected only when the same canonical call repeats. Different calls with semantically related responses are not evaluated for equivalence.
• Separate authority from targeting. Scope checks tool names only. They cannot catch a permitted tool being used on a forbidden resource.
• Use sealing for integrity, not identity. A chain shows whether a file changed after sealing, but it does not prove who created the chain.

Run the Test Suite

The project uses Python's standard-library unittest framework and has no third-party test dependency. From the repository root, run:

python -m unittest discover -s tests

The tests cover strict transcript parsing, canonical encoding, hash-chain creation and tamper detection, replay findings, scope enforcement, CLI behavior, and exit codes.

Conclusion

ToolReplay provides a focused audit workflow for AI-agent sessions. Start with replay to find inconsistent or wasteful behavior, use scope to check declared tool permissions, and use seal plus verify to detect changes after recording. Its strict input validation and deterministic reports make it practical for debugging, review, version control, and CI enforcement.