PAST-Bench Operational

PAST-Bench is a benchmark introduced by Mengdi Wang's team at Princeton University to evaluate the recursive self-improvement capabilities of personal AI agents. By comparing task performance under conditions with and without memory, PAST-Bench determines whether the cross-session experience saved by an agent genuinely improves its subsequent behavior. PAST-Bench covers four capabilities—memory, workflow reuse, information gathering, and state updating—with a total of 26 scenarios and 204 task episodes.

Evaluate recursive self-improvement: Test whether the cross-session experience saved by a personal AI agent (memories, skills, and task histories) genuinely improves its subsequent behavior.
Isolate the effects of experience accumulation: Distinguish whether score improvements come from accumulated experience or from other factors such as a stronger base model, prompt changes, or task difficulty.
Four-dimensional capability diagnosis: Cover four major dimensions—memory, workflow reuse, information gathering, and state updating—to identify which specific capabilities of an agent are deficient.
Mechanism attribution analysis: Look beyond the final score and trace the complete “storage → retrieval → application” pathway to determine whether improvements are supported by a genuine mechanism.

Sequential task-family design: Agents execute multiple rounds of conversations within the same task family in sequence. Early conversations create opportunities to save experience, while later conversations test whether that experience is used correctly.
Matched controlled experiments: A control version is designed for each later conversation with persistence disabled while all other conditions remain identical. The difference in self-evolution, Δ, is obtained through “with-memory score – without-memory score.”
Mechanism evidence tracing: Runtime telemetry such as memory writes, skill calls, session retrievals, and state updates is recorded to calculate the Mechanism-Evidence Score and verify whether improvements follow the expected pathway.
Persistent artifact auditing: Inspect the content actually saved by the agent (memory entries, skill files, and session indexes) and analyze its structure, timeliness, and reusability.

Environment setup: Clone the PAST-Bench repository from GitHub and install the core dependencies and agent adapter.
Model configuration: Configure the API Key for the model being used in the environment variables.
Sandbox construction: Run past-bench build-image --kind sandbox to build the Docker sandbox image required for evaluation.
Quick validation: Run a single task family with past-bench evolve and add the --compare-no-persistence parameter to perform a smoke test.
Full evaluation: Run evaluations across all 26 task families. Each task family automatically executes two controlled experiments: “with persistence” and “without persistence.”
Result analysis: View sequence_comparison.json in the --trace-dir directory to obtain the Δ value and mechanism evidence score.
Custom integration: To evaluate your own agent, implement an adapter in the agents/ directory and ensure that it supports the --compare-no-persistence option.

Genuine attribution capability: PAST-Bench can precisely distinguish whether improvements come from accumulated experience or from the model itself becoming stronger, prompt changes, or simpler tasks.
Matched controlled experiment design: Each task family has a control version with persistence disabled while all other conditions remain identical, enabling a direct causal comparison between memory and no-memory conditions.
Mechanism-level diagnosis: The Mechanism-Evidence Score determines not only whether an agent answered correctly, but also traces the complete “storage → retrieval → application” pathway, distinguishing between “getting the answer right by chance” and “genuinely reusing experience.”
Four-dimensional capability decomposition: Break down the vague concept of self-improvement into four specific capabilities—memory, workflow reuse, information gathering, and state updating—to accurately identify weaknesses.
Diagnosis-driven improvement: Runtime failures exposed by the benchmark directly led to the Hermes+ framework, demonstrating that PAST-Bench is not merely an evaluation tool but also a diagnostic engine for optimizing agent architectures.

GitHub repository: https://github.com/Gen-Verse/PAST-Bench
arXiv technical paper: https://arxiv.org/pdf/2608.04003

Academic research: Provide a standardized, reproducible, and quantitative evaluation environment for recursive self-improvement in agents, supporting objective comparisons across different architectures.
Framework development: Precisely identify weaknesses in agent frameworks through four-dimensional capability decomposition and guide architectural optimization, such as the development of Hermes+.
Model selection: Compare the cross-session experience reuse performance of different base models under a fixed framework and identify each model's capability biases.
Persistence validation: Test the practical effectiveness of different memory structures and retrieval strategies to avoid ineffective persistence where information is “saved but cannot be found” or “saved but not used.”
Product iteration: Determine whether score improvements in a new version come from “cross-session experience accumulation” or “a stronger base model,” ensuring that persistence features deliver genuine value.