E-Bench – An Agent Evaluation Benchmark by Tencent Hunyuan and Others Operational

E-Bench is a benchmark for evaluating multi-step tool use, developed by the Tencent Hunyuan team in collaboration with Tsinghua AIR and Southeast University. Using Honor of Kings, QQ Music, and Tencent Meeting as prototypes, it builds fully synthetic virtual environments containing 323 state-change tasks, 41 database tables, and more than 76,000 data records. Through dual asymmetry in information and tools, it forces agents to proactively gather hidden information and orchestrate multi-step tool calls. It ultimately uses deterministic database state-diff scoring to reveal the capability boundaries of current AI agents in real-world product scenarios.

Fully synthetic product environment construction: Uses graph-guided database population to generate virtual worlds for three products with no isolated records and cross-table consistency.
Dual-gap task generation: The task-generation model has full database access, while the evaluated model can interact only through restricted business tools, creating both information and tool asymmetry.
Deterministic state evaluation: Uses the state diff before and after database operations as the sole evaluation criterion. Only exact matches pass, eliminating partial credit and judge variance.
Joint cost-performance analysis: Records API costs for every task and plots the accuracy-cost Pareto frontier to evaluate model cost-effectiveness.
Code execution extension: The E-Bench-Code version provides an exec_code tool, allowing the orchestration and information-acquisition difficulties to be separated.

Graph-guided database population: Starting from a relational schema, it builds a table-level dependency graph and populates root and downstream tables in topological order. A powerful LLM serves as a constrained synthesizer and writes data only through insertion tools. Downstream tables must query the current database for candidate entities. Deterministic scripts then verify and correct semantic errors such as temporal ordering and aggregate counts, ensuring a complete and self-consistent environment.
Generator-solver asymmetry: The task-generation Agent has query_sql and exec_code privileges, allowing it to inspect the complete database and perform complex calculations. The evaluated Agent can call only business-level tools, creating an information gap, with global state hidden, and a tool gap, with internal computation tools removed. This forces the model to actively explore the environment through multi-step, parallel tool calls.
Intent rewriting and rule substitution: Replaces specific database-derived values with the rules that produce them. For example, a task says “add all songs that have been removed from the collection” rather than directly providing song IDs, ensuring that the task cannot be completed independently of environmental interaction.
Cross-validated task generation: Each task is generated through a product-oriented cycle of inspecting data, determining the target, modifying the state, and summarizing the intent. Three different strong Agent models cross-validate each task, and it is accepted only when at least two reproduce the same result, ensuring solvability and annotation accuracy.

Environment preparation: Obtain the E-Bench fully synthetic virtual environment from the resources accompanying the paper, and load database templates and 323 evaluation tasks covering Honor of Kings, QQ Music, and Tencent Meeting.
Version selection: Choose the base E-Bench or E-Bench-Code according to the research objective.
Model integration: Connect the evaluated LLM Agent to the evaluation framework and configure it to interact indirectly with the environment only through restricted domain tools, such as searching for songs, creating meetings, or adding friends. Direct queries to the underlying database are prohibited.
Task configuration: Set the number of independent runs for each task and configure each run to start with a fresh database copy. Enable API call cost tracking to ensure reproducibility.
Evaluation execution: Start the automated evaluation. After receiving natural-language instructions, the Agent interacts with the virtual environment through multi-step tool calls and modifies its state. The system automatically records the complete operation trace and token consumption.
Result analysis: After evaluation, the system uses deterministic database state diffs to output Pass@1 and Pass³ accuracy, and generates a joint cost-performance analysis report to help assess model cost-effectiveness.

Sufficient real-scale distractions: Each database is populated with more than 76,000 rows and over 600,000 data cells. Target entities are surrounded by many similar records, preventing the model from guessing correctly merely because the environment is sparse.
Decoupled and reusable environments and tasks: A self-consistent product environment can support the generation of approximately 100 tasks, with extremely low marginal cost. The environment layer is controllable, while the task layer is extensible.
Stable and reproducible evaluation: Fully synthetic environments are unaffected by changes to online services, while deterministic diff scoring eliminates the subjective bias of semantic judges.
Clear difficulty levels: The base version tests multi-step orchestration and information gathering, while the Code version additionally tests the ability to offload computation-intensive tasks to code.

arXiv technical paper: https://arxiv.org/pdf/2607.23722

AI agent capability evaluation: Provides LLM Agents with standardized and quantifiable evaluation of multi-step tool use, covering core dimensions such as information retrieval, state changes, and cross-entity orchestration.
Model selection and productization decisions: Uses cost-performance Pareto charts to help enterprises balance accuracy against API costs and identify cost-effective models.
Synthetic training data for agent training: Serves as a controllable and scalable source of training data for continuously improving Agent capabilities in multi-step tool calling and reliability.
Research on code-enhanced Agents: E-Bench-Code supports research into the benefits of code execution for different task types, including the boundary between computation-intensive tasks and reasoning-based decisions.
Cross-domain benchmark expansion: The methodology can be transferred to more product backends and CLI scenarios, advancing the establishment of general-purpose agent evaluation standards.