Build Fast, Task-Specific Decision Pipelines with JevHarness

What JevHarness Does

JevHarness helps an authoring LLM create a task-specific decision harness for Jev. The harness transforms allowed observations into useful features, constructs structured Jev questions and criteria, and combines the answers into legal task actions.

The central idea is to separate expensive strategy development from routine execution:

Reason deeply during development. Freeze the strategy. Let Jev make fast, fuzzy decisions.

During development, an LLM can revise the harness code, questions, graph, state, criteria, and memory. After you select and freeze a harness, runtime decisions use that fixed logic and its Jev calls. The authoring LLM does not need to reason through every action again.

This makes JevHarness suitable for tasks where code can calculate reliable facts but some decisions remain contextual or ambiguous. For example, code can determine legal actions and calculate numeric features, while Jev judges which available action best satisfies task-specific criteria.

Architecture and Responsibility Boundaries

JevHarness separates the trusted task adapter from the decision harness. This boundary prevents the harness from changing its own reward or accessing hidden task state.

• Task adapter: Owns observations, legal actions, side effects, and scoring.
• Harness: Owns feature construction, Jev judgments, and decision logic.
• Environment: Executes actions and returns trusted outcomes or rewards.
• Authoring LLM: Creates or revises the harness during development and optional reflection.

A validated PipelineSpec is executed through PipelineRuntime. Once selected, the pipeline can be frozen together with its specification, runtime, evaluator, and declared task resources.

Key Features

Structured Jev judgments

Jev supports choice, score, and noul answer types. A request can contain multiple named questions, and independent graph nodes can execute concurrently.

Explicit strategy representation

The strategy is preserved in code, computed features, state, instructions, criteria, memory, and control flow. This makes the decision process inspectable and reusable instead of leaving it inside an authoring conversation.

Complete execution traces

Reflection can receive complete records of selected episodes, including observations, decisions, node inputs and outputs, Jev questions and answers, memory, and failures. Lossless deduplication reduces repetition. If reflection input exceeds the configured byte limit, it is archived and rejected rather than silently truncated.

Optional reward-based improvement

The optional GEPA integration selects parents from an instance frontier, compares each parent and proposal on the same training batch, and evaluates accepted proposals on Eval. It records actual ancestry, including rejected proposals.

Inspectable frozen artifacts

Freezing preserves the chosen harness and its declared dependencies. A frozen harness containing Jev nodes still needs Jev at runtime. Also note that a hosted model alias does not guarantee unchanged future provider behavior, so stored responses and fresh calls provide different levels of reproducibility.

Prerequisites

Choose the requirements that match what you want to run:

• Archived website: Node.js and a modern browser. It requires no model credentials, game engine, or Python sandbox.
• Python project: Python 3.11 or later.
• Functional Python nodes: A supported macOS native sandbox. If isolation is unavailable, execution fails closed.
• Version 2 expression and Jev flows: These do not launch the functional Python workers.
• Live Jev calls: A supported provider account and the corresponding API key.
• Optional authoring and reflection: A configured Proposer using a local Claude CLI, OpenAI, Azure, or Anthropic endpoint.

Install the Skill in Claude Code

The fastest Claude Code setup is to add the repository as a plugin marketplace, install the plugin, and reload plugins.

/plugin marketplace add https://github.com/TianyuCodings/JevHarness.git
/plugin install jev-harness@jevharness
/reload-plugins

Installing the plugin downloads the skill and its references, but it does not install Python dependencies or configure model credentials. If the repository is private, GitHub authentication must already work and your account must have read access.

Install the Skill in Codex

In a Codex host that provides the built-in skill-installer, request installation directly from the repository:

$skill-installer Install the jev-harness skill from https://github.com/TianyuCodings/JevHarness/tree/main/skills/jev-harness

Use the skill on a subsequent turn. If Codex does not discover it immediately, restart the session.

Manual Installation

You can install the skill manually for Claude Code, Codex, or both. Replace /path/to/your-project with an existing project directory.

git clone https://github.com/TianyuCodings/JevHarness.git
cd JevHarness
python3 scripts/install-skill.py --target both --scope project --project /path/to/your-project

This creates project-level installations at .agents/skills/jev-harness/ for Codex and .claude/skills/jev-harness/ for Claude Code. Use --target codex or --target claude to install only one.

For a user-level installation available across projects, run:

python3 scripts/install-skill.py --target both --scope user

The personal installation locations are ~/.agents/skills/jev-harness/ and ~/.claude/skills/jev-harness/. The installer refuses to overwrite a different existing installation. Restart the agent session if the skill is not discovered.

Explore the Archived Pokémon Example

The included Pokémon example provides an archived run, a selected harness, a browsable evolution tree, complete decision records, rejected proposals, and replay highlights synchronized with Jev answers. Browsing this archive makes no model requests and starts no games.

1. Clone the repository.
2. Build the website.
3. Start the local preview server.
4. Open the site in a modern browser.

git clone https://github.com/TianyuCodings/JevHarness.git
cd JevHarness
node website/build.mjs
node website/preview.mjs --port 8768

Open localhost:8768. Battle animations download assets from the official Pokémon Showdown renderer. The archived battle log remains in the browser and is not uploaded to a replay server.

The reported Pokémon result improved Eval win rate from 25 percent, or 3 of 12 games, to 75 percent, or 9 of 12 games, after five reflection rounds. The search retained the round-three candidate as its best harness. This was measured on the selection Eval set and should not be treated as an independent estimate of performance on unseen games.

Understand a Basic Decision

A harness writes a task-specific Jev request containing a state, named questions, an answer type, and the instructions and criteria used to judge available actions.

In the recorded Pokémon example, the harness calculates that Slowbro is likely to lose the damage race against Gastrodon. It presents four legal options, including three moves and a switch to Scizor. The action question uses the choice type and requires Jev to return exactly one listed action ID.

{
  "questions": {
    "action": {
      "type": "choice",
      "instructions": "Choose the single action most likely to win the whole battle, not only this turn.",
      "criteria": {
        "switch:2": "switch to Scizor; switching concedes one free attack",
        "move:2": "Psychic; roughly 3 hits to knock out the target",
        "move:3": "Ice Beam; roughly 4 hits to knock out the target",
        "move:4": "Slack Off; predicted incoming damage exceeds current HP"
      }
    }
  }
}

Jev selects switch:2 with an action-choice probability of 0.72, and the harness accepts that choice. That value compares the listed actions; it is not a prediction that the player has a 72 percent chance of winning the battle.

The complete recorded request, answer, timing, and provenance are available in the Pokémon turn-12 JSON record.

Build Your Own Harness

You do not need to handwrite all Jev instructions and criteria. Invoke the installed skill and describe the task, then let the coding agent gather the required contract details.

Claude Code plugin invocation

/jev-harness:jev-harness Build a harness that routes support tickets to the right team.
First clarify my inputs, legal actions, examples, success criteria, and budget.
If we have reliable rewards, add evaluation and reflection optimization.

Standalone Claude Code invocation

/jev-harness Build a harness that routes support tickets to the right team.
First clarify my inputs, legal actions, examples, success criteria, and budget.
If we have reliable rewards, add evaluation and reflection optimization.

Codex invocation

$jev-harness Build a harness that routes support tickets to the right team.
First clarify my inputs, legal actions, examples, success criteria, and budget.
If we have reliable rewards, add evaluation and reflection optimization.

The skill should first clarify the task rather than immediately generating a pipeline. Expect it to gather information about:

• Allowed task observations and hidden state boundaries.
• Legal actions and how actions are validated.
• Representative examples and available data.
• Success criteria, rewards, or evaluation procedures.
• Runtime and isolation requirements.
• Credentials and provider availability.
• Experiment budget and optimization resources.

Once the contract is clear, the agent can build and validate the harness. If reliable feedback exists and you request optimization, it can use trajectories and rewards for reflection before freezing the selected result.

Configure Jev Credentials

JevHarness supports two documented Jev transport configurations:

• Vercel AI Gateway: Create JevClient(transport="vercel") and set AI_GATEWAY_API_KEY.
• TypeSafe directly: Create JevClient(transport="typesafe") and set TYPESAFE_API_KEY.

Provider adapters do not guarantee that every model or endpoint is available to your account. Store credentials in environment variables or an ignored local .env file. Never put credentials in task observations, traces, or frozen artifacts.

Evaluate and Improve the Harness

Reflection is optional. Use it only when you have a trustworthy reward or evaluation method. A practical workflow is:

1. Define the task boundary. Keep hidden state, scoring, legal-action validation, and side effects in the trusted adapter.
2. Create a baseline harness. Compute reliable features in code and reserve Jev for context-sensitive judgments.
3. Validate the pipeline. Confirm that the PipelineSpec and runtime satisfy the task contract.
4. Collect complete traces. Preserve observations, node inputs, outputs, questions, answers, memory, failures, and rewards.
5. Propose revisions. Let the authoring LLM inspect evidence and revise code, prompts, graph structure, or memory.
6. Compare fairly. Compare a parent and proposal on the same training batch.
7. Evaluate accepted proposals. Run full Eval assessment and retain actual candidate ancestry, including rejected proposals.
8. Freeze the selected harness. Bind the winning specification to its runtime, evaluator, and declared task resources.

Keep training evidence and Eval selection results clearly separated. If Eval is used to choose a candidate, report it as selection performance rather than as an unbiased estimate of unseen-task performance.

Latency and Concurrency

In the archived Pokémon Eval run, the selected harness completed a full decision in a median of 568 milliseconds. Individual Jev requests had a median duration of 269 milliseconds. The initial harness had a median full-decision time of 678 milliseconds.

• Initial harness full decision: 678 ms median, 1,495 ms P95, 238 decisions.
• Selected harness full decision: 568 ms median, 657 ms P95, 113 decisions.
• Selected harness individual Jev request: 269 ms median, 348 ms P95, 226 requests.

Full-decision time includes feature computation, parallel Jev calls, and final action selection. Individual request time includes client and network overhead. Parallel call durations overlap, so you should not add them together. Battle simulation and authoring or reflection time are outside these decision measurements.

These archived observations excluded local response-cache hits, but they were not a controlled benchmark against an LLM making every runtime decision.

Advanced Tips

Compute facts before asking for judgment

Use deterministic code or expressions for facts such as legal actions, numeric estimates, thresholds, and derived state. Give Jev a focused state and explicit criteria for the remaining fuzzy choice.

Make action criteria concrete

Describe each available action using task-relevant evidence. In the Pokémon harness, criteria include remaining health, damage estimates, speed, turns required for a knockout, and the cost of conceding a free attack when switching.

Use parallel nodes where independence is real

Independent graph nodes can run concurrently, and multiple questions can share one request. Structure the graph to exploit this without introducing dependencies that the task does not require.

Inspect probabilities correctly

A choice probability ranks the supplied options under the request. Do not reinterpret it as a calibrated probability of overall task success unless your evaluator separately establishes that meaning.

Preserve complete failures

Failures can reveal invalid assumptions, missing observations, and brittle criteria. Keep them in traces alongside successful decisions so reflection receives the full evidence.

Avoid silent truncation

Configure a realistic reflection byte limit. JevHarness archives and rejects oversized reflection inputs rather than truncating them, which protects the integrity of the evidence but requires you to plan experiment sizes appropriately.

Distinguish frozen logic from frozen provider behavior

Freezing the harness fixes your selected pipeline and declared resources, but a hosted model alias may change over time. Retain recorded responses when exact historical inspection matters, and treat fresh provider calls as a different reproducibility condition.

Use the repository map

• auto_jev/ contains specification validation, parallel runtime, Jev transports, reflection, GEPA, storage, and freezing.
• examples/pokemon/ contains the trusted battle adapter, seeded local engine bridge, harnesses, and presentation code.
• examples/pokemon/sample/ contains the selected harness and curated website archive.
• docs/ contains task-authoring material, a recorded Jev call, videos, and screenshots.
• skills/jev-harness/ contains instructions for coding agents that author task-specific harnesses.
• website/ contains the read-only demonstration and deployment adapter.

Conclusion

JevHarness turns an LLM-developed strategy into explicit, inspectable runtime logic. Start by defining a strict task boundary, let the skill build and validate a pipeline, and use complete traces plus trustworthy rewards if you want reflection-based improvement. Once a candidate is selected, freeze it so routine decisions can use its code and focused Jev judgments without invoking the authoring LLM for every action.