Build and Improve AI Functions with jev-align

What is jev-align?

jev-align is an experimental command-line tool from Sutro for building AI Functions with TypeSafe's Jev. It evaluates a dataset, identifies uncertain examples, asks you to label them, and uses GEPA to propose an improved function definition.

The tool is designed for an iterative workflow. You can create an AI Function from local data, review proposed changes, save your progress, use the function in an application, and later improve it with examples captured in production.

Every label comes from you. A higher training score does not cause jev-align to accept a proposal automatically.

Key features

• Uncertainty-driven labeling: The CLI prioritizes ambiguous rows while also including a random audit sample.
• Human-controlled optimization: You decide whether to accept or reject each proposed definition.
• Multiple task types: Build binary, multiclass, multilabel, or ordered scoring functions.
• Local dataset discovery: The guided setup finds CSV, Parquet, and JSONL files.
• Flexible model providers: Run Jev through TypeSafe AI, Vercel AI Gateway, or Cloudflare Workers AI.
• Separate reflection models: Configure OpenAI, Anthropic, Gemini, or another LiteLLM-compatible provider for GEPA.
• Persistent runs: Resume previous optimization sessions and inspect saved AI Functions.
• Production capture: Collect useful application examples and label them during a later optimization round.

Understand the supported task types

Before starting, choose the output format that matches your problem:

• Binary: Returns True or False.
• Multiclass: Returns exactly one label from a fixed set.
• Multilabel: Returns zero or more labels from a fixed set.
• Score: Returns one level from an ordered rubric.

Install jev-align

Prerequisites

You need Python 3.11 or newer. You also need credentials for a Jev provider and a supported reflection model provider.

Install with uv

The recommended quick-start installation uses uv:

uv tool install jev-align

Install with pip

If you do not use uv, install the package with pip:

pip install jev-align

Configure credentials

For the standard TypeSafe AI route, export a TypeSafe API key. GEPA also needs a reflection model provider, such as OpenAI, Anthropic, or Gemini.

export TYPESAFE_API_KEY="..."
export OPENAI_API_KEY="..."

You can use ANTHROPIC_API_KEY or GEMINI_API_KEY instead of an OpenAI key. Vercel AI Gateway and Cloudflare Workers AI can replace TypeSafe AI as the Jev provider; those routes do not require TYPESAFE_API_KEY.

Start the guided setup

Launch the CLI with either command:

jeva

or:

jev-align

The guided setup discovers local CSV, Parquet, and JSONL datasets and includes three ready-to-run examples. Follow the prompts to select a dataset, define the task, choose the relevant fields, and configure the providers.

CLI navigation controls

• Use the arrow keys and Enter to navigate menus.
• Press b to return to the previous label.
• Enter /back to leave the rationale prompt.
• Press Space to toggle choices in a multilabel task.

Run an optimization task with flags

You can bypass parts of the guided workflow by supplying command-line options. The following example builds a binary function that determines whether a post concerns aviation:

jeva optimize posts.csv \
  --question "Is the post related to aviation?" \
  --column title \
  --column text \
  --pool-size 1000

The repeated --column options tell jev-align to evaluate the title and text fields. The --pool-size 1000 option limits the working pool to 1,000 rows.

By default, jev-align uses the first 1,000 rows, or the entire dataset when it contains fewer rows. You can concatenate all fields or choose specific columns.

Configure classes and score levels

For multiclass and multilabel tasks, provide repeated --class "NAME=DESCRIPTION" options. For scoring tasks, provide repeated --score-level options. Because the exact available flags may evolve, consult the built-in reference:

jeva optimize --help

How each optimization round works

1. Evaluate the dataset: jev-align runs the current function against the configured data and measures uncertainty.
2. Select examples: It chooses ambiguous rows and adds a random audit sample.
3. Collect labels: You label the selected examples and can optionally provide rationales.
4. Run GEPA: GEPA uses the accumulated feedback to propose an improved definition.
5. Review the proposal: The CLI displays the score, certainty change, and proposed definition diff.
6. Choose the next action: You can accept, reject, rewind, or stop and resume later.

This review step is important: score improvements remain proposals until you explicitly accept them.

Adjust advanced optimization settings

The guided Advanced menu exposes several controls:

• Choose 5, 10, 15, or 20 training annotations per round.
• Enable an optional 20% held-out evaluation set.
• Set GEPA's metric-call budget, which defaults to 300.

Smaller annotation rounds can make each review cycle quicker. A held-out set can provide an evaluation signal separate from the examples used for optimization. The metric-call budget controls how much work GEPA can perform during a run.

Choose a Jev provider

The guided setup detects configured providers and lets you choose among TypeSafe AI, Vercel AI Gateway, and Cloudflare Workers AI. The selected provider is saved with the AI Function, so later runtime calls use the same route.

Use Vercel AI Gateway

export AI_GATEWAY_API_KEY="..."
jeva optimize data.csv --question "Is this relevant?" --column text \
  --backend vercel

Use Cloudflare Workers AI

export CLOUDFLARE_ACCOUNT_ID="..."
export CLOUDFLARE_API_TOKEN="..."
jeva optimize data.csv --question "Is this relevant?" --column text \
  --backend cloudflare

Neither of these routes requires a TYPESAFE_API_KEY.

Configure the GEPA reflection model

GEPA's reflection model is separate from the Jev model that evaluates dataset rows. OpenAI, Anthropic, and Gemini models are detected automatically. You can also use any LiteLLM provider, including Fireworks, local vLLM, and other OpenAI-compatible endpoints.

For example, configure a hosted vLLM endpoint and select its model explicitly:

export HOSTED_VLLM_API_BASE="http://localhost:8000/v1"
jeva optimize data.csv --question "Is this relevant?" --column text \
  --reflection-model "hosted_vllm/Qwen/Qwen3-8B"

This separation lets you choose one provider for runtime dataset evaluation and another for GEPA's optimization work.

Manage and resume saved AI Functions

List your saved functions with:

jeva functions

To resume a specific run directly, provide its saved directory:

jeva optimize --resume .jev-align/runs/<run-id>

Resuming is useful when you want to continue labeling, revisit a saved function, or process examples captured after deployment.

Use an AI Function in Python

Load a saved function with AIFunction.load. Enable capture so the function records useful production examples for future review:

from jev_align import AIFunction

is_aviation = AIFunction.load(
    ".jev-align/runs/<run-id>",
    capture=True,
)

prediction = is_aviation(
    title="Airport expansion",
    text="A new runway opens next year.",
)

The keyword arguments should correspond to the fields expected by the saved function. In this example, the function evaluates a title and text body.

Keep learning from production data

With capture=True, your application can retain examples that are useful for later optimization. Return to the CLI when you are ready to review them:

jeva functions

Select or resume the relevant AI Function, label the captured examples, and let GEPA propose the next definition. The same human review process applies: inspect the metrics and diff before accepting a change.

Advanced workflow tips

• Select focused columns: Include only fields that provide evidence for the task rather than automatically using every field.
• Write a precise question: A clear decision boundary makes binary classification easier to label consistently.
• Describe fixed labels carefully: For multiclass and multilabel tasks, use the class descriptions to distinguish labels that may otherwise overlap.
• Add rationales when useful: Optional rationales can give GEPA more context about why an example should receive a particular label.
• Use the audit sample: Ambiguous examples are valuable, but random examples help check behavior outside the uncertainty-focused subset.
• Review more than the score: Inspect certainty changes and the proposed definition diff before accepting an iteration.
• Resume instead of restarting: Saved runs preserve accumulated labels and make continued improvement possible.
• Separate provider decisions: Choose the Jev backend for evaluation and runtime needs, then configure the reflection model independently for GEPA.

Conclusion

jev-align provides a human-in-the-loop process for creating and refining Jev AI Functions. Start with a local dataset, label uncertain examples, review GEPA's proposed definition changes, and save the result. By enabling production capture and resuming the same run later, you can continue improving the function with real application examples while retaining control over every accepted version.