Build Reliable Typed LLM Decisions with AnyJev

What AnyJev Does

AnyJev turns an open language model into a Jev-style decision model. Instead of asking the model to generate an answer and then parsing its text, AnyJev reads the model’s next-token distribution from a prefill and returns a typed decision with probabilities.

This is useful for workflows such as routing a request, estimating whether an action is risky, or scoring task completion. Each result records the decision level used, allowing downstream code to distinguish an uncalibrated or zero-label result from one produced by a fitted head.

AnyJev addresses two practical problems with raw logits: predictions may change when option order changes, and raw confidence values may not be calibrated well enough for threshold-based automation. In the README’s Qwen3-8B BANKING77 experiment, L0 reduced the option-order flip rate from 0.230 to 0.073 without labels. L1 reduced expected calibration error from 0.240 to 0.095 with labeled examples.

Decision Levels

AnyJev provides several levels with different data and serving requirements:

• raw: Applies a restricted softmax over label tokens. It does not correct position bias or calibrate confidence.
• L0: Requires no labels. It evaluates cyclic option rotations, averages out position bias, and divides out an estimated label prior.
• L1: Uses approximately 100–500 labels per question to fit temperature scaling over L0. It improves calibration but does not change the ranking.
• L2: Uses approximately 100–300 labels per question to solve a shrunk LDA or ridge head from an intermediate hidden state. It needs a local model and is specific to both the model and question.

L0 requires multiple prefills for a multi-option choice because it evaluates rotations. L2 instead uses one prompt and stops at a fixed intermediate block, making it cheaper than a full forward pass in the reported Qwen3 experiments.

Key Features

• Typed choice, noul, and score questions.
• No text generation or answer parsing.
• Zero-label L0 correction for option-position and label-prior effects.
• L1 temperature calibration with a few hundred labels.
• L2 closed-form heads fitted in seconds without gradients or model fine-tuning.
• Automatic selection of L2, L1, or L0 through level="auto".
• Incremental label collection with observe.
• Batch decisions for many states and one question.
• Small JSON artifacts that can be saved and loaded for later serving.

Installation and Setup

Install the Hugging Face backend

Install AnyJev with its Hugging Face integration:

pip install "anyjev[hf]"

The current release serves all decision levels through the Transformers-based anyjev.backends.hf backend. vLLM and SGLang support is on the roadmap and is not available in this release.

Create a decider

Import the core API and initialize an HFBackend with an open model:

from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend

decider = Decider(HFBackend("Qwen/Qwen3-8B"))

The shipped L2 heads cover five Qwen3 models: 1.7B, 4B, 8B, 30B-A3B, and 32B. L2 heads remain specific to their base model and question, so a head fitted for one model or question cannot automatically be reused for another.

Define Typed Questions

A question describes the output type, valid responses, and a stable name. You can evaluate several question types against the same application state.

route = Question.choice(
    "Which team should handle this?",
    ["billing", "technical", "sales", "other"],
    name="route",
)

risky = Question.noul(
    "Is this tool call destructive or irreversible?",
    name="risky",
)

done = Question.score(
    "How complete is the task?",
    bins=5,
    name="done",
)

Use choice for a categorical decision, noul for a true-or-false decision, and score for a binned numerical score. The letter-token readout currently supports at most 26 options.

Make a Basic L0 Decision

L0 is the default and requires no labeled examples. Construct a state containing the information needed by the questions, then call decide:

state = {
    "conversation": [
        {"role": "user", "content": "I was charged twice."}
    ],
    "tool_call": {
        "name": "refund_payment",
        "arguments": {"payment_id": "pay_123"},
    },
}

result = decider.decide(state, [route, risky, done])

print(result["route"].distribution)
print(result["risky"].p_true)
print(result["done"].value)
print(result.level)

A representative route distribution might map each supplied option to a probability. Boolean decisions expose p_true, while score decisions expose value. The overall result reports L0 when no higher-level artifact is available.

Probabilities should be interpreted according to their level. L0 improves stability and corrects estimated label bias, but it does not make uncertainty fully calibrated.

Add L1 Calibration

If you have roughly 100–500 labeled states for a question, call calibrate to fit an L1 temperature:

decider.calibrate(risky, states, labels)

L1 calibrates the probabilities produced on top of L0. It does not change which answer ranks first, but it can make confidence thresholds more meaningful.

Fit an L2 Closed-Form Head

For higher accuracy, fit a question-specific L2 head using approximately 100–300 labeled examples:

decider.fit_head(route, states, labels)

decider.save_artifacts("qwen3-8b.json")

Fitting performs one model pass over the examples and then solves the head in closed form. It does not use gradients and does not change the language model’s weights. According to the project, a head is typically around 100 KB and can be solved in seconds for the smaller supported Qwen3 models.

Load the saved artifacts in a later process, then request automatic level selection:

decider.load_artifacts("qwen3-8b.json")

result = decider.decide(state, [route], level="auto")
print(result["route"].level)

With level="auto", AnyJev uses L2 when a compatible head can route the question. Otherwise it falls back to L1 when calibration is available, then to L0.

Collect Labels Incrementally

AnyJev can accept labels as they arrive from a review queue, observed outcomes, or another feedback source:

decider.observe(route, state, correct_label)

The project’s automatic loop solves a head at 30 observations and then resolves it at 60, 120, and subsequent milestones. This supports a deployment lifecycle in which a new question begins at L0 and moves to L2 after enough feedback has accumulated.

Batch Inference

When applying one question to many states, use the batch API rather than calling decide repeatedly:

results = decider.decide_batch(states, route)

This is the intended interface for processing many inputs against a single typed question.

Try the Packaged Demo

From a checkout of the repository, run the synthetic demo without downloading a model:

python -m demo.jev_mode --backend fake

To simulate the deployment lifecycle, add the lifecycle option:

python -m demo.jev_mode --backend fake --lifecycle

Remove --backend fake to run the real Qwen3 demonstration with the shipped heads.

Advanced Deployment Tips

Keep question identity stable

L2 is fitted per question and model. A new option set requires new labels and a new head. Rewording the same question or changing the order of the same options can route to an existing head.

Use unlabeled traffic for wording adaptation

For reworded questions, AnyJev can recenter the head’s feature mean and scale using approximately 30 unlabeled requests. This adaptation applies to question wording and option order; it does not detect a shift in the application states themselves.

Monitor real data drift

Because state distribution shifts are invisible to the recentering mechanism, retain a periodically labeled evaluation slice and spot-check performance. Do not treat wording adaptation as a substitute for production monitoring.

Gate actions by decision level

Every decision carries its level. Downstream systems can inspect it before executing a sensitive action, and the project also supports enforcing levels with require=. This prevents a workflow designed for calibrated or L2 decisions from silently acting on a fallback result.

Choose thresholds only after validation

The main benefit of calibration is the ability to send high-confidence cases to automation while escalating uncertain cases. Select thresholds on representative labeled data and measure the resulting error and coverage rather than assuming a displayed probability is automatically trustworthy.

Important Limitations

• L2 heads do not transfer to a different question or base model.
• Only Qwen3 heads ship with the current project release.
• L2 requires access to hidden states, which is currently provided through the Transformers backend.
• Calibration cannot make a model solve a task it fundamentally cannot answer.
• L0 can reduce accuracy when one label strongly dominates the batch prior.
• The current letter readout supports no more than 26 options.
• The reported 5% risk coverage estimate is based on 300 examples and therefore has high variance.
• The published typed-decisions accuracy measures agreement with a teacher LLM, not an independently established ground truth.
• The reported decisions were evaluated in isolation rather than inside a complete agent loop.

Conclusion

AnyJev provides a practical progression from zero-label typed decisions to calibrated probabilities and efficient hidden-state heads. Start with L0, collect real labels, add L1 when calibration is the priority, and fit L2 when you need a more accurate question-specific decision path. Preserve the reported level, validate thresholds, and monitor labeled samples as production data changes.

For implementation details and current plans, see the AnyJev repository, its levels contract, benchmark documentation, and roadmap.