Build Reasoning-Based Decision APIs with Jeeves

What is Jeeves?

Jeeves is a Jev-style classifier that generates a reasoning chain before making a structured decision. It is built from Qwen3.5-9B with LoRA adapters, a pointer head, calibrated probabilities, and a block-4 diffusion drafter. The repository includes the model and drafter code, training pipeline, evaluation tools, and train, development, and test data.

The project is designed for applications that need more than free-form text generation. Given a shared state and one or more questions, Jeeves returns structured answers and probabilities through a Jev-compatible API. Multiple questions can be processed in the same request.

Key features

• Three decision types: binary noul, categorical choice, and ordinal score questions.
• Reasoning before decisions: the model can generate a reasoning chain and then use its pointer head to score the available options.
• Calibrated output: final option probabilities use a temperature fitted on the development set.
• Jev-compatible API: existing Jev clients can submit the core request format, while Jeeves-specific options control reasoning.
• Python SDK: the included sdk/ package is a drop-in replacement for Jev’s typesafe-sdk.
• Diffusion drafting: a block-4 drafter accelerates reasoning-chain generation and supports Qwen3.5’s Gated DeltaNet layers.
• Training code: the repository contains SFT, CISPO, calibration, evaluation, export, and drafter-training components.

With greedy reasoning capped at 2,560 tokens, the README reports 0.889 accuracy on its held-out and out-of-domain test set and 0.935 across 231 public JevBench items. However, its knowledge results trail Jev on MMLU and MMLU-Pro, so benchmark suitability depends on the task.

Prerequisites

You need Python 3.12 and a CUDA GPU. The inference implementation’s FP8 kernel requires a Hopper GPU. Run the following commands from the repository root.

Install the dependencies

pip install -r requirements.txt

Download the released weights

Use the Hugging Face CLI to download the published 9B checkpoint and its block-4 drafter:

hf download PostHog/jeeves --local-dir jeeves-weights

The released model is available from Hugging Face.

Start the API server

python -m inference.serve --model jeeves-weights --drafter jeeves-weights/drafter_k4.safetensors --port 8009

This starts the Jev-compatible service on port 8009. The bundled Python client uses http://127.0.0.1:8009 by default, requires no API key, and waits up to 120 seconds for a response.

Send your first decision request

The /v1/systemone endpoint accepts a state, a map of questions, and optional inference controls. The following request asks Jeeves to route a customer issue, decide whether it requires escalation, and rate the customer’s frustration.

curl -s localhost:8009/v1/systemone \
  -H 'content-type: application/json' \
  -d '{
    "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "returns": "Exchanges, refunds, wrong or damaged items",
          "shipping": "Delivery status, delays, lost packages",
          "billing": "Charges, invoices, payment problems"
        }
      },
      "escalate": {
        "type": "noul",
        "instructions": "Does this need urgent human attention?"
      },
      "frustration": {
        "type": "score",
        "instructions": "How frustrated is the customer?",
        "criteria": ["Calm", "Frustrated", "Very angry"]
      }
    },
    "options": {"max_think": 512}
  }'

Understand the question types

• choice: selects from named alternatives and returns the selected key, probabilities, and confidence.
• noul: represents a yes/no-style decision and returns a numeric noul value.
• score: evaluates an ordered list of labels and returns a numeric score, a legend, probabilities, and confidence.

The response also contains usage information, including input, output, and reasoning token counts, plus total latency. Questions in the same request can reason in parallel.

Use Jeeves from Python

Install the included SDK

pip install ./sdk

Create typed questions

from jeeves_sdk import Choice, Noul, Score, TypeSafeClient

with TypeSafeClient() as client:
    result = client.system_one(
        state="I was charged twice. Please help.",
        questions={
            "billing": Noul(
                instructions="Is this about billing?"
            ),
            "tone": Choice(
                instructions="What is the tone?",
                criteria={"calm": None, "angry": None},
            ),
            "urgency": Score(
                instructions="How urgent is this?",
                criteria=["can wait", "this week", "today"],
            ),
        },
        max_think=768,
        return_reasoning=True,
    )

    print(result.nouls["billing"].noul)
    print(result.choices["tone"].choice)
    print(result.scores["urgency"].score)
    print(result.reasoning["tone"].text)

TypeSafeClient groups returned values by question type. Setting return_reasoning=True also makes each question’s reasoning text available. The README cautions that no language-consistency reward was used during training, so reasoning chains are not necessarily easy to interpret.

Connect to a different server

The client targets http://127.0.0.1:8009 by default. Set the JEEVES_BASE_URL environment variable when the server runs at another address.

export JEEVES_BASE_URL=http://your-server:8009

Control reasoning and latency

The optional options object exposes four reasoning controls. Matching server flags can establish server-wide defaults.

• think: defaults to true. Set it to false to answer directly from the prompt without producing a reasoning chain.
• max_think: defaults to 2560. It truncates each reasoning chain at the specified token count before the model answers.
• nothink_threshold: defaults to null. When configured, Jeeves skips reasoning if its no-think confidence meets or exceeds the threshold.
• return_reasoning: defaults to false. Enable it to include reasoning text in the response.

Choose a latency profile

On the README’s 325-question development evaluation, full reasoning achieved 0.825 accuracy with 1,138 mean reasoning tokens and median latency of 3.3 seconds. Using max_think of 768 with a nothink_threshold of 0.9 produced 0.806 accuracy, 344 mean reasoning tokens, and median latency of 2.0 seconds. Disabling reasoning produced 0.775 accuracy at about 0.3 seconds.

A practical balanced configuration is:

{
  "options": {
    "max_think": 768,
    "nothink_threshold": 0.9,
    "return_reasoning": false
  }
}

Full reasoning has a long latency tail: the reported p90 is 17.1 seconds. Use shorter chains or confidence-based reasoning when response time is important.

How Jeeves makes decisions

Jeeves formats the state, question instructions, and answer options with rare tokens from the Qwen tokenizer. It then generates a reasoning chain beginning with a think marker. After reasoning, the question and options are repeated before a decision marker.

A pointer head computes a scaled dot product between a query projection at the decision marker and key projections at each option boundary. A softmax converts those scores into probabilities, and a temperature fitted on the development set calibrates them. According to the README’s ablations, replacing the special prompt tokens with ordinary labels or omitting the repeated post-reasoning question reduced performance.

Diffusion drafter

The optional drafter accelerates reasoning generation through speculative decoding. It adapts the Orthrus diffusion approach to Qwen3.5’s Gated DeltaNet layers by allowing mask tokens to cross-attend to post-convolution keys and values.

The reported throughput for one question rises from 109 chain tokens per second with plain graphed greedy decoding to 176 tokens per second with block 4. Block 8 reaches 193 tokens per second, but block 4 remains the default because it stays inexpensive when several questions are batched. Eight questions batched with block 4 reach about 960 chain tokens per second in total.

Serve a checkpoint you trained

Jeeves uses LoRA during training. Before serving a trained checkpoint, fuse it into a standalone model and provide the matching drafter file:

python export.py runs/cispo/final --out runs/fused
python -m inference.serve \
  --model runs/fused \
  --drafter runs/drafter_k4/drafter.safetensors \
  --port 8009

Reproduce the training pipeline

Prepare the datasets

The preparation scripts download public datasets from Hugging Face at revisions pinned in prep/public.py. Each source dataset remains under its own license.

python -m prep.prep

Run training and evaluation

The documented reproduction pipeline uses eight GPUs. It performs supervised fine-tuning, CISPO training, test and JevBench evaluation, model export, chain generation, and block-4 drafter training.

torchrun --nproc_per_node 8 train.py sft --run-dir runs/sft

torchrun --nproc_per_node 8 train.py cispo \
  --run-dir runs/cispo \
  --init runs/sft/final

torchrun --nproc_per_node 8 test.py runs/cispo/final
torchrun --nproc_per_node 8 jevbench.py runs/cispo/final
python export.py runs/cispo/final --out runs/fused

torchrun --nproc_per_node 8 -m drafter.gen --model runs/fused

torchrun --nproc_per_node 8 train.py drafter \
  --model runs/fused \
  --block 4 \
  --run-dir runs/drafter_k4

The published recipe first runs two SFT epochs over 19,126 questions from 12 public datasets and synthetic policy data. It then applies CISPO with eight rollouts per question, temperature 1, and a 2,560-token reasoning cap. Training was stopped at step 402 of a planned 624-step schedule because later steps over-sharpened the pointer head on the saturated reinforcement-learning pool. Finally, one calibration temperature was fitted on the development set.

Advanced usage tips

1. Batch related decisions. Put several questions under one shared state so the server can process their reasoning chains in parallel.
2. Use direct answers for easy cases. Configure nothink_threshold to reserve reasoning for decisions whose no-think confidence is below your chosen threshold.
3. Cap reasoning in latency-sensitive services. Lower max_think rather than disabling reasoning entirely when you want a middle ground between speed and accuracy.
4. Return reasoning selectively. Keep return_reasoning disabled unless the application needs the text, especially because the chains are not guaranteed to be readily interpretable.
5. Evaluate on your own domain. Jeeves performs strongly on the project’s test split and public JevBench tiers, but trails Jev on the reported MMLU and MMLU-Pro knowledge evaluations.
6. Interpret comparisons carefully. Outside JevBench, the published Kev and Jev comparisons use different items from the same underlying sources.

Conclusion

Jeeves combines structured Jev-style outputs with test-time reasoning, calibrated probabilities, and accelerated chain generation. Start with the released weights and Jev-compatible HTTP endpoint, use the Python SDK for typed application code, and tune max_think and nothink_threshold to balance accuracy against latency. For deeper customization, the repository provides the complete SFT, CISPO, calibration, export, and drafter pipeline.