Deploy and Use OpenJev for One-Token Classification with SGLang

What is openjev-sglang?

openjev-sglang is a server that implements the TypeSafe/Jev HTTP API using Qwen3.6-35B-A3B on SGLang. It converts classification questions into efficient first-token probability readouts rather than generating long responses.

The default deployment runs one SGLang container per NVIDIA B200 GPU. SGLang 0.5.19 provides its Rust frontend, radix caching, and breakable prefill CUDA graphs. A separate Python process handles the public API with FastAPI, uvloop, a Rust-backed Hugging Face tokenizer, and pooled asynchronous connections to SGLang on localhost.

No requests are sent to TypeSafe. The public endpoint exposes the evaluation API, while SGLang generation and administration routes remain private on localhost.

Key features

• Three evaluation types: Noul for yes/no probability, Choice for categorical selection, and Score for ordered levels.
• One-token inference: each question is evaluated from answer-label log probabilities without a generated chain of thought.
• Shared-prefix caching: a common prompt prefix is warmed before question branches run concurrently.
• Up to 64 questions: Choice and Score questions support between 2 and 64 answers by default.
• Structured state: state and instructions can be strings, JSON objects, or arrays.
• Operational endpoints: health, liveness, model catalogue, limits, OpenAPI schema, Swagger UI, and an interactive Scalar reference are included.
• Modal deployment: the supplied application supports autoscaling, persistent model caches, startup recovery, and scale-to-zero behavior.
• Configurable protection: request limits, concurrency controls, timeouts, and optional Bearer authentication are available.

Prerequisites

Install uv and obtain a Modal account if you plan to use the provided cloud deployment. The first build imports a large SGLang image, and the first GPU start downloads model weights and compiles or captures kernels.

CUDA dependencies remain in the SGLang container. Running uv sync on your workstation installs only the API, deployment utilities, and tests.

Install the project

Clone the repository, enter its directory, and synchronize the Python environment:

uv sync

If Modal has not been authenticated on the current machine, complete its setup:

uv run modal setup

Run a temporary Modal deployment

For an end-to-end test, start a temporary server, run the included inference checks, and shut the server down afterward:

uv run modal run modal_app.py

This command saves the smoke-test report as smoke-result.json. The report includes startup wait time, inference latency, and available cache-usage information.

Deploy a stable endpoint

Deploy the application as a persistent Modal service:

uv run modal deploy modal_app.py

The deployment prints a URL similar to https://YOUR-SERVER.us-west.modal.direct. Store it in an environment variable for convenient access:

export OPENJEV_URL="https://YOUR-SERVER.us-west.modal.direct"

The default Modal endpoint is intentionally unauthenticated. It uses the US West routing region and can schedule compute in configured US regions. There is no explicit container cap, and instances scale to zero after five idle minutes.

A server that has scaled to zero returns HTTP 503 while it starts. The included smoke command retries these startup responses automatically.

Run the smoke test against the deployed URL:

uv run openjev smoke "$OPENJEV_URL"

The smoke test exercises all three answer types, checks a 64-answer question, performs basic semantic checks, and verifies that 65 answers are rejected.

Submit your first evaluation

The primary route is POST /v1/systemone. The following request classifies a customer message with Noul, Choice, and Score questions:

curl "$OPENJEV_URL/v1/systemone" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "jev-latest",
    "state": [
      {"role": "system", "content": "You are a support assistant."},
      {"role": "user", "content": "I was charged twice. Please refund the duplicate."}
    ],
    "questions": {
      "refund": {
        "type": "noul",
        "instructions": "Does the user request a refund?"
      },
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this?",
        "criteria": {
          "billing": "Payments and refunds",
          "technical": "Software bugs"
        }
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is the request?",
        "criteria": ["Routine", "Urgent", "Emergency"]
      }
    }
  }'

You can also keep the JSON in a file and submit it without shell quoting:

curl "$OPENJEV_URL/v1/systemone" \
  -H 'Content-Type: application/json' \
  --data-binary @examples/request.json

Understanding the question types

• Noul: returns the normalized probability of yes.
• Choice: returns the highest-probability option and the full probability distribution. Original option names are preserved in the response.
• Score: returns the zero-based expected level, calculated from each level index and its probability, along with a legend.

Choice options are presented to the model as letter labels followed by descriptions. Response keys are normally hidden from the model. If an option description is null, its key supplies the meaning instead.

Supplying state and instructions

Both state and instructions accept strings, JSON objects, or arrays. A list of chat messages, or an object containing exactly {"messages": [...]}, is rendered with the model's native chat template.

The original message roles and objects are retained, and each classification question is added as another user turn. Other structured values are serialized intact into a user message. An object containing messages plus other fields is also preserved so that metadata is not silently dropped.

Chat state supports text content only. Image, audio, and video content are not supported.

Explore the service endpoints

• POST /v1/systemone performs Noul, Choice, and Score evaluation.
• GET /v1/models returns a model catalogue with TypeSafe and OpenAI-style fields.
• GET /v1/limits reports admission limits.
• GET /health reports readiness, SGLang health, and startup duration.
• GET /health/live checks whether the API process is alive.
• GET / opens a Scalar API reference with an editable request example.
• GET /docs opens the built-in Swagger UI.
• GET /openapi.json returns the generated OpenAPI schema.

The request model name jev-latest is a compatibility alias. The default public model ID is Qwen/Qwen3.6-35B-A3B, while NVIDIA's repository is used internally as the weight source.

How inference works

1. The API validates the schema, answer count, body size, context length, and aggregate token budget.
2. It renders the native chat template once with thinking disabled, then separates a shared prefix from independently tokenized question suffixes.
3. It sends the common prefix to SGLang with max_new_tokens=1. The sampled token is discarded, leaving the prefix available for opportunistic radix-cache reuse.
4. It concurrently submits one branch per question. Each branch contains the prefix, question suffix, assistant header, and Answer: marker.
5. It requests log probabilities only for valid answer labels and renormalizes them with a stable softmax.

For N questions, the server makes N + 1 one-token calls: one cache-warming call and one call for each question. The generated token is ignored, so there is no autoregressive continuation or generated chain of thought. Speculative decoding is not enabled.

Labels run from A through Z and then use verified single-token combinations such as AA and AB. All 64 labels are checked with the active tokenizer during startup. This avoids using multi-token numeric labels such as 10 or 64.

Understand usage and cache metrics

The x-openjev-prefix-tokens response header reports the requested common-prefix size. When SGLang supplies cache counts, x-openjev-cached-tokens contains the sum of branch cache hits.

SGLang 0.5.19's Rust frontend does not expose those cache counts. In that configuration the header is absent and the smoke report records null, not zero. Scheduler logs can still show actual cache hits. The Server-Timing header separates prompt preparation, shared prefill, and branch inference.

usage.input_tokens is the sum of the complete prompt counts reported across warm-up and branch calls, including cached tokens. usage.output_tokens is N + 1. These figures describe backend activity rather than unique computed tokens or TypeSafe billing estimates.

Configure limits and authentication

Default admission limits include 64 questions, 2 to 64 answers per Choice or Score question, a 2 MiB JSON body, 32,768 tokens per branch including output, 262,144 total submitted input tokens, 16 concurrent evaluations, and 64 concurrent backend calls.

Invalid requests return HTTP 422 before inference. Bodies above the size limit return 413, overload returns 529 with Retry-After, and backend timeouts return 504. Failed or cancelled evaluations cancel sibling branches and attempt to abort them in SGLang.

Configuration is available through OPENJEV_* environment variables. Common settings include:

• OPENJEV_MODEL: defaults to nvidia/Qwen3.6-35B-A3B-NVFP4.
• OPENJEV_SERVED_MODEL_NAME: overrides the public model ID and is passed to SGLang.
• OPENJEV_REVISION: selects the pinned checkpoint revision.
• OPENJEV_FRONTEND: defaults to rust; python is an explicit fallback.
• OPENJEV_MAX_INPUT_TOKENS: defaults to 32768.
• OPENJEV_MAX_TOTAL_INPUT_TOKENS: defaults to 262144.
• OPENJEV_MAX_CONCURRENT_REQUESTS: defaults to 16.
• OPENJEV_MAX_CONCURRENT_BRANCHES: defaults to 64.
• OPENJEV_REQUEST_TIMEOUT: defaults to 120 seconds.
• OPENJEV_TEMPERATURE: defaults to 1.0 and affects label normalization.
• OPENJEV_API_KEY: enables optional Bearer authentication for the public API.
• OPENJEV_BACKEND_API_KEY: configures a separate optional Bearer key for SGLang.

The Modal launcher automatically forwards OPENJEV_PROFILE, OPENJEV_FRONTEND, and OPENJEV_SERVED_MODEL_NAME from the local environment. Add other remote settings to image.env(...), or use a Modal Secret for credentials.

Run locally with an existing SGLang backend

To use an already-running backend, point the API at its local URL:

uv run openjev serve --connect http://127.0.0.1:30000

The backend must use a matching model and tokenizer revision. It must also support selected-token log probabilities, radix caching, and a sufficient context length.

On a B200 host or container where SGLang 0.5.19 is installed in a separate Python environment, provide its interpreter explicitly:

uv run openjev serve --sglang-python /path/to/sglang/bin/python

Development and validation

Use the included commands to run offline tests, tokenizer integration checks, linting, and schema generation:

uv run pytest
uv run pytest -m integration
uv run ruff check .
uv run openjev schema

The regular tests cover unit and API behavior offline. Integration tests use a real tokenizer and a small Hugging Face download but do not require a GPU. Schema generation also requires neither a GPU nor a model download.

Advanced deployment tips

Keep a GPU container warm

Modal scales the service to zero after five idle minutes. Set min_containers=1 in modal_app.py when consistently low startup latency is more important than avoiding idle GPU usage.

Expect first-start initialization

The first GPU start downloads weights and compiles or captures kernels. Model weights, SGLang tuning data, and Triton compilation data persist in the openjev-huggingface Modal Volume. Later starts reuse those files, although CUDA graph capture still occurs at startup.

Treat cache reuse as opportunistic

Radix reuse is not a pinned per-request KV session. Recurrent state in the hybrid Qwen model, cache-page boundaries, cache pressure, and concurrent traffic can reduce hits. The backend uses --mamba-radix-cache-strategy extra_buffer.

Interpret confidence carefully

OpenJev defines confidence as 1 - H(probabilities) / log(number_of_options), clamped between zero and one. Uniform probabilities produce zero confidence, while a point mass produces one.

These probabilities are conditioned on the supplied options and can depend on prompt wording and label order. They are not calibrated estimates of correctness.

Plan for backend failure

If SGLang exits unexpectedly, the API exits as well. The Modal launcher then terminates the container so Modal can replace it instead of leaving a public HTTP process connected to a dead inference backend. Normal shutdown disables both watchers cleanly.

Conclusion

openjev-sglang provides a focused way to run Jev-compatible classification on Qwen3.6 with efficient one-token probability readout. Its Modal configuration offers scalable B200 deployment, while its local commands support testing and connection to an existing SGLang service. Start with the smoke test, validate your limits and authentication settings, and monitor timing and cache headers as traffic grows.