Classify CLI Outcomes Locally with Verdict

What Verdict Does

Verdict is a research preview of a semantic outcome layer for coding agents, CI runners, and other tool-using systems. Instead of treating a process exit code as definitive evidence, it examines the command, exit code, standard output, and standard error together.

The model returns one of four labels: success, failure, blocked, or unknown. This distinction is useful because a command can exit with code zero while running no tests, returning an HTTP 500 response, waiting for a worker, or producing incomplete output. Likewise, a nonzero exit code may indicate an environmental blocker rather than a failed assertion.

Important: Verdict is an advisory classifier. It does not prove that a task completed or that an environment changed as intended. Keep deterministic checks and authoritative state verification as the source of truth.

Key Features

• Local inference: Inputs are processed on the local CPU without network calls or telemetry.
• Tiny model: The quantized model is approximately 197 KB.
• Zero-dependency runtime: Inference uses the Python standard library and requires no tokenizer files, GPU, or Python runtime dependencies beyond the package itself.
• Semantic classification: The model considers the command, exit code, stdout, and stderr rather than relying on the exit code alone.
• Fast predictions: The repository reports a median inference time of 0.54 ms in its documented Windows 11, AMD64, and Python 3.12 benchmark environment.
• Python and CLI interfaces: Use the model in an application or directly from a shell pipeline.
• Browser demonstration: The project page runs the same quantized weights in the browser without uploading logs.

Verdict uses hashed character 3–5-grams and word 1–3-grams with multiclass logistic regression. Its inputs are UTF-8 text, with each output stream bounded to 12,288 characters through explicit head-and-tail clipping.

Understand the Four Labels

• success: The output contains positive evidence that the operation completed.
• failure: The operation ran and reported a concrete failure.
• blocked: Permissions, dependencies, networking, quota limits, or the environment prevented execution.
• unknown: Evidence is absent, incomplete, skipped, limited to a dry run, or otherwise inconclusive.

The returned probabilities come from a linear multiclass model. They must not be interpreted as calibrated confidence for safety-critical decisions.

Install Verdict

Clone the repository, enter its directory, and install the package with pip:

git clone https://github.com/Balanov/verdict.git
cd verdict
python -m pip install .

This installs the verdict_tiny Python package and the verdict command-line interface.

Run Your First Python Prediction

Create a Verdict instance and pass it the same evidence your tool runner observed. This example represents a successful pytest process that did not actually run any tests:

from verdict_tiny import Verdict

model = Verdict()
result = model.predict(
    command="pytest tests/",
    exit_code=0,
    stdout="collected 0 items\nno tests ran in 0.01s",
)
print(result)

The documented model output classifies this evidence as unknown, despite the zero exit code:

{
  "label": "unknown",
  "confidence": 0.999981,
  "probabilities": {
    "success": 0.000019,
    "failure": 0.0,
    "blocked": 0.0,
    "unknown": 0.999981
  },
  "advisory": true
}

The label is the selected outcome. The probabilities object exposes the score for every class, while advisory emphasizes that the prediction is not authoritative proof.

Use Verdict from the Command Line

You can pipe command output into Verdict while supplying the original command and exit code:

printf 'collected 0 items\nno tests ran\n' | verdict \
  --command 'pytest tests/' --exit-code 0

For output already saved to files, use --stdout and --stderr:

verdict \
  --command 'pytest tests/' \
  --exit-code 0 \
  --stdout output.log \
  --stderr error.log

You can also load weights from a specific location:

verdict \
  --command 'pytest tests/' \
  --exit-code 0 \
  --stdout output.log \
  --stderr error.log \
  --weights /path/to/verdict-q8.bin

Integrate Verdict Safely

A practical integration should treat Verdict as an additional decision signal, not as the final judge. A coding agent or CI runner can use the label to decide whether to inspect more evidence, retry an operation, or request human review.

1. Run the original command and retain its exit code, stdout, and stderr.
2. Pass all available evidence to Verdict.
3. Inspect both the predicted label and the class probabilities.
4. Apply application-specific thresholds before taking consequential actions.
5. Verify completion with deterministic checks, such as test counts, generated files, API state, or database records.

For example, an unknown result should generally trigger more investigation rather than a completion claim. A blocked result may justify checking credentials, dependencies, connectivity, permissions, or quotas before retrying. A success result still requires authoritative verification when correctness matters.

Advanced Tips

Preserve Complete Execution Context

Provide the actual command, exit code, stdout, and stderr whenever possible. Omitting streams can remove evidence that distinguishes a concrete failure from an environmental blocker or an inconclusive run.

Account for Input Clipping

Each output stream is bounded to 12,288 characters using head-and-tail clipping. When logs are very large, ensure that decisive messages appear near the beginning or end, or perform an additional deterministic scan of the complete logs.

Set Conservative Decision Thresholds

The probabilities are not calibrated confidence in a safety-critical sense. Use thresholds appropriate to your workflow and route uncertain or consequential cases to deterministic checks or human review.

Use Custom or Downloaded Weights

The released quantized weights are stored in weights/verdict-q8.bin and are also available from Hugging Face. The CLI accepts an alternate weight path through --weights.

Try the Browser Demo

The Verdict project page runs the same quantized weights in a browser. According to the project documentation, logs are not uploaded.

Reproduce Training and Evaluation

Install the training and development extras, regenerate the synthetic corpus, train the model, export the int8 weights, evaluate the artifact, and run the tests:

python -m pip install -e '.[train,dev]'
python training/train.py
pytest -q

The public benchmark uses 192 synthetic examples from 48 CLI families absent from training. Training uses 1,440 synthetic examples from 60 different families. The reported quantized model achieved 93.2% accuracy and 93.2% macro-F1, compared with 47.9% accuracy and 31.9% macro-F1 for an exit-code baseline.

The corpus is synthetic and English-biased. These measurements evaluate held-out templates, not reliability on production agent traces, and latency results are not promises across hardware.

Know the Project's Boundaries

Verdict is deliberately narrow. It classifies evidence observed in CLI output, but it does not determine whether a broader task is complete. It also does not establish code correctness, safety, or truthfulness.

The project is therefore best used as a semantic layer between raw process execution and an agent's next action. It can identify evidence that an exit code misses, but it should complement rather than replace tests, state inspection, validation rules, and human judgment.

Conclusion

Verdict provides a small, fast, local way to interpret CLI outcomes more carefully than an exit-code-only check. Install it as a Python package, provide the full execution context, and use its four labels to guide follow-up actions. For reliable automation, always combine its advisory result with deterministic and authoritative verification.