Jev GUI Delegate Tutorial: Let Codex Delegate Windows and Chrome GUI Operations

Project Overview

Jev GUI Delegate is a prototype for delegating Windows and Chrome GUI tasks from Codex. The main model provides the task objective and acceptance criteria, while the local controller observes the interface, matches controls, performs actions, and checks the actual result after each step. The controller calls TypeSafe Jev only when it encounters a semantic choice that cannot be handled through deterministic rules.

The core problem it addresses is that when the main model operates a web page or desktop application step by step, it must repeatedly read the interface, select controls, and process tool responses, creating a costly interaction loop. Jev GUI Delegate gives the local controller a clearly verifiable sequence of operations to execute continuously while retaining state checks, error recovery, and user takeover capabilities.

This project is a prototype personally designed, implemented with AI assistance, and continuously validated. The task contract, tool orchestration, recovery mechanisms, permission checks, tests, and feedback iterations are project contributions; the Jev model, TypeSafe SDK, Codex, and browser plugin are upstream products.

Workflow

1. The Codex main model submits the objective and acceptance criteria.

2. The task is organized into a locally verifiable task contract.

3. The controller observes the current interface and attempts deterministic control matching.

4. When it cannot determine the answer directly, the controller calls Jev for semantic selection.

5. Chrome operations run through the official browser session provided by the host, while Windows operations use UI Automation.

6. The controller verifies the actual result, saves progress, and decides whether to continue, recover, pause, or request additional information from the main model.

This workflow does not copy personal browser settings or extract cookies. The Chrome path depends on a compatible official browser session API provided by the Codex host; the repository itself cannot create or replace that session.

Key Features

Two GUI Execution Paths

• Chrome: Reuses the official browser session provided by the host.

• Windows: Uses Windows UI Automation to operate accessible desktop controls.

Explicit Task Contracts

The task contract explicitly describes the objective, permitted actions, operating scope, required inputs, execution steps, and success criteria. This allows the controller to check permissions and scope before execution and determine whether the task was truly completed based on the success criteria afterward.

Call Jev on Demand

Controls that can be determined through local rules are matched directly in code; Jev is called only when understanding textual meaning or candidate semantics is required. The project records zero model calls when appropriate and does not incorrectly count them as successful Jev inferences.

Continuous Execution and Recovery

The controller can save execution progress and handle missing inputs, local selection errors, pauses, cancellations, and user takeover. If it cannot confirm whether a submission action succeeded, it does not automatically replay the submission, reducing the risk of duplicate submissions.

Result and Safety Checks

The system checks identity, authorization, and scope before an operation, then checks the actual value or page state afterward. Model usage and failure reasons are recorded separately, making it possible to distinguish cost issues from execution issues.

Environment and Prerequisites

The project targets the following environment:

• Windows x64

• Python 3.12

• Node.js

• An interactive desktop for native GUI operations

• For the Chrome path, a compatible official browser session API provided by the Codex host

Installing the project at C:\jev\jev-bridge is recommended. Some historical commands and Codex skill examples use this location by default. If you choose another directory, update the relevant examples and configuration accordingly.

Install the Project

Step 1: Clone the Repository and Create a Virtual Environment

git clone https://github.com/YUTA-fywoo/jev-gui-delegate.git C:\jev\jev-bridge
cd C:\jev\jev-bridge
py -3.12 -m venv .venv

Step 2: Install Locked Dependencies

.\.venv\Scripts\python.exe -m pip install --require-hashes -r requirements.lock

--require-hashes installs dependencies according to the hashes in the lock file, helping ensure that the installed contents match the project record. Third-party dependency directories are not published with the repository and should be installed from their original distribution channels.

Step 3: Prepare the Runtime

.\.venv\Scripts\python.exe scripts/setup-runtime.py

Step 4: Run Offline Tests

.\.venv\Scripts\python.exe -m unittest discover -s gui_delegate/tests -p "test_*.py"

These tests cover the controller, recovery flow, and boundary conditions. Offline regression uses synthetic data and controlled substitutes; it is not equivalent to live GUI operation and does not pretend to make real-time Jev requests. For more complete results, refer to the repository's release validation document.

Configure the Jev Key

The key can be provided through the local TYPESAFE_API_KEY environment variable or Windows Credential Manager. The project recommends using the local input window opened by set-key.cmd to configure it.

set-key.cmd

Do not write the key to the Git repository, task contracts, or chat content. The chrome-runtime.json generated on the installer's machine is ignored, and the repository does not include environment paths from the author's machine.

Register with Codex

To connect the project to Codex, run the following installation scripts from the project directory:

install.cmd
gui-install.cmd

The installer modifies user-level MCP, skill, and routing configuration, while preserving backups and checking for conflicts. If a directory with the same name already exists on the machine, first check its purpose; the installer will not directly overwrite a project currently in use on the local machine.

The personal Codex skill available for installation is located at gui_delegate/skill/SKILL.md. If the project is not installed in the default directory, also check the path-related examples in that file.

Basic Usage and Validation

Check Local Readiness

After installation, run:

health.cmd

health.cmd checks only whether the components are ready. It does not indicate that live Jev validation has been performed or that the current machine has passed acceptance for a real GUI scenario.

Run Live Validation

The project's live validation workflows, such as verify.cmd, call the Jev service:

verify.cmd

Before running it, confirm that the key is configured correctly, and distinguish offline regression results from live service results. The README does not treat “validated on this machine” in a historical development environment as a conclusion that every installer can inherit directly.

Design a Task Suitable for Delegation

This prototype is suitable for GUI operations with clear boundaries, constrained steps, and verifiable results. When delivering a task to the controller, specify at least the following:

• Objective: The final state to achieve.

• Scope: The pages, windows, or objects in which operations are allowed.

• Permitted actions: The types of interactions the controller may perform.

• Inputs: The data required and permitted for task execution.

• Steps: The operation stages that can be executed continuously.

• Success criteria: The actual values or page states that should be observed after completion.

For example, a task might require the controller to find a target control within a specified page scope, enter authorized input, and confirm the result through the page state. The specific contract fields and result model should follow the implementation in gui_delegate/schema.py; do not assume formats that are not defined in the README.

Source Reading Path

To understand the execution mechanism or carry out further development, read the following in order:

1. gui_delegate/schema.py: Understand the task contract, state, and result models.

2. gui_delegate/controller.py: See how the controller organizes the execution loop.

3. gui_delegate/drivers.py: Learn about drivers and structured interface observation.

4. gui_delegate/security.py: Review scope, authorization, and input checks.

5. gui_delegate/chrome_gateway.mjs: Understand the official browser session entry point.

6. jev_client.py: Review the Jev SDK wrapper, version checks, and usage recording.

7. gui_delegate/tests: Understand recovery logic and boundary behavior through the tests.

Advanced Usage Recommendations

Prefer Deterministic Matching

If a control can be identified through stable attributes or structured information, it should be matched by local code first. Call Jev only when semantic judgment is required. This follows the project design and makes model usage easier to record accurately.

Define an Observable Result for Each Step

Do not define only actions such as “click the button.” Also define the expected value, control state, or page change after the click. The project emphasizes checking the actual result rather than treating a successful tool call as task success.

Handle Submission Actions Carefully

If the result after submission is unclear, do not assume failure and automatically repeat the operation. The project explicitly avoids replaying submissions automatically when the result is unknown. In such a state, it is more appropriate to pause, save progress, and ask the user or main model to confirm.

Handle Missing Inputs and Selection Errors Separately

When a required input is missing, provide the input and continue; only repair the specific selection when the problem is a local semantic selection error. This avoids rerunning the task from the beginning and preserves progress that has already been verified.

Revalidate After Upgrading Jev

The current settings.json pins the jev-1.13.0 version used in historical tests. When the server version changes, recheck the setup and update the configuration; do not treat test results from the old version as quality assurance for the new version. The example strategy also does not mean that the current machine has been calibrated.

Interpret Benchmarks Correctly

The README records two rounds of synthetic task testing from September 23, 2026. Each round contained the same 4 tasks and 24 actions, and both task groups completed 4/4.

• In the first round, Astra tokens fell from 2,266,693 to 236,295, a reduction of 89.6%, and the end-to-end task rate was 3.05 times the baseline.

• In the second round, Astra tokens fell from 2,256,091 to 192,752, a reduction of 91.5%, and the end-to-end task rate was 3.53 times the baseline.

Here, Astra tokens include cached input, while additional Jev usage is counted separately. The two rounds repeated the same four tasks and therefore do not represent eight independent scenarios. These figures are not billing savings percentages and cannot prove that all applications, arbitrary tasks, or deployment environments will achieve the same results. The non-cached token results from the earlier three-task version did not improve; see the test notes for the detailed scope.

Known Limitations

• The prototype depends on readable DOM or accessibility information.

• Canvas, image understanding, secure desktops, and some cross-framework controls may not be directly supported.

• Capabilities not exposed by the host may require additional handling by the main model.

• The Chrome path cannot run independently without a compatible official host session.

• Historical machine validation records do not mean that a new installation environment has passed acceptance.

• Runtime logs, browser configuration, keys, personal task materials, machine installation state, and rollback packages are not public.

Summary

Jev GUI Delegate separates verifiable GUI operations from the main model's step-by-step loop, allowing the local controller to perform deterministic matching, on-demand semantic selection, state verification, and fault recovery. During installation, first complete locked dependency installation, runtime setup, and offline tests, then configure the key and connect the project to Codex. In practical use, clear task scope, observable success criteria, cautious submission recovery, and revalidation after version changes are key to achieving reliable results.

To learn more about the upstream components, visit the TypeSafe documentation, MCP Python SDK, and pywinauto.