Run and Explore GPT 6 Astra Embodied-Policy Evaluations

What this project does

This repository contains evaluation code and a bilingual technical report for using GPT 6 Astra in robotic control. It compares GPT 6 Astra Direct with a hybrid policy that reviews and optionally corrects π₀.₅ actions across ten RoboDojo tasks, with five aligned cases per task.

The hybrid policy reports a mean Score of 62.60 and 48% success. GPT corrections were used on 14.4% of executed control steps. GPT 6 Astra Direct achieved 26% success and a mean Score of 37.81 over the 48 episodes with available native scores. Both success rates use all 50 selected cases. The official-model comparisons are reweighted public references rather than same-seed reruns.

The release is a history-free source snapshot. It does not include credentials, private annotations, raw model sessions, original operations logs, simulator assets, or model checkpoints. The required simulator and policy environments must be obtained separately from their upstream projects.

Understand the repository layout

• hybrid_rollout/robodojo/ contains the simulator integration, policy server, controller, skills, and scheduler.
• hybrid_rollout/robolab/ contains the separate RoboLab integration.
• hybrid_rollout/report_site/ contains the bilingual report source, gallery, and packaging tools.
• report_web/ contains the prebuilt report and its 21 article clips.
• public_results/ contains selected case metadata, scores, seeds, and provenance.
• licenses/ contains third-party notices.

The production website includes the complete video gallery and is maintained in a separate repository. The RoboDojo website provides 100 evaluation videos and 43 gallery clips. The RoboLab gallery provides 150 recordings covering three methods, ten tasks, and five trials.

Preview the bilingual report

The quickest way to explore the release is to start the local report preview from the repository root. The preview server supports video byte ranges, so the report can play its included clips efficiently.

python3 -m hybrid_rollout.report_site.preview --directory report_web --port 8768

Open http://127.0.0.1:8768/ in a browser. English is selected by default. Use the language button or add ?lang=zh to switch to Chinese.

The static report_web/ directory can also be hosted with a standard HTTP server. Public GitHub Pages serves only the compiled static report and selected media; it does not expose the annotation API or rollout services.

Rebuild the report frontend

To rebuild the report, use Node.js compatible with the locked Vite version. Node 24 was used for this release. Change into the frontend application, install the locked dependencies, and run the production build.

cd hybrid_rollout/report_site/app
npm ci --ignore-scripts --no-audit --no-fund
npm run build

The build verifies the preserved shared runtime and creates a self-contained dist/index.html. Copy that file to report_web/index.html, while keeping data.json, scores.csv, citation.bib, licenses, and media alongside it.

The public authoring snapshot is included, so a private experiment store is not needed for this frontend rebuild. Do not run build.py merely to rebuild the frontend. That offline evidence-extraction utility requires the original, non-public trajectory archives.

Prepare a new evaluation

New evaluations require externally installed environments. This repository is not a standalone simulator distribution.

1. Install and configure Isaac Sim 5.1, the matching RoboDojo source and assets, and the π₀.₅ OpenPI/JAX checkpoint. RoboLab has its own environment and setup requirements. Obtain all of these from their upstream projects under their respective terms.
2. Review hybrid_rollout/robodojo/SOURCE.json to identify the recorded upstream revisions and checkpoint identity.
3. Prepare a separate tools environment, simulator environment, OpenPI/JAX environment, checkpoint, and Codex CLI. Keep the simulator and policy environments separate from the lightweight repository tools.
4. Copy or review hybrid_rollout/robodojo/cluster.env.example. Set your paths, container image, user and group IDs, credential file paths, and explicit case identity. The paths in the public export are generic examples, not runnable cluster credentials.
5. Use hybrid_rollout.robodojo.codex_backend.profiles to manage isolated rollout authentication. Inspect the CLI with --help before login or lease operations.
6. Use public_results/evaluation_cases.json to align the task, scene, and seed across policies. Each case contains one hybrid result and one Direct result.

The model is fixed to gpt-6-astra with xhigh reasoning. You need your own authorized account or gateway access, and policy or model execution may incur costs.

Select the evaluation method

Set ROLLOUT_EVALUATION_METHOD to choose the policy mode:

ROLLOUT_EVALUATION_METHOD=pi05_plus_gpt

Use pi05_plus_gpt for the hybrid policy, which allows GPT to review and optionally correct π₀.₅ actions. To select the Direct policy, use:

ROLLOUT_EVALUATION_METHOD=gpt_only

gpt_only is an internal CLI identifier for GPT 6 Astra Direct. It does not launch π₀.₅.

The container entry point is hybrid_rollout/robodojo/cluster_entrypoint.sh. Each container runs one explicit case, saves its artifacts, and cleans up its own components. The optional acp.py file is a SenseCore-specific submission adapter. Cluster-specific campaign migration helpers require deployment-local plans and are not a one-command public launch recipe.

Run the reproducible offline tests

Offline tests require the tools dependencies, while simulator and OpenPI dependencies must be installed separately in their own environments. Install requirements-tools.txt in a dedicated virtual environment, then run the report and diagnostic tests.

python3 -m unittest hybrid_rollout.report_site.test_i18n hybrid_rollout.report_site.test_editorial_revision -v
python3 -m pytest hybrid_rollout/robodojo/test_proposal_diagnostics.py hybrid_rollout/robodojo/test_prompt_context.py hybrid_rollout/robodojo/test_network_continue.py

These tests do not perform simulation or model calls. Tests that depend on original shared storage, a proprietary simulator, or historical operations fixtures cannot be reproduced from this public snapshot alone. The release has not rerun paid model or GPU experiments.

Security and deployment guidance

Gateway URLs in codex_backend/profiles.py and codex_backend/config.toml use reserved .invalid placeholders. Configure your own authorized endpoint explicitly in the selected profile and matching configuration. Set ROLLOUT_GATEWAY_NO_PROXY when your network requires it.

Never commit API keys or auth.json, and do not modify your interactive Codex home. Store credentials outside the repository with restricted permissions. Historical profile identifiers exist for compatibility only; this release does not configure a company or relay service.

Keep annotation and review tools on a trusted local network. Retain the default loopback binding, or use an authenticated reverse proxy before making a service accessible on a LAN. Keep Codex tool execution and simulator RPC inside an isolated, trusted environment, and do not provide untrusted programs, serialized RPC data, or trajectory archives.

Advanced usage and research references

Use the case metadata in public_results/evaluation_cases.json to preserve task, scene, and seed alignment when comparing policies. The gallery manifest records each video’s URL, size, and SHA-256, which can help verify downloaded media.

For the report, gallery, and complete video collection, visit the report and video gallery. The production website source is available in the website repository. The RoboLab gallery provides the separate RoboLab recordings.

Project-owned code is MIT licensed. Third-party components, fonts, benchmark assets, and logos retain their own terms; consult THIRD_PARTY_NOTICES.md. Model weights and simulator assets are not redistributed.

Conclusion

This repository is best used as an evaluation and reporting source release: preview the included report, rebuild its static frontend, run the available offline tests, and use the documented environment boundaries to prepare a new RoboDojo or RoboLab run. Before launching any model or simulator workload, obtain the required upstream dependencies, configure authorized services, align cases with the public metadata, and protect credentials and control interfaces.