Skip to main content
AI tutorials

Build and Verify a Legal AI Agent with the Legal Agent Workshop

This tutorial walks through the Legal Agent Workshop’s two sessions: building a general legal agent with the OpenAI Python SDK, applying it to a case, reviewing its omissions, and adding external checklist and memo verification before evidence release and attorney sign-off.

About 18 min read
Build and Verify a Legal AI Agent with the Legal Agent Workshop

What the Legal Agent Workshop Does

The Legal Agent Workshop is a hands-on, two-session project for building and evaluating a legal AI agent. Participants first implement an agentic investigation loop and produce an unverified legal memo. They then add a verification layer that operates outside the agent’s control.

The repository provides a pre-built kit based on plain OpenAI Python SDK code. It includes one file per tool, Files API document reading, and a case-management system that tracks quote-checked facts. The included case is northbridge-v-caldera, but the agent itself remains general and does not contain the case.

Important workspace rule: Launch Muse Code only from the agent/ directory. The cases/, world/, ground_truth/, solutions/, and ui/ directories must remain outside the Muse Code workspace.

Key Features

  • General legal agent: The agent drafts a checklist, creates a plan, investigates documents, and produces a memo.
  • OpenAI Python SDK implementation: The agentic loop uses direct SDK code rather than an additional agent framework.
  • Document analysis: The kit uses the Files API to read case documents.
  • Quote-checked facts: The case-management system records facts with supporting quotations.
  • External verification: Checklist and memo checks run outside the agent’s control.
  • Ground-truth comparison: The checklist verifier compares the agent’s work with a case-specific ground-truth checklist that the agent cannot see.
  • Resumable workflow: A Session 1 run can be branched and continued with verification enabled.
  • Live Agent Console: The pre-built interface displays turns, checklist items, supporting proof, evidence release, and attorney sign-off.

Understand the Repository Structure

  • HANDBOOK.md contains setup instructions, both workshop sessions, and the Muse Code ground-rules prompt in section 1.7.
  • agent/ contains the starter code and is the only directory that should be exposed as the Muse Code workspace.
  • cases/northbridge-v-caldera/ contains the case objective and documents supplied to the agent.
  • ground_truth/ contains the private checklist used by external verification. It must never be included in the case or shown to the agent.
  • world/ represents the simulated outside world, including withheld follow-up documents and the evidence-release script.
  • solutions/ contains snapshots for Session 1 and Session 2.
  • ui/ contains the pre-built Agent Console and remains outside the Muse Code workspace.

Install and Configure the Workshop

1. Clone the repository

Clone the project, then enter the agent/ directory before setting up or launching Muse Code.

git clone https://github.com/meta-models/legal-agent-workshop.git
cd legal-agent-workshop/agent

2. Create the environment file

Copy the example configuration to .env.

cp .env.example .env

Edit .env and add your MODEL_API_KEY. Confirm that CASE_DIR points to the included case:

../cases/northbridge-v-caldera

3. Run the setup helper

./helper setup

The setup command creates a .venv environment with the OpenAI Python SDK, copies the case documents into the expected location, and checks the configured API key.

4. Start Muse Code

While still inside agent/, launch Muse Code:

muse

Your first message must be the ground-rules prompt from section 1.7 of HANDBOOK.md. Submit that prompt again after every restart or new Muse Code session.

Session 1: Build and Run the Legal Agent

Session 1 focuses on implementing the general agentic loop using the supplied kit. Follow the step-by-step instructions in HANDBOOK.md while keeping all work inside agent/.

The intended workflow is:

  1. Draft a checklist: Identify the questions and evidence the investigation should cover.
  2. Plan the investigation: Decide how to use the available tools and case documents.
  3. Investigate: Read documents through the provided document workflow and capture quote-checked facts in the case-management system.
  4. Prepare the result: End the run with an unverified memo based on the investigation.

At this stage, the memo is deliberately unverified. Session 2 demonstrates how apparently complete agent output can still omit required checklist items or violate memo rules.

Watch a Run in the Agent Console

From the repository root, start the pre-built interface:

python3 ui/ui.py

Open http://127.0.0.1:8765 in a browser. The console can show a run live, including each agent turn, its checklist, and how checklist items are proven. It also supports the workshop’s evidence-release and attorney-sign-off stages.

Session 2: Test What the Agent Missed

Session 2 introduces two independent verification checks. One evaluates the checklist against the case’s hidden ground truth. The other evaluates the memo against its required rules.

Verify the memo

From agent/, replace RUN_ID with the identifier of the Session 1 run:

./helper verify RUN_ID --memo

This reveals problems the Session 1 memo check detects.

Verify the checklist

./helper verify RUN_ID --checklist

This compares the agent’s checklist with the case-specific ground-truth checklist. The ground truth remains outside both the case and the agent’s workspace.

Build the verification checks

Follow the Session 2 instructions in HANDBOOK.md to build the checklist and memo checks. Preserve the trust boundary: these checks must run outside the agent’s control rather than relying on the agent to approve its own work.

Continue the Run with Verification Enabled

Branch the original Session 1 run from its existing checklist:

./helper branch RUN_ID

The continued workflow exercises verification-aware behavior, including refusals, escalation, release of new evidence, resumption, and attorney sign-off. Branching preserves the checklist as the starting point instead of beginning an unrelated investigation.

Release withheld evidence

When a run reaches needs_input, use the simulated outside world to release the follow-up material. Run this command from the repository root:

python3 world/release.py

After evidence is released, continue the workshop flow and observe the resumed run in the Agent Console.

Advanced Tips

Protect the workspace boundary

Always start Muse Code from agent/, not from the repository root. This prevents Muse Code from seeing the case source, hidden ground truth, simulated world, solution snapshots, or UI code. The separation is central to the workshop’s verification model.

Never expose ground truth to the agent

The file ground_truth/northbridge-v-caldera.json exists for the external checklist check. Do not copy it into the case, include it in the Muse Code workspace, or supply it to the agent.

Reapply the ground rules after restarts

A restarted or newly created Muse Code session does not remove the requirement to provide the ground-rules prompt. Copy it from section 1.7 of HANDBOOK.md each time.

Add another case using the established layout

A new case is represented by a folder containing an objective.md file and a documents/ directory. Keep the case outside agent/ and point CASE_DIR at it. The README does not specify additional case-authoring or ground-truth-generation steps, so follow the handbook before extending the verification setup.

Use the solution snapshots carefully

The solutions/01_session1/ and solutions/02_session2/ directories provide snapshots for the two sessions. Because they sit outside the Muse Code workspace, they can be used as workshop references without making them part of the agent’s working context.

Command Reference

git clone https://github.com/meta-models/legal-agent-workshop.git
cd legal-agent-workshop/agent
cp .env.example .env
./helper setup
muse

# From the repository root
python3 ui/ui.py
python3 world/release.py

# From agent/
./helper verify RUN_ID --memo
./helper verify RUN_ID --checklist
./helper branch RUN_ID

Conclusion

The Legal Agent Workshop demonstrates both sides of legal-agent development: constructing an agent that can plan, investigate, and draft, and creating external controls that test whether its work is complete and compliant. By keeping cases, ground truth, verification resources, and simulated evidence outside the Muse Code workspace, the project makes the trust boundary explicit and testable.