Extract and Normalize Your AI Coding Assistant Chats with Python

What this project does

AI Coding Assistant Data Extraction collects your locally stored conversations from multiple coding assistants and writes them to a consistent JSONL format. It is useful for backing up chat history, performing personal analytics, preparing fine-tuning data, or preserving conversations before an application clears its local database.

The toolkit automatically discovers storage locations on macOS, Linux, and Windows. It can read JSONL files, JSON session files, Markdown transcripts, SQLite databases, and nested JSON storage trees without requiring third-party Python dependencies.

Supported sources

The project currently supports the following tools:

• Claude Code: JSONL session files under ~/.claude/projects/.
• Codex CLI: JSONL rollout files under ~/.codex/sessions/.
• Cursor: SQLite state.vscdb files in global and workspace storage.
• Windsurf: SQLite storage parsed with a best-effort heuristic.
• Trae: SQLite and JSONL storage parsed with a best-effort heuristic.
• Continue: JSON files in ~/.continue/sessions/.
• Gemini CLI: JSON chat files under ~/.gemini/tmp/.
• OpenCode: JSON session, message, and part trees.
• Cline and Roo Code: JSON task folders stored by the editor extension.
• Aider: Markdown transcripts named .aider.chat.history.md inside project directories.

Installation and setup

Clone or download the repository, then run its scripts from the project root. The extractor uses Python's standard library only. Python 3.9 or newer is required, while Python 3.10 or newer is recommended.

python --version

No virtual environment or dependency installation is required for the core extractor. Before extracting, make sure the relevant AI coding tools have been used on the machine so that local conversation files or databases exist.

Run your first extraction

For an interactive workflow, run the main script without arguments. It presents a numbered menu so you can choose the sources to extract.

python extract.py

To extract every supported source without prompts, use --all.

python extract.py --all

You can also select specific sources with a comma-separated list. The source IDs are shown in the repository's source table or by using --list.

python extract.py --sources cursor,claude_code,aider

To see which installations were discovered without extracting any conversations, run:

python extract.py --list

The included shell shortcut performs an all-source extraction on systems where it is available:

./extract_all.sh

Choose output and search paths

By default, timestamped JSONL files are written to ./extracted_data. Use --output-dir to select a different destination.

python extract.py --all --output-dir ./chat-backup

Most tools have predictable application-data directories, so the extractor searches common operating-system locations automatically. Aider is different: it stores one Markdown transcript inside each project directory rather than using a central application-data folder.

Use one or more --search-path options when your projects or application data live outside the usual locations.

python extract.py --sources aider \
  --search-path ~/client-work \
  --search-path /mnt/data/repos

Extra search paths are also useful for nonstandard installations of other supported tools.

Merge conversations into one file

Each selected source normally receives its own timestamped output file. Add --merge to also concatenate all extracted conversations into all_conversations.jsonl.

python extract.py --all --merge

A typical output directory may contain files such as:

extracted_data/
├── claude_code_conversations_20260816_143022.jsonl
├── cursor_conversations_20260816_143022.jsonl
├── aider_conversations_20260816_143022.jsonl
├── cline_conversations_20260816_143022.jsonl
└── all_conversations.jsonl

Understand the normalized format

Each line in an output file represents one conversation. The common structure includes a messages array, a source identifier, and a session_id. Other fields depend on what the original tool recorded.

{
  "messages": [
    {
      "role": "user",
      "content": "How do I fix this TypeScript error?",
      "code_context": [
        {
          "file": "/Users/you/project/index.ts",
          "code": "const x: string = 123;"
        }
      ],
      "timestamp": "2026-01-16T14:30:22Z"
    },
    {
      "role": "assistant",
      "content": "The error occurs because a number is assigned to a string type.",
      "tool_use": [
        {
          "name": "edit_file",
          "input": {"path": "src/index.ts"}
        }
      ],
      "timestamp": "2026-01-16T14:30:25Z"
    }
  ],
  "source": "cursor-composer",
  "session_id": "c1a2b3...",
  "project_path": "/Users/you/project",
  "name": "TypeScript Type Error Fix",
  "created_at": 1705414222000
}

Fields such as code_context, token usage, project paths, timestamps, tool calls, and diffs are included when the source stores them. The fields you can consistently rely on are messages, source, and session_id.

How extraction works

1. The program detects the operating system and builds a list of likely data roots, including locations such as Application Support, .config, .local/share, %APPDATA%, and %LOCALAPPDATA%.
2. Each source searches those roots for its known folder and file names.
3. The appropriate reader loads JSONL, JSON, SQLite, Markdown, or nested JSON storage.
4. The extractor normalizes the discovered records into the shared message schema.
5. Timestamped JSONL files are written to the output directory.

SQLite databases are opened read-only, which helps avoid blocking a running editor. Readers are also isolated so that a corrupt or locked file produces partial results instead of stopping the entire extraction.

Handle undocumented storage formats

Cursor, Windsurf, and Trae do not publish stable storage schemas. Cursor's extractor explicitly handles several known shapes, including workspace ItemTable chat, inline composer data, and split bubbleId composer data.

Windsurf and Trae use a generic heuristic in extractors/common.py. It searches chat-related keys and walks parsed JSON looking for objects that resemble role-and-text message pairs. This is intentionally best effort. If a future application update changes the storage shape, the relevant KEY_HINTS values may need to be updated.

Use the output for training data

The normalized JSONL files can be loaded with the Hugging Face datasets library. The following example loads all extracted files, keeps conversations containing assistant messages, and formats them with a tokenizer's chat template.

from datasets import load_dataset

dataset = load_dataset(
    "json",
    data_files="extracted_data/*.jsonl",
    split="train"
)
dataset = dataset.filter(
    lambda x: any(m["role"] == "assistant" for m in x["messages"])
)

def format_chat(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False
        )
    }

dataset = dataset.map(format_chat)

Inspect and transform the records further according to your model's requirements. Different sources can contain different optional fields, so training pipelines should primarily depend on the shared message structure.

Extend the extractor with a new source

Each source extractor follows a small two-function interface. Copying continue_ext.py provides a simple starting point.

DISPLAY_NAME = "My Tool"
SOURCE_ID = "my_tool"

def find_installations(extra_paths: list[Path] | None = None) -> list[Path]:
    """Return directories or files worth scanning."""

def extract(installations: list[Path]) -> list[dict]:
    """Return normalized conversation dictionaries."""

After creating the module, register it in the REGISTRY list in extract.py. Shared readers and generic heuristics are available in extractors/common.py.

Protect sensitive conversation data

Local chat histories may contain API keys, proprietary code, personal paths, client information, and tool inputs. Review the extracted files before sharing them or using them for training.

1. Scan the output for secrets.
2. Review code_context and tool_use fields carefully.
3. Do not commit extracted_data/ to a public repository.
4. Keep sensitive backups on encrypted storage.

pip install detect-secrets --break-system-packages
detect-secrets scan extracted_data/*.jsonl

The project includes extracted_data/ in .gitignore, but you should still verify your repository and backup practices before publishing anything.

Troubleshooting

No installation found

The tool may not be installed, may not have any saved chat history, or may use a nonstandard location. Run --list first, then provide the relevant directory with --search-path. You can also inspect the source extractor's SEARCH_DIRS or APP_DIR_NAMES constants.

Cursor or Windsurf reports a locked database

The extractors use read-only SQLite connections so a running application normally will not block extraction. If errors continue, close the application and run the extraction again.

Windsurf or Trae produces zero conversations

Confirm that the application directory was found with --list. If it was found but no chats were extracted, inspect the relevant state.vscdb keys and add matching patterns to KEY_HINTS in the appropriate extractor.

Conclusion

AI Coding Assistant Data Extraction gives you a dependency-free way to preserve and normalize local coding conversations across many storage formats and operating systems. Start with python extract.py --list, select the sources you need, review the generated JSONL for sensitive information, and then use the normalized records for backup, analysis, or carefully prepared training workflows.

This toolkit is intended for data from AI tools installed on your own machine. You are responsible for having the rights to extract the data, protecting sensitive information, complying with each tool's terms, and scanning secrets before sharing or training on the results.