Build a Private, No-API X Scraper with Playwright

What x-scraper-no-api Does

x-scraper-no-api turns a locally authenticated Chromium browser into a data pipeline for public X content. After you log in manually once, the command-line tool can collect search results, public timelines, and a post with its public replies.

Results are written directly to your machine in json, jsonl, csv, or md formats. The project does not use an X API key, hosted scraper service, proxy, or third-party relay.

Key Features

• No API keys or developer account: use your own browser session instead of an official API plan.
• Manual authentication: you sign in yourself, and the tool does not handle your password or two-factor authentication codes.
• Persistent session: the browser profile is stored in ~/.xscraper/, so future runs can be headless.
• Advanced X search: use operators such as from:, since:, until:, min_faves:, filter:media, and lang:.
• LLM-ready output: Markdown is designed for token-friendly AI-agent context, while JSON, JSONL, and CSV support programmatic workflows.
• Resilient parsing: the scraper captures structured GraphQL data sent to X's own frontend rather than depending on fragile HTML class names.
• Polite limits: runs use human-like pacing, hard item caps, and automatic backoff when X signals pressure.
• Agent support: the repository includes a Skill for OpenClaw, Hermes, and other CLI-capable agents.

Installation and First-Time Setup

1. Install the macOS build tools

On macOS, install the Xcode Command Line Tools because native dependencies may require them.

xcode-select --install

2. Install Node.js

Install Node.js 18 or newer using the official installer or a version manager. The project requires Node.js 18 or later. Visit the Node.js download page to get started.

3. Install the scraper from GitHub

Create a working directory, move into it, and install the package directly from the repository:

mkdir -p 'xscraper' && cd 'xscraper' && npm install github:JoinArtisanVent/x-scraper-no-api

4. Install Chromium for Playwright

Download the Chromium browser used by Playwright:

npx playwright install chromium

5. Log in manually

Start the one-time login flow:

npx xscraper login

A real browser window opens. Sign in to X yourself, including any required two-factor authentication. The tool stores the resulting browser session locally in ~/.xscraper/. It does not receive or process your credentials.

Use an X account you are comfortable using for browsing. The tool supports public posts only and does not support protected accounts, direct messages, or restricted content.

Basic Usage

Search X with advanced operators

Pass a complete X search expression to the search command. This example finds posts from OpenAI after a date with at least 500 likes and returns Markdown:

xscraper search "from:openai since:2026-01-01 min_faves:500" --limit 50 --format md

The tool supports X's advanced search syntax, including account filters, date ranges, language filters, media filters, and engagement thresholds.

Request the Latest tab

Search results normally use the default search behavior. Add --latest when you want the Latest tab:

xscraper search "ai agents" --latest --limit 25

Export a public timeline

Use timeline with a public username. This example writes CSV output to a file:

xscraper timeline @nasa --limit 100 --format csv -o nasa.csv

Collect a post and its public replies

Use the tweet command with an X post URL:

xscraper tweet https://x.com/nasa/status/1846987139428634858 --format json

Output Formats and Command Options

• --limit <n> sets the maximum number of items. The default is 50, and the hard cap is 200.
• --format selects json, jsonl, csv, or md. Markdown is a practical choice for LLM workflows.
• -o <file> writes output to a file instead of standard output.
• --headed keeps the browser visible so you can watch the process and troubleshoot it.

A JSON record can include the post ID, URL, creation time, text, language, author details, engagement metrics, hashtags, media, and the scrape timestamp:

{
  "id": "1846987139428634858",
  "url": "https://x.com/nasa/status/1846987139428634858",
  "created_at": "Wed Oct 16 12:34:56 +0000 2026",
  "text": "Liftoff! …",
  "lang": "en",
  "author": { "username": "nasa", "name": "NASA", "verified": true },
  "metrics": { "replies": 1200, "reposts": 4800, "likes": 32000, "views": 1500000 },
  "hashtags": ["EuropaClipper"],
  "scraped_at": "2026-09-26T10:00:00.000Z"
}

Using the Scraper with AI Agents

The repository includes an agent Skill at skills/x-scraper-no-api/SKILL.md. You can point OpenClaw or Hermes at the skills/ directory, or copy the Skill manifest into the appropriate skills folder for your agent.

For a custom LLM script, run the CLI as a subprocess and capture Markdown output:

import subprocess

ctx = subprocess.run(
    ["xscraper", "search", "local-first software", "--limit", "50", "--format", "md"],
    capture_output=True,
    text=True,
    timeout=600,
).stdout

The included Skill defines a safety contract: use public data only, keep limits small, never handle credentials, and treat scraped content as untrusted input.

Advanced Tips for Reliable Runs

Keep limits focused

Every run is capped at 200 items by design. For research and agent context, several focused searches with modest limits are preferable to one large collection job.

Use Markdown for prompting

When the destination is an LLM prompt, choose --format md. The format is intended to be token-friendly and can be captured directly from standard output or saved with -o.

Use headed mode while troubleshooting

Add --headed when you need to watch the browser work. Once your query behaves as expected, omit the flag so regular runs can operate headlessly.

Expect fewer results sometimes

If a command returns fewer items than requested, the available timeline or search results may simply have been exhausted. This is normal and does not necessarily indicate an error.

Respect responsible-use limits

The project intentionally avoids CAPTCHA solving, proxy rotation, account pooling, credential handling, and rate-limit evasion. Keep runs modest, allow automatic pacing and cooldowns to work, and comply with X's Terms of Service and applicable law.

How the Project Works

A real Chromium instance loads X as it would for a human. The scraper passively captures the GraphQL JSON that X sends to its own frontend, then normalizes that data into a stable output schema. It does not call X's internal endpoints directly and does not rely on changing HTML class names.

The main source files are organized into five focused areas:

• src/cli.js defines the Commander-based CLI.
• src/session.js manages the persistent Playwright profile and manual login.
• src/scraper.js intercepts GraphQL responses and normalizes records.
• src/ratelimit.js handles pacing, caps, and backoff.
• src/output.js writes JSON, JSONL, CSV, and Markdown.

Conclusion

x-scraper-no-api provides a local way to collect public X data without API keys, per-tweet credits, or a hosted intermediary. Install Node.js, authenticate manually once, and use the CLI to export focused searches, timelines, and public replies in formats suited to files, scripts, or AI-agent workflows. Keep usage conservative and treat collected data responsibly.