Yakha i-X Scraper Eyimfihlo, Engadingi i-API nge-Playwright

Okwenziwa yi-x-scraper-no-api

x-scraper-no-api iguqula isiphequluli se-Chromium esigunyazwe endaweni sibe yipayiphi yedatha yokuqukethwe komphakathi ku-X. Ngemva kokungena mathupha kanye, leli thuluzi lomugqa wemiyalo lingaqoqa imiphumela yosesho, imigqa yesikhathi yomphakathi, kanye nokuthunyelwe nezimpendulo zako zomphakathi.

Imiphumela ibhalwa ngqo emshinini wakho ngamafomethi e-json, jsonl, csv, noma md. Le phrojekthi ayisebenzisi ukhiye we-X API, isevisi ye-scraper ephethwe, i-proxy, noma i-relay yomunye umuntu.

Izici Eziyinhloko

• Azikho izikhiye ze-API noma i-akhawunti yonjiniyela: sebenzisa iseshini yesiphequluli sakho esikhundleni sohlelo olusemthethweni lwe-API.
• Ukuqinisekisa mathupha: uzingenela wena, futhi ithuluzi aliphathi iphasiwedi yakho noma amakhodi okuqinisekisa okuyizinyathelo ezimbili.
• Iseshini eqhubekayo: iphrofayela yesiphequluli igcinwa ku-~/.xscraper/, ngakho ukuqaliswa okulandelayo kungasebenza ngaphandle kokubonisa isiphequluli.
• Uzoekha oluthuthukile ku-X: sebenzisa ama-opharetha afana ne-from:, since:, until:, min_faves:, filter:media, kanye ne-lang:.
• Okulungele ama-LLM: i-Markdown yakhelwe umongo wama-ejenti e-AI osebenziseka kahle ngamathokheni, kanti i-JSON, i-JSONL, ne-CSV zisekela izinqubo ezisebenza ngohlelo.
• Ukuhlaziya okuqinile: i-scraper ithwebula idatha ehlelekile ye-GraphQL ethunyelwa yi-frontend ka-X uqobo, esikhundleni sokuthembela emagameni amakilasi e-HTML angathembekile.
• Imikhawulo ehloniphekile: ukuqaliswa kusebenzisa ijubane elifana nelomuntu, imikhawulo eqinile yezinto, kanye nokwehlisa ijubane okuzenzakalelayo lapho u-X ekhombisa ingcindezi.
• Ukusekelwa kwama-ejenti: indawo yokugcina ikhodi ihlanganisa i-Skill ye-OpenClaw, i-Hermes, namanye ama-ejenti akwazi ukusebenzisa umugqa wemiyalo.

Ukufaka Nokulungiselela Kokuqala

1. Faka amathuluzi okwakha e-macOS

Ku-macOS, faka i-Xcode Command Line Tools ngoba okuncikile komdabu kungase kuwadinge.

xcode-select --install

2. Faka i-Node.js

Faka i-Node.js 18 noma entsha usebenzisa isifaki esisemthethweni noma umphathi wezinguqulo. Le phrojekthi idinga i-Node.js 18 noma eyakamuva. Vakashela ikhasi lokulanda le-Node.js ukuze uqale.

3. Faka i-scraper ku-GitHub

Dala uhla lwemibhalo lomsebenzi, ungene kulo, bese ufaka iphakheji ngqo endaweni yokugcina ikhodi:

mkdir -p 'xscraper' && cd 'xscraper' && npm install github:JoinArtisanVent/x-scraper-no-api

4. Faka i-Chromium ye-Playwright

Landa isiphequluli se-Chromium esisetshenziswa yi-Playwright:

npx playwright install chromium

5. Ngena mathupha

Qalisa inqubo yokungena eyenziwa kanye:

npx xscraper login

Kuvuleka iwindi lesiphequluli langempela. Ngena ku-X wena, kuhlanganise nanoma yikuphi ukuqinisekiswa okuyizinyathelo ezimbili okudingekayo. Ithuluzi ligcina iseshini yesiphequluli ewumphumela endaweni ku-~/.xscraper/. Alitholi futhi alicutshunguli iziqinisekiso zakho.

Sebenzisa i-akhawunti ye-X okhululekile ukuyisebenzisela ukuphequlula. Ithuluzi lisekela okuthunyelwe komphakathi kuphela futhi alisekeli ama-akhawunti avikelwe, imilayezo eqondile, noma okuqukethwe okunomkhawulo.

Ukusetshenziswa Okuyisisekelo

Sesha ku-X ngama-opharetha athuthukile

Dlulisela umyalo we-search inkulumo ephelele yosesho lwe-X. Lesi sibonelo sithola okuthunyelwe okuvela ku-OpenAI ngemva kosuku oluthile, okuthole okungenani ukuthandwa okungu-500, bese sibuyisa i-Markdown:

xscraper search "from:openai since:2026-01-01 min_faves:500" --limit 50 --format md

Ithuluzi lisekela uhlelo lokusesha oluthuthukile lwe-X, kuhlanganise nezihlungi zama-akhawunti, izigaba zezinsuku, izihlungi zolimi, izihlungi zemidiya, kanye nemikhawulo yokuzibandakanya.

Cela ithebhu ethi Latest

Imiphumela yosesho ivamise ukusebenzisa indlela ezenzakalelayo yosesho. Faka i---latest uma ufuna ithebhu ethi Latest:

xscraper search "ai agents" --latest --limit 25

Khipha umugqa wesikhathi womphakathi

Sebenzisa i-timeline negama lomsebenzisi lomphakathi. Lesi sibonelo sibhala okukhiphayo kwe-CSV efayelini:

xscraper timeline @nasa --limit 100 --format csv -o nasa.csv

Qoqa okuthunyelwe nezimpendulo zako zomphakathi

Sebenzisa umyalo we-tweet one-URL yokuthunyelwe ku-X:

xscraper tweet https://x.com/nasa/status/1846987139428634858 --format json

Amafomethi Okukhiphayo Nezinketho Zemiyalo

• --limit <n> ibeka inani eliphezulu lezinto. Okuzenzakalelayo kungu-50, kanti umkhawulo oqinile ngu-200.
• --format ikhetha i-json, i-jsonl, i-csv, noma i-md. I-Markdown iyinketho esebenzayo yezinqubo zama-LLM.
• -o <file> ibhala okukhiphayo efayelini esikhundleni sokukukhiphela kokujwayelekile.
• --headed igcina isiphequluli sibonakala ukuze ubuke inqubo futhi uxazulule izinkinga.

Irekhodi le-JSON lingafaka i-ID yokuthunyelwe, i-URL, isikhathi sokudalwa, umbhalo, ulimi, imininingwane yombhali, amamethrikhi okuzibandakanya, ama-hashtag, imidiya, kanye nesitembu sesikhathi sokuklwebha:

{
  "id": "1846987139428634858",
  "url": "https://x.com/nasa/status/1846987139428634858",
  "created_at": "Wed Oct 16 12:34:56 +0000 2026",
  "text": "Liftoff! …",
  "lang": "en",
  "author": { "username": "nasa", "name": "NASA", "verified": true },
  "metrics": { "replies": 1200, "reposts": 4800, "likes": 32000, "views": 1500000 },
  "hashtags": ["EuropaClipper"],
  "scraped_at": "2026-09-26T10:00:00.000Z"
}

Ukusebenzisa i-Scraper Nama-ejenti e-AI

Indawo yokugcina ikhodi ihlanganisa i-Skill ye-ejenti ku-skills/x-scraper-no-api/SKILL.md. Ungakhomba i-OpenClaw noma i-Hermes kuhla lwemibhalo lwe-skills/, noma ukopishe i-manifest ye-Skill uyifake kufolda efanele yama-Skill ye-ejenti yakho.

Kuskripthi se-LLM esenziwe ngokwezifiso, sebenzisa i-CLI njenge-subprocess bese uthwebula okukhiphayo kwe-Markdown:

import subprocess

ctx = subprocess.run(
    ["xscraper", "search", "local-first software", "--limit", "50", "--format", "md"],
    capture_output=True,
    text=True,
    timeout=600,
).stdout

Ikhono elifakiwe lichaza isivumelwano sokuphepha: sebenzisa idatha yomphakathi kuphela, gcina imikhawulo incane, ungalokothi uphathe imininingwane yokungena, futhi thatha okuqoqwe kusayithi njengokufakwayo okungathembekile.

Amaqhinga Athuthukile Okusebenzisa Ngokuthembekile

Gcina imikhawulo igxile

Ukusetshenziswa ngakunye kunomkhawulo wezinto ezingu-200 ngokomklamo. Ocwaningweni nasemongweni we-ejenti, ukusesha okuningana okugxile okunemikhawulo emaphakathi kungcono kunomsebenzi owodwa omkhulu wokuqoqa.

Sebenzisa i-Markdown ekwakheni imiyalelo

Uma indawo okuya kuyo kuwumyalezo wemiyalelo we-LLM, khetha --format md. Le fomethi yakhelwe ukusebenzisa amathokheni kahle futhi ingathathwa ngokuqondile kokuphumayo okujwayelekile noma igcinwe nge--o.

Sebenzisa imodi enamakhanda lapho uxazulula izinkinga

Engeza --headed lapho udinga ukubuka isiphequluli sisebenza. Uma umbuzo wakho ususebenza ngendlela elindelekile, susa leli fulegi ukuze ukusetshenziswa okuvamile kusebenze ngaphandle kwesibonisi.

Lindela imiphumela embalwa ngezinye izikhathi

Uma umyalo ubuyisa izinto ezimbalwa kunalezo ozicelile, kungenzeka ukuthi umugqa wesikhathi noma imiphumela yokusesha etholakalayo isiphelile. Lokhu kujwayelekile futhi akusho ukuthi kukhona iphutha.

Hlonipha imikhawulo yokusetshenziswa okufanele

Le phrojekthi igwema ngamabomu ukuxazulula ama-CAPTCHA, ukushintshanisa ama-proxy, ukuhlanganisa ama-akhawunti, ukuphatha imininingwane yokungena, kanye nokugwema imikhawulo yezinga. Gcina ukusetshenziswa kusezingeni eliphansi, vumela ukuhamba kancane okuzenzakalelayo nezikhathi zokuphumula kusebenze, futhi uthobele Imigomo Yesevisi ye-X kanye nomthetho osebenzayo.

Indlela Iphrojekthi Esebenza Ngayo

Isibonelo sangempela se-Chromium silayisha i-X ngendlela umuntu abezokwenza ngayo. I-scraper ithwebula buthule i-JSON ye-GraphQL ethunyelwa yi-X ku-frontend yayo, bese ijwayelanisa leyo datha ibe yisikimu sokuphumayo esizinzile. Ayibizi ngokuqondile ama-endpoint angaphakathi e-X futhi ayinciki emagameni aguqukayo amakilasi e-HTML.

Amayela omthombo amakhulu ahlelwe aba izindawo ezinhlanu ezigxile:

• src/cli.js ichaza i-CLI esekelwe ku-Commander.
• src/session.js ilawula iphrofayela eqhubekayo ye-Playwright nokungena mathupha.
• src/scraper.js ibamba izimpendulo ze-GraphQL futhi ijwayelanisa amarekhodi.
• src/ratelimit.js iphatha ukuhamba kancane, imikhawulo, nokubuyela emuva.
• src/output.js ibhala i-JSON, i-JSONL, i-CSV, ne-Markdown.

Isiphetho

x-scraper-no-api inikeza indlela yasendaweni yokuqoqa idatha yomphakathi ye-X ngaphandle kokhiye be-API, amakhredithi e-tweet ngayinye, noma umlamuli osingathiwe. Faka i-Node.js, gunyaza mathupha kanye, bese usebenzisa i-CLI ukukhipha ukusesha okugxile, imigqa yesikhathi, nezimpendulo zomphakathi ngamafomethi afanele amafayela, imibhalo, noma ukuhamba komsebenzi kwama-ejenti e-AI. Gcina ukusetshenziswa kunengqondo futhi uphathe idatha eqoqiwe ngokuzibophezela.