Does ChatGPT really respect robots.txt?

Yes — OpenAI documents three crawlers that all honor it: OAI-SearchBot (search citations), GPTBot (training data) and ChatGPT-User (live fetch when a user asks about a URL). Blocking the wrong one silently removes you from ChatGPT's answers.

Is llms.txt actually used by AI crawlers?

Adoption is growing — Anthropic, docs platforms and dev-tool companies publish one, and several answer engines read it as a curated index. It's a one-time, ~30-line file, so the cost-benefit is strongly in favor of having it.

What's the difference between SEO and GEO?

SEO aims for a position in a list of links; GEO (generative engine optimization) aims for a mention inside a written answer. The foundations overlap — indexable, well-structured pages — but GEO weighs machine-readability (schema, llms.txt, crawler access) and third-party mentions much more heavily.

Why paste instead of entering a URL?

Two reasons: many sites block server-side scanners with bot protection, so a fetched page isn't what you actually serve; and pasting keeps everything in your browser — staging URLs and internal pages never touch a third-party scanner.

AI Search Readiness Checker — Can ChatGPT & Perplexity Cite You?

Paste your page HTML, robots.txt and llms.txt to see if ChatGPT, Perplexity, Claude and Google AI can actually read, parse and cite your site — 13 checks, all in your browser.

Page HTML (view source of the page you want cited)Open the page → Ctrl+U → copy all → paste. Everything runs locally in your browser — pasted content is never uploaded, logged or stored.robots.txt (optional — for crawler-access checks)From https://yoursite.com/robots.txt — checks OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and Googlebot.llms.txt (optional — if your site has one)From https://yoursite.com/llms.txt — validates structure (title, summary, links).

Waiting

Paste content above

The tool runs instantly, entirely in your browser.

Last reviewed 2026-10-19. Rates change — verify against the official sources linked below before relying on these figures.

What this is based on

Checks the signals answer engines actually use: crawler access (robots.txt), parseability (JSON-LD, headings, canonical), quotability (answer-first text, Q&A structure, tables) and trust signals (dates, authorship).
robots.txt matching is simplified: exact bot names and the * wildcard are checked; a literal `Disallow: /` counts as blocked.
This audits readiness, not actual citations — the full check is running your buyers' questions in ChatGPT/Perplexity/Gemini and counting who gets named.
Everything runs locally in your browser — pasted content is never uploaded, logged or stored.

OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User ↗llms.txt specification ↗Dark Visitors — AI crawler directory ↗Free CLI version on GitHub ↗Full AI visibility audit agent skill ↗

What answer engines actually check

Getting cited by ChatGPT, Perplexity or Google's AI Overviews comes down to four things. Can they reach the page (robots.txt lets their crawlers in)? Can they parse it (clean HTML, one H1, structured data)? Can they lift an answer from it (a direct answer up front, question-shaped headings, tables and lists)? And do they trust it (a named author, an Organization entity, visible dates)?

The mechanics differ from classic SEO in one important way: the prize is not a position in a list of links — it's being named inside a composed answer. A page can rank #1 on Google and still be invisible to ChatGPT if its crawlers are blocked or the content can't be cleanly extracted.

The robots.txt mistake most sites make

Many sites added Disallow rules for AI bots during the 2023–24 scraping backlash — and now block themselves out of AI search. The key detail: OpenAI and Perplexity use separate bots for training and for search. Blocking GPTBot keeps your content out of training data; blocking OAI-SearchBot or PerplexityBot keeps you out of the answers.

If you want citations without contributing to training data, allow the search and user-triggered bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User) while keeping the training crawlers (GPTBot, ClaudeBot) disallowed.

Readiness vs. actual visibility

This tool audits whether your site can be cited — the infrastructure half of GEO. The other half is whether you actually are: take the questions your buyers ask, run them in ChatGPT, Perplexity, Gemini and Google AI, and count how often the answers name you versus your competitors.

Agencies charge $300–1,000 for that full audit, then a monthly retainer to track it. The readiness fixes this tool flags are almost always the first items on their backlog — fix these before paying anyone.

Frequently asked questions

Does ChatGPT really respect robots.txt?: Yes — OpenAI documents three crawlers that all honor it: OAI-SearchBot (search citations), GPTBot (training data) and ChatGPT-User (live fetch when a user asks about a URL). Blocking the wrong one silently removes you from ChatGPT's answers.
Is llms.txt actually used by AI crawlers?: Adoption is growing — Anthropic, docs platforms and dev-tool companies publish one, and several answer engines read it as a curated index. It's a one-time, ~30-line file, so the cost-benefit is strongly in favor of having it.
What's the difference between SEO and GEO?: SEO aims for a position in a list of links; GEO (generative engine optimization) aims for a mention inside a written answer. The foundations overlap — indexable, well-structured pages — but GEO weighs machine-readability (schema, llms.txt, crawler access) and third-party mentions much more heavily.
Why paste instead of entering a URL?: Two reasons: many sites block server-side scanners with bot protection, so a fetched page isn't what you actually serve; and pasting keeps everything in your browser — staging URLs and internal pages never touch a third-party scanner.

Related calculators

Security Headers Checker

Paste HTTP response headers and instantly see which security headers are missing — HSTS, CSP, X-Frame-Options, nosniff, Referrer-Policy, Permissions-Policy — plus cookie flag and Server version checks.

CORS Config Checker

Paste your API's CORS response headers and detect dangerous setups: wildcard origins with credentials, null-origin acceptance, reflected origins and missing Vary: Origin.

Secret Leak Scanner

Paste code or a git diff and detect leaked credentials — AWS keys, GitHub and Slack tokens, private key blocks, hardcoded API keys and JWTs. Matched values are masked, never displayed.