{"title":"Meet the AI Crawlers Reading Your Site","slug":"meet-the-ai-crawlers","type":"page","excerpt":"A field guide to the answer-engine bots — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — and what they actually fetch.","content":"Not all bots are search engines anymore. A fast-growing share of crawler traffic now comes from **answer engines** — the systems behind ChatGPT, Claude, Perplexity, and Google's AI features — fetching your content to train models and answer questions in real time. Here's who they are and what they actually request.\r\n\r\n## The answer engines\r\n\r\n- **GPTBot** (OpenAI) — trains and indexes content for ChatGPT.\r\n- **OAI-SearchBot / ChatGPT-User** (OpenAI) — real-time fetches when ChatGPT browses or grounds an answer.\r\n- **ClaudeBot** (Anthropic) — the training crawler; **Claude-User** fetches live when someone asks Claude about a page.\r\n- **PerplexityBot / Perplexity-User** — index and live-search fetches for Perplexity's answer engine.\r\n- **Google-Extended** — Google's opt-in signal for Gemini training, distinct from classic Googlebot.\r\n\r\n## What they fetch\r\n\r\nSome bots want your rendered HTML. Increasingly, the well-behaved ones look for **machine-readable** versions first: an `llms.txt` index, clean Markdown renderings of your posts, or JSON-LD structured data. Serving those formats makes your content cheaper to parse — and more likely to be cited accurately.\r\n\r\n## How to see it\r\n\r\nYou can't manage what you can't measure. [AI Bot Log](/) aggregates anonymized bot-visit data from sites across the network into a live dashboard, so you can see which crawlers are active, what they request, and how the trends are moving — instead of guessing from raw server logs.","publishedAt":"2026-06-17T00:21:36.092Z","updatedAt":"2026-06-17T00:21:36.092Z","author":{"name":"Michael Janzen"},"categories":[],"tags":[],"featuredImageUrl":null,"aeo":{"summary":"A field guide to the AI crawlers and answer engines (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) now reading the web — what each one is and what content formats they fetch."},"site":{"name":"AI Bot Log","url":"https://aibotlog.com"},"_links":{"canonical":"https://aibotlog.com/post/meet-the-ai-crawlers","markdown":"https://aibotlog.com/post/meet-the-ai-crawlers/llm.txt","json":"https://aibotlog.com/post/meet-the-ai-crawlers/data.json"}}