AI Crawler
An AI crawler is an automated bot that AI companies use to read web pages, either to train models or to fetch current information when answering a question. Examples include OpenAI's GPTBot and OAI-SearchBot, and PerplexityBot. Sites control their access through robots.txt and firewall settings, so a misconfigured site can block the tools it wants to be found by.
Back to top