When to use it
See the live robots.txt decision, winning directive, matching rules, and sitemap declarations for a crawler/path pair.
Crawl & indexing
Robots.txt Tester
Fetch a site’s live robots.txt and test a path for Googlebot, Bingbot, GPTBot, ClaudeBot, or a custom user-agent.
Uses a protected server check
Run the tool
Only public web targets are accepted. Private and reserved network ranges are blocked.
What this check means
Uses transparent longest-match logic against the live public robots.txt file; it does not impersonate a search engine crawler.