Crawl & Indexing Tools
Inspect robots, sitemaps, canonicals, hreflang, noindex, status codes, and redirects.
Robots.txt TesterFetch a site’s live robots.txt and test a path for Googlebot, Bingbot, GPTBot, ClaudeBot, or a custom user-agent.Protected server checkRobots.txt GeneratorCreate a simple robots.txt file from explicit allow, disallow, and sitemap rules.Browser-localSitemap ValidatorFetch a live sitemap or sitemap index, recurse through a bounded set of child sitemaps, and sample discovered URLs for status, canonical, robots, and noindex signals.Protected server checkSitemap GeneratorCrawl a small same-site sample and generate an XML sitemap from reachable HTML pages.Protected server checkCanonical Tag CheckerFetch a page and report its HTML canonical link, duplicates, and whether it resolves against the inspected URL.Protected server checkHreflang CheckerFetch a page and list declared hreflang alternates and resolved target URLs.Protected server checkNoindex CheckerInspect robots meta tags and X-Robots-Tag headers for noindex directives.Protected server checkHTTP Status CheckerReturn the direct HTTP response status for a URL without automatically following redirects.Protected server checkRedirect CheckerInspect the first response and report its redirect target when one is present.Protected server checkRedirect Chain CheckerTrace a bounded redirect chain with numbered hops, status codes, resolved destinations, per-hop timing, total timing, loops, and temporary/permanent redirect warnings.Protected server check