Crawl & Indexing Tools
Inspect robots, sitemaps, canonicals, hreflang, noindex, status codes, and redirects.
Crawl and indexing checks help you understand whether search engines can reach the right URLs, follow the intended directives, and interpret canonical or redirect signals consistently.
Start here when diagnosing robots.txt rules, sitemap coverage, canonical tags, noindex directives, hreflang declarations, HTTP status codes, or redirect behaviour.
Robots.txt TesterFetch a site’s live robots.txt and test a path for Googlebot, Bingbot, GPTBot, ClaudeBot, or a custom user-agent.Protected server checkRobots.txt GeneratorCreate a simple robots.txt file from explicit allow, disallow, and sitemap rules.Browser-localSitemap ValidatorFetch a live sitemap or sitemap index, recurse through a bounded set of child sitemaps, and sample discovered URLs for status, canonical, robots, and noindex signals.Protected server checkSitemap GeneratorCrawl a small same-site sample and generate an XML sitemap from reachable HTML pages.Protected server checkCanonical Tag CheckerFetch a page and report its HTML canonical link, duplicates, and whether it resolves against the inspected URL.Protected server checkHreflang CheckerFetch a page and list declared hreflang alternates and resolved target URLs.Protected server checkNoindex CheckerInspect robots meta tags and X-Robots-Tag headers for noindex directives.Protected server checkHTTP Status CheckerReturn the direct HTTP response status for a URL without automatically following redirects.Protected server checkRedirect CheckerInspect the first response and report its redirect target when one is present.Protected server checkRedirect Chain CheckerTrace a bounded redirect chain with numbered hops, status codes, resolved destinations, per-hop timing, total timing, loops, and temporary/permanent redirect warnings.Protected server check