Free Robots.txt & Sitemap Checker
Make sure you're not accidentally blocking Google from crawling your site, and that your sitemap is reachable and valid. A single wrong line in robots.txt can hide your entire website from search.
Free · No signup required · Results in seconds
Checking …
Couldn't Check That Site
Something went wrong.
What Is a Robots.txt & Sitemap Checker?
robots.txt controls which crawlers are allowed or disallowed from accessing parts of a website — a single misplaced rule can hide your entire site from Google. Your XML sitemap does the opposite job: it hands search engines a list of the URLs you actually want them to discover and crawl.
The two files work together, so checking them side by side during a technical SEO review matters — a perfectly valid sitemap is far less useful if robots.txt is quietly blocking the pages it lists, and a wide-open robots.txt won't help much if your sitemap is broken or missing.
How to Check Your Robots.txt and Sitemap
Enter your website URL.
Run the check.
Review the robots.txt configuration — what's allowed, what's disallowed, and for which crawlers.
Check whether an XML sitemap is available and correctly declared.
Review any issues we detected.
Fix the relevant configuration and run the check again to confirm.
Two Files That Control How Google Crawls Your Site
robots.txt and your sitemap work together to tell search engines what to crawl and index. Here's exactly what the tool looks for in each — nothing more, nothing promised that isn't actually checked.
Robots.txt Checks
robots.txt Availability
Confirms a robots.txt file exists at your domain root and is reachable.
Crawl Directives
Parses Allow/Disallow rules and groups them by user-agent.
Blocked Paths
Lists exactly which paths are disallowed for each crawler.
Site-Wide Blocking
Flags the critical case — "Disallow: /" for all crawlers — that hides your entire site from search.
Sitemap Declaration
Detects whether robots.txt correctly declares your sitemap's location.
Sitemap Checks
Sitemap Availability
Checks the sitemap URL declared in robots.txt, falling back to the common /sitemap.xml and /sitemap_index.xml locations.
XML Format
Confirms the response is valid, parseable XML — not a 404 page or broken markup returned as if it were a sitemap.
HTTP Status
Confirms the sitemap URL itself returns a successful response.
Sitemap Index Support
Detects sitemap index files and lists the child sitemaps they reference.
URL Accessibility
Live-samples up to 10 URLs listed in the sitemap and checks that they actually load, not just that they're listed.
Common Robots.txt & Sitemap Problems
A few issues we see again and again during technical SEO reviews.
Robots.txt Blocks Important Pages
A Disallow rule meant for a staging environment or admin area sometimes accidentally covers real pages too — those pages won't be crawled or indexed until the rule is fixed.
Sitemap Contains Non-Canonical URLs
A sitemap should generally list the preferred, canonical version of each URL. Listing duplicate or parameter-heavy variants sends mixed signals about which version you actually want indexed.
Sitemap Contains Redirected URLs
URLs that redirect shouldn't normally be the final URLs submitted in a sitemap — list the destination URL directly instead, so crawlers don't waste time following the redirect.
Sitemap Returns an Error
If the sitemap URL doesn't load, check the exact URL, the server's response code, and that the XML itself is well-formed — any one of the three can break the whole file.
Robots.txt References the Wrong Sitemap
The Sitemap: line in robots.txt should point to your actual, current sitemap location — a stale or incorrect reference means crawlers may never find the real one.
Robots.txt vs. XML Sitemap
These two files are frequently confused — here's the distinction.
/robots.txt
Can restrict crawling
Can reference sitemaps
/sitemap.xml
Does not control crawling
Lists URLs for discovery
Robots.txt & Sitemaps in Google Search Console
This checker is useful for an initial technical review — catching accidental blocks, broken sitemaps, and misconfigured directives before they become a problem. For the full picture of how Google is actually crawling and indexing your site, Google Search Console is the authoritative source — it shows real crawl stats, indexing status, and any warnings Google itself has flagged.
Need Help Fixing Robots.txt or Sitemap Issues?
If your website has crawl or indexing problems that require development work, Zelthra can audit and implement technical SEO improvements.
More Free Technical SEO Tools
Check another part of your website's technical SEO.
Schema Validator
Validate your structured data and find missing required fields.
Try It FreeBroken Link Checker
Find broken internal and external links before they hurt rankings.
Try It FreeRedirect Chain Checker
Spot redirect chains and loops that waste crawl budget.
Try It FreeMeta Tag Checker
Review titles, descriptions, and Open Graph tags at a glance.
Try It FreeFrequently Asked Questions
Yes. There's no signup, no email required, and no limit on how many sites you can check.
robots.txt is a file at the root of your domain (yoursite.com/robots.txt) that tells search engine crawlers which parts of your site they're allowed to visit.
A line like "Disallow: /" under "User-agent: *" tells every search engine not to crawl any page on your site. This sometimes gets left in place after a staging site goes live, hiding the entire site from Google.
A sitemap (usually sitemap.xml) lists the URLs on your site you want search engines to know about, helping them discover and index your pages more efficiently.
We check up to 10 URLs from your sitemap to confirm they actually load. A sitemap listing pages that return errors sends bad signals to search engines.
No. We fetch robots.txt and your sitemap, check them, and don't save the results anywhere.
Want a Full Technical SEO Audit?
Robots.txt and sitemap issues are just one piece of technical SEO. Our team can audit your entire site's crawlability, indexing, and search visibility.