Free tool by SEO Dispatcher
Robots.txt checker
A small robots.txt mistake can stop a crawler reaching your pages. Check the rules at your site's root and see whether they allow the homepage and the path you entered.
Know what you’re looking at
What this tool checks
- File response and crawler rule groups
- Broad crawl blocks and sitemap declarations
- Crawler permissions for the homepage and your chosen path
What the results mean
Robots.txt controls crawling. It is not a privacy control and does not, by itself, keep a URL out of search results.
Start with anything marked “Review.” Read the evidence before making a change—some settings may be intentional.
Find common crawling and indexing problems →A few things to know
Is this tool free?
Yes. You can run a check without signing in. A free account unlocks all rule checks, crawler permissions for the tested path and the fetched robots.txt file. You don’t need a paid plan for the fuller free report. Each user gets three runs per domain, shared across all tools, with no reset. Sign-in doesn’t add runs. You can reopen a saved report for 24 hours without spending another run.
Do I need to connect Google Analytics or Search Console?
No. This tool uses public sources and works independently of your app workspace. Connected Google insights remain available separately in the paid app.
Will this tool change my website?
No. It reads public information and shows findings for you to review. You decide which changes to make.
Why might a check fail?
The website may block automated requests, require sign-in, or return an unsupported file. Try a public page that opens in your browser. Large files and page samples are limited to keep checks responsive. An accepted attempt uses one run even if it cannot finish; invalid inputs are rejected before a run is spent.