Paste a robots.txt file, pick a crawler and test your paths. Every verdict comes with the exact rule and line number that decided it.
Your files are processed in your browser, on your own device. Nothing is uploaded to a server.
This robots.txt checker tests the file you paste against the paths you care about and shows Allowed or Blocked for each one, with the exact rule and line number that made the call. It follows the RFC 9309 matching rules that crawlers use, and it tests your pasted text only, never a live server.
No account, nothing to install. The whole test runs on the text in front of you.
The lint flags rules written before any User-agent line, because crawlers ignore rules outside a group. It lists unknown directives, extracts your Sitemap lines, notes crawl-delay, and calls out a Noindex line, not a robots.txt directive since September 2019. It also warns when a file passes 500 KB, around the size where Google stops reading.
Most arguments about robots.txt come from one wrong guess: that the first matching rule wins. That is not how the standard works, and this checker follows RFC 9309 step by step.
The checker starts by finding the group that applies to your chosen crawler. The most specific matching group wins, so a group written for Googlebot beats the star group for Googlebot. Groups for the same crawler in different parts of the file are merged and read as one.
Inside that group, every rule is tested against the path. The longest matching pattern wins. Where an Allow and a Disallow match at the same length, Allow takes the tie. A star matches any run of characters, and a dollar sign at the end anchors the match to the end of the path.
Two quiet cases matter. An empty Disallow means the group allows everything, and a path that matches no rule at all is allowed. Before any of this runs, percent-encoding in the path is normalised, so a written path and its encoded twin get the same answer.
The classic case is a section of a site that stopped being crawled after a redesign. Paste the new file, test the section path as Googlebot, and the deciding line usually names the rule a developer added for a different reason entirely.
It also settles the AI question with facts instead of guesses. Test the same paths as GPTBot and then as PerplexityBot and you can see, line by line, whether your file treats training crawlers and search crawlers the way you meant it to. Files written by plugins deserve the same treatment: generate one, paste it here, and check the output blocks what the settings screen promised.
This part matters, so it sits on the page rather than in a footnote. The checker tests the file you paste. It does not fetch the file your server serves, and it cannot see how your server answers when a crawler asks for robots.txt.
That gap has real teeth. If your server answers with a 404, crawlers treat the site as having no rules at all, so everything is allowed. A 500 answer means everything is treated as blocked for a while. Both are facts about the live response, not the text, and no paste based tool can judge them.
Verdicts here are the standard reading of your rules. Real crawlers follow RFC 9309 closely, though single bots differ on odd edge cases now and then. Read a surprising verdict as a prompt to look at the named line, not a promise about one crawl on one day.
Three usual reasons. A longer Allow pattern matched the same path and won. The Disallow sits in a group for a different crawler, so it never applied to the one you tested as. Or the rule sits before any User-agent line, where crawlers ignore it. The named deciding rule points you at the real cause.
It means that group blocks nothing. User-agent: * followed by Disallow: with nothing after it is the standard way to say every crawler may read the whole site.
No. Noindex was never an official robots.txt directive, and Google dropped support for it in September 2019. The lint flags it so you can move that job to a meta robots tag or an X-Robots-Tag header, where crawlers can read it.
No. You paste the file and it tests that text. That is a deliberate limit. A paste cannot show how your server answers, so the checker cannot tell you whether the request returns a 404, a 500 or the file itself.
Google reads roughly the first 500 KB of a file and stops there. Rules past that point may never be seen. The checker warns you when a pasted file passes that size, a good prompt to trim dead rules.
Yes. Paste one URL per line and the checker tests the path and query of each. Percent-encoding is normalised first, so an encoded address and its plain form return the same verdict.
Same rules as this one: free, in your browser, with nothing uploaded.
See if a page repeats a phrase a healthy amount.
Title and description tags, written and copied clean.
A correct robots file without the guesswork.
A fast on-page check with a clear fix list.
Read the title, description and social tags of any page.
The tags that decide how a shared link looks on social media.
See the social preview tags a page already carries.
Card tags so a link posted on X shows its picture and title.