bot?Robots.txt Tester

About the Robots.txt Tester

This tester applies the rules in RFC 9309, the Robots Exclusion Protocol, and the way Google documents its own behavior. For each URL it keeps every Allow and Disallow rule from the group that applies to the crawler and picks the rule with the longest matching path. If an Allow and a Disallow match equally, the Allow wins. An asterisk in a path matches any run of characters, a dollar sign at the end of a rule marks the end of the URL, an empty Disallow blocks nothing, and paths are case sensitive. Rules match from the start of the path, so Disallow: /admin also blocks /admin-guide, while Disallow: /admin/ does not block /admin. For the crawler name, the tester uses the group whose User-agent equals that name, ignoring case, and falls back to the User-agent: * group when no group names it. Groups with the same name are combined, and a named group does not inherit rules from the star group. Paste the file instead of fetching it, so you can test a change before you publish it. The tool also lists warnings, such as a Sitemap line that is not a full address, a path without a leading slash, a rule that comes before any User-agent line, a Crawl-delay line that Google ignores, and files over 500 KiB. It does not test noindex tags, which are a separate mechanism, and it follows the standard, so a crawler with its own quirks may differ in edge cases.