robots.txt tester

Test a crawler path without fetching the site.

Paste robots.txt content, choose a user-agent and path, then see the selected group and longest matching rule. No URL is requested.

Crawler user-agent

Path or complete URL

robots.txt content

Longest matching rule wins

For a selected user-agent group, the most specific matching path normally decides the result; an Allow rule wins when equally specific rules conflict. This local checker follows that common model, but production SEO decisions should also be tested with the crawler operator’s own tooling.