Robots.txt Tester

Test whether a path is allowed or blocked by robots.txt rules for a user-agent.

Help Us Improve
4.9 (0)

How to use Robots.txt Tester

Three quick steps — no sign-up needed

  1. Step 1

    Paste robots.txt rules

    Drop in the contents of your robots.txt file.

  2. Step 2

    Enter a URL path and user-agent

    Test a path as Googlebot, Bingbot, or another agent.

  3. Step 3

    Read allow/block outcome

    Fix rules that accidentally block important sections.

Related SEO & Web tools

View all
Backlink Checker Check backlinks pointing to a domain when a data provider is available. Broken Link Checker Find broken links on a page by checking HTTP status codes. Canonical Tag Generator Generate canonical link tags to reduce duplicate URL issues. Domain Age Checker Find when a domain was registered and how old it is. Google Index Checker Check whether pages appear indexed and generate verification-style search links. Heading Structure Checker Inspect H1-H6 heading hierarchy from pasted HTML. Hreflang Tag Generator Create hreflang link tags for multi-language and multi-region pages. Keyword Combiner Mix keyword lists into every combination with broad, phrase or exact style output.

About Robots.txt Tester

Robots.txt Tester checks whether a given path would be allowed or disallowed for a user-agent based on pasted robots.txt rules. SEOs searching "robots.txt tester", "test robots.txt", or "blocked by robots" need confidence before deploy.

Paste rules, enter a path and user-agent, and review the result. Syntax mistakes and conflicting Disallow/Allow lines are common sources of accidental deindexing pressure (via crawl prevention).

Remember robots.txt controls crawling, not always indexing of URLs discovered other ways. Use noindex when you must keep a crawlable URL out of the index.

Benefits of Robots.txt Tester

Rule + path testing Verify before production.
User-agent specific Test Googlebot vs others.
Catch accidental blocks Protect key sections from Disallow mistakes.
Safer releases Validate sitemap and parameter rules.

Frequently asked questions

Does Disallow remove a page from Google?

It mainly blocks crawling; indexing can still happen without content fetch in some cases - use noindex when needed.

What user-agent should I test?

Test Googlebot and any other bots important to you.

Are blank lines OK?

Yes - keep syntax valid and specificities clear.

Can I test a live robots URL?

This tool tests the rule text you paste.

Where should robots.txt live?

At https://www.example.com/robots.txt on the site root.