Robots.txt Generator & Tester
Build robots.txt with user-agent groups and presets, and test which rule allows or blocks a URL.
How to use the Robots.txt Generator & Tester
- 1Start from a preset such as WordPress or Block AI crawlers, or import your current robots.txt.
- 2Add user-agent groups with Allow and Disallow rules, crawl-delay and your sitemap URLs.
- 3Test any path against any crawler to see whether it is allowed and which line decides it, then download robots.txt.
About Robots.txt Generator & Tester
The Robots.txt Generator & Tester builds a robots.txt file from user-agent groups instead of hand-typed lines. Each group lists one or more crawlers with their Allow and Disallow rules, an optional crawl-delay and a comment, and the file can end with one or more Sitemap lines. Presets cover common setups: allow everything, block everything for a staging site, WordPress, an online shop, and add-on groups that block AI crawlers or backlink and SEO tool crawlers.
The built-in tester answers the question that matters most: is this URL allowed for this crawler? Enter a path or full URL, pick a bot, and the tool shows whether it may be crawled and which line of the file decides it, following the longest-match rule that Google and Bing apply, including the * and $ wildcards. You can also import your current robots.txt to check or extend it before downloading the new file.
It suits site owners who want to keep AI training bots out, developers protecting a staging environment and SEO specialists checking why a section is not being crawled. Keep in mind that robots.txt controls crawling, not indexing, and only crawlers that follow the standard respect it.
Frequently asked questions
How do I block AI crawlers like GPTBot and ClaudeBot?
Add a group with their user-agents and Disallow: /. The Block AI crawlers preset lists the common ones, including GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot. Well-behaved crawlers follow it; robots.txt can't force anyone who ignores it.
Which rule wins when Allow and Disallow both match?
Google and Bing use the most specific rule, the one with the longest path. If both are equally long, Allow wins. The order of the lines doesn't matter.
Does Disallow remove a page from Google?
No. It only stops crawling. A blocked page can still be indexed if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag instead.
What do * and $ mean in robots.txt?
* matches any sequence of characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every URL that ends in .pdf.
Does Google respect Crawl-delay?
No. Google ignores it and adjusts its crawl rate automatically. Bing and some other crawlers honour it; Yandex dropped support in 2018.
Where do I put the robots.txt file?
In the root of the host, so it is reachable at https://example.com/robots.txt. It only applies to that exact protocol, host and port, so a subdomain such as shop.example.com needs its own file.
More in Web & SEO
Write title, description, Open Graph, Twitter and hreflang tags with pixel-based length checks.
See how your page looks in Google on desktop and mobile, truncated close to where Google cuts it.
Preview link cards for Facebook, X, LinkedIn, Slack, Discord, WhatsApp and iMessage before you share.