/en/tool/robots-txt-generator/ works built in the browser, downloaded through the server
Build the file by groups—you can block Yandex and Google differently—and then immediately check a list of URLs against it: the tool reports not only whether each URL is blocked, but also which line matched. Building and checking happen in the browser; only the domain is sent to the server when you ask it to fetch the file from your site or check a sitemap.
A group defines who gets which rules. A robot reads only the rules in its group, so Yandex and Google are configured differently this way: with two groups, not one list for everyone.
The file assembled above is applied to the list of URLs exactly as a search engine does: the longest match wins; if lengths are equal, allow wins; asterisks and dollar signs work. It shows not just yes or no, but the exact rule that matched.
A full URL is trimmed to the path with parameters, so paste it directly from the export. We convert Cyrillic to percent encoding automatically, as the crawler does.
The group is selected by search engine rules: first an exact name, then a more general one (YandexBot uses the Yandex group), and finally an asterisk.
Done
| URL | Matched rule | Result |
|---|
—
Lines that will be removed after replacement are highlighted in the background. Lines with a plus sign are new. Matching lines are shown without a marker so you can see exactly where the changes go.
Click your CMS in the presets—the paths will populate the fields in the first group. Then edit them as regular text: the preset saves you from having to remember that Bitrix uses /bitrix/, rather than silently imposing it on you.
If Yandex and Google need different rules, click «Add group» or «Copy group for Yandex»: the second group gets its own bots and its own list. There is a separate button for AI bots; it adds a group with eight names and blocks the entire site.
Enter your sitemap URLs, one per line, and click «Check sitemaps»: the server will request each URL and return its response code, size, and content type. A Sitemap line that leads to a 404 is the most common silent error.
Enter your domain and click «Fetch file»—we will retrieve the file currently on your site, split it into groups, and show line by line what will change. Nothing will be overwritten unless you click the button.
Check the URLs in the second section. Paste the list from your export there and see which line blocks each one: this is where you can see that «Allow: /cart/rules/» does not work if «Disallow: /cart\/*» appears above it.
Download the file with the «Download file» button and place it at the domain root. Check that it opens at yoursite.com/robots.txt.
The input data is the same as the default field values: you can calculate it mentally and confirm that the tool is telling the truth.
One group produces one User-agent: * line—all bots without their own group read it.
Three Disallow lines, one for each blocked path, and one Allow line.
The Sitemap line goes at the end, outside the groups: it applies to all bots, so you do not need to repeat it in every group.
In the URL check, /cart/ gets «Blocked» with the rule «Disallow: /cart/», while /cart/rules/ gets «Allowed» with the rule «Allow: /cart/rules/»: its match is twelve characters long versus six.
The URL /catalog/shkafy/ does not match any rule, so it is allowed, and the rule column contains a dash—there is nothing blocking it.
Total: 6 non-empty lines—1 User-agent, 3 Disallow, 1 Allow, and 1 Sitemap. Of the six checked URLs, three are blocked: the cart, the administrative page, and search.
Because a robot reads the rules from exactly one group—its own, the one whose name matches best—and ignores the others. When all groups have the same rules, it makes no difference. But when Yandex needs Clean-param and Google needs /wp-content/uploads/ open, one list cannot express both: you need two groups. Previously, this generator cloned one set into every selected group, pretending it could do something it could not.
No. robots.txt blocks crawling, not display. If links lead to a page, it may remain in search results without a description. To remove a page from the index, use noindex in a meta tag or response header. In that case, you must not block the page in robots.txt: the robot will not visit it and will not see the directive.
By the length of the match, not by the order of the lines. The longest match wins; when lengths are equal, the allowing rule wins. That is why «Allow: /cart/rules/» opens the rules page even below «Disallow: /cart/», while «Disallow: /cart\/*» of the same length overrides it. The address checker on this page uses exactly the same logic, including the asterisk and dollar sign.
Clean-param tells Yandex that a parameter does not change the content: pages with and without it are treated as one, and the weight passes to the main URL. Disallow only blocks crawling; no weight is passed from blocked URLs. It is written as «Clean-param: sort&order /catalog/»: parameters go on the left, separated by an ampersand, and the optional path goes on the right. Google does not understand this directive; for Google, block parameters with a pattern.
This is a decision about rights to the text, not about search. GPTBot, ClaudeBot, CCBot, and similar bots collect material for training; Google-Extended is used only for training and does not affect search rankings, while OAI-SearchBot and PerplexityBot bring real visitors—blocking them means losing traffic. The button on this page adds a group with all eight names, but you decide which ones to leave open.
Google reads the first 500 KiB and discards everything after that. In practice, a hundred-line file is already large; if it has grown to hundreds of kilobytes, URLs were probably listed one by one instead of using patterns. The tool calculates the size and warns you as the limit approaches.
Strictly at the root: example.com/robots.txt. A subdomain has its own file, and the rules apply only to that protocol, domain, and port. Bots do not read a file in a subdirectory. Use UTF-8 without BOM. Line endings can be anything, but a BOM at the start breaks the first directive.
Writes the title and description and instantly shows how they appear in search results.
Open
Shows the full path from the URL to the final page and the response codes.
Open