/en/tool/sitemap-generator/ works crawls a site
Enter a site URL. The server follows internal links, builds a list of pages, removes pages blocked from indexing and broken pages, and adds dates from the server response. Download the completed sitemap as an xml file, xml.gz file, or archive containing parts and an index. The server fetches an existing sitemap from its URL and compares it with the crawl.
The same report that people run desktop crawlers for: response code, depth, date, and the reason a URL was not added to the sitemap.
Our server fetches the sitemap by its URL: it decompresses compressed files and reads index files in parts. Then you see how many URLs it contains, what is broken, and how it differs from the site crawl.
Leave blank—we'll use /sitemap.xml at the root of the crawled site. After the crawl, this field is automatically filled with the URL declared in robots.txt. Supports xml, xml.gz, index files linking to parts, and text lists at the URL in the field.
A sitemap.xml, sitemap.xml.gz, or text list will work. Use this when the sitemap has not been published yet.
An index file lists the parts, and each part requires a separate request to the server. For a site with a hundred parts, there is no need to parse them all: the first few already show how the sitemap is structured.
Sitemap not analyzed
| URL | What's wrong | Crawl |
|---|
Sitemap hasn't been analyzed yet.
Enter the site URL and click “Crawl.” The server follows internal links and returns a list of pages with response codes; the list appears below in a field where you can edit it manually.
Click “Check pages in detail.” Each URL is queried separately. The tool retrieves the Last-Modified date, meta robots, X-Robots-Tag, canonical, and language versions. Pages blocked from indexing, broken pages, and pages whose canonical points to a third-party URL are marked and excluded from the sitemap.
Check the “URLs and their status” table. It shows why each excluded URL was excluded; you can remove individual entries with the button or download the entire table as CSV.
Select a priority and frequency and, if needed, set a common date. If URLs have their own dates and priorities, those take precedence over the common setting, except for “Do not specify.”
Download the file. For one file, the result is sitemap.xml or the compressed sitemap.xml.gz; for more than 50,000 URLs, it is an archive with the parts and the sitemap-index.xml index. An HTML sitemap for visitors and a plain-text list, which Google also accepts, are included alongside them.
Place the file in the site root and add the ready-made Sitemap line to robots.txt. It is printed to the right of the sitemap.
If the sitemap is already on the site, start with the last block: click “Parse,” and the server will fetch it. The index file will be expanded part by part, and .gz will be decompressed. The tool will show the number of URLs, format errors, and—if a crawl was run first—what is missing from the sitemap and what remains in it from the previous version of the site.
The input data is the same as the default field values: you can calculate it mentally and confirm that the tool is telling the truth.
Depth is the number of path segments. The root has zero, \/about\/ has one, and \/blog\/first-post\/ has two.
Priority: 1.0 − 0.2 × depth. This gives 1.0 for the root, 0.8 for \/about\/ and \/blog\/, and 0.6 for the blog post.
Each URL takes four lines: the opening url tag, loc, priority, and the closing url tag.
There are two service lines at the top—the XML declaration and the opening urlset tag—and one closing tag at the bottom.
The sitemap has 4 URLs and 19 lines: 2 at the top, 4 × 4 = 16 for the URLs, and 1 at the bottom. The counters show exactly this. After crawling a live site, the same lines will also contain dates from the server response, while excluded URLs will move to the table with a reason.
The server returns a hundred URLs per request, while the selected limit is reached over several passes—up to five hundred. The limit is shared by all tools on the site: three hundred requests per address per hour, so it still cannot fit a complete crawl of someone else's site. For a larger site, export a list from your CMS and paste it into the field: validation, splitting, and downloading work the same way.
4xx and 5xx responses; pages with noindex in a meta tag or the X-Robots-Tag header; pages with a canonical pointing to another URL; and URLs blocked in robots.txt. Each such URL remains in the table with a reason—it is a report, not silent deletion.
From the response's Last-Modified header during a detailed check. If the server does not return it, no tag is added: an invented date is worse than no date. The “date for all URLs” field overrides the dates found—this is a manual override.
The specification requires the W3C Datetime format: YYYY-MM-DD or a full date with a time, in which case the time zone is required—2024-05-17T09:30:00+03:00. Validation flags dates in the wrong format, nonexistent dates such as April 31, and dates in the future. It also flags the case where every URL has the same date: this is usually a build timestamp and does not indicate that a page has changed.
According to the sitemaps.org specification, up to 50,000 URLs and 50 MB in uncompressed form. The tool splits the sitemap automatically: the parts are named sitemap-1.xml, sitemap-2.xml, and so on, while sitemap-index.xml is generated alongside them with links to the parts. Everything is downloaded as one archive.
Only loc is required. Google has said it does not use priority or changefreq. Keeping them does no harm, but do not rely on them: crawl frequency depends on how often the page actually changes, not on what the file says.
The file is fetched by our server, so permission for a third-party domain to be read by the browser is irrelevant: a compressed sitemap is decompressed, an index sitemap is expanded part by part, and each part is parsed as XML—including xhtml:link language links, which are lost when parsed as HTML. The tool then counts URLs, duplicates, relative URLs, Cyrillic characters in loc, third-party domains, malformed lastmod values, priorities outside the allowed range, and changefreq values not in the list. If a crawl was run first, it also adds a two-way comparison: what is missing from the sitemap and what remains in it from the previous version of the site.
No. A sitemap belongs to the site where it is hosted, and URLs from other domains are not used. A subdomain needs its own sitemap in its root; the tool warns if the list or the sitemap being checked contains mixed domains.
The URL is written to the file in the form returned by the server: the domain in punycode and the path using percent encoding. Raw Cyrillic characters are not valid in loc; the generator flags such strings both in your own sitemap and in a third-party one.
Writes the title and description and instantly shows how they appear in search results.
Open
Shows the full path from the URL to the final page and the response codes.
Open