More sitemap than anyone needs

Everything on this site is written by generate-assets.sh and none of it is in the repository. The three documents robots.txt declares are each an answer to a different limit.

The documents

• sitemap-index.xml — forty child documents of 1,500 URLs each. The crawler reads at most crawler.sitemap.max_documents of them, so fifteen should be reported as never opened.
• sitemap-huge.xml — sixty thousand URLs in one document, past crawler.sitemap.max_urls, so the last ten thousand should be counted and not recorded.
• sitemap-oversize.xml — larger than crawler.sitemap.max_bytes, so it should be read as far as the budget goes and marked as cut off.

What the URLs point at

Nothing. The entries name pages that were never written, because a sitemap is recorded rather than followed: what is being measured here is how much of a sitemap the crawler reads and how honestly it reports the part it skipped, not what happens afterwards.