Skip to content

Gather

Sitemap reader

Paste the address of a site or of a sitemap. Snoei finds the sitemap, also through robots.txt, and shows every address in it.

Public pages only, one at a time

Free, up to 30 requests per 10 minutes per visitor.

House rules

How to prune politely

More about Snoei and robots.txt

  • Public only, entered by you

    Snoei only reads pages that anyone can see without logging in, and only the address you enter. It skips login pages.

  • No email addresses or phone numbers

    Snoei refuses to collect them. Links to mailto: and tel: are only counted, never shown or exported.

  • Copyright stays

    Texts, tables and images belong to whoever made them. Use them to check or to quote with a source, not to copy.

  • One page at a time

    No crawling of whole sites. Snoei reads what you give, at most checks the links or language versions of that one page, follows robots.txt and keeps the pace low.

How it works

Sitemap reader in three steps

  1. Paste a site or sitemap

    An ordinary web address is enough: Snoei looks in robots.txt and in the usual places such as /sitemap.xml.

  2. Open the index

    Large sites have a sitemap index with several sitemaps. Open one with a click; Snoei unpacks .gz files itself.

  3. Search and download

    Search the addresses and download the list as CSV, Excel or JSON.

Questions

Good to know

  • Does Snoei also read every page in the sitemap?

    No. Snoei only reads the list itself. It does not visit the pages in it: reading a sitemap is the only thing Snoei does across a whole site.

  • What if Snoei finds no sitemap?

    Then it is not in robots.txt and not at the usual addresses. Paste the address of the sitemap itself, for example from Google Search Console.

  • What do lastmod and priority mean?

    Lastmod is the last change the site reports. Changefreq and priority are hints for search engines; most search engines barely look at them any more.

Spotted a mistake?

A few words are enough. We will check and fix it.

Search

Loading…

Reading comfort

Make the site easier to read. Your choice is saved on this device.

Text size

Prefer your device’s own settings (zoom, screen reader, dark mode)? They work here as well.