How to find all pages on a website

1. Read the sitemap

Most sites publish an XML sitemap, a file that lists the pages they want search engines to find. Try these addresses on the site:

• /sitemap.xml, the most common one (every Shopify store has one here)
• /sitemap_index.xml, used by Yoast SEO and other WordPress plugins
• /wp-sitemap.xml, the one WordPress makes on its own

Many sitemaps are an index: a list of smaller sitemaps (posts, pages, products) instead of pages. Open each child sitemap to get the pages inside it.

Raw XML is slow to read, so let a tool do it. Open the sitemap, view the page source (Ctrl+U in Chrome, or Command+Option+U on a Mac), copy all of it, and paste it into my free sitemap URL extractor. It pulls out every URL, counts child sitemaps apart from pages, removes duplicates, filters by a word, and exports the list as text or CSV. It runs in your browser, so nothing is uploaded.

2. Check robots.txt for the sitemap’s address

If none of those addresses loads, open /robots.txt on the site. Sites often name their sitemap there, on a line that starts with Sitemap:. The same file lists what the site asks crawlers to skip, on Disallow: lines. To see whether a rule blocks a given page, paste the file and the page’s path into my robots.txt tester.

3. Search Google with site:

Type site:example.com into Google to list the pages it has indexed for that site, and add a path to narrow it, like site:example.com/blog. This turns up pages that never made it into the sitemap. It’s rough, though: Google shows only pages it chose to index, and the count at the top is an estimate, not a total.

4. For your own site: Search Console and your CMS

If the site is yours, you have better lists. The Pages report in Google Search Console shows which of your pages Google has indexed and which it found but left out, with the reason for each. Your CMS lists everything you’ve published: Pages and Posts in WordPress, or Products, Collections and Pages in Shopify.

Compare the two. A page in your CMS that Google hasn’t indexed deserves a look, and so does a page Google still shows that you thought you deleted.

5. Crawl the site

A crawler starts at the homepage and follows every link it finds, the way a search engine does. It catches pages the sitemap forgot, and it’s the only way to map a site that has no sitemap at all.

Desktop crawlers such as Screaming Frog do this on your own computer; its free version crawls up to 500 URLs. If you’re a developer, or you need the list inside your own app or AI tool, Firecrawl (affiliate link) does it through an API. Its Map endpoint returns a site’s URLs in one call, drawn from the sitemap, search results and pages it has crawled before. Firecrawl says Map is built for speed and can miss links, so for a complete list its Crawl endpoint follows links page by page and can return each page as clean markdown. The free plan includes 1,000 credits a month with no card needed; a Map call costs 1 credit, and a crawl costs 1 credit per page.

Whatever you use, crawl politely: follow the site’s robots.txt, go slowly, and stay out of anything behind a login.

Pages these methods can miss

• Orphan pages: pages that nothing links to and the sitemap leaves out. A crawler can’t reach them, so on your own site check Search Console or the CMS.
• Pages behind a login, such as account areas and members-only content.
• Noindex pages: pages that ask search engines not to index them won’t appear in a site: search, though a crawler can still reach them through links.
• Pages blocked in robots.txt: polite crawlers skip them, and they’re often missing from the sitemap too.

How many pages does a website have?

For a quick number, count the URLs in the sitemap; the sitemap URL extractor shows the total. Treat a site: search as a rough check only, since Google’s count is an estimate and leaves out pages it didn’t index. For an exact count of your own site, use your CMS or a full crawl.