Crawl Every URL Without Guessing

Let the Sitemap Do the Work

Detect sitemap.xml and sitemap index files quickly, then turn them into crawlable collections. Building a list of URLs by hand is tedious, and missing important pages is easy. A sitemap gives you a structured source of truth that the site itself publishes. When RTILA X reads that sitemap, it can automatically build the URL list your automation needs.

The feature works with both standard sitemaps and sitemap indexes. You point RTILA X at the sitemap URL, and it parses the XML, extracts the available page URLs, and prepares them for your extraction workflow. From there, you can filter, limit, and order the list before the crawl begins.

How Sitemap Discovery Works

RTILA X reads the sitemap.xml file and extracts URLs from standard <loc> entries. If the URL points to a sitemap index, RTILA X follows the child sitemap references and combines their entries into a single crawlable list. This nested discovery happens automatically, so large sites with many sub-sitemaps are handled without extra configuration.

The discovered URLs can be used directly as inputs for a dataset workflow, a form filling routine, or a multi-page scrape. You can also keep the discovered list as a saved project resource for future runs, making it easy to re-run the same crawl later without re-discovering the site structure.

Sitemap Index Support

Sitemap indexes are common on large blogs, e-commerce stores, and enterprise sites. Instead of one enormous XML file, they split URLs across many smaller files. RTILA X recognizes this pattern and processes every child sitemap automatically.

This matters for completeness. If you only parsed the top-level file, you would miss most of the site. With index support, your dataset covers the same URLs the site actually publishes, including category pages, product detail pages, and archived content.

URL Filtering

Filtering helps you focus on the URLs that matter. You can include only paths that match a pattern and exclude paths that are not relevant. For example, a product scraper can include /products/ and exclude /blog/, /account/, or /tag/ pages.

Filters are especially useful when a site publishes many URLs that are not useful for your task. A large sitemap may contain thousands of URLs, but only a small subset may be relevant to your workflow. Including the right subset reduces runtime and makes the dataset more useful.

Combining with Pagination

Sitemap discovery and pagination work well together. You can start with a sitemap to get the initial list of category or landing pages, then let the pagination command follow the next links within each page. This approach gives you broad coverage while still handling modern page structures.

The discovered URLs become the seed list, and the automation expands from there. This is usually more reliable than trying to guess all category URLs manually, especially on sites that add new categories over time.

Use Cases

• E-commerce catalog scraping from category and product URLs
• Content marketing workflows that collect articles by section
• Job boards where each listing page follows a predictable URL pattern
• SEO audits that need a complete list of published pages
• Directory scrapers that start from a sitemap and expand through pagination

Large Sites Made Manageable

Filtering rules keep your dataset focused instead of crawling irrelevant URLs. Sitemap discovery gives you structure, and filters give you control.