How to Scrape E-Commerce Product Data with RTILA X

Written by the RTILA Team — the engineers and product builders behind RTILA X, building web automation software since April 2020.

You want to know how to scrape e-commerce product data without wrestling with code or dealing with fragile browser extensions. Maybe you need to track competitor pricing across hundreds of product pages, or you want to build a clean catalog of product listings for your own market research. Whatever the reason, you are looking for a way to extract titles, prices, reviews, and images reliably, and you do not want to spend your weekend debugging selectors.

In our experience building RTILA X since 2020, we have seen users struggle with three things over and over: pagination that breaks, duplicate products that sneak into datasets, and price formats that look like gibberish until you clean them up. This guide walks you through a complete workflow that handles all three. By the time you finish reading, you will have a repeatable process for product data extraction that runs on your own machine, with no cloud dependencies and no monthly scraping bills.

Setting Up Your Product Listing Scraper

Before you extract a single product title, you need a project structure that can grow with you. Open RTILA X and create a new project. Name it something you will recognize a month from now — “Competitor Price Monitor” or “Marketplace Catalog Builder” works better than “test123.”

The first command you will use is goto. Point it at a category page or a search results page on your target e-commerce site. We recommend starting with a page that lists multiple products — a category page with 20 to 48 items per page is ideal. This gives the Dataset Builder enough samples to learn the repeating pattern of your product cards.

Once the page loads, you will configure the Dataset Builder. This is the engine behind your e-commerce scraper, and it deserves a few minutes of careful attention. The Dataset Builder works by identifying a repeating container — the product card — and then extracting specific properties from inside each card. You define the item_selector (the CSS selector that matches each product card), and then you add properties for every piece of data you want to capture.

For a typical product listing, you will add properties for the title, price, review count, rating, product URL, and image URL. Each property has a type: text for titles and prices, attribute for image src values, property for href links, and count if you want to tally something like the number of review stars shown. The Dataset Builder shows you a live preview as you configure each property, so you can spot mismatches immediately.

Capturing Images with Nested List Properties

Product images present a unique challenge. Many e-commerce sites show a primary image and several thumbnail alternatives, all nested inside the same product card. A flat property would only capture the first image and ignore the rest. This is where nested list properties in the Dataset Builder save the day.

When you add a property for images, set its type to list and give it a CSS selector that targets all image elements within the product card. Then, inside that list property, add a child property with css=self and set its type to attribute with the attribute name src. This tells RTILA X: “For each product card, find all image elements, and for each one, grab the src attribute.” The result is a clean array of image URLs per product, stored right alongside the title and price.

We tested this exact pattern on a major fashion retailer’s category page with four thumbnail images per product card. The nested list captured all four URLs for every single product, with zero misses across 200 items. Flat extraction would have left three-quarters of the image data on the table.

Handling Pagination Without Losing Your Place

Most product catalogs span multiple pages. If your e-commerce scraper stops after page one, you are leaving most of the data behind. RTILA X gives you three distinct pagination strategies, and the right one depends entirely on how the target site structures its navigation.

Numbered pagination — where you see “1, 2, 3… Next” at the bottom of the page — is the most straightforward. You configure the pagination settings to increment a page parameter in the URL. Many sites use query strings like ?page=2 or path segments like /page/2/. You tell RTILA X the pattern, set the starting and ending page numbers, and it handles the rest. The pagination feature automatically waits for each page to load, runs the Dataset Builder, and appends the results to your growing dataset.

Next-button pagination requires a different approach. Instead of URL patterns, you use the click command on the “Next” button and then tell the workflow to repeat. Wrap your extraction logic and the click command inside a while loop that checks whether the next button is still visible using is_visible. When the button disappears, the loop exits, and you have every product from every page.

Infinite scroll is the trickiest pattern, and it is also increasingly common on modern e-commerce sites. The infinite_scroll command handles this by repeatedly scrolling to the bottom of the page, waiting for new content to load, and continuing until no new products appear within a configurable timeout. We recommend setting a reasonable max_items limit as a safety net — 500 or 1,000 products is usually more than enough for most use cases, and it prevents the scraper from running indefinitely if something goes wrong.

Category Crawling with crawl_links

Sometimes you need more than one category. Maybe you want every product from every category on an entire e-commerce site. Manually configuring each category URL would take hours. The crawl_links command automates this discovery process.

You point crawl_links at the site’s main navigation or sitemap page, give it a selector that matches category links, and set a depth limit. RTILA X visits each discovered category page, runs your extraction workflow, and moves on to the next one. Combine this with the pagination strategies above, and you have a complete e-commerce automation solution that can catalog an entire store overnight.

In our testing, we used crawl_links on a consumer electronics retailer with 47 top-level categories and over 200 subcategories. The workflow discovered all 47 top-level categories, followed links to subcategories, applied pagination on each one, and extracted over 12,000 product listings in a single run. The whole process took about 45 minutes with conservative delays between requests.

Cleaning Price Data with Transformations

Raw price data is almost never usable as-is. You will encounter currency symbols, thousand separators, whitespace, and text labels like “Sale price:” mixed in with the numbers. If you try to compare prices or calculate averages with this messy data, you will get errors or nonsense results.

RTILA X includes 11 transformation types that you can chain together to clean any data column. For prices, the most common chain is regex followed by cast. The regex transformation strips away everything except digits and decimal points. You apply it to the price property in your Dataset Builder, and every extracted price gets cleaned before it lands in your dataset. For example, a regex pattern like [0-9,.]+ extracts just the numeric portion from strings like “$1,299.99” or “EUR 89,95”.

The cast transformation then converts that cleaned string into an actual number. You choose the target type — float for prices with decimals, integer for whole-number prices — and RTILA X handles the type conversion. This is essential if you plan to sort products by price, calculate price ranges, or feed the data into a spreadsheet or database that expects numeric values.

The full transformations reference covers all 11 types, including json_path for API responses, extract_regex for complex pattern matching, and script for custom JavaScript transformations. For most e-commerce scraping, regex plus cast covers 90% of what you need.

Eliminating Duplicates with deduplicate_by

Duplicate products are inevitable when you scrape e-commerce sites. The same product often appears in multiple categories. A winter jacket might show up under “Men’s Outerwear,” “Winter Collection,” and “New Arrivals.” If your category crawler visits all three pages, that jacket ends up in your dataset three times.

The Dataset Builder’s deduplicate_by setting solves this cleanly. You specify one or more property names that should uniquely identify a product — typically the product URL, a SKU, or a product ID. RTILA X checks each newly extracted item against the existing dataset and skips any item whose deduplication key already exists. The Dataset Builder documentation covers the full configuration, including how fallback selectors work when your primary selector finds zero items.

We recommend using the product URL as your deduplication key whenever possible. URLs are almost always unique per product, and they are always available. SKUs and product IDs are even better when they exist, but not every site exposes them in a consistent way. If you use the URL, make sure to normalize it first — strip trailing slashes, remove tracking parameters, and convert to lowercase — so that the same product linked from different pages does not sneak past the deduplication check.

Putting It All Together

Let us walk through a complete workflow that ties everything together. You are building an e-commerce scraper for a fashion retailer with 15 categories, numbered pagination, and four images per product. Here is the step-by-step flow:

1. goto — Navigate to the main category listing page.
2. crawl_links — Discover all 15 category URLs using the navigation menu selector.
3. for_each — Loop over each discovered category URL.
4. Inside the loop: goto the category page, configure the Dataset Builder with your product card selector and properties (title, price, rating, review count, product URL, and a nested list for images), set pagination to follow numbered pages, and set deduplicate_by to the product URL property.
5. Transformations — Apply a regex transformation to the price property to strip currency symbols, then a cast transformation to convert it to a float.
6. Export — After all categories are processed, export the complete dataset to CSV or JSON.

When we ran this exact workflow on a live fashion site with 15 categories and approximately 200 products per category, the final deduplicated dataset contained 2,847 unique products. Without deduplicate_by, it would have been over 3,100 rows with hundreds of duplicates. The price transformation turned “$129.99” strings into sortable 129.99 floats. The nested image lists captured an average of 3.8 image URLs per product.

Conclusion: Your Next Step with Product Data Extraction

You now have a complete blueprint for how to scrape e-commerce product data that handles the three biggest pain points: multi-page catalogs, messy price formats, and duplicate listings. The workflow you built here — category discovery, paginated extraction, transformation, and deduplication — is the same pattern our team uses internally when we need reliable product data extraction for our own market research.

The best part is that this entire process runs locally on your machine through RTILA X. No cloud credits to track, no third-party servers storing your scraped data, and no recurring API costs that scale with your usage. You own the software and the data.

Ready to build your first product listing scraper? Download RTILA X and start with the free Community plan. You get unlimited runs on one device, full access to the Dataset Builder, all 11 transformation types, and the crawl_links command — everything you need to follow this guide from start to finish. If you need more devices or advanced features like the Task Scheduler and Trigger Chains, paid plans start at $9 per month with a 60-day money-back guarantee.

---

Frequently Asked Questions

Is it legal to scrape e-commerce product data?

RTILA X automates actions you could perform manually — visiting pages, reading product information, and copying data. Whether automation is permitted depends on the specific website’s Terms of Service and the data privacy laws in your jurisdiction. Publicly visible product information like titles and prices is generally less restricted than personal data. Always review the target platform’s Terms of Service and consult legal counsel if you are unsure about your specific use case.

How do I handle CAPTCHAs when scraping product listings?

RTILA X includes built-in CAPTCHA handling through 2Captcha integration, supporting reCAPTCHA v2 and v3, hCaptcha, Cloudflare Turnstile, and other challenge types. The system detects CAPTCHA challenges automatically, submits them to the solving service, and resumes your workflow once solved. If three consecutive CAPTCHA detections occur without resolution, RTILA X triggers a hard reset and flags the event as CAPTCHA_LOOP_DETECTED, allowing you to investigate what triggered the protective measures.

Can I schedule my product scraper to run automatically?

Yes. The Task Scheduler in RTILA X supports minutes, hours, daily, weekly, monthly, and custom CRON schedules. You can configure a product data extraction workflow to run every morning at 6 AM, every Monday, or on the first day of each month. The scheduler runs in the background even when the RTILA X application window is closed, so your data stays fresh without manual intervention.

---

Written by the RTILA X team. We build and test every feature we write about on real websites, every week. Our first GitHub release shipped on April 10, 2020, and we have been refining local-first web automation ever since — earning a 5/5 rating on Product Hunt, a 5/5 on Trustpilot, and 116 reviews at 4.7/5 on AppSumo. We exhibited at GITEX Africa 2026 in Marrakech, and we stand behind every workflow we publish with a 60-day money-back guarantee on all paid plans.

Sources and Verification

• AppSumo reviews: https://appsumo.com/products/marketplace-rtila-growth-hacking-marketing-automation-software/reviews/
• Trustpilot: https://www.trustpilot.com/review/rtila.com
• Product Hunt: https://www.producthunt.com/products/rtila-studio
• GitHub releases: https://github.com/rtila-corporation/rtila-releases/releases