Competitor Price Extraction: Automate Price Monitoring Across 500 Sites

You run an auto parts e-commerce store. Twenty competitors carry your 50 core references with prices that shift every week. Friday evening you open twenty Chrome tabs, copy prices into a Google Sheet, and compare. Monday morning, three competitors have already adjusted. You do it again on Friday. Eight hours a week swallowed by a task that could run on its own.

Automated price monitoring is neither illegal nor a technique reserved for large corporations. A module that visits each product page, extracts the displayed price, and stores it in a historical database can detect in minutes what a human would take half a day to notice: who lowered, who raised, who went out of stock, who launched a flash promotion.

Why price monitoring is legal in France

Prices displayed publicly on an e-commerce website are data accessible without authentication, published by the site operator for any visitor to see. Collecting them via an automated tool is not an infringement. The DGCCRF (Direction Générale de la Concurrence, de la Consommation et de la Répression des Fraudes — France's competition and consumer affairs authority) actively encourages pricing transparency for the benefit of consumers.

Two limits to keep in mind:

• Target site's terms of service. Some ToS prohibit scraping. Complying with ToS is a contractual obligation, not a statutory one, but violating them can lead to IP blocking and civil disputes. The tolerance threshold depends on the volume and aggressiveness of the crawl.
• Respectful server load. Visiting 500 pages at 10 requests per second resembles a light attack. Spacing out requests (5–30 seconds between two pages on the same site), respecting the robots.txt file, and properly identifying your user agent — these technical best practices prevent blocks.

For targeted competitive intelligence covering a few dozen to a few hundred pages, these two constraints pose no practical issue. For large-scale marketplace scraping, the complexity increases significantly.

The 4 technical difficulty levels of a product page

Not all product pages are equal. Price extraction must be calibrated according to the type of site:

Site type	Difficulty	Specifics
WordPress / WooCommerce store	★ Easy	Price in stable HTML tags (class woocommerce-Price-amount). Direct extraction.
Shopify store	★ Easy	Standard templates, JSON-LD Product with systematic offers.price.
PrestaShop / Magento / custom sites	★★ Medium	Variable structure depending on the merchant's theme. Price sometimes loaded via JavaScript after the initial HTML.
Marketplaces (Amazon, Cdiscount, Leroy Merlin)	★★★ Hard	Aggressive anti-bot measures, JavaScript-heavy, dynamic prices (varying by location and visitor history). Requires a real browser and rotating proxies.

For standard B2B competitive intelligence (direct competitors on e-commerce or catalog sites), 80% of cases fall into levels 1–2. Marketplaces are a project of their own.

Method for building your own price monitoring

1. List the URLs to monitor. One URL per product per competitor. If you track 50 products across 20 competitors = 1,000 URLs. Building this list takes half a day if it doesn't already exist.
2. Identify the CSS selector for the price on each competitor's site. Right-click on the price → Inspect → locate the tag and class. Do this 20 times — once per competitor.
3. Set up the crawl. A cron job that visits each URL once a week (or more, depending on the stakes), extracts the price, compares it to the previous one, and stores the history.
4. Alert on variations. Email or Slack notification when a price moves by more than X% or drops below a threshold.
5. Dashboard. Monthly view of price curves per product, per competitor, with mean and standard deviation.

Step 2 (identifying selectors) is the most time-consuming: 20 competitors = 20 separate calibrations. Once done, the infrastructure runs with almost no intervention.

What outsend's pricing module does

From a list of product URLs, the module visits each page, extracts the displayed price, and returns a structured table. Three operational characteristics:

• Automatic price detection. The module recognizes French-format prices (12,90 €, 12.90 EUR, 12€90, 12,90 €/HT, 12.90 € TTC) across most standard e-commerce CMS platforms (Shopify, WooCommerce, PrestaShop, Magento, and custom sites with consistent structure).
• Contextual metadata. In addition to the price, the module extracts when available: promotional labels (-20%, sale), the crossed-out comparison price, availability status (in stock / out of stock / on order), and the last-updated date if exposed by the site.
• Preserved history. Each extraction is time-stamped and stored. Over 12 weeks of tracking, you can plot the price curve of each product at each competitor and identify patterns (who adjusts monthly, who runs calendar promotions, who never moves).

Standard CSV/XLSX export. No direct ERP integration in alpha — export is the reference format, to be reloaded into your analysis tool (Excel, Google Sheets, Power BI, Metabase).

Use cases that change the game

Dynamic e-commerce repricing. An auto parts merchant tracking 200 references across 15 competitors adjusts prices every Thursday morning based on the latest crawl. A well-run monitoring setup typically recovers 3–7% in margin on references where a competitor drops prices to "steal" visibility in comparison engines.

Supplier negotiation intelligence. You want to renegotiate your purchasing terms. You extract public prices from 30 distributors across 50 references. You walk into the meeting with a factual benchmark, not with "I've heard that…".

Competitor promotion detection. Weekly alerts when competitors launch sales or flash deals. Improved commercial responsiveness — you no longer discover a competitor's promotion by accident when a customer sends it to you.

Pre-launch market study. Before launching a product, mapping actual market prices prevents off-target positioning. A founder claiming "20% cheaper than the competition" without having scraped actual prices often discovers they are 5% more expensive than the cheapest option on the market.

What price monitoring does not do

To set expectations clearly: a price extraction module does not detect hidden discounts (promo codes, contractual rebates, negotiated B2B prices that are not publicly displayed). It extracts what is public.

It does not predict future competitor adjustments — for that, a historical analysis model with seasonality would be required (not included in outsend's current pricing module; that is a post-export task to be done in your own analysis tool).

It does not replace the field knowledge of a sales rep who calls customers to understand why a competitor is lowering prices (clearance, upstream supply issues, strategic repositioning). The number tells you "what"; the human explains "why".

FAQ

How many product pages can be monitored in outsend alpha?

No technical cap in alpha. The practical limit comes from competitors that block overly aggressive IPs — 500–2,000 pages per session with a few seconds between requests gets through everywhere; beyond that, rotating proxies and a site-specific pace are needed.

How do you handle a site that only shows prices after login?

outsend's pricing module in alpha does not include logging into competitor user accounts — that is a legally grey area (almost certain ToS violation). For prices reserved for logged-in professionals, monitoring must go through a sales rep who legitimately creates an account and exports manually.

What extraction frequency is reasonable?

Weekly is sufficient for 90% of standard e-commerce monitoring. Daily is only relevant during sales seasons or for hyper-volatile products (energy, raw materials, event ticketing).

Are prices captured including or excluding VAT, and how does the module distinguish them?

The module captures the displayed price as-is and records the label if it is explicit (incl. VAT / excl. VAT). At export, you filter according to your needs. For comparing B2B sites (excl. VAT) with B2C sites (incl. VAT), downstream reprocessing is required.

Does extraction work on foreign sites (Amazon.de, Cdiscount.it, etc.)?

Technically yes, but international marketplaces activate aggressive anti-bot measures that significantly degrade the success rate. For multi-country monitoring, plan for a setup with geo-located proxies and a very spread-out crawl pace.

How does outsend handle product pages with variants (size, color)?

The module extracts the default price displayed when the page first loads. For variants, you need one URL per variant (often a distinct URL in WooCommerce, a query string in Shopify). The module does not automatically explore the variant selector — it is up to the input URL list to differentiate them.

To see this feature in context, browse all OutSend features.

outsend.xyz — B2B scraping and prospecting platform, currently in alpha. Free application at /demander-acces.