MBScout — the MBase crawler
MBScout is the web crawler of the MBase project. It visits public pages of online shops and brand websites to track prices, promotions and page changes for competitive market monitoring.
Who operates it
MBScout is operated by the MBase project. Contact: bot@mbscout.com.
How to recognise it
What it fetches
- robots.txt, sitemaps, the public pages listed in them, and a few key pages such as the home page, in the site's main language.
- When a site is first added to MBase, a one-time check of its robots.txt and up to seven possible sitemap locations, at any hour.
- For product pages that show reviews from Okendo or Yotpo, the public review counts and ratings, and the text, rating and date of the newest 25 reviews, from those services.
- It never logs in, never submits forms, never uses Shopify's APIs (including product JSON and Ajax endpoints), never solves CAPTCHAs, and never changes its identity or address to get past a block.
How often and how fast
- Pages are fetched at most once a night per site, between 01:30 and 06:30 US Eastern time.
- At least 5 seconds between requests to the same site; longer if your robots.txt sets a Crawl-delay or your site responds slowly.
- A nightly budget per site that starts at 40 requests (robots.txt and sitemaps count), grows slowly while your site responds well, and halves when your site signals load.
- It stops for the night at the first refusal (403, 430, 451 or a challenge page) or after a second 429 or 503, and waits the full Retry-After before trying again. It never pays for access: a 402 is treated as a refusal.
How the data is used
- To track prices, promotions and content changes for competitive market analysis.
- Some page text is analysed by AI models to summarise messaging. MBScout's data is never used to train AI models.
- If your robots.txt opts out of AI use (
Content-Signal: ai-input=no or Content-Usage: ai-use=n), your pages are still tracked for prices and changes but are not sent to AI analysis.
- Reviewer names, email addresses and phone numbers are removed from the review data we extract.
How to block or limit it
MBScout obeys robots.txt and reads it every night; changes apply from the next night. To block it completely:
User-agent: MBScout
Disallow: /
You can also disallow specific paths, set a Crawl-delay for MBScout, or opt out of AI use as above.
Objections
Email bot@mbscout.com. We stop crawling your site within 24 hours of your message and confirm in writing what we have done with the copies we hold.