You currently have no notifications!
Close Menu

How can we help you?

Tell us more about your query, we'd be happy to help.
How did you hear about us?:
IRP Shopper
Common Tasks (1)
Interventions (1)
Languages (1)
Questions & Answers (1)
Reviews (1)
Site Scripts (1)
Subscriptions (1)
Web Services (1)
IRP Freshdesk
Training Videos
Peer Support

Potential Crawlers

You can find this section under Security > Potential Crawlers in the IRP Admin left navigation menu.

The PotentialCrawlers.aspx page lists any IP addresses that may be Crawlers according to the values of the Application Settings that are described below in the section called Crawler Application Settings. The main grid shows all of the IP addresses that have gone over the value of the 'Crawler Page Views Per Minute Threshold' Application Setting:

Potential Crawlers Page

From here, you can either ignore the IP addresses, delete them or add as them as Crawlers (and also exclude them from the Insights functionality).

All of the sessions belonging to the IP address that was flagged as a Potential Crawler will be visible on the PotentialCrawlerActivity.aspx page. To view this page, click the View Activity button as shown in the screen capture above:

Potential Crawlers Activity Page

You can then click the Page Views button for each session to give insight into whether or not the Potential Crawler is in fact a Crawler:

Potential Crawlers Page Views

Crawler Application Settings

The Application Setting 'Crawler Page Views Per Minute Threshold' (in the 'Security' section) considers any customer who views this many pages or more per minute as a Potential Crawler. This value is 60 by default. For example, a value of 60 means 60 page views per minute or one view every second. Ensure that this value is high enough to avoid false positives, such as customers opening several tabs at once. Any customers viewing the site at a rate equal to or faster than the provided value will either be flagged as a Potential Crawler or automatically added as a Crawler depending on the value of the Application Setting called 'Automatically Exclude Suspected Crawlers From Insights' (also in the 'Security' section).

Note that you can also use the 'Automatically Exclude Potential Crawlers' button at the top of the PotentialCrawlers.aspx page to fulfil the same function as the 'Automatically Exclude Suspected Crawlers From Insights' Application Setting. You can also use the 'Check For Potential Crawlers' button at the top of the PotentialCrawlers.aspx page to perform a check right away.

When the 'Automatically Exclude Suspected Crawlers From Insights' Application Setting is enabled, any IP addresses that are detected viewing pages above the provided "Crawler Page Views Per Minute Threshold" Application Setting value will automatically be added as Crawlers and excluded from Insights without having to be reviewed on the PotentialCrawlers.aspx page. When disabled, any suspected crawlers will be listed on the PotentialCrawlers.aspx page and will need to be reviewed and manually added to the list of Crawlers.

Live Market Data
Today v Yesterday 0.11%
July 2024 v July 2023 3.89%
YTD 2024 v 2023 14.41%
Arts and Crafts 0.07%
Baby & Child 0.09%
Cars and Motorcycling 0.43%
Electrical & Commercial Equipment 0.14%
Fashion Clothing & Accessories 0.13%
Food & Drink 0.26%
Health and Wellbeing 0.07%
Kitchen & Home Appliances 0.11%
Pet Care 0.05%
Sports and Recreation 0.04%
Toys, Games & Collectables 0.00%

Copyright © 2023 IRP Commerce. Use of this website constitutes acceptance of the IRP World Acceptable Use Policy, IRP World Terms of Use, IRP Privacy Policy and IRP Cookie Policy

IRP Commerce Limited, Concourse 3, Catalyst, BT3 9DT, UK. Company Number: NI 041856. VAT Number: GB 888249658
A Deloitte Fast 50 Company eight times: 2010, 2011, 2012, 2013, 2014, 2018, 2019 & 2020