To download support materials, please select your product from the list below. You will automatically be shown all downloads for that product, including extensions, brochures, sales presenters and technotes.

  • 1. Select your section, category and chosen product.
  • 2. Click on the Manual, brochure, sales presenter or technote of your choice.
  • 3. Take careful note of any applicable instructions, especially for extensions.
  • 4. Download the file to your hard drive.
Knowledge Base

Knowledge Base

Limiting crawl bots

Categories: Magento 1
Date Created: 4 May 2016
Last Updated: 30 Mar 2020

Limiting crawl bots

There are two mechanisms that can be used to further control the behaviour of both good and bad bots, to give you more granular control of how you want to treat them.

  • Bot-side crawl delay
  • Server-side rate limit

Crawl delay

The crawl-delay value is supported by some crawlers to throttle their visits to the host. Since this value is not part of the standard, its interpretation is dependent on the crawler reading it. Yandex interprets the value as the number of seconds to wait between subsequent visits, Bing uses the value as an opaque number that controls visit frequency with no quantitative specification.

In your robots.txt, specify,

User-agent: *
Crawl-delay: 10
Support Enquiry

Support Enquiry

If you have a question or require technical support for a Magespacex hosting or extensions you have purchased, please fill out the form below and a member of our support team will contact you shortly.

  • * Required Fields