cache

Turn crawler’s cache on or off.

• Type: object

Caching can save bandwidth, as the crawler only crawls pages that changed since the last crawl. With caching, the crawler makes conditional requests

To do so, it relies on the ETag and Last-Modified headers returned by your server during the previous crawl. These values are included in subsequent requests as the If-None-Match and If-Modified-Since headers.

If your server responds with 304 Not Modified, the crawler doesn’t download or parse the page again. Instead, it reuses the existing records in your production index. This means your index stays up to date while minimizing unnecessary data transfers.

The crawler doesn’t send conditional requests:

• If your configuration changed since the last crawl
• If the external data associated with this page changed since the last crawl

Examples

JavaScript

{
  cache: {
    enabled: true;
  }
}

Parameters

enabled · boolean · default: true · required

Whether to turn on the cache.