Skip to main content
  • Type: object
Caching can save bandwidth, as the crawler only crawls pages that changed since the last crawl. With caching, the crawler makes conditional requests To do so, it relies on the ETag and Last-Modified headers returned by your server during the previous crawl. These values are included in subsequent requests as the If-None-Match and If-Modified-Since headers. If your server responds with 304 Not Modified, the crawler doesn’t download or parse the page again. Instead, it reuses the existing records in your production index. This means your index stays up to date while minimizing unnecessary data transfers. The crawler doesn’t send conditional requests:
  • If your configuration changed since the last crawl
  • If the external data associated with this page changed since the last crawl

​
Examples

JavaScript
{
  cache: {
    enabled: true;
  }
}

​
Parameters

​
enabled
boolean
default:true
required
Whether to turn on the cache.