Skip to main content
  • Type: object
The Crawler only accepts headers from the list of allowed headers. The Cookie header is replaced by the cookie fetched in login.

​
Examples

JavaScript
{
  requestOptions: {
    proxy: 'http://algolia:password@1.1.1.1:8080',
    timeout: 10000,
    retries: 2,
    headers: {
      Authorization: 'Bearer Aerehdf==',
      Cookie: 'session=1234;',
    }
  }
}

​
Parameters

​
proxy
string
URL of a proxy server for all crawler requests.
​
headers
object
Headers to add to all requests.Supported headers are:
  • Accept-Language
  • Authorization
  • Cookie
​
retries
number
default:2
Maximum number of retries for all crawler requests.
​
timeout
number
default:30000
Maximum time in milliseconds before canceling the request.