renderJavaScript

Whether to render web pages with a headless browser and crawl the rendered HTML.

• Type: boolean | string[] | object
• Default: false

Rendering JavaScript-based web pages is slower than crawling regular HTML pages. That’s why you can apply this setting to a list URLs that match patterns (using micromatch).

The headless browser has a default timeout of 20 seconds. You can adjust it with the renderJavaScript.waitTime parameter.

Enabling this feature can lead to inconsistent records across crawls, depending on the browser load and the website speed.

Make sure your JavaScript-enabled pages load fast enough for the crawler.

Examples

JavaScript

{
  renderJavaScript: true,
}

JavaScript

{
  renderJavaScript: [
    "http://www.mysite.com/dynamic-pages/**",
  ],
}

JavaScript

{
  renderJavaScript: {
    enabled: true,
    waitTime: {
      min: 7000,
      max: 15000,
    },
    adblock: true,
    patterns: [
      "http://www.mysite.com/dynamic-pages/**",
    ],
  }
}

Parameters

adblock · boolean · default: "false"

Whether to activate the custom ad blocker. It blocks most ads and tracking scripts but may break some websites.

enabled · boolean

Whether to render pages with a headless browser and crawl the rendered HTML.

patterns · string[]

URL patterns to determine which pages to render with a headless browser. If not specified, and enabled is true, all pages are rendered.

waitTime · object

The shortest and longest wait time before considering a page as rendered.

Show child attributes

min · number · default: 0

If the page runs faster than this, the browser remains open at least this long (in milliseconds).

waitTime.max · number · default: 20000

Number of milliseconds to wait before stopping the rendering and processing the HTML as is.