ignoreNoIndex

Whether to ignore the noindex robots meta tag when extracting records.

• Type: boolean
• Default: false

By default, the Crawler doesn’t extract records from pages with the robots meta tag set to noindex or none.

If ignoreNoIndex is true, the Crawler ignores the noindex directive.

Examples

JavaScript

{
  ignoreNoIndex: true,
}