ignoreRobotsTxtRules

Whether to ignore rules set in the robots.txt file.

• Type: boolean
• Default: false

Examples

JavaScript

{
  ignoreRobotsTxtRules: true,
}