ignoreNoFollowTo

Whether to follow links with the nofollow tag.

• Type: boolean
• Default: false

By default, the Crawler doesn’t extract links from pages with the robots meta tag set to nofollow or none. It also ignores links with the rel attribute set to nofollow.

If ignoreNoFollowTo is true, the Crawler ignores any nofollow directives.

Links that don’t match any pattern in your configuration are always ignored.

Examples

JavaScript

{
  ignoreNoFollowTo: true,
}