discoveryPatterns

Patterns for additional web pages to visit for finding links.

• Type: string[]

When visiting a web page, the Crawler looks for these URLs using the micromatch library, and adds all matches to the queue. You can use negations, wildcards, and more.

This is useful when you want to visit pages that contain links to pages to extract, but you don’t want to extract records from these intermediate pages.

Examples

JavaScript

{
  discoveryPatterns: ["https://*.algolia.com/**"],
}