maxUrls

Maximum number of crawled URLs.

• Type: number
• Default: 1,000

Restricting the number of URLs to crawl can be useful for demos or for preventing infinite link holes in the website structure.

maxUrls doesn’t guarantee consistent indexing between crawls. As the crawler processes URLs in parallel, discovered URLs might be processed in different orders for different crawls.

Examples

JavaScript

{
  maxUrls: 30,
}