exclusionPatterns

Patterns for web pages to ignore.

• Type: string[]

The crawler checks this list against page URLs using micromatch wildcard patterns.

Common wildcards are:

• * matches any characters within a single path segment (it can’t cross /).
• ** matches across path segments (it can cross /).
• ! creates an exception. It re-includes URLs that earlier patterns excluded (the crawler evaluates patterns in order).

You can define up to 100 patterns in exclusionPatterns.

In micromatch, some characters (for example, ?) have a special meaning. To match them as normal characters, prefix them with a backslash (for example, \?).

Example exclusion patterns

Exclusion patterns	Skipped by crawler	Still crawled
https://www.example.com/blog/*	https://www.example.com/blog/post-1	https://www.example.com/blog/2025/post-1
https://www.example.com/blog/**	https://www.example.com/blog/post-1 https://www.example.com/blog/2025/post-1	https://www.example.com/about/ny
https://www.example.com/** !https://www.example.com/updates/**	https://www.example.com/blog/post-1 https://www.example.com/about/ny	https://www.example.com/updates/v2/release-notes
https://www.example.com/news/team-changes	https://www.example.com/news/team-changes	https://www.example.com/news/important-announcement

JavaScript

{
  exclusionPatterns: [
    "https://www.example.com/blog/*",
    "https://www.example.com/blog/**",
    "https://www.example.com/**",
    "!https://www.example.com/updates/**",
    "https://www.example.com/news/team-changes",
  ],
}