Configuration settings
Use the visual UI to change the following settings.| Setting | Description | Editor equivalent |
|---|---|---|
| Start URL | Specifies which URLs the crawler should start with. See also: Where to start crawling | startUrls |
| Disable limit/Max URLs | Limit the number of URLs the crawler can process. See also: Maximum URL limit | maxUrls |
| Crawler Schedule | Sets automatic run times for the crawler | schedule |
| URL Exclusion Rules | Excludes specific sections or types of content from the crawl. See also: Exclude and include content | exclusionPatterns |
| Record Loss Policy | Stops the crawler if it finds significantly fewer records than the previous crawl. See also: Safety checks | maxLostRecordsPercentage |
| Max Failed URLs | Stops the crawler if a specified number of pages fail to crawl. See also: Safety checks | maxFailedUrls |
| Website Authentication | Adds authentication details for password protected sites | login |
| Index Backup | Saves a copy of previously crawled content | saveBackup |