Configure crawlers with the visual UI

Configure a crawler’s most often-used configuration settings with a visual UI.

To access the visual UI, click Configuration in the Crawler’s left-hand menu.

For advanced configuration options, use the editor.

Changes made in the visual UI are instantly replicated in the editor and vice versa. You can track all changes in the editor’s Configuration History.

Configuration settings

Use the visual UI to change the following settings.

Setting	Description	Editor equivalent
Start URL	Specifies which URLs the crawler should start with. See also: Where to start crawling	startUrls
Disable limit/Max URLs	Limit the number of URLs the crawler can process. See also: Maximum URL limit	maxUrls
Crawler Schedule	Sets automatic run times for the crawler	schedule
URL Exclusion Rules	Excludes specific sections or types of content from the crawl. See also: Exclude and include content	exclusionPatterns
Record Loss Policy	Stops the crawler if it finds significantly fewer records than the previous crawl. See also: Safety checks	maxLostRecordsPercentage
Max Failed URLs	Stops the crawler if a specified number of pages fail to crawl. See also: Safety checks	maxFailedUrls
Website Authentication	Adds authentication details for password protected sites	login
Index Backup	Saves a copy of previously crawled content	saveBackup