Suggestions

How to interpret the Crawler’s suggestions for configuration improvements.

After each site crawl, the overview page provides “Suggestions” to help optimize your crawler configuration.

To access these suggestions, click View in Suggestions on the Crawler’s overview page or Suggestions in the sidebar menu.

Click Snooze to hide a suggestion for 30 days. Click Dismiss to permanently hide a suggestion.

For information about potential errors, see Error messages.

Potential suggestions

Suggestion	Solution
Automatic crawls schedule isn’t set	To ensure your data is always up-to-date, schedule automatic crawls
Robots.txt file is missing or Algolia Crawler is disallowed	The crawler encountered issues with your site’s robots.txt file. Either the file is missing, or it doesn’t allow the Algolia Crawler
Sitemap not found	Ensure efficient crawling by adding sitemaps to the crawler configuration
Some HTML pages or documents are too big	Records exceed the maximum for your Algolia plan
URLs ignored or failed	Review the crawl status
Website HTML contains <meta name="robots" content="NOINDEX,NOFOLLOW"> tag	Remove these meta tags from the pages or ignore them in the crawler configuration