After each site crawl, the overview page provides “Suggestions” to help optimize your crawler configuration.
To access these suggestions, click View in Suggestions on the Crawler’s overview page or Suggestions in the sidebar menu.
Click Snooze to hide a suggestion for 30 days.
Click Dismiss to permanently hide a suggestion.
For information about potential errors,
see Error messages.
Potential suggestions
| Suggestion | Solution |
|---|
| Automatic crawls schedule isn’t set | To ensure your data is always up-to-date, schedule automatic crawls |
| Robots.txt file is missing or Algolia Crawler is disallowed | The crawler encountered issues with your site’s robots.txt file. Either the file is missing, or it doesn’t allow the Algolia Crawler |
| Sitemap not found | Ensure efficient crawling by adding sitemaps to the crawler configuration |
| Some HTML pages or documents are too big | Records exceed the maximum for your Algolia plan |
| URLs ignored or failed | Review the crawl status |
Website HTML contains <meta name="robots" content="NOINDEX,NOFOLLOW"> tag | Remove these meta tags from the pages or ignore them in the crawler configuration |
Last modified on February 10, 2026
