Troubleshooting monitoring issues

Monitoring issues refer to problems that arise after a site has been crawled and its data sent to Algolia.

You might notice discrepancies in the number of records from one crawl to the next or you want to refresh your site’s content between scheduled crawls while aiming to sidestep potential complications.

For more information, see Monitoring crawlers

Difference in the number of records between crawls

Problem

A noticeable increase or decrease in the number of records between crawls may be due to:

• Temporary site problems may prevent the crawler from finding everything.

• Changes to the site. If your site changed but the crawler configuration wasn’t, the crawl might not fully capture the site content.

  For example, if your company’s job listings page moves from www.example-company.com/careers to www.work-at-example-company.com, the crawl wouldn’t include the new job listing URLs.

• Timing issues with site updates and crawls. For example, you update your site’s content at 8 AM every day, but the crawler schedule is 9 AM. If the updates aren’t completed on time, the crawl may miss the new content, and users might come across outdated information.

Solutions

Issue	Solution
Temporary site problems	Perform a manual re-crawl once the site issues are resolved
Changes to the site	Update your crawler configuration as needed and perform a re-crawl. For large changes, consider updating the safety check threshold
Timing issues	Allow ample time for updates to complete before the crawler schedule

Update between scheduled crawls

Problem

Your crawl might be scheduled to run at specific intervals, but there could be occasions when immediate updates are necessary. For example, a news site might need to publish breaking news immediately.

Solution

To add just a few pages, adjust the crawler configuration and then manually re-crawl from the Crawler dashboard.