Skip to main content
Given a set of start URLs, the Crawler:
  1. Visits these pages and extracts data that’s relevant for search.
  2. Discovers other pages through links (and extracts their data).
  3. Sends your data to your Algolia indices. You can run the Crawler on a schedule to keep your Algolia indices up-to-date.

Explore related content in the Algolia Academy

Algolia Crawler

​
Why you should use the Crawler

The Crawler simplifies uploading your data to Algolia and keeping your indices up-to-date. Compared to using the API clients or other methods to index your data, using the Crawler has these benefits:
  • You don’t have to write and maintain code for extracting content, transforming it to indices, and scheduling periodic updates.
  • It helps you extract data from unstructured content (such as HTML and PDF files).
  • It can index your web pages when it’s difficult to access the sources, for example, due to restricted access, or if you want to index different resources managed by different teams using different tools.

​
Get started

The Algolia Crawler is available as an add-on to your plan. To add, verify, and crawl your site, see Create a new crawler. If you want to use the Crawler to index a technical documentation site, consider DocSearch, which also comes with a search UI. If you use Netlify to host your website, use the Netlify Crawler plugin.
⌘I