Retrieve a configuration version

Retrieves the specified version of the crawler configuration.

You can use this to restore a previous version of the configuration.

GET /1/crawlers/{id}/config/versions/{version}

Retrieve a configuration version

cURL

curl --request GET \
  --url https://crawler.algolia.com/api/1/crawlers/{id}/config/versions/{version} \
  --header 'Authorization: Basic <encoded-value>'

200

{
  "version": 2,
  "config": {
    "actions": [
      {
        "indexName": "algolia_website",
        "recordExtractor": {
          "__type": "function",
          "source": "<string>"
        },
        "autoGenerateObjectIDs": true,
        "cache": {
          "enabled": true
        },
        "discoveryPatterns": [
          "https://www.algolia.com/**"
        ],
        "fileTypesToMatch": [
          "html",
          "pdf"
        ],
        "hostnameAliases": {
          "dev.example.com": "example.com"
        },
        "name": "<string>",
        "pathAliases": {
          "example.com": {
            "/foo": "/bar"
          }
        },
        "pathsToMatch": [
          "https://www.algolia.com/**"
        ],
        "schedule": "<string>",
        "selectorsToMatch": [
          ".products",
          "!.featured"
        ]
      }
    ],
    "appId": "<string>",
    "rateLimit": 4,
    "apiKey": "<string>",
    "exclusionPatterns": [
      "https://www.example.com/excluded",
      "!https://www.example.com/this-one-url",
      "https://www.example.com/exclude/**"
    ],
    "externalData": [
      "testCSV"
    ],
    "extraUrls": [
      "<string>"
    ],
    "ignoreCanonicalTo": true,
    "ignoreNoFollowTo": true,
    "ignoreNoIndex": true,
    "ignorePaginationAttributes": true,
    "ignoreQueryParams": [
      "ref",
      "utm_*"
    ],
    "ignoreRobotsTxtRules": true,
    "indexPrefix": "crawler_",
    "initialIndexSettings": {},
    "linkExtractor": {
      "__type": "function",
      "source": "({ $, url, defaultExtractor }) => {\n  if (/example.com\\/doc\\//.test(url.href)) {\n    // For all pages under `/doc`, only extract the first found URL.\n    return defaultExtractor().slice(0, 1)\n  }\n  // For all other pages, use the default.\n  return defaultExtractor()\n}\n"
    },
    "login": {
      "url": "https://example.com/secure/login-with-post",
      "requestOptions": {
        "method": "POST",
        "headers": {
          "Content-Type": "application/x-www-form-urlencoded"
        },
        "body": "id=my-id&password=my-password",
        "timeout": 5000
      }
    },
    "maxDepth": 5,
    "maxUrls": 250,
    "renderJavaScript": true,
    "requestOptions": {
      "proxy": "<string>",
      "timeout": 30000,
      "retries": 3,
      "headers": {
        "Accept-Language": "fr-FR",
        "Authorization": "Bearer Aerehdf==",
        "Cookie": "session=1234"
      }
    },
    "safetyChecks": {
      "beforeIndexPublishing": {
        "maxLostRecordsPercentage": 10,
        "maxFailedUrls": 123
      }
    },
    "saveBackup": true,
    "schedule": "every weekday at 12:00 pm",
    "sitemaps": [
      "https://example.com/sitemap.xyz"
    ],
    "startUrls": [
      "https://www.example.com"
    ]
  },
  "createdAt": "2023-07-04T12:49:15Z",
  "authorId": "7d79f0dd-2dab-4296-8098-957a1fdc0637"
}

Required ACL: settings

Authorizations

Authorization · string · header · required

Basic authentication header of the form Basic <encoded-value>, where <encoded-value> is the base64-encoded string username:password.

Path Parameters

id · string · required

Crawler ID. Universally unique identifier (UUID) of the crawler.

Example:

"e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"

version · integer · required

This crawler's version nmber.

Response

200

application/json

OK

version · integer · required

Version of the configuration. Version 1 is the initial configuration you used when creating the crawler.

Required range: x >= 1

config · object · required

Crawler configuration.

Show child attributes

config. actions · object[] · required

A list of actions.

Required array length: 1 - 30 elements

Show child attributes

config.actions. indexName · string · required

Reference to the index used to store the action's extracted records. indexName is combined with the prefix you specified in indexPrefix.

Maximum string length: 256

Example:

"algolia_website"

config.actions. recordExtractor · recordExtractor · object · required

Function for extracting information from a crawled page and transforming it into Algolia records for indexing.

The Crawler has an editor with autocomplete and validation to help you update the recordExtractor. For details, see the recordExtractor documentation.

Show child attributes

config.actions.recordExtractor. __type · enum<string>

Available options: function

config.actions.recordExtractor. source · string

A JavaScript function (as a string) that returns one or more Algolia records for each crawled page.

config.actions. autoGenerateObjectIDs · boolean · default: true

Whether to generate an objectID for records that don't have one.

config.actions. cache · object

Whether the crawler should cache crawled pages.

For more information, see Partial crawls with caching.

Show child attributes

config.actions.cache. enabled · boolean · default: true

Whether the crawler cache is active.

config.actions. discoveryPatterns · string[]

Which intermediary web pages the crawler should visit. Use discoveryPatterns to define pages that should be visited just for their links to other pages, not their content. It functions similarly to the pathsToMatch action but without record extraction.

Use micromatch for negation, wildcards, and more.

config.actions. fileTypesToMatch · enum<string>[]

File types for crawling non-HTML documents.

Maximum array length: 100

For more information, see Extract data from non-HTML documents.

Available options: doc, email, html, odp, ods, odt, pdf, ppt, xls

Example:

["html", "pdf"]

config.actions. hostnameAliases · object

Key-value pairs to replace matching hostnames found in a sitemap, on a page, in canonical links, or redirects.

During a crawl, this action maps one hostname to another whenever the crawler encounters specific URLs. This helps with links to staging environments (like dev.example.com) or external hosting services (such as YouTube).

For example, with this hostnameAliases mapping:

{
hostnameAliases: {
    'dev.example.com': 'example.com'
}
}

1. The crawler encounters https://dev.example.com/solutions/voice-search/.

2. hostnameAliases transforms the URL to https://example.com/solutions/voice-search/.

3. The crawler follows the transformed URL (not the original).

hostnameAliases only changes URLs, not page text. In the preceding example, if the extracted text contains the string dev.example.com, it remains unchanged.

The crawler can discover URLs in places such as:

• Crawled pages

• Sitemaps

• Canonical URLs

• Redirects.

However, hostnameAliases doesn't transform URLs you explicitly set in the startUrls or sitemaps parameters, nor does it affect the pathsToMatch action or other configuration elements.

Show child attributes

config.actions.hostnameAliases. {key} · string

Hostname that should be added in the records.

Example:

{ "dev.example.com": "example.com" }

config.actions. name · string

Unique identifier for the action. This option is required if schedule is set.

config.actions. pathAliases · object

Key-value pairs to replace matching paths with new values.

It doesn't replace:

• URLs in the startUrls, sitemaps, pathsToMatch, and other settings.
• Paths found in extracted text.

The crawl continues from the transformed URLs.

For example, if you create a mapping for { "dev.example.com": { '/foo': '/bar' } } and the crawler encounters https://dev.example.com/foo/hello/, it’s transformed to https://dev.example.com/bar/hello/.

Compare with the hostnameAliases action.

Show child attributes

config.actions.pathAliases. {key} · object

Hostname for which matching paths should be replaced.

Show child attributes

config.actions.pathAliases.{key}. {key} · string

Key-value pair of a path that should be replaced.

Example:

{ "example.com": { "/foo": "/bar" } }

config.actions. pathsToMatch · string[]

URLs to which this action should apply.

Uses micromatch for negation, wildcards, and more.

Required array length: 1 - 100 elements

Use micromatch for negation, wildcards, and more.

config.actions. schedule · string

How often to perform a complete crawl for this action.

For mopre information, consult the schedule parameter documentation.

config.actions. selectorsToMatch · string[]

DOM selectors for nodes that must be present on the page to be processed. If the page doesn't match any of the selectors, it's ignored.

Maximum array length: 100

Prefix a selector with ! to ignore matching pages.

Example:

[".products", "!.featured"]

config. appId · string · required

Algolia application ID where the crawler creates and updates indices.

config. rateLimit · integer · required

Determines the number of concurrent tasks per second that can run for this configuration.

A higher rate limit means more crawls per second. Algolia prevents system overload by ensuring the number of URLs added in the last second and the number of URLs being processed is less than the rate limit:

max(new_urls_added, active_urls_processing) <= rateLimit

Start with a low value (for example, 2) and increase it if you need faster crawling. Be aware that a high rateLimit can have a huge impact on bandwidth cost and server resource consumption.

The number of pages processed per second depends on the average time it takes to fetch, process, and upload a URL. For a given rateLimit if fetching, processing, and uploading URLs takes (on average):

• Less than a second, your crawler processes up to rateLimit pages per second.
• Four seconds, your crawler processes up to rateLimit / 4 pages per second.

In the latter case, increasing rateLimit improves performance, up to a point. However, if the processing time remains at four seconds, increasing rateLimit won't increase the number of pages processed per second.

Required range: 1 <= x <= 100

Example:

4

config. apiKey · string

The Algolia API key the crawler uses for indexing records. If you don't provide an API key, one will be generated by the Crawler when you create a configuration.

The API key must have:

• These rights and restrictions: search, addObject, deleteObject, deleteIndex, settings, editSettings, listIndexes, browse
• Access to the correct set of indices, based on the crawler's indexPrefix. For example, if the prefix is crawler_, the API key must have access to crawler_*.

Don't use your Admin API key.

config. exclusionPatterns · string[]

URLs to exclude from crawling.

Maximum array length: 100

Use micromatch for negation, wildcards, and more.

Example:

[
  "https://www.example.com/excluded",
  "!https://www.example.com/this-one-url",
  "https://www.example.com/exclude/**"
]

config. externalData · string[]

References to external data sources for enriching the extracted records.

Maximum array length: 10

For more information, see Enrich extracted records with external data.

config. extraUrls · string[]

The Crawler treats extraUrls the same as startUrls. Specify extraUrls if you want to differentiate between URLs you manually added to fix site crawling from those you initially specified in startUrls.

Maximum array length: 9999

config. ignoreCanonicalTo

Determines if the crawler should extract records from a page with a canonical URL.

If ignoreCanonicalTo is set to:

• true all canonical URLs are ignored.
• One or more URL patterns, the crawler will ignore the canonical URL if it matches a pattern.

config. ignoreNoFollowTo · boolean

Determines if the crawler should follow links with a nofollow directive. If true, the crawler will ignore the nofollow directive and crawl links on the page.

The crawler always ignores links that don't match your configuration settings. ignoreNoFollowTo applies to:

• Links that are ignored because the robots meta tag contains nofollow or none.
• Links with a rel attribute containing the nofollow directive.

config. ignoreNoIndex · boolean

Whether to ignore the noindex robots meta tag. If true, pages with this meta tag will be crawled.

config. ignorePaginationAttributes · boolean · default: true

Whether the crawler should follow rel="prev" and rel="next" pagination links in the <head> section of an HTML page.

• If true, the crawler ignores the pagination links.
• If false, the crawler follows the pagination links.

config. ignoreQueryParams · string[]

Query parameters to ignore while crawling.

All URLs with the matching query parameters are treated as identical. This prevents indexing URLs that just differ by their query parameters.

Maximum array length: 9999

Use wildcards to match multiple query parameters.

Example:

["ref", "utm_*"]

config. ignoreRobotsTxtRules · boolean

Whether to ignore rules defined in your robots.txt file.

config. indexPrefix · string

A prefix for all indices created by this crawler. It's combined with the indexName for each action to form the complete index name.

Maximum string length: 64

Example:

"crawler_"

config. initialIndexSettings · object

Crawler index settings.

These index settings are only applied during the first crawl of an index.

Any subsequent changes won't be applied to the index. Instead, make changes to your index settings in the Algolia dashboard.

Show child attributes

config.initialIndexSettings. {key} · object

Index settings.

Show child attributes

config.initialIndexSettings.{key}. attributesForFaceting · string[]

Attributes used for faceting.

Facets are attributes that let you categorize search results. They can be used for filtering search results. By default, no attribute is used for faceting. Attribute names are case-sensitive.

Modifiers

• filterOnly("ATTRIBUTE"). Allows the attribute to be used as a filter but doesn't evaluate the facet values.

• searchable("ATTRIBUTE"). Allows searching for facet values.

• afterDistinct("ATTRIBUTE"). Evaluates the facet count after deduplication with distinct. This ensures accurate facet counts. You can apply this modifier to searchable facets: afterDistinct(searchable(ATTRIBUTE)).

Example:

[
  "author",
  "filterOnly(isbn)",
  "searchable(edition)",
  "afterDistinct(category)",
  "afterDistinct(searchable(publisher))"
]

config.initialIndexSettings.{key}. replicas · string[]

Creates replica indices.

Replicas are copies of a primary index with the same records but different settings, synonyms, or rules. If you want to offer a different ranking or sorting of your search results, you'll use replica indices. All index operations on a primary index are automatically forwarded to its replicas. To add a replica index, you must provide the complete set of replicas to this parameter. If you omit a replica from this list, the replica turns into a regular, standalone index that will no longer be synced with the primary index.

Modifier

• virtual("REPLICA"). Create a virtual replica, Virtual replicas don't increase the number of records and are optimized for Relevant sorting.

Example:

[
  "virtual(prod_products_price_asc)",
  "dev_products_replica"
]

config.initialIndexSettings.{key}. paginationLimitedTo · integer · default: 1000

Maximum number of search results that can be obtained through pagination.

Higher pagination limits might slow down your search. For pagination limits above 1,000, the sorting of results beyond the 1,000th hit can't be guaranteed.

Required range: x <= 20000

Example:

100

config.initialIndexSettings.{key}. unretrievableAttributes · string[]

Attributes that can't be retrieved at query time.

This can be useful if you want to use an attribute for ranking or to restrict access, but don't want to include it in the search results. Attribute names are case-sensitive.

Example:

["total_sales"]

config.initialIndexSettings.{key}. disableTypoToleranceOnWords · string[]

Creates a list of words which require exact matches. This also turns off word splitting and concatenation for the specified words.

Example:

["wheel", "1X2BCD"]

config.initialIndexSettings.{key}. attributesToTransliterate · string[]

Attributes, for which you want to support Japanese transliteration.

Transliteration supports searching in any of the Japanese writing systems. To support transliteration, you must set the indexing language to Japanese. Attribute names are case-sensitive.

Example:

["name", "description"]

config.initialIndexSettings.{key}. camelCaseAttributes · string[]

Attributes for which to split camel case words. Attribute names are case-sensitive.

Example:

["description"]

config.initialIndexSettings.{key}. decompoundedAttributes · object

Searchable attributes to which Algolia should apply word segmentation (decompounding). Attribute names are case-sensitive.

Compound words are formed by combining two or more individual words, and are particularly prevalent in Germanic languages—for example, "firefighter". With decompounding, the individual components are indexed separately.

You can specify different lists for different languages. Decompounding is supported for these languages: Dutch (nl), German (de), Finnish (fi), Danish (da), Swedish (sv), and Norwegian (no). Decompounding doesn't work for words with non-spacing mark Unicode characters. For example, Gartenstühle won't be decompounded if the ü consists of u (U+0075) and ◌̈ (U+0308).

Example:

{ "de": ["name"] }

config.initialIndexSettings.{key}. indexLanguages · enum<string>[]

Languages for language-specific processing steps, such as word detection and dictionary settings.

You should always specify an indexing language. If you don't specify an indexing language, the search engine uses all supported languages, or the languages you specified with the ignorePlurals or removeStopWords parameters. This can lead to unexpected search results. For more information, see Language-specific configuration.

ISO code for a supported language.

Available options: af, ar, az, bg, bn, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fo, fr, ga, gl, he, hi, hu, hy, id, is, it, ja, ka, kk, ko, ku, ky, lt, lv, mi, mn, mr, ms, mt, nb, nl, no, ns, pl, ps, pt, pt-br, qu, ro, ru, sk, sq, sv, sw, ta, te, th, tl, tn, tr, tt, uk, ur, uz, zh

Example:

["ja"]

config.initialIndexSettings.{key}. disablePrefixOnAttributes · string[]

Searchable attributes for which you want to turn off prefix matching. Attribute names are case-sensitive.

Example:

["sku"]

config.initialIndexSettings.{key}. allowCompressionOfIntegerArray · boolean · default: false

Whether arrays with exclusively non-negative integers should be compressed for better performance. If true, the compressed arrays may be reordered.

config.initialIndexSettings.{key}. numericAttributesForFiltering · string[]

Numeric attributes that can be used as numerical filters. Attribute names are case-sensitive.

By default, all numeric attributes are available as numerical filters. For faster indexing, reduce the number of numeric attributes.

To turn off filtering for all numeric attributes, specify an attribute that doesn't exist in your index, such as NO_NUMERIC_FILTERING.

Modifier

• equalOnly("ATTRIBUTE"). Support only filtering based on equality comparisons = and !=.

Example:

["equalOnly(quantity)", "popularity"]

config.initialIndexSettings.{key}. separatorsToIndex · string · default: ""

Control which non-alphanumeric characters are indexed.

By default, Algolia ignores non-alphanumeric characters like hyphen (-), plus (+), and parentheses ((,)). To include such characters, define them with separatorsToIndex.

Separators are all non-letter characters except spaces and currency characters, such as $€£¥.

With separatorsToIndex, Algolia treats separator characters as separate words. For example, in a search for "Disney+", Algolia considers "Disney" and "+" as two separate words.

Example:

"+#"

config.initialIndexSettings.{key}. searchableAttributes · string[]

Attributes used for searching. Attribute names are case-sensitive.

By default, all attributes are searchable and the Attribute ranking criterion is turned off. With a non-empty list, Algolia only returns results with matches in the selected attributes. In addition, the Attribute ranking criterion is turned on: matches in attributes that are higher in the list of searchableAttributes rank first. To make matches in two attributes rank equally, include them in a comma-separated string, such as "title,alternate_title". Attributes with the same priority are always unordered.

For more information, see Searchable attributes.

Modifier

• unordered("ATTRIBUTE"). Ignore the position of a match within the attribute.

Without a modifier, matches at the beginning of an attribute rank higher than matches at the end.

Example:

[
  "title,alternative_title",
  "author",
  "unordered(text)",
  "emails.personal"
]

config.initialIndexSettings.{key}. userData · any · default: {}

An object with custom data.

You can store up to 32kB as custom data.

Example:

{
  "settingID": "f2a7b51e3503acc6a39b3784ffb84300",
  "pluginVersion": "1.6.0"
}

config.initialIndexSettings.{key}. customNormalization · object

Characters and their normalized replacements. This overrides Algolia's default normalization.

Show child attributes

config.initialIndexSettings.{key}.customNormalization. {key} · object

Show child attributes

config.initialIndexSettings.{key}.customNormalization.{key}. {key} · string

Example:

{ "default": { "ä": "ae", "ü": "ue" } }

config.initialIndexSettings.{key}. attributeForDistinct · string

Attribute that should be used to establish groups of results. Attribute names are case-sensitive.

All records with the same value for this attribute are considered a group. You can combine attributeForDistinct with the distinct search parameter to control how many items per group are included in the search results.

If you want to use the same attribute also for faceting, use the afterDistinct modifier of the attributesForFaceting setting. This applies faceting after deduplication, which will result in accurate facet counts.

Example:

"url"

config.initialIndexSettings.{key}. maxFacetHits · integer · default: 10

Maximum number of facet values to return when searching for facet values.

Required range: x <= 100

config.initialIndexSettings.{key}. keepDiacriticsOnCharacters · string · default: ""

Characters for which diacritics should be preserved.

By default, Algolia removes diacritics from letters. For example, é becomes e. If this causes issues in your search, you can specify characters that should keep their diacritics.

Example:

"øé"

config.initialIndexSettings.{key}. customRanking · string[]

Attributes to use as custom ranking. Attribute names are case-sensitive.

The custom ranking attributes decide which items are shown first if the other ranking criteria are equal.

Records with missing values for your selected custom ranking attributes are always sorted last. Boolean attributes are sorted based on their alphabetical order.

Modifiers

• asc("ATTRIBUTE"). Sort the index by the values of an attribute, in ascending order.

• desc("ATTRIBUTE"). Sort the index by the values of an attribute, in descending order.

If you use two or more custom ranking attributes, reduce the precision of your first attributes, or the other attributes will never be applied.

Example:

["desc(popularity)", "asc(price)"]

config.initialIndexSettings.{key}. attributesToRetrieve · string[]

Attributes to include in the API response To reduce the size of your response, you can retrieve only some of the attributes. Attribute names are case-sensitive

• * retrieves all attributes, except attributes included in the customRanking and unretrievableAttributes settings.
• To retrieve all attributes except a specific one, prefix the attribute with a dash and combine it with the *: ["*", "-ATTRIBUTE"].
• The objectID attribute is always included.

Example:

["author", "title", "content"]

config.initialIndexSettings.{key}. ranking · string[]

Determines the order in which Algolia returns your results.

By default, each entry corresponds to a ranking criteria. The tie-breaking algorithm sequentially applies each criterion in the order they're specified. If you configure a replica index for sorting by an attribute, you put the sorting attribute at the top of the list.

Modifiers

• asc("ATTRIBUTE"). Sort the index by the values of an attribute, in ascending order.
• desc("ATTRIBUTE"). Sort the index by the values of an attribute, in descending order.

Before you modify the default setting, you should test your changes in the dashboard, and by A/B testing.

config.initialIndexSettings.{key}. relevancyStrictness · integer · default: 100

Relevancy threshold below which less relevant results aren't included in the results You can only set relevancyStrictness on virtual replica indices. Use this setting to strike a balance between the relevance and number of returned results.

Example:

90

config.initialIndexSettings.{key}. attributesToHighlight · string[]

Attributes to highlight By default, all searchable attributes are highlighted. Use * to highlight all attributes or use an empty array [] to turn off highlighting. Attribute names are case-sensitive With highlighting, strings that match the search query are surrounded by HTML tags defined by highlightPreTag and highlightPostTag. You can use this to visually highlight matching parts of a search query in your UI For more information, see Highlighting and snippeting.

Example:

["author", "title", "conten", "content"]

config.initialIndexSettings.{key}. attributesToSnippet · string[]

Attributes for which to enable snippets. Attribute names are case-sensitive Snippets provide additional context to matched words. If you enable snippets, they include 10 words, including the matched word. The matched word will also be wrapped by HTML tags for highlighting. You can adjust the number of words with the following notation: ATTRIBUTE:NUMBER, where NUMBER is the number of words to be extracted.

Example:

["content:80", "description"]

config.initialIndexSettings.{key}. highlightPreTag · string · default: <em>

HTML tag to insert before the highlighted parts in all highlighted results and snippets.

config.initialIndexSettings.{key}. highlightPostTag · string · default: </em>

HTML tag to insert after the highlighted parts in all highlighted results and snippets.

config.initialIndexSettings.{key}. snippetEllipsisText · string · default: …

String used as an ellipsis indicator when a snippet is truncated.

config.initialIndexSettings.{key}. restrictHighlightAndSnippetArrays · boolean · default: false

Whether to restrict highlighting and snippeting to items that at least partially matched the search query. By default, all items are highlighted and snippeted.

config.initialIndexSettings.{key}. hitsPerPage · integer · default: 20

Number of hits per page.

Required range: 1 <= x <= 1000

config.initialIndexSettings.{key}. minWordSizefor1Typo · integer · default: 4

Minimum number of characters a word in the search query must contain to accept matches with one typo.

config.initialIndexSettings.{key}. minWordSizefor2Typos · integer · default: 8

Minimum number of characters a word in the search query must contain to accept matches with two typos.

config.initialIndexSettings.{key}. typoTolerance · default: true

Whether typo tolerance is enabled and how it is applied.

If typo tolerance is true, min, or strict, word splitting and concatenation are also active.

config.initialIndexSettings.{key}. allowTyposOnNumericTokens · boolean · default: true

Whether to allow typos on numbers in the search query Turn off this setting to reduce the number of irrelevant matches when searching in large sets of similar numbers.

config.initialIndexSettings.{key}. disableTypoToleranceOnAttributes · string[]

Attributes for which you want to turn off typo tolerance. Attribute names are case-sensitive Returning only exact matches can help when

• Searching in hyphenated attributes.
• Reducing the number of matches when you have too many. This can happen with attributes that are long blocks of text, such as product descriptions Consider alternatives such as disableTypoToleranceOnWords or adding synonyms if your attributes have intentional unusual spellings that might look like typos.

Example:

["sku"]

config.initialIndexSettings.{key}. ignorePlurals

Treat singular, plurals, and other forms of declensions as equivalent. You should only use this feature for the languages used in your index.

ISO code for a supported language.

Available options: af, ar, az, bg, bn, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fo, fr, ga, gl, he, hi, hu, hy, id, is, it, ja, ka, kk, ko, ku, ky, lt, lv, mi, mn, mr, ms, mt, nb, nl, no, ns, pl, ps, pt, pt-br, qu, ro, ru, sk, sq, sv, sw, ta, te, th, tl, tn, tr, tt, uk, ur, uz, zh

Example:

["ca", "es"]

config.initialIndexSettings.{key}. removeStopWords

Removes stop words from the search query.

Stop words are common words like articles, conjunctions, prepositions, or pronouns that have little or no meaning on their own. In English, "the", "a", or "and" are stop words.

You should only use this feature for the languages used in your index.

ISO code for a supported language.

Available options: af, ar, az, bg, bn, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fo, fr, ga, gl, he, hi, hu, hy, id, is, it, ja, ka, kk, ko, ku, ky, lt, lv, mi, mn, mr, ms, mt, nb, nl, no, ns, pl, ps, pt, pt-br, qu, ro, ru, sk, sq, sv, sw, ta, te, th, tl, tn, tr, tt, uk, ur, uz, zh

Example:

["ca", "es"]

config.initialIndexSettings.{key}. queryLanguages · enum<string>[]

Languages for language-specific query processing steps such as plurals, stop-word removal, and word-detection dictionaries This setting sets a default list of languages used by the removeStopWords and ignorePlurals settings. This setting also sets a dictionary for word detection in the logogram-based CJK languages. To support this, you must place the CJK language first You should always specify a query language. If you don't specify an indexing language, the search engine uses all supported languages, or the languages you specified with the ignorePlurals or removeStopWords parameters. This can lead to unexpected search results. For more information, see Language-specific configuration.

ISO code for a supported language.

Available options: af, ar, az, bg, bn, ca, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fo, fr, ga, gl, he, hi, hu, hy, id, is, it, ja, ka, kk, ko, ku, ky, lt, lv, mi, mn, mr, ms, mt, nb, nl, no, ns, pl, ps, pt, pt-br, qu, ro, ru, sk, sq, sv, sw, ta, te, th, tl, tn, tr, tt, uk, ur, uz, zh

Example:

["es"]

config.initialIndexSettings.{key}. decompoundQuery · boolean · default: true

Whether to split compound words in the query into their building blocks For more information, see Word segmentation. Word segmentation is supported for these languages: German, Dutch, Finnish, Swedish, and Norwegian. Decompounding doesn't work for words with non-spacing mark Unicode characters. For example, Gartenstühle won't be decompounded if the ü consists of u (U+0075) and ◌̈ (U+0308).

config.initialIndexSettings.{key}. enableRules · boolean · default: true

Whether to enable rules.

config.initialIndexSettings.{key}. enablePersonalization · boolean · default: false

Whether to enable Personalization.

config.initialIndexSettings.{key}. queryType · enum<string> · default: prefixLast

Determines if and how query words are interpreted as prefixes.

By default, only the last query word is treated as a prefix (prefixLast). To turn off prefix search, use prefixNone. Avoid prefixAll, which treats all query words as prefixes. This might lead to counterintuitive results and makes your search slower.

For more information, see Prefix searching.

Available options: prefixLast, prefixAll, prefixNone

config.initialIndexSettings.{key}. removeWordsIfNoResults · enum<string> · default: none

Strategy for removing words from the query when it doesn't return any results. This helps to avoid returning empty search results.

• none. No words are removed when a query doesn't return results.

• lastWords. Treat the last (then second to last, then third to last) word as optional, until there are results or at most 5 words have been removed.

• firstWords. Treat the first (then second, then third) word as optional, until there are results or at most 5 words have been removed.

• allOptional. Treat all words as optional.

For more information, see Remove words to improve results.

Available options: none, lastWords, firstWords, allOptional

Example:

"firstWords"

config.initialIndexSettings.{key}. mode · enum<string> · default: keywordSearch

Search mode the index will use to query for results.

This setting only applies to indices, for which Algolia enabled NeuralSearch for you.

Available options: neuralSearch, keywordSearch

config.initialIndexSettings.{key}. semanticSearch · object

Settings for the semantic search part of NeuralSearch. Only used when mode is neuralSearch.

Show child attributes

config.initialIndexSettings.{key}.semanticSearch. eventSources · string[] | null

Indices from which to collect click and conversion events.

If null, the current index and all its replicas are used.

config.initialIndexSettings.{key}. advancedSyntax · boolean · default: false

Whether to support phrase matching and excluding words from search queries Use the advancedSyntaxFeatures parameter to control which feature is supported.

config.initialIndexSettings.{key}. optionalWords

Words that should be considered optional when found in the query.

By default, records must match all words in the search query to be included in the search results. Adding optional words can help to increase the number of search results by running an additional search query that doesn't include the optional words. For example, if the search query is "action video" and "video" is an optional word, the search engine runs two queries. One for "action video" and one for "action". Records that match all words are ranked higher.

For a search query with 4 or more words and all its words are optional, the number of matched words required for a record to be included in the search results increases for every 1,000 records:

• If optionalWords has less than 10 words, the required number of matched words increases by 1: results 1 to 1,000 require 1 matched word, results 1,001 to 2000 need 2 matched words.
• If optionalWords has 10 or more words, the number of required matched words increases by the number of optional words divided by 5 (rounded down). For example, with 18 optional words: results 1 to 1,000 require 1 matched word, results 1,001 to 2000 need 4 matched words.

For more information, see Optional words.

config.initialIndexSettings.{key}. disableExactOnAttributes · string[]

Searchable attributes for which you want to turn off the Exact ranking criterion. Attribute names are case-sensitive This can be useful for attributes with long values, where the likelihood of an exact match is high, such as product descriptions. Turning off the Exact ranking criterion for these attributes favors exact matching on other attributes. This reduces the impact of individual attributes with a lot of content on ranking.

Example:

["description"]

config.initialIndexSettings.{key}. exactOnSingleWordQuery · enum<string> · default: attribute

Determines how the Exact ranking criterion is computed when the search query has only one word.

• attribute. The Exact ranking criterion is 1 if the query word and attribute value are the same. For example, a search for "road" will match the value "road", but not "road trip".

• none. The Exact ranking criterion is ignored on single-word searches.

• word. The Exact ranking criterion is 1 if the query word is found in the attribute value. The query word must have at least 3 characters and must not be a stop word. Only exact matches will be highlighted, partial and prefix matches won't.

Available options: attribute, none, word

config.initialIndexSettings.{key}. alternativesAsExact · enum<string>[]

Determine which plurals and synonyms should be considered an exact matches By default, Algolia treats singular and plural forms of a word, and single-word synonyms, as exact matches when searching. For example

• "swimsuit" and "swimsuits" are treated the same
• "swimsuit" and "swimwear" are treated the same (if they are synonyms)
• ignorePlurals. Plurals and similar declensions added by the ignorePlurals setting are considered exact matches
• singleWordSynonym. Single-word synonyms, such as "NY" = "NYC", are considered exact matches
• multiWordsSynonym. Multi-word synonyms, such as "NY" = "New York", are considered exact matches.

Available options: ignorePlurals, singleWordSynonym, multiWordsSynonym, ignoreConjugations

config.initialIndexSettings.{key}. advancedSyntaxFeatures · enum<string>[]

Advanced search syntax features you want to support

• exactPhrase. Phrases in quotes must match exactly. For example, sparkly blue "iPhone case" only returns records with the exact string "iPhone case"
• excludeWords. Query words prefixed with a - must not occur in a record. For example, search -engine matches records that contain "search" but not "engine" This setting only has an effect if advancedSyntax is true.

Available options: exactPhrase, excludeWords

config.initialIndexSettings.{key}. distinct

Determines how many records of a group are included in the search results.

Records with the same value for the attributeForDistinct attribute are considered a group. The distinct setting controls how many members of the group are returned. This is useful for deduplication and grouping.

The distinct setting is ignored if attributeForDistinct is not set.

Example:

1

config.initialIndexSettings.{key}. replaceSynonymsInHighlight · boolean · default: false

Whether to replace a highlighted word with the matched synonym By default, the original words are highlighted even if a synonym matches. For example, with home as a synonym for house and a search for home, records matching either "home" or "house" are included in the search results, and either "home" or "house" are highlighted With replaceSynonymsInHighlight set to true, a search for home still matches the same records, but all occurrences of "house" are replaced by "home" in the highlighted response.

config.initialIndexSettings.{key}. minProximity · integer · default: 1

Minimum proximity score for two matching words This adjusts the Proximity ranking criterion by equally scoring matches that are farther apart For example, if minProximity is 2, neighboring matches and matches with one word between them would have the same score.

Required range: 1 <= x <= 7

config.initialIndexSettings.{key}. responseFields · string[]

Properties to include in the API response of search and browse requests By default, all response properties are included. To reduce the response size, you can select which properties should be included An empty list may lead to an empty API response (except properties you can't exclude) You can't exclude these properties: message, warning, cursor, abTestVariantID, or any property added by setting getRankingInfo to true Your search depends on the hits field. If you omit this field, searches won't return any results. Your UI might also depend on other properties, for example, for pagination. Before restricting the response size, check the impact on your search experience.

config.initialIndexSettings.{key}. maxValuesPerFacet · integer · default: 100

Maximum number of facet values to return for each facet.

Required range: x <= 1000

config.initialIndexSettings.{key}. sortFacetValuesBy · string · default: count

Order in which to retrieve facet values

• count. Facet values are retrieved by decreasing count. The count is the number of matching records containing this facet value
• alpha. Retrieve facet values alphabetically This setting doesn't influence how facet values are displayed in your UI (see renderingContent). For more information, see facet value display.

config.initialIndexSettings.{key}. attributeCriteriaComputedByMinProximity · boolean · default: false

Whether the best matching attribute should be determined by minimum proximity This setting only affects ranking if the Attribute ranking criterion comes before Proximity in the ranking setting. If true, the best matching attribute is selected based on the minimum proximity of multiple matches. Otherwise, the best matching attribute is determined by the order in the searchableAttributes setting.

config.initialIndexSettings.{key}. renderingContent · object

Extra data that can be used in the search UI.

You can use this to control aspects of your search UI, such as the order of facet names and values without changing your frontend code.

Show child attributes

config.initialIndexSettings.{key}.renderingContent. facetOrdering · object

Order of facet names and facet values in your UI.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.facetOrdering. facets · object

Order of facet names.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.facetOrdering.facets. order · string[]

Explicit order of facets or facet values.

This setting lets you always show specific facets or facet values at the top of the list.

config.initialIndexSettings.{key}.renderingContent.facetOrdering. values · object

Order of facet values. One object for each facet.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.facetOrdering.values. {key} · object

Show child attributes

config.initialIndexSettings.{key}.renderingContent.facetOrdering.values.{key}. order · string[]

Explicit order of facets or facet values.

This setting lets you always show specific facets or facet values at the top of the list.

config.initialIndexSettings.{key}.renderingContent.facetOrdering.values.{key}. sortRemainingBy · enum<string>

Order of facet values that aren't explicitly positioned with the order setting.

• count. Order remaining facet values by decreasing count. The count is the number of matching records containing this facet value.

• alpha. Sort facet values alphabetically.

• hidden. Don't show facet values that aren't explicitly positioned.

Available options: count, alpha, hidden

config.initialIndexSettings.{key}.renderingContent.facetOrdering.values.{key}. hide · string[]

Hide facet values.

config.initialIndexSettings.{key}.renderingContent. redirect · object

The redirect rule container.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.redirect. url · string

config.initialIndexSettings.{key}.renderingContent. widgets · object

Widgets returned from any rules that are applied to the current search.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.widgets. banners · object[]

Banners defined in the Merchandising Studio for a given search.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.widgets.banners. image · object

Image to show inside a banner.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.widgets.banners.image. urls · object[]

Show child attributes

config.initialIndexSettings.{key}.renderingContent.widgets.banners.image.urls. url · string

config.initialIndexSettings.{key}.renderingContent.widgets.banners.image. title · string

config.initialIndexSettings.{key}.renderingContent.widgets.banners. link · object

Link for a banner defined in the Merchandising Studio.

Show child attributes

config.initialIndexSettings.{key}.renderingContent.widgets.banners.link. url · string

config.initialIndexSettings.{key}. enableReRanking · boolean · default: true

Whether this search will use Dynamic Re-Ranking This setting only has an effect if you activated Dynamic Re-Ranking for this index in the Algolia dashboard.

config.initialIndexSettings.{key}. reRankingApplyFilter

Restrict Dynamic Re-Ranking to records that match these filters.

config. linkExtractor · linkExtractor · object

Function for extracting URLs from links on crawled pages.

For more information, see the linkExtractor documentation.

Show child attributes

config.linkExtractor. __type · enum<string>

Available options: function

config.linkExtractor. source · string

Example:

"({ $, url, defaultExtractor }) => {\n if (/example.com\\/doc\\//.test(url.href)) {\n // For all pages under/doc, only extract the first found URL.\n return defaultExtractor().slice(0, 1)\n }\n // For all other pages, use the default.\n return defaultExtractor()\n}\n"

config. login · HTTP request · object

Authorization method and credentials for crawling protected content.

The Crawler supports these authentication methods:

• Basic authentication. The Crawler obtains a session cookie from the login page.
• OAuth 2.0 authentication (oauthRequest). The Crawler uses OAuth 2.0 client credentials to obtain an access token for authentication.

Basic authentication

The Crawler extracts the Set-Cookie response header from the login page, stores that cookie, and sends it in the Cookie header when crawling all pages defined in the configuration.

This cookie is retrieved only at the start of each full crawl. If it expires, it isn't automatically renewed.

The Crawler can obtain the session cookie in one of two ways:

• HTTP request authentication (fetchRequest). The Crawler sends a direct request with your credentials to the login endpoint, similar to a curl command.
• Browser-based authentication (browserRequest). The Crawler emulates a web browser by loading the login page, entering the credentials, and submitting the login form as a real user would.

OAuth 2.0

The crawler supports OAuth 2.0 client credentials grant flow:

1. It performs an access token request with the provided credentials
2. Stores the fetched token in an Authorization header
3. Sends the token when crawling site pages.

This token is only fetched at the beginning of each complete crawl. If it expires, it isn't automatically renewed.

Client authentication passes the credentials (client_id and client_secret) in the request body. The Azure AD v1.0 provider is supported.

HTTP request

Show child attributes

config.login. url · string · required

URL with your login form.

Example:

"https://example.com/login"

config.login. requestOptions · object

Options for the HTTP request for logging in.

Show child attributes

config.login.requestOptions. method · string · default: GET

HTTP method for sending the request.

Example:

"POST"

config.login.requestOptions. headers · object

Headers to add to all requests.

Show child attributes

config.login.requestOptions.headers. Accept-Language · string

Preferred natural language and locale.

Example:

"fr-FR"

config.login.requestOptions.headers. Authorization · string

Basic authentication header.

Example:

"Bearer Aerehdf=="

config.login.requestOptions.headers. Cookie · string

Cookie. The header will be replaced by the cookie retrieved when logging in.

Example:

"session=1234"

config.login.requestOptions. body · string

Form content.

Example:

"id=user&password=s3cr3t"

config.login.requestOptions. timeout · integer

Timeout for the request.

Example:

{
  "url": "https://example.com/secure/login-with-post",
  "requestOptions": {
    "method": "POST",
    "headers": {
      "Content-Type": "application/x-www-form-urlencoded"
    },
    "body": "id=my-id&password=my-password",
    "timeout": 5000
  }
}

config. maxDepth · integer

Determines the maximum path depth of crawled URLs.

Path depth is calculated based on the number of slash characters (/) after the domain (starting at 1). For example:

• 1 http://example.com
• 1 http://example.com/
• 1 http://example.com/foo
• 2 http://example.com/foo/
• 2 http://example.com/foo/bar
• 3 http://example.com/foo/bar/

URLs added with startUrls and sitemaps aren't checked for maxDepth..

Required range: 1 <= x <= 100

Example:

5

config. maxUrls · integer

Limits the number of URLs your crawler processes.

Change it to a low value, such as 100, for quick crawling tests. Change it to a higher explicit value for full crawls to prevent it from getting "lost" in complex site structures. Because the Crawler works on many pages simultaneously, maxUrls doesn't guarantee finding the same pages each time it runs.

Required range: 1 <= x <= 15000000

Example:

250

config. renderJavaScript

If true, use a Chrome headless browser to crawl pages.

Because crawling JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specific list of pages. Use micromatch to define URL patterns, including negations and wildcards.

config. requestOptions · object

Lets you add options to HTTP requests made by the crawler.

Show child attributes

config.requestOptions. proxy · string

Proxy for all crawler requests.

config.requestOptions. timeout · integer · default: 30000

Timeout in milliseconds for the crawl.

config.requestOptions. retries · integer · default: 3

Maximum number of retries to crawl one URL.

config.requestOptions. headers · object

Headers to add to all requests.

Show child attributes

config.requestOptions.headers. Accept-Language · string

Preferred natural language and locale.

Example:

"fr-FR"

config.requestOptions.headers. Authorization · string

Basic authentication header.

Example:

"Bearer Aerehdf=="

config.requestOptions.headers. Cookie · string

Cookie. The header will be replaced by the cookie retrieved when logging in.

Example:

"session=1234"

config. safetyChecks · object

Checks to ensure the crawl was successful.

For more information, see the Safety checks documentation.

Show child attributes

config.safetyChecks. beforeIndexPublishing · object

Checks triggered after the crawl finishes but before the records are added to the Algolia index.

Show child attributes

config.safetyChecks.beforeIndexPublishing. maxLostRecordsPercentage · integer · default: 10

Maximum difference in percent between the numbers of records between crawls.

Required range: 1 <= x <= 100

config.safetyChecks.beforeIndexPublishing. maxFailedUrls · integer

Stops the crawler if a specified number of pages fail to crawl.

config. saveBackup · boolean

Whether to back up your index before the crawler overwrites it with new records.

config. schedule · string

Schedule for running the crawl.

Instead of manually starting a crawl each time, you can set up a schedule for automatic crawls. Use the visual UI or add the schedule parameter to your configuration.

schedule uses Later.js syntax to specify when to crawl your site. Here are some key things to keep in mind when using Later.js syntax with the Crawler:

• The interval between two scheduled crawls must be at least 24 hours.
• To crawl daily, use "every 1 day" instead of "everyday" or "every day".
• If you don't specify a time, the crawl can happen any time during the scheduled day.
• Specify times for the UTC (GMT+0) timezone
• Include minutes when specifying a time. For example, "at 3:00 pm" instead of "at 3pm".
• Use "at 12:00 am" to specify midnight, not "at 00:00 am".

Example:

"every weekday at 12:00 pm"

config. sitemaps · string[]

Sitemaps with URLs from where to start crawling.

Maximum array length: 9999

config. startUrls · string[]

URLs from where to start crawling.

Maximum array length: 9999

createdAt · string · required

Date and time when the object was created, in RFC 3339 format.

Example:

"2023-07-04T12:49:15Z"

authorId · string · required

Universally unique identifier (UUID) of the user who created this version of the configuration.

Example:

"7d79f0dd-2dab-4296-8098-957a1fdc0637"