Test crawl a URL

Tests a URL with the crawler’s configuration and shows the extracted records.

POST /1/crawlers/{id}/test

Test crawl a URL

cURL

curl --request POST \
  --url https://crawler.algolia.com/api/1/crawlers/{id}/test \
  --header 'Authorization: Basic <encoded-value>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://www.algolia.com/blog"
}
'

200

{
  "startDate": "2024-04-02T15:34:29Z",
  "endDate": "2024-04-02T15:34:29Z",
  "logs": [
    [
      "Processing url 'https://www.algolia.com/blog'"
    ]
  ],
  "records": [
    {
      "indexName": "testIndex",
      "records": [
        {
          "objectID": "https://www.algolia.com/blog",
          "numberOfLinks": 2
        }
      ],
      "recordsPerExtractor": [
        {
          "index": 0,
          "type": "custom",
          "records": [
            {
              "objectID": "https://www.algolia.com/blog"
            }
          ]
        }
      ]
    }
  ],
  "links": [
    "https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
    "https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
  ],
  "externalData": {
    "externalData1": {
      "data1": "val1",
      "data2": "val2"
    },
    "externalData2": {
      "data1": "val1",
      "data2": "val2"
    }
  },
  "error": {}
}

You can test configuration changes by overriding specific parts before updating the full configuration.

Authorizations

Authorization · string · header · required

Basic authentication header of the form Basic <encoded-value>, where <encoded-value> is the base64-encoded string username:password.

Path Parameters

id · string · required

Crawler ID. Universally unique identifier (UUID) of the crawler.

Example:

"e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"

Body

application/json

url · string · required

URL to test.

Example:

"https://www.algolia.com/blog"

config · object

Crawler configuration to update. You can only update top-level configuration properties. To update a nested configuration, such as actions.recordExtractor, you must provide the complete top-level object such as actions.

Show child attributes

Response

200

application/json

OK

startDate · string · required

Date and time when the test crawl started, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

endDate · string · required

Date and time when the test crawl finished, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

logs · string[][] · required

Logs from the record extraction.

Parameters associated with each log.

records · extractedRecords · object[] · required

Extracted records from the URL.

Show child attributes

links · string[] · required

Links found on the page, which match the configuration and would be processed.

Example:

[
  "https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
  "https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
]

externalData · object

External data associated with the tested URL.

External data is refreshed automatically at the beginning of the crawl.

Example:

{
  "externalData1": { "data1": "val1", "data2": "val2" },
  "externalData2": { "data1": "val1", "data2": "val2" }
}

error · urlTestError · object

An error.

Show child attributes

Example:

{}