Test crawl a URL

Tests a URL with the crawler’s configuration and shows the extracted records.

You can test configuration changes by overriding specific parts before updating the full configuration.

POST /1/crawlers/{id}/test

Test crawl a URL

cURL

curl --request POST \
  --url https://crawler.algolia.com/api/1/crawlers/{id}/test \
  --header 'Authorization: Basic <encoded-value>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://www.algolia.com/blog",
  "config": {
    "actions": [
      {
        "indexName": "algolia_website",
        "recordExtractor": {
          "__type": "function",
          "source": "<string>"
        },
        "autoGenerateObjectIDs": true,
        "cache": {
          "enabled": true
        },
        "discoveryPatterns": [
          "https://www.algolia.com/**"
        ],
        "fileTypesToMatch": [
          "html",
          "pdf"
        ],
        "hostnameAliases": {
          "dev.example.com": "example.com"
        },
        "name": "<string>",
        "pathAliases": {
          "example.com": {
            "/foo": "/bar"
          }
        },
        "pathsToMatch": [
          "https://www.algolia.com/**"
        ],
        "schedule": "<string>",
        "selectorsToMatch": [
          ".products",
          "!.featured"
        ]
      }
    ],
    "appId": "<string>",
    "rateLimit": 4,
    "apiKey": "<string>",
    "exclusionPatterns": [
      "https://www.example.com/excluded",
      "!https://www.example.com/this-one-url",
      "https://www.example.com/exclude/**"
    ],
    "externalData": [
      "testCSV"
    ],
    "extraUrls": [
      "<string>"
    ],
    "ignoreCanonicalTo": true,
    "ignoreNoFollowTo": true,
    "ignoreNoIndex": true,
    "ignorePaginationAttributes": true,
    "ignoreQueryParams": [
      "ref",
      "utm_*"
    ],
    "ignoreRobotsTxtRules": true,
    "indexPrefix": "crawler_",
    "initialIndexSettings": {},
    "linkExtractor": {
      "__type": "function",
      "source": "({ $, url, defaultExtractor }) => {\n  if (/example.com\\/doc\\//.test(url.href)) {\n    // For all pages under `/doc`, only extract the first found URL.\n    return defaultExtractor().slice(0, 1)\n  }\n  // For all other pages, use the default.\n  return defaultExtractor()\n}\n"
    },
    "login": {
      "url": "https://example.com/secure/login-with-post",
      "requestOptions": {
        "method": "POST",
        "headers": {
          "Content-Type": "application/x-www-form-urlencoded"
        },
        "body": "id=my-id&password=my-password",
        "timeout": 5000
      }
    },
    "maxDepth": 5,
    "maxUrls": 250,
    "renderJavaScript": true,
    "requestOptions": {
      "proxy": "<string>",
      "timeout": 30000,
      "retries": 3,
      "headers": {
        "Accept-Language": "fr-FR",
        "Authorization": "Bearer Aerehdf==",
        "Cookie": "session=1234"
      }
    },
    "safetyChecks": {
      "beforeIndexPublishing": {
        "maxLostRecordsPercentage": 10,
        "maxFailedUrls": 123
      }
    },
    "saveBackup": true,
    "schedule": "every weekday at 12:00 pm",
    "sitemaps": [
      "https://example.com/sitemap.xyz"
    ],
    "startUrls": [
      "https://www.example.com"
    ]
  }
}
'

200

{
  "startDate": "2024-04-02T15:34:29Z",
  "endDate": "2024-04-02T15:34:29Z",
  "logs": [
    [
      "Processing url 'https://www.algolia.com/blog'"
    ]
  ],
  "records": [
    {
      "indexName": "testIndex",
      "records": [
        {
          "objectID": "https://www.algolia.com/blog",
          "numberOfLinks": 2
        }
      ],
      "recordsPerExtractor": [
        {
          "index": 0,
          "type": "custom",
          "records": [
            {
              "objectID": "https://www.algolia.com/blog"
            }
          ]
        }
      ]
    }
  ],
  "links": [
    "https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
    "https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
  ],
  "externalData": {
    "externalData1": {
      "data1": "val1",
      "data2": "val2"
    },
    "externalData2": {
      "data1": "val1",
      "data2": "val2"
    }
  },
  "error": {}
}

Authorizations

Authorization · string · header · required

Basic authentication header of the form Basic <encoded-value>, where <encoded-value> is the base64-encoded string username:password.

Path Parameters

id · string · required

Crawler ID. Universally unique identifier (UUID) of the crawler.

Example:

"e0f6db8a-24f5-4092-83a4-1b2c6cb6d809"

Body

application/json

url · string · required

URL to test.

Example:

"https://www.algolia.com/blog"

config · object

Crawler configuration to update. You can only update top-level configuration properties. To update a nested configuration, such as actions.recordExtractor, you must provide the complete top-level object such as actions.

Show child attributes

Response

200

application/json

OK

startDate · string · required

Date and time when the test crawl started, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

endDate · string · required

Date and time when the test crawl finished, in RFC 3339 format.

Example:

"2024-04-02T15:34:29Z"

logs · string[][] · required

Logs from the record extraction.

Parameters associated with each log.

records · extractedRecords · object[] · required

Extracted records from the URL.

Show child attributes

links · string[] · required

Links found on the page, which match the configuration and would be processed.

Example:

[
  "https://www.algolia.com/blog/engineering/challenging-migration-heroku-google-kubernetes-engine",
  "https://www.algolia.com/blog/engineering/a-tale-of-two-engines-algolia-unity"
]

externalData · object

External data associated with the tested URL.

External data is refreshed automatically at the beginning of the crawl.

Example:

{
  "externalData1": { "data1": "val1", "data2": "val2" },
  "externalData2": { "data1": "val1", "data2": "val2" }
}

error · urlTestError · object

An error.

Show child attributes

Example:

{}