login

Parameters for authenticating with protected resources.

• Type: object

This property defines how the Crawler acquires a session to access protected content.

Basic auth

The Crawler extracts the Set-Cookie response header from the login page, stores that cookie and sends it in a Cookie header when crawling all pages of the website defined in the configuration.

This cookie is only retrieved at the beginning of each complete crawl. It won’t be renewed automatically if it expires.

The Crawler can interact with your login page in these ways:

• With a direct request with the credentials to your login endpoint, like a standard curl command (fetchRequest).
• By emulating a web browser, loading your login page, entering the credentials and validating the login form (browserRequest).

OAuth 2.0

The crawler supports the OAuth 2.0 Client Credentials Grant flow. It performs an Access Token Request using the provided credentials, stores the retrieved token in an Authorization header, and sends it when crawling all pages of the website defined in the configuration.

This token is only retrieved at the beginning of each complete crawl. It won’t be renewed automatically if it expires.

Client authentication is performed by passing the client credentials (client_id and client_secret) in the request body as described in RFC 6749.

The following providers are supported.

• Azure AD v1.0

Parameters

fetchRequest · object

Manually create the login request.

Example:

JavaScript

{
  login: {
    fetchRequest: {
      url: "https://example.com/secure/login-with-post",
      requestOptions: {
        method: "POST",
        headers: { "Content-Type": "application/x-www-form-urlencoded" },
        body: "id=my-id&password=my-password",
        timeout: 5000 // in milliseconds
      }
    }
  }
}

Show child attributes

fetchRequest.url · string · required

The URL to target for logging in.

fetchRequest.requestOptions · object

Options for the login request.

Show child attributes

requestOptions.method · string · default: "GET"

HTTP request method to use for the login.

requestOptions.headers · object · default: "{}"

HTTP headers for the login request.

Show child attributes

headers.Content-Type · string

Content-Type header for the login request.

headers.Authorization · string

Authorization header for the login request.

headers.Cookie · string

Cookie header for the login request.

requestOptions.body · string

Request body for the login request as string.

requestOptions.timeout · number

Time in milliseconds to wait before canceling the login request.

browserRequest · object

Use a web browser to visit your login page and validate the login form similar to a human.

Example:

JavaScript

{
  login: {
    browserRequest: {
      url: "https://example.com/secure/login-page",
      username: "my-id",
      password: "my-password",
    }
  }
}

Show child attributes

browserRequest.url · string · required

URL of your login page. The page must have input[type=text] or input[type=email] elements for the username, and input[type=password] for the password.

browserRequest.username · string · required

Your username for logging in.

browserRequest.password · string · required

Your password for logging in.

browserRequest.waitTime · object

Shortest and longest time to wait before considering the login complete.

Show child attributes

waitTime.min · number · default: 0

If the login finishes faster than this time, the browser remains open at least this long before returning the cookies.

waitTime.max · number · default: 2000

Maximum time to wait before returning the cookies.

oauthRequest · object

Use the OAuth 2.0 Client Credentials Grant flow to generate an Authorization header.

Example:

JavaScript

{
  login: {
    oauthRequest: {
      accessTokenRequest: {
        url: "https://example.com/oauth2/token",
        grant_type: "client_credentials",
        client_id: "my-client-id",
        client_secret: "my-client-secret",
        extraParameters: {
          resource: "https://protected.example.com/"
        }
      }
    }
  }
}

Show child attributes

oauthrequest.accessTokenRequest · object · required

Required parameters to perform the Access token request.

Show child attributes

accessTokenRequest.client_id · string · required

The client identifier.

accessTokenRequest.client_secret · string · required

The client secret.

accessTokenRequest.grant_type · string · required

Must be client_credentials.

accessTokenRequest.url · string · required

URL of your access token endpoint.

accessTokenRequest.extraParameters · object

Implementation-specific parameters that aren’t part of RFC 6749.

Show child attributes

extraParameters.resource · string

Required for Azure AD v1.0 implementations.

accessTokenRequest.scope · string

Scope of the access request.