List A/B tests

List A/B tests information and results.

This page documents an earlier version of the API client. For the latest version, see List all A/B tests.

Required ACL: analytics

Examples

C#

AnalyticsClient analytics = new AnalyticsClient("YourApplicationID", "YourWriteAPIKey");

var abTests = analytics.GetABTests(10, 20);

// Asynchronous
var abTests = await analytics.GetABTestsAsync(10, 20);

Parameters

indexPrefix · string

Prefix of index names. Returns A/B test variants whose index names start with this prefix.

indexSuffix · string

Suffix of index names. Returns A/B test variants whose index names end with this suffix.

limit · integer · default: 10

Number of A/B tests to retrieve.

offset · integer · default: 0

Position of the first A/B test to retrieve.

Response

abtests · object[]

List of A/B test objects.

Show child attributes

abTestID · integer

ID of the A/B test.

addToCartSignificance · number

A/B test significance based on addToCart conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.

clickSignificance · number

A/B test significance based on click data. Should be greater than 0.95 to be considered significant, no matter which variant is winning.

configuration · object

Configuration parameters for the A/B test.

Example:

JSON

{
  "outliers": {"exclude": true},
  "emptySearch": {"exclude": true},
  "minimumDetectableEffect": {}
}

Show child attributes

emptySearch · object

Configuration effect of empty search exclusion from metrics calculations.

Show child attributes

exclude · boolean

Whether empty searches are excluded when calculating A/B test results.

minimumDetectableEffect · object

Configuration for minimum detectable effect.

Show child attributes

effect · enum<string>

The effect you want to detect. One of:

• addToCartRate
• clickThroughRate
• conversionRate
• purchaseRate

size · number

The smallest relative difference between the variants you want to detect. For example, to detect a 10% difference between the variants, set this to 0.1. The size must be between 0 and 1.

outliers · object

Configuration effect of outlier exclusion from metrics calculations.

conversionSignificance · number

A/B test significance based on conversion data. Should be greater than 0.95 to be considered significant, no matter which variant is winning.

createdAt · string

Time at which the A/B test has been created. The date will be in the following format: Y-m-d\TH:i:s\Z

endAt · string

Time at which the A/B test will automatically stop. The date will be in the following format: Y-m-d\TH:i:s\Z

name · string

Name of the A/B test.

purchaseSignificance · number

A/B test significance based on purchase conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.

revenueSignificance · object

A/B test revenue significance for each currency. The object has currency codes as keys and significance scores as values.

status · string

Current status of the A/B test.

variants · object[]

List of 2 variants:

• The base index
• The index to test against

Show child attributes

addToCartCount · integer

Distinct number of converted clicks with the eventSubtype set to addToCart and clickAnalytics set to true.

addToCartRate · number

Add-to-cart rate is based on conversion events that have eventSubType set to addToCart, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of add-to-cart conversion events divided by the number of tracked searches.

averageClickPosition · integer

Average click position for the variant.

clickCount · integer

Distinct click count for the variant.

clickThroughRate · number

Click through rate for the variant.

conversionCount · integer

Distinct conversion count for the variant.

conversionRate · number

Conversion rate for the variant.

currencies · object

Revenue metrics for each currency.

Example:

JSON

{
  "USD": {
    "currency": "USD",
    "revenue": "...",
    "mean": "...",
    "standardDeviation": "..."
  },
  "EUR": {}
}

Show child attributes

currency · string

Currency code in ISO 4217 format.

mean · number

Mean revenue for this currency.

revenue · number

Revenue made in this currency.

standardDeviation · number

Standard deviation of revenue for this currency.

description · string

Description of this variant.

estimatedSampleSize · integer

Estimated number of searches needed to achieve confidence for this variant. This number is pre-computed and is based on the minimumDetectableEffect configuration parameter.

filterEffects · object

Effects of the outlier or empty search exclusion. Each effect has these properties:

• usersCount: number of users excluded by this filter
• trackedSearchesCount: number of tracked searches excluded by this filter

Example:

JSON

{
  "outliers": {
        "usersCount": "...",
        "trackedSearchesCount": "..."
  },
  "emptySearch": {
        "usersCount": "...",
        "trackedSearchesCount": "..."
    }
}

index · string

Index name for this variant.

noResultCount · integer

Number of tracked searches without any results.

purchaseCount · integer

Distinct number of converted clicks with the eventSubtype set to purchase and clickAnalytics set to true.

purchaseRate · number

Purchase rate is based on conversion events that have eventSubType set to purchase, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of purchase conversion events divided by the number of tracked searches.

searchCount · integer

Total number of searches for this variant, including untracked searches.

trackedSearchCount · integer

Number of tracked searches. Tracked searches are searches with clickAnalytics enabled.

trackedUserCount · integer

Number of users who searched this variant with at least one search that had clickAnalytics enabled.

userCount · integer

Total number of users who searched this variant, including untracked users.

count · integer

Number of returned A/B tests.

total · integer

Total number of A/B tests that can be fetched.

Response as JSON

This section shows the JSON response returned by the API. Each API client wraps this response in language-specific objects, so the structure may vary. To view the response, use the getLogs method. Don’t rely on the order of properties—JSON objects don’t preserve key order.

JSON

{
  "count": 1,
  "total": 1,
  "abtests": [
    {
      "abtestID": 102,
      "name": "ABTest custom name",
      "createdAt": "2018-06-04T10:25:40.459519Z",
      "endAt": "2018-06-11T23:59:59Z",
      "status": "active",
      "configuration": {
        "outliers": {
          "exclude": true
        },
        "emptySearch": {
          "exclude": false
        },
      },
      "variants": [
        {
          "averageClickPosition": null,
          "clickCount": null,
          "clickThroughRate": null,
          "conversionCount": null,
          "conversionRate": null,
          "description": "my current scenario description",
          "index": "scout_airports",
          "noResultCount": null,
          "searchCount": null,
          "trackedSearchCount": null,
          "trafficPercentage": 61,
          "userCount": null,
          "trackedUserCount": null,
          "filterEffects": {
            "outliers": {
              "usersCount": 0,
              "trackedSearchesCount": 0
            }
          }
        },
        {
          "averageClickPosition": null,
          "clickCount": null,
          "clickThroughRate": null,
          "conversionCount": null,
          "conversionRate": null,
          "description": "the test scenarios is based on locations",
          "index": "airport_demo_airports",
          "noResultCount": null,
          "searchCount": null,
          "trackedSearchCount": null,
          "trafficPercentage": 39,
          "userCount": null,
          "trackedUserCount": null,
          "filterEffects": {
            "outliers": {
              "usersCount": 1,
              "trackedSearchesCount": 300
            }
          }
        }
      ],
      "clickSignificance": null,
      "conversionSignificance": null
    }
  ]
}