Post-indexing transformations

Apply post-indexing transformations to your data already in an Algolia index with the Algolia Index connector.

Post-indexing transformations can help you optimize your data:

• Data cleaning. Correct inconsistencies or errors discovered after indexing, rename attributes, or change data types.
• A/B testing. Create variations of attributes for testing different ranking strategies.
• Index optimizations. Bucket, compute new values, convert units of measure.

To apply post-indexing transformations, use the Algolia Index connector.

The Algolia Index connector transforms the records in your index, not your source data. It can’t repair a transformation that wrote incorrect values, because those values are its input.

Get started

1. Go to the Algolia dashboard and select your Algolia .

2. On the left sidebar, select Data sources.

3. On the Connectors page, select Algolia Index connector, and click Connect.

4. Choose the index to transform.

5. Configure your transformation: create a new transformation or reuse an existing one.

6. Create the task. You can select two types of tasks:

   • On demand. You can manually trigger the task from the Task page in the dashboard, or using the API.
   • Scheduled. You can select a schedule from the list, or enter a custom schedule with a cron expression.

Costs

The Algolia Index connector browses your and re-applies the changes.

Depending on the indexing strategy you choose, this adds read and write operations. These operations might lead to costs, depending on your plan.

To control how often your connector task runs, change the schedule in the task configuration.

Performance considerations

If you don’t need an Algolia-managed transformation step, send records directly to Algolia instead. The Algolia Index connector adds overhead to your ingestion process.

Filter the source index

If your transformation doesn’t need to run over the whole index, set the filters property on the Algolia Index connector source to restrict the records the connector browses. Filtering at the source reduces extraction time and the number of read operations on your index and the overall run duration.

Use the same syntax as the Search API filters parameter. Filtered attributes must be declared in attributesForFaceting on the source index.

Constraints

• Index size: no limit on the index size.
• Duration: each run can’t take longer than 60 minutes.

How to select the right indexing strategy

Data drift happens when you have different sources of truth for your data. To avoid data drift, select your indexing strategy when you use the Algolia Index connector. The connector reads records from an Algolia index and applies transformations. Make sure this workflow doesn’t conflict with your primary ingestion method, such as the API or another connector.

Key limitation for real-time updates and full record updates

Be careful when using the Algolia Index connector with ingestion methods that use the full record updates action with real-time updates. Full record updates replace the entire record. If a real-time update runs after the connector reads data from your index but before it completes the transformations, any transformation applied by the connector is lost. That’s because a real-time update will overwrite the entire record, including the transformed fields.

The following Algolia ecommerce integrations use the full record updates by default:

• Shopify
• Magento
• Salesforce
• BigCommerce
• commercetools

Other connectors and API clients using partial updates

Connectors and API clients using partial updates work with the Algolia Index connector. Partial updates ensure that only the specified attributes are modified, preserving other attributes and avoiding conflicts.

To avoid data loss, select your indexing strategy when using the Algolia Index connector and prefer partial updates whenever possible.

• Partial record updates (recommended) allow creating, updating, and deleting during the connector run. However, concurrent field updates will be overwritten.
• Full record updates replace individual records, overriding changes made during the connector run. Avoid this strategy.
• Full reindexing replaces your index, overriding any changes made during the connector run. Avoid this indexing strategy.

Example scenario of conflicts

The following scenario illustrates a potential issue when two connectors try to update the same index:

1. API client:

   • Ingests product data, including price, category, and inventory.
   • Uses “Partial Update” action.
   • Runs on a cron job every hour.

2. Algolia Index connector:

   • Reads records from the Algolia index and adds a new field, discounted_price, based on the price.
   • Uses “Partial Update” action.

Potential issue

An issue can arise if the Algolia Index connector runs for a long time. For example, if the API client updates a product’s price at 10 AM, the Algolia Index connector starts at 10:07 AM, and finishes after 11 AM, the calculated discounted_price attribute will be outdated. At 11 AM, the API client already updates the price again, which means the discounted_price attribute will be based on an outdated price attribute.

Recommendation

To avoid such issues, ensure that the Algolia Index connector never runs concurrently with any other indexing operation. You can also programmatically trigger your connector after your other indexing operations.