Amazon DynamoDB integrations

Why DynamoDB?

With Amazon DynamoDB, you can easily integrate your workloads with Amazon DynamoDB Accelerator (DAX) for up to 10 times performance improvement, Amazon OpenSearch Service to enable real-time search and analytics capabilities, and a variety of other AWS services. DynamoDB provides you the flexibility to integrate with a broad range of AWS services that meet your use case requirements.

DynamoDB zero-ETL integrations

DynamoDB enables several zero-ETL integrations with other AWS services. These no-code, fully managed integrations automate the extraction, transformation, and loading of data from DynamoDB to other AWS services. They seamlessly replicate your data to the destination service and don’t impact your production workload. Finally, the integrations save you weeks of engineering effort needed to design, build, and manage data pipelines and let you instead focus on your core engineering problems.
1

DynamoDB zero-ETL integration with Amazon SageMaker Lakehouse

This zero-ETL integration allows you to run a broad range of analytics and machine learning (ML) such as SQL analytics, search analytics, business intelligence, generative AI and more. In a few quick steps in the AWS Management Console, you have the flexibility to select the DynamoDB tables to replicate to SageMaker Lakehouse. 

2

DynamoDB zero-ETL integration with Amazon Redshift

This zero-ETL integration makes it easier to perform analytics and ML. You choose the DynamoDB tables you want to analyze and have the DynamoDB data replicated to Amazon Redshift within a few minutes of data being written in DynamoDB. You can then take advantage of the analytics and ML capabilities of Amazon Redshift with rich SQL support, materialized views, data sharing, and schema-less querying capabilities to further transform and enrich your data.

3

DynamoDB zero-ETL integration with Amazon OpenSearch Service

This zero-ETL integration allows you to perform near real-time search analytics. Create indexes to perform full-text search, vector search, geospatial search, and more. OpenSearch Service searches and aggregations work together with DynamoDB queries to enable interactive, performant applications. DynamoDB customers can now use the extensive search and ML capabilities of OpenSearch Service to offer new experiences that boost user engagement and improve satisfaction with their applications.

You can quickly create your first pipeline from the Integrations tab in the DynamoDB console. The integration uses OpenSearch ingestion pipelines to define and process the transforming, routing, and mapping of your DynamoDB data to specific OpenSearch Service indexes.

Event-based triggers with AWS Lambda

AWS Lambda triggers for Amazon DynamoDB enable you to easily set up custom logic to run in response to any changes to an item in DynamoDB. You can quickly create your first trigger from the Exports and Streams tab in the DynamoDB console, after enabling DynamoDB Streams. After configuring your trigger, your Lambda code can trigger within seconds of any changes to the data in your table.

With AWS Lambda triggers for DynamoDB, you can easily build database triggers to validate and enrich data, integrate with other services by replicating the data, or build entire event-driven architectures, such as order fulfillment or media processing, that react to data modification in DynamoDB tables. The Lambda function can perform any actions you specify, such as sending a notification or initiating a workflow.  

DynamoDB Accelerator

Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available caching service built for Amazon DynamoDB. DAX delivers up to a 10 times performance improvement—from milliseconds to microseconds—even at millions of requests per second. DAX manages all the cache invalidation and data population to add in-memory acceleration to your DynamoDB tables. DAX is an inline cache so if the cached item doesn’t exist or is stale, DAX will automatically pass through to the underlying table.

Now you can focus on building great applications for your customers without worrying about performance at scale. As DAX is compatible with existing DynamoDB API calls, no application changes are required making it easy to get started. 

Page topics

General
8

General

DynamoDB zero-ETL integration with OpenSearch Service abstracts away the operational complexity in orchestrating the replication of data from a transactional datastore to a search datastore. Data pipelines that are used to keep transactional and search datastores in sync can be challenging and costly to build and manage, and suffer from intermittent errors which are difficult to track. 

This integration enables Amazon DynamoDB customers to obtain near real-time search results from their transactional data by offering a fully managed solution for making transactional data from DynamoDB available in OpenSearch Service within seconds of being written. Customers simply choose the DynamoDB tables containing the data they want to analyze with OpenSearch Service, and this zero-ETL integration seamlessly replicates the schema and data into OpenSearch Service using OpenSearch Ingestion pipelines. Customers can replicate data from multiple DynamoDB tables into a single OpenSearch Service managed domain or serverless collection to derive holistic insights across several applications, while also consolidating their core analytics assets, gaining significant cost savings and operational efficiencies. 

Customers can get started by using the AWS Management Console for DynamoDB, OpenSearch Service, AWS CLI or AWS SDK or AWS CloudFormation. To enable an integration, customers first choose the DynamoDB table whose data needs to be replicated. Customers then choose either DynamoDB Streams for near real-time replication or DynamoDB Incremental Exports for delayed replication as the CDC mechanism to keep the data between the two systems in sync. 

This zero-ETL integration sets up an OpenSearch Ingestion pipeline in the customer’s account that takes care of replicating the data to an OpenSearch Service managed cluster or serverless collection. OpenSearch Ingestion understands the structure of the DynamoDB tables and then creates an equivalent OpenSearch Service managed domain or serverless collection and bootstraps the destination with the existing data from the DynamoDB tables. Optionally, customers can specify a schema for the indices that will be created in OpenSearch Service. 

This zero-ETL integration provides you a dashboard where you can monitor the state of your end-to-end integration with real time metrics and logs Amazon CloudWatch. You can set up alerting in case of breach of user defined thresholds. This integration also continually monitors the state of the DynamoDB tables and OpenSearch Service indices and immediately notifies users in case of regressions with any of these entities.

In order to ensure that OpenSearch Ingestion has the necessary permissions to replicate data across both these systems, DynamoDB zero-ETL integration with OpenSearch Service creates an IAM role with the necessary permissions to read data from DynamoDB tables and writes to an OpenSearch domain or collection. This role is then assumed by OpenSearch Ingestion pipelines to ensure that the right security posture is always maintained when moving the data from source to destination.

This zero-ETL integration uses the native data transformational capabilities of OpenSearch Ingestion pipelines to aggregate and filter the data while it is in motion. When moving the data from a DynamoDB table, customers may want to drop a few fields or create new fields based on aggregations across existing fields. 

Optionally, customers can also write custom logic for OpenSearch Ingestion to achieve bespoke transformational capability. For other users, who just want to move their entire data from source to sink, this zero-ETL integration will provide out-of-the box OpenSearch Ingestion blueprints so that they can perform the integrations with just a few button clicks.

This zero-ETL integration provides customers options to specify their custom data schema along with index mappings that is used to by OpenSearch Ingestion when writing data from DynamoDB to OpenSearch Service. This experience is added to the UI console within DynamoDB so that customers have full control on the format of indices that are created on OpenSearch Service.

There is no additional cost to use DynamoDB zero-ETL integration with OpenSearch Service apart from the cost of the existing underlying components. This zero-ETL integration uses Amazon OpenSearch Ingestion to read data in DynamoDB tables and replicates to OpenSearch Service. The cost involved in using DynamoDB zero-ETL integration with OpenSearch Service is the cost of OpenSearch Compute Units (OCUs) needed for OpenSearch Ingestion to replicate the data across the systems. Furthermore, customers have an option to choose either DynamoDB streams or incremental exports as the choice of CDC. For incremental exports, there is cost associated with writing data to S3 buckets. For DynamoDB streams, customers would be charged the standard charges for using DynamoDB streams.

DynamoDB zero-ETL integration with Amazon OpenSearch Service is available in all the regions that OpenSearch Ingestion is available in today.