Integrating using the new API destinations of Amazon eventbridge

Use case with Magento, Mailchimp, Salesforce and MS Teams.

Amazon EventBridge is the central event bus in your AWS-based environment. A comprehensive integration messaging service that includes error handling, filtering, content-based routing, replay functionality, event schemas, and OTB SaaS integration capabilities. This in form of a developer-friendly integration service ingested with iPaaS capabilities.

Need an introduction to EventBridge itself, check out this video.

Last week (March 2021) AWS extended the service with the “API Destinations” feature. This allows integrating incoming events towards downstream REST APIs without the need to code a Lambda function. Further reducing complexity, development time and costs.

In this blog, I’ll try out the API Destination functionality using 3 commonly used downstream services: Salesforce, Mailchimp, and MS Teams.

Integration architecture with Mailchimp, Salesforce, and MS Teams as targets and simplified ingestion flow from Magento. (Full diagram, see below)

From the depicted diagram, you may already conclude that not all 3 connections were successful. More on that later …

UPDATE 05/2021: Salesforce integration also works! Detailed blog: EventBridge to Salesforce using API Destinations

Downstream service ingestion rate control

Interestingly, “API Destinations” allows you to control the ingestion rate into the downstream services. This is very crucial for enterprise integration because not all downstream services scale like a Lambda or DynamoDB ?

This feature provides a significant simplification of the common “Eventbridge -> SQS -> Lambda -> Downstream service” pattern. A Pattern where SQS is used in combination with Lambda to control the ingestion rate. And Lambda by itself for Transformation, Enrichment, and connectivity.

Today Amazon EventBridge takes care of rate control, transformation, and REST API connectivity. At least for some scenarios

Let’s build the proposed integration solution

Let’s validate if the service lives up to its promise. If you plan to follow along, the tutorial prerequisites:

Magento “New Customer” events published into a custom event bus
Mailchimp account with API Key
MSTeams channel webhook URL

Note that the web console allows us to create the full setup. Though in this blog we’ll use CloudFormation. The used template: https://gist.github.com/Samuelvdc/96d1f0c91eee6b7fed58a3dc755f5704#file-cloudformation-eb-apidestinations-demo-yaml

Creating the “AWS::Events::Connection” resources

There are currently 3 supported authorization types: Basic authentication, API Key and OAuth Client credentials flow. It’s also possible to set static (or from the event payload) HTTP headers, query parameters or body fields.

For both Mailchimp (Basic Auth) and MS Teams (no Auth needed) the setup is straightforward. Unfortunately, Salesforce does not support any of the currently supported authorization types. To push to Salesforce, we still need an intermediate Lambda.

UPDATE 05/2021: Salesforce integration also works without intermediate Lambda! Detailed blog: EventBridge to Salesforce using API Destinations

Creating the “AWS::Events::ApiDestination” resources

Using the API Destination resource you’ll configure the actual REST API endpoint of the downstream service. You should only provide the method and endpoint for this. Also, the discussed rate-limiting capability is configured here.

Two thoughts on this:

There is no support for dynamically creating the “InvocationEndpoint”. For example, I wanted to upsert a member in Mailchimp. So it also pushes changes of the customer to Mailchimp. For this you need to use following endpoint: PUT /lists/list_id/members/subscriber_hash.

The “subscriber_hash” id is the MD5 hash of the email address. This requires custom code and still requires a Lambda function.
Is it possible to map “list_id” from the event payload? I haven’t found a possibility yet (though for this scenario not relevant). UPDATE 05/2021: yes it’s possible, example see: EventBridge to Salesforce using API Destinations

Secondly, it is not possible to define what a successful call means. EventBridge will assume this from the returned HTTP status code. In most cases this will not be an issue, but it’s certainly something to be aware of.

Creating the “AWS::Events::Rule” resource

Nothing new here, I’ve used the Input Transform functionality to build a new JSON payload with input from the incoming event.

Note that this demo CloudFormation template doesn’t include a dead letter queue integration for messages which failed to deliver. When using this in production, look into that ?

Afterthoughts

API Destinations is an interesting addition to the EventBridge offering. And paves the way to further simplify Event-driven integration architectures.

The current capabilities are rather limited for most scenarios to eliminate the need for Lambda functions: basic transformations, only JSON based, no hooks for integration business monitoring, no response inspection, no enrichment, … . However, this addition strengthens the positioning of the service for enterprise integration use cases. And for some simple scenarios, it already provides direct added value.

The rate control is a required capability when integrating with downstream services. It’s very interesting to see this queuing capability incorporated in the EventBridge offering. As of last week, the service was still a Push-Push only messaging service. Hopefully, the rate control capability will also be introduced for a Lambda target ??? Enabling an “Eventbridge -> Lambda -> non-scalable downstream service” pattern.

Below would be the current solution architecture for the initially proposed use case: