Salesforce Magento integration in AWS

– Salesforce a well known SaaS application that has a myriad way of integrating, connecting to their system and consuming the data.

– The Magento version we used is slightly older and confronted us with some challenges towards authenticating and thus connecting before being able to consume data. We were bounded to version 1 of Magento.

In AWS it is quite easy to spin up a cluster. This consists of two big steps. Creating the actual cluster and then creating a client to manage the cluster. Following the instructions listed on Getting started using MSK Serverless clusters – Amazon Managed Streaming for Apache Kafka brings you pretty quickly to a working kafka cluster on which we can start creating topics right off the bat.

One thing to keep in mind is to always make sure that the right security group and the appropriate IAM role has been attributed to the EC2 client.

On the consumer side we used lambda functions as it is set up pretty easily and works very neatly. Set up is described here.

One topic per business object (Customer, Product, …) and per side (Salesforce, Magento).

Which API Solution to choose?

Salesforce has lots of integration API’s to choose from. Out of the box they support the following list:

So, the first problem is choosing the right API for your particular use case.

Since our use case is all about getting new accounts from the Magento web shop solution and pushing them to salesforce, we decided to use the REST API with json as dataformat. Other APIs are providing other functionalities that are not needed for this exercise.

Alternatively, we could have chosen the bulk api but since we aimed for an event driven realtime scenario, we decided not to use the bulk api in this case as this would bring additional complexity to the solution and this API is better suited for classic batch operations, not really our use case in this exercise.

The streaming api works best when you need near-realtime streams of data based on changes in Salesforce records or custom payloads. Subscribers can receive notifications using CometD (CometD)—an implementation of the Bayeux protocol that simulates push technology.

Setting up your environment to use the Salesforce REST API.

Getting to use this API can be quite challenging, especially if you’re not very familiar with Salesforce. There’s plenty of good developer documentation and a sandbox environment for experimenting and testing but it can sometimes be a bit daunting to search for the nuggets of information you need.

In order to test this API, you have to set up a developer edition Salesforce test organization. To create a Developer Edition org, go to developer edition org and follow the instructions for signing up for a Developer Edition organization.

The second important prerequisite is enabling the API permission. This permission is enabled by default, but an admin might have changed it.

The next step is setting up a Salesforce “connected app”. A connected app is a framework that enables an external application to integrate with Salesforce using APIs and standard protocols such as SAML, OAuth and OpenID Connect. Connected apps use these protocols to authenticate, authorize and provide single sign-on (SSO) for external systems.

When you create a connected app, make sure that you understand how it’s going to be used so you can configure the appropriate settings. In our case, we needed to configure the connected app with OAuth authorization settings.

This integration scenario is a server-to-server flow, so we choose the OAuth 2.0 client credentials flow, which means that our implementation exchanges its client credentials defined in the connected app—its consumer key and consumer secret—for an access token.

For creating or updating new Salesforce accounts, we also need to set the ‘Modify all data’ user permission for the connected app.

Configuring and enabling OAuth settings is the next step. We wanted to use JWT tokens so we selected  ‘use digital signatures’. We also need to configure the correct OAuth scopes, in our case the ‘Manage Customer Data Platform Ingestion API data’ (cdp_ingest_api).

Finally, we need to set up an execution user for the flow.

Although there’s no user interaction in the client credentials flow, Salesforce still requires you to specify an execution user. By selecting an execution user, you allow Salesforce to return access tokens on behalf of this user.

Design

One Lambda function per Kafka topic to POST update events to Salesforce:

– trigger: kafka topic event
– mapping from the topic to Salesforce
– post to Salesforce REST API.

One MSK Connect source connector per Salesforce business object to publish to a Kafka topic

– MSK Connect
– connector (+OAuth): Salesforce Push Topic Connector (one Salesforce object to one topic): Salesforce PushTopic Source Connector Configuration Properties | Confluent Documentation
– Salesforce streaming API: Bayeux Protocol, CometD, and Long Polling | Streaming API Developer Guide | Salesforce Developers

We could not use the Magento Kafka integration available on the Magento Market Place. It is not maintained anymore, and we could not get it to work.

Instead, we have developed simple AWS Lambda function.

One Lambda function per business object to read from Magento and publish to a Kafka topic

– Node.js
– trigger: scheduled CloudWatch EventBridge
– read from Magento over HTTPS with OAuth
o this should be optimized to avoid pulling all the objects every time the lambda is triggered
– publish to MSK topic using KafkaJS https://kafka.js.org/

One Lambda function per Kafka topic to POST it to Magento

– Node.js
– trigger: kafka topic event
– mapping from topic event to Magento object
– post to Magento over HTTPS with OAuth

– Don’t use Magento 1, please!

– Don’t eat too many pastries (at the i8c FastTrack Day) if you want to make it past 3 PM.

– Choose the right Salesforce API for your particular use case.

– Altternatively, you can also use the MSK Connect Salesforce connector to send updates to Salesforce.

– Salesforce has all tools you need to integrate with. Magento needs more custom work.