Introduction

Implementing new MIG6 market communication at Engie.

Let’s start with a brief introduction about what MIG6 actually represents, before diving in the more technical details in how Engie, as a supplier, plays a role in this.

So MIG6, which stands for ‘market implementation guidelines’, is actually a guideline setup by Atrias which describes the information-exchange between all the different distribution grid operators and other market parties. The goal of MIG6 is to accelerate the market processes, and to arrive at a more cost-efficient, customer friendly and less sensitive-to-error working of the energy market.

To arrive at such a goal, a new centralized system had to be put in place which supports market development and modernizes the way this information is exchanged between all these different parties. And that is where MIG6 came into the picture.

The role of Engie

So, now you have an idea of what MIG6 is and how it plays a role in the energy market, we will dive a little more in how Engie as a supplier, plays it’s part in this whole story.

So currently, MIG4 is still being used as of today. This is the predecessors of MIG6. There, Engie still exchanges market messages with various external partners (DGO’s).

As of MIG6, which plans to go live in 2021, this will all change and Engie will only have a single external partner, which is the CMS (Central Market System). CMS will facilitate the market processes by routing and managing MIG6 message exchanges between the different actors for the Belgian energy market.

In the next parts of this post, we will go a little deeper into this exchange of messages between Engie and CMS, as well as the internal application/integration setup at Engie.

Message exchange between Engie and CMS

The MIG6 interaction models support three types of message exchange patterns, or also described as MEP’s, from a business point of view.

First off, there is the ‘Synchronous request/response’ mechanism, which is where a commercial market party sends a business request to the CMS and a single business response will be returned. In this case, the requestor will wait for the response before continuing with other activities

Secondly, there is the ‘Asynchronous request/response’ where a commercial market party sends a request to CMS and one or more business responses will be sent back. Unlike with the synchronous request/response, the requestor does not wait for the response before continuing with other activities.

Finally, there is the ‘Asynchronous event’ mechanism, where the CMS just sends business data to the market parties, unrelated to any earlier requests.

Now, at Engie, this exchange of messages all goes through IBM DataPower gateway, which is used as the proxy / reverse proxy. Here, all the web services to send and receive information from/to CMS are hosted.

Apart from the IBM DataPower Gateway, we use the IBM Integration Bus (IIB) as the Integration Platform or ESB. IIB is used to transform messages where needed and route them to the internal applications like SAP, SAS, etc.. in the picture below, you find an overview of the internal applications at Engie, and the type of communication/data format used to talk to those applications.

Connections

As described earlier in this post, there are actually two different types of connections: ‘Synchronous’ and ‘Asynchronous’. Both of these pass through the DataPower gateway, which serves as the (reverse) proxy, as CMS exposes webservices to exchange market messages with suppliers.

The connection between internal applications and the integration platform goes through different types of connectors such as IDOC connectors, Bapi calls, HTTPS calls etc.

Below, I go a little more in depth in these different types of connection technologies used by each involved party.

Synchronous connection from SAP to CMS

First off is the synchronous connection, where a request is triggered by an internal SAP system. This request will then go through the integration platform and route it towards CMS. Afterwards, the reply will go back to that requesting SAP system again trough the integration platform.

Asynchronous connections

For the Asynchronous connections, the request is again triggered by internal SAP systems for solicited messages. This again goes through the integration platform via specific SAP connectors. This requests data is provided in IDOC format where it is then transformed to XML, to be sent by our webservices using the SOAP protocol.

To receive the asynchronous replies from CMS, a polling mechanism is put in place. At CMS, they have different queues where different type of asynchronous replies reside on. This polling mechanism makes sure we continuously pull those replies from the queues at CMS. Those replies are then transformed back to IDOC format, and delivered back to SAP.

Connections for file transfer

Apart from SAP, SAS is another internal application at Engie, which focuses more on data analytics, big data and artificial intelligence. SAS requires data to be transformed into CSV format so they can upload it into their system. Thus, with the help of IBM integration bus, the XML date we receive from CMS is transformed to CSV. SAS also provides us with mapping specifications, so we can transform the XML according to their needs. After the transformation, this CSV is then put on a NAS where SAS picks it up.

SFTP connections

Some XML files meant for SAS are too big to be sent via SOAP in XML format and validated. These sets of files are zipped and put on an SFTP server at CMS side.  Engie continuously checks for the presence of these zip files, and pulls them from the SFTP server. Another mechanism is put in place to then unzip, validate and transform them to CSV format. Afterwards, these CSV’s are sent to SAS.

Routing of messages to the correct system

Of course, when a message from the central market system enters our integration platform, it has to know to which of multiple ERP/DWH system it has to go to. This is defined via the following rules and mechanisms, in the order in which it is presented.

1. The message type is checked, and based on that, the messages meant for SAS are already filtered out and sent to SAS.
2. For the ones that are not meant for SAS, the response content is checked for a specific related Identifier, which if present is the same as the one included in the request. Based on that identifier, the integration platform knows which SAP system to send it to.
3. If the previous two steps are not an option, the message will be scanned for some other specific tags. A lookup using these values will is done in a database, and based on the result of that query, the message will be routed to the correct SAP system.

Logging and monitoring

For the last part of this post, I will give you a brief summary of our logging/monitoring mechanism that has been put in place.

Author: Rob Engelen