Mapping for Exceptions

There are a lot of integration providers out there that offer a list of API endpoints as well as support for linking those endpoints together. They make it seem simple enough to build an integration by connecting the two endpoints (and adding some transformations, if you have the time to figure out how to do that on their integration platform ). After some effort you should be able to get everything configured so that the data in one system starts appearing in the next.  Great, you’re all done, right?  Hardly.

One thing you may not realize is that those API endpoints that are mapped together do not call themselves. 

APIs can be called, but they do not call themselves.  You need what we call an Execution Layer to run the map that connects those API endpoints.  This is not always made clear with all integration providers.  There is an execution layer that has to run in the background in order to make everything happen; and the problem with that is that when errors arise, it is virtually always because of a problem encountered in the execution layer (that layer that no one told you about).  What sorts of problems?  Here are the top 3 problems that we see every day:

  1. Connectivity – The network is down.  Happens to everyone, even integration providers.
  2. Availability – The source API or target API is not responding.  Could be a planned maintenance window, or it could be a network or other problem at their end.
  3. Data problems – The data being integrated caused an error.  Maybe your are trying to update an order that is already closed.  Maybe you are referencing an item that does not exist in the target system.  Maybe there is a dependent map that is behind or down, causing cascading problems with other maps.

How your integration provider deals with these issues is really what differentiates integration providers from one another.  Anyone can provide access to an API endpoint, that is a given.  That does not make them a quality integration provider.  The real challenge is how they deal with problems.

Some real differentiators between integration providers are:

How do you deal with connectivity or availability errors?

Here are Good, Better, Best options available in the integration service market today:

  1. Just log them somewhere and leave it up to me to check the logs regularly? (not so good)
  2. Log them and periodically send out reports? (better, but still not the best)
  3. Send out notifications in real-time or near real-time. (getting better)
  4. Perform diagnostic, remediation and retry mechanisms to attempt to resolve the error before sending out notifications to the customer. (still better)
  5. Perform diagnostic, remediation and retry mechanisms using heuristics and/or formulas that the customer can adjust to their liking, and then send out a notification or custom action per the customer’s specification (BEST.  Everyone should meet this standard)

For data errors, there are additional considerations since many are situational, or based on timing, or other customer-specific rules.

How do you deal with failures based on data? 

You should have mechanisms available where the errors can be grouped and different remediation rules applied based on the type of error that occurred.  Is your provider telling you that if a transaction fails import into the target system then it is up to you to provide manual follow-up?  Terrible answer. 

You should be able to setup retry heuristics or customer-supplied failover rules for retry.  Better yet, you should be able to create separate data maps specifically for automated remediation of error scenarios – data maps that you can program with your rules to understand the difference between when a data error is truly fatal and when it can be adjusted and retried.

The bottom line here is that you should never receive an error notification (warnings notwithstanding) for a situation that was possible to remediate.  The only time you should be notified of exceptions is where no remediate was possible, based on any set of rules that you could accept.  In addition, you should never get error messages that you can ignore (another shortcoming of some integration providers).

Your time is valuable.  You don’t have to accept integration vendor shortcomings.

When you create an integration, the integration should be fully automated INCLUDING EXCEPTIONS.  Do not accept that your integrations require manual follow-up for situations that you know could be handled automatically (in another more capable system).  If you can describe how an error could be accommodated, then your integration solution should be able to accommodate it.  Do not accept solutions that place the burden on you, and do not accept solutions that send out error messages that can be ignored.

Your time is valuable.  You expect that your integration solution should be fully automated.  Do not accept less.