Batch versus Real-Time: What is the best choice?
In some cases, it makes sense to automate integration via batch processes, and in other cases real-time integration makes more sense. A mature and capable integration solution will allow for each integration map to be integrated as you see fit. A typical implementation will see a mixture of maps automated as batch processes and real-time processes.
Batch is a good choice if …
- The data being integrated is not time-sensitive
- The data you are integrating has little impact from being hours or days old.
- You have limited integration between systems and you need to be sure that all changes from many areas are kept in sync with as few processes as possible.
- You are unable or unwilling to create data maps for all of the individual elements requiring integration. In this case, a batch process that integrates many fields could save time and get the integration process up and running sooner.
- You have to integrate a large number of data elements, but not very many transactions (records).
- The data to be integrated is closely coupled with other data that is being updated on a batch basis. If you are integrating data that is closely coupled to data that is being updated via a batch process, then it only makes sense that the closely coupled data should be updated via a batch process as well. For instance, if you are using an ERP system and integrating forecast orders (that are updated via a batch process) to another system, there may be no advantage to monitor the forecast orders in real-time since they are only updated with each ERP batch run. In this case, it might be more efficient to set up a batch integration that is triggered just after the forecast orders are updated.
- You don’t mind a little manual intervention.
- You have the necessary processing power to run large batch processes without causing system hangups. Batch processes can take a long time to run and take up a lot of resources. Since these processes usually have a time window in which they need to complete, adding more processes can make for a time crunch or a processing power crunch. Before choosing to integrate systems using batch processes, be sure that you have excess capacity and/or time to allow the integration to complete in an acceptable amount of time.
Batch is not a good choice if …
- You have large record sets to transfer and It takes a long time for them to process.
- You need to refresh the data outside of the set batch processing schedule. Batch processes can be automated just like real-time processes can be, but batch processes have a catch. If a special situation arises where a refresh of the data is required that falls out of the normal batch processing times, then the batch synchronization will need to be triggered manually. This can be a big deal if the person with all of the system smarts is not available or if the batch takes an excessively long time to complete. This situation doesn’t come up every day, but it will eventually happen when dealing with batch processes.
Real-Time is a good choice if …
- The data being integrated is time-sensitive
- The data you are integrating will have an impact on other operations if it is not current.
- You need to integrate many transactions (records). Real-time processes monitor and integrate data one transaction at a time as they appear. This means that real-time integration processes are always “caught-up”, so the transactions per second capabilities are of much less concern for these processes.
- The data to be integrated is loosely coupled with other data that is being updated on a batch basis. Most environments have a mix of data that is updated in real-time and data that is updated via batch processes. If the data you are integrating is not tied to a batch processes, then you are free to choose real-time integration to integrate it with other systems as efficiently as possible.
- You want to avoid manual intervention. Real-time processes are always caught-up, so there is rarely a need for manual intervention to get them caught-up or synchronized with other activities. Real-time integrations tend to “just work” and require the least amount of manual attention.
- You are integrating systems with high transactional throughput. In this case, real-time integration is a better choice because it is always caught-up with processing. This removes the problem of excessively long batch runs that can hog machine resources and interfere with other processes.
- You do not have the necessary power to run large batch processes. Batch processes can take a long time to run and take up a lot of resources. Since these processes usually have a time window in which they need to complete, adding more processes can make for a time crunch or a processing power crunch. If you do not have the excess capacity and/or time to allow a batch integration to complete in an acceptable amount of time, then you should choose real-time integration. Real-time integration is much less processor-intensive and does not require a time window to complete.
We are the creators of The LINK®, the go-to resource for your integration needs.
The LINK allows all maps to be deployed as either a batch process or a real-time process. There is no change in the map itself, it is simply a deploy-time decision. Batch maps can be scheduled. Real-time maps can be associated with up-time and down-time schedules so that they do not run at inappropriate times.
If you would like to explore your integration options, we are here to help.