Integration Architecture

When architecting a solution to integrate two systems, it's useful to realize that only three categories are possible. Applying the correct approach is crucial to avoiding extra coding, operational issues and higher costs. The three categories are:

• Batch - A set of requests will be processed together at regular intervals. This is typically done to process requests more efficiently or during a window when resources are more available. This is a time-based mechanism.


• Asynchronous - A response to a request is needed as soon as possible. The data will be delivered when the other side is ready and will never be lost. This is an event-based trigger mechanism.


• Real-Time - A response to a request is required in real-time. In case the request is not fulfilled within a reasonable amount of time, the data will be discarded.

Batch interfaces have been around since the advent of mainframes and punch cards. A single method to handle batch data exchange should be used throughout the data center to simplify all operational aspects from security to recovery. One folder structure could be developed for all Production data and a second for Test data. These folders should be mountable to all systems, so multiple protocols (e.g. NFS, SMB) may be required. File system renaming is a useful practice to keep in-flight work from causing issues. As an example, perhaps we have a folder named Payroll. Within the Payroll folder, we create an Inbound, Ready, InProcess, Processed and Error folders. Data being created is put in the Inbound folder and when complete is renamed to put it in the Ready folder. A batch job runs looks in the Ready folder, renames it to put it in the InProcess folder and renames it again upon successful completion to the Processed folder or to the Error folder if unsuccessful. This simple example may not be enough for your requirements, so expand the concept with as many folders and subfolders needed.

Asynchronous interfaces are typically built upon a messaging queuing infrastructure using products such as Microsoft's MSMQ or IBM's MQSeries. Like Batch, you should establish only one method for this type of exchange. Asynchronous interfaces are typically used to process data that needs very quick turnaround, but the data can't be lost if the target system is unavailable.

There are a number of reasons that individual transactions can fail, special attention needs to be paid to build a notification system with the needed data available to take quick action to resolve the error. An approach to this is to make a copy of any transaction that fails and put it into a message queue where a program will read that data, send the appropriate notifications and make the data and error code available to the person performing the troubleshooting. A method of "replaying" the transaction by putting the transaction back on its original queue is useful to avoid manually performing each failed transaction.

Real-time interfaces come in lots of shapes and sizes and will vary from one vendor to the next. Standardization is coming slowly with the adoption of web standards, so you're likely stuck with supporting a variety of proprietary and open standards, fortunately in most cases with the aid of the vendor who knows their own choices well. Still, it will cause a great deal of operational support issues and these interfaces will tend to be your most critical. Put together a team to attack these issues before they become a business issue.

Your standards can be drawn using a generic Source System and Target System on either side of a diagram and your Real-Time, Asynchronous and Batch solutions connecting the two Systems. Each of the three paths describe the specific hardware and software that is standard in your environment. The simple diagram can then be expanded with the specific details for a particular set of Systems.

A few number of highly reusable components will speed the delivery of your interfaces at a greatly reduced cost, and most important, with the least amount of operational issues and business impact.