Who Has A Bigger Problem

Your manager has just assigned you a difficult project or problem and you're at a loss to envision any possible solution. Is it time to hit the pervasive search engine and starting looking? Perhaps a call to a favorite vendor or perhaps an unknown one? An approach might be to stop looking for a solution and find someone that has a bigger problem than you do, preferably a much larger problem. Depending on the problem, you might even find that person inside your own company.

Odds are someone has a bigger problem than you do and looking outside your normal field of vision is often needed. Looking to reduce the cost of your email system? Try an organization such as a university, other large not-for-profit corporation or a particularly financially distressed company for ideas. Want to improve the integrity of your data center? Can you find a company where revenue dead stops when they're down. How about the stock market? If they make the Wall Street Journal headlines when problems occur, that's a good place to start.

Money, or lack thereof, is a good place to start hunting. Following the money trail is also useful. Are you looking for management support for a new security idea and the CIO isn't very receptive? Who else in your organization is rewarded when incidents are reduced or eliminated? With the advent of SOX compliance putting more people on the chopping block for deficiencies, perhaps Audit, Compliance or the CFO has that bigger problem.

Sales people are rarely a good source, simply because they sell a solution to your problem, not to their problem. Their problem is making their quota, and rightfully so. Helping solve your problem may be aligned at times, but in most cases they sell stuff, not solutions, and their reward is not directly tied to your problem being solved, but in getting the contract signed. Sales people can be a good conduit into making connections into the companies you want to investigate. A better way is to join and be active in one or two few large user groups or leverage a company subscription to The Hackett Group, Gartner Group or other advisory and benchmarking firms. Using LinkedIn, Plaxo or other web-based social networks can also lead to making the right contact.

The idea is not just to implement everything another company does, but to generate new insights into your specific problem. Perhaps one or two components of their solution is enough to satisfy your current needs. Stretching your mind in the direction of the problem, not an immediate solution, may be the key to your next "breakthrough".

Integration Architecture

When architecting a solution to integrate two systems, it's useful to realize that only three categories are possible. Applying the correct approach is crucial to avoiding extra coding, operational issues and higher costs. The three categories are:

• Batch - A set of requests will be processed together at regular intervals. This is typically done to process requests more efficiently or during a window when resources are more available. This is a time-based mechanism.


• Asynchronous - A response to a request is needed as soon as possible. The data will be delivered when the other side is ready and will never be lost. This is an event-based trigger mechanism.


• Real-Time - A response to a request is required in real-time. In case the request is not fulfilled within a reasonable amount of time, the data will be discarded.

Batch interfaces have been around since the advent of mainframes and punch cards. A single method to handle batch data exchange should be used throughout the data center to simplify all operational aspects from security to recovery. One folder structure could be developed for all Production data and a second for Test data. These folders should be mountable to all systems, so multiple protocols (e.g. NFS, SMB) may be required. File system renaming is a useful practice to keep in-flight work from causing issues. As an example, perhaps we have a folder named Payroll. Within the Payroll folder, we create an Inbound, Ready, InProcess, Processed and Error folders. Data being created is put in the Inbound folder and when complete is renamed to put it in the Ready folder. A batch job runs looks in the Ready folder, renames it to put it in the InProcess folder and renames it again upon successful completion to the Processed folder or to the Error folder if unsuccessful. This simple example may not be enough for your requirements, so expand the concept with as many folders and subfolders needed.

Asynchronous interfaces are typically built upon a messaging queuing infrastructure using products such as Microsoft's MSMQ or IBM's MQSeries. Like Batch, you should establish only one method for this type of exchange. Asynchronous interfaces are typically used to process data that needs very quick turnaround, but the data can't be lost if the target system is unavailable.

There are a number of reasons that individual transactions can fail, special attention needs to be paid to build a notification system with the needed data available to take quick action to resolve the error. An approach to this is to make a copy of any transaction that fails and put it into a message queue where a program will read that data, send the appropriate notifications and make the data and error code available to the person performing the troubleshooting. A method of "replaying" the transaction by putting the transaction back on its original queue is useful to avoid manually performing each failed transaction.

Real-time interfaces come in lots of shapes and sizes and will vary from one vendor to the next. Standardization is coming slowly with the adoption of web standards, so you're likely stuck with supporting a variety of proprietary and open standards, fortunately in most cases with the aid of the vendor who knows their own choices well. Still, it will cause a great deal of operational support issues and these interfaces will tend to be your most critical. Put together a team to attack these issues before they become a business issue.

Your standards can be drawn using a generic Source System and Target System on either side of a diagram and your Real-Time, Asynchronous and Batch solutions connecting the two Systems. Each of the three paths describe the specific hardware and software that is standard in your environment. The simple diagram can then be expanded with the specific details for a particular set of Systems.

A few number of highly reusable components will speed the delivery of your interfaces at a greatly reduced cost, and most important, with the least amount of operational issues and business impact.