Intro into Integration

I am often surprised how often people create solutions from scratch. Or start solutioning before they have identified the problem at hand. Sometimes solutions are built without finding out how it currently being handled. Did they even talk to the people that are currently doing the work that the new system is going to support or replace?

Why re-invent the wheel? There are well documented reusable patterns for commonly occurring problems. And your problem is probably not as unique or special as you probably think it is.

Extract, Transform, Load (ETL) is a time tested integration pattern of moving data from a source system to a target system where data is transformed--updated for format and/or purpose of target system. Most modern Enterprise Application Integration (EAI), Enterprise Service Bus (ESB) and Data Warehouse (DW) solutions include ETL functionality. EAI is the middleware that enables the enterprises applications to be integrated together. ESB is the communication system between applications in a Service-Oriented Architecture (SOA). DW is the central repository of the enterprises data for data analysis and reporting that supports Business Intelligence (BI).

Do you need a near real-time solution or batch? Should it be synchronous or asynchronous? And yes, you can have near real-time and asynchronous. Though it requires more work, it can be much more fault and load tolerant.

For specific intergration patterns see http://www.enterpriseintegrationpatterns.com/ and specifically http://www.enterpriseintegrationpatterns.com/patterns/messaging/ where you drill down into each of the 65 patterns.

For more information I would suggest their books. Their first is Enterprise Integration Patterns which is over 700 pages and was written over a decade ago. They have also recently released a new book titled 37 Things One Architect Knows About IT Transformation - A Chief Architect's Journey which is available from https://leanpub.com/37things.