Real Time Integration aka EDA

Brenda Michelson has another excellent post on Event Driven Architecture. For those of you from the EAI world, we use to call this Real Time Integration. Jump over and read her article on Event Driven Architecture and then come back and read about some issues associated with it. I've posted below an excerpt from a white paper I wrote a while back on Real Time Integration challenges. Feel free to chime in with other challenges you have run into when doing Real Time Integration.

Excerpt

Issues associated with real time integration are numerous and can have a large impact on the success of a real time integration project. This white paper is designed to give the developer an overview of these issues before undertaking an integration project.

Background

What is meant by real time? For the purpose of this discussion real time is a business event that is to be processed as it happens or within a short time period there after by another application(s). This time lag could be few seconds or several minutes. This differs from a batch type of integration which is typically defined as a grouping of multiple of events or data points in a scheduled one time event.

This need for real time integration introduces multiple issues and challenges which must be overcome to produce a successful, reliable integration project. A thorough understanding of these design issues is necessary before beginning an integration project.

Integration Challenges

Real time integration brings forth a host of challenges driven by the need to consume business events as they happen. These challenges can be categorized into the following topics: Event Notification, Event Delivery, Event Persistence, Event Recovery, Event Consumption, and Event Monitoring.

Event Notification

In a typical integration scenario, there is at least one event producer and at least one event consumer (multiple consumers are discussed under Event Consumption). The first challenge is notification a business event in the producing application has occurred. How is this event generated, how is it captured by the integration layer, how is it persisted in case of connectivity failure between producing application and the integration layer, and how is it rolled back if needed?.

Most applications do not provide an internal mechanism for providing event notification and storing event information for use by integration platforms. Applications typically have application programming interfaces (API's) for interfacing however these API's usually do not address real time event notification (capture). Api's typically allow event generation (i.e. create an invoice), data interrogation i.e. return a specific invoice, and even import and export of data.

Third Party integration platforms such as webMethods provide "adapters" for common types of third party applications. I.e. Oracle, DB2, Peoplesoft, etc. These adapters are designed to be aware of events in the applications, persist the events at the source when possible, and aid in the deliver of the event and its content to the integration layer. So to summarize, the adapter is responsible for observing the event, capturing the event, persisting the event and aiding in the delivery to the integration layer. The adapter also may be responsible for assisting in "at least once"delivery to the integration layer and/or "once and only once" delivery to the integration layer.

Event Persistence

Persisting the event once it has happened, is the responsibility of the integration adapter. Event persistence is important to maintain the history of the event, the contents of event and the notification of the event occurring. To insure that the event data is recoverable in the event of a transient infrastructure type failure, the persistence should always occur on the source system. This also aids in de-coupling the source application from the rest of the integration layer infrastructure as well as any target receiving system. For example if the integration platform were to be down for maintenance, failure etc then the event would be collected on the source system and sent to the integration layer once connectivity was re-established automatically.

Event Delivery

Delivering the event data to the integration layer once the event has occurred is generally the responsibility of the adapter. There are multiple ways to do this and different adapter types use different methods.

Event Recovery

There are multiple recovery situations that can occur within a typical integration. Designing the integration to be recoverable is a key step in insuring the event does not get lost.

Event Recovery - Infrastructure Failure

Infrastructure failures can come in a variety of forms that the integrations have to take into account. Network failures, application failures, database failures, integration software failures, hardware failures. These types of failures can be accounted for during the design and event recovery can be automated within the integration.

Event Recovery - Logic Failure/Error

Logic failures can occur in the source application, integration layer and target application(s). These failures are typically not associated with automatic recovery. However notification of the event logic error is critical. Once notification of the error is received, the event can be repaired and resubmitted/reprocessed within the integration layer, if the error occurred within the integration layer.

Event Persistence and Event Recovery are tightly tied together during the design phase. If the event is not persisted correctly and at the right place(s) then recovery becomes difficult.

Event Consumption

Event consumption can be separated into two parts: Integration layer event consumption and target application event consumption.

Integration layer event consumption occurs after the hand off from the event source application. Typically in well designed integrations, the event is persisted before it is consumed. Consumption may consist of simply forwarding the event on to interested target applications. In most cases however, the event needs to be translated into the target application's desired format.

The transformation process is an important part of the integration layer. It is at this layer that a common data model can be inserted, also called a common document. This common document plays an important role in the de-coupling of source and target interfaces. By translating the source data into this common document, the target applications do not know about or care about the data structure of the event producing application.

As the common document is passed through the integration layer to the target applications, it will be translated again into the format supported by the individual target application. These different persistence points allow for further de-coupling of the source and target applications.

Target application event consumption occurs when the event has finished its route through the integration layer. The event data has been translated into the target applications data model. The interface into the target application is dependent on how the target application has exposed its methods.

Important design considerations: What happens if the update of the target application fails due to a data error? What happens if two separate applications are being updated and one succeeds and one fails? Is there an event sequencing requirement? What happens if the update of the target application fails due to infrastructure failures?

Event Monitoring

As events are happening in real time, event monitoring becomes critical. Event monitoring can be very complex depending on the overall design of the integration layer. Event monitoring can be divided into several layers: Source and Target Application monitors, Infrastructure monitors, Integration Layer Component monitors, and Transaction monitors.

Application Monitors

Application monitors are specifically designed to monitor the running application. I.e. an Oracle database monitor. These monitors typically have specialized knowledge for the application they are monitoring however they can be as simple as is the application up or down.

Infrastructure Monitors

Infrastructure monitors typically monitor the underlying server hardware, operating system and network.

Integration Component Monitors

Integration component monitors typically monitor the individual pieces of an integration. An example would be the custom queue monitor which alerts when queue depths for a particular integration have exceeded a certain limit.

Transaction Monitors

Transaction monitors are used to monitor an individual transaction from end to end. This is the most sophisticated type of monitor and yields the best results for monitor an integration. It is also the most difficult to implement.

Conclusion

Real time integration can provide great business value to the corporation. Events consumed as they happen can reduce cycle times, make the business more responsive to customers and increase the competitive advantage. Real time integration can also bring a host of issues as this paper has outlined. Understanding these issues prior to undertaking the project will significantly increase the integration's chance of success.