Work on the EPTS Reference Architecture is moving forward, with excellent contributions from the event processing industry and academia (EPTS members can observe the work-in-progress on the EPTS working group wiki). One interesting angle of discussion revolves around the compatibility and overlap of “event processing” with conventional “data processing” technologies, especially when it is possible to view “events” as “data” and process them as such. Of course, critics of CEP will counter that all event processing tools simply re-use existing data processing techniques (such as extended versions of SQL engines, or inference engines, or neural nets etc…), and to some extent they are right – what’s new is the idea of re-arranging these techniques to exploit higher-performance CPUs and larger amounts of available memory for in-situ event pattern detection in everyday business problems.
Defining a generalized event processing (and complex event processing) architecture that covers all use cases, even of a single (albeit ambidextrous) tool like TIBCO BusinessEvents, is rather challenging. There are a number of different architecture patterns possible, covering different functional and performance requirements. One generalized effort is included here, based on TIBCO’s work for customers in this area:
The various tiers can be described as:
Tier 0 (not displayed): The sources of events could be anything from SCADA control systems, RFID detectors, BPM or ERP systems, or whatever.
Tier 1: Event channel (or transmission medium). In TIBCO’s world this is usually EMS (topic-based or queue-based) or RV message buses, although it could also easily be some other middleware mechanism.
Tier 2: Event PreProcessor. Optional. Handles basic event filtering, and roles such as concept enrichment (when the event is just adding information to an existing event object), concept creation (the event is transformed into an event object, or concept in BusinessEvents), and forwarding to local and/or distributed memory (a.k.a. event store, for processing elsewhere).
Tier 3: Distribution Agent(s) or Event Store. Optional – used for high-performance, high volume, fault-tolerence and large-scale CEP. Provides storage for events (and event objects) required for the event patterns representing complex (aggregate, abstract, derived) events – where these event patterns may occur over periods of time. Needed (in conjunction with Tier 4…n) where the volume of events, or amount of event processing, exceeds the limits of a single process node’s memory or CPU limits. Used also when information needs to be replicated across nodes to support fault tolerance and failovers. For performance reasons this is often some kind of high-performance cache or data grid (as is the case of BusinessEvents’ cache component), but it could also be a common-or-garden file-oriented database… in TIBCO BusinessEvents we support both approaches.
Tier 4…n: Event Processing Agent(s). In the simplest case, a single EP Agent can attach to a channel and process events (2-tier). Or it can involve a Preprocessor either in-process or as a separate agent (2.5-tier). Or be used with the cache (4-tier). Ultimately these can be organised as a hierarchy of EP services, with these services themselves constituting multiple, load-balanced, identical agents. These EP Agents define event patterns (through rule conditions, state transitions, and queries) and communicate either directly to event channels [Tier 1] or, more often, through the event store [Tier 3]. Large workloads can be distributed through queries (i.e. pull model) or through shared event / event object updates (i.e. push model). In BusinessEvents, EP Agents can be one of 2 base types: inference agents (for rules and states), and query agents.
Tier n+1: Services. Optional. Existing SOA services may also contribute data to the event processing agents, either as-required or cached in the (event) store to be used and updated as needed. Typically service invocation (which could also be a BPM workflow invocation) involves a call out to some mechanism like TIBCO BusinessWorks, and possible some TIBCO Adapter.
Tier n+2: Persistence. Optional. When the event store [Tier 3] is using high-performance cache there is a need to provide back-up persistence for tasks like archiving, start-up load and shut-down un-load, and standard analytics. You might also have operational data stores you need to query directly (rather than through the services layer [Tier n+1]) to enrich event objects for processing. If you want to visualise all your events over time to identify new patterns you would typically use something like TIBCO Spotfire against this layer, and perhaps S+ Miner to extract some rules to update some Tier 4…n agent with…
