One of the probable future trends in event processing will be the merger of Predictive Analytics (per Wikipedia: “analyze current and historical facts to make predictions about future events“) and Complex Event Processing… or basically the increasing use of advanced statistical techniques in event processing. Of course, for “current facts” (in Wikipedia) read “events” in the Event Processing community…
As an example, consider the diagram on the right, which details some of the algorithms available in such a predictive analytics tool (TIBCO Spotfire Miner), and how they relate to event processing (as when embedded in a CEP tool like TIBCO BusinessEvents). One thing to consider is that these algorithms are not likely to be specified by Subject Matter Experts or Business Analysts – who might ordinarily be relatively happy with the logical constructs in rule-based event patterns or queries.
Interesting thread, thanks, Paul for starting it.
A tool like Spotfire Miner would be used offline in a garden-variety data mining mode. As you’ve pointed out, Hans, this would involve studying the past, recognizing and learning patterns, and making educated predictions. These prediction rules would be translated into CEP rules; predictions that become inaccurate become an early indicator that the underlying patterns have shifted and might trigger an appropriate event.
The time (attention) span applied in a CEP application would then balance the stability of prediction vs how quickly the application reacts to real changes.
The idea of summaries of events touches on Luckham’s CEP aggregation events and event abstraction hierarchies.
Agreed about the question of online vs the usual offline mining. With these particular mining methods, each piece of data might not contribute enough information to make incremental/online changes worth while. Online methods are possibly more useful for models that make use of time.
Also, most of what people think of as “data mining” will work on summaries of events, but maybe not so well on the events themselves. For example, it would likely use the number of purchases this month, rather than the individual purchase events.
Great topic. Thanks for starting it, Paul.
There are various ways this could work, but I think the simplest case would be using a product like Miner offline, doing analysis on historical data, to create a model to define event types (e.g., predict when a transaction might be fraudulent or a piece of equipment might be tending towards failure). These models would be exported (probably through an intermediary like PMML, as Michael suggests), and translated to CEP/BE rules where they would be used to “score” each individual event to see whether further action should be taken.
While not all models could be easily translated to CEP rules, many could (e.g., linear models are just a simply equation with coefficients, decision tree models are really a series of nested if/then statements, etc.)
Good points here, but not clear how in-the-loop these should or will be (versus off-line with results leveraged algorithmically but not with real-time induction). Also, with regard to usability, listing algorithms rather than guiding towards their use is clearly “before crossing the chasm”. There is nothing that precludes analytics from being more actively intelligent and usable by those without PhDs.
Hi Paul,
What you’re calling the span of attention depends on the use case.
But [re] use cases, there are many possible issues about taking models from a mining tool and applying them to online events. There’s the model development, learning from historical data, prediction and interpretation and then feedback from events into the learning.
Currently, there is not a lot of guidance for miners to get from the world of static data into the world of online processing. If you could partner with the mining group to provide some guidance, I think you’d get some good attention from the practitioners that you’re targeting as customers.
[edit: combined posts]
Hi Hans – watch this space… there certainly are use cases out there, but none that we can publicise yet… !
On the other hand, as one who has a “foot in both camps” (AFAIK), you can probably describe the issues and advantages as well as anyone.
Cheers
Hi Hans – you are right – these are data mining tools, where data = historic events:
- the “span of attention” (history period in time to be analysed) for CEP use is typically less than that when used in traditional business intelligence and analytics. For these algorithms the unit of time used is relatively independent, and the key factor is quality and quantity of historic data
- the “mining process” eg using Spotfire Miner is a “development activity”, whereas the application of the model into the event stream (calling the Spotfire S+ server from a BusinessEvents catalog) is the runtime activity…
Possible more interesting are the different use cases for the different models you can define in Miner / predictive analytics, versus the “conventional event processing” we see in BE and other CEP tools. Plus these statistical methods are great for event discovery (defining event types) versus event detection (defining events)…
Cheers
This looks like data mining – might need some explanation about how it would be used with BE.
Paul,
Great blog! I absolutely agree, Predictive Analytics definitely has a place in Complex Event Processing, just as it should be a part of an Enterprise Decision Management strategy in the more traditional sense. In either case, adding advanced statistical models will lead to smarter decisions.
Key prerequisite, however, is the ability for on-demand / real-time execution of Predictive Analytics as well as the operational deployment and integration of such models. It is excellent to see that TIBCO Spotfire Miner supports the Predictive Model Markup Language (PMML – http://www.dmg.org ) which is the de-facto standard for statistical and data mining models. Using PMML, you can deploy your models in the Zementis ADAPA predictive analytics engine ( http://www.zementis.com ) and in turn leverage it for on-demand execution within your CEP framework.
The future is already here, we just need to plug the right pieces together!