BI | The TIBCO Blog

ACM Overview of BI Technology misleads on CEP

July 28, 2011 by Paul Vincent Leave a Comment

The latest edition of the Communications of the ACM had an article covering CEP from the BI perspective, written by researchers from Microsoft and HP. On Complex Event Processing they write:

The competitive pressure of today’s businesses has led to the increased need for near real-time BI. The goal of near real-time BI (also called operational BI or just-in-time BI) is to reduce the latency between when operational data is acquired and when analysis over that data is possible. … A class of systems that enables such real-time BI is Complex Event Processing (CEP) engines…

They then quickly make the mistake of assuming CEP = Event Stream Processing or ESP…

Applications define declarative queries that can contain operations over streaming data such as filtering, windowing, aggregations, unions, and joins. The arrival of events in the input stream(s) triggers processing of the query. These are referred to as “standing” or “continuous” queries …

Of course, rule-based CEP, using constructs like Event Condition Action rules, and indeed other languages / algorithms can be used to define aggregate events to support real-time BI tasks. Ooops.

There are several open technical problems in CEP; we touch upon a few of them here.

This should be interesting…

One important challenge is to handle continuous queries that reference data in the database (for example, the query references a table of customers stored in the database) without affecting near real-time requirements.

I’m pretty sure most stream processing engines provide some ability to do this (if you want to). But normally the last thing you want to do in (near) real-time business applications is waste time querying a RDBMS. Common practice is to put the data you need first into memory or a datagrid / cache, where access times are much lower.

The problem of optimizing query plans over streaming data has several open challenges. In principle, the benefit of an improved execution plan for the query is unlimited since the query executes “forever.” This opens up the possibility of more thorough optimization than is feasible in a traditional DBMS. Moreover, the ability to observe execution of operators in the execution plan over an extended period of time can be potentially valuable in identifying suboptimal plans.

Again, I’m pretty sure all the main CEP technology provides have optimised CEP engines – for example TIBCO uses a high-performance version of the Rete algorithm. I’m pretty sure Microsoft’s stream processing engine has optimizations too!

Finally, the increasing importance of real-time analytics implies that many traditional data mining techniques may need to be revisited in the context of streaming data. For example, algorithms that require multiple passes over the data are no longer feasible for streaming data.

This is actually an issue for the analytics guys: continuous analytics do indeed imply computing statistical models an event-at-a-time rather than “all at once against some vast data store”. The good news is that the analytics world has been doing some of this for some time, in order to accomodate complex processing of large data sets by batching and recombining data as it is processed.

The BI Overview also mentions several other “BI technology” aspects that are also often combined in CEP solutions: in-memory, distributed (map reduce a.k.a. “divide and conquer”), and analytics. So although it wasn’t emphasised in this paper, it seems the interesting development here is a certain amount of convergence in using these technologies together!

Filed Under: Complex Event Processing (CEP) Tagged With: ACM, BI, Queries

Real-time, Event-driven BI as an EA Trend for 2011 and on

January 7, 2011 by Paul Vincent Leave a Comment

Thanks to Brenda Michelson for an interesting post on Data Quality in Real Time that points to a Forrester report by Gene Leganza on The Top 15 Technology Trends EA Should Watch: 2011 To 2013 (from Oct last year and commented also on here).

To quote from Brenda quoting Gene: “The shift from historical to real-time analytics will require that related processes such as data quality services also move to real time.” In practice poor data quality is typically due to manual system inputs – something that technology like TIBCO Netrics can help address at source – or historic information (with previous manual system inputs). Very occasionally it could be due to problems with event sensors. In any such case, real-time events and data can be cleansed during event processing as CEP technologies usually include the necessary filter and transformation rule capabilities.

I haven’t seen Gene’s report yet but for sure, real-time analytics (monitoring, dashboards, rule-based predictions etc) are increasingly augmenting traditional BI (reporting), visual analytics (data exploration) and predictive analytics (data mining) – and such CEP technology is another powerful weapon in the business analysts’ armory in the quest for business optimisation.

Disclaimer: although we usually use the term operational intelligence for the real-time analytic capabilities provided by CEP and associated technologies, others use terms like continuous intelligence, real-time BI, and so on. And today (thanks to a TIBCO customer) I heard another new term – event-driven Business Intelligence or edBI. Which I must admit I rather like!

Filed Under: Complex Event Processing (CEP) Tagged With: Analytics, BI, Brenda Michelson, EA, edBI, Forrester

Do Businesses need 63 different types of event?

November 23, 2010 by Paul Vincent 8 Comments

One of the drivers of success in the BPM world has been BPMN – the Business Process Modelling Notation – developed under BPMI before it got absorbed into OMG, and probably a bigger success than any other modelling standard – and now as BPMN 2.o (beta, at this time) slightly reconfigured in name as the Business Process Model And Notation. BPMN is used to provide a “process” perspective (versus, say, a data or an event perspective) of a system and shows the flow or orchestration of process tasks and activities. It is made up of (per the excellent bpme.de BPMN2 summary poster):

2 flow subtypes: sequence is the normal flow, with default and conditional conditional subtypes
7 task types: send, receive, user, manual, “business rule” (really decision), service, and script
6 activity markers: sub-process, loop, parallel, sequential, ad-hoc and compensation
7 gateways: exclusive, event-based, parallel, inclusive, exclusive event-based, parallel event-based, and complex
6 types of data: input, output, object, collection, store, and message
and
63 types of event…

OK, “63 types of event” needs a bit of explanation (and justification). These consist of:

13 main event types: untyped, message, timer, escalation, conditional, link, error, cancel, compensation, signal, multiple, parallel multiple, and terminate

… across 8 situations classified by location in a process:

Start: top-level, event sub-process interrupting, and event sub-process non-interrupting
Intermediate: catching, boundary interrupting, boundary non-interrupting, and throwing
End

… but with some situations not requiring certain types of event (e.g. there is no “start cancel process” event) leaving 58 63 event types defined [*1].

Presumably BPMN tools will let the user specifiy the main event type and associate the correct symbol from the context in most cases, leaving us to consider just the 13 main event types.

So let us analyse these event types from an event processing perspective:

Message and timer events: these are probably most familiar with to those with an event processing perspective, with a message relating to what we consider either a source or a derived event.
Escalation: this is really a deferal to a different process (e.g. a supervisory process), and can be handled by some internal message in event processing.
Conditional: this is “reacting to conditions becoming true” – in other words equivalent to a rule-firing (or event query) success. In a rule system this might be handled just by setting some fact or data, which in an inferencing rule system will use some internal event to signal to the rule engine that other rules may “fire” – however there is no need to explicitly model this behavior. I may also of course create an event (or in a query system, insert / update some event), which is again is not explicitly modelled as “conditional” per se…
Link: some means of connecting models: this is more a “page break” than a business event!
Error: I prefer the term “exception” here, which of course is a local context – an error event in one process is a normal event in an error-handling process, etc.
Cancel: this is more a transaction-specific event: if dealing with transactions then some events may indeed be “cancel transaction X” requests, which may or may not be satisfiable, etc.
Compensation: this is where some process route may require some compensatory process – for example in relation to a “cancel” event.
Signal: signalling information across processes: this is might be used as a control event, such as in controlling a choreography.
Multiple: one of a set of events is “caught” or provides the cue to continue the process.
Parallel Multiple: all of a set of events is “caught” to continue the process.
Untyped events are used for start and state changes. Typically, though, a state change is the main event we are interested in (in event processing)!
Terminate event – process complete! From an event perspective this (process end) might never happen of course…

The extra detail might be explained by the fact that in BPMN you are detailing *how* to implement a process, and the “event symbol” is providing you with some specific context for that event. In event processing languages, you usually describing “what* is to be processed, and where all events contribute to some situation or context. It will be interesting to see how these 58 symbols work out in practice, as a hindrance or a help for process designers…

Note:

*1: the table of events above, from the BPMN2 poster from bpmb.de, has 63 event symbols, and is slightly re-organised from the equivalent table in the BPMN2 beta specification (Table 10.93 pp 269-270, in 10-06-04.pdf). Interestingly the specification document also includes a table to detail the event symbols (Table 12.23 ppp 405-411, in 10-06-04.pdf) but this shows only 58 symbols! Let’s just say “lots of event types” in BPMN2 then…

Filed Under: BPM, Complex Event Processing (CEP) Tagged With: BI, BPMN2, Choreography, Event notation, models, Standards, UML

How does CEP fit into BPM and SOA environments?

March 4, 2010 by Paul Vincent Leave a Comment

There were some customer discussions recently on the conceptual relationships between the CEP and BPM and SOA “software stacks”. This coincided with the announcement of the OMG-backed Event Processing Consortium being set up (alongside the merger of the SOA and BPM consortia), events which themselves can be interpreted as that event processing has some special role to play alongside BPM and SOA.

[Disclosure: note TIBCO is not a member of either the OMG's EP or BPMSOA consortia - they seem to be focused on end-user rather than vendor participation - but is an OMG member and participant in standards development. We currently see the EPTS as the main advocacy group across vendors, academics, analysts and end-users, but will be monitoring the progress of the EP Consortium].

So here are a few simplistic patterns on how CEP (event processing) relates to BPM (processes) and SOA (services)…

CEP Pattern	Example	Diagram
1a. Standalone CEP This is a bit of a misnomer – you don’t identify event patterns without some intent to use them, and at the very least such a use would be in a BAM type role displaying interesting correlations for some business person – who of course is engaged in some kind of business process, which may or may not be managed by BPM…	Monitor a production process to provide an “additional view” or dashboard for the process control manager.
1b. CEP enriching BPM processes and/or SOA services. This is the conventional view of CEP, detecting the complex events that are of interest to, and useful in, appropriate processes and services. Complex events in these cases can be as straightforward as deducing that a deliverable has been completed, or some process truly initiated. Typically the CEP system is transforming source events into business events, for onward use in (the) business processes.	Identifying exception events in a business that need to handed to a workflow or case management system for resolution.
2a. CEP monitoring processes and services This is where the sources of events are the managed processes and services themselves. This process and service monitoring is used to detect exceptions, disparities across systems, and system performance… Note that effectively this pattern is a combination of patterns 1a and 1b above.	Detect when response times exceed some metrics and suggest corrective actions such as reallocating resources.
2b. CEP-based decisions for processes and services This is where I need to make intelligent decisions for the process and service layers, using the CEP layer as a monitoring, shared decision management component. Note that effectively this is a slight extension of pattern 2a above.	A BPMN “rule activity” sends a decision request to the CEP engine to get a valid decision for a process decision point; the CEP engine monitors the decisions made.
3a. Dynamic process and service control This is where the events from processes and services, and external services, are combined to select which processes and services are relevant to use for the current context. In effect, the CEP engine becomes the controlling agent for the business processes and service engine, handling for example dynamic process selection. Note that this pattern is a further evolution of patterns 2a and 2b.	In a complex business process for ever-changing fulfillment problems, CEP-based rules determine which sub-processes are valid based on incoming information.
3b. Embedded processes and services within CEP The final evolution of the above is when you argue that the functions of the BPM and SOA stacks can be subsumed into the CEP layer. In reality this is usually only a partial subsumption, as otherwise the centralization of services into just 1 layer could be perceived as contrary to the very idea of SOA! So this covers things like event-based policy implementations being embedded as CEP rules rather than as external services, but alongside some external services such as an operational database. Indeed, one could argue that in this case the CEP event processing agents are themselves really part of the SOA layer, not the other way round!	A service gateway controlling access to existing services, but embedding decisions, service policies, and business rules.

So, did I miss anything?

Filed Under: BPM, Complex Event Processing (CEP), SOA Tagged With: BI, OMG EP Consortium

Processes = event-driven decisions and actions?

February 24, 2010 by Paul Vincent 2 Comments

In a previous post we commented on IBM’s Conceptual Model for Event Processing Systems, but failed to observe that IBM’s rule engine group, Ilog, was noticably absent from its contributor list. So perhaps it was both fortuitous and timely that Ilog’s Daniel Selman recently added another “viewpoint” on the position of event processing (aka the rules viewpoint).

Daniel’s take on the “primary users (sic) of rules technology” are the (use cases for) automating decisions, event responses, processes and inferences. I think these might be better classified by renaming then as decision processing, event processing, and business process processing. But inferencing, which Daniel notes as being a technology to support Artificial Intelligence, is not so much a (user or) use case, but a means of providing knowledge-based reasoning to support any of decisions, processes, etc – “AI” is not (or should not be) a means unto itself.

Standard Event Processing Design Pattern

Indeed, one could probably argue that:

all business processes are driven by events of some kind, and involve decisions and (re)actions;
all processes, decisions and event processing are context (i.e. state) driven; however, some processes and algorithms are used in a subordinate fashion to a stateful process (consider a typical decision service that is subordinate to its application server and database layers);
inferencing can be used to enhance any part of the event-handling process as a form of declarative rule control.

In TIBCO’s experience, an event driven rule engine (like TIBCO BusinessEvents) can be used to provide dynamic business processes, event-driven decisions, and real-time control mechanisms – and often all 3 – exploiting and building on the fundamentals of complex event processing.

So, on the utility of rule engines, we totally agree with Daniel!

Filed Under: BPM, Complex Event Processing (CEP) Tagged With: AI, BI, event-decision-action, Ilog, Rules, Standards

WfMC calls for “Dynamic event-based BI driven by BPM”

February 15, 2010 by Paul Vincent Leave a Comment

An email from Layna Fischer, who is editor of the annual WfMC‘s “BPM and Workflow Handbook”, tells us that the theme for the 2010 handbook is a “Spotlight on Process-Driven Business Intelligence (BI)”. To wit:

In 2010 we include a special spotlight on “Process-Driven BI” to illustrate how Business Process Management (BPM) and Business Intelligence (BI) are increasingly intertwined. Linking business intelligence and business process management creates stronger operational business intelligence.

Users seek more intelligent business process capabilities in order to remain competitive within their fields and industries. BPM vendors realize they need to improve their business processes, rules and event management offerings with greater intelligence or analytics capabilities.

We welcome submissions that include issues, case studies, guidance, solutions or research such as:

* BPM integration with collaboration, portals and BI
* BPM-BI with application assembly and deployment
* Dynamic event-based BI driven by BPM
* Embedding BI applications into business processes

Taking the 4 points above:

BPM integration with BI – this is quite an easy one as reporting (BI) on the process statistics gives insight into the long term performance of your processes – and thence areas for improvement. An obvious TIBCO example is TIBCO iProcess Spotfire – although this is more analytics than simple paper reporting…
BPM-BI with application assembly and deployment – I have to confess I have no idea what this phrase means! Is this BPM and BI for the software development process, perhaps? I have a feeling it is something to do with the aspect of BPM that is “visual coding” of straight-through-processes (for the event community: using BPMN to write single-event-handling applications). Traditionally there is usually insufficient data around to justify regular changes to traditional business processes, but this changes with the advent of analysis. Indeed the need to change processes often can lead to the use of things like rule-driven processes… a pretty common use case for TIBCO BusinessEvents in fact!
Dynamic event-based BI driven by BPM – this too can have a number of interpretations!
- “Event-based BI” could be “event-based analytics”, through CEP. You would normally expect such “events” to drive process changes or business processes directly (e.g. “oooh look, the market trend for “wimbles” has started to go up!”). So this would be detecting events to “drive BPM”, not the other way round.
- One could have a workflow to control manual views of your BI reports on incoming events – and this is again “BI as a process”.
Embedding BI into business processes – although this implied some “BI process” in a business process, I would expect it to be the simpler case of embedding BI results (decision changes, for example) into an existing business process – surely the classic use of BI in BPM… a slightly specialised case of the first point above.

But the above analysis is somewhat moot. More interesting is Layna’s assumption as indicated by the phrase:

BPM vendors realize they need to improve their business processes, rules and event management offerings …

So at least one BPM expert has come to believe that business process management encompasses managing events, rules, and processes… not just “process diagrams”!

If anyone is interested in submitting something on this, note abstracts are due Feb 17.

Filed Under: BPM, Complex Event Processing (CEP) Tagged With: BI, book, Trends, WfMC

TIBCO CEO on 2009: the shift to Event Driven Architectures

January 11, 2010 by Paul Vincent 4 Comments

The end-of-year analyst report (aka “earnings call”) is where financial analysts listen to and question the corporate officers of a company like TIBCO on their end-of-year report card. TIBCO’s 2009 report (recorded Dec 22 2009) delivered by the CEO, COO and CFO had some comments relevant to the CEP business:

TIBCO innovation in event store-and-forward approaches: the release of TIBCO ActiveSpaces
“[I]t is actually substantially more valuable to have just a little bit of the right information, at the right place, at the right time and in the right context than having all the information in the world six months after the fact.” This followed 3 customer examples doing TIBCO-based CEP (a “Western states utility”, a “major Asian bank”, and a “major Indian mobile company” - covering energy, finance and telco industries).
On EDA: “[T]here’s going to be a systematic shift from transactional to event driven architectures … transactions don’t pickup threats and opportunities … there is a systematic shift, it’s like a change that’s taking place …”
On BPM: “We’ve seen a number of situations where a customer actually went away from say an ERP CRM type implementation as in the case of the Asian bank and went through an event driven inbound marketing approach.”
On BI: “The traditional BI players are largely reporting systems … they allow you to analyze and mine data after the fact and what we do is we look at streaming events before and allow you to anticipate what’s going to happen as it is about to happen. Then, we also with our Spotfire product have taken the visualization of it through a whole different level. I think those are the two elements, the predictive real time nature versus the reporting after the fact nature and the visual technology that goes with it.”

Filed Under: BPM, Complex Event Processing (CEP) Tagged With: annual report, BI, EDA, Trends

Seth Grimes on Event Analytics

September 7, 2009 by Paul Vincent 1 Comment

Although I prefer to differentiate CEP-based analytics between event-driven (some complex events cause you to run some analytic function) and event-based (analytic operations working event-at-a-time or on the event stream directly), Seth Grimes has just published an interesting article titled “CEP+BI= Event Analytics” in Intelligent Enterprise.

Seth had some useful data to back up his comments from a survey on “data analysis software”: “Responses to the question, “What business value can you see in real-time, continuous analysis?” demonstrate not only that current and potential users see value in “faster decisions” (68%) and “more accurate and timely information” (61%) but also that these capabilities are directly linked to concrete business measures that affect the profitability such as “higher customer satisfaction” (42%), “lower personnel and operational costs” (32%), and “increased sales” (26%).”

PS: partly related and very insightful was Tony Baur’s related observation via Twitter: “So in essence, CEP makes BI literally actionable. IOW BI is embedded in execution.”

Filed Under: Complex Event Processing (CEP) Tagged With: Analytics, BI, data analysis, decisions, event analytics

CEP reshaping BI per Intelligent Enterprise

September 2, 2009 by Paul Vincent Leave a Comment

IE just published an article on “4 Technologies That Are Reshaping Business Intelligence”, namely

Predictive Analytics (see TIBCO S+ and Miner)
Real-time monitoring (ergo Complex Event Processing, see TIBCO BusinessEvents)
In-memory processing (see the above tools, and also TIBCO Spotfire)
SaaS (a matter of packaging and deployment tooling, see also “cloud”)

On CEP, the article says
“Complex event processing is a technology that companies are starting to use more broadly to make monitoring more real time. Sprung from the lab projects and custom deployments of fast-trading Wall Street brokerages in the 1990s, commercial off-the-shelf products have taken shape the last five years. Mainstream uses have emerged in supply chains, shipping and logistics, retail, utilities, and other time-sensitive applications.“

Of course, this is why we use the term Business Optimization, and sometimes Operational Intelligence, to differentiate CEP-driven systems over the “reports over tea and biscuits” type of historical BI… but then businesses never ask for “complex events”, they want insight. For better decisions. Basically what BI promised.

Filed Under: Complex Event Processing (CEP) Tagged With: Analytics, BI, BI industry, Intelligent Enterprise, Trends

CEP gives insight now, helped by BI looking at yesterday

July 20, 2009 by Paul Vincent Leave a Comment

Colleague Saul Caganoff has posted some great diagrams explaining the relative values of CEP (i.e. for operational intelligence) and BI (i.e. for longer-term business intelligence) on his blog. Note the conclusion – they are really complementary.

Indeed, I would go further and explain that BI data exploration (as typified by TIBCO Spotfire) is one type of manual BI, and Analytics are another (sometimes more automatable) type (as typified by TIBCO S+ and Miner) – which can identify or tune correlations and detections in the CEP rules (as typified by TIBCO BusinessEvents).

Filed Under: Complex Event Processing (CEP) Tagged With: BI

The TIBCO Blog