The latest edition of the Communications of the ACM had an article covering CEP from the BI perspective, written by researchers from Microsoft and HP. On Complex Event Processing they write:

The competitive pressure of today’s businesses has led to the increased need for near real-time BI. The goal of near real-time BI (also called operational BI or just-in-time BI) is to reduce the latency between when operational data is acquired and when analysis over that data is possible. … A class of systems that enables such real-time BI is Complex Event Processing (CEP) engines…

They then quickly make the mistake of assuming CEP = Event Stream Processing or ESP…

Applications define declarative queries that can contain operations over streaming data such as filtering, windowing, aggregations, unions, and joins. The arrival of events in the input stream(s) triggers processing of the query. These are referred to as “standing” or “continuous” queries …

Of course, rule-based CEP, using constructs like Event Condition Action rules, and indeed other languages / algorithms can be used to define aggregate events to support real-time  BI tasks. Ooops.

There are several open technical problems in CEP; we touch upon a few of them here.

This should be interesting…

One important challenge is to handle continuous queries that reference data in the database (for example, the query references a table of customers stored in the database) without affecting near real-time requirements.

I’m pretty sure most stream processing engines provide some ability to do this (if you want to). But normally the last thing you want to do in (near) real-time business applications is waste time querying a RDBMS. Common practice is to put the data you need first into memory or a datagrid / cache, where access times are much lower.

The problem of optimizing query plans over streaming data has several open challenges. In principle, the benefit of an improved execution plan for the query is unlimited since the query executes “forever.” This opens up the possibility of more thorough optimization than is feasible in a traditional DBMS. Moreover, the ability to observe execution of operators in the execution plan over an extended period of time can be potentially valuable in identifying suboptimal plans.

Again, I’m pretty sure all the main CEP technology provides have optimised CEP engines - for example TIBCO uses a high-performance version of the Rete algorithm. I’m pretty sure Microsoft’s stream processing engine has optimizations too!

Finally, the increasing importance of real-time analytics implies that many traditional data mining techniques may need to be revisited in the context of streaming data. For example, algorithms that require multiple passes over the data are no longer feasible for streaming data.

This is actually an issue for the analytics guys: continuous analytics do indeed imply computing statistical models an event-at-a-time rather than “all at once against some vast data store”. The good news is that the analytics world has been doing some of this for some time, in order to accomodate complex processing of large data sets by batching and recombining data as it is processed.

The BI Overview also mentions several other “BI technology” aspects that are also often combined in CEP solutions: in-memory, distributed (map reduce a.k.a. “divide and conquer”), and analytics. So although it wasn’t emphasised in this paper, it seems the interesting development here is a certain amount of convergence in using these technologies together!

Thanks to Brenda Michelson for an interesting post on Data Quality in Real Time that points to a Forrester report by Gene Leganza on The Top 15 Technology Trends EA Should Watch: 2011 To 2013 (from Oct last year and commented also on here).

To quote from Brenda quoting Gene: “The shift from historical to real-time analytics will require that related processes such as data quality services also move to real time.” In practice poor data quality is typically due to manual system inputs - something that technology like TIBCO Netrics can help address at source - or historic information (with previous manual system inputs). Very occasionally it could be due to problems with event sensors. In any such case, real-time events and data can be cleansed during event processing as CEP technologies usually include the necessary filter and transformation rule capabilities.

I haven’t seen Gene’s report yet but for sure, real-time analytics (monitoring, dashboards, rule-based predictions etc) are increasingly augmenting traditional BI (reporting), visual analytics (data exploration) and predictive analytics (data mining) - and such CEP technology is another powerful weapon in the business analysts’ armory in the quest for business optimisation.

Disclaimer: although we usually use the term operational intelligence for the real-time analytic capabilities provided by CEP and associated technologies, others use terms like continuous intelligence, real-time BI, and so on. And today (thanks to a TIBCO customer) I heard another new term - event-driven Business Intelligence or edBI. Which I must admit I rather like!

One of the drivers of success in the BPM world has been BPMN - the Business Process Modelling Notation - developed under BPMI before it got absorbed into OMG, and probably a bigger success than any other modelling standard - and now as BPMN 2.o (beta, at this time) slightly reconfigured in name as the Business Process Model And Notation. BPMN is used to provide a “process” perspective (versus, say, a data or an event perspective) of a system and shows the flow or orchestration of process tasks and activities. It is made up of (per the excellent bpme.de BPMN2 summary poster):

• 2 flow subtypes: sequence is the normal flow, with default and conditional conditional subtypes
• 7 task types: send, receive, user, manual, “business rule” (really decision), service, and script
• 6 activity markers: sub-process, loop, parallel, sequential, ad-hoc and compensation
• 7 gateways: exclusive, event-based, parallel, inclusive, exclusive event-based, parallel event-based, and complex
• 6 types of data: input, output, object, collection, store, and message
  and
• 63 types of event…

OK, “63 types of event” needs a bit of explanation (and justification). These consist of:

• 13 main event types: untyped, message, timer, escalation, conditional, link, error, cancel, compensation, signal, multiple, parallel multiple, and terminate

… across 8 situations classified by location in a process:

• Start: top-level, event sub-process interrupting, and event sub-process non-interrupting
• Intermediate: catching, boundary interrupting, boundary non-interrupting, and throwing
• End

… but with some situations not requiring certain types of event (e.g. there is no “start cancel process” event) leaving 58 63 event types defined [*1].

Presumably BPMN tools will let the user specifiy the main event type and associate the correct symbol from the context in most cases, leaving us to consider just the 13 main event types.

So let us analyse these event types from an event processing perspective:

• Message and timer events: these are probably most familiar with to those with an event processing perspective, with a message relating to what we consider either a source or a derived event.
• Escalation: this is really a deferal to a different process (e.g. a supervisory process), and can be handled by some internal message in event processing.
• Conditional: this is “reacting to conditions becoming true” - in other words equivalent to a rule-firing (or event query) success. In a rule system this might be handled just by setting some fact or data, which in an inferencing rule system will use some internal event to signal to the rule engine that other rules may “fire” - however there is no need to explicitly model this behavior. I may also of course create an event (or in a query system, insert / update some event), which is again is not explicitly modelled as “conditional” per se…
• Link: some means of connecting models: this is more a “page break” than a business event!
• Error: I prefer the term “exception” here, which of course is a local context - an error event in one process is a normal event in an error-handling process, etc.
• Cancel: this is more a transaction-specific event: if dealing with transactions then some events may indeed be “cancel transaction X” requests, which may or may not be satisfiable, etc.
• Compensation: this is where some process route may require some compensatory process - for example in relation to a “cancel” event.
• Signal: signalling information across processes: this is might be used as a control event, such as in controlling a choreography.
• Multiple: one of a set of events is “caught” or provides the cue to continue the process.
• Parallel Multiple: all of a set of events is “caught” to continue the process.
• Untyped events are used for start and state changes. Typically, though, a state change is the main event we are interested in (in event processing)!
• Terminate event - process complete! From an event perspective this (process end) might never happen of course…

The extra detail might be explained by the fact that in BPMN you are detailing *how* to implement a process, and the “event symbol” is providing you with some specific context for that event. In event processing languages, you usually describing “what* is to be processed, and where all events contribute to some situation or context. It will be interesting to see how these 58 symbols work out in practice, as a hindrance or a help for process designers…

Note:

*1: the table of events above, from the BPMN2 poster from bpmb.de, has 63 event symbols, and is slightly re-organised from the equivalent table in the BPMN2 beta specification (Table 10.93 pp 269-270, in 10-06-04.pdf). Interestingly the specification document also includes a table to detail the event symbols (Table 12.23 ppp 405-411, in 10-06-04.pdf) but this shows only 58 symbols! Let’s just say “lots of event types” in BPMN2 then…

There were some customer discussions recently on the conceptual relationships between the CEP and BPM and SOA “software stacks”. This coincided with the announcement of the OMG-backed Event Processing Consortium being set up (alongside the merger of the SOA and BPM consortia), events which themselves can be interpreted as that event processing has some special role to play alongside BPM and SOA.

[Disclosure: note TIBCO is not a member of either the OMG's EP or BPMSOA consortia - they seem to be focused on end-user rather than vendor participation - but is an OMG member and participant in standards development. We currently see the EPTS as the main advocacy group across vendors, academics, analysts and end-users, but will be monitoring the progress of the EP Consortium].

So here are a few simplistic patterns on how CEP (event processing) relates to BPM (processes) and SOA (services)…

CEP Pattern	Example	Diagram
1a. Standalone CEP This is a bit of a misnomer - you don’t identify event patterns without some intent to use them, and at the very least such a use would be in a BAM type role displaying interesting correlations for some business person - who of course is engaged in some kind of business process, which may or may not be managed by BPM…	Monitor a production process to provide an “additional view” or dashboard for the process control manager.
1b. CEP enriching BPM processes and/or  SOA services. This is the conventional view of CEP, detecting the complex events that are of interest to, and useful in, appropriate processes and services. Complex events in these cases can be as straightforward as deducing that a deliverable has been completed, or some process truly initiated. Typically the CEP system is transforming source events into business events, for onward use in (the) business processes.	Identifying exception events in a business that need to handed to a workflow or case management system for resolution.
2a. CEP monitoring processes and services This is where the sources of events are the managed processes and services themselves. This process and service monitoring is used to detect exceptions, disparities across systems, and system performance… Note that effectively this pattern is a combination of patterns 1a and 1b above.	Detect when response times exceed some metrics and suggest corrective actions such as reallocating resources.
2b. CEP-based decisions for processes and services This is where I need to make intelligent decisions for the process and service layers, using the CEP layer as a monitoring, shared decision management component. Note that effectively this is a slight extension of pattern 2a above.	A BPMN “rule activity” sends a decision request to the CEP engine to get a valid decision for a process decision point; the CEP engine monitors the decisions made.
3a. Dynamic process and service control This is where the events from processes and services, and external services, are combined to select which processes and services are relevant to use for the current context. In effect, the CEP engine becomes the controlling agent for the business processes and service engine, handling for example dynamic process selection. Note that this pattern is a further evolution of patterns 2a and 2b.	In a complex business process for ever-changing fulfillment problems, CEP-based rules determine which sub-processes are valid based on incoming information.
3b. Embedded processes and services within CEP The final evolution of the above is when you argue that the functions of the BPM and SOA stacks can be subsumed into the CEP layer. In reality this is usually only a partial subsumption, as otherwise the centralization of services into just 1 layer could be perceived as contrary to the very idea of SOA! So this covers things like event-based policy implementations being embedded as CEP rules rather than as external services, but alongside some external services such as an operational database. Indeed, one could argue that in this case the CEP event processing agents are themselves really part of the SOA layer, not the other way round!	A service gateway controlling access to existing services, but embedding decisions, service policies, and business rules.

So, did I miss anything?

In a previous post we commented on IBM’s Conceptual Model for Event Processing Systems, but failed to observe that IBM’s rule engine group, Ilog, was noticably absent from its contributor list. So perhaps it was both fortuitous and timely that Ilog’s Daniel Selman recently added another “viewpoint” on the position of event processing (aka the rules viewpoint).

Daniel’s take on the “primary users (sic) of rules technology” are the (use cases for) automating decisions, event responses, processes and inferences. I think these might be better classified by renaming then as decision processing, event processing, and business process processing. But inferencing, which Daniel notes as being a technology to support Artificial Intelligence, is not so much a (user or) use case, but a means of providing knowledge-based reasoning to support any of decisions, processes, etc - “AI” is not (or should not be) a means unto itself.

Standard Event Processing Design Pattern

Indeed, one could probably argue that:

• all business processes are driven by events of some kind, and involve decisions and (re)actions;
• all processes, decisions and event processing are context (i.e. state) driven; however, some processes and algorithms are used in a subordinate fashion to a stateful process (consider a typical decision service that is subordinate to its application server and database layers);
• inferencing can be used to enhance any part of the event-handling process as a form of declarative rule control.

In TIBCO’s experience, an event driven rule engine (like TIBCO BusinessEvents) can be used to provide dynamic business processes, event-driven decisions, and real-time control mechanisms - and often all 3 - exploiting and building on the fundamentals of complex event processing.

So, on the utility of rule engines, we totally agree with Daniel!