At EPTS5 last week, a 2nd vendor proclaimed that CEP was merging with BPM (as a technology). So lets check this claim out against Connie Moore (Forrester analyst)’s recent blog post on “When to use a BPM Suite” and see how this compares to a CEP tool like TIBCO BusinessEvents (which we have previously compared with its TIBCO BPM+ cousins).

Connie gives the following indicators for readiness for a “BPM suite”:

1. Processes change frequently.
   That’s an interesting one. BPMN is the preferred model for processes, and it is a good way of documenting processes. And in model-driven BPM, you can elaborate a BPMN model with IT artifacts and thence generate appropriate automation.
   However, another trick is to use dynamic process models - typically rule-driven or plan-driven, and both event-driven. So CEP tools like BusinessEvents are at least as good for “changing processes” with the slight caveat that the type of change is significant: pre-ordained dynamic processes, or just good old manual changes by a business analyst?
   E.G. TIBCO BusinessEvents uses rules and state models to define (event) processing, as well as managed decisions.
2. Automate and monitor processes across the “value chain”.
   Monitoring capabilities may be built-in at the BPM level, but usually you want to monitor processes against external metrics (like your SOA services, external events, etc). That’s a role for CEP.
   E.G. TIBCO BusinessEvents can model a value chain as a state model, and correlate events in various ways to support business activity monitoring - including monitoring BPM.
3. Support cross-functional processes.
   This is the “enterprise view” of BPM. Probably its true to say that CEP technologies have few enterprise modeling aspirations.
   E.G. I have no “organization model” for example in TIBCO BusinessEvents, although I can usually define the appropriate metamodel as concepts, and enjoy full flexibility in how that interacts with events and rules and decisions…
4. Allow monitoring of status of work in progress.
   CEP *does* allow for long-running stateful processes - and “self monitoring” is quite common - but again this is a general capability not an out-of-the-box fixed feature.
   E.G. TIBCO BusinessEvents can query continuously the attributes of entities it is processing, and deliver the results, as they change, as events.
5. Extend the life of legacy apps.
   The message here is that BPM can layer on existing apps (/ services). The same is true for CEP, which can invoke and control existing applications just as well.
   E.G. TIBCO BusinessEvents can invoke services in TIBCO ActiveMatrix and BusinessWorks…
6. Collect work metrics or SLAs.
   Metrics: event counters. SLAs: rules. CEP: check.
   E.G. TIBCO BusinessEvents provides rules to operate against events, and detect when, for example, SLAs are trending to be breached… an example of SOA SLA is the TIBCO SPM product based on CEP.
7. Improve inaccurate, inconsistent or laborious manual work.
   OK, CEP is not a workflow monitor or manager (although rule-driven workflows applications have been done). CEP is mostly about automated processes, or monitoring events from manual processes.
   E.G. you could implement a state model for workflow queue handling in TIBCO BusinessEvents, but most people prefer to use the out-of-the-box variant from a BPM tool.
8. Capture process knowledge.
   This comes down to documenting (fixed) processes in BPMN. Fine and dandy, but decision knowledge and inferred knowledge are missing. Some CEP tools can help fill this gap.
   E.G. TIBCO BusinessEvents provides inference rules for deeper knowledge chains (think Zachmann “what” rather than “how”).

So: “CEP as a part of BPM” is not a slam dunk by the current narrow view of business processing. CEP augments and provides capabilities that help in BPM, but also provides other capabilities (BAM, operational intelligence, …). BPM 3.0? BPM+? edBPM? We’ll see.

UK retailer Carphone Warehouse was in the news recently, announcing their transition to a TIBCO-based CRM system - in this case conventional workflows augmented by event- and rule-driven processes.

This use case has a good write-up (via MWD) available from TIBCO’s web site: some choice quotes are:

Background:
The Carphone Warehouse is a European telecommunications retailing and service provision group, based in the UK. The company is large, and has a record of fast growth: in its last financial year it reported revenues of over £3.9bn, but is still growing over 30% year­-on­-year. It currently employs around 20,000 staff in eleven markets. As its name implies, the company was founded as a mobile phone retailer. …

As is the case for so many large organisations, there came a point where the weight of “legacy” functionality became effectively unmaintainable in the face of continuing business change …  as a result, The Carphone Warehouse decided that as well as implementing a new billing system, it would also implement a new, distinct and separate CRM system. …

A service­-oriented approach to implementing the renewed billing capability and integrating it with the new CRM capability was identified and promoted early on … TIBCO products (Enterprise Messaging System and BusinessWorks) were selected for the SOA infrastructure … initially [the chosen CRM vendor's] inbuilt workflow functionality was selected as the platform for this. However as the project unfolded, the importance of long­-running processes, and asynchronous event-based flows … became increasingly apparent. It was at this point that the project team decided to extend The Carphone Warehouse’s investment in TIBCO technology to include the iProcess Suite (TIBCO’s BPM technology offering) and BusinessEvents (TIBCO’s Complex Event Processing offering).

Implementation:
The processes currently being implemented in the billing/CRM replacement implementation are focused on automating broadband service provisioning, and are prime examples of our “straight through processing” scenario …

The Carphone Warehouse is using TIBCO’s iProcess technologies to define and coordinate long­ running, stateful, system­-to-system integrations; it’s using TIBCO’s BusinessEvents technology to handle other stateful interactions, where there’s a need for near-­real-­time processing. …

Results:
…There’s already been one major success from this approach. In its retail business, the team has used the iProcess technology to deliver a reusable telecoms service activation process which has enabled the company to progress business expansion across Europe much more broadly and quickly than anticipated. The implementation cost just 20% of  the original estimate, and the reduced cost for retail operations “in territory” has made it much easier to roll out new network deals and offers.

Note: Carphone Warehouse are no doubt using TIBCO BusinessStudio to develop iProcess applications - so now is a good time to mention that BusinessStudio 3.2.0 is now shipping. Although this doesn’t cover CEP (or CEP-driven BPM), it does now include some iProcess Conductor (Gantt-based goal-driven BPM) features…

Tibco’s Jacky Yap has posted a nice YouTube video of CEP-BPM-SOA interaction. Enjoy!

Work on the EPTS Reference Architecture is moving forward, with excellent contributions from the event processing industry and academia (EPTS members can observe the work-in-progress on the EPTS working group wiki). One interesting angle of discussion revolves around the compatibility and overlap of “event processing” with conventional “data processing” technologies, especially when it is possible to view “events” as “data” and process them as such. Of course, critics of CEP will counter that all event processing tools simply re-use existing data processing techniques (such as extended versions of SQL engines, or inference engines, or neural nets etc…), and to some extent they are right - what’s new is the idea of re-arranging these techniques to exploit higher-performance CPUs and larger amounts of available memory for in-situ event pattern detection in everyday business problems.

Defining a generalized event processing (and complex event processing) architecture that covers all use cases, even of a single (albeit ambidextrous) tool like TIBCO BusinessEvents, is rather challenging. There are a number of different architecture patterns possible, covering different functional and performance requirements. One generalized effort is included here, based on TIBCO’s work for customers in this area:

The various tiers can be described as:

Tier 0 (not displayed):  The sources of events could be anything from SCADA control systems, RFID detectors, BPM or ERP systems, or whatever.

Tier 1: Event channel (or transmission medium). In TIBCO’s world this is usually EMS (topic-based or queue-based) or RV message buses, although it could also easily be some other middleware mechanism.

Tier 2: Event PreProcessor. Optional. Handles basic event filtering, and roles such as concept enrichment (when the event is just adding information to an existing event object), concept creation (the event is transformed into an event object, or concept in BusinessEvents), and forwarding to local and/or distributed memory (a.k.a. event store, for processing elsewhere).

Tier 3: Distribution Agent(s) or Event Store. Optional - used for high-performance, high volume, fault-tolerence and large-scale CEP. Provides storage for events (and event objects) required for the event patterns representing complex (aggregate, abstract, derived) events - where these event patterns may occur over periods of time. Needed (in conjunction with Tier 4…n) where the volume of events, or amount of event processing, exceeds the limits of a single process node’s memory or CPU limits. Used also when information needs to be replicated across nodes to support fault tolerance and failovers. For performance reasons this is often some kind of high-performance cache or data grid (as is the case of BusinessEvents’ cache component), but it could also be a common-or-garden file-oriented database… in TIBCO BusinessEvents we support both approaches.

Tier 4…n: Event Processing Agent(s). In the simplest case, a single EP Agent can attach to a channel and process events (2-tier). Or it can involve a Preprocessor either in-process or as a separate agent (2.5-tier). Or be used with the cache (4-tier). Ultimately these can be organised as a hierarchy of EP services, with these services themselves constituting multiple, load-balanced, identical agents. These EP Agents define event patterns (through rule conditions, state transitions, and queries) and communicate either directly to event channels [Tier 1] or, more often, through the event store [Tier 3]. Large workloads can be distributed through queries (i.e. pull model) or through shared event / event object updates (i.e. push model). In BusinessEvents, EP Agents can be one of 2 base types: inference agents (for rules and states), and query agents.

Tier n+1: Services. Optional. Existing SOA services may also contribute data to the event processing agents, either as-required or cached in the (event) store to be used and updated as needed. Typically service invocation (which could also be a BPM workflow invocation)  involves a call out to some mechanism like TIBCO BusinessWorks, and possible some TIBCO Adapter.

Tier n+2: Persistence.  Optional. When the event store [Tier 3] is using high-performance cache there is a need to provide back-up persistence for tasks like archiving, start-up load and shut-down un-load, and standard analytics. You might also have operational data stores you need to query directly (rather than through the services layer [Tier n+1]) to enrich event objects for processing. If you want to visualise all your events over time to identify new patterns you would typically use something like TIBCO Spotfire against this layer, and perhaps S+ Miner to extract some rules to update some Tier 4…n agent with…