Many of TIBCO’s Complex Event Processing customers use CEP for operational intelligence types of tasks in industries like transportation and logistics as well as financial services, healthcare and government. Many of these CEP users also use Operations Research techniques, usually in separate and specialist departments or groups, exploiting technologies like constraint programming / CP and  mathematical programming / MP to optimize issues such as route schedules, configurations, marketing, etc - the classical example being the “travelling salesman problem“.

The interesting area to note is that “optimizations” are event-driven - some sequence of events will normally occur to require “re-optimizing”. This could be something as simple as a route perturbation disrupting a daily travel schedule, or an anticipation or prediction of some complex event. This is of course the domain of CEP. Furthermore, the event *data* can be the same data required for the optimization solvers.

Interestingly, at the INFORMS OR event this week, it is clear that the OR tools have to handle multiple aspects of data access. Some recognize that in future they may be dealing with data grids (or even clouds of data / events). They are especially working on parallel-based (albeit not necessarily distributed-based) algorithms for maximizing performance. This, of course, is yet another parallel with the CEP world - tools like TIBCO BusinessEvents provide a high performance multi-agent compute grid + data grid mechanism that allows for different agent technologies - including “optimization agents” - to exploit the same platform as the event processing… subject maybe to controlling event data stability during optimization. Could it be that CEP provides the means by which OR breaks into the “mainstream” of business consciousness?

In TIBCO BusinessEvents 3.0 TIBCO introduced the BusinessEvents Query Language (BQL) [*1] for continuous (and static) queries against cached events and concepts - thereby adding a “stream processing” component to the “rule processing” BusinessEvents event processing platform, like the bespoke event stream processing engines out there.

Interestingly, there seems to be a trend in real-time event store / database technologies adopting continuous query languages. For example, apart from the event stream processing engines, there are also continuous query languages for both Oracle’s and IBM’s cache technologies. Even Sybase has apparently licensed an ESP engine. One wonders when the the big DB tools and ANSI SQL guys will start supporting and promoting stream processing constructs?

Notes:

[1] BQL is based on the object query language OQL but with continuous query extensions.

Joe McKendrick over on eBizQ has claimed that CEP is on the verge of enterprise acceptance, subject to “prices being lowered”. That’s an interesting comment - taken it seems from the recent IEEE report on CEP - it’s as if the author spoke to some CIOs who said they would buy into CEP only when the price comes down. But who’s price? Compared to what? Versus what ROI?

Taking TIBCO BusinessEvents as an example of an enterprise CEP system, included “out of the box” you get:

• inference engine
• continuous query engine
• state machine engine
• UML-based modeling tools with model-to-event-driven-system generator (model driven engineering and all that)
• decision management system with business user interface
• high performance+resilient distributed agent platform with multiple configuration options
• distributed in-memory OODB with connections to a RDBMS for backup
• event channels for popular event sources like JMS and TIBCO’s RV
• powerful XML processing tools
• extensibility mechanism via Java.

Seemingly, there are not many complaints about the value proposition here. Sourcing these components separately would cost an arm and a leg, and integrating them would cost guts and all.

So perhaps “prices are lowering” was referring to the availability of tools like BusinessEvents versus enterprises having to build and support their own solutions at far greater cost?

Jim Sinur at Gartner has posted his views on CEP on his blog. Jim comments on “event discovery” (a.k.a. detection?) via rules, patterns (maybe meaning streaming or continuous queries), process (maybe meaning process exceptions or manual detections), algorithms, and “context”. For the latter he proposes a connection between CEP and “Complexity Theory” which according to Wikipedia is either Chaos Theory or maybe Complex Adaptive Systems (”complex, self-similar collection of interacting adaptive agents … focuses on complex, emergent and macroscopic properties of the system”). That one is for the lab, methinks.

One of the interesting requirements for XTP/XEP (as mentioned in the previous blog posting) is that such performance (and the need for fault tolerance / failover) requires co-operation across multiple systems, ideally generalised in a multi-agent architecture [*1]. The problem here is usually in re-factoring to change performance characteristics as the project develops, to optimize the architecture to the performance needs, available compute nodes, and network (as well as changes to event throughputs).

One solution is to take the agent-based approach, where agents can be re-configured (through model parameters) very simply. Agents can:

• execute declarative production rules “as required”, removing most of the need to specify / re-engineer (/ debug etc) “entry” and “exit” points;
• use queries to partition data (/event objects) and process loads to separate agents;
• rely on shared distributed cache to provide a “co-operative model” of events and data for processing.

The re-factoring / re-architecting process is, to some extent, as simple as specifying which rulesets are assigned to which agents, and defining more or fewer agents as required:
a. Revise named agents (aka agent types and roles)
b. Re-allocate rulesets (or queries) to appropriate agents
c.  Deploy (or re-deploy, with appropriate care) in numbers required.

So the agent modeling aspect is to map the event processing elements first to agent base types (e.g. inference or query [*2]) and subtype (e.g. event preprocessor, event router, event/data processor, batched event/data processor,  etc)  and verify the event flow / data flow is complete and without contention. In this way you can quickly re-architect a TIBCO BusinessEvents application from single-node to multi-node, high performance to extreme performance…

Notes:

[1] Agent representations and modelling is currently the subject of an OMG standard, AMP, currently under development.

[2] CEP agents in BusinessEvents come in 2 forms: inference agents and query agents, roughly equating to CEP agents and Event Stream Processing agents…