Defining what is “eXtreme” is a difficult topic. For example, if I am simply “processing” simple events as incoming data with no correlation requirements, I have a significantly easier job than if I am doing complex event processing, abstracting higher level events over time from incoming events.

Here is an interesting example that was being discussed last week: during an on-site application test, an enterprise event processing application (using TIBCO BusinessEvents) was comfortably processing events at their arrival rate of ~750 eps (events per second) when a “testing glitch” caused the EMS (JMS) queue to back up; when the offending component (i.e. agent) was corrected and restarted (in situ) the CEP application maxed out its CPUs to clear the queue at a reported ~5K eps. This 5K figure compares quite well “as a number” with other complex applications (as opposed, maybe, to event stream processing of simple events), although probably it will be up to someone like the EPTS Use Case Working Group to make some classifications here.

My guess is that “eXtreme Event Processing” or “XEP” (in which we really mean “extreme CEP” involving correlation across events, XML processing, some number of business rules, large number of fields per base event message, 24+ hour retention, fault tolerance, and so forth) probably starts at ~1000 events per second. But ultimately its not just a matter of “how fast”, but also “work done”…

I see from Tim Bass’ blog that Brian Husted of Pyxis has published a paper titled “Event Driven Grid Computing“. This is regarding a US DOD SOA application that exploited “CEP to enable scalability for SOA“, handling tasks like intelligent routing, governance, monitoring and complex workflows.

Brian comes across as pretty enthusiastic about the use of the TIBCO BusinessEvents event processing platform, for example commenting “… a small number of people (2 or 3) can turn out big things with BE”. Hopefully he’ll find time in the near future to write about some of his other BusinessEvents applications.

Notes: See also this past post on the topic of CEP, EDA and SOA.

IT strategist Brenda Michelson has posted an interesting blog titled “Smart Roads, Bridges and Grids call for Smart Event Processing”, referring to a similarily titled WSJ article. The article concentrates on the availability and technology of the sensors: from a Complex Event Processing perspective this just adds to the “event cloud” of course. Also, we can expect the latest CEP technologies to lower the entry cost for processing the events from these new sources and make all these new applications feasible.

So soon we may have “your tax dollars at work, using CEP”. Maybe we can also request CEP to “monitor the appropriate spending of your tax dollars”, too…

Notes:

I would probably define “Smart Event Processing” as making smart or intelligent decisions based on simple and complex events being detected and identified respectively. Probably you need to combine CEP and especially event stream processing with intelligent decision-making rules in an eXtreme Transaction (Event) Processing environment.

TIBCO announced today the delivery of the first TIBCO Messaging Appliance - basically an RV agent in a box - and this has been widely reported and blogged about. It’s quite feasible that the same approach could be used for “basic” complex event processing operations, especially those that don’t require history (or much persistence) - examples that spring to mind include simple streaming queries, aggregations, identifying missing events over time, and so forth. Indeed, there are academic studies of such approaches in progress today.

The TIBCO Messaging Appliance works with the TIBCO BusinessEvents CEP product, whose multi-agent architecture may well be required to consume such high messaging rates, as well as any other RV-enabled tool.

Just to show how close the BRE/BRMS and CEP fields are to converging was ably demonstrated by Kevin Forbes of Healthways in his presentation on the massive data requirements and event loads being handled by a conventional BRMS, albeit with much of the “heavy lifting” being done by a Coherance cache, with traceability being handled by Amberpoint. Points of note were:

• “Network and data access are the limiting factors” in decision automation.
• Distributed cache and data grid approaches are “key for decision service performance” and “multi event correlation”.
• They are looking at / planning to do real-time analytics in terms of updating rules from analysis of data on the datagrid. Which makes perfect sense.

In some respects these guys have constructed their own custom equivalent (albeit with a web services level for invoking the rules, on a Microsoft platform) of a CEP tool like TIBCO BusinessEvents. Needless to say I thought it was a great presentation…