Database pundit Curt Monash made a brief mention of event processing (/event stream processing) in his discussion on “big data terminology”, presumably as a response to the discussion he started with Forrester’s Brian Hopkins where Brian (very reasonably IMHO) defined “big data” as:

techniques and technologies that make handling data at extreme scale economical.

with “extreme scale” being defined mainly by the metrics of volume and “velocity” - with the latter being the obvious area of interest from an event processing perspective, as stated by Curt:

Low-volume/high-velocity problems are commonly referred to as “event processing” and/or “streaming”.

Ignoring what might constitute high volume / high velocity problems (see later), Curt replaces “velocity” with “structure” to the “big data metrics” chart (with “velocity” being included in his “bigness” metric). But of course the argument over whether “structure” or “velocity” (or neither or both) are relevant metrics for Big Data is entirely perspective-based:

1. both are characteristics of data / events and
2. both affect processing and storage techniques,
3. … along with other metrics like data lifecycles and data value.

From an event perspective, event payloads (real-time data) can be simple values, tuples (such as the equivalent of a database record), or complex explicit data (such as an XML document), for which something like TIBCO BusinessEvents rules, continuous queries or patterns can be applied. For unstructured text then you may want to add TIBCO Patterns, and for non-deterministic data something like TIBCO Spotfire S+ (think neural nets and the like).

From a “big data” perspective, event processing use cases can include customer purchase records, credit card transactions, phone voice packets or text messages, inventory updates, operational sensor reports, etc etc. But from the event processing perspective (i.e. actually exploiting “big data”) there is another dimension to consider: the scale and velocity of the incoming events versus the scale and velocity (and structure) of the existing data it needs to be related to and/or processed against. Some examples might be:

• large volumes of data at high velocities, compared to large volume of data
  = national security applications
• large volumes of data at high velocities, compared to normal volume of data
  = sensor processing like Radar
• normal volumes of data at high velocities, compared to large volume of data
  = web search
• normal volumes of data at high velocities, compared to normal volume of data
  = automated trading in Capital Markets

This might be a useful way of comparing Big Data requirements against the multitude of different IT technologies and solutions out there. Today, CEP is mostly dealing with normal volumes of data at low to high velocities being tested against normal(ish) volumes of data (maybe up to Terabytes but not Petabytes), with the higher end values requiring fast datagrid solutions such as TIBCO ActiveSpaces. But as always, it would be interesting to have some metrics against the Big Data use cases  to see what we are all talking about…

Last week Forrester analysts Mike Gualtieri and John Rymer published a briefing on the “Future of Business Rules Platforms”, looking at the rules engines that have been so successful in providing business logic platforms for the BPM and SOA worlds. Their opinions are worth noting as they reflect what we are seeing in the marketplace… and their findings are somewhat given away in the report’s subtitle “Customers Are Moving To Event And Decision Management”…

Mike and John mention 2 patterns for CEP and business rule convergence:

1. CEP can drive business rules via separate platforms. This is analagous to CEP (complex events) driving business processes and decisions as required, and where bolting CEP onto the front of mature BPM and SOA environments can make sense (note [1]).
2. CEP can use rules to provide event pattern detection merged with business (decision) rules. This is the use of the same platform to provide an event-decision-action process (note [2]).

Notes.

[1] These patterns will be familiar to TIBCO customers through TIBCO BusinessEvents calling TIBCO BusinessWorks, and TIBCO ActiveMatrix BPM calling TIBCO BusinessEvents Decision Manager.

[2] Forrester also say, on the second pattern, “only Tibco is in a strong position to expand from use of rules to define event handlers into general business rules applications“!

Forrester analyst Jon Rymer is asking for examples of increasing volumes of events being exploited in companies.

We’ve gathered examples from logistics and supply chain management, online marketing, gaming and gambling, payments processing, process management, and physical security, in addition to the well-known financial trading, network and systems management, and RFID-processing examples. We’d like to know yours.

Well, those examples look similar to the TIBCO client use cases for CEP, some of which are mentioned here. And if anyone wants to provide information on their corporate exploitation of event clouds, consider also submitting more details to the EPTS use case survey…

Thanks to Brenda Michelson for an interesting post on Data Quality in Real Time that points to a Forrester report by Gene Leganza on The Top 15 Technology Trends EA Should Watch: 2011 To 2013 (from Oct last year and commented also on here).

To quote from Brenda quoting Gene: “The shift from historical to real-time analytics will require that related processes such as data quality services also move to real time.” In practice poor data quality is typically due to manual system inputs - something that technology like TIBCO Netrics can help address at source - or historic information (with previous manual system inputs). Very occasionally it could be due to problems with event sensors. In any such case, real-time events and data can be cleansed during event processing as CEP technologies usually include the necessary filter and transformation rule capabilities.

I haven’t seen Gene’s report yet but for sure, real-time analytics (monitoring, dashboards, rule-based predictions etc) are increasingly augmenting traditional BI (reporting), visual analytics (data exploration) and predictive analytics (data mining) - and such CEP technology is another powerful weapon in the business analysts’ armory in the quest for business optimisation.

Disclaimer: although we usually use the term operational intelligence for the real-time analytic capabilities provided by CEP and associated technologies, others use terms like continuous intelligence, real-time BI, and so on. And today (thanks to a TIBCO customer) I heard another new term - event-driven Business Intelligence or edBI. Which I must admit I rather like!

Forrester analyst Mike Gualtieri presented on his view of the Complex Event Processing space versus the BRE space - Mike is co-author of both the BRE and CEP Forrester reports. So, going into this, the audience were wondering: is Mike going to say “keep them separate - they are tackling different problems”? Or “they are doing much the same thing - combine them”?

Mike gave a simplistic version of the differences between BREs and CEP engines: CEP handling multiple event channels for input and output, and using (mostly) different algorithms. Mike used the term “event handlers” for the equivalent to BRE rulesets (roughly equivalent to EPTS‘ Event Processing Elements). as well as the term “temporal cache” for CEP event stores. He also coined the term “Event Processing Architecture” to describe CEP-based architectures and event processing networks. Although I quite like some of this terminology, Mike was called out by the audience for his use of some terms - for example comparing CEP with “Business Rules” (when he meant “Business Rule Engines”, and would have been even more accurate if he just said data-driven rule engines…). He was also called out by the audience when he claimed no BREs had temporal constructs for the example he showed (which was recognisable as an example from a mostly-financial-services CEP vendor)  - when a few do…

Also mentioned were 2 examples of where CEP and BRE technology are already being combined - including TIBCO BusinessEvents (described as “poaching from the BRE market”, although I prefer to think the customer base is just becoming better educated… ), and the open source Drools offering (described as a BRE introducing some CEP features).

Mike ended with his comment that real-time / temporal business rules / decisions should all be processed on a single platform, not on separate CEP and BRE platforms. Which of course is what TIBCO does today - so kudos to Mike for calling this out at a BRE conference.

As a side note I see the parallel session by Gartner’s Jim Sinur (on BRManagement as a partner to Process) also mentioned how “rules technology was being included in CEP”…