I see that last year’s Dagstuhl workshop report has been published, titled “Executive Summary and Manifesto — Event Processing”. This is an excellent 60-page group effort by some of the great-and-the-good in event processing, and recommended reading!

The top-level Table Of Contents is:

Chapter 1: Why event processing?
Chapter 2: What are the characteristics of event processing?
Chapter 3: Synergies and relations to other areas
Chapter 4: Event processing related standards
Chapter 5: Grand challenge: The global event processing fabric and its applications
Chapter 6: Near-term research

Disclaimer/disclosure: TIBCO was a co-author of Chapter 4.

Of the various topics discussed at Dagstuhl last week, one was to look at the possibility of standards for CEP. The Dagstuhl event in 2007 apparently concluded that the event processing space was too new or immature to consider standards - 3 years on has anything changed?

Since 2007, the CEP market and acceptance has continued to grow, as has the understanding of how and where Complex Event Processing fits.

I was fortunate to be joined by Prof. Adrian Paschke from Free University of Berlin, and Florian Springer from business consultancy Senacor, in the “Standards Task” at Dagstuhl in 2010. Between us we defined a “framework” of the CEP system development process from which we could better  understand the role and potential for both existing, extended and new standards.

This follows the OMG viewpoint (via Model Driven Architecture / MDA CIM, PIM and PSM levels representing Computer Independent, Platform Independent and Platform Specific Models) supported by Use Cases, Domain Reference Models, and associated Functional Models etc. As such, it is only one of many viewpoints - but a reasonable starting point nontheless.

The team reported to the Dagstuhl attendees a mapping of some of the available standards to this model. Others we missed (like “Benchmarking standards”). There was also interest in “Academic Standards” - which probably maps to a Reference Model for the terminology and practices in CEP that could be used for a common curriculum - effectively a special domain (education) reference model (for the CEP definitions in the framework).

One intriguing “standards gap” is that of the business definition of event patterns - the high level pattern language at the CIM-PSM boundary that defines complex event patterns. Earlier it was disclosed that although CEP vendors offer such pattern specification languages, some developers shun them for more detailed control of event processing (using queries, rules etc) - although it is likely these patterns are not yet defined at the “business model” level. Such a pattern language, incidentally, is also available in the recently announced TIBCO BusinessEvents 4.0.

Opher Etzion has blogged a good summary of one of the more “interesting” sessions at Dagstuhl, a.k.a. the “industry feedback” evening session. My list of the “most interesting points raised” includes:

1. Marco Sierio from Sweden talked about his ASP/SaaS model for delivering event processing - proof if needed that event processing applications can be deployed as services or in the cloud. His RuleCore application does track n trace for trans-Europe shipments, with customer rules sent as events directly from his customer to the system.
   • The “rule updates are just events” philosophy makes perfect sense in my mind, albeit one requiring considerably more event validation / security / etc etc than the simple sensor events that drive the application. A similar approach is used to update the template-defined rules in TIBCO Active Service Gateway, for example.
   • Marco mentioned one side effect of the application being that the shipping company discovered truckers were apt to make stops outside - and therefore effectively associating their ad-emblazoned trailers with-  er, “establishments they would not normally want to be associated with”. Real life meets monitored life?
   • Some of the “issues” in learning to “do CEP” he finds in customers are
     • understanding the declarative (rather than procedural) model, and
     • understanding asynchronous events (rather than “request-reply”, a.k.a. simple SOA).
2. Richard Tibbetts talked about some of his “observations” about developer practices in building CEP systems from his customers at Streambase - and applicable to all CEP development and customers, for sure.
   • Lack of discipline in MOM message design: developers create too many message types and replicate information inside messages. I have heard this before, and continue to wonder whether there is now a need for “Master Event Management” …
   • A tendency towards premature optimisation: some EP operations are simply not used enough to warrant much effort in optimising them. Use a profiler to see what needs optimising *after* the system is running …
   • Use incremental development - don’t try and replicate the “big bang” and wait until functional completion before unit-testing!
   • Event patterns tend NOT to be favored by developers, compared to lower level developer constructs like queries.
3. Badrish Chandramouli from Microsoft explained the genesis of the Microsoft stream processing engine (MSSI) from its routes in the CEDR research project in Microsoft Research.
   • MSSI is used internally for web analytics on the Microsoft Hotmail system, doing user classification for the on-line ad system at a rate of 1M events per day (i.e. probably 20-50 per sec), as a part of a system that uses map reduce as a part of the machine learning.
   • The MSSI product is validated internally by translating sample continuous queries into SQLServer queries via some temporal algebra translator, and event streams into database tables; if the outputs from SQLServer and MSSI don’t match they know they have a problem!
   • Although MSSI is brand new to the market, they are already looking at High Availability features for a future release, and topics like real-time data mining…
4. IBM covered 2 of their (at least 3, maybe 4) CEP technologies:
   • Martin Hirzel from the IBM System S (a.k.a. Infosphere Streams) research group talked about their progression from research tool to product, and the development of their Streams Processing Language.
   • Udo Pletat talked about using the IBM RFID tool that exploits the IBM Labs’ Amit ECA rule engine. I hadn’t realised Amit had gone into real-world use, so will need to do more updates on the CEP Market time-graph!

This week is the Dagstuhl Event Processing “seminar” (a.k.a. workshop) in the pretty Saarland region of Germany, attended by vendors, researchers and Gartner analyst Roy Schulte. Dagstuhl is a kind of non-profit research collaboration centre that runs “seminars” on various computer science topics throughout the year - its like a kind of academic retreat, with principles like “no one can lock their bedroom door” when they leave their room  (which must be a mainland European concept…).

The Dagstuhl campus is actually in an old provincial palace (think château) with modern extensions, under a hill with a ruined 13th century castle (populated by giant snails, and apparently destroyed by Napoleonic invaders - facts which may or may not be related).

The group here (40+ folks) is being managed by Opher Etzion, and we are arranged into 5 groups covering the “why, “what”, “relative IT context”, possible standards, and Grand Challenges in Event Processing. Various numbers signed up for these groups, with varying from 3 of us on “standards” to 10+ looking at the “why” and “what” groups.

Today the “Why use Event Processing” group presented their early thoughts - based on looking at benefits (like latency of response) versus costs (like acquiring additional tools, skills etc). Probably they will need to look at real comparative customer use cases to get actual data on this stuff… but they’ve made a good start.

Time to start the 2010 list of “interesting” CEP and rule-related conferences…so far I have 4 5 on the list, covering 2 3 weeks in May to July…  (yes I had forgotten Dagstuhl…). So, by date order:

• Week starting 16 May, location Europe:
  • 2010 Dagstuhl Event Processing seminar, Dagstuhl in Germany, 16-21 May, theme = task-oriented, focused on a joint effort between the academic participants and the industrial participants (vendors, customers, analysts) … output will be a manifesto for the event processing discipline…
• Week starting 21 June, location USA:
  • OMG Business Rules Symposium, co-located with the OMG Technical Meeting in Minneapolis, MN, day TBA, likely to cover the OMG rule standards such as SBVR (for documenting business rules) and PRR (for production rules)…
  • Semantic Technology Conference / RuleML co-organised Rules Track, San Francisco, June 21-25, theme = semantic rules [...are] more powerful, flexible, and active forms of “structured” knowledge…
• Week starting 12 July, location UK:
  • ACM’s DEBS2010, Cambridge, July 12-15, theme = dissemination of original research, the discussion of practical insights, and the reporting on relevant experience relating to event-based computing… which was very interesting last year, and is “in cooperation with” the EPTS.
  • Rule2010 workshop, Edinburgh, July 14, theme = rule-based programming in Industry and the Semantic Web… which seems a bit wide to me (i.e. there is not much “semantic web” overlap with “industry”).