Earlier this month Dr Neil Geddes gave a fascinating presentation at the BCS on “Data Processing at the Large Hadron Collider”, describing how LHC experiments create 1 Billion events per sec of which they can record in detail 100 events per sec. Neil’s backgrounder on particle physics was certainly worthy of a an award for the best Dummys Guide, covering everything from molecules and atoms to leptons, mesons, baryons etc. The role of the LHC is to help discover new particles like the Higgs boson, as current astronomical observations don’t fit current physics theory (such as the rotational speeds of stars indicating some unknown gravitational force on galaxies).

The LHC uses a proton collider to create new particles: the rates of the experiment are approximately:

• 40MHz of “proton bunch” collisions
• which give 10E9 Hz proton collisions
• which give 10E-5 Hz particle production

So one part of LHC experiments are to do with creating these particles, and the other part is “Complex Event Detection” - detecting and tracking the particles that fly out due to the proton collisions - and then doing data processing to try and reconstruct the “collision event” as it happened…

From an event detection perspective, as one would expect, the detector hardware is complex and layered to detect different types of particles:

1. Silicon CCD strips (similar conceptually to a digital camera sensor) providing 66M channels (c.f. camera pixels) detecting at 45MHz; these measure the curvature of particle tracks in a magnetic field, from which particle momentum can be calculated
2. Calorimeter: measures the energy of particles by their absorption (and subsequent heat generation)
3. Lead Tungsten crystals which create measurable light when heavier particles are absorbed, using layers of brass and crystals (with the brass apparently sourced from redundent Soviet artillery shells !)
4. Drift chambers that provide the same function as the layer-1 silicon strips but are much larger and hence with fewer readout channels

The instrumentation of these sensors also has to deal with the issue of detector latency, where for example a single calorimeter reading could include the impacts of many events. The design of the sensors tries to mitigate this as much as possible - for example the average occupancy of the silicon detectors is designed to be 2%.

In terms of event rates, the numbers are staggering: 1 Bn proton-proton interactions per sec means 1000 particle tracks created every 25 ns which leads to 100Pb event data per sec which needs to be mapped to 100Mb per sec. So the approach in event processing is to make successively more complex decisions on successively lower data rates. These conversions are:

1. 40Mhz reduced to 100KHz via custom electronics
2. 100KHz reduced to 1KHz initial processing
3. 1KHz reduced to O(100Hz) or 3-10ms per event.

For one of the LHC particle colliders (CMS: the others are known as Atlas, Alice, and LHCb) the event processing is done via a PC grid (up to 150K CPUs) using the approach of (a)  identify high energy particles and (b) network routes each event data to a specific process. They use a huge disk buffer at the expirement source in case of network failure, but otherwise distribute processing in a cross-Europe WAN (such as the CERN-UK connection handling 10Gb per sec).

Interestingly they have found that:

• the network has proved much better than expected so the nodes are increasingly being used for data caches
• CPU use has more than doubled in the last ~18months

Also interesting is how much more hype there is for “big data” (MapReduce, Hadoop, et al) versus developments in “extreme event processing” like that done at LHC.

Note: the presenter used the term “data processing” whereas I deliberately used the term “event processing”. Actually both are involved: event detection and initial processing, then distribution / storage and ~batch-type data processing.

… was the title of the Business Rules Forum session presented earlier this month, targeting the Business Analysts attending the BBC2011 conference. This includes a quick overview of (some) of the various ways to model events (in business models), including using the EPTS Reference Architecture as a guide to event operations required or events desired. Sandy Kemsley did a great overview on her blog, by the way (and Sandy also covers event modelling in her tutorials and training).

Feedback welcome, and of course I’ll be adding things like VPEC-T to version 2 of this.

As surely as Business Rules Forum follows RulesFest, so RuleML follows Business Rules Forum. This is the “academic” rules conference, which boasted a good selection of CEP-related papers this year covering some interesting topics. My (first) session here was a keynote covering “Experiences in CEP” - a slightly different slant on the session presented at RulesFest. Today we explored more the different roles and rule types used in 2 rule-driven TIBCO event processing technologies:

• TIBCO Hawk: event filtering for alarm detection: ECA rules providing “reflex” behaviors for systems
• TIBCO BusinessEvents: full event / event stream processing including Rete-based inferencing / event pattern processing

We’ll do a comparison in a future post. But the general message for the RuleML community was one of encouragement: the use of rules / rules technology / event-driven rules is succeeding, and growing, in industry.

With the end of the BBC conference, the hotel has returned to its normal industry-specific event schedule. By the pool an open area has been turned into an outdoor wedding chapel.  Downstairs there is some kind of gynaecology conference being set up - complete with a curtained off booth that looks like it might actually be a “demo area”… no, surely not?

IDC’s Stephen Hendrick presented on Decision Management at BBC2011, and (of course!) commented that the future of business rule tooling was likely to be sense and respond (CEP) event processing capabilities. Stephen also presented an IDC model on social / collaborative development - something also touched upon by Paul Haley in his talks on the future of  “knowledge management” in the BBC2011 and RulesFest conferences.

As TIBCO is also a major provider of social media technology (tibbr), it is worth maybe suggesting what “collaboration” might mean in terms of operational systems.

• Social collaboration starts with text, not images / diagrams (/voice / video)

Although it is possible to display / allow constrained edits  to / verify / validate visual models like BPMN, state models and so on, these don’t fit too neatly on the average smartphone that is the base platform for social media. So one is more likely to collaborate against a textual representation of such models, but more often than not that textual model will look like… a ruleset.

• Representing text forms on social media like tibbr is easy

Displaying a rule editor in a chat session is not difficult. Collaboration models could also involve adding refactoring tools to allow rules to be split, joined, extended, subclassed etc. But at some point you need a specialist editor in your chat session - if only to keep track of the suggested changes / evolution. IDC’s model also adds more user roles that will need specialist controls too, like the discussion “leader”, “guider”, “critic” etc.

While some such collaboratively-developed rulesets might be considered as overly complex for social-media development, there are certainly areas that could benefit from this form of interactive development: consider specifying complex event definitions using an event pattern language. Or providing examples for a non-discrete event pattern mechanism like Netrics (now TIBCO Patterns) to learn from. This could be very interesting for certain industries to explore!

Note: this session also blogged about here.

As is now “tradition”, the Business Rules Forum / Building Business Capability conference followed RulesFest (from last week), each covering the opposite side of the rules spectrum in terms of audience (analysts versus rules programmers), size (~1000 vs ~120), and even location (California vs Florida). It’s an interesting fracturing of the “decision and rules community”, which actually makes sense (i.e. business vs IT) - and both deserve to grow next year (RulesFest should be 2x-3x in size, BBC could probably easily grow 30%).

One of the interesting trends here at BBC2011 is the growth of the business analyst focus, which of course means “rules engine vendors” make much less sense to be seen here versus “rules management” or even MDM vendors, and those less so than the business modelling community. In the CEP space, vendors tend to cover both business and IT camps: for example TIBCO BusinessEvents includes event / concept modeling, state modeling and decision modeling, but for automation not business documentation. And both IBM and Red Hat have business user interfaces with event processing (with a rumor of a 4th CEP vendor about to buy their way into the decision management space).

So the main focus for business analysts attending BBC were very much at the TIBCO Nimbus level of process documenting / discovery (and indeed, decision documenting / discovery). And the surprise was the lack of business modeling tools (with the notable exception of Sparx systems - folks known from the UML world). Maybe this will change next year?

TIBCO Software exhibited this year too (with the aforementioned TIBCO Nimbus as well as ActiveMatrix BPM), and we contributed to 2 sessions (”The OMG Decision Model and Notation standard” co-presented with IBM, and “What Analysts need to understand about Business Events“). The latter was interesting to present, as events participate in many analyst models: I’ll post a precis of this session later.