Opher Etzion just made an interesting classification of the CEP tools market in his observations on the Bloor Research comments on CEP and Big Data, part of an increasing amount of coverage on CEP. To wit:

• Event Processing Platform is a software that enables the creation of event processing network, handle the routing of events among agents, management, and other common infrastructure issues.
• Event Processing Engine is a software that enables the creation of the actual function - in the EPN term implementing agents.

In the CEP Market analysis we don’t try to distinguish between these - probably because it would be contentious. For example, to some folks an “event processing network” is managed as a single process - possibly multi-threaded, but bounded on a single machine instance. To others (like TIBCO) the network is a message or event distribution mechanism for breaking the constraints of a single process or system (e.g. performance, scalability, and fault tolerance constraints). Furthermore “event processing agents” might be viewed as “event processing operations” - like a single pattern detection query, or a pattern matching rule, arranged in some kind of activity or business process diagram - or as more autonomous processing agents that can handle a number of operations and cooperate declaratively towards some solution.

If one views an Event Processing Platform as one that handles routing across multiple processes and distributed systems, then the potential candidates is reduced somewhat [*1]. Of course, any CEP engine can be used acoss multiple systems with a shared middleware infrastructure, but individually they are “blind” to the other agents and the design tools do not handle the cooperative nature of the agents. Of course, one can set up a message type to include management information to allow for some semblance of distributed control, but this is more likely to be a developer task than a platform capability.

Looking at something like TIBCO BusinessEvents, we can see this satisfies the requirements of a (physically distributed) Event Processing Platform:

1. Enables a (computer) network of event processing agents - typically as a minimum of rule agents and cache /datagrid agents, in pretty much any configuration.
2. Enables a (single process) network of event processing operations - typically the network is implemented as  declarative rules, but can be visualised as a network in a report.
3. Enables different types of Event Processing Engines - apart from the rule agents, you can also have (continuous) query agents.  Rule agents can also be customised as “decision agents” (executing decision rules,  or decision tables), “analytics agents” (executing predictive analytics models in Spotfire S+ or R), or “optimization agents” (executing NuOpt optimization routines in  Spotfire Statistical Services) [*2]

Notes:

[*1] Other candidates for an Event Processing Platform across distributed systems include IBM Infosphere Streams (although IBM is very quiet these days about that), and EventZero. If there are any others please comment them, and if enough we’ll update the  Market Analysis with this classification…

[*2] Note that invoking Spotfire services involves invoking the Spotfire platform under the control of a rules agent; from an architecture point of view these are just SOA services, like calling BusinessWorks services during event processing.

One of the other interesting meetings at the recent OMG standards event was by the Finance Domain Task Force on the requirement for a “Trade Transaction Traceability” standard to meet the regulatory requirements of Dodd-Frank (full name: Dodd-Frank Wall Street Reform and Consumer Protection Act). The idea here is to provide an audit trail (or log), to some SEC-approved XBRL- based XML format, of all the activities and processes that are involved in a “trade” in an investment bank. Thence the need for track-and-trace across front, middle and back office systems.

From a CEP perspective, this is more of a risk management problem than the typical low-latency algo-trading application usually associated with stream-processing CEP in the financial markets. A typical solution for trade track-and-trace would combine some combination of TIBCO Hawk monitoring and TIBCO BusinessEvents CEP to correlate trade processing throughout the bank into a single audit trail while minimising footprints and impacts on existing processes and execution times. Another solution, also CEP-based, would adapt TIBCO Active Service Gateway as a “financial service router” or Services Monitor / Message Interceptor (doing discrete monitoring to create the audit trail).

The OMG is continuing to work on this proposal, and their use of trade and process event correlations (using CEP) could also provide an input to the planned OMG Event Metamodel and Profile standard. And a similar, existing CEP use case is planned as a presentation by an existing TIBCO customer at TUCON this year (in conjunction with an update on TIBCO Hawk technology for the “edge of the Event Processing Network”)…

The EPTS Event Processing Glossary talks about an “Event Processing Network” or EPN - the event processing agents and their connecting channels, with event sources or producers leading through to event consumers in a connected graph (which can be a report or a model). Usually we talk here about TIBCO BusinessEvents - sets of agents doing event processing and storage in a distributed network that corresponds quite nicely to the idea of an EPN. However, there are many other types of event processing agents, and we will start to cover some of those on this blog, starting now with TIBCO Hawk.

Compared to TIBCO BusinessEvents, TIBCO Hawk is a very different type of event processor; that is it is:

• much closer to the event sources (typically system events)
• simpler in its operations (typically event detectors that create alert events) - no inferencing, continuous queries, joins across event types, etc
• much more distributed (smaller agents, 1 per system, with various microagents for collecting or interfacing to different event types) with associated controls and services (distributed rulebase updates on the fly, dashboard display, event capture server, and so on)
• targeting monitoring and management tasks, rather than the more business-oriented “track and trace”. For example, Hawk might be used to monitor SCADA systems, rather than distribute the SCADA data itself. Alternatively, if the SCADA system is itself being used for a monitoring task, then indeed it could work in conjunction with Hawk.

Nonetheless it also shares characteristics with BusinessEvents: it is rule-based, distributed, agent-based, event-driven, event processing. On the subject of whether Hawk is a “CEP solution” then at the academic level probably it is: it is deriving alerts from input events. But at the tool level, it is best to consider Hawk as a high performance filtering and trigger-rule operating system with the emphasis on rule execution simplicity and small agent size.

What is interesting is that Hawk collects information events from sources such as SNMP, JMX, databases, log files, operating systems etc, as well as the means to influence these through control events. Ergo it is very much complementary rather than overlapping with CEP technologies as provided in tools like TIBCO BusinessEvents.

Over the coming weeks we will dive into TIBCO Hawk in more detail to see what it does for some of the larger IT organisations on the planet.

In a blog post about CEP being considered a supporting act for BPM and other technologies, Raimer von Ammon of CITT (a spinoff from University of Regensburg, Germany) commented about the EU proposal for a “U-CEP” (Ubiquitous CEP) research programme. In that comment Rainer mentioned the connection between CEP (an IT paradigm around events) and Biology (covering event-driven neural and evolutionary systems - Epigenetics and Brain Research) which form part of the U-CEP EU research programme proposal. Rainer has kindly elaborated some more on this topic, which IMHO is worthy of a separate post on this blog … and I note has parallels to one of James Odell’s posts on the “inside of an agent”…

Two or three recently emerging disciplines yield to a similar paradigm in order to explain how the environment (actually the universe of global event clouds) and their processing by sensors (respectively receptors and the activation of effectors) drive our daily life, but also the whole evolutionary process of life on our earth since the Big Bang.

The discipline of the New Biology or Epigentics has detected - on the basis of the nanotechnology, microbiology and cell biology – within the last years that the life of cells is determined by its physical and energetic environment and not by their genes. Genes are only the molecular “blue print”, the design pattern, on which the structure or architecture of cells, tissue and organs is based. But the environment as a global cloud of signals, energies or so called events are at the end responsible for the way of life of a cell. In each single cell the mechanisms of life are triggered by the processing of the “events” of its environment and not by its genes.

Since “quantum physics” and the “Heisenberg uncertainty relation” we know today that such events are at last energy as eddies of quarks and photons, but we can define different types of events, also on the physical level. Around 3.5 billion years ago the first monads as bacilli and algae lived on the earth, 650 millions of years ago the multicellular organisms as trees and other plants were created as a more complex and more intelligent life, and since around only 100,000 years the today’s human is a united cell structure of 50 trillions of single cells where each cell is doing event processing on the base of its receptors of its cell membrane. So, a human being can allegedly process 120,000 events per second unconsciously (by the right side of the brain), but logically a human can only process around (I guess) 7 or a few information units at the same time (by the left side of the brain).

Against this background CEP is based on a very similar model. EPAs (event processing agents) are actually cells where event adaptors are the receptors of a cell membrane and the event processing logic based on an EPL (event processing language) are the effectors of a cell. EPNs (event processing networks) are actually a united multicellular structure and so on.

The Epigenetics explains how the environmental signals (events) control the activity of the genes. The primacy of the DNA is no longer valid, and the new found information flow is now called the “primacy of the environment”. Recent experiments of The Epigeneitcs have proved that our beliefs and thinking is energy in the sense of environmental signals. The Epigenetics found that all these kinds of environmental signals influence the regulating proteins which control the activity of the genes and that the global event cloud as environmental signals influences and changes the DNS (so called reverse transcriptase).

What is an individual?
According to the Epigenetics an individual is determined by the sum of its receptors in the cell membrane of all its 50 trillions of (specialized) cells. Each individual is unique because it is controlled or driven by the protein machines of its cells as a result of the received events (signals) of the global event cloud of the universe. The global event cloud does always exist and each event is an undestroyable energy which exists always and forever (see Unus Mundus and space-time continuum).

The Epigenetics explains an individual as an analogy of a broadcasting of television programs where the adjustment buttons are the receptors of the cell membranes which determine which program we receive in which manner. If we add or switch off receptors, we receive a different program or we see the same program differently. This can be understood as the incarnation of an individual. On the basis of such a model we can explain phenonemas like reincarnation or time travel as well.

Results of the recent Brain Research
On the basis of recent experiments of the Brain Research, researchers like Wolf Singer, Gerhard Roth et al. claim that a free will does not exist. They believe that their experiments have proven that the brain region which is responsible for a deliberate decision is only later activated when a signal or event is received while the protein machinery as the activity activator was already started “long” before. This cognition that there is no free will could be supported by the Epigenetics and the functionality of a cell membrane and its receptors and effectors/event adaptors and EPA’s as the basis of the whole protein machinery and as the motor of life.

“If you think you think you only think you think” versus “Cogito ergo sum” (Descartes) …”

For the U-CEP course/textbook I’m gathering some materials since some time which I’ll add to the U-CEP doc within the next days; here are some nice video lectures as a basis for U-CEP enhancements/projects, e.g. from Bruce Lipton, I like him as a great guy and great lecturer:

http://epigenetics.uni-saarland.de/en/home/ (German language only)
http://www.tonyb.freeyellow.com/id68.html
http://www.veoh.com/collection/AgriculturalNews/watch/v378752sbRFQa3F# (video)

Great to hear that TIBCO might support such U-CEP ideas, perhaps Larry Ellison or also Bill Gates would be interested in supporting such things … as an investment in the future

Part 4 in a series by James Odell on agent technologies and event processing. Part 1 covered Why Agent Technology for Event Processing, Part 2 was on the  Relationship between Agents and Events, and Part 3 was on What goes on inside an Agent? …

Conventional software objects can be thought of as passive, because they wait for a message before performing an operation. Once invoked, they execute their “method” and “rest” until the next message. A current trend in software architecture is to design objects that not only react to events but are also proactive. In UML 2.0, these are known as active objects; in the agent community, they are known as agents. Whether they are called active objects or agents, this new direction is radically changing how we design systems.

The Basic Properties of Agents

Figure 1 – Agents have degrees of autonomy, interactivity, and adaptivity

At a system level, an agent can be a person, a machine, a piece of software, or a variety of other things. The basic dictionary definition of agent is something that acts (c.f. an actor in some use case). However, for developing business and IT systems, such a definition may be too general. While an industry-standard definition of agent has yet to emerge, most agree that agents in an IT context are not useful without the following three important properties:

• Autonomous - capability to act without direct external intervention. It has some degree of self-control over its internal state and actions based on its own knowledge or rules.
• Interactive - communicates with the environment and other agents, typically via sensors or interfaces and events.
• Adaptive - capable of responding to other agents and/or its environment during interactions. An agent can also modify its behavior based on past events.

In short, a basic working definition useful for IT purposes is:
An agent is an autonomous entity that can adapt to and interact with its environment.

As illustrated in Fig. 1, agents can be autonomous, interactive, and adaptive - to various degrees. It is important to note that these are not “all or nothing” characteristics - there is no “agent minimum specification” qualification list! These three key agent properties will be discussed in more detail below.

Agents are Autonomous

Figure 2 – Degrees of autonomy

Agents can be thought of as autonomous because each is capable of governing its own behavior (to some extent). Autonomy is best characterized by degrees. At one extreme, an agent could be completely self-governing and self-contained. However, an agent that is fully self-sufficient and does not require the resources of, or interaction with, other entities is very rare-primarily because it would be unmanageably large and unnecessarily complex for any common role. At the other extreme, an agent that is barely able to perform the simplest self-management is of limited use in complex systems. Even traditional objects have some degree of autonomy within their methods.

An agent’s level of autonomy can be designed depending on its role. For example in Fig. 2, conventional objects are not completely dependent on outside resources-having some state and behavior of their own. However, they are traditionally simple in that they are far from being completely independent. Because a “supply-chain agent” is able to reach certain conclusions and make decisions on its own, it has more autonomy than an “object”. By their very nature, supply-chain agents require interactions with other entities to enable a fully functional supply-chain system, since one agent can not adequately support an entire organization’s supply-chain processing - interactions are also a characteristic for the class of agents called event processing agents (EPA). Even more autonomous are manager agents, because their role can involve a high degree of internal decision making through activities such as monitoring and delegation, which depend on outside resources.

Agents are Interactive

Figure 3 – Degrees of interaction in Agents

Interaction implies the ability to communicate with the environment and other entities. As illustrated in Fig. 3, interaction can also be expressed in degrees.

On one end of the scale, method invocations can be seen as the most basic form of interaction. More sophisticated would include those agents that can react to observable events within the environment. For example, food-gathering ants do not invoke methods on each other. Their interaction is indirect through the direct physical effects of their pheromones on the environment. Even more complex interactions are found where agents derive information - complex events - from multiple events over time. Here agents begin to act with “intelligence”. And then there are multiagent systems (MAS) where agents engage in multiple, parallel interactions with other agents. Here agents begin to act as a society.

Finally, the ability to interact becomes most complex when systems involving many heterogeneous agents can coordinate cooperatively and/or competitively through such mechanisms as negotiation and planning.

Agents are Adaptive

An agent is considered adaptive if it can respond to other agents and/or its environment. Entities that fail to adapt to the ever-changing world become extinct: organisms die, companies go out of business, and so forth. For an organization, adaptation enables the system to react effectively to changes in areas such as the market and business environment. For IT systems, adaptation enables systems to react appropriately to enable such effects as system balancing, integrity assurance, and self-healing. The goals and motivations for adaptation are around remaining competitive and useful, and deriving value. When designed properly, individual parts of systems can be empowered to change based on their environment and market conditions.

The primary forms of adaptivity include:

• Simple reaction rules - At a minimum, this means that an agent must be able to react to a simple stimulus-to make a direct, predetermined response to a particular event or environmental signal. Such a response is usually expressed by an IF-THEN form. Thermostats, robotic sensors, and simple events can fall into this category. From atoms to ants, the reactive mode is quite evident. A carbon atom has a rule that states in effect, “If I am alone, I will only bond with oxygen atoms.” Simple reactions can also include business decision rules - such as event-condition-action (ECA) rules.
• Inference rules - Beyond the simple reactive agent is the agent that can appear to reason by following chains of rules. For example, agents can react by making inferences and include patient-diagnosis agents, certain kinds of data-mining agents, and identifying complex events. Such inferences are computed by following a chain of inference rules, which can be described as goal-directed or data and event-driven.
• Learning - Learning is change that occurs during the lifetime of an agent and can take many forms. Popular learning techniques employ reinforcement learning include “credit assignment”, Bayesian and classifier rules, and neural networks. Examples of learning agents would be agents that can approve credit applications, analyze speech, or recognize and track targets.
• Evolution - Evolution is change that occurs over successive generations of agents. A primary technique for agent evolution usually involves genetic algorithms and genetic programming. Here, agents’ adaptive mechanisms can literally be “bred” to fit specific purposes. For example, operation plans, circuitry, and software programs can prove to be more optimal that any product that a human can make in a reasonable amount of time.

Other Agent Properties

In the sections above, three key agent properties were discussed: autonomy, interactivity, and adaptivity. These properties are important because they are essential for agents deployed by IT systems. However, agents may possess various combinations of other properties whose usefulness depends on the application requirements and the agent designer. Here, agent characteristics may include being:

• Sociable - interaction that is marked by friendliness or pleasant social relations, that is, where the agent is affable, companionable, or friendly.
• Mobile - able to transport itself from one environment to another, and geo-aware.
• Proxy - may act on behalf of someone or something, that is, acting in the interest of, as a representative of, or for the benefit of some other entity.
• Intelligent - state is formalized by knowledge (e.g., beliefs, goals, plans, assumptions) and interacts with other agents using symbolic language.
• Rational - able to choose an action based on internal goals and the knowledge that a particular action will bring it closer to its goals.
• Temporally continuous - is a continuously running process.
• Credible - believable personality and emotional state.
• Transparent and accountable - must be transparent when required, yet must provide a log of its activities upon demand.
• Coordinative - able to perform some activity in a shared environment with other agents. Activities are often coordinated via plans, workflows, or some other process management mechanism.
• Cooperative - able to coordinate with other agents to achieve a common purpose; non-antagonistic agents that succeed or fail together. (Collaboration is another term used synonymously with cooperation.)
• Competitive - able to coordinate with other agents such that the success of one agent implies the failure of others (the opposite of cooperative).
• Rugged - able to deal with errors and incomplete data robustly.
• Trustworthy - adheres to Laws of Robotics and is truthful.