Main Page

From Complex Events Processing and Anomaly Detection

Jump to: navigation, search

1 The Anomaly Detection problem
2 Types of Anomaly Detection
3 Detection Techniques
4 Measuring Performance
5 Key Organisations
6 Latest News
7 Search

The Anomaly Detection problem

The problem of anomaly detection has several key characteristics:

It is rare. Typically less than 0.1% of the transactions will be anomolous. This means that there are very few examples in the data. The data is said to be highly skewed and this is the root cause of many of the problems associated with this class of problem sometime known as Rare Event or Anomaly detection.

Reliability of the data. The marking (aka tagging) of transactions as being anomalous is usually a manual task and is often subject to various sources of error. These errors effectively introduce a level of ‘noise’ into the data. This can be a major problem.

Measuring success. The overall effectiveness of a rare event detection system means very little. In the case of fraud detection, even if it missed all the fraud it could still claim to be 99.9% accurate. The application of good metrics is essential when evaluating these systems. The same is true when measuring the error-rate of the system.

Types of Anomaly Detection

Detection Techniques

An overview of some of the detection techiques:

Detection Techniques

Measuring Performance

Measuring the performance of classifiers of rare events needs care. The article below provides an introduction to some of the issues and how to deal with them.