异常检测方法简介Anomaly Detection A Tutorial.ppt

Anomaly Detection: A Tutorial Arindam Banerjee, Varun Chandola, Vipin Kumar, Jaideep Srivastava University of Minnesota Aleksandar Lazarevic United Technology Research Center Outline Introduction Aspects of Anomaly Detection Problem Applications Different Types of Anomaly Detection Case Studies Discussion and Conclusions Introduction We are drowning in the deluge of data that are being collected world-wide, while starving for knowledge at the same time* Anomalous events occur relatively infrequently However, when they do occur, their consequences can be quite dramatic and quite often in a negative sense What are Anomalies? Anomaly is a pattern in the data that does not conform to the expected behavior Also referred to as outliers, exceptions, peculiarities, surprise, etc. Anomalies translate to significant (often critical) real life entities Cyber intrusions Credit card fraud Real World Anomalies Credit Card Fraud An abnormally high purchase made on a credit card Cyber Intrusions A web server involved in ftp traffic Simple Example N1 and N2 are regions of normal behavior Points o1 and o2 are anomalies Points in region O3 are anomalies Related problems Rare Class Mining Chance discovery Novelty Detection Exception Mining Noise Removal Black Swan* Key Challenges Defining a representative normal region is challenging The boundary between normal and outlying behavior is often not precise The exact notion of an outlier is different for different application domains Availability of labeled data for training/validation Malicious adversaries Data might contain noise Normal behavior keeps evolving Aspects of Anomaly Detection Problem Nature of input data Availability of supervision Type of anomaly: point, contextual, structural Output of anomaly detection Evaluation of anomaly detection techniques Input Data Most common form of data handled by anomaly detection techniques is Record Data Univariate Multivariate Input Data – Nature of Attributes Nature of attributes