outlier 数据仓储与数据挖掘 教学 讲义.pptVIP

* Contents Outliers as Noises Distribution-based Outliers Depth-based Outliers Distance-based Outliers Density-based Outliers Deviation-Based Outlier Case Study Deviation-Based Outlier Detection Simulate a mechanism familiar to human being: after seeing a series of similar data, an element disturbing the series is considered an exception Sequential Exception Techniques OLAP Data Cube Techniques Sequential Exception Select subsets of data Ij (j=1,2,…,n) from the dataset I Compare the dissimilarity of I and (I-Ij) Find out the minimum subset Ij that reduce the disimuliarity the most Smoothing factor D is a dissimilarity function C is a cardinality function, for example, the number of elements in the dataset Example Ij I- Ij C(I- Ij) D(I- Ij) SF(Ij) {} {1,4,4,4} 4 1.69 0.00 {4} {1,4,4} 3 2.00 -0.93 {4,4} {1,4} 2 2.25 -1.12 {4,4,4} {1} 1 0.00 1.69 {1} {4,4,4} 3 0.00 5.07 {1,4} {4,4} 2 0.00 3.38 {1,4,4} {4} 1 0.00 1.69 Let the data set I be the set of integer values {1,4,4,4} Note, when Ij = {}, D(I) = D(I-Ij) = 1.69, SF(Ij)=0 When Ij={1}, SF(Ij) has the maximum value, so {1} is the outlier set OLAP Data Cube Technique Deviation detection process is overlapped with cube computation Precomputed measures indicating data exceptions are needed A cell value is considered an exception if it is significantly different from the expected value, based on a statistical model Use visual cues such as background color to reflect the degree of exception * Contents Outliers as Noises Distribution-based Outliers Depth-based Outliers Distance-based Outliers Density-based Outliers Deviation-Based Outlier Case Study * E-bay Fraud detection Virtual marketplaces: online auctions have been thriving as a business over the past decade. People from all over the world trade goods worth millions of dollars every day using these. EBay, the world’s largest auction site, reported a third quarter revenue of $1,449 billion, with over 212 million registered users . These figures represent a 31%