iLOF一种改进局部异常检测算法①.PDFVIP

2015 年 第 24 卷 第 12 期 计 算 机 系 统 应 用 ① iLOF*: 一种改进的局部异常检测算法 王 飞 (复旦大学 计算机科学技术学院, 上海 201203) (复旦大学 上海市数据科学重点实验室, 上海 201203) 摘 要: 异常检测是数据挖掘领域研究的基本问题之一, 已被广泛应用于气象预报、网络入侵检测、电信和信用 卡欺诈侦察等领域. 基于密度的异常检测算法LOF 具有较好的检测效果和适用性, 但其计算量较大, 运行效率不 够高, 且在进行对象之间的距离计算时忽略了不同属性对异常值的不同影响. 针对以上不足, 本文提出了一种高 效的 LOF 改进算法 iLOF*. 该算法利用网格进行数据约简, 从而提高了算法的运行效率; 同时, 在进行对象之间 的距离计算时, 引入信息熵, 给不同属性赋予不同的权值, 从而提高了算法的准确率. 另外, 用 MapReduce 计算 框架将 iLOF*算法并行化, 进一步提高了算法在大规模数据集上的运行效率. 最后的实验结果验证了 iLOF*算法 的有效性和高效性. 关键词: 数据挖掘; 异常检测; 局部异常因子; 信息熵; 并行化 iLOF*: An Optimized Local Outlier Detection Algorithm WANG Fei (School of Computer Science, Fudan University, Shanghai 201203, China) (Shanghai Key Laboratory of Data Science, Fudan University, Shanghai 201203, China) Abstract: Outlier detection is an important branch in the areaof data mining,It has been widely used in weather forecasting, network intrusion detection, telecommunications and credit card fraud detection,etc. LOF algorithm has good detection effect and availability, but its computation is very high, whose efficiency is not good enough,And when calculating the distance between two objects, LOF algorithm ignores the different influence of different properties.To solve above disadvantages, we put forward an improved outlier detection algorithmiLOF*. iLOF* algorithm usesgrid to reduce the data sets, so as to improve the efficiency of the algorithm; at the same time, when calculating the distance between the object, iLOF* algorithm gives different weights to different properties through the introduction of information entropy, which improve the accuracy of the algorithm.In addition, we use the parallel computing framework MapReduce to parallel iLOF * algo