基于YARN和Spark框架的数据挖掘算法并行研究-计算机技术专业论文.docxVIP

ABSTRACTWith ABSTRACT With the rapid development of science and technology， the depth of Internet applications and the popularization of personal computers，tablets，smart phones，smart home and other terminal equipment，the amount of worldwide data is increasing at a rapid rate，we have entered an era of Big Data． Faced with such a 1arge—scale mass data， the traditional model (stand—alone model and the traditional paral lel computing model) has been difficult to deal with it， and distributed computing platform emerged at the right moment proVides a new way for massive data processing． Compared with the traditional model of parallel computation， the distributed computing platform completes segmentation data， task allocation， parallel processing，fault tolerance and other features by the underlying data， and it has the characteristics of easy to expand， learn， use and deploy， etc． The distributed computing platform is a simple abstract parallel progra皿ming model． Because the users only need to concentrate on solving paral lel comput ing tasks that they need to s01ve and do not need to concern for parallel implementation details， so it greatly simplifies the design of parallel programs． Used in parallel III 万方数据 algorithm algorithm design， the model has a high practical application Value for enhancing the efficiency of the algorithm．This paper use the model for Parallel study DBSCAN algorithm of elustering analysis， the results are as follows： (1)It proposed a data sub—grid algorithm based on grid unit． The algorithm diVides data sets of each partition into the rectangular block whose side is DBSCAN Eps radius，and this will fast accelerate the speed of data centralized data object to find Eps neighborhood． This eliminates the need to find Eps neighborhood of a data object in all the data set of the entire partition，so the range of finding Eps Neighborhood is narrowed to eight adjacent cells of data object． The experimental results show that the algorithm iⅡlproves the clu