基于Mahout框架的Hadoop平台作业日志分析平台设计与实现.PDFVIP

2015 年 软 件 2015, Vol. 36, No. 11 第 36 卷 第 11 期 COMPUTER ENGINEERING SOFTWARE 国际 IT 传媒品牌 基于Mahout 框架的Hadoop 平台作业 日志分析平台设计与实现 曹政 100876 摘 要：随着 Hadoop 的流行与 Hadoop Yarn 的出现，集群的规模越来越大。在 Hadoop 生态圈中对集群运行状 态的开源实现已经很成熟，但是尚未有对一个对 Hadoop 作业的运行趋势进行统计分析的平台。本文介绍了一个面向 Hadoop Yarn 的作业资源统计分析平台，面向集群管理员与普通用户，以时间、用户双维度对作业进行统计分析，得出 一个 Hadoop 作业运行的标准 关键词：关键词；数据分析；Hadoop ；Yarn ；作业分析；机器学习；Mahout 中图分类号: TP399 文献标识码: A DOI ：10.3969/j.issn.1003-6970.2015.11.012 本文著录格式：曹政. 基于 Mahout 框架的 Hadoop 平台作业日志分析平台设计与实现[J].软件，2015 ，36 （11）：43-47 Design and Implementation of Hadoop’s Log Statistics Analysis System Based on Mahout CAO Zheng School of Computer Science and TechnologyBeijing University of Posts and Telecommunications Beijing 100876China AbstractWith the wild use of Hadoop and appearance of Hadoop Yarnthe scale of clusters is getting larger.And the open source implementation of the clusters operating state monitor system in the Hadoop ecosystem has been very ma- turebut there is not yet a platform for statistical analysis of the running trend of the Hadoop Jobs.In this papera job resource statistical analysis platform Yarn Hadoop is presentedwhich is for the Cluster Administrator and the ordinary userswith the double dimension of date and user to analyze the joband get the standard of Hadoop operation. Key wordsBig data Hadoop Yarn Job analyzer Machine learning Mahout 0 引言 [2]Hadoop Google 2004 MapReduce Hadoop Hadoop Hadoop Apache Hadoop History Server Google MapReduce Hadoop Hadoop