基于Hadoop处理小文件的优化策略.PDFVIP

软件 2015 年第 36 卷 第 2 期 SOFTWARE IT 基于 Hadoop 处理小文件的优化策略 左大鹏，徐薇 (北京交通大学计算机与信息技术学院，北京市 100044) : HDFS(Hadoop Distributed File System)作为开源系统广泛地适用于各类存储服务中，具有高容错，易 扩展， 廉价存储等特点。然而，HDFS 基于单一的服务器 NameNode 来处理元数据信息管理，当处理海量小文件时会造成 NameNode 内存过分消耗以及存储和读取性能并不理想，使 NameNode 成为系统瓶颈。本文提出一种基于 HAR(Hadoop Archive)的优化机制来提高 NameNode 存储元数据信息的内存利用效率和提高读取小文件的访问效率。另外，该策略也 扩展了 HAR 文件追加的优化和为提高访问效率采用索引预取机制。实验结果表明该优化策略能够提高现有 HAR 处理 小文件的能力和访问海量小文件的效率。 : HDFS; 小文件; HAR; 索引策略; 索引预取 中图分类号: TP391.1 文献标识码: A DOI： 10.3969/j.issn.1003-6970.2015.02.023 本文著录格式：左大鹏，徐薇. 基于 Hadoop 处理小文件的优化策略[J]. 软件，2015，36(2)：107111 Optimization Strategy for Handling Small Files on Hadoop ZUO Da-peng, XU Wei (School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China) 【Abstract 】: Hadoop Distributed File System(HDFS) is an open source system which has advantages of high fault-tolerance, scalability and low-cost storage capability and has been widely suitable for storage applications. How- ever, HDFS based on single master (NameNode) to handle metadata management, NameNode would have the memory overhead problem and suffer the performance penalty in both storage and accessing while handling massive small files, NameNode would become bottleneck. This paper proposes a mechanism based on Hadoop Archive (HAR) to improve the memory utilization for metadata and enhance efficiency of accessing small files. In addition, this strategy also ex- tends HAR capability to allow additional files to be inserted into the existing archive files and adopts the preload of index files to improve the access efficiency. Experimental results show that thi