基于云存储的分布式文件系统分析与优化-analysis and optimization of distributed file system based on cloud storage.docxVIP

摘 要随着互联网的飞速发展，全球数据量每年以指数增长，使得云计算成为了当 前研究与应用的热点。云存储作为云计算的底层服务，是一种架构复杂的分布式 文件系统。因为它具有结构灵活、响应效率高、管理方便等优点，因而成为世界 各国解决数据爆炸性增长方案的首选。Hadoop 分布式文件系统（HDFS）作为当 今最流行的基于云存储的分布式文件系统具有开源、廉价、高容错以及高可扩展 性的特点，在云存储领域占居了者举足轻重的地位。然而，HDFS 因其结构和性能 上的局限性，也存单点失效、并发用户的高延时访问、负载均衡不足等的问题。本文在系统、全面的学习和总结分布式存储系统发展现状和特点的基础上， 重点分析了几种常用的分布式存储系统架构的优缺点，同时设计了一个部分对等 式的多 Namenode 系统架构。该架构通过增加元数据服务器层中部分对等的多个 Namenode，改变了以 HDFS 为代表的集中式存储系统对主节点的单点依赖，降低 了并发用户的等待时延和元数据服务器的平均内存占用率。同时，本文还深入研 究了常用的负载均衡方法，针对 HDFS 存储服务器负载均衡不足的缺点，建立了 磁盘利用率模型和服务阻塞率模型，设计了一种基于本文架构的自适应反馈负载 均衡算法。通过算法性能分析与实验仿真进一步论证了本文设计的算法比 HDFS 系统中的负载均衡算法在系统性能和负载均匀度方面都有一定的优化。关键词：云存储HDFS部分对等 Namenode单点依赖负载均衡AbstractWith the rapid development of the Internet，the growth of global data is exponential per year, which demands cloud computing to be a key issuse in recent research and applications. As an underlying service of cloud computing, cloud storage is a distributed file system with complicated architecture. Because of its advantages of flexible structure, high efficient response, and easy management, cloud storage becomes the first choice for all the countries in the world to solve the problem of explosive growth data. As one of the most popular cloud storage based distributed file system, Hadoop Distributed File System (HDFS) has characteristics of open source, low-cost, high fault tolerance andhigh scalability. HDFS plays an important role in the field of cloud storage. However, because of its structure and performance limitations, HDFS has the problems of single point of failure, high delay of concurrent users, and insufficient load balancing.In this paper, based on studying on and summarizing the current development and characteristics of the distributed file system systematically and comprehensively, we analyze the advantages and disadvantages of several common distributed storage system architectures. Then, we propose a partial peer-to-peer (P2P) Namenodes architecture. The proposed architecture reduces the single point dependence of main node in t