基于hadoop企业海量数据存储与计算平台设计与实现-design and implementation of enterprise mass data storage and computing platform based on hadoop.docxVIP

摘要摘要近年来，随着数据存储与分布式计算技术的发展，国内外掀起了研究海量数 据存储与计算的新高潮，以 Google、Amazon、IBM 为代表的信息技术公司陆续推 出了自己的海量数据存储与计算的构架，其中最引人注目的新理念就是“云计算” 解决方案。作者所在的企业为互联网数据服务公司，每天产生的数据是 TB 级别，数据的 存储与管理成了公司面临一大难题。结合海量数据存储与计算的相关技术，吸纳 业界领先公司的工作成果，设计一套符合公司业务要求、支持海量数据应用服务 的数据存储与计算平台，寻求适合企业海量数据的存储和计算的“云计算”解决方案 是本文的目标。本文主要的工作是海量数据与计算平台设计、开发和测试。首先基于 Hadoop 框架体系，设计了符合海量数据存储的平台架构以及海量数据处理的算法与机制； 其次，根据平台的设计方案，给出了基于 Hadoop 的分布式存储与计算平台的实施 方案，并针对支持海量大文件存储的 GHFS 和支持海量小文件存储的 TGHFS 给出 了存取接口的实现，构建了海量数据查重和海量数据分布式索引和检索的的海量 数据分布式计算应用；最后对平台的文件读写功能、存储性能和检索的速度进行 了测试。测试表明，海量数据存储与计算平台完全达到设计总体目标，为公司的 海量数据存储与计算提供了理想的解决方案。关键词：Hadoop海量数据存储与索引AbstractAbstractDuring these years, with the development of data storage and distributed computing technology, the research area of large-scale data storage and computing is coming to a new high tide. Information technology companies such as Google, Amazon and IBM publish their large-scale data storage and computing framework one after another, cloud computing solution is the most new concept among these framework which catches the world developer’s eyes.The author’s company is an Internet data service company at TB-scale data increasing speed every day, so data storage and management becomes a difficult problem in our company. Combining large-scale data storage and computing technology with the outstanding companies achievement in this area, we design a suit of data storage and computing platform which is suitable for company’s business requirements and supporting large-scale data applications. And look forward to get a large-scale data storage and computing solution to apply to company.The main job of this thesis is large-scale data and computing platform design，development and test. Firstly, the thesis designs a platform construction for large-scale data storage based on Hadoop, and gives a storage framework construction for large structural and unstructured data and large-scale data processing algorithms and mechanism. Secondly, According to the platform design proposal, the thesis expoun