数据模型与存储设计-北京大学.DOC

北京大学硕士研究生学位论文 题目： 二00九 年 六 月版权声明 任何收存和保管本论文各种版本的单位和个人，未经本论文作者同意，不得将本论文转借他人，亦不得随意复制、抄录、拍照或以任何方式传播。否则，引起有碍作者著作权之问题，将可能承担法律责任。 摘要 “中国Web信息博物馆”Web InfoMall)[4]，是针对中国互联网信息的搜集、存储与历史浏览服务的海量信息系统，5 年来已经积累超过25 亿中国互联网上出现过的网页，数据量已经超过30TBBighive: An Optimized Distributed Data Storage System on time dimension Abstract Chinese Web Museum (Web InfoMall)[4] is a system for crawling, storing and exhibiting all the web pages being on or once been on the web. For the past 5 years, the system has stored 2.5 billion web pages, and the overall data size is more than 30TB. As its load continues growing, the storage and access of data become more and more difficult. Since the current system cannot meet our daily request due to the specific character of its data distribution, an optimized distributed data storage system is in urgent need. The present Chinese Web Museum has several problems. First, the overall data has a huge size on both space and time dimensions with rapid growth. Second, all requests to this system suffer great limitations on the above 2 dimensions. To resolve the problems mentioned above, this paper propose a brand-new data storage format called TSFile, qualify its suitableness for the InfoMall. After that, The paper describes the design, implementation and evaluation of Bighive, and discusses issues related to the system in detail. Keywords: Bigtable, Chinese Web Museum, Distributed System, Structured Data, Storage System 目录 第一章 引言 - 6 - 1.1 工作背景与动机 - 6 - 1.2 问题描述 - 7 - 1.3 术语定义 - 8 - 1.4 本文结构 - 9 - 第二章 相关工作与研究 - 10 - 2.1 相关系统 - 10 - 2.2 近期相关研究 - 12 - 第三章 数据模型与存储设计 - 13 - 3.1 数据模型 - 13 - 3.1.1 Web InfoMall的数据特征 - 13 - 3.1.2 Web InfoMall的访问特征 - 15 - 3.1.3 Bigtable存储方案在InfoMall应用上的不足 - 17 - 3.2 Bighive存储设计 - 17 - 3.2.1 带时间维度索引的存储格式(TSFile) - 18 - 3.2.2 Tablet管理 - 22 - 3.2.3 TSFile的有效性 - 23 - 第四章 Bighive设计与实现 - 25 - 4.1 体系结构 - 25 - 4.2 Debby 和Tablet元数据的管理 - 25 - 4.3 Tianwang File System和后台数据的存储 - 27 - 4.4 Master主控节点 - 28 - 4.4.1 启动流程 - 29 - 4.4.2 主要功能 - 30 - 4.4.3 负载均