面向归档存储的重复数据删除优化方法研究-计算机系统结构专业毕业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract With the improvement of the level of social information, data is becoming increasingly important. At the same time, the demand of storage requirements for enterprise data center is explosively growing. The current storage system is mainly designed for data read and write performance and reliability, ignoring the association between the data and redundant characteristics. This not only causes a waste of storage space, but also makes it difficult for users to effectively manage a large number of complex data structures. For this, De-duplication technology has been proposed in recent years. On the basis of analyzing the characteristics of the metadata access and querying, the layout of data and data reading and writing, a system architecture scheme is proposed, of which the metadata and data access are separated from each other: (1) it uses a tripartite structure consisting of client, metadata server and storage node; (2) access to metadata is separated between client and metadata server, while access to the contents of files is separated between client and storage node, so that the scheme has a high scalability and high access concurrency. In the de-emphasis function, (1) fixed block data partitioning method is used and the hash algorithm MD5 and SHA-1 is used as the fingerprint of a data block; (2) a two layers of Bloom filter is used for the rapid identification and filtering of the hash fingerprints of data blocks, and a B + tree index structure is used as the persistent data storage solution for the metadata of fingerprints. In order to further optimize the I / O performance, (1) a data layout strategy of storing data in accordance with the data flow sub regional is adopted, which can obtain the spatial locality of data access; (2) with the combination of client metadata and data caching mechanism, cache hit rate of file access and file read / write performances can be improved. Finally, a de-duplication system p