面向非结构化数据查询优化的存储系统-计算机系统结构专业毕业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract In order to solve the issue of unstructured-data storage and provide data storage infrastructure for upper layer applications to improve the development, an unstructured-data query optimization oriented storage system has been designed and implemented. The system provides a unified, simple, transparent, secure data access interface and query methods, and effective organization and management of unstructured-data to guarantee low latency, high throughput and high availability data service. The main idea of unstructured-data query optimization oriented storage system is to learn from the data model and architecture ideas from Bigtable and make improvements. It uses {key: value} format to extract metadata from unstructured-data and build indexes, index information and unstructured-data are persisted to document database, meanwhile, REST architecture is adopted to provide operating system and programming language independent data access interfaces. The system tackles the complexity existed in Bigtable that data processing and control are implemented by users, the work is major focusing on following aspects: (1) Based on the NWR model, make balance between consistency and availability, and build hierarchical cache structure with fine- grained data scale. Meanwhile, pre-caching mechanism is established according to the relevance of data; (2) In order to guarantee eventually consistent, message queue is introduced to synchronize between cache and persistent storage, and it’s also responsible for synchronization and backup in data server nodes to ensure the whole system is always available; (3) Digital signature authentication method based on URL is adopted to ensure the security, and finally, it provides complex query syntax to meet diverse users’ requirements by translating the syntax in the key-value format to SQL-like statement. The storage system is implemented based on fully exploiting the system requirements and cha