海量多结构数据智能检索中的存取方法研究-计算机技术专业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract With the rapid development of Internet, Data Information showed exponential growth, particularly in the smart retrieval, in terms of the way of Data–Handling, the traditional stand-alone mode has already could not be well adapt to the current massive data processing. With Hadoop Distributed File System as well as the emergence of MapReduce distributed parallel processing of vast amounts of information, that way has gradually transformed into a distributed parallel processing mode. In the intelligent retrieval of the mass of information, it can be efficiently processed by MapReduce. After the feasibility analyzing of the hadoop and mapreduce distributed parallel processing environment for storage and retrieval of the massive-data, based on the requirement of intelligent retrieval for semantic and content features of multi-structed massive-data , we proposed a kind of organization policy and access method. In terms of the features of Lucene’s full-text retrieval technology, we designed to establish the inverted index and the positive index for the text-file of the characteristics of massive-data. Because of Hbase distributed database can be well adapted to the storage of multi-structured data, we designed the high-dimensional feature library, including semantic and content features. In terms of distributed parallel processing of the massive-information, the larger files can be obtained higher efficiency, so we proposed and implemented the way of dealing with massive large file access method for small files, combined with the characteristics of distributed file system and the small files. Based on the above access methods, design and implementation the intelligent retrieval, including single-mode and multi-modal clustering similarity retrieval, and optimized for caching. In terms of MapReduce distributed processing, we proposed and implemented a method of optimization for it’s shuffle. After a number of experiments, t