海量数据分析处理方法的研究-计算机系统结构专业论文.docxVIP

第 第 PAGE IV 页 RESEARCH ON APPROACHES TO LARGE-SCALE DATA ANALYSIS ABSTRACT With the development of information technology, the construction of information systems is in transition phase for many fields. Take the financial field for example. In the past, the core of the IT construction is about business trading system. But now, lots of companies have pay more attention on information management system aimed at customers、rick control and gain analysis. This kind of transition needs to collect all business systems data and share the data between departments、platforms. MapReduce is a simple and flexible parallel programming model proposed by Google for large scale data processing in a distributed computing environment. Users specify a map function that processes a key/value pair to generate a set of intermediate values associated with the same intermediate key. Many large-scale data analysis tasks in real world are expressible in this model. Being a type of high performance database management system, parallel database is a product combined between parallel technology and database technology. Parallel database improves the efficiency of working for rational database. Common parallel database can divide into three kinds of architecture method. It includes shared memory、shared disk and shared nothing. Based on the analysis of MapReduce and parallel database, this article proposes a more common and more extensional architecture for large-data processing. Then we make a test on related product. At first, we analyze the MapReduce and parallel database and know their development and practical thinking. Next we make a comparison between them. Then we propose three kinds of method for combining MapReduce and SQL. The methods are making a SQL layer built on a MapReduce engine 、 MapReduce invoking SQL and SQL invoking MapReduce. We think the last method is the best one. Next we propose the general architecture combined MapReduce and parallel database. This architect