面向大数据的ETL设计与实现-计算机软件与理论专业毕业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract With the development of information technology, more and more data are generated. These data, both contain large amounts of structured data, and also contains large amounts of unstructured data and semi-structured data. Data capacity becomes bigger, data growth rate becomes faster, the format of the data becomes more complex, the demand of data processing becomes more urgent, all these bring new challenges to ETL. Design a ETL which can effectively handle big data has important practical significance. First, according to the characteristics of big data, we puts forward the system functional objectives and performance goals on the basis of the needs analysis. According to the big data processing requirements , we designs a ETL architecture with effective support for big data processing and designs the ETL workflow. In order to optimize ETL workflow, improve the efficiency of data processing, we designs the rules of ETL by classification, merger which is suitable for big data environment. The same time, according to the characteristics of the MapReduce,we give the design of MapReduce workflow and the mapping rules between MapReduce workflow and ETL workflow. Again, the realization of the system is introduced. Universal data access module implements the data extraction and loading, especially for unstructured data extraction. The workflow module is used to parse the metadata to generate local workflow and MapReduce workflow model. The execution module is used to complete the procedure from data extraction to data loading. Metadata management module realize the storage of metadata. Finally, the experiment showed that the system realizes the function of big data processing and meets the design goals.Through the use of MapReduce can enhance the ETL data processing efficiency in the certain degree. Key words：data processing, extraction transformation loading, big data, workflow PAGE IV PAGE IV 目 录 摘要I Abstract II 1