基于增量etl的分布式数据交换平台的分析与实现-analysis and implementation of distributed data exchange platform based on incremental etl.docxVIP

摘要数据交换平台是一个在不同数据库、不同数据格式之间，进行数据交换服务 的平台。它要解决的是现阶段不同应用系统之间信息数据无法自由转换的问题。本文的在综合研究了目前数据交换平台所采用的相关技术的基础上，利用增 量 ETL 技术和分布式思想，建立了基于增量 ETL 的分布式数据交换平台，并在实 际工作中得到了有效的应用。具体完成了以下工作：1、提出了基于关键属性比对的增量数据抽取方法，包括关键属性提取算法和 基于关键属性比对的增量数据抽取方法模型，介绍了方法的适用范围，并与全表 比对方法和全表删除插入方法进行对比，分析了空间复杂度和时间复杂度，说明 了其优越性，用实验验证了分析的正确性。2、提出了基于按行数据分片的分布式增量 ETL 优化方法，在增量数据抽取的 基础上，对前人的 ETL 优化框架进行了改进，提出了增量 ETL 优化框架；在同顺 序作业理论的基础上，提出了数据的按行分片机制以及分片数的最佳取值方法， 从理论上证明了其正确性，然后给出了具体算法，用实验验证了算法的正确性并 对实验结果进行了分析。3、在理论研究的基础上，提出了数据交换平台的整体框架，并对关键模块进 行了设计与实现。最后对系统进行了测试，测试结果证明系统具有较好的实用性 和稳定性。主题词： 数据交换；ETL；增量数据抽取；关键属性；数据分片ABSTRACTData Exchange Platform processes data exchange service between different databases and different data formats. It is to solve the problem that information and data can not transform freely between different application systems at present stage.After a systemic study on relate technologys which was adopted in current data exchange platform.Utilizing incremental ETL technology and distribution ideas, we establish distributed data exchange platform based on the incremental ETL which has been effectively applied in practical work. We specifically completed the following work:1、Proposed an new incremental data extraction method based on pivotal attributescomparison, which includes pivotal attributes extraction algorithm and incremental data extraction method model based on key attributes comparison, introduced the available scope of the method, compared with the whole table comparison method and the whole table deleting inserting method, analyzed its space complexity and time complexity, demonstrated its superiority, verified the correctness of the analysis by experiments.2、Proposed a distributed incremental ETL optimization method based on datapartitioning by row. On the basis of incremental data extraction, we have an improved the previous ETL optimization framework, proposed incremental ETL optimization framework; on the basis of same sequence work theory, we proposed mechanism of data partitioning by lin