数据清洗解决方案和其元数据管理.pdfVIP

数据库元数据 能让用户对其中的过程 规则和知识进行定义 编辑和 增删 用户通过该工具能方便地实现对清洗过程的管理 具体 Cockpit 数据清洗过程的实现采用了上述基于元数据管理的模 型 包括对Cockpit 数据清洗过程中的业务逻辑进行分析 设计Cockpit 的数据清洗流程 实现数据清洗算法 该流程和算法实现了异常数据的 判断规则 自动对数据进行检验 对异常数据进行分类 用简单的数据 挖掘算法实现对数据字典自动更新 并以Email 形式自动通知相关人员 最终达到增强数据清洗能力 提供更准确的数据并简化维护工作量的目 的 通过实验比较 本文的数据清洗解决方案相对原有的数据清洗过程 具有自动化程度高 人为管理少 结构清晰 便于维护和运行效率高等 特点 关键词 数据清洗 数据质量 元数据 ETL 数据仓库 Cockpit V DATA CLEANSING SOLUTION ABSTRACT Data cleansing, also called data scrubbing, deals with detecting and removing errors and inconsistencies from data in order to improve data quality. Present issue is the dirty data problem in Intel DSS tool, Cockpit, especially for data of product definition. Regarding key data cleansing technology, this essay provides data cleansing solution based on metadata management from two main objects. One is metadata management, the other is practical development based on its metadata model. Cockpit data cleansing metadata management includes building data cleansing metadata model, designing and developing metadata management software and establishing data cleansing structure based on metadata management. Idea of design based on metadata management is described as following: Metadata describes contents, quality, details, and other characteristics of data. Regarding metadata model, this essay classifies knowledge metadata into two parts: logic and information. Logic is a kind of rule, a detailed method processing information, detailed design and accomplishment done by designer. Information is abstract reflection from detailed object of real world, which can be updated by program automatically. Logic means data cleansing operation, vali