data quality and data cleaning an overview.pptVIP

Data Quality and Data Cleaning: An Overview Theodore Johnson johnsont@ ATT Labs – Research (Lecture notes for CS541, 02/12/2004) Based on: Recent book  Exploratory Data Mining and Data Quality Dasu and Johnson  (Wiley, 2004) SIGMOD 2003 tutorial. Tutorial Focus What research is relevant to Data Quality? DQ is pervasive and expensive. It is an important problem. But the problems are so messy and unstructured that research seems irrelevant. This tutorial will try to structure the problem to make research directions more clear. Overview Data quality process Where do problems come from How can they be resolved Disciplines Management Statistics Database Metadata Overview The meaning of data quality (1) The data quality continuum The meaning of data quality (2) Data quality metrics Technical tools Management Statistical Database Metadata Case Study Research directions Meaning of Data Quality (1) Generally, you have a problem if the data doesn’t mean what you think it does, or should Data not up to spec : garbage in, glitches, etc. You don’t understand the spec : complexity, lack of metadata. Many sources and manifestations As we will see. Data quality problems are expensive and pervasive DQ problems cost hundreds of billion $$$ each year. Resolving data quality problems is often the biggest effort in a data mining study. Example Can we interpret the data? What do the fields mean? What is the key? The measures? Data glitches Typos, multiple formats, missing / default values Metadata and domain expertise Field three is Revenue. In dollars or cents? Field seven is Usage. Is it censored? Field 4 is a censored flag. How to handle censored data? Data Glitches Systemic changes to data which are external to the recorded process. Changes in data layout / data types Integer becomes string, fields swap positions, etc. Changes in scale / format Dollars vs. euros Temporary reversion to defaults Failure of a processing st