Data Mining and Scalability数据挖掘和可扩展性.ppt

Data Mining and Scalability Lauren Massa-Lochridge Nikolay Kojuharov Hoa Nguyen Quoc Le Outline Data Mining Overview Scalability Challenges Approaches. Overview – Association rules. Case study - BIRCH – An Efficient Data Clustering Method for VLDB. Case Study – Scientific Data Mining. QA DATA MINING Data Mining: Rationale Data size Data in databases is estimated to double every year. Number of people who look at the data stays constant Complexity The analysis is complex. The characteristics and relationships are often unexpected and unintuitive. Knowledge discovery tools and algorithms are needed to make sense and use of data Data Mining: Rationale (cont’d) As of 2003, France Telecom has largest decision-support DB, ~30 TB; ATT was 2nd with 26 TB database. Some of the largest databases on the Web, as of 2003, include Alexa () internet archive: 7 years of data, 500 TB Internet Archive (),~ 300 TB Google, over 4 Billion pages, many, many TB Applications Business – analyze inventory, predict customer acceptance, etc. Science – find correlation between genes and diseases, pollution and global warming, etc. Government – uncover terrorist networks, predict flu pandemic, etc. Data Mining: Definition Semi-automatic discovery of patterns, changes, anomalies, rules, and statistically significant structures and events in data. Nontrivial extraction of implicit, previously unknown, and potentially useful information from data Data mining is often done on targeted, preprocessed, transformed data. Targeted: data fusion, sampling. Preprocessed: Noise removal, feature selection, normalization. Transformed: Dimension reduction. Data Mining: Evolution Data Mining: Approaches Clustering - identify natural groupings within the data. Classification - learn a function to map a data item into one of several predefined classes. Summarization – describe groups, summary statistics, etc. Association – identify data items that occur frequently together. Prediction – predict values or