From Data Mining toKnowledge Discovery An Introduction.ppt

COM 307: Machine Learning and Data Mining From Data Mining toKnowledge Discovery: An Introduction Gregory Piatetsky-Shapiro KDnuggets Outline Introduction Data Mining Tasks Application Examples Trends leading to Data Flood More data is generated: Bank, telecom, other business transactions ... Scientific Data: astronomy, biology, etc Web, text, and e-commerce More data is captured: Storage technology faster and cheaper DBMS capable of handling bigger DB Examples Europes Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation session storage and analysis a big problem Walmart reported to have 24 Tera-byte DB ATT handles billions of calls per day data cannot be stored -- analysis is done on the fly Growth Trends Moore’s law Computer Speed doubles every 18 months Storage law total storage doubles every 9 months Consequence very little data will ever be looked at by a human Knowledge Discovery is NEEDED to make sense and use of data. Knowledge Discovery Definition Knowledge Discovery in Data is the non-trivial process of identifying valid novel potentially useful and ultimately understandable patterns in data. from Advances in Knowledge Discovery and Data Mining, Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy, (Chapter 1), AAAI/MIT Press 1996 Related Fields Outline Introduction Data Mining Tasks Application Examples Data Mining Tasks: Classification Classification: Linear Regression Classification: Decision Trees Classification: Neural Nets Data Mining Central Quest Data Mining Tasks: Clustering Major Data Mining Tasks Classification: predicting an item class Clustering: finding clusters in data Associations: e.g. A B C occur frequently Visualization: to facilitate human discovery Estimation: predicting a continuous value Deviation Detection: finding changes Link Analysis: finding relationships … www.KDData Mining Software Guide Outline Introduction Data Mining