Mining from Data Streams LUMS挖掘从数据流中发现.pptVIP

CS 636 - Adv. Data Mining (Wi 04/05) CS 636 - Adv. Data Mining (Wi 04/05) Mining from Data Streams - Competition between Quality and Speed Adapted from: Wei-Guang Teng (鄧維光) and S. Muthukrishnan’s presentations Streaming: Finding Missing Numbers Paul permutes numbers 1…n, and shows all but one to Carole, in the permuted order, one after the other. Carole must find the missing number. Carole can not remember all the numbers she has been shown. Streaming: Finding Missing Numbers Carole cumulates the sum of all the numbers that she has been shown. At the end she can subtract this sum from n(n+1)/2 Analysis Takes O(log n) bits to store the partial sum Performs one addition each time a new number is shown (takes O(log n) time per number) Performs one subtraction at the end (takes O(log n time) Data Streams (1) Traditional DBMS – data stored in finite, persistent data sets New Applications – data input as continuous, ordered data streams Network monitoring and traffic engineering Telecom call detail records (CDR) ATM operations in banks Sensor networks Web logs and click-streams Transactions in retail chains Manufacturing processes Data Streams (2) Definition Continuous, unbounded, rapid, time-varying streams of data elements Application Characteristics Massive volumes of data (can be several terabytes) Records arrive at a rapid rate Goal Mine patterns, process queries and compute statistics on data streams in real-time Data Stream Algorithms Streaming involves Small number of passes over data. (Typically 1?) Sublinear space (sublinear in the universe or number of stream items?) Sublinear time for computing (?) Similar to dynamic, online, approximation or randomized algorithms, but with more constraints. Data Streams: Analysis Model Motivation 3 Billion Telephone Calls in US each day 30 Billion emails daily, 1 Billion SMS, IMs Scientific data: NASAs observation satellites generate billions of readings each day. IP Network Traffic: up to 1 Billion packets per hour pe