CS 345AData MiningLecture 1.ppt

CS 345AData MiningLecture 1 Introduction to Web Mining What is Web Mining? Discovering useful information from the World-Wide Web and its usage patterns Web Mining v. Data Mining Structure (or lack of it) Textual information and linkage structure Scale Data generated per day is comparable to largest conventional data warehouses Speed Often need to react to evolving usage patterns in real-time (e.g., merchandising) Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues Size of the Web Number of pages Technically, infinite Much duplication (30-40%) Best estimate of “unique” static HTML pages comes from search engine claims Until last year, Google claimed 8 billion(?), Yahoo claimed 20 billion Google recently announced that their index contains 1 trillion pages How to explain the discrepancy? The web as a graph Pages = nodes, hyperlinks = edges Ignore content Directed graph High linkage 10-20 links/page on average Power-law degree distribution Structure of Web graph Let’s take a closer look at structure Broder et al (2000) studied a crawl of 200M pages and other smaller crawls Bow-tie structure Not a “small world” Bow-tie Structure What can the graph tell us? Distinguish “important” pages from unimportant ones Page rank Discover communities of related pages Hubs and Authorities Detect web spam Trust rank Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues Power-law degree distribution Power-laws galore Structure In-degrees Out-degrees Number of pages per site Usage patterns Number of visitors Popularity e.g., products, movies, music The Long Tail The Long Tail Shelf space is a scarce commodity for traditional retailers Also: TV networks, movie theaters,… The web enables near-ze