ICS 278 Data MiningLecture 17 Web Log Mining.ppt

ICS 278: Data MiningLecture 17: Web Log Mining Padhraic Smyth Department of Information and Computer Science University of California, Irvine Outline Basic concepts in Web log data analysis Predictive modeling of Web navigation behavior Markov modeling methods Analyzing search engine data Ecommerce aspects of Web log mining Introduction Useful to study human digital behavior, e.g. search engine data can be used for Exploration e.g. # of queries per session? Modeling e.g. any time of day dependence? Prediction e.g. which pages are relevant? Applications Understand social implications of Web usage Design of better tools for information access E-commerce applications How our Web navigation is recorded… Web logs Record activity between client browser and a specific Web server Easily available Can be augmented with cookies (provide notion of “state”) Search engine records Text in queries, which responses were clicked on, etc Client-side browsing records Produced for research purposes as part of a study Automatically recorded by client-side software Harder to obtain, but much more accurate than server-side logs Other sources Web site registration, purchases, email, etc ISP recording of Web browsing Web Server Log Files  Server Transfer Log: transactions between a browser and server are logged IP address, the time of the request Method of the request (GET, HEAD, POST…) Status code, a response from the server Size in byte of the transaction Referrer Log: where the request originated Agent Log: browser software making the request (spider) Error Log: request resulted in errors (404) Example of Web Log entries Apache web log: 0 - - [29/Mar/2002:03:58:06 -0800] GET /~sophal/whole5.gif HTTP/1.0 200 9609 /~sophal/whole.html Mozilla/4.0 (compatible; MSIE 5.0; AOL 6.0; Windows 98; DigExt) 6 - - [29/Mar/2002:03:59:40 -0800] GET /~alexlam/resume.html HTTP/1.0 200 2674 - Mozilla/5.0 (Slurp/cat; slurp@; /slurp.html)“ 42 - - [29/Mar/2002:03:00:14 -0800] GET /~tahir/in