[计算机软件及应用]web文本挖掘.ppt

Data Mining: Concepts and Techniques 第九章 WEB挖掘 什么是万维网 万维网是目前一个巨大的、分布广泛的全球性信息服务中心，它涉及新闻、广告、消费信息、金融管理、教育、政府、电子商务和许多其它信息服务。WEB还包了丰富和动态的超链接信息，以及WEB页面的访问和使用信息，这为数据挖掘提供了丰富的资源。 WEB对有效的资源和知识发现还是具有极大的挑战性。 Finding Information On the Web 两种获得web信息的方法: Browsing: From a starting point, navigate through hyperlinks to find desired documents. Yahoo’s category hierarchy facilitates browsing. Searching: Submit a query to a search engine to find desired documents. Many well-known search engines on the Web: Google, MSN, Yahoo, AltaVista, Fast, Lycos, , etc. Searching is the second most popular activities on the Web behind email. 浏览和查找方式的比较 Category hierarchy is built mostly manually while search engine databases can be created automatically. Search engines can index much more documents than a category hierarchy. Browsing is more accurate and more focused (less junk will be encountered) than searching. The Web Has Many Other Rich Structures 搜索引擎—为什么查询很重要？ 信息就在你的指尖 Fundamental pervasive 在线广告 查询是一个在线广告的分布渠道 搜索引擎—为什么查询很重要？ 搜索引擎的发展 Web Search 1.0 – Traditional Text Retrieval Web Search 2.0 – level Relevance Ranking Web Search 3.0 – Object-level Structured Search The Trend in Web Search Web Search 1.0: Traditional Text Retrieval Relevance ranking based on term distribution – Term frequency (TF) * Inverse document frequency (IDF) – Language models – …… Web Search 1.0: Traditional Text Retrieval Web Page Has Richer Structure Than Plain Text ? Different term types and formats ? Hyperlink structure ? 2D visual layout structure Web Search 1.0 ? Web Search 2.0 The first major improvement in the history of Web search – Link analysis ? PageRank HITS – Relevance ranking = IR Score + PageRank Web Search 2.0 ???Web Search 3.0 Object Level Vertical Search (MSRA Libra) Object Level Vertical Search (MSRA Libra) Web Object Identification Object-level Link Analysis 网络蜘蛛 网络蜘蛛即Web Spider，是一个很形象的名字。把互联网比喻成一个蜘蛛网，那么Spider就是在网上爬来爬去的蜘蛛。网络蜘蛛是通过网页的链接地址来寻找网页，从网站某一个页面（通常是首页）开始，读取网页的内容，找到在网页中的