爬虫技术介绍（Crawler technology introduction）.docVIP

爬虫技术介绍（Crawler technology introduction） The web crawler technology introduced in 2008 05 on Sunday 04 August 20:07 original link: /viewthread.php tid=5? The web crawler technology With the rapid development of the network, the web becomes the carrier of a large amount of information, how to effectively extract and use these information become a huge challenge. The search engine (Search Engine), for example, the traditional general search engine AltaVista, Baidu, Yahoo and Google, and the entrance! As a guide to help people retrieve information users to access web tools. However, these general search engines also exist certain limitations, such as:? (1) in different areas, different background users tend to have different retrieval purposes and needs, returned by the general search engine results contain a large number of users dont care . ? (2) the general search engine is as far as possible coverage of the network, the contradiction between the limited search engine server resources and unlimited network data resources will further deepen. ? (3) the development of Web data form rich and network technology, database, pictures, audio / video multimedia data appear large, general search engines tend to the information content of intensive and has a certain data structure is not well incapable of action, discovery and access. ? (4) general search engines provide most keyword retrieval based on semantic information to support according to the proposed query. ? In order to solve the above problems, focused crawler to crawl directly related web resources came into being. Focused crawler is a program to automatically download page, it crawl under the established goals, choose to access the world wide web and related links, to obtain the required information. With the general crawler (general? Purpose web crawler), a focused crawler does not pursue large coverage, but aim to capture related to a specific topic , prepare data resource for subject oriented user query. ? 1 foc