基于增量式爬虫的搜索引擎系统的设计.docVIP

基于增量式爬虫的搜索引擎系统的设计与实现 摘 要 随着社会的不断发展，信息增长的也越来越快，大量的数据涌现在我们面前，对于这些数据，我们对信息的查找和提取也会更加困难。如何更快速、更精准的找到我们所需要的信息，获取到有用的信息成为了很重要的一个技术。与传统的搜索引擎对比，本次开发的搜索引擎具有针对性，更新频率快，可以实时的对数据进行爬取，使用户每次获取到的数据都是最新的。 本次课题在Windows平台下基于Python中的Scrapy框架对网络数据进行爬取，将获取到的数据进行本地保存以及Redis的分布式保存。通过当下最流行的搜索引擎elasticsearch进行索引与数据进行连接，然后通过Django框架快速搭建搜索网站，讲解如何完成Django与elasticsearch的搜索查询交互，最后通过Scrapyd完成对Scrapy的线上部署。使用户可以针对性的对信息进行搜索与查询。通过一系列的实验验证，获取到的数据都很好的存储在了本地以及Redis，证明了增量式爬虫在传统搜索引擎上的优点以及优势。 关键词：Scrapy、搜索引擎、增量式、Django框架、爬虫 Design and Implementation of Search Engine System Based on Incremental Crawler ABSTRACT With the continuous development of society, information is growing faster and faster, and a large amount of data appears in front of us. For these data, we will find it more difficult to find and extract information. How to find the information we need faster and more accurately and obtain useful information has become an important technology. Compared with traditional search engines, the search engine developed this time is targeted and updated frequently. It can crawl data in real time, so that the data obtained by users every time is the latest. This topic crawls network data based on the Scrapy framework in Python under the Windows platform, and saves the obtained data locally and Redis distributed preservation. Through the most popular search engine elasticsearch for indexing and data connection, then quickly build a search website through the Django framework, explain how to complete the search query interaction between Django and elasticsearch, and finally complete online deployment of Scrapy through Scrapyd. Users can search and query information in a targeted manner. Through a series of experimental verifications, the data obtained is well stored locally and in Redis, proving the advantages and advantages of incremental crawlers in traditional search engines. Keywords: Scrapy, search engine, incremental, Django framework, crawler 目录 TOC \o 1-3 \h \z \u 1 绪论 1 1.1搜索引擎的背景 1 1.2搜索引擎