基于Scrapy框架的威胁情报抓取以及处理系统的设计与实现.docxVIP

内容摘要 随着安全技术的发展,网络安全防御措施成为了一种重要资产,如何通过现有的威胁情报数据进行快速有效的提取和分析数据是目前的一个研究热点。针对开源威胁网站以及博客等海量安全数据采用Scrapy框架设计网络爬虫对数据进行提取,并将其以知识图谱所需要的底层数据存储，通过scrapy爬虫技术，可以使得通过对各个数据源的爬虫文件，针对性的爬取所需要的信息以及信息的解析整合入库，从而在这个基础上进行多方向的研究。安全防范的目标不仅仅是在攻击到达前阻挡它，还应该包括阻断它达到其最终目的。 本次的毕业设计课题将围绕基于对开源威胁情报网站的爬取与数据解析系统进行设计与实现。系统将实现由Scrapy爬虫框架爬取开源威胁情报网站上公开的数据进行解析入库，通过Flask_admin+nginx以及pyecharts来展示所抓取的数据以及相对于的查询与导出功能；本系统由爬虫模块、数据解析模块以及数据展示模块组成，很大程度上实现了APT知识图谱的数据需求。 关键词：APT知识 Scrapy 爬虫技术 Abstract With the development of security technology, network security defense measures have become an important asset. How to extract and analyze data quickly and effectively through the existing threat intelligence data is a research hotspot at present. For massive security data such as open-source threat websites and blogs, we use the scrapy framework to design web crawlers to extract data, It can store the bottom data needed by knowledge map, and through the technology of crawler, it can integrate the crawler files of each data source, the information needed for targeted crawling and the analysis of information into the database, so as to carry out multi-directional research on this basis. The goal of security is not only to block the attack before it arrives, but also to block it to achieve its ultimate goal. This graduation project will focus on the design and implementation of crawling and data analysis system based on open source threat information website. The system will be implemented by scrapy crawler framework to crawl open-source threat information website open data for analysis and storage, and display the captured data and relative query and export functions by flask_admin + nginx and pyecharts; the system is composed of crawler module, data analysis module and data display module, which largely realizes the data requirements of APT knowledge map. Key words: APT knowledge Scrapy Crawler Technology 目 录 TOC \o 1-3 \h \z \u 第一章 绪论 1 1.1 课题背景及意义 1 1.2 国内外研究现状 1 1.3 课题研究内容 2 第二章 开发技术与工具 3 2.1 Scr