科研项目数据入库模块的设计与实现-电子与通信工程专业毕业论文.docxVIP

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract The pace of informatization is increasing rapidly. Implementing “Research management database”, as a means of information construction, to improve information management status of current research and management departments, has very important significance. Informatization has an important early work, the data building, which contains regulating data in different formats from different places, and then organized in a uniform manner, and finally store in the database. Start from the demand, the data warehousing work is divided into two parts: data acquisition and data processing. Based on the characteristics of the existing data source data acquisition is divided into two modes of importing and crawling. This paper analyzes the characteristics of importing and crawling separately then designed and implemented a code framework. For data processing, three representatives as an entry point to detail the source of this problem, the solution is to set forth the data processing needs to pay attention to all aspects of. Finally, through the detailed correctness testing and performance testing, the reliability and stability of the system are verified. Technically, all the work in this article is based on JavaEE platform using Hibernate and Spring Framework, and transaction processing technique to ensure accuracy and consistency of the data warehousing. For the data crawling aspects, we use HttpClient to visit the external sites with java thread pool to execute in parallel crawling task to improve efficiency. Through the appropriate use of design patterns, the entire code framework has a good design and scalability. Framework to achieve in the subject has an important role to play in the actual work completed in advance of the data construction work, guiding significance for the subsequent construction work. Keywords: Data import; Transaction processing; Crawler; Multithreading III III 目 录 摘 要 I Abstract II 缩略语V 1 绪论