分布式微博采集系统的设计与实现-通信与信息系统专业论文.docx

华中科技大学硕士学位论文 华 中 科 技 大 学 硕 士 学 位 论 文 II II Abstract As a rising internet platform, microblog has great influence on the way people use media and the information propagation pattern. It has become a very important media platform with the most instant news and the most active users among all the social media. To the end of December 2012, the microblog user number in China has reached 309 million, which is 54.7 percentages of all the Internet users in China. The researches on microblog have great significance to society and research by helping understanding the trends of public opinion, tracing hot topics and dividing social groups in social networking services. All these studies require large amount of microblog data for support. Although there are already many organizations focusing on microblog data collection, there is still no mature collecting method as for the traditional internet applications. Therefore, research on microblog data collection is of great significance. This research designs and implements a distributed microblog crawling system, including the following parts: 1) Designing and implementing the method of microblog data collection through application programming interface of open platform, mainly focusing on the research and use of authorization in open platform and programming interface. 2) Designing and implementing the method of microblog data collection through login simulation and webpage parsing, mainly focusing on the research and use of single sign on and webpage structure. 3) Combining the two methods above, designing the general framework, modules and database, and implementing an efficient and expandable microblog data collecting system by using a distributed strategy. Using this system, the user can simply input the microblog accounts that need to be collected, and select the type of data to be collected, the results will be feedback quickly. It’s also convenient to adjust the collection rate by modifying the amount of crawlers.