基于倒排索引的微博话题检测-计算机技术专业论文.docxVIP

哈尔滨工业大学工程硕士学位论文 哈尔滨工业大学工程硕士学位论文 - - II - Abstract With the rapid development of micro-blog, an emerging social network, more and more users use it and post news. There is a great demand in detecting hot events in micro-blog data, because of the fast propagation of micro-blog and great significance of news spread. However there is huge challenge in topic detection and tracking because of the large-scale, much noise and shorter text of micro-blog data. After analysis the shortcomings of the traditional topic detection and tracking algorithm, this paper proposed an inverted index based method to increase the processing of the algorithm, but have no harm to the accuracy of the algorithm. Some artificial rules are established to remove the noise after analysis the micro-blog data. Topic detection and tracking algorithm is processed after inverted index is build. When get the new events, sort the events based on the entropy of the event and number of user. The top 20 events of the list are merged with the old events. The old events are collected based on the aging theory. This paper also analysis the result of AP cluster algorithm on micro-blog data. In order to verify the efficiency increasing of the algorithm, the processing time of inverted index based algorithm and traditional SINGLE-PASS algorithm are compared on several different scale data set. The inverted index algorithm is 6 times faster than the SINGLE-PASS algorithm. A test set was build by artificial label because of no standard corpus. The test set include 26 events, total of 2817 documents. The algorithm shows to get a better result after analysis the experiment results. The effective of different weight compute method is also compared. And the disadvantages and advantages of AP clustering algorithm and SINGLE-PASS algorithm is compared. Because of the large scale of micro-blog data, some event get by the algorithm is not interested. After sort the event by using the entropy and the number of user, the eve