基于主题树的微博突发话题检测.docVIP

基于主题树的微博突发话题检测 　　摘要：针对传统话题检测方法不能很好处理微博中用语不规范、随意性强、指代不明确以及存在大量网络用语的问题，提出了一种基于潜在狄利克雷分配（LDA）模型的主题树检测方法。首先，运用自然语言处理（NLP）中增大信息熵的方法将相关微博整理成一棵主题树，配合狄利克雷先验α与经验值β随主题数目动态变化的设计思想，结合该模型独特的双重概率统计模式，实现了对文本中每个词“贡献度”的统计，提前处理掉干扰信息，排除垃圾数据对话题检测的影响；然后，利用该“贡献度”作为空间向量模型（VSM）改进后的参数值计算文档间相似度来提取突发话题，达到提高突发话题检测精准度的目的。提出的基于LDA模型的主题树检测方法从F值比对与人工检测两个角度进行了相关实验，实验数据显示该算法不仅可以检测到突发话题，而且获得的结果与原有方法知网模型和TFIDF算法相比分别高出3%、7%原有方法是指什么？是指“人工分类”、“知网模型”还是“TF-IDF”方法。需在摘要中说明清楚。另外，高出3%与7%，是与一种方法比，还是与两种方法比，也需要交代清楚。，且更符合人的判断逻辑。实验结果表明，使用基于LDA算法的主题树检测方法可以用于微博突发话题的检测。 　　关键词：潜在狄利克雷分配；主题树；语义相似度；空间向量模型；话题检测 　　中图分类号： TP391； TP18 　　文献标志码：A 　　Abstract： A kind of topic tree detection method based on Latent Dirichlet Allocation （LDA） model was put forward， in order to solve the problems of nonstandard terms， randomness， uncertainty of reference and large number of network terms in microblog texts， which can not be solved in traditional detection method. Relevant microblogs were reorganized into a topic tree by increasing information entropy in Natural Language Processing （NLP）， combining with the design idea that Dirichelet prior experience value α and experience value β vary with the topic number， then the contribution statistics of every word in the text was achieved using the specific dual probability statistical method of this model. Thus， the interference information would be disposed in advance and the influence of garbage data on topic detection was excluded. Using this contribution as the parameter value of the improved Vector Space Model （VSM）， bursty topics were extracted through calculating the similarity between texts， in order to improve the detection precision of bursty topics. Experiments of the proposed detection method were made from two aspects： comparison of the value of F and the manual detection. The experimental data show that， this algorithm not only can detect the bursty topics， 　　but also can improve the precision about 3% and 7% respectively compared with the HowNet model and the TFIDF （Te