基于频繁项集的海量短文本聚类与主题抽取.pdfVIP

计算机研究与发展 D()I：10．7544／issnlOOO一1239．2015 JournalofComputerResearchandDevelopment 52(9)：1941一l953，2015 基于频繁项集 的海量短文本聚类与主题抽取 彭 敏 黄佳佳 朱佳晖。黄济民 刘纪平 (武汉大学计算机学院 武汉 430072) (武汉大学深圳研究院 广东深圳 518057) 。(软件工程国家重点实验室(武汉大学) 武汉 430072) (pengm@whu．edu．cn) M assofShortTextsClusteringandTopicExtractionBasedonFrequentItemsets PengMin ，HuangJiajia，ZhuJiahui。，Huangdimin ，andLiudiping (ComputerSchool，WuhanUniversity，Wuhan430072) (ShenzhenResearch，WuhanUniversity，Shenzhen，Guangdong 518057) 。(StateKeyLaboratoryofSoftwareEngineering(WuhanUniversity)，Wuhan430072) Abstract Shorttextsgenerated in socialmediahavethe characteristicsofvolume。velocity。low qualityandvariety，thusmakethevector-space—basedclusteringmethodsfacethechallengesofhigh— dimensions，featuressparsityandnoisydisturbing．Inthispaper，weproposeashorttextsclustering andtopicextraction (STC—TE)frameworkbasedonthefrequentitemsetsm inedfrom thetexts．This framework firstly studiesthe impactofmulti—featureson the shorttexts’quality．Then，a Iarge amountoffrequentitemsetsaredugoutfrom thehighqualityshorttextsetviasettingalow support level，andasimilaritemsetsfiltering strategyisdevisedto discardmostoftheunimportantfrequent itemsets．Furthermore，based on the frequentitemsets similarity evaluatedbyrelevanttexts，we proposed aclusterself—adaptive spectralclustering (CSA SC)algorithm to form the itemsetsinto — differenttopicclusters．Atlast，thelarge—scaleofshorttextsareclassified into associated clusters accordingtothetopicwordsextractedfrom thefrequentitemsetclusters．Theframeworkistestedon onem illion of SinaW eibo dataset to evaluate the performance ofthe important frequentitemset selectionandclustering，thetopicwordsextraction，andthelargescaleofshorttextsclassification． Experimentalresul