文本标签自动抽取的算法优化与应用.pdfVIP

文本标签自动抽取的算法优化和应用 吴斌 224227 （华为 ，深圳 518129 ） 摘 要：本文简述了对文档内容自动抽取标签的各类方法。在“按标签进行文档分类”这个应用场景中，标签不 但需要表现文档主题，还需要有一定的文档覆盖率，以便对类似文档按标签进行分类显示。而监督学习的分类方 法标签范围不能灵活选取，无监督学习的关键词抽取方法，如TFIDF，标签又缺乏文档的覆盖率。所以，在该应 用场景中，本文提出 TFNDDF 算法进行优化，既能灵活地抽取关键词标签，又能使标签在文档列表中有一定的覆 盖率。在实际应用中测试，对比TFIDF 和TFDF 算法都有一定的提高。 另外，如果文档有很多个不同来源的多个标签，我们需要能够自动挑选出比较好的标签，就需要判断标签的 有效性，并对标签与文本的相关性进行计算和排序。现有相关性计算的方法，往往面临数据稀疏，噪音大等问题， 本文基于 LSA 算法，提出LSA-TF 算法，来计算标签与文档的相关性，可以提供一种自动计算标签有效性，并做 相关性排序的方法。 关键词：文档；标签；关键词；TFIDF；TFNDDF ; LSA; 相关性; 排序; LSA-TF 中图分类号： 文献标识码： 文章编号： Text tag auto extract application by algorithm optimization Wu Bin 224227 (HuaWei, Shenzhen 518129, China) Abstract: This paper describes a variety of methods to automatically extract tags for document content. In the scenario of document classification by tag, the tags not only need to show the topic of document, but also need to have some documentation coverage, similar documents in order to be classified according to the tag. The supervised learning classification for tags can not be selected with flexible range, keyword extraction by unsupervised learning methods, such as TFIDF, tag is lack of coverage in document list. So, in this scenario, the TFNDDF algorithm, both flexibly extract keywords tag, but also make the tag has coverage in the document list. In practice, TFNDDF compared to TFIDF and TFDF algorithm has some improvement. In addition, if the document has a number of multiple tags from different sources, we need to be able to automatically pick out some better tags, we need to get the correlations form the tag to the document. The method of calculation correlations, often faced with sparse data, noise and other i