互信息改进方法在术语抽取中应用.docVIP

互信息改进方法在术语抽取中的应用 　　摘要：为了确定改进互信息（PMIk）方法的参数k取何值时能够克服互信息（PMI）方法过高估计两个低频且总是一起出现的字串间结合强度的缺点，解决术语抽取系统采用经过分词的语料库时由于分词错误导致的某些术语无法抽取的问题，以及改善术语抽取系统的可移植性，提出了一种结合PMIk和两个基本过滤规则从未经过分词的语料库中进行术语抽取的算法。首先，利用PMIk方法计算两个字之间的结合强度，确定2元待扩展种子；其次，利用PMIk方法计算2元待扩展种子分别和其左边、右边的字的结合强度，确定2元是否能扩展为3元，如此迭代扩展出多元的候选术语；最后，利用两个基本过滤规则过滤候选术语中的垃圾串，得到最终结果。理论分析表明，当k≥3（k∈N+）时，PMIk方法能克服PMI方法的缺点。在1GB的新浪财经博客语料库和300MB百度贴吧语料库上的实验验证了理论分析的正确性，且PMIk方法获得了比PMI方法更高的精度，算法有良好的可移植性。 　　关键词：术语抽取；专业术语；知识获取； 互信息 　　中图分类号：TP391.1 　　文献标志码：A 　　Abstract：The traditional Pointwise Mutual Information （PMI） method has shortcoming of overvaluing the cooccurrence of two lowfrequency words. To get the proper value of k of improved PMI named PMIk to overcome the shortcoming of PMI， and solve the problem that the term extraction cannot be obtained from a segmented corpus with segmentation errors， as well as maintaining the portability of term extraction system， combining with the PMIk method and two fundamental rules， a new method was put forward to identity terms from an unsegmented corpus. Firstly， 2gram extended seed was determined by computing the bonding strength of two adjoining words by PMIk method. Secondly， whether the 2gram extended seed could be extended to 3gram was determined by respectively computing the bonding strength between the seed and the word in front of it and the word located behind it， and then getting multigram term candidates iteratively. Finally， the garbage of term candidates were filtered using the two fundamental rules to obtain terms. The theoretical analysis shows that PMIkcan overcome the shortcoming of PMI when k≥3（k∈N+）. The experiments on 1GB SINA finance Blog corpus and 300MB Baidu Tieba corpus verify the theoretical analysis， and PMIk outperforms PMI with good portability. 　　英文关键词Key words：term extraction； technical term； knowledge acquisition； Pointwise Mutual Information （PMI） 　　0 引言 　　术语抽取在中文信息处理领域中是一项重要的基础性研究课题。随着科技、经济、文化的快速发展，各个学科领域中的术语也发生了很大变化，为了及时了解学科的发展动态，术语抽取的需求应运而生。 　　术语抽取方法总体上有两种：基