隐马尔可夫模型和贝叶斯模型词义消歧对比研究.pdfVIP

隐马尔可夫模型和贝叶斯模型词义消歧对比研究 隐马尔可夫模型和贝叶斯模型词义消歧对比研究 丁江伟 刘挺 卢志茂 李生 哈尔滨工业大学计算机学院信息检索研究 (哈尔滨工业大学 321 信箱，哈尔滨 150001) E-mail:{djw, tliu, lzm, ls}@ir.hit.edu.cn 摘 要：词义消歧是自然语言处理中的一个难点和热点问题。现阶段，多义词消歧的研究大多采用几 个有代表性的歧义词作为研究与测试的对象，与实际应用还存在一定的距离，作者针对真实的应用情况， 对大规模文本进行了词义消歧的研究。本文对两个经典的统计消歧模型解决大规模的词义消歧难题的优缺 点进行了考察，一阶隐马尔可夫模型考察了邻接的上下文，有些时候距离歧义词较远的词语往往对词义的 确定起着至关重要的作用，所以这种方法的消歧正确率比较低，开放测试在 85%左右；单纯贝叶斯概率模 型的消歧方法在抽取上下文特征时加大了上下文的窗口，使与多义词消歧相关的信息充分考虑进来，这种 方法的开放消歧正确率最高可达 92%，取得了比较好的效果。由此证明了贝叶斯模型词义消歧的比较优势 和有效性。 关键词：词义消歧，自然语言处理，隐马尔可夫模型，单纯贝叶斯模型 Comparative Study on Word Sense isambiguation Based on HMM and Naive Bayes Model Ding Jiangwei Liu Ting Lu Zhimao Li Sheng School of Computer Science and Technology of HIT (Harbin Institute of Technology 321 Box，Harbin 150001) E-mail:{djw, tliu, lzm, ls}@ir.hit.edu.cn Abstract: Word Sense Disambiguation has always been a key problem and one of the difficult points in Natural Language Processing. Presently, only some ambiguous words are selected as disambiguated objects in many word sense disambiguation researches, which have great limitations in real application. In this paper, differently, large-scale real texts are researched applying supervised word sense disambiguation approach based on statistical model. By a mass of contrastively experiments based on Hidden Markov Model and Naive Bayes Model, we find that the accuracy of the former model is only 85.05% in open test but 92.00% of the latter, substantiating the wonderful performance of Naive Bayes Model. c 哈尔滨工业大学