基于统计的语言处理模型的局限性 - 北京大学中国语言学研究中心.pdf

2004 5 May, 2004 2 Applied Linguistics No12 * 袁毓林 ( 北京大学中文系 北京 100871) [ ] 本文通过介绍和评论基于统计的语言处理模型的工作原理和有关的 用 实例, 从语言学理论的角度来说明统计模型的局限性通过讨论N 元语法模型及其 在词类标注上的 用, 展示基于统计的语言处理模型的工作原理及其 用讨论了 语言结构的递归性特点和语言学知识的结构依赖性特点, 指出递归嵌入会使得统计 规律被任意数目的嵌入词语打乱, 语言学知识的结构依赖性将使得统计模型赖以实 现的独立性假设失效 [ ] 统计模型; 有限状态语法; 马尔科夫过程; 递归性; 结构依赖性 [ ] H08[ ]A [ ] 1003-5397( 2004) 02-0099-10 The Limitations of the Statistically-based NLP Models Yuan Yulin Abstract: This pape demonst ates the limitat ions of the statist ically-based natu al lan- guage p ocessing (NLP) models in the pe spective of linguist ic theo y by int oducing and com- menting the mechanism of the statist ical language models ( SLM) and thei applying cases . Fi stly, it int oduces the studies of the statist ical st uctu e of language unde the influence of info mation theo y, especially Chomsky. s demonst ation that finite state g amma ( FSG) based on Ma kov p ocess is not suited to desc iption of natu al language. Then, it eveals mechanism and possible applying fields of SLM by discussing N-g amm and its applying in pa ts-o-f speech tagging. It discusses the ecu sion p ope ty of linguistic st uctu e and the st uctu e-dependent p ope ty of linguist ic knowledge, and a gues that ecu sive nested const uctions would upset the statistic egula ity and the st uctu e-dependent p ope ty of linguistic knowledge would make the [ ] 2003- 06- 20 [ ] , , , * / 0 , / 0 ( 01JB740006) , 2004 2 # 100 # independence assumption, whe eby SLM can be ealized, l