中文文本自动校对系统的研究与实现.pdfVIP

中文文本自动校对系统的研究与实现 龚永罡，付俊英，廉小亲，李玉莹 (北京工商大学计算机与信息工程学院, 北京 100048) 摘要：随着“互联网+”时代的到来和快速发展，新媒体（微博、微信公众号、博 客、论坛、新闻客户端等）已成为人们生活中不可分割的一部分,很多新闻媒体平台，每天 原创新闻发布量巨大，采用人工审核已经不切实际。因此，高速处理海量内容的中文文本 自动校对系统的研究具有重要的实际意义和广泛的应用领域。本文对中文文本的校对内容 进行了分析，将其分为两大类，分别为错别字和敏感信息。本文采用政府网站及今日头条 的文本内容作为训练语料，利用n-gram 模型对分词后的语料进行统计分析形成二元知识模 型库与上下文语境库，再构建错别字混淆集，然后通过计算目标词对应的混淆集中的混淆 词在二元知识模型库与上下文语境库中的支持度，来实现中文文本自动校对系统的自动查 错与纠错。该设计方案目前已成功应用于一些政府网站，现已部署五台服务器，每台服务 器每天能够处理一百万万篇文章，抽取分析结果显示，该系统的召回率为 78.9%，准确率 为 85.6%，具有较好的应用前景。 关键字：n-gram 模型，混淆集， 支持度，自动校对 Research and Implementation of Chinese Text Automatic Proofreading System GONG Yong-Gang, Fu Jun-Ying, LIAN Xiao-Qin, Li Yu-Ying (School of Information Engineering and Computer, Beijing Technology and Business University, Beijing 100048, China) Abstract: With the advent and rapid development of the “Internet+” era, new media (microblog, WeChat public account, blogs, forums, news clients, etc.) have become an integral part of people’s lives. Many news media platforms have huge amounts of original news releases every day, so manual review is not practical. Therefore, the study of the Chinese text automatic proofreading system for high-speed processing of massive contents has important practical significance and a wide range of applications. This article analyzes the proofreading content of Chinese texts and classifies them into two categories, namely typos and sensitive information. The article uses the text of government websites and headlines today as training corpus, uses the n-gram model to perform statistical analysis on the segmented corpora to form a binary knowledge model library and context library, then Build a typo confusion set, and by calculating the support degree of the confused words in the confusion se