URL模式与HTML结构相结合的平行网页获取方法AParallelPages.PDFVIP

URL 模式与HTML 结构相结合的平行网页获取方法 刘奇，刘洋，孙茂松 （清华大学计算机科学与技术系 智能技术与系统国家重点实验室，北京 100084） 摘要： 平行语料库是对机器翻译、跨语言信息检索等应用技术具有重要支撑作用的基础数据资源。虽然互 联网上的平行网页数量巨大且持续增长，但由于平行网站的异构性和复杂性，如何快速自动获取高质量的 平行网页进而构造平行语料库仍然是巨大的挑战。本文提出了一种URL 模式与HTML 结构相结合的平行网页 获取方法，首先利用HTML 结构实现平行网页的递归访问，其次使用URL 模式优化遍历平行网站的拓扑顺序， 1 从而实现高效准确的平行网页获取。在联合国与香港政府两个平行网站上的实验表明，我们的方法相对传 统获取方法在获取时间上减少50%以上，准确率提高 15%，并显著提高了机器翻译的质量（BLEU 值分别提 高1.6 和0.7 个百分点）。 关键词：平行网页获取；平行语料库；URL 模式；HTML 结构 A Parallel Pages Mining Approach : Combining URL Patterns and HTML Structures Qi Liu, Yang Liu and Maosong Sun (Department of Computer Science and Technology, State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing 100084, China) Abstract: Parallel corpus are important supporting basic resource for statistical machine translation, cross-lingual information retrieval and others information processing technologies. Although the amount of parallel data on the web is continually increasing, the heterogeneity and complexity of parallel website make it still a large challenge to build parallel corpus by automatically extracting parallel text. This paper presents a new parallel web pages mining approach, which combines URL patterns and HTML structure together. First, we use HTML structure to recursively visit parallel pages. Then, URL patterns are used to optimize the traverse sequence of parallel web site topology. Thus we realized a both efficient and accurate parallel pages mining system. Compared with traditional approach, experiments on two parallel web sites( 1 and .hk ) show that this approach reduces more than 50% processing time，improves 15% accuracy and significantly improves the translation quality of MT System(BLEU score achieved 1.6 and 0.7 percent point increase，respectively). Key words