一种构建信息检索系统评测集的方法.doc

本科生学位论文 题目：一种构建信息检索系统评测集的方法 姓 名：戴 梦 学 号院 系：信息科学技术学院 专 业：计算机科学 导 师：闫宏飞 教授 二零零九年五月 摘要 随着互联网的飞速发展，信息检索技术在实际生活中的作用越来越重要，在学术界也引起了研究学者的重视。参考国外测试集的构建经验，天网实验室构建了大规模中文网页信息检索测试集CWT，并组织了SEWM中文网页检索评测，希望在国内外各个研究小组的共同参与下建立并完善CWT，一起推动中文网页信息检索技术的发展。 测试集是信息检索评测的一项决定性因素之一。测试集包括查询主题集、文档集和相关文档集。查询主题集可以从网络日志中获取抑或由人工设定；文档集是信息检索系统返回的检索结果的集合；而相关文档集是对应相应的查询主题相关的文档的集合，一般相关文档需要人工进行判断，由于文档集的规模都不小，而相关文档集是文档集的子集，建造其工作量巨大，甚至有可能难以实施。所以我们想尝试得出判断两个信息检索系统孰优孰劣最小的工作量，及所需的最小查询主题集和相关文档集。 本论文通过变形AP(Average Percision)公式的表达方式，研究其与相关文档集的关系，说明了在给定查询主题的情况下，如何找到最小的相关文档集，并作实验加以证明。结合查询主题和相关文档集合，利用MAP(Mean Average Precision)使得两个集合共同最小化。 关键词：信息检索，评测，测试集，文档集，查询主题，相关判断，最小相关文档集，算法 A Research on Constructing Information Retrieval Test Collection DAI Meng (Computer Science Dept.) Instructed by Prof. YAN Hongfei Abstract With the rapid development of World Wide Web, Information Retrieval technique is playing a more and more important role in people’s life. Meanwhile, it has become a crucial research topic across different research entities. Following the existing test collection framework established by foreign research conference (especially TREC Test REtrieval Conference), TianWang research team has constructed a large-scale Chinese Web Test collection (CWT), and is organizing SEWM Chinese Web search evaluation on a yearly basis. Test collections is of great importance in the study of Information Retrieval, it encompass corpora of documents, sets of topics and relevance judgment indicating which documents are relevant to which topics. Topics could be retrieved from current web user log, or established by annotators experienced in certain field of study. And accurate estimation of information retrieval evaluation metrics such as Average Precision require large sets of relevance judgment. Building sets large enough for evaluation of real world implementation is at best inefficient, at worst infeasible. In the work, we tried to come up with an algorithm that requires minimal human effort in g