systematic characterizations of text similarity in full text biomedical publications系统特征的文本相似度在生物医学全文出版物.pdfVIP

Systematic Characterizations of Text Similarity in Full Text Biomedical Publications 1 2 1 2 2 1 Zhaohui Sun , Mounir Errami , Tara Long , Chris Renard , Nishant Choradia , Harold Garner * 1Virginia Bioinformatics Institute, Blacksburg, Virginia, United States of America, 2 Department of Math and Natural Sciences, Collin College, Plano, Texas, United States of America Abstract Background: Computational methods have been used to find duplicate biomedical publications in MEDLINE. Full text articles are becoming increasingly available, yet the similarities among them have not been systematically studied. Here, we quantitatively investigated the full text similarity of biomedical publications in PubMed Central. Methodology/Principal Findings: 72,011 full text articles from PubMed Central (PMC) were parsed to generate three different datasets: full texts, sections, and paragraphs. Text similarity comparisons were performed on these datasets using the text similarity algorithm eTBLAST. We measured the frequency of similar text pairs and compared it among different datasets. We found that high abstract similarity can be used to predict high full text similarity with a specificity of 20.1% (95% CI [17.3%, 23.1%]) and sensitivity of 99.999%. Abstract similarity and full text similarity have a moderate correlation (Pearson correlation coefficient: 20.423) when the similarity ratio is above 0.4. Among pairs of articles in PMC, method sections are found to be the most repetitive (frequency of similar pairs, methods: 0.029, introduction: 0.0076, results: 0.0043). In contrast, among a set of manually verified duplicate articles, results are the most repetitive sections (frequency of s