一种改进的社交媒体文本规范化方法animprovingmethodforsocial.pdfVIP

一种改进的社交媒体文本规范化方法 1 2 宋亚军 ，于中华 （1. 四川大学，610000 ；2. 四川大学，610000） 摘 要：社交媒体文本书写不规范的特点，使得现有的自然语言处理工具直接应用于社交媒体文本时效果 不甚理想，并且基于关键词的算法和应用也达不到预期的效果。因此，研究如何更好的对社交媒体文本进 行规范化是非常有意义和价值的。本文基于社交媒体文本中非规范词与其规范形式具有相似上下文的假设， 引入词嵌入模型更好地刻画上下文的相似性，提出了一种改进的基于图的社交媒体文本规范化方法，该方 法是无监督并且语言无关的，可以处理不同类型语言的大规模无标注社交媒体文本。实验结果表明，该方 法能够改进前人方法的不足，并且在与相关方法的对比实验中取得了最好的 F 值。 关键词：社交媒体；文本规范化；自然语言处理；词嵌入 中图分类号：TP391 文献标识码：A An improving Method for Social Media Text Normalization 1 2 Song Yajun , Yu Zhonghua (1. Sichuan University, 610000 ; 2. Sichuan University, 610000) Abstract: The informal nature of social media text makes it difficult for many natural language processing tools to process social media text effective, and many keywords-based methods proposed on social media text works not as well as expected. Therefore, Social media text normalization is very meaningful and valuable. Based on the assumption that lexical variants occur in similar contexts to their standard forms, we proposed an improved graph-based social media text normalization method by introducing word embedding model to better capture the context similarity, the proposed method is unsupervised and language independent, and can be used to process large-scale different language social media text. Experimental results show that the proposed method improved the shortcomings of previous methods, and achieves state-of-the-art F-score. Keywords: social media; text normalization; natural language process; word embedding 1 引言