Distributed Representations of Words and Phrases and their….pdfVIP

Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov Ilya Sutskever Kai Chen Google Inc. Google Inc. Google Inc. Mountain View Mountain View Mountain View mikolov@ ilyasu@ kai@ Greg Corrado Jeffrey Dean Google Inc. Google Inc. Mountain View Mountain View gcorrado@ jeff@ Abstract The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large num- ber of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alterna- tive to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of “Canada” and “Air” cannot be easily combined to obtain “Air Canada”. Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. 1 Introduction Distributed representations of words in a vector space help le