Tom′asˇMikolov1,2,MartinKara′.PDF

INTERSPEECH 2010 Recurrent neural network based language model ˇ ˇ ˇ Tomas Mikolov , Martin Karafiat , Lukas Burget , Jan “Honza” Cernocky , Sanjeev Khudanpur ´ ´ ´ ´ Speech@FIT, Brno University of Technology, Czech Republic Department of Electrical and Computer Engineering, Johns Hopkins University, USA imikolov,karafiat,burget,cernocky@fit.vutbr.cz, khudanpur@jhu.edu Abstract INPUT(t) OUTPUT(t) A new recurrent neural network based language model (RNN CONTEXT(t) LM) with applications to speech recognition is presented. Re- sults indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empiri- cal evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high com- putational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition