《Autoencoders》.pdf

Extracting and Composing Robust Features with Denoising Autoencoders Pascal Vincent vincentp@iro.umontreal.ca Hugo Larochelle larocheh@iro.umontreal.ca Yoshua Bengio bengioy@iro.umontreal.ca Pierre-Antoine Manzagol manzagop@iro.umontreal.ca Universit´e de Montr´eal, Dept. IRO, CP 6128, Succ. Centre-Ville, Montral, Qubec, H3C 3J7, Canada Abstract to ponder the difficult problem of inference in deep directed graphical models, due to “explaining away”. Previous work has shown that the difficul- Also looking back at the history of multi-layer neural ties in learning deep generative or discrim- networks, their difficult optimization (Bengio et al., inative models can be overcome by an ini- 2007; Bengio, 2007) has long prevented reaping the ex- tial unsupervised learning step that maps in- pected benefits of going beyond one or two hidden lay- puts to useful intermediate representations. ers. However this situation has recently changed with We introduce and motivate a new training the successful approach of (Hinton et al., 2006; Hinton principle for unsupervised learning of a rep- Salakhutdinov, 2006; Bengio et al., 2007; Ranzato resentation based on the idea of making the et al., 2007; Lee et al., 2008) for training Deep Belief learned representations robust to partial cor- Networks and stacked autoencoders.