Annealeddropouttrainingofdeepnetworks-Centerfor.PDFVIP

ANNEALED DROPOUT TRAINING OF DEEP NETWORKS Steven J. Rennie, Vaibhava Goel, and Samuel Thomas IBM Thomas J. Watson Research Center NY, USA sjrennie, vgoel, sthomas@ ABSTRACT Conventional dropout improves test-time performance when there is limited data relative to the size of the model being trained. Dropout Recently it has been shown that when training neural networks on a training allows one to gain performance by avoiding overfitting, so limited amount of data, randomly zeroing, or ”dropping out” a fixed that a larger model than would otherwise maximize test performance percentage of the outputs of a given layer for each training case can can be utilized. But what about the more usual scenario where the improve test set performance significantly. Dropout training discour- size of the model and training time, rather than the amount of train- ages the detectors in the network from co-adapting, which limits the ing data, are the dominant constraints? This question has not been capacity of the network and prevents overfitting. In this paper we previously studied. In this paper, we show that annealing the drop