《一天理解深度学习》.ppt

推文接龍 * 中翻英 英翻中 沒有哪一個一定比較長或比較短 More applications * A woman is throwing a Frisbee in a part. * Different techniques are used to deal with different problems. 不同方法間的互相搭配!!!!!!!!!!!!!!!! * * * * * * CNN is widely used Very good example for designing your network * /docs/LDC93S4B/corpus.html * Can be very large * * * General idea * Output yi depends on x1, x2, …… xi The same input can have totoally different output. * To determine yi, you have to consider a lot …… You should see the whole sequence * Normal neuron 1 input, 1 output This one 4 input, 1 output * * Identiy C is vector * Identiy C is vector * Identiy C is vector * Usually when people say RNN, they mean LSTM! * * General idea * * Not because you have bug haha * Even adagrad can not handle this problem, maybe RMS prop is better Source: /proceedings/papers/v28/pascanu13.pdf Large or small is fine. Change quickly is bad * Gradient Explode/Gradient Vanishing Echo state network - ….. * * * * English is successful, ASRU’15 better than HMM + N-gram * 中翻英 英翻中 沒有哪一個一定比較長或比較短 * 中翻英 英翻中 沒有哪一個一定比較長或比較短 * * * THEANO_FLAGS=device=gpu0 python YourCode.py import os os.environ[THEANO_FLAGS] = device=cpu * * The three steps can also apply on other machine learning methods * Other methods do not emphasize this. * * * * Hopefully, when the match size is large enough, Not that stateble * * * * Parametric ReLU, - PReLU Ref: Glorot, Xavier, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks.?International Conference on Artificial Intelligence and Statistics. 2011. * * 3,4,5 * Advanced Idea: Eta Can we give each parameters different learning rates? * 反差 /mediawiki/images/6/6a/Adagrad.pdf /courses/cse547/15sp/slides/adagrad.pdf * * * Momentum: 動量 * What can we see? * In speech recognition: add noise warping * * * * ??=∑?〖??_?? ??_?? 〗 * * 0,0 - 0 1,0 - 1 0,-1 - -2 1,-1 - -1 ?, -1/2 - -0.5 1,2 Geometric Mean? * * Rate = 0.5! * * For example, if we modify “1” to