常见的深度学习的随机矩阵理论模型_v0.1.ppt

深度学习的随机矩阵理论模型 邱才明 10/30/2018 深度学习理论- A Review 神经网络将许多单一的神经元连接在一起 一个神经元的输出作为另一个神经元的输入 多层神经网络模型可以理解为多个非线性函数“嵌套” 多层神经网络层数可以无限叠加 具有无限建模能力, 可以拟合任意函数 多层神经网络 前向传播 Sigmoid Tanh Rectified linear units (ReLU) 常用激活函数 层数逐年增加 误差逐年下降 层数逐年增加 今天1000 layers Features are learned rather than hand-crafted More layers capture more invariances More data to train deeper networks More computing (GPUs) Better regularization: Dropout New nonlinearities Max pooling, Rectified linear units (ReLU) Theoretical understanding of deep networks remains shallow 为什么深度学习性能如此之好? [1] Razavian, Azizpour, Sullivan, Carlsson, CNN Features off-the-shelf: an Astounding Baseline for Recognition. CVPRW’14 [2] Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958. [3] Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456. Experimental Neuroscience uncovered: Neural architecture of Retina/LGN/V1/V2/V3/ etc Existence of neurons with weights and activation functions (simple cells) Pooling neurons (complex cells) All these features are somehow present in Deep Learning systems 神经科学带来的启示 Neuroscience Deep Network Simple cells First layer Complex cells Pooling Layer Grandmother cells Last layer Olshausen and Field demonstrated that receptive fields learned from image patches. Olshausen and Field showed that optimization process can drive learning image representations. Olshausen and Field’s Work (Nature, 1996) Olshausen-Field representations bear strong resemblance to defined mathematical objects from harmonic analysis wavelets, ridgelets, curvelets. Harmonic analysis: long history of developing optimal representations via optimization Research in 1990s: Wavelets etc are optimal sparsifying transforms for certain classes of ima