人工智能中的深度学习——从机器感知到机器认知.ppt

6 0 12 24 18 BENCHMARK: WMT’14 EN-DE 30 June 2015 OOV Replacement (Jean et al., 2015; Luong et al., 2015) Attention-based NMT (Bahdanau et al., 2015) Large Target Vocabulary (Jean et al., 2015; Luong et al., 2015) Location+Content, Local+Global Attention (Luong et al., 2015a) Phrase-based MT (Buck et al., 2014) Dec 2014 (modified from: Kyunghyun Cho) ? SP-P14.5: “SCALABLE TRAINING OF DEEP LEARNING MACHINES BY INCREMENTAL BLOCK TRAINING WITH INTRA-BLOCK PARALLEL OPTIMIZATION AND BLOCKWISE MODEL-UPDATE FILTERING,” by Kai Chen and Qiang Huo (Slide credit: Xuedong Huang) *Google updated that TensorFlow can now scale to support multiple machines recently; comparisons have not been made yet ? Recent Research at MS (ICASSP-2016): -“SCALABLE TRAINING OF DEEP LEARNING MACHINES BY INCREMENTAL BLOCK TRAINING WITH INTRA- BLOCK PARALLEL OPTIMIZATIONAND BLOCKWISE MODEL-UPDATE FILTERING” -“HIGHWAY LSTM RNNs FOR DISTANCE SPEECH RECOGNITION” -”SELF-STABILIZED DEEP NEURAL NETWORKS” CNTK/Phily 23 Deep Learning also Shattered Image Recognition (since 2012) 24 Microsoft Research 3.567% 3.581% Super-deep: 152 layers 4th year 25 Microsoft Research 11x11conv,96,/4,pool/2 5x5conv,256,pool/2 3x3conv,384 3x3conv,384 3x3conv,256,pool/2 fc,4096 fc,4096 fc,1000 AlexNet, 8 layers (ILSVRC 2012) 3x3 conv, 64 3x3 conv, 64, pool/2 3x3 conv, 128 3x3 conv, 128, pool/2 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256, pool/2 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512, pool/2 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512, pool/2 fc, 4096 fc, 4096 fc, 1000 VGG, 19 layers (ILSVRC 2014) LocalRespNorm Conv 3x3+ 1(S) Conv 1x1+ 1(V) LocalRespNorm MaxPool 3x3+ 2(S) Conv 7x7+ 2(S) input MaxPool 3x3+ 2(S) Conv 1x1+ 1(S) Conv 3x3+ 1(S) Conv 1x1+ 1(S) Conv 5x5+ 1(S) Conv 1x1+ 1(S) Conv 1x1+ 1(S) MaxPool 3x3+ 1(S) DepthConcat Conv Conv Conv Conv 1x1+ 1(S) 3x3+ 1(S) Conv 1x1+ 1(S) 5x5+ 1(S) Conv 1x1+ 1(S) 1x1+ 1(S) MaxPool 3x3+ 1