自注意力机制.pptxVIP

Self-attentionHung-yi Lee 李宏毅Sophisticated Input ModelScalar or ClassModelScalars or Classes(may change length)Input is a vector Input is a set of vectorsVector Set as Input this is a cat One-hot EncodingWord Embeddingapple = [ 1 0 0 0 0 …… ]dograbbitrunbag = [ 0 1 0 0 0 …… ]jumpcatcat = [ 0 0 1 0 0 …… ]treeflowerdog = [ 0 0 0 1 0 …… ]elephant = [ 0 0 0 0 1 …… ]To learn more: https://youtu.be/X7PH3NuYW0Qhttps://youtu.be/X7PH3NuYW0Q (in Mandarin)Vector Set as Input 10ms1s 100 frames?25ms400 sample points (16KHz)39-dim MFCC80-dim filter bank outputframe/analytics-vidhya/social-network-analytics-f082f4e21b16Vector Set as Input Each profile is a vectorGraph is also a set of vectors (consider each node as a vector)/wiki/%E5%88%86%E5%AD%90Vector Set as Input Graph is also a set of vectors (consider each node as a vector)H = [ 1 0 0 0 0 …… ]C = [ 0 1 0 0 0 …… ]O = [ 0 0 1 0 0 …… ]……One-hot vectorWhat is the output?Each vector has a label. ModelNNExample ApplicationsI saw a sawnotDETNVNbaabbuybuyPOS taggingHW2What is the output?Each vector has a label. ModelNNThe whole sequence has a label. ModelExample Applicationsthis is goodSentiment analysisHW4speakerpositivehydrophilicityWhat is the output?focus of this lectureEach vector has a label. ModelNNThe whole sequence has a label. Modelseq2seqModel decides the number of labels itself.ModelTranslation (HW5)N’NSequence Labeling Is it possible to consider the context?Fully-connectedFCFC can consider the neighbor How to consider the whole sequence? a window covers the whole sequence?FCFCFCFCwindowIsawsawaSelf-attentionFCFCFCFCwith contextSelf-attentionFCFCFCFCSelf-attentionFCFCFCFCSelf-attentionAttention is all you need./abs/1706.03762Self-attention????????Can be either input or a hidden layerSelf-attention??relevant?????Find the relevant vectors in a sequence ?Self-attention?Additive?Dot-product?????????Self-attention??????attention scoreke