Vector quantization - MIT - Massachusetts Institute of Technology矢量量化的麻省理工学院-麻省理工学院.pptVIP

Clustering Hypothesis: Hebbian synaptic plasticity enables a perceptron to compute the mean of its preferred stimuli. Unsupervised learning Sequence of data vectors Learn something about their structure Multivariate statistics Neural network algorithms Brain models Data can be summarized by a few prototypes. Vector quantization Many telecom applications Codebook of prototypes Send index of prototype rather than whole vector Lossy encoding A single prototype Summarize all data with the sample mean. Multiple prototypes Each prototype is the mean of a subset of the data. Divide data into k clusters. One prototype for each cluster. Assignment matrix Data structure for cluster memberships. k-means algorithm Alternate between computing means and computing assignments. Objective function Why does it work? Method of minimizing an objective function. Rubber band computer Attach rubber band from each data vector to the prototype vector. The prototype will converge to the sample mean. The sample mean maximizes likelihood Gaussian distribution Maximize Objective function for k-means Local minima can exist Model selection How to choose the number of clusters? Tradeoff between model complexity and objective function. Neural implementation A single perceptron can learn the mean in its weight vector. Many competing perceptrons can learn prototypes for clustering data. Batch vs. online learning Batch Store all data vectors in memory explicitly. Online Data vectors appear sequentially. Use one, then discard it. Only memory is in learned parameters. Learning rule 1 Learning rule 2 “weight decay” Learning rule 2 again Is there an objective function? Stochastic gradient following The average of the update is in the direction of the gradient. Stochastic gradient descent Convergence conditions Learning rate vanishes slowly but not too slowly Every limit point of the sequence wt is a stationary point of E(w) Competitive learning Online version of k-means Competition with WTA If the wa