2806 Neural Computation Temporal Processing Lecture 11.pptVIP

2806 Neural ComputationTemporal Processing Lecture 11 2005 Ari Visa Agenda Some historical notes Some theory Some network architecture Conclusions Some Historical Notes z-transform A single neuron - an adaptive filter Widrow Hoff (1960): least-mean square algorithm (LMS) = delta rule The need for temporal processing arises in numerous applications [prediction modeling (Box Jenkins, 1976), noise cancellation (Widrow and Stearns, 1985), adaptive equalization (Proakis, 1989), adaptive control (Narendra Annaswamy, 1989), system identification (Ljung, 1987)] Some Theory How do we build time into the operation of a neural network? Implicit representation The temporal structure of the input signal is embedded in the spatial structure of the network. Explicit representation Time is given its own particular representation. For a neural network to be dynamic, it must be given memory. Memory may be divided into short-term and long-term. Long-term memory = neural network Short-term memory = time delays Some Theory Short-time memory can be implemented in continuous time or in discrete time. z-transform, z-1 is the unit delay operator. We may define a discrete-time memory as a linear time invariant, single input-multiple output system (causal, normalized). The junction points, to which the output terminals of the memory are connexted, are commonly called taps. Some Theory Memory depth is defined as the first time moment of gp(n) (generating kernel). Memory resolution is defined as the number of taps in the memory structure per unit time. Most commonly used form of short-time memory is called a tapped delay line memory. It consists of p unit delay operators. The generating kernel is g(n) = ?(n-1), where ?(n) is the unit impulse. Some Theory Gamma memory, each section of this memory structure consists of a feedback loop with unit delay and adjustable parameter ?. g(n) = ?(1-?)n-1, 0 ?2, n≥1 gp(n) represents a discrete version of the integrand of the gamma func