2.7 Hopfield Network
In the beginning of the 1980s Hopfield published two scientific papers, which attracted much interest. This was the starting point of the new era of neural networks, which continues today.
Hopfield showed that models of physical systems could be used to solve computational problems. Such systems could be implemented in hardware by combining standard components such as capacitors and resistors.
The importance of the different Hopfield networks in practical application is limited due to theoretical limitations of the network structure but, in certain situations, they may form interesting models. Hopfield networks are typically used for classification problems with binary pattern vectors.
The Hopfield network is created by supplying input data vectors, or pattern vectors, corresponding to the different classes. These patterns are called class patterns. In an ndimensional data space the class patterns should have n binary components {1,1}; that is, each class pattern corresponds to a corner of a cube in an ndimensional space. The network is then used to classify distorted patterns into these classes. When a distorted pattern is presented to the network, then it is associated with another pattern. If the network works properly, this associated pattern is one of the class patterns. In some cases (when the different class patterns are correlated), spurious minima can also appear. This means that some patterns are associated with patterns that are not among the pattern vectors.
Hopfield networks are sometimes called associative networks since they associate a class pattern to each input pattern.
The Neural Networks package supports two types of Hopfield networks, a continuoustime version and a discretetime version. Both network types have a matrix of weights W defined as
where D is the number of class patterns {, ..., }, vectors consisting of +/1 elements, to be stored in the network, and n is the number of components, the dimension, of the class pattern vectors.
Discretetime Hopfield networks have the following dynamics:
Eq. (2.26) is applied to one state, x(t), at a time. At each iteration the state to be updated is chosen randomly. This asynchronous update process is necessary for the network to converge, which means that x(t)=Sign[W x(t)].
A distorted pattern, x(0), is used as initial state for the Eq. (2.0), and the associated pattern is the state toward which the difference equation converges. That is, starting with x(0) and then iterating Eq. (2.0) gives the associated pattern when the equation converged.
For a discretetime Hopfield network, the energy of a certain vector x is given by
It can be shown that, given an initial state vector x(0), x(t) in Eq. (2.26) will converge to a value having minimum energy. Therefore, the minima of Eq. (2.27) constitute possible convergence points of the Hopfield network and, ideally, these minima are identical to the class patterns {, ..., }. Hence, one can guarantee that the Hopfield network will converge to some pattern, but one cannot guarantee that it will converge to the right pattern.
Note that the energy function can take negative values; this is, however, just a matter of scaling. Adding a sufficiently large constant to the energy expression it can be made positive.
The continuous Hopfield network is described by the following differential equation
where x(t) is the state vector of the network, W represents the parametric weights, and is a nonlinearity acting on the states x(t). The weights W are defined in Eq. (2.25). The differential equation, Eq. (2.28), is solved using an Euler simulation.
To define a continuoustime Hopfield network, you have to choose the nonlinear function . There are two choices supported by the package, SaturatedLinear and the default nonlinearity of Tanh.
For a continuoustime Hopfield network, defined by the parameters given in Eq. (2.25), one can define the energy of a particular state vector x as
As for the discretetime network, it can be shown that given an initial state vector x(0) the state vector x(t) in Eq. (2.28) converges to a local energy minimum. Hence, the minima of Eq. (2.29) constitute the possible convergence points of the Hopfield network and ideally these minima are identical to the class patterns {, ..., }. However, there is no guarantee that the minima will coincide with this set of class patterns.
Examples with Hopfield nets can be found in Section 9.2.
