13.2 Fixed Parameters
For FF and RBF networks, you can modify the network structure by setting some of the parameters to predefined values. Then you can exclude the predefined parameters from the training with the FixedParameters option. These parameters will maintain their defined values during the training; thus, fewer parameters are adapted to the data. This might be good from a biasvariance tradeoff perspective, as described in Section 7.5, Regularization and Stopped Search.
In some problems, you might know that the dependence is linear with respect to some inputs but nonlinear with respect to other inputs. The following example demonstrates how this characteristic can be built into the neural network model so that no more parameters than necessary have to be estimated. A more advanced example can be found in Section 8.2.4, Fix Some Parameters—More Advanced Model Structures.
The default of the FixedParameters option is None, which means that all parameters are trained. You can set FixedParameters to a list of integers specifying the parameters to hold fixed during the initialization of the network. You can also fix parameters when training with NeuralFit. A specification of FixedParameters at the training stage overrides any earlier specification at initialization. The information about fixed parameters is stored in the network model. If you want to train all parameters at a later stage, you must give the option FixedParameters None.
The fixed parameters are indicated by their position in the flattened parameter structure of the network—that is, the position in the list Flatten[net[[1]]], where net is a FF or RBF network model.
Suppose you know, through physical insight or just by assumption, that the unknown function looks something like the following, where f is an unknown nonlinear function and a is an unknown constant. This problem has two inputs and one output.
Therefore, you know that the function is linear in the second input . If the model is specified to be linear in from the beginning, then the training should produce a better model.
Load the Neural Networks package.
In[1]:=
The next two steps produce data from a function of the form shown in Eq. (13.3).
Obtain a "true" function.
In[2]:=
Out[2]=
Generate data and plot the output.
In[3]:=
From the plot, it is evident that the true function is linear in one of the inputs.
Now, only the data and the knowledge that the true function is linear in the second input will be used to develop an FF network model. A linear part must be included in the model. This is done with the option LinearPart.
Initialize an FF network including a linear part.
In[7]:=
Out[7]=
The nonlinear dependence of the second input has to be removed. This is accomplished by setting its components in the w1 matrix to zero. The position of the w1 matrix in the network model is evident from the previous output. Also consult the figures and the description of the w1 matrix in Section 2.5.1, Feedforward Neural Networks.
View the matrix w1.
In[8]:=
Out[8]=
The first row is the dependence on the first input , the second row is the dependence on the second input , and the third row contains the bias parameters. It is the second row that has to be set to zero. This can be done in two steps. First, the position of the row is identified.
Find the position of the second row of w1.
In[9]:=
Out[9]=
Next, the elements of the second row are set to zero.
Set the second row of w1 to zero.
In[10]:=
Out[10]=
Whenever you manipulate the parameters directly, it is wise to check that the structure of the parameter part of the network has been changed correctly.
Check that the manipulation is correct.
In[11]:=
Out[11]=
Only the second row of w1, now set to 0, has changed. For the parameters to remain 0, they have to be excluded from the training. This is done by using the option FixedParameters.
To find the parameter numbers to be held fixed, search for the 0 in the parameter structure.
Find the parameters to be fixed and assign them to a variable.
In[12]:=
Out[12]=
Now the initialized FF network is ready to be trained. The two parameters to be held fixed are indicated in the FixedParameters option.
Train the network with some fixed parameters.
In[13]:=
Check that the two parameters have not changed.
In[14]:=
Out[14]=
The second row of w1 is still {0, 0}. The fact that some parameters were fixed in the training is now specified as a rule in the second argument of the network object.
Fixed parameters are indicated as a rule in the network.
In[15]:=
Out[15]=
If you submit the network fdfrwrd2 to NeuralFit again for more training iterations, then the fixed parameters do not have to be specified since they are already indicated in the network. If you want to change the fixed parameters, simply indicate the new set of fixed parameters in the call to NeuralFit. This will override any specification in the initial model.
Complete this example with a plot of the estimated function. If you repeat the training without forcing the network to be linear in one direction, then the plot of the estimated function will probably not match the defined function as well as the following plot.
In[16]:=
