NetInitialize

NetInitialize[net]

gives a net in which all uninitialized learnable parameters in net have been given initial values.

NetInitialize[net,All]

gives a net in which all learnable parameters have been given initial values.

Details and Options

  • NetInitialize[net,All] overwrites any existing training or preset learnable parameters in net.
  • NetInitialize typically assigns random values to parameters representing weights and zero to parameters representing biases.
  • The following optional parameters can be included:
  • MethodAutomaticwhich initialization method to use
  • Possible settings for Method include:
  • "Xavier"choose weights to preserve variance of random tensors when propagated through affine layers
    "Orthogonal"choose weights to be orthogonal matrices
    "Random"choose weights from a given univariate distribution
    "Identity"choose weights so as to preserve components of tensors when propogated through affine layers
  • Suboptions for specific methods can be specified using Method{"method",opt1val1,}.
  • For the method "Xavier", the following suboptions are supported:
  • "FactorType""Mean"one of "In", "Out", or "Mean"
    "Distribution""Normal"either "Normal" or "Uniform"
  • For the method "Random", the following suboptions are supported:
  • "Weights"NormalDistribution[0,1]random distribution to use to initialize weight matrices
    "Biases"Nonerandom distribution to use to initialize bias vectors
  • For the method "Identity", the following suboption is supported:
  • "Distribution"NormalDistribution[0,0.01]random distribution used to add noise to the initial identity matrices in order to break symmetries
  • For any suboption that expects a distribution, a numeric value stddev can be specified and is taken to mean NormalDistribution[0,stddev].
  • By default, all methods initialize bias vectors to zero.

Examples

open allclose all

Basic Examples  (1)

Create an uninitialized layer:

In[1]:=
Click for copyable input

Initialize the layer with random weights:

In[2]:=
Click for copyable input
Out[2]=

Extract the new initialized weights:

In[3]:=
Click for copyable input
Out[3]=

Scope  (1)

Options  (1)

Properties & Relations  (2)

Possible Issues  (1)

Neat Examples  (1)

See Also

NetChain  NetGraph  NetTrain  NetExtract  RandomVariate

Introduced in 2016
(11.0)