LossFunction

LossFunction

is an option for NetTrain that specifies how to compare actual and requested outputs from a neural net.

Details

  • In NetTrain[net,data,], possible settings for LossFunction include:
  • Automaticautomatically attach loss layers to all outputs, or use existing loss outputs (default)
    "port"interpret the given port as a loss
    losslayerattach a preconfigured loss layer to the net's output
    "port"losslayercompute loss from one output of a multi-output net
    {lspec1,lspec2,}minimize several losses simultaneously
    {,lpeciScaled[r],}scale individual losses by a factor r
  • Layers that can be used as preconfigured loss layers include:
  • MeanAbsoluteLossLayer[]mean of distance between output and target
    MeanSquaredLossLayer[]mean of squared distance between output and target
    CrossEntropyLossLayer[form]distance between output class probabilities and target class
    ContrastiveLossLayer[]how well output is maximized or minimized conditioned on target
    NetGraph[], NetChain[], etc.any network with an "Input" and optional "Target" port
  • When specifying a single preconfigured loss layer with LossFunction->losslayer, the net being trained must have exactly one output port.
  • When specifying a loss for a specific port with LossFunction->"port"lspec, the loss specification lspec can be a preconfigured loss layer or Automatic.
  • When a loss layer is chosen automatically for a port, the loss layer to use is based on the layer within the net whose output is connected to the port, as follows:
  • SoftmaxLayer[]use CrossEntropyLossLayer["Index"]
    ElementwiseLayer[LogisticSigmoid]use CrossEntropyLossLayer["Binary"]
    NetPairEmbeddingOperator[]use ContrastiveLossLayer[]
    other non-loss layersuse MeanSquaredLossLayer[]
    loss layers use unchanged
  • If the attached loss layer has one input port ("Input"), it will be attached to the output of the net, and the keys of the training data should supply ports to feed the input of the net only (first figure).
  • If the attached loss layer has two input ports ("Input" and "Target"), the input will be attached to the output of the net and the target will be fed from the training data, using the name of the output port of the net (second figure). Typically, this name is "Output", and so the usual case is training data of the form <|"Input"->{in1,in2,},"Output"->{out1,out2,}|>, which is often written as {in1,in2,}->{out1,out2,} or {in1->out1,in2->out2,}.
  • If multiple such layers are connected, there should be one port in the training data to feed the target of each layer (third figure).

Examples

Basic Examples  (4)

Train a simple net using MeanSquaredLossLayer, the default loss applied to an output when it is not produced by a SoftmaxLayer:

In[1]:=
Click for copyable input
Out[1]=

Evaluate the trained net on a set of inputs:

In[2]:=
Click for copyable input
Out[2]=

Specify a different loss layer to attach to the output of the network. First create a loss layer:

In[3]:=
Click for copyable input
Out[3]=

The loss layer takes an input and a target and produces a loss:

In[4]:=
Click for copyable input
Out[4]=

Use this loss layer during training:

In[5]:=
Click for copyable input
Out[5]=

Evaluate the trained net on a set of inputs:

In[6]:=
Click for copyable input
Out[6]=

Create a net that takes a vector of length 2 and produces one of the class labels Less or Greater:

In[1]:=
Click for copyable input
Out[1]=

NetTrain will automatically use a CrossEntropyLossLayer object with the correct class encoder:

In[2]:=
Click for copyable input
Out[2]=

Evaluate the trained net on a set of inputs:

In[3]:=
Click for copyable input
Out[3]=

Create an explicit loss layer that expects the targets to be in the form of probabilities for each class:

In[4]:=
Click for copyable input
Out[4]=

The training data should now consist of vectors of probabilities instead of symbolic classes:

In[5]:=
Click for copyable input
Out[5]=

Evaluate the trained net on a set of inputs:

In[6]:=
Click for copyable input
Out[6]=

Start with an "evaluation net" to be trained:

In[1]:=
Click for copyable input
Out[1]=

Create a "loss net" that explicitly computes the loss of the evaluation net (here, the custom loss is equivalent to MeanSquaredLossLayer):

In[2]:=
Click for copyable input
Out[2]=

Train this net on some synthetic data:

In[3]:=
Click for copyable input
Out[3]=

Obtain the trained "evaluation" network using NetExtract:

In[4]:=
Click for copyable input
Out[4]=

Part syntax can also be used:

In[5]:=
Click for copyable input
Out[5]=

Plot the net's output on the plane:

In[6]:=
Click for copyable input
Out[6]=

Create a training net that computes both an output and multiple explicit losses:

In[1]:=
Click for copyable input
Out[1]=

This network requires an input and a target in order to produce the output and losses:

In[2]:=
Click for copyable input
Out[2]=

Train using a specific output as loss, ignoring all other outputs:

In[3]:=
Click for copyable input

Measure the loss on a single pair after training:

In[4]:=
Click for copyable input
Out[4]=

Specify multiple losses to be trained jointly:

In[5]:=
Click for copyable input
In[6]:=
Click for copyable input
Out[6]=

Use NetTake to remove all but the desired output and input:

In[7]:=
Click for copyable input
Out[7]=

Evaluate this network on a set of inputs:

In[8]:=
Click for copyable input
Out[8]=

See Also

NetTrain  MeanAbsoluteLossLayer  MeanSquaredLossLayer  CrossEntropyLossLayer  CTCLossLayer  ContrastiveLossLayer  NMinimize

Tutorials

Introduced in 2018
(11.3)