TrainingStoppingCriterion

TrainingStoppingCriterion

is an option for NetTrain that specifies a criterion for stopping training early in order to prevent overfitting.

Details

  • With the default of TrainingStoppingCriterionNone, no early stopping is performed.
  • Setting TrainingStoppingCriterion"measurement" specifies that training should be stopped if measurement stops improving. Possible values for measurement include:
  • "Loss"network training loss
    keya TrainingProgressMeasurements key
  • The values available for key depend on the specification of TrainingProgressMeasurements. Often, the key is the same as the measurement. For example, with TrainingProgressMeasurements"ErrorRate", key can be "ErrorRate". However, this is not always the case.
  • TrainingStoppingCriterionAutomatic is equivalent to TrainingStoppingCriterion"Loss".
  • By default, TrainingStoppingCriterion is based on the metrics and loss from the validation set. If the ValidationSet option of NetTrain is None, then a warning will be issued and the training set will be used. If the validation set is present, the stopping criterion is checked whenever the validation loss and metrics are calculated (once per round by default); otherwise, the stopping criterion is checked once per round.
  • TrainingStoppingCriterion has a number of suboptions that can be specified using the <|"Criterion""measurement",opt1val1,opt2val2,|> syntax.
  • Setting TrainingStoppingCriterion<|"Criterion""measurement","Patience"n|> specifies that training should be stopped if an improvement in measurement is not seen for n rounds in a row. The default value for n is 0.
  • Setting TrainingStoppingCriterion<|"Criterion""measurement","InitialPatience"n |> specifies that the stopping criterion is only checked for the first time after n rounds. The default value for n is 0.
  • Setting TrainingStoppingCriterion<|"Criterion""measurement","improvement"v|> specifies the minimum change in measurement that is considered an improvement. Possible values for improvement are:
  • "AbsoluteChange"stop training if measurement improves by less than v
    "RelativeChange"stop training if measurement improves by less than a factor v of the current best value
  • If an improvement specification is not given, then "AbsoluteChange"0 is used.
  • The form "RelativeChange"Quantity[q,"Percent"] can also be given.
  • It is also possible to specify a function as the stopping criterion using TrainingStoppingCriterion<|"Criterion"func,|>, where func should return True if training should be stopped.
  • The function func is provided an association with the following keys:
  • RoundLosstotal loss for the training set
    ValidationLosstotal loss for the validation set
    RoundMeasurementsassociation of requested measurements for the training set
    ValidationMeasurementsassociation of requested measurements for the validation set
  • The validation properties are only available if there is a validation set.
  • The "RelativeChange" and "AbsoluteChange" options cannot be used with a function criterion.
  • If a measurement is specified for TrainingStoppingCriterion, the value of the measurement will be used to select the optimal trained net to return from NetTrain. Note that the choices of "Patience", "InitialPatience", "AbsoluteChange" and "RelativeChange" have no effect on this selection. If a function is specified for TrainingStoppingCriterion, the default behavior will be used when selecting the optimal trained net.

Examples

open all close all

Basic Examples  (1)

Prevent overfitting by stopping training when the validation loss stops improving. Set up a simple net as well as some training and validation data:

In[1]:=
Click for copyable input

Use TrainingStoppingCriterion to stop training when the validation loss stops improving:

In[2]:=
Click for copyable input
Out[2]=

Compare this to the case without early stopping:

In[3]:=
Click for copyable input
Out[3]=

Scope  (2)

Properties & Relations  (2)

Introduced in 2019
(12.0)