TrainingProgressMeasurements

TrainingProgressMeasurements

is an option for NetTrain that specifies measurements to make while training is in progress.

Details

  • In TrainingProgressMeasurements->spec, the following forms for spec are allowed:
  • "measurement"a named, built-in measurement
    NetPort["output"]the value of an output port of the net
    NetPort["tdata"]the value of training data for the net
    NetPort[{lspec,"output"}]the value of an interior activation of the net
    NetPort[{lspec,"weight"}]the value of a weight array
    <|"Measurement"spec,|>a measurement with suboptions
    <|"Measurement"Function[],|>a custom function to measure
  • Setting TrainingProgressMeasurements{spec1,spec2,} will result in multiple measurements being made.
  • With the default setting of TrainingProgressMeasurementsAutomatic, the "ErrorRate" measurement will be automatically included for nets that are trained using a single CrossEntropyLossLayer.
  • For nets that contain a CrossEntropyLossLayer, the following built-in measurements are available:
  • "Accuracy"fraction of correctly classified examples
    "Accuracy"nfraction of examples with the correct result in the top n
    "AreaUnderROCCurve"area under the ROC curve for each class
    "CohenKappa"Cohen's kappa coefficient
    "ConfusionMatrix"counts cij of class i examples classified as class j
    "ConfusionMatrixPlot"plot of the confusion matrix
    "Entropy"entropy measured in nats
    "ErrorRate"fraction of incorrectly classified examples
    "ErrorRate"nfraction of examples with the incorrect result in the top n
    "F1Score"F1 score for each class
    "FScore"βFβ score for each class
    "FalseDiscoveryRate"false discovery rate for each class
    "FalseNegativeNumber"number of false negative examples
    "FalseNegativeRate"false negative rate for each class
    "FalseOmissionRate"false omission rate for each class
    "FalsePositiveNumber"number of false positive examples
    "FalsePositiveRate"false positive rate for each class
    "Informedness"informedness for each class
    "Markedness"markedness for each class
    "MatthewsCorrelationCoefficient"Matthews correlation coefficient for each class
    "NegativePredictiveValue"negative predictive value for each class
    "Perplexity"exponential of the entropy
    "Precision"precision for each class
    "Recall"recall rate for each class
    "ROCCurve"receiver operating characteristics (ROC) curve for each class
    "ROCCurvePlot"plot of the ROC curve
    "ScottPi"Scott's pi coefficient
    "Specificity"specificity for each class
    "TrueNegativeNumber"number of true negative examples
    "TruePositiveNumber"number of true positive examples
  • For nets that contain a MeanSquaredLossLayer or MeanAbsoluteLossLayer, the following built-in measurements are available:
  • "FractionVarianceUnexplained"the fraction of output variance left unexplained by the net
    "IntersectionOverUnion"intersection over union for bounding boxes
    "MeanDeviation"mean absolute value of the residuals
    "MeanSquare"mean square of the residuals
    "RSquared"coeficient of determination
    "StandardDeviation"root mean square of the residuals
  • The pure function for the custom function measurement will be supplied with the association described in TrainingProgressFunction.
  • Suboptions can be specified for a given measurement using the association form <|"Measurement"measurement,opt1val1,|>.
  • When the source of a measurement is ambiguous, the suboption "Source""outname" can be used to select a specific output port of the net from which to take the measurement.
  • The suboption "ClassAveraging"type can be specified for measurements that are calculated per-class, such as "Precision", "Recall", "F1Score" and "ROCCurve". Possible types include:
  • Nonereturn the per-class measurements (default)
    "Macro"return the macro-average of the per-class measurements
    "Micro"return the micro-average of the per-class measurements
  • The suboption "Direction"direction specifies which direction of change in the measurement is an improvement. Possible values for directions are:
  • Automaticmeasurement direction is chosen to correspond to improved accuracy (default)
    "Decreasing"measurement should decrease
    "Increasing"measurement should increase
  • The "Aggregation"aggregation option can be used to aggregate non-scalar NetPort measurements. Possible aggregations are:
  • Nonereturn the array unchanged (default)
    "L1Norm"return the L1 norm of the flattened array
    "L2Norm"return the L2 norm of the flattened array
    "Max"return the maximum of the flattened array
    "Mean"return the mean of the flattened array
    "Min"return the minimum of the flattened array
    "RootMeanSquare"return the root mean square of the flattened array
    "StandardDeviation"return the standard deviation of the flattened array
    "Total"return the sum of the flattened array
  • The "Interval"->"interval" option specifies how often to collect a measurement. Possible intervals are:
  • Automaticautomatically choose an appropriate interval (default)
    "Batch"collect the measurement every batch
    "Round"collect the measurement every round
  • If a TrainingProgressFunction is supplied to NetTrain, associations of the most recent values of the requested measurements are available using the keys "RoundMeasurements" and "ValidationMeasurements".
  • Associations of the final value of all measurements can be obtained after training by specifying "RoundMeasurements" and "ValidationMeasurements" as properties in NetTrain[net,data,properties] or in a NetTrainResultsObject.
  • The default key used to access a particular measurement is based on the measurement specification:
  • "measurement""measurement"
    "measurement"x"measurement"
    NetPort["name"]"name"
    NetPort[{part1,part2,,"port"}]"part1/part2//port"
  • The "Key"->"key" option can be used to specify a key for a measurement. This is primarily useful to uniquely identify measurements that have the same default key.
  • These measurement keys are also used for accessing round and validation measurements in TrainingStoppingCriterion.
  • When a textual reporting form is specified via TrainingProgressReporting, some measurements may be reported in aggregated form or not reported at all. For class-based measurements like "Precision", "Recall", "F1Score", etc., the macro-average is reported. For non-scalar NetPort[] measurements, the scalar mean is reported. For "ROCCurve" and "ConfusionMatrix", no textual summary is reported.
  • The "PlotType"type suboption can be used to specify how to scale the axis of the measurement plot. Possible types are:
  • Automaticadaptively choose the scale as appropriate (default)
    "Linear"plot the measurement on a linear scale
    "Log"plot the measurement on a log scale

Examples

open all close all

Basic Examples  (1)

Examine the accuracy for LeNet trained on FashionMNIST during training:

In[1]:=
Click for copyable input
Out[1]=

Scope  (6)

Properties & Relations  (2)

Possible Issues  (5)

Neat Examples  (1)

Introduced in 2019
(12.0)