TrainingProgressMeasurements

TrainingProgressMeasurements

is an option for NetTrain that specifies measurements to make while training is in progress.

Details

  • In TrainingProgressMeasurements->spec, the following forms for spec are allowed:
  • "measurement"a named, built-in measurement
    NetPort["output"]the value of an output port of the net
    NetPort["tdata"]the value of training data for the net
    NetPort[{lspec,"output"}]the value of an interior activation of the net
    NetPort[{lspec,"weight"}]the value of a weight array
    <|"Measurement"spec,…|>a measurement with suboptions
    <|"Measurement"Function[…],…|>a custom function to measure
  • Setting TrainingProgressMeasurements{spec1,spec2,…} will result in multiple measurements being made.
  • With the default setting of TrainingProgressMeasurementsAutomatic, the "ErrorRate" measurement will be automatically included for nets that are trained using a single CrossEntropyLossLayer.
  • For nets that contain a CrossEntropyLossLayer, the following built-in measurements are available:
  • "Accuracy"fraction of correctly classified examples
    "Accuracy"nfraction of examples with the correct result in the top n
    "AreaUnderROCCurve"area under the ROC curve for each class
    "CohenKappa"Cohen's kappa coefficient
    "ConfusionMatrix"counts cij of class i examples classified as class j
    "ConfusionMatrixPlot"plot of the confusion matrix
    "Entropy"entropy measured in nats
    "ErrorRate"fraction of incorrectly classified examples
    "ErrorRate"nfraction of examples with the incorrect result in the top n
    "F1Score"F1 score for each class
    "FScore"βFβ score for each class
    "FalseDiscoveryRate"false discovery rate for each class
    "FalseNegativeNumber"number of false negative examples
    "FalseNegativeRate"false negative rate for each class
    "FalseOmissionRate"false omission rate for each class
    "FalsePositiveNumber"number of false positive examples
    "FalsePositiveRate"false positive rate for each class
    "Informedness"informedness for each class
    "Markedness"markedness for each class
    "MatthewsCorrelationCoefficient"Matthews correlation coefficient for each class
    "NegativePredictiveValue"negative predictive value for each class
    "Perplexity"exponential of the entropy
    "Precision"precision for each class
    "Recall"recall rate for each class
    "ROCCurve"receiver operating characteristics (ROC) curve for each class
    "ROCCurvePlot"plot of the ROC curve
    "ScottPi"Scott's pi coefficient
    "Specificity"specificity for each class
    "TrueNegativeNumber"number of true negative examples
    "TruePositiveNumber"number of true positive examples
  • For nets that contain a MeanSquaredLossLayer or MeanAbsoluteLossLayer, the following built-in measurements are available:
  • "FractionVarianceUnexplained"the fraction of output variance left unexplained by the net
    "IntersectionOverUnion"intersection over union for bounding boxes
    "MeanDeviation"mean absolute value of the residuals
    "MeanSquare"mean square of the residuals
    "RSquared"coeficient of determination
    "StandardDeviation"root mean square of the residuals
  • The pure function for the custom function measurement will be supplied with the association described in TrainingProgressFunction.
  • Suboptions can be specified for a given measurement using the association form <|"Measurement"measurement,opt1val1,…|>.
  • When the source of a measurement is ambiguous, the suboption "Source""outname" can be used to select a specific output port of the net from which to take the measurement.
  • The suboption "ClassAveraging"type can be specified for measurements that are calculated per-class, such as "Precision", "Recall", "F1Score" and "ROCCurve". Possible types include:
  • Nonereturn the per-class measurements (default)
    "Macro"return the macro-average of the per-class measurements
    "Micro"return the micro-average of the per-class measurements
  • The suboption "Direction"direction specifies which direction of change in the measurement is an improvement. Possible values for directions are:
  • Automaticmeasurement direction is chosen to correspond to improved accuracy (default)
    "Decreasing"measurement should decrease
    "Increasing"measurement should increase
  • The "Aggregation"aggregation option can be used to aggregate non-scalar NetPort measurements. Possible aggregations are:
  • Nonereturn the array unchanged (default)
    "L1Norm"return the L1 norm of the flattened array
    "L2Norm"return the L2 norm of the flattened array
    "Max"return the maximum of the flattened array
    "Mean"return the mean of the flattened array
    "Min"return the minimum of the flattened array
    "RootMeanSquare"return the root mean square of the flattened array
    "StandardDeviation"return the standard deviation of the flattened array
    "Total"return the sum of the flattened array
  • The "Interval"->"interval" option specifies how often to collect a measurement. Possible intervals are:
  • Automaticautomatically choose an appropriate interval (default)
    "Batch"collect the measurement every batch
    "Round"collect the measurement every round
  • If a TrainingProgressFunction is supplied to NetTrain, associations of the most recent values of the requested measurements are available using the keys "RoundMeasurements" and "ValidationMeasurements".
  • Associations of the final value of all measurements can be obtained after training by specifying "RoundMeasurements" and "ValidationMeasurements" as properties in NetTrain[net,data,properties] or in a NetTrainResultsObject.
  • The default key used to access a particular measurement is based on the measurement specification:
  • "measurement""measurement"
    "measurement"x"measurement"
    NetPort["name"]"name"
    NetPort[{part1,part2,…,"port"}]"part1/part2/…/port"
  • The "Key"->"key" option can be used to specify a key for a measurement. This is primarily useful to uniquely identify measurements that have the same default key.
  • These measurement keys are also used for accessing round and validation measurements in TrainingStoppingCriterion.
  • When a textual reporting form is specified via TrainingProgressReporting, some measurements may be reported in aggregated form or not reported at all. For class-based measurements like "Precision", "Recall", "F1Score", etc., the macro-average is reported. For non-scalar NetPort[…] measurements, the scalar mean is reported. For "ROCCurve" and "ConfusionMatrix", no textual summary is reported.
  • The "PlotType"type suboption can be used to specify how to scale the axis of the measurement plot. Possible types are:
  • Automaticadaptively choose the scale as appropriate (default)
    "Linear"plot the measurement on a linear scale
    "Log"plot the measurement on a log scale

Examples

open allclose all

Basic Examples  (1)

Examine the accuracy for LeNet trained on FashionMNIST during training:

Scope  (7)

Examine the final validation precision and recall for LeNet trained on FashionMNIST:

Examine the final validation and training confusion matrices for LeNet trained on FashionMNIST (note that the training progress panel will show the confusion matrix for the training data by default; click the plot to toggle between the two):

Examine the final-round, macro- and micro-averaged specificity for a small toy problem, using the "Key" option to uniquely identify the measurements:

Examine the mean round output of an internal layer of the network by creating a new output port:

Alternatively, examine the same internal output using without creating a new output:

Examine the error rate of one of the outputs for a multitask trained network for the CIFAR-100 dataset:

Examine data produced by the training data generator function:

Use the custom function measurement and ClassifierMeasurements to measure properties not yet supported by TrainingProgressMeasurements:

Properties & Relations  (2)

Some of the built-in measurements, such as "Precision", "Recall", "F1Score" and "AreaUnderROCCurve", are not defined for batches and will only be measured after every completed round:

For non-batch measurements, setting "Interval"->"Batch" will have no effect:

Validation measurements will only be computed if a validation set is provided and will include measurements not defined on batches:

The "IntersectionOverUnion" measurement expects the input and target bounding boxes to be supplied as lists with the form {x1,y1,x2,y2}, where (x1,y1) and (x2,y2) are the coordinates describing the bottom-left and top-right corners of a bounding box.

The calculation of intersection over union is equivalent to the following Mathematica function:

Possible Issues  (5)

The shorthand syntax TrainingProgressMeasurementsmeasurement can only be used if there is exactly one source for measurement:

Specify which output to measure:

If two or more measurements that have the same default key are requested, the "Key"->"key" suboption must be used to uniquely identify them:

Custom function measurements must have the "Key" suboption specified:

Custom function measurements must return a numeric result:

Trying to measure an output port being used for the LossFunction can fail:

A simple workaround is to create an additional output port to measure and explicitly label the training data:

Neat Examples  (1)

Animate the history of the confusion matrix for a LeNet trained on a sample of FashionMNIST:

Wolfram Research (2019), TrainingProgressMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html.

Text

Wolfram Research (2019), TrainingProgressMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html.

CMS

Wolfram Language. 2019. "TrainingProgressMeasurements." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html.

APA

Wolfram Language. (2019). TrainingProgressMeasurements. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html

BibTeX

@misc{reference.wolfram_2024_trainingprogressmeasurements, author="Wolfram Research", title="{TrainingProgressMeasurements}", year="2019", howpublished="\url{https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html}", note=[Accessed: 21-November-2024 ]}

BibLaTeX

@online{reference.wolfram_2024_trainingprogressmeasurements, organization={Wolfram Research}, title={TrainingProgressMeasurements}, year={2019}, url={https://reference.wolfram.com/language/ref/TrainingProgressMeasurements.html}, note=[Accessed: 21-November-2024 ]}