PredictorMeasurements

PredictorMeasurements[predictor,testset,prop]

gives measurements associated with the property prop when predictor is evaluated on testset.

PredictorMeasurements[…,{prop1,prop2,…}]

gives properties prop1, prop2, etc.

PredictorMeasurements[predictor,testset]

yields a PredictorMeasurementsObject[…] that can be applied to any property.

Details and Options

  • The predictor is typically a PredictorFunction object as generated by Predict.
  • PredictorMeasurements[…,opts] specifies that the predictor should use the options opts when applied to the test set. Possible options are as given in PredictorFunction.
  • PredictorMeasurementsObject[…][prop] can be used to look up prop from a PredictorMeasurementsObject. When repeated property lookups are required, this is typically more efficient.
  • PredictorMeasurementsObject[…][prop,opts] specifies that the predictor should use the options opts when applied to the test set. It supersedes options given to PredictorMeasurements.
  • PredictorMeasurements has the same options as PredictorFunction[…], with the following additions:
  • WeightsAutomaticweights to be associated with test set examples
    ComputeUncertaintyFalsewhether measures should be given with their statistical uncertainty
  • When ComputeUncertaintyTrue, numerical measures will be returned as Around[result,err], where err represents the standard error (corresponding to a 68% confidence interval) associated with measure result.
  • Possible settings for Weights include:
  • Automaticassociates weight 1 with all test examples
    {w1,w2,…}associates weight wi with the i^(th) test examples
  • Changing the weight of a test example from 1 to 2 is equivalent to duplicating the example.
  • Weights affect measures as well as their uncertainties.
  • Properties returning a single numeric value related to prediction abilities on the test set include:
  • "StandardDeviation"root mean square of the residuals
    "StandardDeviationBaseline"standard deviation of test set values
    "LogLikelihood"log-likelihood of the model given the test data
    "MeanCrossEntropy"mean cross entropy over test examples
    "MeanDeviation"mean absolute value of the residuals
    "MeanSquare"mean square of the residuals
    "Perplexity"exponential of the mean cross entropy
    "RejectionRate"fraction of examples predicted as Indeterminate
    "GeometricMeanProbabilityDensity"geometric mean of the actual-class probability densities
  • Test examples classified as Indeterminate will be discarded when measuring properties related to prediction abilities on the test set, such as "StandardDeviation" or "MeanCrossEntropy".
  • Properties returning graphics include:
  • "Report"panel reporting main measurements
    "ComparisonPlot"plot of predicted values versus test values
    "ProbabilityDensityHistogram"histogram of actual-class probability densities
    "ResidualHistogram"histogram of residuals
    "ResidualPlot"plot of the residuals
  • Timing-related properties include:
  • "EvaluationTime"time needed to predict one example of the test set
    "BatchEvaluationTime"marginal time to predict one example in a batch
  • Properties returning one value for each test-set example include:
  • "Residuals"list of differences between predicted and test values
    "ProbabilityDensities"actual-class prediction probability densities
  • Properties returning examples from the test set include:
  • "BestPredictedExamples"examples having the highest actual-class probability density
    "Examples"all test examples
    "Examples"{i1,i2}all examples in the interval i1 predicted in the interval i2
    "LeastCertainExamples"examples having the highest distribution entropy
    "MostCertainExamples"examples having the lowest distribution entropy
    "WorstPredictedExamples"examples having the lowest actual-class probability density
  • Examples are given in the form inputivaluei, where valuei is the actual value.
  • Properties such as "WorstPredictedExamples" or "MostCertainExamples" output up to 10 examples. PredictorMeasurementsObject[…][propn] can be used to output n examples.
  • Other properties include:
  • "PredictorFunction"PredictorFunction[…] being measured
    "Properties"list of measurement properties available

Examples

open all close all

Basic Examples  (1)

Define a training set and a test set:

In[1]:=
Click for copyable input
In[2]:=
Click for copyable input

Create a predictor with the training set:

In[3]:=
Click for copyable input
Out[3]=

Visualize the residual values:

In[4]:=
Click for copyable input
Out[4]=

Compute the root mean square of the residuals:

In[5]:=
Click for copyable input
Out[5]=

Generate a PredictorMeasurementsObject of the predictor with the test set:

In[6]:=
Click for copyable input
Out[6]=

Perform the previous measurements using the PredictorMeasurementsObject:

In[7]:=
Click for copyable input
Out[7]=

Scope  (1)

Options  (5)

Applications  (2)

Introduced in 2014
(10.0)
|
Updated in 2019
(12.0)