PredictorMeasurements

PredictorMeasurements[predictor,testset,prop]

gives measurements associated with the property prop when predictor is evaluated on testset.

PredictorMeasurements[predictor,testset]

yields a PredictorMeasurementsObject[…] that can be applied to any property.

Details and Options

  • The predictor is typically a PredictorFunction object as generated by Predict.
  • The following properties can be measured:
  • "BatchEvaluationTime"marginal time to predict one example when a batch is given
    "BestPredictedExamples"examples having the highest actual-class probability density
    "ComparisonPlot"plot of predicted values versus test values
    "EvaluationTime"time needed to predict one example
    "Examples"all test examples
    "Examples"{i1,i2}all examples in the interval i1 predicted in the interval i2
    "GeometricMeanProbabilityDensity"geometric mean of the actual-class probability densities
    "LeastCertainExamples"examples having the highest distribution entropy
    "LogLikelihood"log-likelihood of the model given the test data
    "MeanCrossEntropy"mean cross entropy over test examples
    "MeanDeviation"mean absolute value of the residuals
    "MeanSquare"mean square of the residuals
    "MostCertainExamples"examples having the lowest distribution entropy
    "Perplexity"exponential of the mean cross entropy
    "PredictorFunction"PredictorFunction[…] being measured
    "ProbabilityDensities"actual-class prediction probability densities
    "ProbabilityDensityHistogram"histogram of actual-class probability densities
    "Properties"list of measurement properties available
    "RejectionRate"fraction of examples predicted as Indeterminate
    "Report"panel reporting main measurements
    "ResidualHistogram"histogram of residuals
    "ResidualPlot"plot of the residuals
    "Residuals"list of differences between predicted and test values
    "StandardDeviation"root mean square of the residuals
    "StandardDeviationBaseline"standard deviation of test set values
    "WorstPredictedExamples"examples having the lowest actual-class probability density
  • PredictorMeasurements[…,opts] specifies that the predictor should use the options opts when applied to the test set. Possible options are as given in PredictorFunction.
  • PredictorMeasurementsObject[…][prop] can be used to look up prop from a PredictorMeasurementsObject. When repeated property lookups are required, this is typically more efficient.
  • PredictorMeasurementsObject[…][prop,opts] specifies that the predictor should use the options opts when applied to the test set. It supersedes options given to PredictorMeasurements.
  • Test examples classified as Indeterminate will be discarded when measuring properties "MeanDeviation", "MeanSquare", "Residuals", "ResidualPlot", "ResidualHistogram" and "StandardDeviation".
  • In properties giving test examples such as "Examples" or "WorstPredictedExamples", examples are in the form inputivaluei where valuei is the actual value.
  • Properties such as "WorstPredictedExamples" or "MostCertainExamples" output up to 10 examples. PredictorMeasurementsObject[…][propn] can be used to output n example.
  • PredictorMeasurements has the same options as PredictorFunction[…], with the following additions:
  • WeightsAutomaticweights to be associated with test set examples
    "Uncertainty"Falsewhether measures should be given with their statistical uncertainty
  • With the setting "Uncertainty"True, numerical measures will be given in the form measure±err, where err represents the standard error (corresponding to a 68% confidence interval) associated with measure.
  • Possible settings for weights include:
  • Automaticassociates weight 1 with all test examples
    {w1,w2,…}associates weight wi with the i^(th) test examples
  • Changing the weight of a test example from 1 to 2 is equivalent to duplicating this example.
  • Weights affect measures as well as their uncertainties.

Examples

open allclose all

Basic Examples  (1)

Define a training set and a test set:

In[1]:=
Click for copyable input
In[2]:=
Click for copyable input

Create a predictor with the training set:

In[3]:=
Click for copyable input
Out[3]=

Visualize the residual values:

In[4]:=
Click for copyable input
Out[4]=

Compute the root mean square of the residuals:

In[5]:=
Click for copyable input
Out[5]=

Generate a PredictorMeasurementsObject of the predictor with the test set:

In[6]:=
Click for copyable input
Out[6]=

Perform the previous measurements using the PredictorMeasurementsObject:

In[7]:=
Click for copyable input
Out[7]=

Scope  (1)

Options  (5)

Applications  (2)

See Also

PredictorMeasurementsObject  Predict  PredictorInformation  PredictorFunction  ClassifierMeasurements  LinearModelFit  Histogram

Introduced in 2014
(10.0)
| Updated in 2018
(11.3)