WOLFRAM

Wolfram Language & System Documentation Center
Wolfram Language Home Page »

PredictorMeasurements
✖
PredictorMeasurements

✖
PredictorMeasurements[predictor,testset,prop]

gives measurements associated with the property prop when predictor is evaluated on testset.

✖
PredictorMeasurements[predictor,testset]

yields a measurement report that can be applied to any property.

✖

use predictions data instead of a predictor.

✖
PredictorMeasurements[…,{prop1,prop2,…}]

gives properties prop1, prop2, etc.

Details and Options

  • Measurements are used to determine the performance of a classifier on data that was not used for training purposes (the test set).
  • Possible measurements include classification metrics (accuracy, likelihood, etc.), visualizations (confusion matrix, ROC curve, etc.) or specific examples (such as the worst classified examples).
  • The predictor is typically a PredictorFunction object as generated by Predict.
  • In PredictorMeasurements[data,…], the predictions data can have the following forms:
  • {y1,y2,…}predictions from a predictor (human, algorithm, etc.)
    {dist1,dist2,…}predictive distributions obtained by a predictor
  • PredictorMeasurements[…,opts] specifies that the predictor should use the options opts when applied to the test set. Possible options are as given in PredictorFunction.
  • PredictorMeasurements[predictor,testset] returns a PredictorMeasurementsObject[…] that displays as a report panel, such as:
  • PredictorMeasurementsObject[…][prop] can be used to look up prop from a PredictorMeasurementsObject. When repeated property lookups are required, this is typically more efficient than using PredictorMeasurements every time.
  • PredictorMeasurementsObject[…][prop,opts] specifies that the predictor should use the options opts when applied to the test set. These options supersede original options given to PredictorMeasurements.
  • PredictorMeasurements has the same options as PredictorFunction[…], with the following additions:
  • Weights Automaticweights to be associated with test set examples
    ComputeUncertaintyFalsewhether measures should be given with their statistical uncertainty
  • When ComputeUncertaintyTrue, numerical measures will be returned as Around[result,err], where err represents the standard error (corresponding to a 68% confidence interval) associated with measure result.
  • Possible settings for Weights include:
  • Automaticassociates weight 1 with all test examples
    {w1,w2,…}associates weight wi with the i^(th) test examples
  • Changing the weight of a test example from 1 to 2 is equivalent to duplicating the example.
  • Weights affect measures as well as their uncertainties.
  • Properties returning a single numeric value related to prediction abilities on the test set include:
  • "StandardDeviation"root mean square of the residuals
    "StandardDeviationBaseline"standard deviation of test set values
    "LogLikelihood"log-likelihood of the model given the test data
    "MeanCrossEntropy"mean cross entropy over test examples
    "MeanDeviation"mean absolute value of the residuals
    "MeanSquare"mean square of the residuals
    "RSquared"coefficient of determination
    "FractionVarianceUnexplained"fraction of variance unexplained
    "Perplexity"exponential of the mean cross entropy
    "RejectionRate"fraction of examples predicted as Indeterminate
    "GeometricMeanProbabilityDensity"geometric mean of the actual-class probability densities
  • Test examples classified as Indeterminate will be discarded when measuring properties related to prediction abilities on the test set, such as "StandardDeviation" or "MeanCrossEntropy".
  • Properties returning graphics include:
  • "ComparisonPlot"plot of predicted values versus test values
    "ICEPlots"Individual Conditional Expectation (ICE) plots
    "ProbabilityDensityHistogram"histogram of actual-class probability densities
    "Report"panel reporting main measurements
    "ResidualHistogram"histogram of residuals
    "ResidualPlot"plot of the residuals
    "SHAPPlots"Shapley additive feature explanations plot for each class
  • Timing-related properties include:
  • "EvaluationTime"time needed to predict one example of the test set
    "BatchEvaluationTime"marginal time to predict one example in a batch
  • Properties returning one value for each test-set example include:
  • "Residuals"list of differences between predicted and test values
    "ProbabilityDensities"actual-class prediction probability densities
    "SHAPValues"Shapley additive feature explanations for each example
  • "SHAPValues" assesses the contribution of features by comparing predictions with different sets of features removed and then synthesized. The option MissingValueSynthesis can be used to specify how the missing features are synthesized. SHAP explanations are given as deviation from the training output mean. "SHAPValues"n can be used to control the number of samples used for the numeric estimations of SHAP explanations.
  • Properties returning examples from the test set include:
  • "BestPredictedExamples"examples having the highest actual-class probability density
    "Examples"all test examples
    "Examples"{i1,i2}all examples in the interval i1 predicted in the interval i2
    "LeastCertainExamples"examples having the highest distribution entropy
    "MostCertainExamples"examples having the lowest distribution entropy
    "WorstPredictedExamples"examples having the lowest actual-class probability density
  • Examples are given in the form inputivaluei, where valuei is the actual value.
  • Properties such as "WorstPredictedExamples" or "MostCertainExamples" output up to 10 examples. PredictorMeasurementsObject[…][propn] can be used to output n examples.
  • Other properties include:
  • "PredictorFunction"PredictorFunction[…] being measured
    "Properties"list of measurement properties available

Examples

open allclose all

Basic Examples  (3)Summary of the most common use cases

Train a predictor on a training set:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-cgrfnb
Out[1]=1

Define a test set:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-zefo51

Measure the standard deviation of the predictor on the test set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-4svu8i
Out[3]=3

Visualize the residuals:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-ianu4o
Out[4]=4

Visualize a scatter plot of the actual and predicted values:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-k7fr3n
Out[5]=5

Measure several properties at once:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-ow3v5p
Out[5]=5

Train a predictor on a training set:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-btpoj7
Out[1]=1

Generate a measurement object of the predictor on the test set:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-njqae6
Out[2]=2

Obtain the list of measurement properties available:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-mmkv1w
Out[3]=3

Measure the standard deviation of the predictor on the test set:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-iwve7t
Out[4]=4

Obtain the standard deviation along with its statistical uncertainty due to finite test-set size:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-eo6qwz
Out[5]=5

Obtain several measures:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-xvbf6j
Out[6]=6

Measure the standard deviation directly from classified examples:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-zlbna9
Out[1]=1

Obtain a measurement report:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-ilqudd
In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-prvy29

Measure the log-likelihood from predictive distributions:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-dtmenz
Out[4]=4

Scope  (3)Survey of the scope of standard use cases

Residual-Based Metrics  (1)

Visualize the residuals of predicted examples against their true values:

In[14]:=14
✖
https://wolfram.com/xid/0fq5rt0583bcj-mdxba4
Out[14]=14

Measure the standard deviation:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-cw1rai
Out[5]=5

This is equivalent to computing the root mean square of the residuals:

In[15]:=15
✖
https://wolfram.com/xid/0fq5rt0583bcj-pgeez2
Out[15]=15

Obtain the statistical uncertainty on this measure:

In[16]:=16
✖
https://wolfram.com/xid/0fq5rt0583bcj-1j8guf
Out[16]=16

Compare the standard deviation to a baseline (always predicting the mean of the test-set values):

In[18]:=18
✖
https://wolfram.com/xid/0fq5rt0583bcj-hzxbnj
Out[18]=18

Measure the -squared value:

In[24]:=24
✖
https://wolfram.com/xid/0fq5rt0583bcj-rizj8d
Out[24]=24

This is equivalent to the proportion of variance that is explained:

In[25]:=25
✖
https://wolfram.com/xid/0fq5rt0583bcj-0t654v
Out[25]=25

Measure the fraction of variance unexplained:

In[26]:=26
✖
https://wolfram.com/xid/0fq5rt0583bcj-pa2e0g
Out[26]=26

These two values sum to one:

In[27]:=27
✖
https://wolfram.com/xid/0fq5rt0583bcj-tbd8d3
Out[27]=27

Comparison Plot and Example Extraction  (1)

Create a training set and a test set on the Boston homes data:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-pyoo4i
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-7aoe1o
Out[2]=2

Train a model on the training set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-21d6pb
Out[3]=3

Create a classifier measurements object for this classifier on the test set:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-4q52n7
Out[4]=4

Find the two test examples that have the worst predictions:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-rpwh8c
Out[5]=5

Compute their predictions:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-uvq0w0
Out[6]=6

Find the two test examples that have the best predictions:

In[7]:=7
✖
https://wolfram.com/xid/0fq5rt0583bcj-vpr8tw
Out[7]=7

Compute their predictions:

In[8]:=8
✖
https://wolfram.com/xid/0fq5rt0583bcj-y48qyk
Out[8]=8

Compare the predicted and correct values:

In[9]:=9
✖
https://wolfram.com/xid/0fq5rt0583bcj-ujsxrz
Out[9]=9

Extract the examples whose true values are between 30 and 40 and whose predictions are between 40 and 50:

In[10]:=10
✖
https://wolfram.com/xid/0fq5rt0583bcj-dtzgm1
Out[10]=10

Probabilistic Metrics  (1)

Create and visualize an artificial dataset from the expression Cos[x*y]:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-j0yb8r
Out[1]=1
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-lpxlmd
In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-gpi61p
Out[3]=3

Split the dataset into a training set and a test set:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-jxalvz

Train a predictor on the training set:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-pvobk1
Out[5]=5

Measure the log-likelihood of the test set (total log-PDF of actual value):

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-i8c4e6
Out[6]=6

Measure the mean cross-entropy:

In[7]:=7
✖
https://wolfram.com/xid/0fq5rt0583bcj-wlydgl
Out[7]=7

The mean cross-entropy is the average negative log-likelihood:

In[8]:=8
✖
https://wolfram.com/xid/0fq5rt0583bcj-z09nmu
Out[8]=8

Options  (5)Common values & functionality for each option

IndeterminateThreshold  (1)

Create an artificial dataset and visualize it:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-80hqll
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-3mw01s
Out[2]=2

Split the dataset into a training set and a test set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-ltivz7

Train a predictor on the training set:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-07q9td
Out[4]=4

Plot the predicted distribution for a few feature values:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-eqpzvo
Out[5]=5

Compute the root mean square of the residuals:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-dkyl33
Out[6]=6

Perform the same computation with a different threshold value for the predictor:

In[7]:=7
✖
https://wolfram.com/xid/0fq5rt0583bcj-670wy6
Out[7]=7

This operation can also be done on the PredictorMeasurementsObject:

In[8]:=8
✖
https://wolfram.com/xid/0fq5rt0583bcj-vlbmy9
Out[8]=8
In[9]:=9
✖
https://wolfram.com/xid/0fq5rt0583bcj-nthx6z
Out[9]=9

Plot the standard deviation and the rejection rate as a function of the threshold:

In[10]:=10
✖
https://wolfram.com/xid/0fq5rt0583bcj-tz4f79
Out[10]=10
In[11]:=11
✖
https://wolfram.com/xid/0fq5rt0583bcj-o8xjwj
Out[11]=11

TargetDevice  (1)

Train a predictor using a neural network:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-gnlysk

Measure the standard deviation of the predictor on a test set for different setting of TargetDevice:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-4fgmey
In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-dqg5pp
In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-osicmu

UtilityFunction  (1)

Define a training and a test set:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-d5e35w
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-c1yd0z

Train a predictor on the training set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-fg10uw
Out[3]=3

Define and visualize a utility function that penalizes the predicted value's being smaller than the actual value:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-0p1sgu
In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-ygfxmi
Out[5]=5

Compute the residuals of the predictor on the test set with this utility function:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-ynz2ar
Out[6]=6

The residuals with the default utility function are higher:

In[7]:=7
✖
https://wolfram.com/xid/0fq5rt0583bcj-m2orow
Out[7]=7

The utility function can also be specified when using the PredictorMeasurementsObject:

In[8]:=8
✖
https://wolfram.com/xid/0fq5rt0583bcj-qg0op0
Out[8]=8
In[9]:=9
✖
https://wolfram.com/xid/0fq5rt0583bcj-5j54kc
Out[9]=9

"Uncertainty"  (1)

Train a predictor on the "WineQuality" dataset:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-nfgpho
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-xxoqad
Out[2]=2

Generate a PredictorMeasurements[…] object using a test set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-vpffp
In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-72st1s
Out[4]=4

Obtain a measure of the standard deviation along with its uncertainty:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-6iopsc
Out[5]=5

Obtain a measure of other properties along with their uncertainties:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-mat67g
Out[6]=6

Weights  (1)

Create a predictor on a training set:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-undf2n
Out[1]=1

Generate a measurement object while specifying the weights that each test example has:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-u0r98b
Out[2]=2

Compute the standard deviation:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-ttvh6t
Out[3]=3

Weights can also be modified when using the measurement object:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-4oromz
Out[4]=4

Uncertainties are also affected by weights:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-us8urh
Out[5]=5
In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-yyigru
Out[6]=6

Applications  (2)Sample problems that can be solved with this function

Load a dataset of the average monthly temperature as a function of the city, the year and the month:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-38fkv
In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-lw5ujc

Split the dataset into a training set and a test set:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-txgfzg
In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-0qu7d1

Train a predictor on the training set:

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-oinmn3
Out[5]=5

Generate a PredictorMeasurementsObject from the predictor and the test set:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-c0qrii
Out[6]=6

Compute the mean cross entropy of the classifier on the test set:

In[7]:=7
✖
https://wolfram.com/xid/0fq5rt0583bcj-fn1wqz
Out[7]=7

Visualize the scatter plot of the test values as a function of the predicted values:

In[8]:=8
✖
https://wolfram.com/xid/0fq5rt0583bcj-8zeap
Out[8]=8

Extract the test examples that are in a given region of the comparison plot:

In[9]:=9
✖
https://wolfram.com/xid/0fq5rt0583bcj-1mu2vg
Out[9]=9

Extract the 20 worst predicted examples:

In[10]:=10
✖
https://wolfram.com/xid/0fq5rt0583bcj-9hbb76
Out[10]=10

Train a predictor that predicts the median value of properties in a neighborhood of Boston, given some features of the neighborhood:

In[1]:=1
✖
https://wolfram.com/xid/0fq5rt0583bcj-cs8a11
Out[1]=1

Generate a predictor measurements object to analyze the performance of the predictor on a test set:

In[2]:=2
✖
https://wolfram.com/xid/0fq5rt0583bcj-fiomr4
Out[2]=2

Plot the residuals:

In[3]:=3
✖
https://wolfram.com/xid/0fq5rt0583bcj-47j7fi
Out[3]=3

Plot a histogram of the residuals:

In[4]:=4
✖
https://wolfram.com/xid/0fq5rt0583bcj-oh305
Out[4]=4

Compute the standard deviation of the predicted values from the actual values (root mean square of the residuals):

In[5]:=5
✖
https://wolfram.com/xid/0fq5rt0583bcj-lxrlde
Out[5]=5

Obtain the statistical uncertainty of the above measure:

In[6]:=6
✖
https://wolfram.com/xid/0fq5rt0583bcj-sd02k8
Out[6]=6
Wolfram Research (2014), PredictorMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/PredictorMeasurements.html (updated 2021).
✖
Wolfram Research (2014), PredictorMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/PredictorMeasurements.html (updated 2021).

Text

Wolfram Research (2014), PredictorMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/PredictorMeasurements.html (updated 2021).

✖
Wolfram Research (2014), PredictorMeasurements, Wolfram Language function, https://reference.wolfram.com/language/ref/PredictorMeasurements.html (updated 2021).

CMS

Wolfram Language. 2014. "PredictorMeasurements." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2021. https://reference.wolfram.com/language/ref/PredictorMeasurements.html.

✖
Wolfram Language. 2014. "PredictorMeasurements." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2021. https://reference.wolfram.com/language/ref/PredictorMeasurements.html.

APA

Wolfram Language. (2014). PredictorMeasurements. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/PredictorMeasurements.html

✖
Wolfram Language. (2014). PredictorMeasurements. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/PredictorMeasurements.html

BibTeX

@misc{reference.wolfram_2025_predictormeasurements, author="Wolfram Research", title="{PredictorMeasurements}", year="2021", howpublished="\url{https://reference.wolfram.com/language/ref/PredictorMeasurements.html}", note=[Accessed: 16-April-2025 ]}

✖
@misc{reference.wolfram_2025_predictormeasurements, author="Wolfram Research", title="{PredictorMeasurements}", year="2021", howpublished="\url{https://reference.wolfram.com/language/ref/PredictorMeasurements.html}", note=[Accessed: 16-April-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_predictormeasurements, organization={Wolfram Research}, title={PredictorMeasurements}, year={2021}, url={https://reference.wolfram.com/language/ref/PredictorMeasurements.html}, note=[Accessed: 16-April-2025 ]}

✖
@online{reference.wolfram_2025_predictormeasurements, organization={Wolfram Research}, title={PredictorMeasurements}, year={2021}, url={https://reference.wolfram.com/language/ref/PredictorMeasurements.html}, note=[Accessed: 16-April-2025 ]}