ClassifierMeasurements

ClassifierMeasurements[classifier,testset,prop]

gives measurements associated with property prop when classifier is evaluated on testset.

ClassifierMeasurements[classifier,testset]

yields a ClassifierMeasurementsObject[] that can be applied to any property.

Details and Options

  • The classifier is either a ClassifierFunction object as generated by Classify, or a NetGraph, NetChain, etc. in which the output is a "Class" NetDecoder.
  • Properties supported include:
  • "Accuracy"fraction of correctly classified examples
    "Accuracy"ntop-n accuracy
    "AccuracyRejectionPlot"plot of the accuracy as function of the rejection rate
    "AreaUnderROCCurve"area under the ROC curve for each class
    "BestClassifiedExamples"examples having the highest actual-class probability
    "ClassifierFunction"ClassifierFunction[] being measured
    "ClassMeanCrossEntropy"mean cross entropy for each class
    "ClassRejectionRate"rejection rate for each class
    "CohenKappa"Cohen's kappa coefficient
    "ConfusionFunction"function giving confusion matrix values
    "ConfusionMatrix"counts cij of class i examples classified as class j
    "ConfusionMatrixPlot"plot of the confusion matrix
    "ConfusionMatrixPlot"{c1,c2,}confusion matrix plot restricted to classes c1, c2, etc.
    "ConfusionMatrixPlot"nconfusion matrix plot for the worst n-class subset
    "CorrectlyClassifiedExamples"examples correctly classified
    "DecisionUtilities"value of the utility function for each example
    "Error"fraction of incorrectly classified examples
    "Examples"all test examples
    "Examples"{i,j}all class i examples classified as class j
    "F1Score"F1 score for each class
    "FalseDiscoveryRate"false discovery rate for each class
    "FalseNegativeExamples"false negative test examples for each class
    "FalseNegativeRate"false negative rate for each class
    "FalsePositiveExamples"false positive test examples for each class
    "FalsePositiveRate"false positive rate for each class
    "GeometricMeanProbability"geometric mean of the actual-class probabilities
    "IndeterminateExamples"test examples classified as Indeterminate
    "LeastCertainExamples"examples having the highest distribution entropy
    "LogLikelihood"log-likelihood of the model given the test set
    "MatthewsCorrelationCoefficient"Matthews correlation coefficient for each class
    "MeanCrossEntropy"mean cross entropy over test examples
    "MeanDecisionUtility"mean utility over test example
    "MisclassifiedExamples"examples misclassified
    "MostCertainExamples"examples having the lowest distribution entropy
    "NegativePredictedValue"negative predicted value for each class
    "Perplexity"exponential of the mean cross entropy
    "Precision"precision of classification for each class
    "Probabilities"actual-class classification probabilities
    "ProbabilityHistogram"histogram of actual-class probabilities
    "Properties"list of measurement properties available
    "Recall"recall rate of classification for each class
    "RejectionRate"fraction of examples classified as Indeterminate
    "ROCCurve"Receiver Operating Characteristics curve
    "ScottPi"Scott's pi coefficient
    "Specificity"specificity for each class
    "TopConfusions"pairs of classes that are most confused
    "TopConfusions"nn most confused class pairs
    "TrueNegativeExamples"true negative test examples for each class
    "TruePositiveExamples"true positive test examples for each class
    "WorstClassifiedExamples"examples having the lowest actual-class probability
  • ClassifierMeasurements[,opts] specifies that the classifier should use the options opts when applied to the test set. Possible options are as given in ClassifierFunction.
  • ClassifierMeasurementsObject[][prop] can be used to give properties prop from a ClassifierMeasurementsObject. When repeated property lookups are required, this is typically more efficient.
  • ClassifierMeasurementsObject[][prop,opts] specifies that the classifier should use the options opts when applied to the test set. It supersedes options given to ClassifierMeasurements.
  • Test examples classified as Indeterminate are discarded when measuring properties such as "Accuracy", "Error", "Precision", "Recall" or "F1Score".
  • Properties such as "Precision" or "Recall" give one measure for each possible "positive class". The "negative class" is the union of all the classes that are not the "positive class". ClassifierMeasurementsObject[][propclass] specifies that class is the "positive class".
  • In properties giving test examples such as "Examples" or "WorstClassifiedExamples", examples are in the form inputiclassi, where classi is the actual class.
  • Properties such as "WorstClassifiedExamples" or "MostCertainExamples" output up to 10 examples. ClassifierMeasurementsObject[][propn] can be used to output n examples.
  • ClassifierMeasurements has the same options as ClassifierFunction[], with the following additions:
  • WeightsAutomaticweights to be associated with test set examples
    "Uncertainty"Falsewhether measures should be given with their statistical uncertainty
  • With the setting "Uncertainty"True, numerical measures will be given in the form measure±err where err represents the standard error (corresponding to a 68% confidence interval) associated with measure.
  • Possible settings for Weights include:
  • Automaticassociates weight 1 with all test examples
    {w1,w2,}associates weight wi with the i^(th) test examples
  • Changing the weight of a test example from 1 to 2 is equivalent to duplicating this example.
  • Weights affect measures as well as their uncertainties.

Examples

open allclose all

Basic Examples  (1)

Define a training set and a test set:

In[1]:=
Click for copyable input
In[2]:=
Click for copyable input

Create a classifier on the training set:

In[3]:=
Click for copyable input
Out[3]=

Measure the accuracy of the classifier on the test set:

In[4]:=
Click for copyable input
Out[4]=

Measure the F1 score of each class:

In[5]:=
Click for copyable input
Out[5]=

Obtain a function giving the number of test examples for each pair of actual class and predicted class:

In[6]:=
Click for copyable input
Out[6]=

Use the function to obtain the number of examples of class "B" predicted as "A":

In[7]:=
Click for copyable input
Out[7]=

Generate a ClassifierMeasurementsObject of the classifier with the test set:

In[8]:=
Click for copyable input
Out[8]=

Perform the previous measurements using the ClassifierMeasurementsObject:

In[9]:=
Click for copyable input
Out[9]//TableForm=

Scope  (2)

Options  (5)

Applications  (2)

See Also

ClassifierMeasurementsObject  Classify  ClassifierInformation  ClassifierFunction  PredictorMeasurements  LogitModelFit

Introduced in 2014
(10.0)