ClassifierMeasurementsObject

ClassifierMeasurementsObject[…]

represents an object generated by ClassifierMeasurements that can be applied to properties.

Details and Options

  • ClassifierMeasurementsObject[…][prop] is used to look up property prop from the ClassifierMeasurementsObject.
  • ClassifierMeasurementsObject[…][{prop1,prop2,…}] is used to look up many properties.
  • ClassifierMeasurementsObject[…][prop,opts] specifies that the classifier should use the options opts when applied to the test set. These options supersede original options given to ClassifierMeasurements.
  • Possible options are as given in ClassifierFunction[…], with the following addition:
  • ComputeUncertaintyFalsewhether measures should be given with their statistical uncertainty
  • When ComputeUncertaintyTrue, numerical measures will be returned as Around[result,err], where err represents the standard error (corresponding to a 68% confidence interval) associated with measure result.
  • Properties returning a single numeric value related to classification abilities on the test set include:
  • "Accuracy"fraction of correctly classified examples
    "Accuracy"ntop-n accuracy
    "AccuracyBaseline"accuracy if predicting the commonest class
    "CohenKappa"Cohen's kappa coefficient
    "Error"fraction of incorrectly classified examples
    "GeometricMeanProbability"geometric mean of the actual-class probabilities
    "LogLikelihood"log-likelihood of the model given the test set
    "MeanCrossEntropy"mean cross entropy over test examples
    "MeanDecisionUtility"mean utility over test example
    "Perplexity"exponential of the mean cross entropy
    "ScottPi"Scott's pi coefficient
    "RejectionRate"fraction of examples classified as Indeterminate
  • Examples classified as Indeterminate are discarded when measuring properties related to classification abilities on the test set, such as "Accuracy", "Error" or "MeanCrossEntropy".
  • Confusion matrix–related properties include:
  • "ConfusionMatrix"counts cij of class i examples classified as class j
    "ConfusionMatrixPlot"plot of the confusion matrix
    "ConfusionMatrixPlot"{c1,c2,…}confusion matrix plot restricted to classes c1, c2, etc.
    "ConfusionMatrixPlot"nconfusion matrix plot for the worst n-class subset
    "ConfusionFunction"function giving confusion matrix values
    "TopConfusions"pairs of classes that are most confused
    "TopConfusions"nn most confused class pairs
  • Timing-related properties include:
  • "EvaluationTime"time needed to classify one example of the test set
    "BatchEvaluationTime"marginal time to classify one example in a batch
  • Properties returning one value for each test-set example include:
  • "DecisionUtilities"value of the utility function for each example
    "Probabilities"actual-class classification probabilities for each example
    "SHAPValues"Shapley additive feature explanations for each example
  • "SHAPValues" assesses the contribution of features by comparing predictions with different sets of features removed and then synthesized. The option MissingValueSynthesis can be used to specify how the missing features are synthesized. SHAP explanations are given as odds ratio multipliers with respect to the class training prior. "SHAPValues"n can be used to control the number of samples used for the numeric estimations of SHAP explanations.
  • Properties related to probability calibration include:
  • "CalibrationCurve"probability calibration curve in logit scale
    "LinearCalibrationCurve"probability calibration curve in linear scale
    "CalibrationData"probability calibration curve data
  • Properties returning graphics include:
  • "Report"panel reporting main measurements
    "ROCCurve"Receiver Operating Characteristics curve for each class
    "ProbabilityHistogram"histogram of actual-class probabilities
    "AccuracyRejectionPlot"plot of the accuracy as function of the rejection rate
    "ReliabilityDiagram"probability calibration curve in logit scale
    "ClassicReliabilityDiagram"probability calibration curve in linear scale
  • Properties returning examples from the test set include:
  • "Examples"all test examples
    "Examples"{i,j}all class i examples classified as class j
    "BestClassifiedExamples"examples having the highest actual-class probability
    "WorstClassifiedExamples"examples having the lowest actual-class probability
    "CorrectlyClassifiedExamples"examples correctly classified
    "MisclassifiedExamples"examples misclassified
    "TruePositiveExamples"true positive test examples for each class
    "FalsePositiveExamples"false positive test examples for each class
    "TrueNegativeExamples"true negative test examples for each class
    "FalseNegativeExamples"false negative test examples for each class
    "IndeterminateExamples"test examples classified as Indeterminate
    "LeastCertainExamples"examples having the highest distribution entropy
    "MostCertainExamples"examples having the lowest distribution entropy
  • Examples are given in the form inputiclassi, where classi is the actual class from the test set.
  • Properties such as "WorstClassifiedExamples" or "MostCertainExamples" output up to 10 examples. ClassifierMeasurementsObject[…][propn] can be used to output n examples.
  • Properties returning one measure for each class include:
  • "AreaUnderROCCurve"area under the ROC curve for each class
    "ClassMeanCrossEntropy"mean cross entropy for each class
    "ClassRejectionRate"rejection rate for each class
    "F1Score"F1 score for each class
    "FalseDiscoveryRate"false discovery rate for each class
    "FalseNegativeRate"false negative rate for each class
    "FalsePositiveRate"false positive rate for each class
    "MatthewsCorrelationCoefficient"Matthews correlation coefficient for each class
    "NegativePredictiveValue"negative predictive value for each class
    "Precision"precision of classification for each class
    "Recall"recall rate of classification for each class
    "Specificity"specificity for each class
    "TruePositiveNumber"number of true positive examples
    "FalsePositiveNumber"number of false positive examples
    "TrueNegativeNumber"number of true negative examples
    "FalseNegativeNumber"number of false negative examples
  • ClassifierMeasurementsObject[…][propclass] can be used to only return the measure associated with the specified class.
  • ClassifierMeasurementsObject[…][prop<|class1w1,class2w2,…|>] can be used to return a weighted average of each class measure.
  • ClassifierMeasurementsObject[…][propf] can be used to apply function f to the returned class measures (e.g. ClassifierMeasurementsObject[…][propMean]).
  • Properties such as "Precision" or "Recall" give one measure for each possible "positive class". The "negative class" is the union of all the classes that are not the positive class. For such properties, one can average the measures for all possible positive classes using ClassifierMeasurementsObject[…][propaverage], where average can be:
  • "MacroAverage"takes the mean of the measures
    "WeightedMacroAverage"weights each measure by its related class frequency
    "MicroAverage"joins true positive/true negative, etc. examples of all classes to give a unique measure
  • Other properties include:
  • "ClassifierFunction"ClassifierFunction[…] being measured
    "Properties"list of measurement properties available

Examples

Basic Examples  (1)

Define a training set and a test set:

Create a classifier on the training set:

Generate a ClassifierMeasurementsObject of the classifier with the test set:

Measure accuracy and F-score and plot the confusion matrix from the ClassifierMeasurementsObject:

Wolfram Research (2015), ClassifierMeasurementsObject, Wolfram Language function, https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html (updated 2020).

Text

Wolfram Research (2015), ClassifierMeasurementsObject, Wolfram Language function, https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html (updated 2020).

CMS

Wolfram Language. 2015. "ClassifierMeasurementsObject." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html.

APA

Wolfram Language. (2015). ClassifierMeasurementsObject. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html

BibTeX

@misc{reference.wolfram_2024_classifiermeasurementsobject, author="Wolfram Research", title="{ClassifierMeasurementsObject}", year="2020", howpublished="\url{https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html}", note=[Accessed: 21-December-2024 ]}

BibLaTeX

@online{reference.wolfram_2024_classifiermeasurementsobject, organization={Wolfram Research}, title={ClassifierMeasurementsObject}, year={2020}, url={https://reference.wolfram.com/language/ref/ClassifierMeasurementsObject.html}, note=[Accessed: 21-December-2024 ]}