TestReport

TestReport["file"]

gives a report of the results of the tests from a file.

TestReport[{test1,test2,…}]

gives a report of the results of the testi.

TestReport[{report1,report2,…}]

gives a unified report by merging all test reports reporti.

Details and Options

  • TestReport returns a TestReportObject.
  • TestReport handles both plain text Wolfram Language files and notebooks that use the Testing Notebook framework.
  • TestReport only runs tests with outcome "NotEvaluated", skipping ones that have already been evaluated.
  • TestReport has the following options:
  • HandlerFunctions <||>how to handle events generated
    HandlerFunctionsKeys Automaticwhat parameters to supply to handler functions
    MemoryConstraint Infinitymemory (in bytes) each test is allowed to use
    ProgressReporting $ProgressReportingwhether to report the progress
    SameTest SameQfunction to compare actual and expected outputs
    TestEvaluationFunction TestEvaluatefunction to evaluate created tests
    TimeConstraint Infinitytime (in seconds) each test is allowed to use
  • TestReport[File["file"]] is also supported.
  • During the execution of TestReport, the following events can be generated:
  • "FileStarted"test file started
    "FileCompleted"test file completed
    "ReportStarted"test report started
    "ReportCompleted"test report completed
    "RuntimeFailure"runtime failure encountered
    "TestCreated"test created
    "TestEvaluated"test evaluated
  • With the specification HandlerFunctions-><|…,"eventi"->fi,…|>, fi[assoc] is evaluated whenever eventi is generated. The elements of assoc have keys specified by the setting for HandlerFunctionsKeys.
  • Possible keys specified by HandlerFunctionsKeys include:
  • "EventName"name of the event being handled
    "EventID"unique ID of the event
    "Failure"failure object associated with the event
    "FailureType"failure type associated with the test
    "Outcome"outcome associated with the test
    "TestFileName"test file name associated with the event
    "TestObject"test object associated with the event
    "Title"name of the test report
  • TestReport sets $TestFileName to the name of the file from which a test is being run.
  • CreateNotebook["Testing"] opens a blank notebook set up to use the Testing Notebook framework.

Examples

open allclose all

Basic Examples  (2)

Run a list of TestObject:

Run tests from a notebook:

Extract information on tests that failed and present it in a TabView:

Merge the existing TestReportObject with a new one:

Scope  (1)

Basic Uses  (1)

Run a list of tests:

Properties of the TestReportObject:

List of individual test results:

Compare the expected and actual outputs of tests where those differed:

Options  (10)

HandlerFunctions  (1)

Use HandlerFunctions to log test events happening during the execution of TestReport:

Use the special key "UnhandledEvent" to log all events:

HandlerFunctionsKeys  (1)

Use HandlerFunctionsKeys to select only the keys you are interested in when using HandlerFunctions:

If a certain event cannot provide the key you are requesting, it will not be present in the Association:

MemoryConstraint  (2)

Apply a memory constraint to each test:

Options for a specific TestCreate override the options set at the TestReport level:

ProgressReporting  (1)

Long test runs will automatically display a progress report:

Use the option ProgressReporting to disable it:

In order to disable the progress reporting globally, you can simply override $ProgressReporting:

SameTest  (2)

Apply SameTest to all given tests:

Options for a specific TestCreate override the options set at the TestReport level:

TestEvaluationFunction  (1)

Print all tests in the notebook without evaluating them:

Print all tests in the notebook before evaluation and then evaluate them:

TimeConstraint  (2)

Apply a time constraint to all given tests:

Options for a specific TestCreate override the options set at the TestReport level:

Applications  (1)

Run a list of individual tests and test files at the same time:

Use TestReport again to merge an existing TestReportObject into a new one:

When merging multiple TestReportObject instances using TestReport, duplicated tests will be automatically deleted from the report:

Use the property "ResultsByTestFileName" to group together tests from the same file:

Properties & Relations  (2)

Use Throw and Catch in HandlerFunctions to stop the execution of the test run when a test fails:

You can also use Throw and Catch in TestEvaluationFunction:

Use Reap and Sow in HandlerFunctions to collect all event names:

Possible Issues  (2)

A TestReport with no tests is considered a success:

TestReport will execute and collect unique tests once:

In order to execute the same test multiple times, the test needs to be created multiple times:

Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).

Text

Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).

CMS

Wolfram Language. 2014. "TestReport." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2023. https://reference.wolfram.com/language/ref/TestReport.html.

APA

Wolfram Language. (2014). TestReport. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TestReport.html

BibTeX

@misc{reference.wolfram_2025_testreport, author="Wolfram Research", title="{TestReport}", year="2023", howpublished="\url{https://reference.wolfram.com/language/ref/TestReport.html}", note=[Accessed: 13-May-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_testreport, organization={Wolfram Research}, title={TestReport}, year={2023}, url={https://reference.wolfram.com/language/ref/TestReport.html}, note=[Accessed: 13-May-2025 ]}