Enable JavaScript to interact with content and submit forms on Wolfram websites. Learn how

TestReport

TestReport["file"]

gives a report of the results of the tests from a file.

TestReport[{test₁,test₂,…}]

gives a report of the results of the test_i.

TestReport[{report₁,report₂,…}]

gives a unified report by merging all test reports report_i.

Details and Options

TestReport returns a TestReportObject.
TestReport handles both plain text Wolfram Language files and notebooks that use the Testing Notebook framework.
TestReport only runs tests with outcome "NotEvaluated", skipping ones that have already been evaluated.
TestReport has the following options:

	HandlerFunctions	<\|\|>	how to handle events generated
	HandlerFunctionsKeys	Automatic	what parameters to supply to handler functions
	MemoryConstraint	Infinity	memory (in bytes) each test is allowed to use
	ProgressReporting	$ProgressReporting	whether to report the progress
	SameTest	SameQ	function to compare actual and expected outputs
	TestEvaluationFunction	TestEvaluate	function to evaluate created tests
	TimeConstraint	Infinity	time (in seconds) each test is allowed to use

TestReport[File["file"]] is also supported.
During the execution of TestReport, the following events can be generated:

	"FileStarted"	test file started
	"FileCompleted"	test file completed
	"ReportStarted"	test report started
	"ReportCompleted"	test report completed
	"RuntimeFailure"	runtime failure encountered
	"TestCreated"	test created
	"TestEvaluated"	test evaluated

With the specification HandlerFunctions-><|…,"event_i"->f_i,…|>, f_i[assoc] is evaluated whenever event_i is generated. The elements of assoc have keys specified by the setting for HandlerFunctionsKeys.
Possible keys specified by HandlerFunctionsKeys include:

	"EventName"	name of the event being handled
	"EventID"	unique ID of the event
	"Failure"	failure object associated with the event
	"FailureType"	failure type associated with the test
	"Outcome"	outcome associated with the test
	"TestFileName"	test file name associated with the event
	"TestObject"	test object associated with the event
	"Title"	name of the test report

TestReport sets $TestFileName to the name of the file from which a test is being run.
CreateNotebook["Testing"] opens a blank notebook set up to use the Testing Notebook framework.

Examples

open allclose all

Basic Examples (2)

Run a list of TestObject:

Run tests from a notebook:

Extract information on tests that failed and present it in a TabView:

Merge the existing TestReportObject with a new one:

Scope (1)

Basic Uses (1)

Run a list of tests:

Properties of the TestReportObject:

List of individual test results:

Compare the expected and actual outputs of tests where those differed:

Options (10)

HandlerFunctions (1)

Use HandlerFunctions to log test events happening during the execution of TestReport:

Use the special key "UnhandledEvent" to log all events:

HandlerFunctionsKeys (1)

Use HandlerFunctionsKeys to select only the keys you are interested in when using HandlerFunctions:

If a certain event cannot provide the key you are requesting, it will not be present in the Association:

MemoryConstraint (2)

Apply a memory constraint to each test:

Options for a specific TestCreate override the options set at the TestReport level:

ProgressReporting (1)

Long test runs will automatically display a progress report:

Use the option ProgressReporting to disable it:

In order to disable the progress reporting globally, you can simply override $ProgressReporting:

SameTest (2)

Apply SameTest to all given tests:

Options for a specific TestCreate override the options set at the TestReport level:

TestEvaluationFunction (1)

Print all tests in the notebook without evaluating them:

Print all tests in the notebook before evaluation and then evaluate them:

TimeConstraint (2)

Apply a time constraint to all given tests:

Options for a specific TestCreate override the options set at the TestReport level:

Applications (1)

Run a list of individual tests and test files at the same time:

Use TestReport again to merge an existing TestReportObject into a new one:

When merging multiple TestReportObject instances using TestReport, duplicated tests will be automatically deleted from the report:

Use the property "ResultsByTestFileName" to group together tests from the same file:

Properties & Relations (2)

Use Throw and Catch in HandlerFunctions to stop the execution of the test run when a test fails:

You can also use Throw and Catch in TestEvaluationFunction:

Use Reap and Sow in HandlerFunctions to collect all event names:

Possible Issues (2)

A TestReport with no tests is considered a success:

TestReport will execute and collect unique tests once:

In order to execute the same test multiple times, the test needs to be created multiple times:

Top