WOLFRAM

Wolfram Language & System Documentation Center
Wolfram Language Home Page »

TestReport

✖
TestReport["file"]

gives a report of the results of the tests from a file.

✖
TestReport[{test1,test2,…}]

gives a report of the results of the testi.

✖
TestReport[{report1,report2,…}]

gives a unified report by merging all test reports reporti.

Details and Options

  • TestReport returns a TestReportObject.
  • TestReport handles both plain text Wolfram Language files and notebooks that use the Testing Notebook framework.
  • TestReport only runs tests with outcome "NotEvaluated", skipping ones that have already been evaluated.
  • TestReport has the following options:
  • HandlerFunctions <||>how to handle events generated
    HandlerFunctionsKeys Automaticwhat parameters to supply to handler functions
    MemoryConstraint Infinitymemory (in bytes) each test is allowed to use
    ProgressReporting $ProgressReportingwhether to report the progress
    SameTest SameQfunction to compare actual and expected outputs
    TestEvaluationFunction TestEvaluatefunction to evaluate created tests
    TimeConstraint Infinitytime (in seconds) each test is allowed to use
  • TestReport[File["file"]] is also supported.
  • During the execution of TestReport, the following events can be generated:
  • "FileStarted"test file started
    "FileCompleted"test file completed
    "ReportStarted"test report started
    "ReportCompleted"test report completed
    "RuntimeFailure"runtime failure encountered
    "TestCreated"test created
    "TestEvaluated"test evaluated
  • With the specification HandlerFunctions-><|…,"eventi"->fi,…|>, fi[assoc] is evaluated whenever eventi is generated. The elements of assoc have keys specified by the setting for HandlerFunctionsKeys.
  • Possible keys specified by HandlerFunctionsKeys include:
  • "EventName"name of the event being handled
    "EventID"unique ID of the event
    "Failure"failure object associated with the event
    "FailureType"failure type associated with the test
    "Outcome"outcome associated with the test
    "TestFileName"test file name associated with the event
    "TestObject"test object associated with the event
    "Title"name of the test report
  • TestReport sets $TestFileName to the name of the file from which a test is being run.
  • CreateNotebook["Testing"] opens a blank notebook set up to use the Testing Notebook framework.

Examples

open allclose all

Basic Examples  (2)Summary of the most common use cases

Run a list of TestObject:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-cn9xhl
Out[1]=1

Run tests from a notebook:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-f7el2z
Out[1]=1

Extract information on tests that failed and present it in a TabView:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-o4thxf
Out[2]=2

Merge the existing TestReportObject with a new one:

In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-ojktty
Out[3]=3

Scope  (1)Survey of the scope of standard use cases

Basic Uses  (1)

Run a list of tests:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-wa4gl3
Out[1]=1

Properties of the TestReportObject:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-076129
Out[2]=2

List of individual test results:

In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-zp2rk7
Out[3]=3

Compare the expected and actual outputs of tests where those differed:

In[4]:=4
✖
https://wolfram.com/xid/0b8c14vkou-ukdmvl
Out[4]=4

Options  (10)Common values & functionality for each option

HandlerFunctions  (1)

Use HandlerFunctions to log test events happening during the execution of TestReport:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-7aai8i
Out[1]=1

Use the special key "UnhandledEvent" to log all events:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-fre54h
Out[2]=2
In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-2rnvj1
Out[3]=3

HandlerFunctionsKeys  (1)

Use HandlerFunctionsKeys to select only the keys you are interested in when using HandlerFunctions:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-ym6cfc
Out[1]=1

If a certain event cannot provide the key you are requesting, it will not be present in the Association:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-glhkf1
Out[2]=2

MemoryConstraint  (2)

Apply a memory constraint to each test:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-7n3n5r
Out[1]=1
In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-e171cf
Out[2]=2
In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-fm0ujn
Out[3]=3

Options for a specific TestCreate override the options set at the TestReport level:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-qedemh
Out[1]=1

ProgressReporting  (1)

Long test runs will automatically display a progress report:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-lzargd
Out[1]=1

Use the option ProgressReporting to disable it:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-2e2560
Out[2]=2

In order to disable the progress reporting globally, you can simply override $ProgressReporting:

In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-4txmop
Out[3]=3

SameTest  (2)

Apply SameTest to all given tests:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-r6b0is
Out[1]=1
In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-hbnvo7
Out[2]=2

Options for a specific TestCreate override the options set at the TestReport level:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-iljv40
Out[1]=1

TestEvaluationFunction  (1)

Print all tests in the notebook without evaluating them:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-gscluc
Out[1]=1

Print all tests in the notebook before evaluation and then evaluate them:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-nwk0n9
Out[2]=2

TimeConstraint  (2)

Apply a time constraint to all given tests:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-rbhtbx
Out[1]=1
In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-0duiq7
Out[2]=2
In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-v2eg79
Out[3]=3
In[4]:=4
✖
https://wolfram.com/xid/0b8c14vkou-nodcwr
Out[4]=4

Options for a specific TestCreate override the options set at the TestReport level:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-ikxub1
Out[1]=1

Applications  (1)Sample problems that can be solved with this function

Run a list of individual tests and test files at the same time:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-9fihr7
Out[1]=1

Use TestReport again to merge an existing TestReportObject into a new one:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-t4nhjg
Out[2]=2

When merging multiple TestReportObject instances using TestReport, duplicated tests will be automatically deleted from the report:

In[3]:=3
✖
https://wolfram.com/xid/0b8c14vkou-4ph0hf
Out[3]=3
In[4]:=4
✖
https://wolfram.com/xid/0b8c14vkou-8pdrby
Out[4]=4
In[5]:=5
✖
https://wolfram.com/xid/0b8c14vkou-vowscl
Out[5]=5

Use the property "ResultsByTestFileName" to group together tests from the same file:

In[6]:=6
✖
https://wolfram.com/xid/0b8c14vkou-1uwd42
Out[6]=6

Properties & Relations  (2)Properties of the function, and connections to other functions

Use Throw and Catch in HandlerFunctions to stop the execution of the test run when a test fails:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-d8nfc6
Out[1]=1

You can also use Throw and Catch in TestEvaluationFunction:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-edp8nq
Out[2]=2

Use Reap and Sow in HandlerFunctions to collect all event names:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-keajtv
Out[1]=1

Possible Issues  (2)Common pitfalls and unexpected behavior

A TestReport with no tests is considered a success:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-td87sn
Out[1]=1
In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-ir8ul7
Out[2]=2

TestReport will execute and collect unique tests once:

In[1]:=1
✖
https://wolfram.com/xid/0b8c14vkou-txbvqk
Out[1]=1

In order to execute the same test multiple times, the test needs to be created multiple times:

In[2]:=2
✖
https://wolfram.com/xid/0b8c14vkou-e6toab
Out[2]=2
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).
✖
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).

Text

Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).

✖
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).

CMS

Wolfram Language. 2014. "TestReport." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2023. https://reference.wolfram.com/language/ref/TestReport.html.

✖
Wolfram Language. 2014. "TestReport." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2023. https://reference.wolfram.com/language/ref/TestReport.html.

APA

Wolfram Language. (2014). TestReport. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TestReport.html

✖
Wolfram Language. (2014). TestReport. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TestReport.html

BibTeX

@misc{reference.wolfram_2025_testreport, author="Wolfram Research", title="{TestReport}", year="2023", howpublished="\url{https://reference.wolfram.com/language/ref/TestReport.html}", note=[Accessed: 26-March-2025 ]}

✖
@misc{reference.wolfram_2025_testreport, author="Wolfram Research", title="{TestReport}", year="2023", howpublished="\url{https://reference.wolfram.com/language/ref/TestReport.html}", note=[Accessed: 26-March-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_testreport, organization={Wolfram Research}, title={TestReport}, year={2023}, url={https://reference.wolfram.com/language/ref/TestReport.html}, note=[Accessed: 26-March-2025 ]}

✖
@online{reference.wolfram_2025_testreport, organization={Wolfram Research}, title={TestReport}, year={2023}, url={https://reference.wolfram.com/language/ref/TestReport.html}, note=[Accessed: 26-March-2025 ]}