TestReport
✖
TestReport
Details and Options
- TestReport returns a TestReportObject.
- TestReport handles both plain text Wolfram Language files and notebooks that use the Testing Notebook framework.
- TestReport only runs tests with outcome "NotEvaluated", skipping ones that have already been evaluated.
- TestReport has the following options:
-
HandlerFunctions < > how to handle events generated HandlerFunctionsKeys Automatic what parameters to supply to handler functions MemoryConstraint Infinity memory (in bytes) each test is allowed to use ProgressReporting $ProgressReporting whether to report the progress SameTest SameQ function to compare actual and expected outputs TestEvaluationFunction TestEvaluate function to evaluate created tests TimeConstraint Infinity time (in seconds) each test is allowed to use - TestReport[File["file"]] is also supported.
- During the execution of TestReport, the following events can be generated:
-
"FileStarted" test file started "FileCompleted" test file completed "ReportStarted" test report started "ReportCompleted" test report completed "RuntimeFailure" runtime failure encountered "TestCreated" test created "TestEvaluated" test evaluated - With the specification HandlerFunctions-><…,"eventi"->fi,… >, fi[assoc] is evaluated whenever eventi is generated. The elements of assoc have keys specified by the setting for HandlerFunctionsKeys.
- Possible keys specified by HandlerFunctionsKeys include:
-
"EventName" name of the event being handled "EventID" unique ID of the event "Failure" failure object associated with the event "FailureType" failure type associated with the test "Outcome" outcome associated with the test "TestFileName" test file name associated with the event "TestObject" test object associated with the event "Title" name of the test report - TestReport sets $TestFileName to the name of the file from which a test is being run.
- CreateNotebook["Testing"] opens a blank notebook set up to use the Testing Notebook framework.
Examples
open allclose allBasic Examples (2)Summary of the most common use cases
Run a list of TestObject:
https://wolfram.com/xid/0b8c14vkou-cn9xhl
https://wolfram.com/xid/0b8c14vkou-f7el2z
Extract information on tests that failed and present it in a TabView:
https://wolfram.com/xid/0b8c14vkou-o4thxf
Merge the existing TestReportObject with a new one:
https://wolfram.com/xid/0b8c14vkou-ojktty
Scope (1)Survey of the scope of standard use cases
Basic Uses (1)
https://wolfram.com/xid/0b8c14vkou-wa4gl3
Properties of the TestReportObject:
https://wolfram.com/xid/0b8c14vkou-076129
List of individual test results:
https://wolfram.com/xid/0b8c14vkou-zp2rk7
Compare the expected and actual outputs of tests where those differed:
https://wolfram.com/xid/0b8c14vkou-ukdmvl
Options (10)Common values & functionality for each option
HandlerFunctions (1)
Use HandlerFunctions to log test events happening during the execution of TestReport:
https://wolfram.com/xid/0b8c14vkou-7aai8i
Use the special key "UnhandledEvent" to log all events:
https://wolfram.com/xid/0b8c14vkou-fre54h
https://wolfram.com/xid/0b8c14vkou-2rnvj1
HandlerFunctionsKeys (1)
Use HandlerFunctionsKeys to select only the keys you are interested in when using HandlerFunctions:
https://wolfram.com/xid/0b8c14vkou-ym6cfc
If a certain event cannot provide the key you are requesting, it will not be present in the Association:
https://wolfram.com/xid/0b8c14vkou-glhkf1
MemoryConstraint (2)
Apply a memory constraint to each test:
https://wolfram.com/xid/0b8c14vkou-7n3n5r
https://wolfram.com/xid/0b8c14vkou-e171cf
https://wolfram.com/xid/0b8c14vkou-fm0ujn
Options for a specific TestCreate override the options set at the TestReport level:
https://wolfram.com/xid/0b8c14vkou-qedemh
ProgressReporting (1)
Long test runs will automatically display a progress report:
https://wolfram.com/xid/0b8c14vkou-lzargd
Use the option ProgressReporting to disable it:
https://wolfram.com/xid/0b8c14vkou-2e2560
In order to disable the progress reporting globally, you can simply override $ProgressReporting:
https://wolfram.com/xid/0b8c14vkou-4txmop
SameTest (2)
Apply SameTest to all given tests:
https://wolfram.com/xid/0b8c14vkou-r6b0is
https://wolfram.com/xid/0b8c14vkou-hbnvo7
Options for a specific TestCreate override the options set at the TestReport level:
https://wolfram.com/xid/0b8c14vkou-iljv40
TestEvaluationFunction (1)
TimeConstraint (2)
Apply a time constraint to all given tests:
https://wolfram.com/xid/0b8c14vkou-rbhtbx
https://wolfram.com/xid/0b8c14vkou-0duiq7
https://wolfram.com/xid/0b8c14vkou-v2eg79
https://wolfram.com/xid/0b8c14vkou-nodcwr
Options for a specific TestCreate override the options set at the TestReport level:
https://wolfram.com/xid/0b8c14vkou-ikxub1
Applications (1)Sample problems that can be solved with this function
Run a list of individual tests and test files at the same time:
https://wolfram.com/xid/0b8c14vkou-9fihr7
Use TestReport again to merge an existing TestReportObject into a new one:
https://wolfram.com/xid/0b8c14vkou-t4nhjg
When merging multiple TestReportObject instances using TestReport, duplicated tests will be automatically deleted from the report:
https://wolfram.com/xid/0b8c14vkou-4ph0hf
https://wolfram.com/xid/0b8c14vkou-8pdrby
https://wolfram.com/xid/0b8c14vkou-vowscl
Use the property "ResultsByTestFileName" to group together tests from the same file:
https://wolfram.com/xid/0b8c14vkou-1uwd42
Properties & Relations (2)Properties of the function, and connections to other functions
Use Throw and Catch in HandlerFunctions to stop the execution of the test run when a test fails:
https://wolfram.com/xid/0b8c14vkou-d8nfc6
You can also use Throw and Catch in TestEvaluationFunction:
https://wolfram.com/xid/0b8c14vkou-edp8nq
Use Reap and Sow in HandlerFunctions to collect all event names:
https://wolfram.com/xid/0b8c14vkou-keajtv
Possible Issues (2)Common pitfalls and unexpected behavior
A TestReport with no tests is considered a success:
https://wolfram.com/xid/0b8c14vkou-td87sn
https://wolfram.com/xid/0b8c14vkou-ir8ul7
TestReport will execute and collect unique tests once:
https://wolfram.com/xid/0b8c14vkou-txbvqk
In order to execute the same test multiple times, the test needs to be created multiple times:
https://wolfram.com/xid/0b8c14vkou-e6toab
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).
Text
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).
Wolfram Research (2014), TestReport, Wolfram Language function, https://reference.wolfram.com/language/ref/TestReport.html (updated 2023).
CMS
Wolfram Language. 2014. "TestReport." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2023. https://reference.wolfram.com/language/ref/TestReport.html.
Wolfram Language. 2014. "TestReport." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2023. https://reference.wolfram.com/language/ref/TestReport.html.
APA
Wolfram Language. (2014). TestReport. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TestReport.html
Wolfram Language. (2014). TestReport. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TestReport.html
BibTeX
@misc{reference.wolfram_2024_testreport, author="Wolfram Research", title="{TestReport}", year="2023", howpublished="\url{https://reference.wolfram.com/language/ref/TestReport.html}", note=[Accessed: 19-December-2024
]}
BibLaTeX
@online{reference.wolfram_2024_testreport, organization={Wolfram Research}, title={TestReport}, year={2023}, url={https://reference.wolfram.com/language/ref/TestReport.html}, note=[Accessed: 19-December-2024
]}