Wolfram Research, Inc.

1.12.5 Testing and Verification

Every version of Mathematica is subjected to a large amount of testing before it is released. The vast majority of this testing is done by an automated system that is written in Mathematica.

The automated system feeds millions of pieces of input to Mathematica, and checks that the output obtained from them is correct. Often there is some subtlety in doing such checking: one must account for different behavior of randomized algorithms and for such issues as differences in machine-precision arithmetic on different computers.

The test inputs used by the automated system are obtained in several ways:

For every Mathematica function, inputs are devised that exercise both common and extreme cases.

Inputs are devised to exercise each feature of the internal code.

All the examples in this book and in other books about Mathematica are used.

Standard numerical tables are optically scanned for test inputs.

Formulas from all standard mathematical tables are entered.

Exercises from textbooks are entered.

For pairs of functions such as Integrate and D or Factor and Expand, random expressions are generated and tested.

When tests are run, the automated testing system checks not only the results, but also side effects such as messages, as well as memory usage and speed.

There is also a special instrumented version of Mathematica which is set up to perform internal consistency tests. This version of Mathematica runs at a small fraction of the speed of the real Mathematica, but at every step it checks internal memory consistency, interruptibility, and so on.

The instrumented version of Mathematica also records which pieces of Mathematica source code have been accessed, allowing one to confirm that all of the various internal functions in Mathematica have been exercised by the tests given.

All standard Mathematica tests are routinely run on each version of Mathematica, on each different computer system. Depending on the speed of the computer system, these tests take a few days to a few weeks of computer time.

In addition, huge numbers of tests based on random inputs are run for the equivalent of many years of computer time on a sampling of different computer systems.

Even with all this testing, however, it is inevitable in a system as complex as Mathematica that errors will remain.

The standards of correctness for Mathematica are certainly much higher than for typical mathematical proofs. But just as long proofs will inevitably contain errors that go undetected for many years, so also a complex software system such as Mathematica will contain errors that go undetected even after millions of people have used it.

Nevertheless, particularly after all the testing that has been done on it, the probability that you will actually discover an error in Mathematica in the course of your work is extremely low.

Doubtless there will be times when Mathematica does things you do not expect. But you should realize that the probabilities are such that it is vastly more likely that there is something wrong with your input to Mathematica or your understanding of what is happening than with the internal code of the Mathematica system itself.

If you do believe that you have found a genuine error in Mathematica, then you should contact Wolfram Research at the addresses given in the front of this book so that the error can be corrected in future versions.