Testing and Verification

Every version of the Wolfram System is subjected to a large amount of testing before it is released. The vast majority of this testing is done by an automated system that is written in the Wolfram Language.

The automated system feeds millions of pieces of input to the Wolfram System, and checks that the output obtained from them is correct. Often there is some subtlety in doing such checking: one must account for different behavior of randomized algorithms and for such issues as differences in machineprecision arithmetic on different computers.

The test inputs used by the automated system are obtained in several ways:

  • For every Wolfram Language function, inputs are devised that exercise both common and extreme cases.
  • Inputs are devised to exercise each feature of the internal code.
  • All the examples in the Wolfram Language's documentation system are used, as well as those from many books about the Wolfram System.
  • Tests are constructed from mathematical benchmarks and test sets from numerous websites.
  • Standard numerical tables are optically scanned for test inputs.
  • Formulas from standard mathematical tables are entered.
  • Exercises from textbooks are entered.
  • For pairs of functions such as Integrate and D or Factor and Expand, random expressions are generated and tested.

When tests are run, the automated testing system checks not only the results, but also side effects such as messages, as well as memory usage and speed.

There is also a special instrumented version of the Wolfram System which is set up to perform internal consistency tests. This version of the Wolfram System runs at a small fraction of the speed of the real Wolfram System, but at every step it checks internal memory consistency, interruptibility, and so on.

The instrumented version of the Wolfram System also records which pieces of the Wolfram Language source code have been accessed, allowing one to confirm that all of the various internal functions in the Wolfram System have been exercised by the tests given.

All standard Wolfram System tests are routinely run on current versions of the Wolfram System, on each different computer system. Depending on the speed of the computer system, these tests take from a few hours to a few days of computer time.

Even with all this testing, however, it is inevitable in a system as complex as the Wolfram System that errors will remain.

The standards of correctness for the Wolfram System are certainly much higher than for typical mathematical proofs. But just as long proofs will inevitably contain errors that go undetected for many years, so also a complex software system such as the Wolfram System will contain errors that go undetected even after millions of people have used it.

Nevertheless, particularly after all the testing that has been done on it, the probability that you will actually discover an error in the Wolfram System in the course of your work is extremely low.

Doubtless there will be times when the Wolfram System does things you do not expect. But you should realize that the probabilities are such that it is vastly more likely that there is something wrong with your input to the Wolfram System or your understanding of what is happening than with the internal code of the Wolfram System itself.

If you do believe that you have found a genuine error in the Wolfram System, then you should contact Wolfram Research Technical Support, so that the error can be corrected in future versions.