Introduction

Parallel Computing with the Wolfram Language

Parallel computing in the Wolfram Language is based on launching and controlling multiple Wolfram Language kernel (worker) processes from within a single master Wolfram Language, providing a distributed-memory environment for parallel programming. Every copy of the Wolfram System comes with all the components and tools to run and create parallel applications.

The parallel computing features are written almost entirely in the Wolfram Language and are therefore machine independent. They have been tested on Unix, Linux, Windows, and Macintosh platforms and are well suited to working locally on a multicore machine, in a grid of machines, or in a heterogeneous network. All client and application code is distributed, so no common file system is necessary.

Parallel computing is now provided as a standard part of the Wolfram Language, thereby facilitating the use of parallel programming techniques. These can benefit from many of the advances in technology, for example, multicore machines, usage of specialized hardware for parallel computation, and fast network technology that helps with running a grid of connected computers.

To perform computations in parallel, you need to be able to perform the following tasks:

• start processes and wait for processes to finish

• schedule processes on available processors

• exchange data between processes and synchronize access to common resources

In the Wolfram Language environment, the term processor refers to a running Wolfram Language kernel, whereas a process is an expression to be evaluated.

WSTP

Underlying parallel computation in the Wolfram Language is the Wolfram Symbolic Transfer Protocol (WSTP). This is a general and flexible interface for programs to communicate with the Wolfram Language. It is a fundamental component of the Wolfram Language and is used extensively in the Wolfram Language itself. One common mode of operation is for different components of the Wolfram Language to communicate using WSTP, which is exactly how parallel computation uses WSTP.

WSTP provides some core features, it is platform and architecture independent, it works both locally and across the network, it can transmit anything that the Wolfram Language can represent, and it provides an interface for controlling the Wolfram Language. These are all features that are key to parallel computation, allowing local and network communication between both homogeneous and heterogeneous networks, and allowing transmission of both data and programs.

Connection Methods

The Wolfram Language can run parallel workers in a number of different ways, locally on the same machine or remotely on a network. In addition, the network might be a homogeneous grid controlled by some dedicated management application, or it might be a heterogeneous grid. To run in these situations, the Wolfram Language needs to launch parallel workers and then communicate with them.

To carry out parallel computation in these different situations, the Wolfram Language provides a number of different connection methods, each of which is specialized to a particular style of usage. Some of the main connection methods are described here.

Local Kernels

The local kernels connection method is used to run parallel workers on the same computer as the master Wolfram Language. It is suitable for a multicore environment and is the easiest way to get up and running with parallel computation.

The Lightweight Grid

The Lightweight Grid connection method is used to run parallel workers on different computers from the master Wolfram Language. It uses Wolfram Lightweight Grid technology to launch the Wolfram Language on the remote machines. It is suitable for a heterogeneous network and where there is no management technology.

Cluster Integration

The Cluster Integration connection method is used to run parallel workers on different computers from the master Wolfram Language. It integrates with a large number of third-party cluster management technologies.

Remote Kernels

The remote kernels connection method is used to run parallel workers on different computers from the master Wolfram Language. It relies on using a remote shell invocation technology for launching and is typically harder to configure and maintain.

Features

The main features of parallel computing in the Wolfram Language are:

• distributed memory, master/slave parallelism

• written in the Wolfram Language

• machine independent

• WSTP communication with remote kernels

• exchange of symbolic expressions and programs with remote kernels, not only numbers and arrays

• heterogeneous network, multiprocessor machines, LAN and WAN

• virtual process scheduling or explicit process distribution to available processors

• virtual shared memory, synchronization, locking

• latency hiding

• parallel functional programming and automatic parallelization support

• failure recovery, automatic reassignment of stranded processes on failed remote computers

Lower-Level Features

Additional parallel programming features are available in the Parallel`Developer` context. They allow you to get and set properties of parallel kernels and implement your own scheduler for concurrency.

For a single use, it is sufficient to refer to developer functions with their full name, such as Parallel`Developer`ClearKernels[].