PARALLEL PACKAGE TUTORIAL

Introduction

Parallel Computing with Mathematica

Parallel computing in Mathematica is based on launching and controlling multiple Mathematica kernel (worker) processes from within a single master Mathematica, providing a distributed-memory environment for parallel programming. Every copy of Mathematica comes with all the components and tools to run and create parallel applications.

The parallel computing features are written almost entirely in the Mathematica language and are therefore machine independent. They have been tested on Unix, Linux, Windows, and Macintosh platforms and are well suited to working locally on a multicore machine, in a grid of machines, or in a heterogeneous network. All client and application code is distributed, so no common file system is necessary.

1.gif

Parallel computing is now provided as a standard part of Mathematica, thereby facilitating the use of parallel programming techniques. These can benefit from many of the advances in technology, for example, multicore machines, usage of specialized hardware for parallel computation, and fast network technology that helps with running a grid of connected computers.

To perform computations in parallel, you need to be able to perform the following tasks:

  • start processes and wait for processes to finish
  • schedule processes on available processors
    • exchange data between processes and synchronize access to common resources

    In the Mathematica environment, the term processor refers to a running Mathematica kernel, whereas a process is an expression to be evaluated.

MathLink

Underlying parallel computation in Mathematica is MathLink. This is a general and flexible interface for programs to communicate with Mathematica. It is a fundamental component of Mathematica and is used extensively in Mathematica itself. One common mode of operation is for different components of Mathematica to communicate using MathLink, which is exactly how parallel computation uses MathLink.

MathLink provides some core features, it is platform and architecture independent, it works both locally and across the network, it can transmit anything that Mathematica can represent, and it provides an interface for controlling Mathematica. These are all features that are key to parallel computation, allowing local and network communication between both homogeneous and heterogeneous networks, and allowing transmission of both data and programs.

Connection Methods

Mathematica can run parallel workers in a number of different ways, locally on the same machine or remotely on a network. In addition, the network might be a homogeneous grid controlled by some dedicated management application, or it might be a heterogeneous grid. To run in these situations, Mathematica needs to launch parallel workers and then communicate with them.

To carry out parallel computation in these different situations, Mathematica provides a number of different connection methods, each of which is specialized to a particular style of usage. Some of the main connection methods are described here.

Local Kernels

The local kernels connection method is used to run parallel workers on the same computer as the master Mathematica. It is suitable for a multicore environment and is the easiest way to get up and running with parallel computation.

Whenever your processor has more than one core, you can use local kernels for parallel computations.
In[25]:=
Click for copyable input
Out[25]=

The Lightweight Grid

The Lightweight Grid connection method is used to run parallel workers on different computers from the master Mathematica. It uses Wolfram Lightweight Grid technology to launch Mathematica on the remote machines. It is suitable for a heterogeneous network and where there is no management technology.

Cluster Integration

The Cluster Integration connection method is used to run parallel workers on different computers from the master Mathematica. It integrates with a large number of third-party cluster management technologies.

Remote Kernels

The remote kernels connection method is used to run parallel workers on different computers from the master Mathematica. It relies on using a remote shell invocation technology for launching and is typically harder to configure and maintain.

Features

The main features of parallel computing in Mathematica are:

  • distributed memory, master/slave parallelism
  • written in Mathematica
  • machine independent
  • MathLink communication with remote kernels
  • exchange of symbolic expressions and programs with remote kernels, not only numbers and arrays
  • heterogeneous network, multiprocessor machines, LAN and WAN
  • virtual process scheduling or explicit process distribution to available processors
  • virtual shared memory, synchronization, locking
  • latency hiding
  • parallel functional programming and automatic parallelization support
  • failure recovery, automatic reassignment of stranded processes on failed remote computers

Lower-Level Features

Additional parallel programming features are available in the Parallel`Developer` context. They allow you to get and set properties of parallel kernels and implement your own scheduler for concurrency.

The lower-level features are best made available by adding their context to your $ContextPath.
In[13]:=
Click for copyable input

For a single use, it is sufficient to refer to developer functions with their full name, such as Parallel`Developer`ClearKernels[].

New to Mathematica? Find your learning path »
Have a question? Ask support »