Legacy Documentation

Parallel Computing Toolkit (2000)

This is documentation for an obsolete product.
Current products and services
Previous section-----Next section

Parallel Evaluation

Sending Commands to Remote Kernels

Recall that connections to remote kernels, as opened by LaunchSlave, are represented as kernel objects. See the chapter Starting Remote Kernels for details. The commands in this section take such kernel objects as arguments and send or receive Mathematica expressions to remote kernels using these links. In the following, the variable link is a kernel returned by LinkLaunch or LinkConnect.

Low-Level Send and Receive

Send[link, cmd]sends cmd for evaluation to the remote kernel connected to link; returns link
Send[{links...}, cmd]sends cmd for evaluation to all remote kernels listed
ReceiveIfReady[link]returns a result waiting on the given link; returns $NotReady if no result is waiting
ReceiveIfReady[link, h]wraps the result in h[...] before returning it
Receive[link]waits for a result on the given link and returns it
Receive[{links...}]waits for a result on each of the links and returns them in a list
Receive[links, h]wraps the results in h[...] before returning them
RemoteEvaluate[cmd, link]sends cmd for evaluation to the remote kernel connected to link, then waits for the result and returns it
RemoteEvaluate[cmd, {links...}]sends cmd for evaluation to the remote kernels given, then waits for the results and returns them
RemoteEvaluate[cmd]sends cmd for evaluation to all remote kernels and returns the list of results. Equivalent to RemoteEvaluate[cmd, $Slaves]

Sending and receiving commands to and from remote kernels.

Send has the attribute HoldRest so that the given command is not evaluated before it is sent to the remote kernel. RemoteEvaluate has the attribute HoldFirst.
RemoteEvaluate[cmd, link] is equivalent to the combination Send[link, cmd]; Receive[link].
With ReceiveIfReady[link], you can poll several open links for results.
You cannot use RemoteEvaluate while a concurrent computation involving Queue or Wait is in progress. See the chapter Concurrency: Managing Parallel Processes for details.

Values of Variables

Values of variables defined on the local master kernel are usually not available to remote kernels. If a command you send for evaluation refers to a variable, it usually will not work as expected. The following piece of code will return False because the symbol a will most likely not have any value at all on the remote kernel.
A convenient way to insert variable values into unevaluated commands is to use With, as demonstrated in the following command.
The symbol a is replaced by 2, then the expression 2=2 is sent to the remote kernel where it evaluates to True.
If you need variable values and definitions carried over to the remote kernels, use ExportEnvironment or the package Parallel`VirtualShared`, which is part of Parallel Computing Toolkit.
Iterators, such as Table and Do, work in the same way with respect to the iterator variable. Therefore, a statement like the following will not do the expected thing. The variable i will not have a value on the remote kernel.
You can use the following command to accomplish the intended iteration on the remote kernel. This substitutes the value of i into the argument of RemoteEvaluate.
Pattern variables, constants, and pure function variables will work as expected on the remote kernel. Each of the following three examples will produce the expected result.

Parallel Evaluation of Expressions

Dispatching Evaluations to Remote Kernels

ParallelDispatch[h[e1,e2,...,en],{k1,k2,...,km}]
evaluateseion kernel ki and returns h[ r1, r2, ... rn],whereriis the result of evaluatingei. The default list of kernels is $Slaves
ParallelEvaluate[h[e1,e2,...,en],f,comb]
evaluates f[h[e1,e2,...,en]] in parallel by distributing chunks f[h[ei,ei+1,...,ei+k]] to all kernels and combining the results with comb[].
ParallelEvaluate[h[e1,e2,...,en],f]
the default combiner comb is h, if h has attribute Flat, and Join otherwise
ParallelEvaluate[h[e1,e2,...,en]]
the default function f is Identity

Basic parallel dispatch of evaluations.

In ParallelDispatch[h[e1,e2,...,en],{k1,k2,...,km}], the number m of kernels must be at least as large as the number n of expressions. ParallelDispatch has the attribute HoldFirst so that h[e1,e2,...,en] is not evaluated on the master kernel before the parallelization.
ParallelDispatch[{e1,e2,...,en},{k1,k2,...,kn}] is equivalent to Receive[Inner[Send,{k1,k2,...,kn},{e1,e2,...,en}]].
ParallelEvaluate[h[e1,e2,...,en],f,comb] breaks h[e1,e2,...,en] into as many pieces h[ei,ei+1,...,ei+k] as there are remote kernels, evaluates f[h[ei,ei+1,...,ei+k]] in parallel (using ParallelDispatch[]), then combines the results ri using comb[r1,r2,...,rm]. ParallelEvaluate has the attribute HoldFirst so that h[e1,e2,...,en] is not evaluated on the master kernel before the parallelization.
The size of the pieces of the input expression is chosen to be proportional to the remote processor speed estimates for optimal load balancing.

ParallelEvaluate

ParallelEvaluate is a general and powerful command with default values for its arguments that are suitable for evaluating elements of containers such as lists and associative functions.

Evaluating List-like Containers

If the result of applying the function f to a list is again a list, ParallelEvaluate[{e1,e2,...,en},f] simply applies f to pieces of the input list and joins the partial results together.
In[2]:=
Out[2]=
The result is the same as that of Prime[{1,2,3,4,5,6,7,8,9}], but the computation is done in parallel.
The default function is Identity, therefore, ParallelEvaluate[{e1,e2,...,en}] simply evaluates the elements ei in parallel.
In[3]:=
Out[3]=
If the result of applying the function f to a list is not a list, a custom combiner has to be chosen.
The function Function[li,Count[li,_?OddQ] counts the number of odd elements in a list. To find the total number of odd elements, add the partial results together.
In[4]:=
Out[4]=

Evaluating Associative Operations

If the operation h in h[e1,e2,...,en] is associative (has attribute Flat), the identity
holds; with the default combiner being h itself, the operation is parallelized in a natural way. Here all numbers are added in parallel.
In[5]:=
Out[5]=
In[6]:=
Out[6]=

Parallel Mapping and Iterators

The commands in this section are fundamental to parallel programming in Mathematica.
ParallelEvaluate[h[e1,e2,...]]
evaluates the elements ei in parallel and returns h[ r1, r2, ...], where ri is the result of evaluating ei
ParallelMap[ f, h[ e1, e2, ...]]evaluates h[ f[e1], f[e2] ,...] in parallel.
ParallelTable[expr,{i,i0,i1,di},{j,j0,j1,dj},...]
builds Table[expr,{i,i0,i1,di},{j,j0,j1,dj},...] in parallel; parallelization occurs along the first (outermost) iterator {i, i0, i1,di }
ParallelSum[...], ParallelProduct[...]
computes sums and products in parallel

Parallel evaluation, mapping, and tables.

ParallelEvaluate has attribute HoldFirst, so that its argument is not evaluated on the local kernel. After receiving the results, ri, the expression h[r1,r2,...] is further evaluated normally on the local kernel. The symbol List is often used as the head h.
is a parallel version of
evaluating the individual f[ei]in parallel rather than sequentially.
ParallelEvaluate and related commands use all available remote kernels on the list $Slaves.

Side Effects

Unless you use shared variables, the parallel evaluations performed are completely independent and cannot influence each other. Furthermore, any side effects, such as assignments to variables, that happen as part of evaluations will be lost. The only effect of a parallel evaluation is that its result is returned at the end.

Examples

First, load the package and then start several remote kernels.
In[1]:=
The sine function is applied to the given arguments. Each computation takes place on a remote kernel.
In[2]:=
Out[2]=
This particular computation is almost certainly too trivial to benefit from parallel evaluation. The overhead required to send the expressions Sin[0], Sin[], and so on to the remote kernels and to collect the results will be larger than the gain obtained from parallelization.
Factoring integers of the form 111...1 takes more time, so this computation can benefit from parallelization.
In[3]:=
Out[3]=
Alternatively, you can use ParallelTable. Here a list of the number of factors in is generated.
In[4]:=
Out[4]//TableForm=

Automatic Parallelization

ParallelEvaluate[cmd[list,arguments...]] recognizes if cmd is a Mathematica function that operates on a list or other long expression in a way that can be easily parallelized and performs the parallelization automatically. You do not need to figure out suitable f and comb for ParallelEvaluate[list,f,comb]. The list of commands that can be parallelized is kept in the variable $ParallelCommands.
In[5]:=
Out[5]=
In[6]:=
Out[6]=
In[7]:=
Out[7]=
In[8]:=
Out[8]=
Not all uses of these commands can be parallelized. A message is generated and the evaluation is performed sequentially on the master kernel if necessary.
In[9]:=
Out[9]=