Configuring Lightweight Grid Kernels for Parallel Computation

Specifying Machines with Lightweight Grid ManagerDiscovering Lightweight Grid Managers

The Wolfram Lightweight Grid Manager makes Wolfram Engines, or kernels, available over the network to use as parallel subkernels. This note discusses how to configure the parallel master kernel to use such Lightweight Grid kernels for parallel computation.

In Version 13.1 of the Wolfram System, the way these Lightweight Grid kernels are configured has changed. This document show how to discover, configure and use Lightweight Grid kernels.

There are two ways to configure kernels provided by Lightweight Grid Manager:

Specifying Machines with Lightweight Grid Manager

$DefaultKernelslist of kernel configurations to launch by default
KernelConfigurationdetailed configuration for a parallel subkernel
LaunchKernelslaunch configured kernels

Configuration of kernels for parallel computing.

To use kernels from a machine, specify an lwg: URL, including the desired number of kernels:
When no kernels are running, these default kernels will be launched:
Use the running kernels for parallel computations:

Once you have a desired default kernel configuration, you can make it persistent, by setting an initialization value. Such a value is stored in a file in your home directory and causes the variable $DefaultKernels to be initialized in every new session.

Assign a list of kernel specifications as an initialization value for $DefaultKernels:
After quitting and restarting the notebook's kernel, $DefaultKernels is set to the given initialization value:

If you ever want to revert to the default, you should remove any initialization values given.

Remove any initialization value for $DefaultKernels:
After quitting and restarting the notebook's kernel, $DefaultKernels reverts to its default:

Each Lightweight Grid Manager has a management interface, a web URL where you can monitor and configure the manager. You can directly use these management URLs as kernel configurations, in place of the lwg: URLs discussed above.

Specify a full Lightweight Grid Manager URL:

Because the http: scheme does not uniquely identify Lightweight Grid Manager, not every http: URL is recognized. The port 3737 must be given, and the path component must not be empty. Alternatively, you can specify the lwg: scheme in place of http: if you have a nondefault manager configuration.

Discovering Lightweight Grid Managers

Lightweight Grid Managers advertise themselves (much like printers do) in the local network. Using the LightweightGrid client package, you can obtain kernel configurations from these discovery results. First, find any servers on the local or remote network, then ask each one in turn for the services it offers and use these services as kernel configurations.

Load the client package:

The discovery protocol works only on the local network. Remote Lightweight Grid Managers perform discovery in their own networks, and you can obtain these results by querying them.

List discovered Lightweight Grid Managers on the local network:
Or list discovered Lightweight Grid Managers in the vicinity of a known remote server:

Each agent may offer several services, for example, different versions of the Wolfram Engine.

Find the services of all discovered managers:

You can list properties of these services to help in deciding which ones to use.

Tabulate service properties:
Choose desired services:
Turn the services into kernel configurations:
Launch kernels on the desired services:

Discovered services cannot be used directly as initialization values. The recommended method is to extract the agent URLs and manually set up the desired list of $DefaultKernels.

Extract the "Agent" property:
Optionally add the number of desired kernels for each agent:
Use the agent URLs for your $DefaultKernels:
Test your setting:
Optionally, make your setting persistent:

If you ever want to revert to the default, you should remove any initialization values given.

Remove any initialization value for $DefaultKernels: