WOLFRAM

Wolfram Language & System Documentation Center
Wolfram Language Home Page »

"AWSBatch" (Batch Computation Provider)

[Experimental]

Details

AWS Batch is a container-based batch computation service that schedules jobs across managed or unmanaged pools of Amazon EC2 compute instances.
To configure the "AWSBatch" batch computation provider for use in the Wolfram Language, follow the instructions in the Set Up the AWS Batch Computation Provider workflow.
AWS Batch packs jobs into instances based on each job's vCPU count requirement. Multiple jobs can execute concurrently on a single EC2 instance, with each job running in a dedicated Docker container.
The "AWSBatch" batch computation provider stores input and output data for jobs in a user-specified Amazon S3 bucket.
The "AWSBatch" batch computation provider supports Linux-based jobs only.

Environment Properties

The following properties are supported in a RemoteBatchSubmissionEnvironment object for the "AWSBatch" provider:
  • "IOBucket"(required)name of the Amazon S3 bucket in which to store job input and output data
    "JobDefinition"(required)ARN for the AWS Batch job definition to use for submitted jobs
    "JobQueue"(required)ARN for the AWS Batch job queue to which jobs are submitted
    "ServiceObject"AutomaticServiceObject for the "AWS" service
    • The "IOBucket", "JobDefinition" and "JobQueue" properties are required to construct a valid RemoteBatchSubmissionEnvironment object.
    If the "ServiceObject" property is omitted (or is set to Automatic), a service object will be automatically constructed and connected with ServiceConnect["AWS"].
    If the "ServiceObject" property is set to "New", a new service connection will be created with ServiceConnect["AWS","New"], disregarding any saved connections or credentials.
    The Set Up the AWS Batch Computation Provider workflow provides instructions for creating all of the required environment resources in your AWS account using an automated AWS CloudFormation template.

    Job Settings

    The following settings are supported by the RemoteProviderSettings option when using the "AWSBatch" provider:
  • "GPUCount"Inheritedinteger number of GPUs »
    "Memory"Automatic"InformationUnit" quantity or integer number of mebibytes »
    "VCPUCount"Inheritedinteger number of vCPUs »
    • The value Inherited for the "GPUCount", "Memory" and "VCPUCount" settings corresponds to the value in the AWS Batch job definition specified by the supplied RemoteBatchSubmissionEnvironment object.
    The "GPUCount" setting instructs AWS Batch to schedule the job to an instance that has, at minimum, the specified number of GPUs available and to make that number of GPUs available within the job container.
    The "Memory" setting instructs AWS Batch to reserve the specified amount of memory for the job. If the job exceeds this limit, it will be terminated. »
    With the default setting "Memory"Automatic, the minimum memory limit is set to the larger of the value in the job definition and the expression Max[vCPUCount*,GPUCount*,]. »
    The value of the "VCPUCount" setting is used as a weight for the Linux kernel scheduler when allocating CPU time on the host instance to processes within the job container. It does not necessarily result in any fixed number of vCPUs being dedicated to the job container.
    Within a "Single"-type job submitted with RemoteBatchSubmit, functions such as ParallelEvaluate and ParallelMap will automatically launch a number of subkernels equivalent to the value of the "VCPUCount" setting. This behavior can be overridden by calling LaunchKernels with an explicit number of kernels.
    When using the "AWSBatch" provider, the value of the TimeConstraint option of RemoteBatchSubmit and RemoteBatchMapSubmit must be at least 60 seconds. The TimeConstraint option defaults to the "Execution timeout" value in the job definition being used.

    Job Statuses

    The following are possible values of the "JobStatus" job property when using the "AWSBatch" provider, listed in the order through which a typical job will pass:
  • "Submitted"the submitted job has not yet been evaluated by the AWS Batch scheduler
    "Pending"the job is waiting for dependencies to be satisfied
    "Runnable"the job is waiting for compute resources to be available
    "Starting"the job has been scheduled to an instance and its container image is being downloaded
    "Running"the job's container has started
    "Succeeded"the job's execution has succeeded and its output has been uploaded
    "Failed"the job's execution has failed
    • A job in the "Running" state may be in the process of downloading input files, evaluating job code or uploading output data.
    An array job will remain in the "Pending" state until either all of its constituent child jobs have succeeded (at which point it will transition to the "Succeeded" state) or at least one child job has failed (at which point it will transition to the "Failed" state).
    More information about job states is available in the AWS Batch documentation.

    Job Properties

    When using the "AWSBatch" provider, the following properties are available from "Single"-type job objects, in addition to the standard properties supported by RemoteBatchJobObject:
  • "JobExitCode"exit code returned by the kernel within the job container
    "JobLog"console logs from the job container
    "JobStatusReason"string describing the reason for which the job is in its current state
    "ProviderJobID"AWS-provided unique identifier for the job
    • When using the "AWSBatch" provider, the following properties are available from "Array"-type job objects, in addition to the standard properties supported by RemoteBatchJobObject:
  • "ChildJobExitCodes""JobExitCode" property of each array child job
    "ChildJobStatusReasons""JobStatusReason" property of each array child job
    "JobStatusReason"string describing the reason for which the array job is in its current state
    "ProviderJobID"AWS-provided unique identifier for the array job
    • When using the "AWSBatch" provider, the following properties are available from "ArrayChild"-type job objects in addition to the standard properties supported by RemoteBatchJobObject:
  • "JobExitCode"exit code returned by the kernel within the job container
    "JobLog"console logs from the child job container
    "JobStatusReason"string describing the reason for which the child job is in its current state
    "ProviderJobID"AWS-provided unique identifier for the child job
    • The meanings of some possible values of the "JobExitCode" property are listed on the reference page for Exit.
    The value of the "JobStatusReason" property is supplied by AWS Batch and may be absent from a given job, depending on its current state.
    The value of the "JobLog" property is retrieved from Amazon CloudWatch Logs and will be absent until the job has reached the "Running" state.
    The value of the "ProviderJobID" property identifies a job to AWS Batch, in contrast to the "JobUUID" property, which identifies a job within the Wolfram System.
    AWS Batch expires and deletes a job's metadata 24 hours after the job reaches a completed state ("Succeeded" or "Failed"). »
    RemoteBatchJobObject expressions representing jobs that have expired cannot be queried for status and will not be listed by RemoteBatchJobs. Output data from expired jobs will remain in Amazon S3 and accessible from the RemoteBatchJobObject until manually deleted or expired by the bucket's lifecycle policy.

    Examples

    open allclose all

    Basic Examples  (2)Summary of the most common use cases

    Create an "AWSBatch" RemoteBatchSubmissionEnvironment object, after configuring the "AWSBatch" batch computation provider as described in the Set Up the AWS Batch Computation Provider workflow:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bkqbsj
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-h14fw1

    Submit a job using the created environment:

    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-jyh8rr
    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bepkqd

    Query the job’s status:

    In[5]:=5
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-qizr2
    Out[5]=5

    Query the job’s status again, after it has completed:

    In[6]:=6
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-fln9ob
    Out[6]=6

    Download the job’s output:

    In[7]:=7
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-h66qpm
    Out[7]=7

    Create an environment object using an explicitly specified "AWS" service object:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-b6wpwo
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-zl8po
    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-g57drg
    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-hiu7n1

    Job Settings  (4)

    "GPUCount"  (1)

    Submit an array job to AWS Batch that uses GPU computation to perform inference with a pretrained neural net:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-mlkj7
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-ba6vw3

    Download the array job output:

    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-glvxje
    Out[3]=3

    "Memory"  (2)

    The memory limit for a job can be adjusted with the "Memory" job setting:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-il1x98
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-gc51ad
    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-fihutk
    Out[3]=3

    If not otherwise specified, the default memory limit is based on the configured vCPU and GPU counts:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-cneu2n
    Out[1]=1

    "VCPUCount"  (1)

    Instruct AWS Batch to allocate four vCPUs to a submitted job:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bvt2n
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bpiila
    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-h4w7dx
    Out[3]=3

    Job Properties  (2)

    "Single" Jobs  (1)

    Submit a batch job to AWS Batch using RemoteBatchSubmit:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-df9rlo
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-4cxf6

    Query the job’s status along with the reason why the job transitioned to that state:

    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-ecxdi
    Out[3]=3

    Obtain the console output from the job container:

    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-i43qt
    Out[4]=4

    "Array" and "ArrayChild" Jobs  (1)

    Submit an array job to AWS Batch using RemoteBatchMapSubmit:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-capz2h
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-dv73e5

    Query the status of each child job along with the reason why each transitioned to its current state:

    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-ih7mxw
    Out[3]=3

    Obtain a RemoteBatchJobObject expression representing the first child job:

    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-e69pyd
    Out[4]=4

    Query the child job’s status along with the reason why the child job transitioned to that state:

    In[5]:=5
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bk9auq
    Out[5]=5

    Obtain the console output from the child job container:

    In[6]:=6
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-eauidf
    Out[6]=6

    Properties & Relations  (1)Properties of the function, and connections to other functions

    If a job was terminated with RemoteBatchJobAbort, the value of the "JobStatusReason" property will indicate that the termination request originated from the Wolfram Language:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-6p2du
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-is4v1y
    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bf4sch
    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-jter1s
    In[5]:=5
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-co85uc
    Out[5]=5

    Possible Issues  (2)Common pitfalls and unexpected behavior

    AWS Batch terminates jobs that exceed their memory limit:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-uliad
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-fhgdjr
    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-nfqcpk
    Out[3]=3

    The memory limit for a job can be adjusted with the "Memory" job setting:

    In[4]:=4
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-b00ugz
    In[5]:=5
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bgjyjq
    In[6]:=6
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-bz9g0r
    Out[6]=6

    AWS Batch deletes a job’s metadata 24 hours after the job reaches a completed state ("Succeeded" or "Failed"). This will cause metadata queries to return a missing value:

    In[1]:=1
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-cdysar
    In[2]:=2
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-r4jjn
    Out[2]=2

    The job’s output data will still be accessible until it is either manually deleted or automatically expired by the lifecycle policy on the submission environment's I/O bucket, if such a policy is configured:

    In[3]:=3
    ✖
    https://wolfram.com/xid/0ihah4xlexcllho87hjuda310pqpy0uqdloyx40-b5exw8
    Out[3]=3