Creating a Job

Create and run a job in Data Science.

Ensure that you have created the necessary policies, authentication, and authorization for your jobs.

Before you begin:

Create a job artifact file or build a custom container.
To store and manage job logs, learn about logging.
To use storage mounts, you must have an Object Storage bucket or OCI File Storage Service (FSS) mount target and export path.

To use FSS, you must first create the file system and the mount point. Use the custom networking option and ensure that the mount target and the notebook are configured with the same subnet. Configure security list rules for the subnet with the specific ports and protocols.

Ensure that service limits are allocated to file-system-count and mount-target-count.
To use storage mounts, you must have an Object Storage bucket or OCI File Storage Service (FSS) mount point.

Add basic information for the job you're creating.
1. From the jobs list page, select Create job. If you need help finding the list of jobs, see Listing Jobs.
2. Select Single Node if using a single machine for the job or Multi Node for demanding jobs that are to be run over several nodes.
3. (Optional) Select a different compartment for the job.
4. (Optional) Enter a name and description for the job (limit of 255 characters). If you don't provide a name, a name is automatically generated.
  
  For example, job20210808222435
5. Single or multi node configuration steps
  
  If you selected Single Node, follow the steps in Using the Console for Single Node Jobs.
  
  If you selected Multi Node, follow the steps in Using the Console for Multi Node Jobs.
Using the Console for Single Node Jobs
Using the Console for Single Node Jobs

The required and optional configuration steps for a single node job.

Required Configuration Steps

Optional Configuration Steps

Review and Create

Required Configuration Steps

Required configuration steps specific for creating a single node job.

Create a default job configuration that's used when the job is run using these options.

Enter or select any of the following values:

Custom environment variable key

The environment variables that control the job.

Note

If you uploaded zip file or compressed tar file, add the JOB_RUN_ENTRYPOINT as a custom environment variable to point to the file.

Value

The value for the custom environment variable key.

You can select Additional custom environment key to specify more variables.

Command line arguments

The command line arguments that you want to use for running the job.

Maximum runtime (in minutes)

The maximum number of minutes that the job can run. The service cancels the job run if its runtime exceeds the specified value. The maximum runtime is 30 days (43,200 minutes). We recommend that you configure a maximum runtime on all job runs to prevent runaway job runs.

Select a Compute shape.
(Optional) Change the Compute shape by selecting Change shape. Then, follow these steps in the Select compute panel.

Select an instance type.

Select a shape series.

Select one of the supported Compute shapes in the series.

Select the shape that best suits how you want to use the resource.

For each OCPU, select up to 64 GB of memory and a maximum total of 512 GB. The minimum amount of memory allowed is either 1 GB or a value matching the number of OCPUs, whichever is greater.

If using burstable VMs, toggle Burstable.
In Baseline utilization per OCPU select the percentage of OCPUs that you usually want to use. The supported values are 12.5% and 50%.

Select Select shape.

For Storage, enter the amount of block storage to use between 50 GB and 10, 240 GB (10 TB). You can change the value by 1 GB increments.
Select one of the following options to configure the network type:

Default networking—The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

Custom networking—Select the VCN and subnet that you want to use for the job.

For egress access to the public internet, use a private subnet with a route to a NAT gateway.

If you don't see the VCN or subnet that you want to use, click Change Compartment, and then select the compartment that contains the VCN or subnet.

Important

Custom networking must be used to use a file storage mount

This step is optional only if BYOC is configured. Upload the job artifact by dragging the required job artifact file into the box.
Select Next.

Optional Configuration Steps

Configuration steps specific for creating a single node job.

(Optional) To use logging, select Select, and then ensure that Enable logging is selected.

Select a log group from the list. You can change to a different compartment to specify a log group in a different compartment from the job.

Select one of the following to store all stdout and stderr messages:

Enable automatic log creation

Data Science automatically creates a log when the job starts.

Select a log

Select a log to use.

Select Select to return to the job run creation page.

(Optional) To use Bring Your Own Container, in Environment configuration select Select.
In the Set your BYOC environment panel, follow these steps:

In Repository select a repository from the list. If the repository is in a different compartment, select Change compartment.

In Image select an image from the list.

(Optional) In Entrypoint enter an entry point. To add another, select +Add parameter.

(Optional) In CMD enter a CMD. To add another, select +Add parameter.
Note

Use CMD as arguments to the ENTRYPOINT or the only command to run in the absence of an ENTRYPOINT.

(Optional) In Image digest enter an image digest.

(Optional) In Signature ID, if using signature verification, enter the OCID of the image signature. For example, ocid1.containerimagesignature.oc1.iad.aaaaaaaaab....

Select Select.

(Optional) To use storage mounts, select +Add storage mount.
Ensure you have the authorization to use storage mounts.

Select a storage mount type, OCI Object Storage or OCI File Storage.

Select a compartment that contains the storage resource that you want to mount.

Select one of the following:

Object Storage

If using custom networking:

Create the service gateway in the VCN.

For the route table configurations in the private subnet, add the service gateway.

Change the egress rules of security list of the required subnet to let traffic to all services in the network.

The bucket you want to use.

You can add an object name prefix. The prefix must start with an alphanumeric character. The allowed characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).

File Storage

The mount target and export path you want to use. You must have created the file system. Specify an existing mount target, or create a new one.

You must use a custom network to use file storage.

Follow the security list rules of the subnet.
Note

Before mounting a file system, you must configure security rules in the VCN subnet to let network traffic to and from the mount target. You can set up security rules in subnet security lists, network security groups, or by using a combination of both. For more information, see Configuring VCN Security Rules for File Storage.

Stateful ingress to TCP ports 111, 2048, 2049, and 2050, and UDP ports 111 and 2048.

Stateful egress for TCP source ports 111, 2048, 2049, and 2050, and UDP source port 111.

Enter the path under which the storage is to be mounted.

Storage is mounted under the specified mount path. The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

Select Submit.

Repeat these steps to add up to two storage mounts for notebook sessions.

Add a start up probe. Select Select.

Enter a command.

(Optional) To add another command, select +Add command and repeat step a.

(Optional) Enter a value for the initial delay (in seconds).

(Optional) Enter a value of the period.

(Optional) Enter a value for the failure threshold.

Select Save.

(Optional) Select Show advanced options to add tags to the job.
(Optional) In the Tags section, add one or more tags to the job. If you have permissions to create a job, then you also have permissions to apply free-form tags to that job. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
Select Next.

Review and Create

Review and create the job.

Review the job.
To update any part of the job configuration, select Edit in the appropriate section.

When you're happy with the configuration, select Create to create the job.

After the job is in an active state, you can use job runs to repeatedly run the job.
Using the Console for Multi Node Jobs
Using the Console for Multi Node Jobs

The required and optional configuration steps for a multi node job.

Required Configuration Steps

Optional Configuration Steps

Review and Create

Required Configuration Steps

Required configuration steps specific for creating a multi node job.

Select one of the following options to configure the network type:

Default networking—The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

Custom networking—Select the VCN and subnet that you want to use for the job.

For egress access to the public internet, use a private subnet with a route to a NAT gateway.

If you don't see the VCN or subnet that you want to use, click Change Compartment, and then select the compartment that contains the VCN or subnet.

Important

Custom networking must be used to use a file storage mount.

This step is optional only if BYOC is configured. Upload the job artifact by dragging the required job artifact file into the box.
(Optional) Enter the maximum runtime (in minutes between 5 and 43200).
Select Add node group.
Enter a unique name for the node group.
Enter the number of replicas.
Enter the minimum number of replicas that must succeed.
For Storage, enter the amount of block storage to use between 50 GB and 10, 240 GB (10 TB). You can change the value by 1 GB increments.
(Optional) Enter the default configuration:

Enter the custom environment variable key and a value for it.

Select +Additional custom environment key to add another custom environment variable key and repeat step a.

Enter the command line arguments.

Note

If you uploaded a zip file or compressed tar file, add the JOB_RUN_ENTRYPOINT as a custom environment variable to point to the file.

Select a Compute shape.
(Optional) Change the Compute shape by selecting Change shape. Then, follow these steps in the Select compute panel.

Select an instance type.

Select a shape series.

Select one of the supported Compute shapes in the series.

Select the shape that best suits how you want to use the resource.

For each OCPU, select up to 64 GB of memory and a maximum total of 512 GB. The minimum amount of memory allowed is either 1 GB or a value matching the number of OCPUs, whichever is greater.

If using burstable VMs, toggle Burstable.
In Baseline utilization per OCPU select the percentage of OCPUs that you usually want to use. The supported values are 12.5% and 50%.

Select Select shape.

To use Bring Your Own Container, in Environment configuration select Select.
In the Set your BYOC environment panel, follow these steps:

In Repository select a repository from the list. If the repository is in a different compartment, select Change compartment.

In Image select an image from the list.

(Optional) In Entrypoint enter an entry point. To add another, select +Add parameter.

(Optional) In CMD enter a CMD. To add another, select +Add parameter.
Note

Use CMD as arguments to the ENTRYPOINT or the only command to run in the absence of an ENTRYPOINT.

(Optional) In Image digest enter an image digest.

(Optional) In Signature ID, if using signature verification, enter the OCID of the image signature. For example, ocid1.containerimagesignature.oc1.iad.aaaaaaaaab....

Select Select.

Add a start up probe. Select Select.

Enter a command.

(Optional) To add another command, select +Add command and repeat step a.

(Optional) Enter a value for the initial delay (in seconds).

(Optional) Enter a value of the period.

(Optional) Enter a value for the failure threshold.

Select Save.

Select Add.
(Optional) To add another node group, select Add node group and repeat the previous steps. You can add up to five node groups.
(Optional) If you have more than one node group, then specify the node group start up order, either in parallel or in sequence.

Optional Configuration Steps

Optional configuration steps when creating a multi node job.

(Optional) To use logging, select Select, and then ensure that Enable logging is selected.

Select a log group from the list. You can change to a different compartment to specify a log group in a different compartment from the job.

Select one of the following to store all stdout and stderr messages:

Enable automatic log creation

Data Science automatically creates a log when the job starts.

Select a log

Select a log to use.

Select Select to return to the job run creation page.

(Optional) To use storage mounts, select +Add storage mount.
Ensure you have the authorization to use storage mounts.

Select a storage mount type, OCI Object Storage or OCI File Storage.

Select a compartment that contains the storage resource that you want to mount.

Select one of the following:

Object Storage

If using custom networking:

Create the service gateway in the VCN.

For the route table configurations in the private subnet, add the service gateway.

Change the egress rules of security list of the required subnet to let traffic to all services in the network.

The bucket you want to use.

You can add an object name prefix. The prefix must start with an alphanumeric character. The allowed characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).

File Storage

The mount target and export path you want to use. You must have created the file system. Specify an existing mount target, or create a new one.

You must use a custom network to use file storage.

Follow the security list rules of the subnet.
Note

Before mounting a file system, you must configure security rules in the VCN subnet to let network traffic to and from the mount target. You can set up security rules in subnet security lists, network security groups, or by using a combination of both. For more information, see Configuring VCN Security Rules for File Storage.

Stateful ingress to TCP ports 111, 2048, 2049, and 2050, and UDP ports 111 and 2048.

Stateful egress for TCP source ports 111, 2048, 2049, and 2050, and UDP source port 111.

Enter the path under which the storage is to be mounted.

Storage is mounted under the specified mount path. The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

Select Submit.

Repeat these steps to add up to two storage mounts for notebook sessions.

(Optional) Select Show advanced options to add tags to the job.
(Optional) In the Tags section, add one or more tags to the job. If you have permissions to create a job, then you also have permissions to apply free-form tags to that job. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
Select Next.

Review and Create

Review and create the job.

Review the job.
To update any part of the job configuration, select Edit in the appropriate section.

When you're happy with the configuration, select Create to create the job.

After the job is in an active state, you can use job runs to repeatedly run the job.

These environment variables control the job.

Use the Data Science CLI to create a job as in this example:

Create a job with:

oci data-science job create \
--display-name <job_name>\
--compartment-id <compartment_ocid>\
--project-id <project_ocid> \
--configuration-details file://<jobs_configuration_json_file> \
--infrastructure-configuration-details file://<jobs_infrastructure_configuration_json_file> \
--log-configuration-details file://<optional_jobs_infrastructure_configuration_json_file>

Use this jobs configuration JSON file:

{
  "jobType": "DEFAULT",
  "maximumRuntimeInMinutes": 240,
  "commandLineArguments" : "test-arg",
  "environmentVariables": {
    "SOME_ENV_KEY": "some_env_value" 
  }
}

Use this jobs infrastructure configuration JSON file:

{
  "jobInfrastructureType": "STANDALONE",
  "shapeName": "VM.Standard2.1",
  "blockStorageSizeInGBs": "50",
  "subnetId": "<subnet_ocid>"
}

(Optional) Use this jobs logging configuration JSON file:

{
  "enableLogging": true,
  "enableAutoLogCreation": true,
  "logGroupId": "<log_group_ocid>"
}

Upload a job artifact file for the job you created with:

oci data-science job create-job-artifact \
--job-id <job_ocid> \
--job-artifact-file <job_artifact_file_path> \
--content-disposition "attachment; filename=<job_artifact_file_name>"

The ADS SDK is also a publicly available Python library that you can install with this command:
```
pip install oracle-ads
```
It provides the wrapper that makes the creation and running jobs from notebooks or on your client machine easy.

Use the ADS SDK to create and run jobs.

Oracle Cloud Infrastructure Documentation

Creating a Job

Using the Console for Single Node Jobs

Required Configuration Steps

Optional Configuration Steps

Review and Create

Using the Console for Multi Node Jobs

Required Configuration Steps

Optional Configuration Steps

Review and Create