Win $20,000. Help build the future of education. Answer the call. Learn more

Run a batch job on a managed serverless platform

This article is part of the Introduction to IBM Cloud Code Engine series.

What is a batch job in IBM Cloud Code Engine

In simple terms, a batch job is a program that is assigned to run on a computer without further user interaction. IBM Cloud Code Engine is designed for a developer to run any type of batch job on a unified, serverless platform without provisioning, configuring, managing, or securing any clusters, networks, or VMs. And only pay when your batch job is running.

Batch jobs in IBM Cloud Code Engine run as containers. That means you only need to package your batch job as an image and submit it to IBM Cloud Code Engine. It will make sure your batch job is isolated for security and guarantee your batch job CPU and memory request which you specified.

IBM Cloud Code Engine can also run your batch job as an array of job instances. All instances will start and run in parallel, and each instance will get a unique index value in the array list. The instance will run until completed. IBM Cloud Code Engine will monitor the instance and retry failure instances automatically.

With batch job IBM Cloud Code Engine, you can easily and efficiently run large-scale batch job instances in parallel to support various kinds of workloads, such as heavy compute tasks, ETL workloads (for example, transcoding), map and reduce workloads, simulations (scientific compute), rendering, and any type of parallel data processing.

Benefits of using a batch job in IBM Cloud Code Engine

Easy user experience

Compared with a job in Kubernetes, where you have to deal with complex YAML files, IBM Cloud Code Engine abstracts two concepts, job and job run, which is designed for end-users to easily manage batch job. IBM Cloud Code Engine also provides an easier UI and CLI to guide you on how to create a job by providing a few input parameters from the user perspective. After the job is created, you can submit the job anytime to kick off the job running repeatedly.

Run multiple batch job instances in parallel

Target to support distributed computing for a heavy workload, when define or batch job, you can specify “Array indices” to run multiple instances in parallel. For example, the user can specify the values from 1 to 10, which results in 10 job instances that are created and run in parallel, where each job instance gets an index value between 1 and 10 assigned as an environment variable for the partitioning workload.

Serverless model

IBM Cloud Code Engine is a serverless platform that runs your batch job. You need only to focus on developing your batch job, instead of spending effort to maintain infrastructure, such as setting up your compute environment, keeping the latest security patches, or scaling the compute environment for a heavy workload. IBM Cloud Code Engine covers all of the infrastructure stuff for you and provides isolated, secure, and scalable compute resources to run your batch job. And you only pay for the resources that your batch job actually uses. If you do not run a no batch job, than you do not get billed.

Observability

IBM Cloud Code Engine provides operation capabilities for you to check log and metrics for your batch job by integrating with the IBM Log Analysis with LogDNA and IBM Cloud Monitoring with Sysdig services.

How to use a batch job in IBM Cloud Code Engine

IBM Cloud Code Engine simplifies the UI and CLI experience for you. Following is information about how to use the IBM Cloud Code Engine CLI to walk through your batch job lifecycle.

Batch job input and output

IBM Cloud Code Engine provides compute resources to run your batch job. When your batch job needs to read or output data and send out a notification, you must integrate your batch job with external services, such as object storage, database, or notification services. If it is an IBM Cloud service, you can try IBM Cloud Code Engine Service Binding to bind service to your batch job. Or you can populate environment variables when you creating a batch job to access other external services.

Define batch job

To create a batch job, you need only to provide a container image as a mandatory input property. You can also provide other optional properties, such as command and argument, environment variables, CPU and memory, and array indices.

For example, the ibmcloud code-engine job create --name batchjob --image docker.io/ibmcom/testjob command defines a batch job that you want, but does not actually run the batch job.

Submit batch job

To start a batch job, you can specify the batch job that we defined previously, such as ibmcloud code-engine jobrun submit --job batchjob. Or you can also merge these two steps with ibmcloud code-engine jobrun submit --image docker.io/ibmcom/testjob --name batchjobrun.

Query status and logs

IBM Cloud Code Engine provides other commands for you to query job status, manage the job, and query logs. For example, the command ibmcloud code-engine jobrun get --name batchjobrun obtains job run status. The command ibmcloud code-engine jobrun logs --jobrun batchjobrun gets logs.

Summary

IBM Cloud Code Engine batch job is truly designed for a developer to help you focus only on batch job development. IBM Cloud Code Engine covers all of the infrastructure and security aspects for you. Also, IBM Cloud Code Engine is a pay-as-you-go model, so you only pay for the resources that you actually use. Get started by registering for an IBM Cloud Code Engine trial and running your first IBM Cloud Code Engine batch job.