Taxonomy Icon

Cloud

One of the questions that often comes up when developing with IBM Cloud is “How do I schedule tasks?” If you are using IBM bare metal servers, the answer is fairly simple: Use cron or a similar scheduler. But if you are using the IBM Cloud platform as a service (PaaS), the answer is a little more involved.

The IBM Cloud PaaS doesn’t have a system scheduler like cron. Instead, it offers the Workload Scheduler service, which is a special service instance you can connect to your IBM Cloud application. This service instance allows you to define custom processes that run on a pre-determined schedule and can be integrated with a variety of other tools, such as Cloudant, MQTT, Hadoop, and OpenWhisk.

In this tutorial, I introduce you to the Workload Scheduler service with a real-world example: saving a daily backup of an application database. By working through this example, you’ll understand the steps involved and can adapt them to meet your own use cases for process scheduling.

What you need for this tutorial

For applications that see a lot of activity, it’s important to maintain a regular backup of application data. Most of the time, that data is stored in a database, but it may also take the form of file assets or activity logs. Depending on the volume of data being generated, backups may need to be taken hourly, daily, weekly, or at custom intervals.

For this example, I assume that the application’s data is stored in a MySQL database and that this data needs to be backed up daily, as an SQL file. For safety, the backup should be stored separately from the application itself. This example uses Dropbox as the backup storage area, but you can use any storage service, including IBM Cloud’s own Object Storage service.

This example uses the Workload Scheduler service’s REST integration to perform the backup by invoking a specified URL endpoint at a specified time. This URL endpoint is a PHP script, which internally does the following:

  • Authenticates the REST request
  • Connects to the application database
  • Exports the database contents as an SQL file
  • Transfers the SQL file to Dropbox
  • Sends a notification email to the application administrator using SendGrid

Before starting, make sure you have everything you need:

Note: Any application that uses the Dropbox and SendGrid services must comply with the Dropbox and SendGrid terms of service, respectively. Similarly, any application that uses the IBM Cloud Platform must comply with IBM Cloud’s terms of use. Before beginning your project, spend a few minutes reading these requirements and ensuring that your application complies with them.

This tutorial covers just one possible example. You could use the process outlined here to perform other scheduled tasks, such as sending daily activity summaries, cleaning up unwanted files, archiving logs, and so on.

Step 1. Install the dependencies

To perform its tasks, the PHP script uses the following:

  • The MySQLDump-PHP client, to export a MySQL database in SQL format
  • The Dropbox API to store backup files
  • The SendGrid API to send email notifications

  • The first step is to obtain all the dependencies for the PHP script, including the MySQLDump-PHP client, a Dropbox PHP client, and the SendGrid PHP client. These can be easily downloaded and installed using Composer, the PHP dependency manager. Use the following Composer configuration file, which should be saved to $APP_ROOT/composer.json. ($APP_ROOT refers to your project directory.)

    
    {
        "require": {
            "ifsnop/mysqldump‑php": "2.",
            "sendgrid/sendgrid": "6.",
            "kunalvarma05/dropbox‑php‑sdk": "0.2.1"
        }
    }
                  

  • Install using Composer with the following command:
    
    shell> php composer.phar install
                  
  • Once the necessary components have been downloaded via Composer, create the directories $APP_ROOT/public for all web-accessible files and $APP_ROOT/data for temporary storage, where $APP_ROOT refers to the application directory.
    
    shell> cd myapp
    shell> mkdir public data
                  
  • Next, create the file $APP_ROOT/config.php file with the following information (you’ll fill in the placeholders as you progress).
    
    <?php
    $config = 
      'settings' => [
        'db' => [
          'name' => 'DATABASE‑NAME',
          'hostname' => 'DATABASE‑HOST',
          'username' => 'DATABASE‑USERNAME',
          'password' => 'DATABASE‑PASSWORD',
        ],
        'dropbox' => [
          'app‑key' => 'DROPBOX‑APP‑KEY',
          'app‑secret' => 'DROPBOX‑APP‑SECRET',
          'app‑token' => 'DROPBOX‑APP‑TOKEN',
          'app‑folder' => 'DROPBOX‑APP‑FOLDER',
        ],
        'sendgrid' => [
          'key' => 'SENDGRID‑API‑KEY',
          'to' => 'RECIPIENT‑EMAIL‑ADDRESS'
        ],
        'token' => 'guessme',
      ];
                  

Step 2. Obtain a Dropbox access token

To upload files to Dropbox, the PHP script needs to use the Dropbox API PHP client from the previous step. This client requires three key pieces of information: a Dropbox application key, the Dropbox application secret, and an application token. This information can be obtained from the Dropbox web dashboard. To begin:

  1. Log in to the Dropbox developer dashboard and create a new application by clicking Create app.
  2. Select the Dropbox API, set the type of access to App folder, and enter a name for your application, as shown below. Click Create app to create the new application. Dropbox application creation
  3. Once the Dropbox application has been created, you are redirected to the application settings page. This page displays an application key and secret. Note these values, together with the application folder name, and enter them into the configuration file from Step 1.
  4. Under ordinary circumstances, every access to the Dropbox API must go through an OAuth authorization process. However, in this case, because it will be your own trusted PHP script accessing your own account in the Dropbox API, you can bypass the normal authorization flow by creating an access token and using it in every request made to the Dropbox API by the PHP script. To do this, generate an access token from the application settings page and enter it into the configuration file from Step 1.

Important: Do not share the Dropbox access token and application secret with anyone.

Step 3. Obtain a SendGrid API key

To send email notifications, the PHP script needs to use the SendGrid API. SendGrid provides a PHP SDK that can be used for this purpose. To use it, you must first obtain a SendGrid API key.

  1. Log in to your SendGrid account and browse to the Settings > API Keys menu. Click Create API Key.
  2. Enter a name for the API key and choose the Full access option. Click Create & View to generate the key.
  3. Note the API key (it will be displayed only once) and enter it into the configuration file from Step 1. This is a good time to specify the recipient email address for status notifications in the same configuration file.

Step 4. Create the REST service endpoint

You are now ready to begin building the PHP script that will work as a REST service and perform the backup. Use the code below, which should be saved to $APP_ROOT/public/backup.php.


<?php
// load classes
require '../vendor/autoload.php';

// load configuration
require '../config.php';

use Ifsnop\Mysqldump as Mysqldump;
use Kunnu\Dropbox\DropboxApp;
use Kunnu\Dropbox\Dropbox;
use Kunnu\Dropbox\DropboxFile;

try {

  // look for token in POST data
  // reject request if not found or mismatch
  $json = file_get_contents('php://input');
  $obj = json_decode($json);

  if (isset($obj‑>token) && ($obj‑>token == $config['settings']['token'])) {

    // initialize MySQL client
    // connect to database and dump contents to file
    $dump = new Mysqldump\Mysqldump(
      'mysql:host=' . $config['settings']['db']['hostname'] .
      ';dbname=' . $config['settings']['db']['name'],
      $config['settings']['db']['username'],
      $config['settings']['db']'password'    );
    $filename = '../data/' . $config['settings']['db']['name'] . '‑' .
      date('Y‑m‑d‑h‑i‑s', time()) . '.sql';
    $dump‑>start($filename);

    // initialize Dropbox client
    $app = new DropboxApp(
      $config['settings']['dropbox']['app‑key'],
      $config['settings']['dropbox']['app‑secret'],
      $config['settings']['dropbox']'app‑token'    );
    $dropbox = new Dropbox($app);
    $dropboxFile = new DropboxFile($filename);
    $uploadedFile = $dropbox‑>upload($dropboxFile, "/" .
      basename($filename));
    unlink($filename);

    // initialize SendGrid client
    $sg = new \SendGrid($config['settings']['sendgrid']['key']);
    $from = new SendGrid\Email(null, "no‑reply@example.com");
    $subject = "Database backup notification";
    $to = new SendGrid\Email(null, $config['settings']['sendgrid']['to']);
    $content = new SendGrid\Content(
      "text/plain",
      "Your database backup was successful. Your backup filename is: " .
      basename($filename));
    $mail = new SendGrid\Mail($from, $subject, $to, $content);
    $response = $sg‑>client‑>mail()‑>send()‑>post($mail);
    if ($response‑>statusCode() != 200 &&
      $response‑>statusCode() != 202) {
      throw new \Exception('SendGrid failure, response code ' .
        $response‑>statusCode()
      );
    }

    http_response_code(200);
    echo 'Backup successful';

  } else {
    throw new Exception ('Invalid authentication token');
  }
} catch (\Exception $e) {
    http_response_code(500);
    echo $e‑>getMessage();
}

exit;
              

This script does a number of things, so let’s step through it:

  1. The script begins by loading the Composer auto-loader, which takes care of loading all the necessary dependencies, and the configuration file.
  2. Next, it looks for an incoming POST request in JSON format. If a request is found, it looks for a 'token' key in the request. The supplied token is matched against the token specified in the configuration file. The request is processed further only if the tokens match. This provides a minimal layer of security and ensures that only valid requests are accepted.
  3. Assuming a valid request, the script initializes the MySQLDump-PHP client and connects to the target database using the configured credentials. The client’s start() method is then used to export the contents of the target database to an SQL file, which is (temporarily) stored in the data/ directory.
  4. The script then initializes the Dropbox PHP API client, using the Dropbox authentication credentials from Step 2. The client’s upload() method is used to transfer the SQL file from the data/ directory to the application folder on Dropbox. The temporary file is then deleted.
  5. Finally, the script initializes the SendGrid PHP client using the SendGrid API key from Step 3. It creates a new SendGrid/Mail object by specifying the sender address, recipient address (obtained from the configuration file), subject, and body. The body includes a status message and the filename of the generated SQL file on Dropbox. The client’s post() method is then used to transmit the message through SendGrid’s servers.
  6. The script terminates with either a 200 (OK) or 500 (error) HTTP response code.

Step 5. Configure database credentials and token

To connect to the target MySQL database, the PHP script needs the database host name, username, password, and database name. You have two options for specifying these values:

  • You can specify them as static values in the application configuration file created in Step 1. If using this approach, go ahead and add these values to the configuration file. Remember that you can obtain the necessary credentials from the Credentials page of the database instance in the IBM Cloud dashboard or third-party dashboard. Service credentials
  • Alternatively, you can import the needed values from the IBM Cloud environment by connecting the corresponding database service to the application. If using this approach, add the following lines to the PHP script in Step 4, before the try block (you may need to use different JSON keys depending on the service you’re connecting to; this example uses ClearDB).


<?php
// if IBM Cloud VCAP_SERVICES environment available
// overwrite local credentials with IBM Cloud credentials
if ($services = getenv("VCAP_SERVICES")) {
  $services_json = json_decode($services, true);
  $config['settings']['db']['hostname'] = $services_json['cleardb'][0]['credentials']['hostname'];
  $config['settings']['db']['username'] = $services_json['cleardb'][0]['credentials']['username'];
  $config['settings']['db']['password'] = $services_json['cleardb'][0]['credentials']['password'];
  $config['settings']['db']['name'] = $services_json['cleardb'][0]['credentials']['name'];
}
              

By default, the configuration file created in Step 2 includes a simple authentication token. This token is used to validate incoming POST requests. You should update this token to a different value known only to you before deploying the script.

Step 6. Deploy to IBM Cloud

The PHP script that performs the backup can be deployed to IBM Cloud as a standalone application or as part of another application. So long as it is accessible at a public URL, the Workload Scheduler service will be able to invoke it. This gives you two primary deployment options:

  • You can deploy this script as a standalone application, to back up a single application database. This is the deployment option covered in the remainder of this section.
  • You can bundle this backup script as part of a larger application. In this case, move the backup script from Step 4 to your application’s public-facing directory and re-deploy the application as usual.

Note that in both options, you can connect the database instance to your application and read key variables (such as database credentials) directly from the IBM Cloud environment instead of using a configuration file. Refer to Step 5 for details on how to do this.

  1. First, create the application manifest file, remembering to use a unique host and application name by appending a random string to it (such as your initials).
    
    ‑‑‑
    applications:
    ‑ name: myapp‑initialsmemory: 256M
    instances: 1
    host: myapp‑initialsbuildpack: https://github.com/cloudfoundry/php‑buildpack.git
    stack: cflinuxfs2
                  
  2. Configure the build pack to use the public/ directory of the application as the Web server directory. Create a $APP_ROOT/.bp-config/options.json file with the following content:
    
    {
        "WEB_SERVER": "httpd",
        "WEBDIR": "public",
        "PHP_VERSION": "{PHP_70_LATEST}"
    }
                  
  3. Load the PDO and MySQL PHP extensions, required by the MySQLDump-PHP client. Create a $APP_ROOT/.bp-config/php/php.ini.d/php.ini file with the following content:
    
    extension=mysqli.so
    extension=mbstring.so
    extension=pdo.so
    extension=pdo_mysql.so
                  
  4. Push the application to IBM Cloud:
    
    shell> cf api https://api.ng.bluemix.net
    shell> cf login
    shell> cf push
                  
  5. If you’re planning to read the target database’s credentials directly from the IBM Cloud environment, and you have made the necessary changes to the PHP script as shown in Step 5, bind the database service instance to the application. Remember to use the correct ID for the service instance to ensure that the right instance is bound to the application. You can obtain the service ID from the service instance page in the IBM Cloud dashboard.
    
    shell> cf bind‑service myapp‑[initials] "[database‑service‑id]"
                  
  6. Restage the application for the changes to take effect:
    
    shell> cf restage myapp‑initials              
  7. The backup script should now be accessible at the public URL http://myapp-_[initials]_.mybluemix.net/backup.php. (If you are using the script as part of a larger application, this public URL will be different.) Before proceeding to the next step, test the script by sending a POST request to the above URL, using a tool such as Postman or RESTer. Include a JSON document in the POST request containing the token you defined in Step 5, as shown below:
    
    {
    "token": "guessme"
    }
                  

If all goes well, the PHP script should perform the backup and respond with an HTTP 200 success code. In case of an error, the response will be an HTTP 500 error code and the response output will include a description of the error.

REST response

If the backup succeeds, you should be able to see the SQL file in your Dropbox application folder. In case of an error, you can debug the script using “Debugging PHP errors on IBM Cloud” (see “Related topics” below).

Step 7. Configure the Workload Scheduler service

Now that you have a way to manually trigger a backup with a POST request, all that’s left is to automate it:

  1. Log in to the IBM Cloud dashboard and click Create resource. From the Application Services section, find and create a new Workload Scheduler instance using the Standard plan. Workload Scheduler instance creation
  2. Launch the service instance by clicking Launch on the service detail page. Accept the terms and conditions to proceed.
  3. Select the My processes folder on the left and click New to create a new process. Define the new process as follows: a. In the General tab, enter a name for the process and specify the period during which it is valid. Workload Scheduler process details b. On the Triggers tab, click New and define a schedule for the process by specifying the frequency (once, daily, weekly, or monthly), the start and end day, the start time, and the repeat interval. Workload Scheduler process frequency c. On the Steps tab, click New and define a new RESTful step. In the Action section, enter the public URL to the PHP script and specify the method as POST. In the Body section, specify the request content type as “application/json” and create a JSON document containing the token as the body content. Workload Scheduler REST configuration
  4. Click OK to save the changes, then click Enable to enable the process in the process library.

The Workload Scheduler should now run the process on the defined schedule, invoking the REST endpoint and creating a fresh backup of the database each time. The History tab of the process can be used to monitor process execution.

Conclusion

This tutorial explained how to schedule tasks on IBM Cloud. I considered a common use case — performing a regular backup of application data — and described how to satisfy it with an IBM Cloud Workload Scheduler instance and a REST-accessible PHP script. I used two external services — Dropbox for file storage and SendGrid for notification delivery — to illustrate how you can use the Workload Scheduler to harness both IBM Cloud services and third-party services to execute tasks on a recurring schedule.

Obviously, this tutorial covers just one possible example. You could use the process outlined here to perform other scheduled tasks, such as sending daily activity summaries, cleaning up unwanted files, archiving logs, and so on. The PHP script used here is available on GitHub for you to use as a starting point. Try it and then start modifying it for your specific requirements. Happy coding!