Deploy number-crunching Python applications to the IBM Cloud

This tutorial illustrates typical workflows and tools in cloud computing, addressing key concepts and technologies. By the end of this tutorial, you should be able to deploy a Python application to the cloud, both as a Cloud Foundry app and as a Kubernetes pod. You will also gain some familiarity with the following:

  • IBM Cloud
  • GitHub
  • Docker containers
  • Visual Studio Code
  • Continuous Delivery
  • Kubernetes containers
  • Microservices starter kits
  • Cloud Foundry applications

Prerequisites

  • Internet access
  • A computer running either MacOS, Windows, or Linux

Estimated time

Completing this tutorial should take 1 to 2 hours, depending on your familiarity with the concepts.

Steps

To complete this tutorial, you’ll need to complete the following steps:

  1. Install the developer tools and environments
  2. Configure the DevOps tools
  3. Play with your Python application
  4. Develop a number-crunching application
  5. Containerize the Python application

1. Install the developer tools and environments

In this section, you build your local development environment as well as the IBM Cloud infrastructure for deploying your application. You start by creating an IBM Cloud account and requesting an instance of a free trial Kubernetes cluster. Then, you create a Github account and configure your local git client. You also install a number of software development tools that you need in your journey toward your first cloud-based Python application: the IBM Cloud CLI, Docker, Kubernetes, and Visual Studio Code.

Set up the IBM Cloud environment

  1. Log in to IBM Cloud. If you do not have an account, create an IBM Cloud account using your email address.
  2. Click Create resource + in the top right.
  3. Open the Services menu on the left, select the Containers category, and click the Kubernetes Service card. kubernetes
  4. On the Kubernetes Cluster page, select the Free plan and default resource group.
  5. Name your cluster username-cluster and click Create.
  6. In the Overview tab on the left, watch as your cluster is deployed. It can take up to 30 minutes until the Normal status is reached.

Set up the GitHub environment

  1. Log in to GitHub. If you do not have an account, sign up for free using your email address.
  2. Update your profile page.
  3. Install a Git client:
  4. (macOS and Windows): Follow the instructions and set up GitHub Desktop. For Windows users, if Firefox cannot handle the operation, copy and paste the URL into a different browser.
  5. Define your Git identification credentials according to your profile page:
    • macOS and Windows: Under Preferences / Options, go to the Git tab and fill in your name and email address.
    • Fedora: Open a terminal and enter the following:
        git config --global user.name "Your Name"
        git config --global user.email "your@email.com"
      
  6. (macOS) Click the GitHub Desktop menu and select Install Command Line Tool….
  7. Follow these instructions to generate an SSH key.
  8. Add the SSH key to your GitHub profile according to the instructions.

Install the IBM Cloud CLI

  1. Go to the IBM Cloud CLI download page.
  2. Choose the install command according to your operational system and follow the instructions:
    • macOS users: You can also do brew cask install ibm-cloud-cli.
    • Windows users: A reboot might be required after installing the IBM Cloud CLI.
  3. Install IBM Cloud CLI plugins:
    • IBM Cloud Developer Tools: ibmcloud plugin install dev.
    • IBM Cloud Container Service: ibmcloud plugin install kubernetes-service.
    • IBM Cloud Container Registry: ibmcloud plugin install container-registry.
  4. In a terminal window, enter ibmcloud --version to check the installation.
  5. In a terminal window, enter ibmcloud plugin list to check the plugins.

Install Docker Community Edition

  1. Go to the Docker Store and follow the instructions according to your operational system:
  2. (Windows) You must enable virtualisation in the BIOS. Go to Security, then Virtualization, and enable both.
  3. In a terminal window, enter docker --version to check the installation.

Install the Kubernetes client

  1. Go to the Install and Set Up kubectl page and follow the instructions appropriate to your operational system:

    • macOS users: You can also do brew install kubernetes-cli.
    • Fedora users: You can also do sudo dnf install kubernetes.
    • Windows users: To add the folder containing kubectl.exe to the system PATH, follow these command-line or graphical interface instructions.
  2. In a terminal window, enter kubectl version to check the installation.

Install Visual Studio Code

  1. Go to Visual Studio Code download page.
  2. Choose the installer according to your operational system and follow the instructions.
    • macOS users: You can also do brew cask install visual-studio-code.
    • Fedora users: You can find detailed instructions here.
  3. Open VS Code and install extensions:
    • Python: Click Install or press Ctrl + P and enter ext install ms-python.python.
    • IBM Developer Tools: Click Install or press Ctrl + P and enter ext install IBM.ibm-developer.
    • Output Colorizer: Click Install or press Ctrl + P and enter ext install IBM.output-colorizer.
    • Docker: Click Install or press Ctrl + P and enter ext install PeterJausovec.vscode-docker.
    • Kubernetes: Click Install or press Ctrl + P and enter ext install ms-kubernetes-tools.vscode-kubernetes-tools.

2. Configure DevOps tools

In this section, you create a Python Flask starter kit and link it to a Github repository. Later, you configure a Continuous Delivery Pipeline that tracks the repository and redeploys your application at every git push.

Create a Python Starter Kit

  1. Log in to IBM Cloud.
  2. Click Create resource + in the top right corner.
  3. Open the Software menu on the left, then select the Developer Tools and Starter Kits filter boxes.
  4. Select the Python Flask App card. python_flask_app
  5. In the next page, click Get started.
  6. Name it username-python-microservice and, leaving everything else untouched, click Create.

Create the GitHub repository

  1. Log in to IBM Cloud.
  2. Click username-python-microservice in the Apps link on the Resource summary panel.
  3. In the Deployment Automation card, click the Deploy your app button to configure the Continuous Delivery feature.
  4. In the Select the deployment target panel, pick Cloud Foundry.
  5. Click the New + button to generate a new IBM Cloud API key and click OK in the pop-up window. Click Next at the bottom of the screen.
  6. In the Configure the DevOps toolchain panel, name the toolchain as username-python-microservice-cf (where cf refers to “Cloud Foundry”) and click Create.
  7. Wait until the Delivery Pipeline Status moves from In progress to Success. (You might need to refresh your browser window so that it shows the App URL.) delivery_pipeline
  8. In the Details card, copy the Source URL.
  9. In the Deployment Automation card, open the link containing the Name of your Continuous Delivery service.
  10. In the top-right corner, click Add tool and choose GitHub from the Version Control category in the catalog.
  11. Authorise access from IBM Cloud to your GitHub account by selecting the “I understand” checkbox.
  12. Under Repository type, choose Clone from the menu.
  13. Paste the Source URL you copied to Source repository URL.
  14. Choose your own username as Owner and name the repository as username-python-microservice. (The default Repository Name contains an unwanted -cf suffix. Consider removing it.)
  15. Make sure the Enable GitHub Issues and Track deployment of code changes boxes are selected.
  16. Click Create Integration.

Configure the Continuous Delivery Pipeline

  1. Log in to IBM Cloud.
  2. Click username-python-microservice in the Apps link on the Resource summary panel.
  3. In the Deployment Automation card, click the name of your Delivery Pipeline.
  4. In the next screen, click the gear icon in the Build Stage card, and then click Configure Stage.
  5. On the Input tab, choose the option in Git repository that leads to Github as Git URL.
  6. Save your work and run the Build Stage by clicking the forward arrow icon.
  7. Watch as the Build Stage completes and triggers the Deploy Stage.

3. Play with your Python application

In this section, you clone the repository to your local machine and become familiar with the structure of the application. After that, you add two new simple API calls to your application.

Clone the repository

  1. Log in to GitHub.
  2. Choose username-python-microservice from the Repositories menu on the left.
  3. Click the green Clone or download button and copy the address starting with git@github.com.
  4. Open Visual Studio Code and summon the Command Palette with:
  5. Type clone and select Git: Clone from the list.
  6. Paste the git@github.com address you copied before and provide the destination as required.
  7. Open the newly cloned repository.

Test the example implementation

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Open the server/routes/index.py file.
  3. Read the @app.route('/') code block.
  4. Go to http://username-python-microservice.mybluemix.net/ and observe the response: congratulations
  5. Read the @app.route('/error404') code block.
  6. Go to http://username-python-microservice.mybluemix.net/error404 and observe the response.
  7. Open the server/routes/health.py file.
  8. Read the @app.route('/health') code block.
  9. Go to http://username-python-microservice.mybluemix.net/health and observe the response.
  10. Open the server/routes/swagger.py file.
  11. Read the @app.route('/swagger/api') code block.
  12. Go to http://username-python-microservice.mybluemix.net/swagger/api and observe the response.
  13. Read the @app.route('/explorer') code block.
  14. Go to http://username-python-microservice.mybluemix.net/explorer and observe the response.
  15. Click the /health endpoint and the Try it out! button.

Customize the example implementation

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Open the health.py file.
  3. Add the following code block and save the modifications:
     @app.route('/answer')
     def answer():
         answer = {'The Answer to Life the Universe and Everything': 42}
         return jsonify(answer)
    
  4. Open the Source Control panel in the left corner.
  5. Click the + icon next to health.py to Stage Changes.
  6. Write a meaningful commit message, such as Created '/answer' endpoint and click the icon.
  7. In the bottom left corner, click the counterclockwise arrow icon to Synchronize Changes.
  8. Track the deployment progress in the Continuous Delivery Pipeline until the Deploy Stage passes.
  9. Go to http://username-python-microservice.mybluemix.net/answer and observe the response.

Customize the example implementation again

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Open the health.py file.
  3. Add the following code block to health.py and save the modifications.
     @app.route('/answer/<number>')
     def CheckAnswerToLifeTheUniverseAndEverything(number):
         content = {
             'statement': 'The Answer to Life the Universe and Everything is ' + number + '.',
             'check': 'The statement is ' + str(number == '42') + '!'
         }
         return jsonify(content)
    
  4. Open the Source Control panel in the left corner.
  5. Click the + icon next to health.py to Stage Changes.
  6. Write a meaningful commit message, such as Created '/answer/<number>' endpoint and click the icon.
  7. In the bottom left corner, click the counterclockwise arrow to Synchronize Changes.
  8. Track the deployment progress in the Continuous Delivery Pipeline.
  9. Go to http://username-python-microservice.mybluemix.net/answer/42 and observe the response.
  10. Go to http://username-python-microservice.mybluemix.net/answer/24 and observe the response.

4. Develop a number-crunching application

In this section, you use a new Python library that automatically generates Swagger API documentation from your source code. Later, you add new services and routes to your application, adding new number-crunching features to the cloud-based API.

Generate Swagger API documentation

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Install the latest Flask-RESTX by adding flask-restx = "*" to the bottom of Pipfile.
  3. Delete the Pipfile.lock file to force pipenv to read your modified Pipfile.
  4. Open the server/__init__.py file, perform the following modifications, and save your work:
    • Add from flask_restx import Api to the import list at the top of the file.
    • Add the following line below the definition of the app object in line 6.
        api = Api(app, title='My first Python API', version='1.0', doc='/apidocs/', description='A number-crunching API')
      
  5. Create a new math.py file in server/routes/ with the following code:

     from server import api
     from flask import jsonify
     from flask_restx import Resource
    
     @api.route('/double/<int:number>')
     @api.doc(params={'number': 'Number to be doubled.'}, description='This method doubles the input.')
     class DoubleNumber(Resource):
         def get(self, number):
             return jsonify(result=2 * number)
    
  6. Commit your changes to the Pipfile, server/__init__.py and math.py files.
  7. Sync your commits with the GitHub repository.
  8. Track the deployment progress in the Continuous Delivery Pipeline.

Explore the Swagger API documentation

  1. Go to http://username-python-microservice.mybluemix.net/apidocs/.
  2. Observe how the header in the top reflects the line recently added to server/__init__.py.
  3. Click on the Default namespace and the /double/{number} endpoint.
  4. Observe how the blue card reflects the contents of server/routes/math.py.
  5. Try to guess from the documentation the purpose and data input format of this method.
  6. Click Try it out, enter an input in the text box, and click Execute.
  7. Try several values for the required field and observe the response:
    • A string: word
    • A symbol: +
    • A floating point number: 1.5
    • An integer number: 42
  8. Go to http://username-python-microservice.mybluemix.net/double/42 and observe the response.

Create number-crunching services

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Install the latest NumPy by adding numpy = "*" to the bottom of Pipfile.
  3. Create a new file inside server/services/ and name it math_services.py.
  4. Start math_services.py by importing the Python module you have just added to Pipfile:
     import numpy as np
    
  5. Implement a service that converts a string with comma-separated integer values to a Python list of integers. Hint: Copy csv_to_list() from app/math_services.py.
  6. Implement a service that converts a list of integer values to an integer NumPy array. Hint: Copy list_to_array() from app/math_services.py.
  7. Implement a service that calculates the mean of a list of integer values. Hint: Copy calculate_mean() from app/math_services.py.
  8. Commit your changes to the Pipfile and math_services.py files.
  9. Sync your commits with the GitHub repository.
  10. Track the deployment progress in the Continuous Delivery Pipeline.

Configure Python deployment environment

  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Create a new file in the main repository directory named runtime.txt by clicking the New File button.
  3. Check the latest Python buildpack version available at IBM Cloud.
  4. Add python-X.Y.Z to runtime.txt reflecting the latest available version (for example: python-3.6.4).
  5. Commit your changes to the runtime.txt file.
  6. Sync your commits with the GitHub repository.
  7. Track the deployment progress in the Continuous Delivery Pipeline.
  1. Open the username-python-microservice repository in Visual Studio Code.
  2. Open math.py and remove the @api.route('/double/<int:number>') code block.
  3. Add from server.services import math_services as ms to the import list at the top of the file.
  4. Add below my_list = list() to create an empty Python list.
  5. Create an API method that prints the content of the list. Hint: Copy lines 9-13 from app/custom-math.py.
  6. Create an API method that resets the content of the list. Hint: Copy lines 16-21 in app/custom-math.py.
  7. Create an API method that reverses the content of the list. Hint: Copy lines 24-29 in app/custom-math.py.
  8. Create an API method that sorts the content of the list. Hint: Copy lines 32-37 in app/custom-math.py.
  9. Create an API method that extends the content of the list. Hint: Copy lines 40-46 in app/custom-math.py.
  10. Create an API method that replaces the content of the list. Hint: Copy lines 49-56 in app/custom-math.py.
  11. Create an API method that calculates the mean of the content of the list. Hint: Copy lines 59-64 in app/custom-math.py.
  12. Place your mouse cursor over csv_to_list() and calculate_mean() to read the documentation.
  13. In the end, your file should look like app/custom-math.py.
  14. Commit your changes to the math.py file.
  15. Sync your commits with the GitHub repository.
  16. Track the deployment progress in the Continuous Delivery Pipeline. cf_build_stages

5. Containerise the Python application

In this section, you replace the Cloud Foundry-based delivery pipeline by one that uses Kubernetes as a deployment strategy. You create and configure the new Kubernetes-based pipeline and, optionally, learn how to perform the Kubernetes deployment manually.

Set up a private registry namespace

  1. Log in to IBM Cloud.
  2. In the top-left corner, click the three horizontal lines to open the IBM Cloud menu.
  3. Select Kubernetes from the menu.
  4. On the Registry tab, click the Namespaces card.
  5. Click Create + on the right side.
  6. In the Create namespace screen, use your username as name.
  7. Click Create.

Create a new Kubernetes-based toolchain

  1. Log in to IBM Cloud.
  2. Click username-python-microservice in the Apps link on the Resource summary panel.
  3. Click Disconnect toolchain in the top-right corner of the Deployment Automation card to remove the old Cloud Foundry-based toolchain.
  4. Click Deploy your app to re-enable the Continuous Delivery feature.
  5. In the Choose a deployment environment panel, click Deploy to Kubernetes and hit Next.
  6. In the Configure toolchain panel, name the toolchain username-python-microservice-k8s (where k8s refers to “Kubernetes”) and click Create.
  7. In the Deployment Automation card, click the name of your toolchain.
  8. On the top right, click Add a Tool and choose GitHub from the catalog.
  9. Click Authorize to authorise access from IBM Cloud to your GitHub account.
  10. Under Repository type, choose Link from the menu.
  11. Select the username-python-microservice repository from the Repository URL menu.
  12. Make sure the Enable GitHub Issues and Track deployment of code changes boxes are selected.
  13. Click Create Integration.

Deploy a container with Continuous Delivery Pipeline

  1. Log in to IBM Cloud.
  2. Click username-python-microservice in the Apps link on the Resource summary panel.
  3. In the Deployment Automation card, click the name of your Delivery Pipeline.
  4. In the next screen, click the gear icon in the Build Stage card and then click Configure Stage.
  5. In the Input tab, choose the option in Git repository that leads to Github as Git URL.
  6. Save your changes and run the Build Stage by clicking the forward arrow icon. k8s_build_stages

Access the containerised application

  1. Log in to IBM Cloud.
  2. Click username-python-microservice in the Apps link on the Resource summary panel.
  3. In the Deployment Automation card, click the name of your Delivery Pipeline.
  4. Track the progress of your latest commit from the Build Stage to the Deploy Stage.
  5. Click the View logs and history link in the Deploy Stage card.
  6. After a successful deploy, at the very bottom of the page, you will find a link like http://IP_ADRESS:PORT.
  7. Click the link and add /apidocs/ to the URL in order to open the Swagger API documentation.

Deploy a container manually using the command-line

  1. Open Visual Studio Code and summon the Command Palette with
  2. Type terminal and choose View: Toggle Integrated Terminal.
  3. In the terminal window, issue ibmcloud login --sso and follow the on-screen instructions to log in to IBM Cloud.
  4. Execute ibmcloud ks init, then ibmcloud cr login. If the second command fails, make sure Docker is running locally.
  5. Configure your Kubernetes cluster by executing the command
     ibmcloud ks cluster config --cluster <username>-cluster
    
  6. Create a Kubernetes namespace.
     kubectl create namespace <username>
    
  7. Configure your kubectl environment.
     kubectl config set-context <username>-cluster --cluster=<username>-cluster --namespace=<username>
    
  8. Export your image registry secrets.
     kubectl get secret bluemix-default-secret-regional -n default -o yaml | sed 's/default/<username>/g' | kubectl create -f -
     kubectl get secret bluemix-default-secret-international -n default -o yaml | sed 's/default/<username>/g' | kubectl create -f -
    
  9. Take note of the Repository URL and image Tag number in the output of ibmcloud cr images.
  10. Create a Pod based on your image and expose it with a NodePort service.
     kubectl run <username>-python-microservice --image=<repository>:<tag> --restart=Never --image-pull-policy=Always --overrides='{ "apiVersion": "v1", "spec": { "imagePullSecrets": [{"name": "bluemix-<username>-secret-regional"}] } }'
     kubectl expose pod <username>-python-microservice --port=3000 --type=NodePort --name=<username>-python-microservice
    
  11. Take note of the Public IP from the output of ibmcloud ks workers --cluster <username>-cluster.
  12. Take note of the secondary Port number (after the :) in the output of kubectl get all.
  13. Open a browser window and access <http://PUBLIC_IP:PORT/apidocs/>.

Summary

Congratulations. You have now deployed your own Python application to IBM Cloud using two deployment modes: Cloud Foundry apps and Kubernetes pods. You also learned how to document your API using the Swagger specification. If you want to learn more ways to deploy your Python application in the cloud, explore Python on IBM developer.