Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Invoking serverless functions through a message broker

Many legacy applications drive workload through RabbitMQ queues, whereby distributed sub-components exchange messages to collaboratively fulfill the applications requirements. With the advent of serverless computing, an alternative mechanism for distributed applications is now available.

IBM Cloud Functions (backed by Apache Openwhisk) is an implementation of the serverless paradigm, in which individual functions are invoked on demand. However, porting legacy RabbitMQ applications to IBM Cloud Funtions poses a challenge. How can messages contained in RabbitMQ queues be dispatched to Cloud Functions?

RabbitWhisker solves this problem, by treating messages as events and invoking an appropriate IBM Cloud Function to handle the messages.

RabbitWhisker is an open source component on GitHub, that provides:

  • RabbitWhisker itself, which listens to a RabbitMQ queue and invokes Cloud Functions
  • A sample Cloud Function that receives messages and sends replies
  • A sample client application that drives the workload

Learning objectives

After you complete this tutorial, you will know how to:

  • Provision RabbitMQ on IBM Cloud
  • Obtain credentials for RabbitMQ and IBM Cloud Functions
  • Build and start a Docker container
  • Build and deploy an IBM Cloud Function
  • Publish messages to a RabbitMQ queue
  • Receive messages from a RabbitMQ queue

Prerequisites

To complete this tutorial, you need an IBM Cloud account; you can request one here. Also, the following software or services must be installed or provisioned:

  • A RabbitMQ service must be provisioned on IBM Cloud
  • Service credentials for this RabbitMQ service must be created and be available
  • The IBM Cloud CLI must be installed
  • An IBM Cloud organization and space must be targeted (see bx target command)
  • Docker 18.09 or higher must be installed
  • Python 3.5 or higher must be installed

Estimated time

It will take about an hour to complete this tutorial.

Steps

  1. Review the design of the sample application
  2. Specify the credentials
  3. Test your local environment
  4. Build and deploy RabbitWhisker
  5. Run the sample client application
1

Review the design of the sample application

Because IBM Cloud Functions are invoked on demand, there is no definitive concept of an “always-up” Cloud Function waiting to process workload. IBM Cloud Functions can be triggered on a cron-like schedule, but a schedule is not necessarily responsive to the peaks and troughs of typical application workloads.

RabbitWhisker is designed to stay up, for immediate response to workload. It uses a Dockerfile to build a Docker image, which can be specified in a docker run command or in a Kubernetes pod.

When RabbitWhisker is running, it stays up indefinitely, subscribing to a RabbitMQ queue, listening for messages, and invoking a Cloud Function in response to received messages. The contents of each message that is received is passed as a parameter to a Cloud Function.

To respond to increasing workload, RabbitWhisker has a built-in scalability feature. You can establish multiple connections to RabbitMQ, with each connection receiving a dedicated thread. Because all threads subscribe to the same RabbitMQ queue, RabbitWhisker can handle higher throughput of messages on the queue.

The sample client application that we included in the RabbitWhisker open source component and the sample Cloud Function show how to drive the workload (messages) and respond to the original messages. The sample client application assumes that a RabbitMQ instance is already provisioned.

A request-reply message flow pattern is shown in the following Figure.

Figure 1. RabbitWhisker Messaging Flow

Message flow for RabbitWhisker

A description of the messaging flow is as follows:

  1. The client application publishes a number of messages to the feed queue (see the table in the next step for a description of each variable).
  2. The running RabbitWhisker container, which has already subscribed to the same feed queue, receives a message.
  3. A Cloud Function is invoked, passing the message contents as a parameter to the function.
  4. RabbitWhisker continues to listen for more messages.
  5. The invoked Cloud Function processes the message and replies to the reply queue (this step is optional, but helps to indicate an end-to-end message flow).
  6. The client application, now subscribed to the reply queue, receives the reply messages.

Certain client applications will not likely require a reply to some messages. In this case, the handling Cloud Function, need not publish a reply message.

2

Specify credentials

With secure cloud-based service instances, the availability of access credentials is required. In this case, credentials for both a RabbitMQ instance and IBM Cloud Functions are necessary. After you retrieve these credentials, which are described in the following table, you need to add them to a local script (env.sh):

Name Type Description
RABBIT_BROKER URL The base URL for the RabbitMQ service
RABBIT_PORT String The port number for the RabbitMQ service
RABBIT_VHOST String The virtual host, often ‘/’
RABBIT_USER String Username for the RabbitMQ service
RABBIT_PWD String Password for the RabbitMQ service
CERT String Path to the certificate PEM file for the RabbitMQ service
REPLY_QUEUE String Queue to retrieve messages from, used by the client application
FEED_QUEUE String Queue to retrieve messages from, used by RabbitWhisker
WHISK_ACTION String The IBM Cloud Function name to invoke on receipt of a message
WHISK_SPACE String The full namespace for IBM Cloud Function invocation
WHISK_URL URL The base URL for IBM Cloud Functions
WHISK_AUTH String Authentication token for accessing IBM Cloud Functions

Specify RabbitMQ credentials

To populate the RabbitMQ variables, connection details to a RabbitMQ service are required. Two different RabbitMQ services are available on IBM Cloud:

While both services provide a RabbitMQ cloud service, CloudAMQP has a free plan, whereas Messages for RabbitMQ does not. Choose the most appropriate service for your needs.

The REPLY_QUEUE and FEED_QUEUE environment variables are queue names; for example, FEED_QUEUE might be ‘requests’ and REPLY_QUEUE might be ‘replies’.

Specify credentials for the Messages for RabbitMQ service

After this service is initially provisioned, the password for the admin user must be changed. You can change this password on the IBM Cloud Dashboard for the service, on the Settings tab, in the Change Password panel, or you can change it with the IBM Cloud CLI as follows:

bx plugin install cloud-databases
bx cdb user-password "YOUR RABBITMQ SERVICE" admin <newpassword>

The remaining credentials are obtained from the IBM Cloud by using the IBM Cloud CLI plug-in.

bx cdb deployment-connections "YOUR RABBITMQ SERVICE"

This command will output credentials to the terminal. See the AMQPS output for the relevant RabbitMQ credentials, and use the admin user with the password that you already created. The port number must be the AMQPS port, not an HTTPS port.

The SSL certificate for the service can be retrieved as follows:

bx cdb deployment-cacert "YOUR RABBITMQ SERVICE" > cert.pem

Specify CloudAMQP credentials

The service credentials are available immediately after the service is provisioned and can be retrieved directly from the provisioned service page. This service always uses the default RabbitMQ port of 5672.

Specify IBM Cloud Functions credentials

Now details for the IBM Cloud Functions service are retrieved, which are available by using these commands:

bx plugin install cloud-functions
bx wsk property get

The output will look similar to:

whisk auth              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx:xxxxxx
whisk API host          eu-de.functions.cloud.ibm.com
whisk API version       v1
whisk namespace
whisk CLI version       2019-02-04T22:28:01+00:00
whisk API build         2019-03-07T16:26:34Z
whisk API build number  whisk-build-11573

As before, these values are added to the env.sh script.

The WHISK_SPACE variable is a combination of bx organisation + whisk namespace + bx space, where bx organisation and bx space are found from running the bx target command, and whisk namespace is found from the bx wsk property get command (whisk namespace in the previous output). For example:

WHISK_SPACE=myorg_myspace

The WHISK_URL variable is a combination of whisk API host + /api/ + whisk API version, from the bx wsk property get command. For example:

WHISK_URL=https://eu-de.functions.cloud.ibm.com/api/v1

3

Test local environment

At this stage, you have an env.sh script with the required environment variables specified.

Now, you need to test the connection details in these variables. But first, you need to install the appropriate Python package dependencies, if they’re not available already.

pip3 install -r invoker/requirements.txt

If the credentials are correct, the test should successfully connect to the RabbitMQ service.

./test_connect.sh

If this fails, the credentials might not be specified correctly.

4

Build and deploy

Now, you can build and deploy RabbitWhisker and the Cloud Function. First, deploy the Cloud Function. This function is invoked by RabbitWhisker, and upon invocation, publishes a response to a RabbitMQ queue, the REPLY_QUEUE.

cd action
./install.sh
cd ..

You should see output similar to the following:

  adding: __main__.py (deflated 58%)
  adding: messenger/rabbitmq.py (deflated 69%)
ok: updated action RabbitFeedEndpoint

Next, you need to build the docker image that will run RabbitWhisker:

docker build -t rabbitmq_feed:latest -t rabbitmq_feed:$(cat invoker/VERSION) .

Then, start RabbitWhisker:

./start_listener.sh

The logs from the running container can be viewed with this command:

docker logs <container-id>

5

Run the sample client application

The sample client application shows how to use the RabbitMQ Python class to do the following:

  • Establish a connection to RabbitMQ
  • Publish a number of messages to the feed queue (specified in env.sh)
  • Subscribe to a reply queue to receive replies

You now have a deployed IBM Cloud Function that is ready to be invoked as well as a running RabbitWhisker that is waiting for messages. You can now start the sample client application, which publishes messsages to the FEED_QUEUE, and waits for the appropriate number of responses on the REPLY_QUEUE.

./round_trip.sh

You should see output similar to the following output:

2019-03-19 12:07:04.129 INFO   root 139834587285248 :: Starting...
2019-03-19 12:07:04.129 INFO   root 139834587285248 :: Sending requests to: requests
.
.
.
2019-03-19 12:07:04.886 INFO   root 139834587285248 :: Dispatched messages: 10
2019-03-19 12:07:04.886 INFO   root 139834587285248 :: Now wait for replies on: responses
.
.
.
2019-03-19 12:07:05.908 INFO   root 139834587285248 :: Received messages: 10
2019-03-19 12:07:05.908 INFO   root 139834587285248 :: Duration 1.78
2019-03-19 12:07:05.910 INFO   root 139834587285248 :: Done

Troubleshooting

Most problems arise as a result of incorrectly specified credentials for the IBM Cloud services. Double-checking these credentials is the first place to start. Also, make sure that the RabbitMQ port number is the AMQPS port, not the HTTPS port.

Often when you use RabbitMQ with multiple queues, it is easy to specify the wrong queue name when you run the sample application, which will result in an incorrect number (could be zero) of messages being received.

Summary and next steps

Now that you’ve seen how RabbitWhisker works, you can port your own RabbitMQ applications to the serverless paradigm using IBM Cloud Functions. If you use RabbitWhisker, we’d be happy to receive any feedback, enhancement requests, or GitHub pull requests.

Mark Purcell