Time is running out! Call for Code submissions due July 29 ›
Mark Purcell | Updated June 20, 2019 - Published June 19, 2019
Many legacy applications drive workload through RabbitMQ queues, whereby distributed sub-components exchange messages to collaboratively fulfill the applications requirements. With the advent of serverless computing, an alternative mechanism for distributed applications is now available.
IBM Cloud Functions (backed by Apache Openwhisk) is an implementation of the serverless paradigm, in which individual functions are invoked on demand. However, porting legacy RabbitMQ applications to IBM Cloud Funtions poses a challenge. How can messages contained in RabbitMQ queues be dispatched to Cloud Functions?
RabbitWhisker solves this problem, by treating messages as events and invoking an appropriate IBM Cloud Function to handle the messages.
RabbitWhisker is an open source component on GitHub, that provides:
After you complete this tutorial, you will know how to:
To complete this tutorial, you need an IBM Cloud account; you can request one here. Also, the following software or services must be installed or provisioned:
It will take about an hour to complete this tutorial.
Because IBM Cloud Functions are invoked on demand, there is no definitive concept of an “always-up” Cloud Function waiting to process workload. IBM Cloud Functions can be triggered on a cron-like schedule, but a schedule is not necessarily responsive to the peaks and troughs of typical application workloads.
RabbitWhisker is designed to stay up, for immediate response to workload. It uses a Dockerfile to build a Docker image, which can be specified in a docker run command or in a Kubernetes pod.
When RabbitWhisker is running, it stays up indefinitely, subscribing to a RabbitMQ queue, listening for messages, and invoking a Cloud Function in response to received messages. The contents of each message that is received is passed as a parameter to a Cloud Function.
To respond to increasing workload, RabbitWhisker has a built-in scalability feature. You can establish multiple connections to RabbitMQ, with each connection receiving a dedicated thread. Because all threads subscribe to the same RabbitMQ queue, RabbitWhisker can handle higher throughput of messages on the queue.
The sample client application that we included in the RabbitWhisker open source component and the sample Cloud Function show how to drive the workload (messages) and respond to the original messages. The sample client application assumes that a RabbitMQ instance is already provisioned.
A request-reply message flow pattern is shown in the following Figure.
Figure 1. RabbitWhisker Messaging Flow
A description of the messaging flow is as follows:
Certain client applications will not likely require a reply to some messages. In this case, the handling Cloud Function, need not publish a reply message.
With secure cloud-based service instances, the availability of access credentials is required. In this case, credentials for both a RabbitMQ instance and IBM Cloud Functions are necessary. After you retrieve these credentials, which are described in the following table, you need to add them to a local script (env.sh):
To populate the RabbitMQ variables, connection details to a RabbitMQ service are required. Two different RabbitMQ services are available on IBM Cloud:
While both services provide a RabbitMQ cloud service, CloudAMQP has a free plan, whereas Messages for RabbitMQ does not. Choose the most appropriate service for your needs.
The REPLY_QUEUE and FEED_QUEUE environment variables are queue names; for example, FEED_QUEUE might be ‘requests’ and REPLY_QUEUE might be ‘replies’.
After this service is initially provisioned, the password for the admin user must be changed.
You can change this password on the IBM Cloud Dashboard for the service, on the Settings tab, in the Change Password panel, or you can change it with the IBM Cloud CLI as follows:
bx plugin install cloud-databases
bx cdb user-password "YOUR RABBITMQ SERVICE" admin <newpassword>
The remaining credentials are obtained from the IBM Cloud by using the IBM Cloud CLI plug-in.
bx cdb deployment-connections "YOUR RABBITMQ SERVICE"
This command will output credentials to the terminal. See the AMQPS output for the relevant RabbitMQ credentials, and use the admin user with the password that you already created. The port number must be the AMQPS port, not an HTTPS port.
The SSL certificate for the service can be retrieved as follows:
bx cdb deployment-cacert "YOUR RABBITMQ SERVICE" > cert.pem
The service credentials are available immediately after the service is provisioned and can be retrieved directly from the provisioned service page. This service always uses the default RabbitMQ port of 5672.
Now details for the IBM Cloud Functions service are retrieved, which are available by using these commands:
bx plugin install cloud-functions
bx wsk property get
The output will look similar to:
whisk auth xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx:xxxxxx
whisk API host eu-de.functions.cloud.ibm.com
whisk API version v1
whisk CLI version 2019-02-04T22:28:01+00:00
whisk API build 2019-03-07T16:26:34Z
whisk API build number whisk-build-11573
As before, these values are added to the env.sh script.
The WHISK_SPACE variable is a combination of bx organisation + whisk namespace + bx space, where bx organisation and bx space are found from running the bx target command, and whisk namespace is found from the bx wsk property get command (whisk namespace in the previous output). For example:
bx wsk property get
The WHISK_URL variable is a combination of whisk API host + /api/ + whisk API version, from the bx wsk property get command. For example:
whisk API host
whisk API version
At this stage, you have an env.sh script with the required environment variables specified.
Now, you need to test the connection details in these variables. But first, you need to install the appropriate Python package dependencies, if they’re not available already.
pip3 install -r invoker/requirements.txt
If the credentials are correct, the test should successfully connect to the RabbitMQ service.
If this fails, the credentials might not be specified correctly.
Now, you can build and deploy RabbitWhisker and the Cloud Function. First, deploy the Cloud Function. This function is invoked by RabbitWhisker, and upon invocation, publishes a response to a RabbitMQ queue, the REPLY_QUEUE.
You should see output similar to the following:
adding: __main__.py (deflated 58%)
adding: messenger/rabbitmq.py (deflated 69%)
ok: updated action RabbitFeedEndpoint
Next, you need to build the docker image that will run RabbitWhisker:
docker build -t rabbitmq_feed:latest -t rabbitmq_feed:$(cat invoker/VERSION) .
Then, start RabbitWhisker:
The logs from the running container can be viewed with this command:
docker logs <container-id>
The sample client application shows how to use the RabbitMQ Python class to do the following:
You now have a deployed IBM Cloud Function that is ready to be invoked as well as a running RabbitWhisker that is waiting for messages. You can now start the sample client application, which publishes messsages to the FEED_QUEUE, and waits for the appropriate number of responses on the REPLY_QUEUE.
You should see output similar to the following output:
2019-03-19 12:07:04.129 INFO root 139834587285248 :: Starting...
2019-03-19 12:07:04.129 INFO root 139834587285248 :: Sending requests to: requests
2019-03-19 12:07:04.886 INFO root 139834587285248 :: Dispatched messages: 10
2019-03-19 12:07:04.886 INFO root 139834587285248 :: Now wait for replies on: responses
2019-03-19 12:07:05.908 INFO root 139834587285248 :: Received messages: 10
2019-03-19 12:07:05.908 INFO root 139834587285248 :: Duration 1.78
2019-03-19 12:07:05.910 INFO root 139834587285248 :: Done
Most problems arise as a result of incorrectly specified credentials for the IBM Cloud services. Double-checking these credentials is the first place to start. Also, make sure that the RabbitMQ port number is the AMQPS port, not the HTTPS port.
Often when you use RabbitMQ with multiple queues, it is easy to specify the wrong queue name when you run the sample application, which will result in an incorrect number (could be zero) of messages being received.
Now that you’ve seen how RabbitWhisker works, you can port your own RabbitMQ applications to the serverless paradigm using IBM Cloud Functions. If you use RabbitWhisker, we’d be happy to receive any feedback, enhancement requests, or GitHub pull requests.
May 27, 2019
Back to top