Invoking serverless functions through a message broker
Use RabbitWhisker to drive workload to IBM Cloud Functions by using RabbitMQ
Many legacy applications drive workload through RabbitMQ queues, whereby distributed sub-components exchange messages to collaboratively fulfill the applications requirements. With the advent of serverless computing, an alternative mechanism for distributed applications is now available.
IBM Cloud Functions (backed by Apache Openwhisk) is an implementation of the serverless paradigm, in which individual functions are invoked on demand. However, porting legacy RabbitMQ applications to IBM Cloud Funtions poses a challenge. How can messages contained in RabbitMQ queues be dispatched to Cloud Functions?
RabbitWhisker solves this problem, by treating messages as events and invoking an appropriate IBM Cloud Function to handle the messages.
RabbitWhisker is an open source component on GitHub, that provides:
- RabbitWhisker itself, which listens to a RabbitMQ queue and invokes Cloud Functions
- A sample Cloud Function that receives messages and sends replies
- A sample client application that drives the workload
After you complete this tutorial, you will know how to:
- Provision RabbitMQ on IBM Cloud
- Obtain credentials for RabbitMQ and IBM Cloud Functions
- Build and start a Docker container
- Build and deploy an IBM Cloud Function
- Publish messages to a RabbitMQ queue
- Receive messages from a RabbitMQ queue
To complete this tutorial, you need an IBM Cloud account; you can request one here. Also, the following software or services must be installed or provisioned:
- A RabbitMQ service must be provisioned on IBM Cloud
- Service credentials for this RabbitMQ service must be created and be available
- The IBM Cloud CLI must be installed
- An IBM Cloud organization and space must be targeted (see bx target command)
- Docker 18.09 or higher must be installed
- Python 3.5 or higher must be installed
It will take about an hour to complete this tutorial.
- Review the design of the sample application
- Specify the credentials
- Test your local environment
- Build and deploy RabbitWhisker
- Run the sample client application
Review the design of the sample application
Because IBM Cloud Functions are invoked on demand, there is no definitive concept of an “always-up” Cloud Function waiting to process workload. IBM Cloud Functions can be triggered on a cron-like schedule, but a schedule is not necessarily responsive to the peaks and troughs of typical application workloads.
RabbitWhisker is designed to stay up, for immediate response to workload. It uses a Dockerfile to build a Docker image, which can be specified in a
docker run command or in a Kubernetes pod.
When RabbitWhisker is running, it stays up indefinitely, subscribing to a RabbitMQ queue, listening for messages, and invoking a Cloud Function in response to received messages. The contents of each message that is received is passed as a parameter to a Cloud Function.
To respond to increasing workload, RabbitWhisker has a built-in scalability feature. You can establish multiple connections to RabbitMQ, with each connection receiving a dedicated thread. Because all threads subscribe to the same RabbitMQ queue, RabbitWhisker can handle higher throughput of messages on the queue.
The sample client application that we included in the RabbitWhisker open source component and the sample Cloud Function show how to drive the workload (messages) and respond to the original messages. The sample client application assumes that a RabbitMQ instance is already provisioned.
A request-reply message flow pattern is shown in the following Figure.
Figure 1. RabbitWhisker Messaging Flow
A description of the messaging flow is as follows:
- The client application publishes a number of messages to the feed queue (see the table in the next step for a description of each variable).
- The running RabbitWhisker container, which has already subscribed to the same feed queue, receives a message.
- A Cloud Function is invoked, passing the message contents as a parameter to the function.
- RabbitWhisker continues to listen for more messages.
- The invoked Cloud Function processes the message and replies to the reply queue (this step is optional, but helps to indicate an end-to-end message flow).
- The client application, now subscribed to the reply queue, receives the reply messages.
Certain client applications will not likely require a reply to some messages. In this case, the handling Cloud Function, need not publish a reply message.
With secure cloud-based service instances, the availability of access credentials is required. In this case, credentials for both a RabbitMQ instance and IBM Cloud Functions are necessary. After you retrieve these credentials, which are described in the following table, you need to add them to a local script (
|RABBIT_BROKER||URL||The base URL for the RabbitMQ service|
|RABBIT_PORT||String||The port number for the RabbitMQ service|
|RABBIT_VHOST||String||The virtual host, often ‘/’|
|RABBIT_USER||String||Username for the RabbitMQ service|
|RABBIT_PWD||String||Password for the RabbitMQ service|
|CERT||String||Path to the certificate PEM file for the RabbitMQ service|
|REPLY_QUEUE||String||Queue to retrieve messages from, used by the client application|
|FEED_QUEUE||String||Queue to retrieve messages from, used by RabbitWhisker|
|WHISK_ACTION||String||The IBM Cloud Function name to invoke on receipt of a message|
|WHISK_SPACE||String||The full namespace for IBM Cloud Function invocation|
|WHISK_URL||URL||The base URL for IBM Cloud Functions|
|WHISK_AUTH||String||Authentication token for accessing IBM Cloud Functions|
Specify RabbitMQ credentials
To populate the RabbitMQ variables, connection details to a RabbitMQ service are required. Two different RabbitMQ services are available on IBM Cloud:
While both services provide a RabbitMQ cloud service, CloudAMQP has a free plan, whereas Messages for RabbitMQ does not. Choose the most appropriate service for your needs.
FEED_QUEUE environment variables are queue names; for example,
FEED_QUEUE might be ‘requests’ and
REPLY_QUEUE might be ‘replies’.
Specify credentials for the Messages for RabbitMQ service
After this service is initially provisioned, the password for the
admin user must be changed.
You can change this password on the IBM Cloud Dashboard for the service, on the Settings tab, in the Change Password panel, or you can change it with the IBM Cloud CLI as follows:
bx plugin install cloud-databases bx cdb user-password "YOUR RABBITMQ SERVICE" admin <newpassword>
The remaining credentials are obtained from the IBM Cloud by using the IBM Cloud CLI plug-in.
bx cdb deployment-connections "YOUR RABBITMQ SERVICE"
This command will output credentials to the terminal. See the
AMQPS output for the relevant RabbitMQ credentials, and use the
admin user with the password that you already created. The port number must be the AMQPS port, not an HTTPS port.
The SSL certificate for the service can be retrieved as follows:
bx cdb deployment-cacert "YOUR RABBITMQ SERVICE" > cert.pem
Specify CloudAMQP credentials
The service credentials are available immediately after the service is provisioned and can be retrieved directly from the provisioned service page. This service always uses the default RabbitMQ port of 5672.
Specify IBM Cloud Functions credentials
Now details for the IBM Cloud Functions service are retrieved, which are available by using these commands:
bx plugin install cloud-functions bx wsk property get
The output will look similar to:
whisk auth xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx:xxxxxx whisk API host eu-de.functions.cloud.ibm.com whisk API version v1 whisk namespace whisk CLI version 2019-02-04T22:28:01+00:00 whisk API build 2019-03-07T16:26:34Z whisk API build number whisk-build-11573
As before, these values are added to the
WHISK_SPACE variable is a combination of
bx organisation +
whisk namespace +
bx space, where
bx organisation and
bx space are found from running the
bx target command, and
whisk namespace is found from the
bx wsk property get command (
whisk namespace in the previous output). For example:
WHISK_URL variable is a combination of
whisk API host +
whisk API version, from the
bx wsk property get command. For example:
Test local environment
At this stage, you have an
env.sh script with the required environment variables specified.
Now, you need to test the connection details in these variables. But first, you need to install the appropriate Python package dependencies, if they’re not available already.
pip3 install -r invoker/requirements.txt
If the credentials are correct, the test should successfully connect to the RabbitMQ service.
If this fails, the credentials might not be specified correctly.
Build and deploy
Now, you can build and deploy RabbitWhisker and the Cloud Function. First, deploy the Cloud Function. This function is invoked by RabbitWhisker, and upon invocation, publishes a response to a RabbitMQ queue, the
cd action ./install.sh cd ..
You should see output similar to the following:
adding: __main__.py (deflated 58%) adding: messenger/rabbitmq.py (deflated 69%) ok: updated action RabbitFeedEndpoint
Next, you need to build the docker image that will run RabbitWhisker:
docker build -t rabbitmq_feed:latest -t rabbitmq_feed:$(cat invoker/VERSION) .
Then, start RabbitWhisker:
The logs from the running container can be viewed with this command:
docker logs <container-id>
Run the sample client application
The sample client application shows how to use the RabbitMQ Python class to do the following:
- Establish a connection to RabbitMQ
- Publish a number of messages to the feed queue (specified in env.sh)
- Subscribe to a reply queue to receive replies
You now have a deployed IBM Cloud Function that is ready to be invoked as well as a running RabbitWhisker that is waiting for messages. You can now start the sample client application, which publishes messsages to the
FEED_QUEUE, and waits for the appropriate number of responses on the
You should see output similar to the following output:
2019-03-19 12:07:04.129 INFO root 139834587285248 :: Starting... 2019-03-19 12:07:04.129 INFO root 139834587285248 :: Sending requests to: requests . . . 2019-03-19 12:07:04.886 INFO root 139834587285248 :: Dispatched messages: 10 2019-03-19 12:07:04.886 INFO root 139834587285248 :: Now wait for replies on: responses . . . 2019-03-19 12:07:05.908 INFO root 139834587285248 :: Received messages: 10 2019-03-19 12:07:05.908 INFO root 139834587285248 :: Duration 1.78 2019-03-19 12:07:05.910 INFO root 139834587285248 :: Done
Most problems arise as a result of incorrectly specified credentials for the IBM Cloud services. Double-checking these credentials is the first place to start. Also, make sure that the RabbitMQ port number is the AMQPS port, not the HTTPS port.
Often when you use RabbitMQ with multiple queues, it is easy to specify the wrong queue name when you run the sample application, which will result in an incorrect number (could be zero) of messages being received.
Summary and next steps
Now that you’ve seen how RabbitWhisker works, you can port your own RabbitMQ applications to the serverless paradigm using IBM Cloud Functions. If you use RabbitWhisker, we’d be happy to receive any feedback, enhancement requests, or GitHub pull requests.