Andy Emmett
Published on 16/11/2018 / Updated on 21/11/2018
Overview
Hello, and welcome to my very first post on the subject of Queue Manager Clusters. The aim of this three part article, is to provide some knowledge regarding the integration of on-premise queue managers that are part of a cluster, alongside queue managers hosted in the IBM MQ on Cloud service.
Scope : What is and isn’t covered
It’s important to say up front, that the target audience for this series of posts, is primarily those with a working understanding of clustering. If however, you are here looking to pick up some knowledge, then familiarity with the terminology would be handy. Information about IBM Cloud accounts and setting them up, is not detailed here and thus prior knowledge is assumed. A good starting point if required, is the IBM Cloud home page. Otherwise, the article is broken down into several parts with some walk-through of the scenario. Here is what will be covered …
- Creating a new cluster queue manager in IBM Cloud
- Creating a Secure Gateway service endpoint to allow connection to on-premises queue managers / Installing a Secure Gateway Client
- Adding the Cloud queue manager to queue manager cluster and updating the existing configuration
Please note, it is beyond the scope of these article to provide a prescribed set of steps or commands that you can use to create your own queue managers, secure gateways or alter you existing configurations. What has been documented here should be reviewed with respect to your own requirements and modified appropriately to achieve the integration you require. This is especially true of your security requirements for queue managers. Also, please be aware that the images and designs of the pages shown in this article are subject to change without notice.
Introduction
To get us started, we are going to use a fictitious scenario that readers will hopefully be able to relate to.
Perhaps you have been thinking about or indeed plans are already afoot to bring your enterprise messaging applications into a cloud environment. Imagine you already have applications which are using queue manager clusters to distribute messages, and now you’d like your cloud applications to communicate with on-premise back-end services which are not planned to be moved (yet at least). What you want to know is, How do I do that? Essentially, this set of articles will illustrate that with careful use of multiple connection names supplied in cluster receiver channels, combined with use of the Secure Gateway service to allow connections back into the on-premises network, an MQ queue manager running in the cloud can easily participate in and with an existing on-premises MQ cluster configuration. The key aspect is the specification of multiple connection names of the cluster receiver channels on the on-premise queue managers. This is detailed in Part Three.
It’s said that a picture paints a thousand words. Looking at this one however, there may be some terms that are unfamiliar and have no relation with queue manager clusters as you know it. Also, you may notice that there are many lines and arrows in the cluster topology, that should be flowing in the opposite direction to those that are there. We won’t worry too much about those and for now focus on the background for the diagram that will help throughout the series. And so, allow me to present, the ‘big picture’.
Fig 1. The Big Picture
Background
Our fictional scenario begins with some-bank.com. They are developing a new application for their Auto Teller Machines (ATM) with a new web based user interface. This will be served up by application servers based on the Liberty for Java run-time, and hosted in IBM Cloud. The idea is much the the same as it was originally, user interactions will send messages for processing to the existing proven back end services (hosted on-premise). These back end services take in requests and provide replies using messaging. In this case, IBM MQ in a clustered configuration.
Precisely because IBM MQ because provides a secure, reliable communication across different network zones, a decision was made to have a new queue manager run alongside the Liberty for Java runtime in IBM Cloud. This would allow this and future cloud application migrations, integrate better with the back end services that already exist.
Knowing that a lot of these services already employ clustering, its not yet clear, how to implement this. However, that does not prevent the message flow from being drawn out.
Fig 2. Application Message Flow
Creating A Queue Manager
If you are already familiar with creating queue managers in IBM Cloud, you may want to skip ahead to the Testing Connectivity section.
Creating a queue manager in IBM Cloud is to be the first task. IBM Cloud has a catalog of services which users can provision to build their applications in the cloud. So we’ll assume that (if you are actually following along) you’ve logged in and have navigated to the MQ Service in the IBM Cloud catalog and have provisioned an MQ Service instance for your organization. You’ll see a page which (at the time of writing) looks like the following
Fig 3. A Provisioned MQ Service (called MQ-ANDY)
Selecting a plan
Creating a queue manager in IBM Cloud is simple. You will need a name for your new queue manager and an idea of your messaging requirements in terms of connectivity and throughput. A simple page is presented when you begin the provision wizard for MQ that asks the name and the size for your queue manager. For each option in the Size selection panel, there is a description providing an idea of the throughput and connectivity that can be achieved with the queue manager.
You can add more queue managers if you need to scale up, of perhaps you’ll select a large queue manager plan to begin with, knowing that other applications will soon be moving their workloads to it. For our discussion here, we would be using the Large plan. This is because in our scenario here, we are providing a web application with potential for a lot of concurrent users interacting at any given time, and thus, the messaging requirements will need to be adequate. However, we have shown the Trial version being used. (If you’re trying this yourself for demo purposes you may wish to select a Trial or Small instance in order to limit your expenses as below)
Fig 4. Example ‘Plan Selection’
Deployment
Once you’ve selected your plan and clicked on the Create button, deployment of your new queue manager in IBM Cloud begins. The deployment process finds a location for the queue manager to run before creating it. In a few minutes, the queue manager will be ready for use.
Fig 5. Queue Manager being deployed
Up and Running
When your queue manager is ready, it will be started automatically for you, there’s no need to launch a command prompt each time you want to start it as with traditional servers. IBM cloud manages the queue manager on your behalf or helps you manage it through the user interface. In our case, we now have a queue manager called CLOUD1. The queue manager is created with a single listener and this too is automatically managed for you. Please note, no other listeners can be created using the various administration tools such as the MQ Console, MQ Explorer, remote runmqsc or your own Programmable Command Format (PCF) client applications.
Fig 6. A running Queue Manager
Important information
Because your queue manager is a dynamically created resource in IBM Cloud, hostnames and ports will have been allocated automatically. As a result, you will need to make a note of these details for later use (though they are available to obtain at any time) with channel definitions. Remember, channel definitions (in particular CLUSRCVR channels) are the key to queue managers joining and participating in cluster configurations. You can click on the table row which represents your queue manager (such as CLOUD1 in the picture above) to see these Connection Details details.
A new page is presented to you showing the hostname and associated port which other queue managers can use to send messages to our CLOUD1 queue manager. Please note, a single listener is created in the queue manager. We would recommend that you download the Connection information file but clicking the button in the top-right of the screenshot (choice of plain text for humans, or JSON formatted) so that it can be put in a safe place for when it is needed in the future.
Fig 7. Important connection information for a provisioned queue manager
Testing Connectivity
Connection Information in MQExplorer
Now might be a good time to test that you’re able to connect to the new cloud queue manager. For instance, you will likely already have an installation of MQExplorer that’s used to administer the queue managers hosted on the IBM MQ Appliances. If so, start the Add Queue Manager wizard and supply the details of the hostname and port from your queue manager.
Fig 8. Queue Manager connection details wizard page
Credential Information
When presented with the wizard page asking for credentials, naturally you will need to supply your own credentials (IBM Cloud ID) for it to successfully connect. For the password you need to supply an apiKey. This key was originally generated for you when you created your user account. Users typically download the key and save it in a safe place for later use. (The key can be reset if you have forgotten it by accessing you account setting once you are logged into IBM Cloud). Complete the wizard by adding other information as required.
Important Note : Please ensure that the “User identification compatibility mode” is unchecked as shown in the diagram below.
If User identification compatibility mode remains checked, this will send a truncated portion of the password (apiKey) to the queue manager. This will result in a Not Authorized error.
Fig 9. MQExplorer Add Queue Manager wizard
Security Consideration
Once connected, you should review the Channel Authentication Records of CLOUD1. To start with, the queue manager is as you would expect, ultra secure. No connections are permitted until the administrator adds and removes their required connectivity rules. For the purpose of this article (and simplicity), I removed the predefined Block User List and Address Map rules that prevent total access to the queue manager via the other predefined server connection channels, namely CLOUD.ADMIN.SVRCONN and CLOUD.APP.SVRCONN
This article does not endorse this behavior, it is for example purposes only. Instead, we recommend planning of your own security requirements in order to implement a secure environment
Fig 10. Authentication records global blocking rules
Summary
So far we have only described the creation of a queue manager called CLOUD1. We are going to be using this from now on with the other articles in this series. We continue in Part Two by discussing two of the new aspects shown in the diagram at the beginning of the article, (see Fig 1. the Secure Gateway and the Secure Gateway Client.