This post is long overdue, but better late than never! This post will show how a user can view Keystone CADF notifications in both Ceilometer and RabbitMQ. To start, let’s go over some basic concepts first.


Events: An event is triggered when a user accesses certain APIs (i.e.: an administrator creates a new user) Auditing: We want to be able to figure out who did what, from where, and to what resource. Ceilometer: OpenStack’s Telemetry service – provides support for metering, alarms, notifications, and other services. Keystone: OpenStack’s Identity service – provides authentication, authorization and identity management CADF: The Cloud Auditing Data Federation (CADF) standard defines a full event model anyone can use to fill in the essential data needed to certify, self-manage and self-audit application security in cloud environments. pyCADF: A python implementation of CADF events, maintained by OpenStack.

Background, or Why is Keystone so special?

Why not just use pyCADF’s audit middleware? Most OpenStack services (nova, glance, cinder, etc…) use keystonemiddleware to establish communication with Keystone. This is performed by creating a service user and saving credentials in the [keystone_authtoken] section and including keystonemiddleware in the service’s pipeline. Services can also include pyCADF‘s middleware, which piggybacks off the content it sees in keystonemiddleware to easily include auditing support for every API request that hits the service. We can’t take this approach in Keystone. keystonemiddleware gets its information from a Keystone token (using the service user). If we try to include pyCADF‘s middleware into Keystone’s pipeline, then we’ll run into a circular reference. What the Keystone team has decided to do is to instead emit notifications for specific events. I thought Keystone already emits notifications, why do I need CADF? Keystone did already emit notifications, but these were lacking in content and made auditing impractical. Take for instance this example of a basic notification.
    "event_type": "identity.user.created",
    "message_id": "0156ee79-b35f-4cef-ac37-d4a85f231c69",
    "payload": {
        "resource_info": "671da331c47d4e29bb6ea1d270154ec3"
    "priority": "INFO",
    "publisher_id": "identity.host1234",
    "timestamp": "2013-08-29 19:03:45.960280"
We can tell a user was created, but who triggered the event? From which host machine? What project did they use as their scope? Using CADF notifications will tell us this information (and more), and additionally it’ll standardize the way events are seen in OpenStack. On to the fun stuff!

Viewing CADF notifications in Keystone logs

Setup DevStack with Ceilometer added Pull down devstack, add ceilometer, here’s the localrc I used:
Update Keystone’s configuration file Make the following changes to keystone.conf. Use cadf as the notification_format instead of basic. Also, set notification_driver twice, once to messaging and again to log. This will make the notifications be emitted to the rabbitmq service, and be written to Keystone’s log.
notification_format = cadf
notification_driver = messaging
notification_driver = log
Create a user Now trigger an event that will generate a notification, creating a user should do the trick.
vagrant@vagrant-ubuntu-trusty-64:~/devstack$ . openrc admin admin
vagrant@vagrant-ubuntu-trusty-64:~/devstack$ openstack user create test_user --os-identity-api-version 3 --os-auth-url --os-default-domain default
Examine the Keystone logs If you look at the logs, you’ll see something like this:
oslo.messaging.notification.identity.user.created [-] {"priority": "INFO", "event_type": "identity.user.created", "timestamp": "2015-07-05 00:36:27.337617", "publisher_id": "identity.vagrant-ubuntu-trusty-64.localdomain", "payload": {"typeURI": "", "initiator": {"typeURI": "service/security/account/user", "host": {"agent": "python-keystoneclient", "address": ""}, "project_id": "ae7e3ebab8f442a8a7f19bc25c8c09f0", "id": "ce14a6f356ef4c40ad93339e6e424a7f"}, "target": {"typeURI": "data/security/account/user", "id": "22a21766c47247028be3fda7ee0b712b"}, "observer": {"typeURI": "service/security", "id": "openstack:3bc912f3-5746-4e69-8e11-2510e9d78db2"}, "eventType": "activity", "eventTime": "2015-07-05T00:36:27.337390+0000", "action": "created.user", "outcome": "success", "id": "openstack:11fc2c26-0d67-4875-90d0-09186863e502", "resource_info": "22a21766c47247028be3fda7ee0b712b"}, "message_id": "9767db8f-1891-4566-b3a7-a217a37c515f"}
It looks like a CADF event to me! Here it is with better formatting:
    "priority": "INFO",
    "event_type": "identity.user.created",
    "timestamp": "2015-07-05 00:36:27.337617",
    "publisher_id": "identity.vagrant-ubuntu-trusty-64.localdomain",
    "payload": {
        "typeURI": "",
        "initiator": {
            "typeURI": "service/security/account/user",
            "host": {
                "agent": "python-keystoneclient",
                "address": ""
            "project_id": "ae7e3ebab8f442a8a7f19bc25c8c09f0",
            "id": "ce14a6f356ef4c40ad93339e6e424a7f"
        "target": {
            "typeURI": "data/security/account/user",
            "id": "22a21766c47247028be3fda7ee0b712b"
        "observer": {
            "typeURI": "service/security",
            "id": "openstack:3bc912f3-5746-4e69-8e11-2510e9d78db2"
        "eventType": "activity",
        "eventTime": "2015-07-05T00:36:27.337390+0000",
        "action": "created.user",
        "outcome": "success",
        "id": "openstack:11fc2c26-0d67-4875-90d0-09186863e502",
        "resource_info": "22a21766c47247028be3fda7ee0b712b"
    "message_id": "9767db8f-1891-4566-b3a7-a217a37c515f"
You’ll see here that we have access to the necessary information we need for auditing. The initiator block includes: the user who performed the action, the project they used to authenticate with, and the host machine they used. For a full list of actions that will emit notifications, refer to Keystone’s notification documentation.

Viewing CADF notifications with Ceilometer

Use Ceilometer’s CLI
vagrant@vagrant-ubuntu-trusty-64:~/devstack$ ceilometer event-list --query event_type=identity.user.created
| Event Type            | Traits                                                                    |
| identity.user.created | +--------------+--------+-----------------------------------------------+ |
|                       | |     name     |  type  |                     value                     | |
|                       | +--------------+--------+-----------------------------------------------+ |
|                       | | initiator_id | string |        ce14a6f356ef4c40ad93339e6e424a7f       | |
|                       | |  project_id  | string |        ae7e3ebab8f442a8a7f19bc25c8c09f0       | |
|                       | | resource_id  | string |        22a21766c47247028be3fda7ee0b712b       | |
|                       | |   service    | string | identity.vagrant-ubuntu-trusty-64.localdomain | |
|                       | +--------------+--------+-----------------------------------------------+ |

Viewing CADF notifications in RabbitMQ

Alternatively instead of using Ceilometer, you can use RabbitMQ’s management plugin to view the same notifications. Note, you should stop all Ceilometer processes to ensure that it won’t read the messages on the notification queue. Setup RabbitMQ management plugin When installed, the RabbitMQ management plugin will automatically add a web interface, users will be able to easily read notifications.
vagrant@vagrant-ubuntu-trusty-64:~/devstack$ sudo rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
Restart RabbitMQ and ensure the management plugins are installed
vagrant@vagrant-ubuntu-trusty-64:~/devstack$ sudo service rabbitmq-server restart 
 * Restarting message broker rabbitmq-server                                                                                                                                              [ OK ] 

vagrant@vagrant-ubuntu-trusty-64:~/devstack$ rabbitmq-plugins list
[e] amqp_client                       3.2.4
[ ] cowboy                            0.5.0-rmq3.2.4-git4b93c2d
[ ] eldap                             3.2.4-gite309de4
[e] mochiweb                          2.7.0-rmq3.2.4-git680dba8
[ ] rabbitmq_amqp1_0                  3.2.4
[ ] rabbitmq_auth_backend_ldap        3.2.4
[ ] rabbitmq_auth_mechanism_ssl       3.2.4
[ ] rabbitmq_consistent_hash_exchange 3.2.4
[ ] rabbitmq_federation               3.2.4
[ ] rabbitmq_federation_management    3.2.4
[ ] rabbitmq_jsonrpc                  3.2.4
[ ] rabbitmq_jsonrpc_channel          3.2.4
[ ] rabbitmq_jsonrpc_channel_examples 3.2.4
[E] rabbitmq_management               3.2.4
[e] rabbitmq_management_agent         3.2.4
[ ] rabbitmq_management_visualiser    3.2.4
[ ] rabbitmq_mqtt                     3.2.4
[ ] rabbitmq_shovel                   3.2.4
[ ] rabbitmq_shovel_management        3.2.4
[ ] rabbitmq_stomp                    3.2.4
[ ] rabbitmq_tracing                  3.2.4
[e] rabbitmq_web_dispatch             3.2.4
[ ] rabbitmq_web_stomp                3.2.4
[ ] rabbitmq_web_stomp_examples       3.2.4
[ ] rfc4627_jsonrpc                   3.2.4-git5e67120
[ ] sockjs                            0.3.4-rmq3.2.4-git3132eb9
[e] webmachine                        1.10.3-rmq3.2.4-gite9359c7
View CADF notifications in RabbitMQ’s web console You’ll need to trigger another event, so create a user, project or role using OpenStackClient. When that is done, navigate to: and log in with your RabbitMQ credentials. The default username and password are guest/guest. Note, in the screenshots shown below RabbitMQ is shown running on port 8081, this because I had port forwarding enabled in my VM. 1. Check the Queues tab at the top Screen Shot 2015-07-05 at 12.31.26 AM 2. Look for unread messages in the queue. Screen Shot 2015-07-05 at 12.30.43 AM 3. Get the messages, there will be authentication messages and a identity.user.created message Screen Shot 2015-07-05 at 12.32.06 AM 4. View the payload of each message, which should be a CADF event Screen Shot 2015-07-05 at 12.30.18 AM

4 Comments on "Viewing Keystone CADF notifications with Ceilometer and RabbitMQ"

  1. […] post Viewing Keystone CADF notifications with Ceilometer and RabbitMQ appeared first on IBM […]

  2. […] Steve Martinelli: Viewing Keystone CADF notifications with Ceilometer and […]

  3. […] Steve Martinelli: Viewing Keystone CADF notifications with Ceilometer and […]

  4. […] But if you had an async way to get events from Keystone, you could solve this yourself. That was my idea with my Keystone CADF Event Logger tool. Before we dive into the tool, some quick background on CADF events. You can read the DMTF mumbo-jumbo at the link in the previous sentence, but just know, Keystone CADF events log anything interesting that happens in Keystone. They also tell you who did it, from where they did it, and when they did it. All important things for auditing. (This article from Steve Martinelli has some more great background) […]

Join The Discussion

Your email address will not be published. Required fields are marked *