Kafka ACLs - Hadoop Dev

Overview

In Apache Kafka, the security feature is supported from version 0.9. When Kerberos is enabled, we need to have the authorization to access Kafka resources. In this blog, you will learn how to add authorization to Kafka resources using Kafka console ACL scripts. In addition, when SSL is enabled in Kafka, ACLs (access control list) can be enabled to authorize access to Kafka resources.

Kafka ACLs are defined in the general format of “Principal P is [Allowed/Denied] Operation O From Host H On Resource R”.

Kafka resources that can be protected with ACLS are:

• Topic
• Consumer group
• Cluster

Operations on the Kafka resources are as below:

Cluster operations (CLUSTER_ACTION) refer to operations necessary for the management of the cluster, like updating broker and partition metadata, changing the leader and the set of in-sync replicas of a partition, and triggering a controlled shutdown.

Kafka Kerberos with ACLs

To enable Kerberos in an IOP 4.2 cluster, you can follow the steps mentioned in the link Enable Kerberos on IOP 4.2

After Kerberos is enabled, the following properties are automatically added to custom Kafka broker configuration.

Kafka console commands running as super user kafka

By default, only the super.user will have the permissions to access the Kafka resources. The default value for super.users is kafka.

The Kafka home directory in IOP is located at /usr/iop/current/kafka-broker. The Kafka console scripts referenced in this article are located under /usr/iop/current/kafka-broker.

List Kafka service keytab

[kafka@hostname kafka]# klist -k -t /etc/security/keytabs/kafka.service.keytab   Keytab name: FILE:/etc/security/keytabs/kafka.service.keytab  KVNO Timestamp         Principal  ---- ----------------- --------------------------------------------------------     1 06/22/16 13:53:01 kafka/hostname.abc.com@IBM.COM     1 06/22/16 13:53:01 kafka/hostname.abc.com@IBM.COM     1 06/22/16 13:53:01 kafka/hostname.abc.com@IBM.COM     1 06/22/16 13:53:01 kafka/hostname.abc.com@IBM.COM     1 06/22/16 13:53:01 kafka/hostname.abc.com@IBM.COM  

Perform kinit to obtain and cache the Kerberos ticket

[kafka@hostname kafka]# kinit -f -k -t /etc/security/keytabs/kafka.service.keytab kafka/hostname.abc.com@IBM.COM  

Create a topic

[kafka@hostname kafka]# bin/kafka-topics.sh --create --zookeeper hostname.abc.com:2181 --replication-factor 1 --partitions 1 --topic mytopic  Created topic "mytopic".

Run Kafka producer

[kafka@hostname kafka]# bin/kafka-console-producer.sh --broker-list hostname.abc.com:6667 --topic mytopic --producer.config producer.properties   Hi  Sending Message to Kafka topic  Message 1  Message 2  Message 3  ^C  [kafka@hostname kafka]$ cat producer.properties   security.protocol=SASL_PLAINTEXT

Run Kafka consumer

[root@hostname kafka]# bin/kafka-console-consumer.sh --new-consumer --zookeeper hostname.abc.com:2181 --topic mytopic --from-beginning --bootstrap-server hostname.abc.com:6667 --consumer.config consumer.properties  Hi  Sending Message to Kafka topic  Message 1  Message 2  Message 3  ^CProcessed a total of 5 messages  [root@hostname kafka]# cat consumer.properties   security.protocol=SASL_PLAINTEXT

As we have run the commands with super user kafka, we have access to Kafka resources without adding any ACLs.

How to add a new user as a super user?

• Update the super.users property in the “Custom kafka-broker” configuration to add additional users as super users. The list is a semicolon-separated list of user names in the format “User:”. The example shows how to configure the users kafka and kafkatest as super users.
• This will allow the user to access resources without adding any ACLs.

• Restart Kafka

How to add ACLs for new users?

The following example shows how to add ACLs for a new user “kafkatest”.

Create a user kafkatest

[root@hostname kafka]# useradd  kafkatest 

Note: In the example shown here the KDC server, Kafka broker and Producer/Consumer running are on the same machine. If the KDC server is setup on a different node in your environment, copy the keytab files to /etc/security/keytabs where Kafka producer and consumer are running.

Create a principal for kafkatest user

[root@hostname kafka]# kadmin.local  Authenticating as principal kafka/admin@IBM.COM with password.  kadmin.local:  addprinc "kafkatest"

Create  a Kerberos keytab file

kadmin.local: xst -norandkey -k /etc/security/keytabs/kafkatest.keytab kafkatest@IBM.COM

Quit from kadmin

kadmin.local:  quit

List and cache the kafkatest Kerberos ticket

[kafkatest@hostname kafka]$ klist -k -t /etc/security/keytabs/kafkatest.keytab     Keytab name: FILE:/etc/security/keytabs/kafkatest.keytab  KVNO Timestamp         Principal  ---- ----------------- --------------------------------------------------------     1 06/22/16 16:24:15 kafkatest@IBM.COM     1 06/22/16 16:24:15 kafkatest@IBM.COM     1 06/22/16 16:24:15 kafkatest@IBM.COM     1 06/22/16 16:24:15 kafkatest@IBM.COM    [kafkatest@hostname kafka]$ kinit -f -k -t /etc/security/keytabs/kafkatest.keytab kafkatest@IBM.COM

Create a topic

[kafkatest@hostname kafka]$ bin/kafka-topics.sh --create --zookeeper hostname.abc.com:2181 --partitions 1 --replication 1 --topic kafka-testtopic  Created topic "kafka-testtopic".

Add write permission for user kafkatest for topic kafka-testtopic:

[kafkatest@hostname kafka]$ bin/kafka-acls.sh --topic kafka-testtopic --add -allow-host 9.30.150.22 --allow-principal User:kafkatest --operation Write --authorizer-properties zookeeper.connect=hostname.abc.com:2181  Adding ACLs for resource `Topic:kafka-testtopic`:    	User:kafkatest has Allow permission for operations: Write from hosts: 9.30.150.22     Current ACLs for resource `Topic:kafka-testtopic`:    	User:kafkatest has Allow permission for operations: Write from hosts: 9.30.150.22 

Run Kafka producer

[kafkatest@hostname kafka]$ bin/kafka-console-producer.sh --broker-list hostname.abc.com:6667 --topic kafka-testtopic --producer.config producer.properties  Hi  Writing Data as kafkatest user  Message 1  Message 2  Message 3  ^C  [kafkatest@hostname kafka]$ cat producer.properties   security.protocol=SASL_PLAINTEXT

Add read permission for user kafkatest for topic kafka-testtopic and consumer group kafkatestgroup

[kafkatest@hostname kafka]bin/kafka-acls.sh --topic kafka-testtopic --add -allow-host 9.30.150.22 --allow-principal User:kafkatest --operation Read --authorizer-properties zookeeper.connect=hostname.abc.com:2181 --group kafkatestgroup  Adding ACLs for resource `Topic:kafka-testtopic`:    	User:kafkatest has Allow permission for operations: Read from hosts: 9.30.150.22     Adding ACLs for resource `Group:kafkatestgroup`:    	User:kafkatest has Allow permission for operations: Read from hosts: 9.30.150.22     Current ACLs for resource `Topic:kafka-testtopic`:    	User:kafkatest has Allow permission for operations: Write from hosts: 9.30.150.22  	User:kafkatest has Allow permission for operations: Read from hosts: 9.30.150.22     Current ACLs for resource `Group:kafkatestgroup`:    	User:kafkatest has Allow permission for operations: Read from hosts: 9.30.150.22 

Run Kafka consumer

[kafkatest@hostname kafka]$ bin/kafka-console-consumer.sh --new-consumer --zookeeper hostname.abc.com:2181 --topic kafka-testtopic --from-beginning --bootstrap-server hostname.abc.com:6667 --consumer.config consumer.properties  Hi  Writing Data as kafkatest user  Message 1  Message 2  Message 3  ^CProcessed a total of 5 messages  [kafkatest@hostname kafka]$ cat consumer.properties   security.protocol=SASL_PLAINTEXT  group.id=kafkatestgroup

Information about kafka_jaas conf file:

When Kerberos is enabled in Kafka, this configuration file is passed as a security parameter (-Djava.security.auth.login.config=”/usr/iop/current/kafka-broker/conf/kafka_jaas.conf”) to Kafka console scripts.

[root@hostname kafka]# cat /usr/iop/current/kafka-broker/conf/kafka_jaas.conf   KafkaServer {     com.sun.security.auth.module.Krb5LoginModule required     useKeyTab=true     keyTab="/etc/security/keytabs/kafka.service.keytab"     storeKey=true     useTicketCache=false     serviceName="kafka"     principal="kafka/hostname.abc.com@IBM.COM";  };  KafkaClient {     com.sun.security.auth.module.Krb5LoginModule required     useTicketCache=true     renewTicket=true     serviceName="kafka";  };  Client {     com.sun.security.auth.module.Krb5LoginModule required     useKeyTab=true     keyTab="/etc/security/keytabs/kafka.service.keytab"     storeKey=true     useTicketCache=false     serviceName="zookeeper"     principal="kafka/hostname.abc.com@IBM.COM";  };

• The KafkaServer section is used by the Kafka broker and inter-broker communication, for example during the creation of topic.

• The KafkaClient is used when running Kafka producer or consumers. Because in the example KafkaClient is using the ticket cache, we have to run the kinit command to cache the Kerberos ticket before running the Kafka producer and consumer.

• The Client section is used for Zookeeper connection. Kafka ACLs are stored in the Zookeeper.

What to do when the SASL username (operating system user name) is different from the principal name

Generally, the SASL username  is the same as the primary name of the Kerberos principal. However, if that’s not the case, we need to add a property sasl.kerberos.principal.to.local.rules to the Kafka broker configuration, to map the principal name to the user name. In the following example, a mapping from the principal name ambari-qa-bh to the user name (operating system user name) ambari-qa is added.

When Kerberos is enabled from Ambari, the principal name generated for the user “ambari-qa” will be of the form ambari-qa-[Cluster Name]. In the example shown here, I have provided my cluster name as “bh”, the principal name generated for user “ambari–qa” is generated as ambari-qa-bh.

  [root@hostname kafka]# klist -k -t /etc/security/keytabs/smokeuser.headless.keytab   Keytab name: FILE:/etc/security/keytabs/smokeuser.headless.keytab  KVNO Timestamp         Principal  ---- ----------------- --------------------------------------------------------     1 06/22/16 13:53:00 ambari-qa-bh@IBM.COM     1 06/22/16 13:53:00 ambari-qa-bh@IBM.COM     1 06/22/16 13:53:00 ambari-qa-bh@IBM.COM     1 06/22/16 13:53:00 ambari-qa-bh@IBM.COM     1 06/22/16 13:53:00 ambari-qa-bh@IBM.COM

For the user ambari-qa, we need to add the following rule::
RULE:[1:$1@$0](ambari-qa-bh@IBM.COM)s/.*/ambari-qa/

• Add sasl.kerberos.principal.to.local.rules in custom Kafka-broker configuration.

• Restart Kafka.

More information about the mapping between principal and username can be found in the section auth_to_local in the following article: auth to local

Kafka SSL with ACLs

In this section, we will see how to work with ACLs when SSL is enabled. For information on how to enable SSL in Kafka, follow the steps in the sections Setup SSL and Enable SSL  in the Kafka Security Blog

There is an issue in IOP 4.2 when setting up SSL is enabled in Kafka with ACLs.  Follow the steps mentioned in the technote, to resolve the issue.
Add the below properties in custom-kafka-broker section to enable authorization with SSL.

• authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
• super.users=User:CN=hostname.ibm.com,OU=iop,O=ibm,L=san jose,ST=california,C=US

Restart the Kafka service from Ambari UI for the changes to take effect.
Note: Add the output of the command below, used to generate the key and certificate for the broker, to the list of super users in Kafka. This allows the Kafka broker to access all Kafka resources. As mentioned above, by default only the super user has access to all Kafka resources. The output of the below command provides the SSL username which is used as the value for super.users.

[root@hostname security]# keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey  Enter keystore password:    Re-enter new password:   What is your first and last name?    [Unknown]:  hostname.ibm.com  What is the name of your organizational unit?    [Unknown]:  iop  What is the name of your organization?    [Unknown]:  ibm  What is the name of your City or Locality?    [Unknown]:  san jose  What is the name of your State or Province?    [Unknown]:  california  What is the two-letter country code for this unit?    [Unknown]:  US  Is CN=hostname.ibm.com, OU=iop, O=ibm, L=san jose, ST=california, C=US correct?    [no]:  yes    Enter key password for   	(RETURN if same as keystore password):

By default, the SSL username will be of the form “CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown”. This can be changed by adding the property principal.builder.class to the Kafka broker configuration in the Ambari UI, and setting the value to a class that needs to implement the interface PrincipalBuilder interface (org.apache.kafka.common.security.auth.PrincipalBuilder).

How to add ACLs for a new SSL user?

Create a topic

[root@hostname kafka]# bin/kafka-topics.sh --create --zookeeper hostname.ibm.com:2181 --replication-factor 1 --partitions 1 --topic ssltopic  Created topic "ssltopic".

Add write permission for SSL user (CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US) for topic ssltopic

[root@hostname kafka]# bin/kafka-acls.sh --topic ssltopic --add -allow-host 9.30.150.20 --allow-principal "User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US" --operation Write --authorizer-properties zookeeper.connect=hostname.ibm.com:2181  Adding ACLs for resource `Topic:ssltopic`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Write from hosts: 9.30.150.20     Current ACLs for resource `Topic:ssltopic`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Write from hosts: 9.30.150.20

The user name provided above is the output when running the below command, which is used to generate Key and Certificate for Kafka Client (Producer/Consumer).

[root@hostname security]# keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365 -genkey  Enter keystore password:    Re-enter new password:   What is your first and last name?    [Unknown]:  hostname.ibm.com  What is the name of your organizational unit?    [Unknown]:  biginsights  What is the name of your organization?    [Unknown]:  ibm  What is the name of your City or Locality?    [Unknown]:  san jose  What is the name of your State or Province?    [Unknown]:  california  What is the two-letter country code for this unit?    [Unknown]:  US  Is CN=hostname.ibm.com, OU=biginsights, O=ibm, L=san jose, ST=california, C=US correct?    [no]:  yes    Enter key password for   	(RETURN if same as keystore password): 

Run Kafka producer

[root@hostname kafka]# bin/kafka-console-producer.sh --broker-list hostname.ibm.com:6667 --topic ssltopic --producer.config client-ssl.properties  Testing Acl with SSl  Message 1  Message 2  ^C  [root@hostname kafka]# cat client-ssl.properties   security.protocol=SSL  ssl.truststore.location=/etc/kafka/conf/security/kafka.client.truststore.jks  ssl.truststore.password=bigdata  ssl.keystore.location=/etc/kafka/conf/security/kafka.client.keystore.jks  ssl.keystore.password=bigdata  ssl.key.password=bigdata  

Add read permission for SSL user (CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US) for topic ssltopic and consumer group ssl-group

[root@hostname kafka]# bin/kafka-acls.sh --topic ssltopic --add -allow-host 9.30.150.20 --allow-principal "User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US" --operation read --authorizer-properties zookeeper.connect=hostname.ibm.com:2181 --group ssl-group  Adding ACLs for resource `Topic:ssltopic`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Read from hosts: 9.30.150.20     Adding ACLs for resource `Group:ssl-group`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Read from hosts: 9.30.150.20     Current ACLs for resource `Topic:ssltopic`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Read from hosts: 9.30.150.20  	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Write from hosts: 9.30.150.20     Current ACLs for resource `Group:ssl-group`:    	User:CN=hostname.ibm.com,OU=biginsights,O=ibm,L=san jose,ST=california,C=US has Allow permission for operations: Read from hosts: 9.30.150.20

Run Kafka consumer

[root@hostname kafka]# bin/kafka-console-consumer.sh --zookeeper hostname.ibm.com:2181 --topic ssltopic --from-beginning --new-consumer --bootstrap-server hostname.ibm.com:6667 --consumer.config client-consumer-ssl.properties  Testing Acl with SSl  Message 1  Message 2  ^CProcessed a total of 3 messages  [root@hostname kafka]# cat consumer-client-ssl.properties   group.id=ssl-group  security.protocol=SSL  ssl.truststore.location=/etc/kafka/conf/security/kafka.client.truststore.jks  ssl.truststore.password=bigdata  ssl.keystore.location=/etc/kafka/conf/security/kafka.client.keystore.jks  ssl.keystore.password=bigdata  ssl.key.password=bigdata  

How to give everyone permission to access a resource if no ACLs are set for the resource.

• Add allow.everyone.if.no.acl.found=true in the “Custom kafka-broker” configuration.

• Restart Kafka

Conclusion:

This blog described how to configure ACLs in Kafka when SSL and Kerberos are enabled in IOP 4.2. For more information, see the Kafka documentation