Starting from Kafka 0.10.x Kafka Broker supports username/password authentication. This Mechanism is called SASL/PLAIN. The steps below describe how to set up this mechanism on an IOP 4.2.5 Kafka Cluster.
1. Configure the Kafka brokers and Kafka Clients
Add a JAAS configuration file for each Kafka broker. Create a kafka_plain_jaas.conf file as specified below:
KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" password="kafka-secret" user_kafka="kafka-secret" user_ibm="ibm-secret"; }; KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required username="ibm" password="ibm-secret"; };Let’s understand the content of kafka_plain_jaas.conf file and how Kafka Brokers and Kafka Clients use it.
KafkaServer Section:
The KafkaServer section defines two users: kafka and ibm. The properties username and password are used by the broker to initiate connections to other brokers. In this example, kafka is the user for inter-broker communication. The set of properties user_{userName} defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.
KafkaClient Section:
The Kafka Client section describes how the clients, Producer and Consumer, can connect to Kafka Broker. The properties username and password in the Kafka Client section are used by clients to configure the user for client connections. In this example, clients connect to the broker as user “ibm”.
2. Pass the JAAS configuration location as a JVM Parameter to the Kafka broker
Update the Advanced kafka-env section to add the properties below: export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/usr/iop/4.2.5.0-0000/kafka/conf/kafka_plain_jaas.conf" export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS" 3. Add the properties below to the custom Kafka broker configuration.
4. Update the listeners property in the Kafka Broker section to SASL_PLAINTEXT://localhost:6667
5. Restart Kafka.
Run the Kafka Client Console Producer / Console Consumer to authenticate to the Kafka broker with the specified username and password. In this example, the username used to connect to brokers is “ibm”. The JVM parameter java.security.auth.login.config for console producer and console consumer is set by the Step 2 above.
All the commands shown below are run from the path /usr/iop/4.2.5.0-0000/kafka
Create a topic:
[root@heel1 kafka]# bin/kafka-topics.sh --create --topic plain-topic --zookeeper heel1.fyre.ibm.com:2181 --partitions 1 --replication-factor 1 [2017-04-10 12:06:22,239] WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/iop/4.2.5.0-0000/kafka/conf/kafka_plain_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn) Created topic "plain-topic".Run Kafka console producer
Before running the Kafka console Producer configure the producer.properties file as shown:
[root@heel1 kafka]# cat producer.properties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN[root@heel1 kafka]# bin/kafka-console-producer.sh --broker-list heel1.fyre.ibm.com:6667 --topic plain-topic --producer.config producer.properties Message 1 Message 2 Message 3 ^CRun Kafka console consumer
Before running Kafka console consumer configure the consumer.properties file as shown:
[root@heel1 kafka]# cat consumer.properties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN[root@heel1 kafka]# bin/kafka-console-consumer.sh --bootstrap-server heel1.fyre.ibm.com:6667 --topic plain-topic --from-beginning --consumer.config consumer.properties Message 1 Message 2 Message 3 ^CProcessed a total of 3 messagesSASL/PLAIN with ACLs:
The username is used as the authenticated Principal, which is used in authorization (e.g. ACLs).
For usage of ACLS refer to the blog Kafka ACLs
For more information refer to the kafka documentation