Co-author, Gobi Sankaranarayanan

This article aims at providing a tool (a standalone Java Program) to simplify setting up Kerberos authentication with Kafka nodes. The tool enables you to create a setup and test it outside of the IIB/ACE environment and once you have it working, then to adopt the same configurations to IIB/ACE.

Pre-reqs:

Instructions to setup and run the tool outside of the IIB/ACE environment.

  1. Download the IIBKafkaPubTestV2.jar

    IIBKafkaPubTestV2.jar

  2. Create a standalone Kafka client properties file with the name “producer.properties” and configure the options below based on your Kafka Server settings

    #The name of the Kerberos service used by Kafka. This name must match
    #the principal name of the Kafka brokers.
    sasl.kerberos.service.name=kafkaServerPrincipal
    
    #Set the mechanism to GSSAPI to enable Kerberos KAFKA authentication.
    sasl.mechanism=GSSAPI
    
    #JASS config for Kerberos authentication (all in one line).
    #The useKeytab value is the full path to the Kerberos keytab file.
    #The principal value is the Kerberos principal, for
    #example user/host@REALM. Here, host is the host of the center for key
    #distribution and REALM is the Kerberos REALM.
    
    sasl.jaas.config=com.ibm.security.auth.module.Krb5LoginModule required 
    credsType=both useKeytab="/etc/krb5.keytab" 
    principal="kafkaServerPrincipal/vmhost@DEFAULT_REALM";


    Please find the attached producer.properties file for reference

    producer.properties

    Note : You can use a debug=true at the end of sasl.jaas.config if you would like to debug the Kerberos activities.

    For example:

    sasl.jaas.config=com.ibm.security.auth.module.Krb5LoginModule required 
    credsType=both useKeytab="/var/mqsi/mquser.keytab" principal="mquser 
    @CN.CA" debug=true;
  3. Execute the IIBKafkaPubTestV2.jar with below syntax and collect the result.

    java -jar IIBKafkaPubTestV2.jar <full path to producer.properties
    file> <full path to krb5.conf file> <bootstrap server addr>
    <publish topic> <security protocol>


    For example:

    Java -jar IIBKafkaPubTestV2.jar C:\producer.properties 
    C:\krb5.conf localhost:9094 testtopic SASL_PLAINTEXT > 
    kafkalog.txt 2>&1


    Note: if you are using security protocol as SSL or SASL_SSL, you can provide the truststore to the program as

    Java -Djavax.net.ssl.trustStore=<path> -
    Djavax.net.ssl.trustStorePassword=<trust_password> -jar  
    IIBKafkaPubTestV2.jar

    Similarly, if you want to include a keystore as well for a 2-way SSL handshake, you can use

    Java -Djavax.net.ssl.trustStore=<path> -
    Djavax.net.ssl.trustStorePassword=<trust_password>
    -Djavax.net.ssl.keyStore=<path> -
    Djavax.net.ssl.keyStorePassword=<keystore_password>
    -jar  IIBKafkaPubTestV2.jar
  4. Check kafkalog.txt logs to identify the status of Kafka Publish using the Kerberos authentication. On successful publish you will see the below message

    “successfully published”

    If it throws any error related to SSL Handshake, Kerberos Authentication or any other errors related to Kafka with Kerberos setup then fix the issue until you receive “successfully published” from the IIBKafkaPubTestV2.jar execution.

    Once you have “successfully published” then you are ready to incorporate the setup in to your IIB configuration.

Instructions for configuring Kafka with Kerberos authentication in IIB v10.

Once you have a successful run of the tool, use the steps below to adopt the arguments used in the tool to configure Kafka in IBM Integration Bus.

  1. Set the Kerberos configuration file. It is the second argument used while running the tool on Integration server

    mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n kerberosConfigFile -v <full path to krb5.conf file>


  2. Set the producer.properties file.
    mqsichangeproperties IBNODE -c ConnectorProviders -o Kafka -n property1 -v  <full path to
    producer.properties file>

    (Optional) For Kafka consumer nodes if they are using the same properties file, please run

    mqsichangeproperties IBNODE -c ConnectorProviders -o Kafka -n property2 -v <full path to 
    producer.properties file>

  3. Configure the Kafka node with the following properties

    • Specify the topic name on the node property as <publish topic>

    • Set the 'Bootstrap servers' node property as <bootstrap server addr>

    • Set node property 'Security protocol' as <security protocol>

  4. (Optional) If you have used -Djavax.net.ssl.trustStore or -Djavax.net.ssl.keyStore while running the tool, then you need to configure the same keystore and truststore in IBM integration Bus using the following commands.

Commands to configure truststore under the Integration sever.

  1. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n truststoreFile -v 
    <Truststore value used to run the program IIBKafkaPubTestV2.jar >
  2. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n truststorePass -v 
    kafkaTrust::password
  3. mqsisetdbparms IBNODE  -n kafkaTrust::password -u temp -p <truststore password used to run
    the program IIBKafkaPubTestV2.jar >

Commands to configure keystore under the Integration server.

  1. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n keystoreFile -v <keystore
    value used to run the program IIBKafkaPubTestV2.jar >
  2. mqsichangeproperties IBNODE -e IBSERVER -o ComIbmJVMManager -n keystorePass  -v 
    kafkaKeystore::password
  3. mqsisetdbparms IBNODE  -n kafkaKeystore::password -u temp -p <keystore  password used to 
    run the program IIBKafkaPubTestV2.jar >

Instructions for configuring Kafka with Kerberos authentication in ACE V11.

You will need App Connect Enterprise version 11.0.0.5 or higher for Kafka nodes to use Kerberos authentication.

  1. Define the Kerberos configuration file

    a) For Integration node, run the command

    mqsichangeproperties NODE -e SERVER -o ComIbmJVMManager -n kerberosConfigFile -v <full path to 
    krb5.conf file>


    b) For independent integration server, set the property inside server.conf.yaml under the JVM ResourceManager.

  2. Set the producer.properties file using environment variable

    MQSI_KAFKA_PRODUCER_PROPERTIES_FILE and MQSI_KAFKA_CONSUMER_PROPERTIES_FILE


    a) setting producer.properties file for KafkaProducer node

    export MQSI_KAFKA_PRODUCER_PROPERTIES_FILE=<full path to producer.properties file>

    b) (optional) setting the consumer.properties file for KafkaConsumer node

    export MQSI_KAFKA_CONSUMER_PROPERTIES_FILE=<full path to consumer.properties file>


    Please make sure that you restart the integration node or independent integration server after setting the environment variable.

  3. Configure the Kafka node with following properties

    • Specify the topic name on the node property as <publish topic>
    • Set the 'Bootstrap servers' node property as <bootstrap server addr>
    • Set node property 'Security protocol' as <security protocol>

Conclusion

The above procedure is to configure Kerberos authentication with Kafka using the producer.properties file. For those who are using plain username and password authentication instead of Kerberos authentication, you can use the same stand-alone program for testing in the following way;

  1. Change the sasl.jaas.config in the producer.properties file in the following form;
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";
  2. Run the program. You can use a dummy file for the argument <full path to krb5.conf file>
  3. Once you have a successful result, configure the username and password credentials as per the following Knowledge Center page for Kafka nodes to pick them.
    https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/bz91050_.htm

The stand alone program is built using Java KafkaProducer class. You can refer to the Javadoc for further information;
https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

Acknowledgement and thanks to Amar Shah for his technical support and advice in the construction of this article.

Join The Discussion

Your email address will not be published. Required fields are marked *