Kerberos plays an important role in Hadoop security, mainly doing the job of authentication. Kerberos is well documented already, so here we will introduce some key tasks to give a big picture of Kerberos usage in IOP 4.2.

1. Kerberos setup in IOP 4.2

To turn on Kerberos authentication, in the Ambari UI, go to Admin – Kerberos – Enable Kerberos as shown below

Screen Shot 2016-08-03 at 2.51.17 PM

Click the Enable Kerberos button to open the Enable Kerberos wizard.

Screen Shot 2016-08-03 at 3.27.29 PM

In the Enable Kerberos wizard, you can see that IOP supports three types of KDC, each having different prerequisites as listed in the checkbox.

1) For setting up a MIT KDC server on a Linux machine, the topic Setting up a KDC manually.

Using the setup MIT KDC to enable Kerberos follow the procedure in Setting up Kerberos for IBM Open Platform with Apache Spark and Apache Hadoop clusters.

2) Setup Active Directory and using it to enable Kerberos, follow the procedure in Enable Kerberos in Ambari with an existing Active Directory.

3) Your third option is to enable Kerberos manually with KDC Type None (Such as FreeIPA).

4) Refer to the topic Securing the BigInsights calue-added services to setup Kerberos for IOP 4.2 value-added services.

2. During the process of enabling Kerberos, IOP 4.2 manages the following tasks:

1) Install Kerberos client and libraries on each node.

2) Generate principal on KDC (or Active Directory) you have setup.

3) Generate Keytab files and distribute to the appropriate hosts for different services according to kerberos descriptor (will go to detail in part 3).

4) Update relevant configurations.

3. Verify Kerberos setup

i. After enabling Kerberos, you can check your cluster status using REST API by curl command or by entering the query URL directly in a web browser after login to the cluster: http(s)://${hostname}:${portnumber}/api/v1/clusters/${clustername}

In the JSON file that is returned, you can see the “security_type” is “Kerberos”. Without Kerberos this value is “none”, and it is a flag of whether or not the cluster is secured.

ii. Query for http(s)://${hostname}:${portnumber}/api/v1/clusters/${clustername}/artifacts/kerberos_descriptor

By looking at the Kerberos descriptor of the cluster, you can get an idea of which service and component in your cluster has automated kerberization by IOP. For further information about Kerberos descriptors refer to the topic Automated Kerberization.

iii. Check directory /etc/security/keytabs, you will see all keytabs for services.
Screen Shot 2016-08-04 at 1.51.01 PM

The headless keytabs are for the user principal and the service keytabs are for the service principal.

iv. Run the following command:
# klist -kte hdfs.headless.keytab
It will show all the principal names using this keytab, timestamp and supported encryption methods such as:
Screen Shot 2016-08-04 at 3.20.09 PM
In IOP 4.2, the default HDFS headless user principal name is ${hdfsusername}-${cluster_name}@${realm}.

v. Check the Kerberos client configuration file: /etc/krb5.conf

[libdefaults] renew_lifetime = 7d
forwardable = true
default_realm = EXAMPLE.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[logging] default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log

[realms] EXAMPLE.COM = {
admin_server =
kdc =

[domain_realm] = EXAMPLE.COM

IOP will do the proper client configuration by copying from the KDC server configuration and based on the information that you provided during the enable Kerberos process.

4. Troubleshooting

1) Currently all services except Ranger and Ranger KMS are able to have automated kerberization by IOP 4.2 in RPC communication, but the manual steps are well documented in IOP 4.2 knowledge center.

To set up and configure Ranger on a Kerberized cluster refer to the topic Setting up Ranger on a Kerberized cluster.

To set up Ranger KMS on a Kerberized cluster refer to the topic Ranger KMS set up and usage.

2) Without the automation of Ambari, you may encounter problems due to authentication issues. Usually errors will be of the type GSSException.

Here is a good example of dealing with this situation.

When you enable HDFS High Availability, the wizard will ask you to manually run some HDFS commands. Without Kerberos, the commands could be run without error. However, with security enabled, the HDFS user will need to be authenticated on KDC first. To manually authenticate an HDFS user, you would need to run the kinit command:

# sudo su hdfsuser -c ‘kinit –kt principal keytab’

Running this command as the HDFS user, you will get a tgt ticket and the ticket will be cached under /tmp/krb5cc_uid. Running commands su hdfsuser and klist will return something like the following:

Ticket cache: FILE:/tmp/krb5cc_uid
Default principal: principal

Valid starting Expires Service principal
XXX XXX krbtgt/${realm}@${realm}
renew until XXXX

The expire and renew until times are based on ticket_lifetime and renew_lifetime in the krb5.conf file. Once you run kinit, it will give new lifetime for both the expire date and the renew until date.

3) In IOP 4.2, SPNEGO(HTTP authentication) is not automatically enabled after enabling Kerberos (except HBase REST), but refer to the topic Enabling SPNEGO authentication for IBM® Open Platform with Apache Spark and Apache Hadoop to manually set up SPNEGO.

4) Basically, to manually create a principal, follow the processes below:

A. Login to the Kerberos admin console
# kadmin or kadmin.local(on the kdc machine)
This step will show: Authenticating as principal *** with password. Then, type in the password.

B. Create a principal for the user. For example
# addprinc user@realm
This step will ask you to enter a password for this principal.

If you want to authenticate by typing a password, you are done here. If you want to authenticate using keytabs, follow the two steps below.

C. # xst –k user.keytab user@realm
If the keytab is generated successfully, it will show the encryption type added to keytab user.keytab, and you can find the keytab file under current directory.

D. Copy the keytab file to the node you need to use this keytab under /etc/security/keytabs.

E. To authenticate a Ranger user, if you did not already generate a keytab file, run

# sudo su user -c ‘kinit user@realm’
and enter the password you setup

If you have generated keytab file, run
# sudo su user -c ‘kinit –kt user.keytab user@realm’

You can also use addprinc -randkey principal to randomly generate a password for the principal and use keytab to do kinit.

The examples above demonstrated certain key tasks for using Kerberos with IOP 4.2. To better understand the Kerberos setup in IOP, take a look at this video on YouTube, BigInsights Kerberos Setup.

Join The Discussion

Your email address will not be published. Required fields are marked *