Slider HBase Kerberos configuration

Apache Slider is used to deploy existing distributed applications on a yarn cluster. This article describes how to run the Slider HBase app-package in a secure (kerberized) cluster using IBM Open Platform with Apache Spark and Apache Hadoop (IOP) 4.2. To configure Slider HBase in a secure cluster, a user with both headless and service security tabs is required. It is recommended to use the hbase user for creating HBase app configurations in a secure cluster.

Setup
In case of a secure cluster extract and copy appConfig-secured-default.json as appConfig-secured.json instead of appConfig-default.json from HBase app-package

unzip /usr/iop/4.2.0.0/slider/app-package/HBase/slider-HBase-app-package-pkg-version.zip
   appConfig-secured-default.json -d /home/hbase
cp appConfig-secured-default.json appConfig-secured.json

Procedure
There are two ways to configure an HBase app in a secure cluster. This article explains the configuration using local keytabs.

1.Edit the appConfig-secured.json file and provide the proper values relevant to your cluster for .kerberos.principal and .keytab.file parameter for the HBase environment. For slider.keytab.principal.name value use the hbase-(clustername). The following appConfig-secured.json snippet demonstrates the values used for IBM.COM realm

"site.hbase-site.hbase.regionserver.kerberos.principal": "hbase/_HOST@IBM.COM",
"site.hbase-site.hbase.regionserver.keytab.file":  "/etc/security/keytabs/hbase.service.keytab",
"site.hbase-site.hbase.master.kerberos.principal": "hbase/_HOST@IBM.COM",
"site.hbase-site.hbase.master.keytab.file": "/etc/security/keytabs/hbase.service.keytab",
"site.hbase-site.hbase.rest.kerberos.principal":  "hbase/_HOST@IBM.COM",
"site.hbase-site.hbase.rest.keytab.file":  "/etc/security/keytabs/hbase.service.keytab",
"site.hbase-site.hbase.thrift.kerberos.principal": "hbase/_HOST@IBM.COM",
"site.hbase-site.hbase.thrift.keytab.file": "/etc/security/keytabs/hbase.service.keytab" 
"components": {
       "slider-appmaster": {
       "jvm.heapsize": "1024M",
       "slider.am.keytab.local.path": "/etc/security/keytabs/hbase.headless.keytab",
       "slider.keytab.principal.name": "hbase-bdakerberos@IBM.COM"
       }
 } 

Note : Do not replace the _HOST in the above values with actual host name. The _HOST value will be replaced at runtime automatically. Replacing _HOST value with host names will result in login failures and yarn container errors.

2. Install Slider HBase app-package

slider package -–install --name HBASE --package  /usr/iop/4.2.0.0/slider/app-package/hbase/slider-hbase-app-package-pkg-version.zip

3. Create HBase Slider app

slider create hbase1 --template /home/hbase/appConfig-secured.json --resources /home/hbase/resources.json

4. Get HBase site config hbase-site.xml

slider registry --name hbase1 --getconf hbase-site --format xml --out hbase-site.xml

5. Edit the hbase-site.xml and change hbase.tmp.dir to a directory for which user has write permissions

 <property>
 <name>hbase.tmp.dir</name>
 <value>/tmp</value> 
</property>

6. Launch HBase shell using the edited hbase-site.xml

hbase --config <path-to-hbase-site.xml> shell

Problems
If the Slider configuration is not correct, the slider.log might contain yarn errors like the following example:
1. hbase(main):001:0> list
TABLE
16/04/29 07:33:44 ERROR client.ConnectionManager$HConnectionImplementation: The node /yarnapps_hbase_slider_hbase1 is not in ZooKeeper. It should have been written by the master. Check the value configured in ‘zookeeper.znode.parent’. There could be a mismatch with the one configured in the master
Check the slider.log and see if there are authentication failures. The most common reason for this problem is that slider agent does not have permissions to login to cluster and create zookeeper node. The Slider log looks something similar to
java.io.IOException: Login failure for hbase-bdakerberos@IBM.COM from keytab /hadoop/yarn/local/usercache/hbase/appcache/application_1462809282136_0010/container_e03_1462809282136_0010_02_000001/keytabs/etc/security/keytabs/hbase.headless.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Solution: Make sure the appConfig-Secured.json has all the Kerberos values relevant to the realm being used. Check if the hbase user on the node running Slider has login permissions on the RM node.

References
For Kerberos related commands and setup refer to Kerberos documentation at
http://web.mit.edu/kerberos/krb5-1.12/doc/

1 comment on"Slider HBase Kerberos configuration"

  1. Hi Sailaja – Getting this error. When I start the shell:
    ERROR: Can’t get master address from ZooKeeper; znode data == null
    Any tips?

Join The Discussion

Your email address will not be published. Required fields are marked *