IBM Support

Migrating Kerberos Principals to a new KDC server in BigInsights cluster - Hadoop Dev

Technical Blog Post


Abstract

Migrating Kerberos Principals to a new KDC server in BigInsights cluster - Hadoop Dev

Body

This article is intended for customers who would want to migrate their Kerberos server due to performance reasons, KDC corruption, KDC server data loss with minimal downtime. It can also be applied in scenarios where Kerberos principals could not be recovered due to catastrophic failures.

Please note

Prior Knowledge of Kerberos is mandatory before proceeding with this article do not attempt the steps on production cluster unless you have tested on development cluster and you are comfortable with the nuances of Kerberos server.
While every attempt has been made to ensure the script is bug free. It is always advised to backup the existing keytabs in /etc/security/keytabs from all the nodes. The usage of script is on as-is basis any queries related to the script will be addressed only in IBM hadoop forum https://developer.ibm.com/answers

Steps for migrating the Kerberos principals.

  • Stop all the BigInsights services from Ambari if not already stopped.
  • Make sure the new server has Kerberos installed and has connectivity to BigInsights cluster nodes.
  • Modify the following properties in file /etc/krb5.conf on all BigInsihgts nodes to point to the new kerberos server.
           EXAMPLE.COM = {               kdc = kerberos-1.example.com               admin_server = kerberos.example.com           }  
  • Download the csv file with list of kerberos principals configured for BigInsights cluster. Use the following URL to download the file from Ambari. A valid login session is required to access this url.
    http://<ambari_hostname>:<ambari_port>/api/v1/clusters/<cluster_name>/kerberos_identities?fields=*&format=csv

    The parameters <ambari_hostname>, <ambari_port> and <cluster_name has to be replaced with corresponding values.
    A csv file with name “kerberos_identities” will be downloaded. We will be using this file for creating/migrating the identities.

  • Download the kerberos_setup.sh script, the script has one parameters which the file download from ambari “kerberos_identities”
  • Upload the script and identities file to the new Kerberos server using scp or ftp.
  • Executing the script is a 2 step process. The following action should be performed as root user on Kerberos server.
    • Creation of generator script, which is achieved by running following command.
      ./kerberos-setup.sh kerberos_identities
    • EXecuting the generator script as follows.
      ./generate_keytabs.sh
    • At the end of the script execution, the keytabs are created as following directory structure in the current directory.
        keytabs_host1.ibm.com  keytabs_host1.ibm.com.tar  keytabs_host2.ibm.com  keytabs_host2.ibm.com.tar  keytabs_host2.ibm.com  keytabs_host2.ibm.com.tar  
    • The keytabs pertaining to individual hosts of BigInsights cluster are organized in the respective folders and an archived version is also created. The hostnames are appended to simplify identification. The folder structure within each directory mimics that of original structure created by Ambari Kerberos Wizard i.e /etc/security/keytabs.
    • As root user of the respective hosts overwrite the keytabs by either doing scp or extracting the archive. This ensures the permissions are intact. If the old keytabs were discarded it is recommended the permissions of the keytabs should be restored as before.
    • Start all the services from Ambari , run service checks to confirm services are running fine.

      The script used in this article is a modified version of kerberos_setup.sh script shipped in Ambari.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

UID

ibm16260019