Introduction

IBM Open Platform (IOP) is a 100% open source platform. It uses Ambari for provisioning and managing Hadoop clusters.

Apache Knox gateway is used in IOP in providing perimeter security. It provides a single point of authentication and access for Hadoop cluster services. It can integrate with Kerberos and provides the following features:

  • Single REST API access point
  • Centralized authentication, authorization and audit for Hadoop REST/HTTP services
  • LDAP/Active Directory authentication, authorization and audit
  • Eliminate SSH edge node risks
  • Hides Network topology

Objective

This technical document is intended to show viewers step by step instructions on setting up LDAP with Ambari, Knox, and BigInsights v4.x.

Version Tested

  • IOP v4.0.0.0 (Ambari v1.7, Knox v0.5.0)
  • BigInsights v4.0.0.1 (FP1)
  • OpenLDAP v2.4.39

LDAP / Ambari Setup Steps

  1. Collect the LDAP server information
    • LDAP server FQDN. Example shown in this post is: ldap2.ibm.com
    • LDAP server port. Usually default is 389. Ex. 389
    • LDAP server BaseDN. (Currently does not support case sensitive LDAP entries) Ex. dc=ibm,dc=com
    • LDAP server Admin account and its credential. Ex. Account: cn=Manager,dc=ibm,dc=com
  2. Install LDAP client on all Hadoop cluster nodes
    • Run the attached script “setup_ldap_client_v3_rhel.sh”. Usage: $ sudo setup_ldap_client_v3_rhel.sh ldap2.ibm.com 389 dc=ibm,dc=com   **Make sure there is no spaces in BaseDN parameter value. (ie. dc=ibm,dc=com)
  3. Test LDAP client install and connectivity to the server
    • Login to Ambari node as root
    • Run connectivity test: Ex. $ sudo openssl s_client -connect ldap2.ibm.com:636
      Note: If connectivity test failed, you should verify if LDAP server’s firewall is opened for port 389 and 636 (Ex. $ sudo netstat -anp|grep 636)
    • Test if a LDAP user can login to Ambari node without an existing local Unix account
      1. $ ssh ldapuser1@n1.ibm.com
      2. You may get an error message indicating that command failed to change home directory for ldap user. This is a correct behavior since ldap user does not exist on the local machine.
      3. If you get the following status, you have successfully logged in using LDAP user id and it’s credential. You are done setting up LDAP client and successfully tested connections. Now you are ready to move on to configure Ambari LDAP.
        ldap user login test
        LDAP User Login Test
  4. Setup Ambari LDAP configuration
    1. Login to Ambari node as user “root”
    2. Either using CLI or direct modify Ambari property file for advanced users. Here is showing via CLI commands.
      • $ sudo ambari-server setup-ldap
      • Answer each question from the prompt. Notice I chose anonymous bind as false so no users can login anonymously.
      • Once you save the settings, the CLI program saves the parameter values into file “/etc/ambari-server/conf/ambari.properties”
        ambari ldap setup
        Ambari LDAP Setup
    3. Restart Ambari process. Ex. $ sudo ambari-server restart
    4. Sync up LDAP users into Ambari metadata store so that Ambari Web UI can allow LDAP users to login.
      ** Be careful of this step. In the example is showing sync all users. You do not want to do this in a production large enterprise environment, you need to narrow it down to a selected targeted users or groups. Ex. $ sudo ambari-server sync-ldap –all (notice there is a double dashes before word “all”)

      ambari ldap sync
      Ambari LDAP Sync Up
  5. LDAP User Test in Ambari
    Now let’s test Ambari Web UI login as a LDAP user. (Screen shot below). Notice that Ambari Admin user needs to assign appropriate permission to each ldap user before s/he can see any screens when they logging in. If not, s/he still can login but will see a blank page.Screen shot #1:Notice all ldap users showed up after sync command. None of those users exist on current cluster nodes nor do you need to create them on cluster servers.

    ambari ldap user list
    Ambari LDAP User List

    Screen shot #2:Ambari administrator must assign permissions to the ldap users.

    ambari assign user permission
    Ambari Assign User Permission

    Screen shot #3:When Ambari Admin assigned LDAP users operator permission. S/He can then login and act as a Hadoop admin to monitoring and maintaining the cluster.

    ambari hadoop admin
    Ambari Hadoop Admin

LDAP / Knox Setup Steps

  1. Login to Ambari Web UI as administrator. (ie. “http://<ambari-server>:8080” default port is 8080. You can change the port in ambari.properties file)
  2. Click on “Knox” from left panel. Select “Configs” tab from right panel.
  3. Extend the section “Advanced topology”, and modify the following entries:
    • main.ldapRealm.userDnTemplate=uid={0},ou=people,dc=ibm,dc=com (modify to match with your ldap server settings)
    • main.ldapRealm.contextFactory.url=ldap://ldap2.ibm.com:389
    knox ldap setup
    Knox LDAP Setup
  4. Click on “Save” button from the upper right corner
  5. Restart “Knox” service via Ambari Web UI

BigInsights v4.x — Testing Knox Application LDAP Login with BigInsights Home Page

    1. URL: https://<biginsights_home_FQDN>:8443/gateway/default/BigInsightsWeb/index.html
    2. Login as a LDAP user ID and its password
BigInsights Home Page with LDAP User Login
BigInsights Home Page with LDAP User Login

Content of setup_ldap_client_v3_rhel.sh

#!/bin/bash
############################################
# File name: setup_ldap_client_v3_rhel.sh
# Version: 3.0
# Last Updated Date: 6/2/2015
############################################
if [ $# -ne 3 ]
then
  echo ""
  echo "Usage: setup-ldap-client.sh   <ldap-server-FQDN> <ldap-server-port> <BASE_DN>"
  echo "Example: setup-ldap-client.sh ldap2.ibm.com 389 dc=ibm,dc=com"
  echo ""
  exit 1
fi 

yum install openldap-clients sssd perl-LDAP.noarch nss-pam-ldapd -y

cat > /etc/openldap/ldap.conf << EOFDELIM
URI ldap://LDAP_SERVER:PORT
BASE BASEDN
TLS_CACERTDIR /etc/openldap/certs
EOFDELIM

sed -i "s/LDAP_SERVER/$1/" /etc/openldap/ldap.conf
sed -i "s/PORT/$2/" /etc/openldap/ldap.conf
sed -i "s/BASEDN/$3/" /etc/openldap/ldap.conf

cat > /etc/nslcd.conf << EOFDELIM
uri ldap://LDAP_SERVER:PORT
base BASEDN
ssl no
tls_cacertdir /etc/openldap/cacerts
EOFDELIM

sed -i "s/LDAP_SERVER/$1/" /etc/nslcd.conf
sed -i "s/PORT/$2/" /etc/nslcd.conf
sed -i "s/BASEDN/$3/" /etc/nslcd.conf

cat > /etc/pam_ldap.conf << EOFDELIM
uri ldap://LDAP_SERVER:PORT
base BASEDN
ssl no
tls_cacertdir /etc/openldap/cacerts
pam_password md5
EOFDELIM

sed -i "s/LDAP_SERVER/$1/" /etc/pam_ldap.conf
sed -i "s/PORT/$2/" /etc/pam_ldap.conf
sed -i "s/BASEDN/$3/" /etc/pam_ldap.conf

cat > /etc/pam.d/system-auth << EOFDELIM
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so
auth        sufficient    pam_ldap.so use_first_pass

account     required      pam_unix.so
account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3 type=
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

password    sufficient    pam_ldap.so use_authtok

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_ldap.so
# add if you need ( create home directory automatically if it's none )
session     optional      pam_mkhomedir.so skel=/etc/skel umask=077
EOFDELIM

cat > /etc/nsswitch.conf << EOFDELIM
passwd:     files ldap
shadow:     files ldap
group:      files ldap

#hosts:     db files nisplus nis dns
hosts:      files dns

# Example - obey only what nisplus tells us...
#services:   nisplus [NOTFOUND=return] files
#networks:   nisplus [NOTFOUND=return] files
#protocols:  nisplus [NOTFOUND=return] files
#rpc:        nisplus [NOTFOUND=return] files
#ethers:     nisplus [NOTFOUND=return] files
#netmasks:   nisplus [NOTFOUND=return] files

bootparams: nisplus [NOTFOUND=return] files

ethers:     files
netmasks:   files
networks:   files
protocols:  files
rpc:        files
services:   files

netgroup:   ldap

publickey:  nisplus

automount:  files ldap
aliases:    files nisplus
EOFDELIM

cat > /etc/sysconfig/authconfig << EOFDELIM
IPADOMAINJOINED=no
USEMKHOMEDIR=no
USEPAMACCESS=no
CACHECREDENTIALS=yes
USESSSDAUTH=no
USESHADOW=yes
USEWINBIND=no
USEDB=no
FORCELEGACY=no
USEFPRINTD=no
FORCESMARTCARD=no
PASSWDALGORITHM=md5
USELDAPAUTH=yes
USEPASSWDQC=no
IPAV2NONTP=no
USELOCAUTHORIZE=yes
USECRACKLIB=yes
USEIPAV2=no
USEWINBINDAUTH=no
USESMARTCARD=no
USELDAP=no
USENIS=no
USEKERBEROS=no
USESYSNETAUTH=no
USESSSD=no
USEHESIOD=no
USEMD5=yes
EOFDELIM

service nslcd start
chkconfig nslcd on

echo 'Welcome to LDAP World!!'

 

3 comments on"LDAP Integration for IOP and BigInsights"

  1. Thanks for your notification on script errors! -Linda Liu

  2. […] how to configure LDAP integration with IBM Open Platform on a BigInsights Cluster. Here is the link.In this post, it concentrates on the missing content from the previous post for the Microsoft […]

  3. […] how to configure LDAP integration with IBM Open Platform on a BigInsights Cluster. Here is the link.In this post, it concentrates on the missing content from the previous post for the Microsoft […]

Join The Discussion

Your email address will not be published. Required fields are marked *