Using SSL for Apache Ambari Server & Agent Communication - Hadoop Dev

Introduction

Apache Ambari provides options to secure all its communication using SSL.  An Ambari administrator can configure the Ambari server to use the HTTPS protocol for all end-user interaction both from the browser and REST APIs.  For additional security, the administrator can configure the Ambari server to secure all its internal communication with its agents  using 2-way SSL.  In this post, the process of securing Apache Ambari’s communication using SSL for communications between Ambari server and users will be described. In addition, the process for configuring SSL communication between Ambari server and its agents using both self-signed certificate as well as CA-signed certificate will also be discussed.

Configure Ambari Server to use HTTPS

For HTTPS setup, it’s best to use a CA-signed certificate for better security.  Though a self-signed certificate can be used, it should be used for development and testing purpose only.   If a self-signed certificate is to be used, it must first be generated using a number of tools. Below is an example of how to generate a certificate using openssl.

First, generate the private key.  Note that an optional password can be supplied, and the sample name “nodename” can be customized if desire:

openssl genrsa -out nodename.key 2048

Next, generate certificate signing request using the newly created key file:

openssl req -new -key nodename.key -out nodename.csr

Thirdly, generate and sign the certificate:

openssl x509 -req -in nodename.csr -signkey nodename.key -out nodename.crt

Note that Ambari Server only handles PEM-encoded certificate.  If needed in the case where the certificate was DER-encoded, modify the following sample command to convert a DER-encoded certificate to a PEM-encoded certificate:

openssl x509 -in cert.der -outform pem -out cert.crt

With the self-signed or CA-signed certificate and its associated key file, Ambari server can be configured for HTTPS:

• Log into the Ambari server host.
• Stop the Ambari server by issuing: ambari-server stop
• Locate the self–signed or CA-signed certificate artifacts that you want to use.
• Enter the the Ambari security setup interface ambari-server setup-security command and select option [1] Enable HTTPS for Ambari server:

# ambari-server setup-security  Using python  /usr/bin/python2.6 Security setup options...  ==========================================================================  Choose one of the following options:    [1] Enable HTTPS for Ambari server.    [2] Encrypt passwords stored in ambari.properties file.   [3] Setup Ambari kerberos JAAS configuration.    [4] Setup truststore.    [5] Import certificate to truststore.  ==========================================================================  Enter choice, (1-5): 1

• Answer the prompts that appear:

Do you want to configure HTTPS [y/n] (y)? y  SSL port [8443] ? 9443  Enter path to Certificate: /path/to/cert-file.crt  Enter path to Private Key: /path/to/key-file.key  Please enter password for Private Key:  Generating random password for HTTPS keystore...done.  Importing and saving Certificate...done.  Adjusting ambari-server permissions and ownership...

— When asked whether you want to configure HTTPS, type y.
— Select the port that you want to use for SSL. The default is 8443.
    Important: Make sure that you choose a port that is not being used by any services on the machine. For example, the default port for Knox is also 8443.
— Provide the path to your certificate and your private key.
— Provide the password for the private key if configured.

• Start the Ambari Server:

# ambari-server start  Using python  /usr/bin/python2.6  Starting ambari-server  Ambari Server running with administrator privileges.  Organizing resource files at /var/lib/ambari-server/resources...  Server PID at: /var/run/ambari-server/ambari-server.pid  Server out at: /var/log/ambari-server/ambari-server.out  Server log at: /var/log/ambari-server/ambari-server.log  Waiting for server start....................  Ambari Server 'start' completed successfully.

Note: after the HTTPS setup, by default Ambari will store the certificate, key, and the created keystore in the /var/lib/ambari-server/keys directory under the these file names:  https.crt, https.key, https.keystore.p12.  The keystore password is stored in the https.pass.txt file

# pwd  /var/lib/ambari-server/keys  [root@beep1 keys]# ls -l https*  -rwx------ 1 root root  847 May 25 13:44 https.crt  -rwx------ 1 root root  963 May 25 13:44 https.key  -rwx------ 1 root root 2229 May 25 13:44 https.keystore.p12  -rwx------ 1 root root    5 May 25 13:44 https.pass.txt

Once the Ambari server is restarted successfully, all Web UI and REST APIs communication with Ambari Server will need to use the HTTPS URL with the correct configured port.

Configure Ambari Server and Ambari Agents SSL communication

By default, Ambari server will always generate its own self-signed certificate for 1-way SSL usage between the agent and server.  Upon startup, Ambari server will check for the existence of the SSL artifacts using the default values of the following properties defined in the Ambari server properties file /etc/ambari-server/conf/ambari.properties (default values are used if the properties are not defined):

security.server.cert_name=ca.crt  security.server.key_name=ca.key  security.server.keystore_name=keystore.p12  security.server.truststore_name=keystore.p12  security.server.crt_pass_file=pass.txt

If the named SSL artifacts do not exist, Ambari server will generate them automatically.

# pwd  /var/lib/ambari-server/keys  [root@beep1 keys]# ls -l  total 32  -rwx------ 1 root root  803 May 25 09:45 ca.config  -rwx------ 1 root root 7134 May 25 09:53 ca.crt  -rwx------ 1 root root 1647 May 25 09:53 ca.csr  -rwx------ 1 root root 3311 May 25 09:53 ca.key  drwx------ 3 root root  114 May 25 09:53 db  -rwx------ 1 root root 5677 May 25 09:53 keystore.p12  -rw------- 1 root root   50 May 25 09:46 pass.txt

If there is a need to customize these values,  administrator can define these properties with the customized values in the /etc/ambari-server/conf/ambari.properties file on the Ambari server node.

Ambari agents, at startup, will communicate with Ambari server using 1-way SSL via the URL: https://$AMBARI_SERVER_ADDRESS:8440.   The agents will check with whether 2-ways SSL is configured. If so, it will configure itself for 2-way SSL.

Using self-signed certificate for 2-way SSL

To enable Ambari Server and Agents 2-way SSL communication, the simplest way is to let Ambari manage all the certificates between the server and the agents.   All that is needed is to configure the Ambari Server’s 2-way SSL property.

On the Ambari server node, edit the /etc/ambari-server/conf/ambari.properties file and make ensure this property is properly set:

security.server.two_way_ssl=true

Once the property is configured, restart Ambari-server on the server node, and Ambari-agents on each nodes:

ambari-server restart  ambari-agent restart

Once the server and agents are restarted, Ambari would generate the proper certificates for each Ambari agent signed by the Ambari server’s self-signed certificate.   The SSL certificate related artifacts can be found in /var/lib/ambari-server/keys and /var/lib/ambari-agent/keys directories.

Using CA-signed certificate for 2-way SSL

In order to use CA-signed certificate, Ambari needs to be told about the CA-signed certificate and its associated artifacts since the self-signed certificate is already created on the very first Ambari-server start.

First, stop Ambari server and agents on each node:

ambari-server stop  ambari-agent stop

Next, cleanup any existing Ambari server keys information by:

• Backing up and delete all the crt and csr files that starts with hostname at /var/lib/ambari-server/keys.
• Empty /var/lib/ambari-server/keys/db/index.txt file
• Delete any certificates under /var/lib/ambary-server/keys/db/newcerts/

After that, configure CA-signed certificate artifacts in Ambari Server by:

• Copy the CA-signed certificate, key files to /var/lib/ambari-server/keys/ e.g. your_CA.crt, your_CA.key
• Create PKCS keystore file from your cert and key files.

openssl pkcs12 -export -in your_CA.crt -inkey /var/lib/ambari-server/keys/your_CA.key -certfile your_CA.crt -out keystore_CA.p12 -password pass:KEYSTOREPASSWORD -passin pass:KEYPASSWORD

• Create a text file with appropriate password that has been provided for the keystore in /var/lib/ambari-server/keys directory:

echo KEYSTOREPASSWORD > pass_CA.txt

Next, configure Ambari to use the new cert, keys and keystore files:

Edit the /etc/ambari-server/conf/ambari.properties and set the following keys and values:

security.server.cert_name=your_CA.crt  security.server.key_name=your_CA.key  security.server.keystore_name=keystore_CA.p12  security.server.truststore_name=keystore_CA.p12  security.server.crt_pass_file=pass_CA.txt  security.server.two_way_ssl=true

After adding the properties, cleanup existing Ambari agent keys information by removing all files in the /var/lib/ambari-agent/keys/ directory on each node that has Ambari agent installed.

The last step is to start Ambari server and agents.

ambari-server start  ambari-agent start

The ambari-server log in /var/log/ambari-server/ambari-server.log will contain SSL errors if there is any issue

If everything was done correctly, Ambari server and agents should start successfully and communicating using SSL.