Overview:

Ambari Server is a single point of failure, so any kind of a situation where-in the current Ambari Server is down due to bad disk or disk corruption/ abrupt network outage etc. , will lead the system to be unusable. This document provides detail steps involved to deal restoring Ambari Server onto a new node when the ambari DB is on MySQL. This is valid for BigInsights 4.0 onwards.

Prerequisites:

1. Since we are restoring the data from an existing Ambari Server node to a new node, one is expected to take regular back-ups.
2. It is mandatory to have root user access
3. All system level user exists on the new host and the UIDs must be consistent with the UIDs on the rest of the Ambari Agent nodes.
4. MySQL has to be installed on the new ambari server node.

Considering the above prerequisites are met, please use the steps mentioned below to restore Ambari Server on the new node for MySQL. For Postgres, please use the link https://developer.ibm.com/hadoop/blog/2015/10/08/back-up-and-restore-ambari-server-postgresql-database/

References:

FQDN – Fully Qualified Domain Name refers to the full host name of the node.

Procedure:

A) Backup the database Mysql on current Ambari Server

As ambari user, on the Ambari Server node (node6.ibm.com), run the following command to stop Ambari Server : ambari-server stop
As ambari user, on each Ambari Agent node, run the following command to stop Ambari Agent : ambari-agent stop
On the Ambari Server node, as the root user, run the following command to export the databases used by Ambari Server.
If the Ambari-server metadata is in mysql perform the below step with root user.
mysqldump -u ambari -pbigdata ambari > /tmp/ambari_md_bkp
Enter bigdata if prompted for password.

B) Steps to configure MySQL database on the new host

After installation of MySQL, follow the below mentioned steps to create users, databases and grant privileges for ambari, hive and oozie respectively.

Run mysql command, this will create the mysql prompt.

create database ambari;
CREATE USER ‘ambari’@’%’ IDENTIFIED BY ‘bigdata’;
CREATE USER ‘ambari’@’localhost’ IDENTIFIED BY ‘bigdata’;
CREATE USER ‘ambari’@” IDENTIFIED BY ‘bigdata’;
GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@’%’;
GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@’localhost’;
GRANT ALL PRIVILEGES ON *.* TO ‘ambari’@”;
create database hive;
CREATE USER ‘hive’@’%’ IDENTIFIED BY hive’;
CREATE USER ‘hive’@’localhost’ IDENTIFIED BY hive’;
CREATE USER ‘hive’@” IDENTIFIED BY ‘hive’;
GRANT ALL PRIVILEGES ON *.* TO ‘hive’@’%’;
GRANT ALL PRIVILEGES ON *.* TO ‘hive’@’localhost’;
GRANT ALL PRIVILEGES ON *.* TO ‘hive’@”;
create database oozie;
CREATE USER ‘oozie’@’%’ IDENTIFIED BY ‘oozie’;
CREATE USER ‘oozie’@’localhost’ IDENTIFIED BY ‘oozie’;
CREATE USER ‘oozie’@” IDENTIFIED BY ‘oozie’;
GRANT ALL PRIVILEGES ON *.* TO ‘oozie’@’%’;
GRANT ALL PRIVILEGES ON *.* TO ‘oozie’@’localhost’;
GRANT ALL PRIVILEGES ON *.* TO ‘oozie’@”;

C) Install Ambari Server, Ambari Agent and setup the Ambari Server to use MySQL on this new node.

Setup the ambari repo under /etc/yum.repos.d
service mysql start
yum install ambari-server
yum install ambari-agent
ambari-server setup (Select mysql as the database else default is postgres)
ambari-server start
ambari-agent start

D) Restore the databases on the newly configured Ambari Server

MYSQL:

On the new Ambari Server node, as the root user, run the following command to restore the mysql database created earlier.
mysql -u ambari -p ambari < /tmp/metadata/ambari_md_bkp
Enter bigdata when prompted for password.

E) On each Ambari Agent node, please verify that the /etc/ambari-agent/conf/ambari-agent.ini refers to the hostname of the new node.
ex:
[server] hostname=FQDN

F) Start Ambari Server using commands “ambari-server restart”.

G) Start Ambari Agents on all other nodes using command “ambari-agent restart”

H) Login to new Ambari Server to view the status and monitor the cluster.
Ex: http:// FQDN:8080/

Note:

The steps mentioned above is generic and can be re-used once the original Ambari node is repaired. This document is to Ambari Server restore only. After moving the Ambari Server to a new node, the Ambari web UI will show other services installed on the old node as heartbeat unknown. It has to be done manually which is outside the scope of the document.

Document owners:

Raviprasad Pentakota (rapentak@in.ibm.com)
Ramesh G Srinivasan (ramesrin@in.ibm.com)
Samir Katti(samikatt@in.ibm.com)
Mohan N Dani(mohadani@in.ibm.com)

Join The Discussion

Your email address will not be published. Required fields are marked *