INTRODUCTION:

The log directories are defined to contain the standard output/ errors and standard messages from the services. Sometimes, the current log directories may get filled and users may have a need to change their existing log directories. This article addresses this requirement.

The default log location is located under the “Configs → Advanced” tab on the AmbarI UI for all IOP services. This location can be changed for a few services directly from the UI and few other services the log location field is disabled. This article is to demonstrate changing log directories of those IOP services where the default log location is disabled.

The article was developed by Mohan Dani (mohadani@in.ibm.com), Samir Katti (samikatt@in.ibm.com) and Ramesh Gopal (ramesrin@in.ibm.com)

ASSUMPTION:

1. IOP is pre-installed with default log locations for services
2. All the scripts are executed as root user
3. Restart of services is done via Ambari GUI
4. Version applicable : BigInsights 4.1.0.0 and above

SERVICES APPLICABLE

The services mentioned in Table 1 have their log field location disabled ( i.e. not editable by ambari user ) . A snapshot of such a service is shown below:

Advanced hbase-env

HBase Log Dir 
HBase PID Dir 

Table 1

SERVICE NAME LOG FIELD NAME DEFAULT LOCATION
HDFS Hadoop Log Dir Prefix /var/log/hadoop
Mapreduce Mapreduce Log Dir Prefix /var/log/hadoop-mapreduce
Yarn Yarn Log Dir Prefix /var/log/hadoop-yarn
Hive Hive Log Dir /var/log/hive
Hbase Hbase Log Dir /var/log/hbase
Oozie

Oozie Log Dir /var/log/oozie
Zookeeper Zookeeper Log Dir /var/log/zookeeper
Flume Flume Log Dir /var/log/flume

Changing the default Log locations

1) As root user ensure the following script exists with the right permissions:

       /var/lib/ambari-server/resources/scripts/configs.sh 

  ll /var/lib/ambari-server/resources/scripts/configs.sh
  -rwxr-xr-x 1 root root 9391 Oct 28  2015 /var/lib/ambari-server/resources/scripts/configs.sh   

2) Stop the service for which the default log directory needs to be modified

3) Execute the above script as root user as follows :

configs.sh [-u userId] [-p password] [-port port] [-s] [CONFIG_FILENAME | CONFIG_KEY [CONFIG_VALUE]]

Eg for hive service:

  configs.sh [-u userId] [-p password] [-port port] [-s] set   hive-env hive_log_dir  
[-u userId]: Optional user ID to use for authentication. Default is 'admin'.
[-p password]: Optional password to use for authentication. Default is 'admin'.
[-port port]: Optional port number for Ambari server. Default is '8080'. Provide empty string to not use port.
[-s]: Optional support of SSL. Default is 'false'. Provide empty string to not use SSL.
 : One of 'get', 'set', 'delete'. 'Set' adds/updates as necessary.
 : Server external host name
 : Name given to cluster. Ex: 'c1'
 : One of the various configuration types in Ambari. Ex:global, core-site, hdfs-site, mapred-queue-acls, etc.
[CONFIG_FILENAME]: File where entire configurations are saved to, or read from. Only applicable to 'get' and 'set' actions
[CONFIG_KEY]: Key that has to be set or deleted. Not necessary for 'get' action.
[CONFIG_VALUE]: Optional value to be set. Not necessary for 'get' or 'delete' actions.

To provide corresponding values for services listed in Table 1, refer to Table 2 for CONFIG_FILENAME , CONFIG_KEY

Table 2

SERVICE NAME CONFIG_FILENAME CONFIG_KEY
HDFS hadoop-env hdfs_log_dir_prefix
Mapreduce mapred-env mapreduce_log_dir_prefix
Yarn yarn-env yarn_log_dir_prefix
Hive hive-env hive_log_dir / hcat_log_dir
Hbase hbase-env hbase_log_dir
Oozie

oozie-env oozie_log_dir
Zookeeper zookeeper-env zk_log_dir
Flume flume-env flume_log_dir

4) Restart the service and ensure the new log directory is reflected back on the Ambari UI. A sample snapshot is shown below :

Hive Log Dir 

1 comment on"Changing Default Log Directories for BigInsights IOP Services"

  1. Nice article, thank you!

Join The Discussion

Your email address will not be published. Required fields are marked *