INTRODUCTION:
The log directories are defined to contain the standard output/ errors and standard messages from the services. Sometimes, the current log directories may get filled and users may have a need to change their existing log directories. This article addresses this requirement.
The default log location is located under the “Configs → Advanced” tab on the AmbarI UI for all IOP services. This location can be changed for a few services directly from the UI and few other services the log location field is disabled. This article is to demonstrate changing log directories of those IOP services where the default log location is disabled.
The article was developed by Mohan Dani (mohadani@in.ibm.com), Samir Katti (samikatt@in.ibm.com) and Ramesh Gopal (ramesrin@in.ibm.com)
ASSUMPTION:
1. IOP is pre-installed with default log locations for services
2. All the scripts are executed as root user
3. Restart of services is done via Ambari GUI
4. Version applicable : BigInsights 4.1.0.0 and above
SERVICES APPLICABLE
The services mentioned in Table 1 have their log field location disabled ( i.e. not editable by ambari user ) . A snapshot of such a service is shown below:
Advanced hbase-env
HBase Log Dir HBase PID Dir
Table 1
| SERVICE NAME | LOG FIELD NAME | DEFAULT LOCATION |
|---|---|---|
| HDFS | Hadoop Log Dir Prefix | /var/log/hadoop |
| Mapreduce | Mapreduce Log Dir Prefix | /var/log/hadoop-mapreduce |
| Yarn | Yarn Log Dir Prefix | /var/log/hadoop-yarn |
| Hive | Hive Log Dir | /var/log/hive |
| Hbase | Hbase Log Dir | /var/log/hbase |
| Oozie | Oozie Log Dir | /var/log/oozie |
| Zookeeper | Zookeeper Log Dir | /var/log/zookeeper |
| Flume | Flume Log Dir | /var/log/flume |
Changing the default Log locations
1) As root user ensure the following script exists with the right permissions:
/var/lib/ambari-server/resources/scripts/configs.sh ll /var/lib/ambari-server/resources/scripts/configs.sh -rwxr-xr-x 1 root root 9391 Oct 28 2015 /var/lib/ambari-server/resources/scripts/configs.sh
2) Stop the service for which the default log directory needs to be modified
3) Execute the above script as root user as follows :
configs.sh [-u userId] [-p password] [-port port] [-s] [CONFIG_FILENAME | CONFIG_KEY [CONFIG_VALUE]]
Eg for hive service:
configs.sh [-u userId] [-p password] [-port port] [-s] set hive-env hive_log_dir
[-u userId]: Optional user ID to use for authentication. Default is 'admin'. [-p password]: Optional password to use for authentication. Default is 'admin'. [-port port]: Optional port number for Ambari server. Default is '8080'. Provide empty string to not use port. [-s]: Optional support of SSL. Default is 'false'. Provide empty string to not use SSL. : One of 'get', 'set', 'delete'. 'Set' adds/updates as necessary. : Server external host name : Name given to cluster. Ex: 'c1' : One of the various configuration types in Ambari. Ex:global, core-site, hdfs-site, mapred-queue-acls, etc. [CONFIG_FILENAME]: File where entire configurations are saved to, or read from. Only applicable to 'get' and 'set' actions [CONFIG_KEY]: Key that has to be set or deleted. Not necessary for 'get' action. [CONFIG_VALUE]: Optional value to be set. Not necessary for 'get' or 'delete' actions.
To provide corresponding values for services listed in Table 1, refer to Table 2 for CONFIG_FILENAME , CONFIG_KEY
Table 2
| SERVICE NAME | CONFIG_FILENAME | CONFIG_KEY |
|---|---|---|
| HDFS | hadoop-env | hdfs_log_dir_prefix |
| Mapreduce | mapred-env | mapreduce_log_dir_prefix |
| Yarn | yarn-env | yarn_log_dir_prefix |
| Hive | hive-env | hive_log_dir / hcat_log_dir |
| Hbase | hbase-env | hbase_log_dir |
| Oozie | oozie-env | oozie_log_dir |
| Zookeeper | zookeeper-env | zk_log_dir |
| Flume | flume-env | flume_log_dir |
4) Restart the service and ensure the new log directory is reflected back on the Ambari UI. A sample snapshot is shown below :
Hive Log Dir