Lab 3: Exploring and administering your cluster with the BigInsights Web console

As you saw in the previous lab, Apache Hadoop users typically work through a command line interface to perform many common tasks. This lab introduces you to the BigInsights Web console, which enables you to administer your cluster, work with HDFS, launch jobs, and perform many other tasks using a graphical interface.

After completing this hands-on lab, you’ll be able to:

  • Launch the web console.
  • Work with popular resources accessible through the Welcome page.
  • Administer BigInsights by inspecting the status of your cluster and accessing tools for open source components provided with BigInsights.
  • Work with the distributed file system. In particular, you’ll explore the HDFS directory structure, create subdirectories, and upload files to HDFS.
  • Manage and launch pre-built applications from a Web catalog.
  • Inspect the status of previously launched applications (jobs) and review their output.

Allow 30 minutes to complete this section of lab. Prior to this lab, you should have set up a working environment. See Getting Started with Hadoop and BigInsights for details.

This lab is an introduction to a subset of console functions. Real-time monitoring, dashboards, alerts, and application linking are among the more advanced console functions that are beyond this lab’s scope.

Please post questions or comments about this lab to the forum on Hadoop Dev at https://developer.ibm.com/answers?community=hadoop.

3.1. Getting started with the web console

In this exercise, you will launch the console and inspect its Welcome page.

__1. Launch the BigInsights web console. Direct your browser to http://bivm.ibm.com:8080 or click the Web Console icon on your desktop.

image001

__2. Log in with your user name and password (biadmin / biadmin).

image002

__3. Verify that your console appears similar to this:

image003

__4. Briefly skim through the links provided in these sections to become familiar with resources available to you:

Tasks: Quick access to popular BigInsights tasks
Quick Links: Links to internal and external quick links and downloads to enhance your environment
Learn More: Online resources available to learn more about BigInsights

3.2. Administering BigInsights

The web console allows administrators to inspect the overall health of the system as well as perform basic functions, such as starting and stopping specific servers or components, adding nodes to the cluster, and so on. You’ll explore a subset of these capabilities here.

__5. Click on the Cluster Status tab at the top of the page.

image004

__6. Inspect the overall status of your cluster. The figure below was taken on a single-node cluster that had several services running. One service – Monitoring — was unavailable. Your display may differ somewhat. It’s not necessary for all BigInsights services to be running to complete the exercises in this lab.

image005

__7. Click on the Hive service and note the detailed information provided for this service in the pane at right. For example, you can see the URL for Hive’s Web interface and its process ID. In addition, note that you can start and stop services (such as the Hive service) from the Cluster Status page of the console.

image006

__8. Optionally, cut-and-paste the URL for Hive’s web interface into a new tab of your browser. You’ll see an open source tool provided with Hive for administration purposes, as shown below.

image007

Other open source tools provided with Apache Hadoop are also available through IBM’s packaged distribution (BigInsights), as you’ll see shortly. Close this browser tab.

__9. Click on the Welcome page of your web console.

__10. Click on the Access secure cluster servers button in the Quick Links section at right.

image008

If nothing appears, verify that the pop-up blocker of your browser is disabled; a prompt should appear at the top of the page if pop-ups are blocked.

__11. Inspect the list of server components for which there are additional web-based tools. The BigInsights console displays the URLs you can use to access each of these web sites directly. (This information will only appear if the pop-up blocker is disabled on browser.)

image009

__12. Click on the jobtracker alias. The display should be familiar to you — it’s the same one you saw in the previous lab that introduced you to some basic Hadoop facilities.

image010

3.3. Working with the distributed file system (HDFS)

In this section, you’ll learn how to use the console to create directories in HDFS, navigate the file system, and upload small files — tasks you performed earlier through a command-line interface. In addition, you’ll perform a few other file-related tasks as well. Many people find the console’s graphical interface to be easier to use than the command-line interface.

__1. Click on the Files tab at the top of the page.

image011

__2. Expand the DFS directory tree in the left pane to display the contents of /user/biadmin. Note the presence of the /WordCount_output and /test subdirectories, which you created in an earlier lab. If desired, expand each directory and inspect its contents.

image012

__3. Become familiar with the functions provided through the icons at the top of this pane, as we’ll refer to some of these in subsequent sections of this module. Simply position your cursor on each icon to learn its function. From left to right, the icons enable you to copy a file or directory, move a file, create a directory, rename a file or directory, upload a file to HDFS, download a file from HDFS to your local file system, remove a file or directory from HDFS, set permissions, open a command window to launch HDFS shell commands, and refresh the web console page.

image013

__4. Delete the /user/biadmin/test directory and its contents. Position your cursor on this directory, click the red X icon, and click Yes when prompted.

image014

image015

__5. Create a new subdirectory in /user/biadmin. With your cursor positioned on /user/biadmin, click the create directory icon.

image016

__6. When a pop-up window appears, specify test2 as the new directory’s name and click OK.

image017

__7. Expand the directory hierarchy to verify that your new subdirectory was created.

image018

__8. Upload a file into this directory from your local file system. Click the upload icon.

image019

__9. When a pop-up window appears, click the Browse button to navigate through your local file system to /home/biadmin/licenses. Select the BIlicense_en.txt file and click Open.

image020

__10. Expand the /user/biadmin/test2 directory and verify that the BIlicense_en.txt file was successfully copied into HDFS. Note that the right pane of the Web console previews the file’s contents.

image021

3.4. Managing and launching pre-built applications from the web catalog

The web console includes a catalog of ready-made applications that users can launch through a graphical interface. Each application’s status, execution history, and output are easy to monitor from this page as well. In this exercise, you’ll first manage the catalog’s contents, selecting one of more than 20 pre-built applications provided with BigInsights to deploy on your cluster. Once deployed, the application will be visible to all authorized users. You’ll then launch the application, monitor its execution status, and inspect its output.

As you might have guessed, the sample application used in this lab is Word Count — the same application you ran from a command line earlier.

__1. Click the Applications tab of the Web console. No applications are deployed on a new cluster, so there won’t be much to see yet.

image022

__2. In the upper left corner, click Manage. A list of applications available for deployment are displayed.

image023

__3. Expand the Test category and click on the Word Count application.

image024

__4. Click Deploy.

__5. When a pop-up window appears, accept the defaults for all settings and click Deploy.

image025

__6. After the application has been deployed, you’re ready to run it. Click Run in the upper left pane.

image026

__7. Verify that the Word Count application appears in the catalog. (Any other applications that were previously deployed to the Web catalog will also appear.)

image027

__8. Click on the Word Count icon. The pane at right prompts you to enter appropriate information. For this application, you need to specify an execution name for your application’s run, the HDFS directory containing the input document(s) for the Word Count application, and an output directory in HDFS.

image028

__9. For the Execution name, enter My Test Run 1.

__10. For the Input path, click Browse and navigate to /user/biadmin/test2. Click OK.

image029

__11. For the Output path, type /user/biadmin/WordCount_console_output. (Recall that the Word Count application creates this output directory at run time. If you specify an existing HDFS directory for the output, the application will fail.)

__12. Verify that your display appears similar to this and click Run.

image030

__13. As your application executes, monitor its status through the Application History pane at lower right.

image031

__14. When the application completes successfully, click the link provided in the Output column to see the application’s output.

image032

__15. Optionally, return to the Applications page of the console and click on the link provided in the Details column for your application’s run.

image033

__16. Note that the console displays the Application Status page, which contains information about the Oozie workflow for your application as well as the application itself. If desired, click on one or more available links to explore details available for your review.

image034

To find the other tutorials in this series, go to Overview tutorial.

Join The Discussion

Your email address will not be published. Required fields are marked *