As you saw in the previous lab, Apache Hadoop users typically work through a command line interface to perform many common tasks. This lab introduces you to the BigInsights Web console, which enables you to administer your cluster, work with HDFS, launch jobs, and perform many other tasks using a graphical interface.
After completing this hands-on lab, you’ll be able to:
- Launch the web console.
- Work with popular resources accessible through the Welcome page.
- Administer BigInsights by inspecting the status of your cluster and accessing tools for open source components provided with BigInsights.
- Work with the distributed file system. In particular, you’ll explore the HDFS directory structure, create subdirectories, and upload files to HDFS.
- Manage and launch pre-built applications from a Web catalog.
- Inspect the status of previously launched applications (jobs) and review their output.
Allow 30 minutes to complete this section of lab. Prior to this lab, you should have set up a working environment. See Getting Started with Hadoop and BigInsights for details.
This lab is an introduction to a subset of console functions. Real-time monitoring, dashboards, alerts, and application linking are among the more advanced console functions that are beyond this lab’s scope.
Please post questions or comments about this lab to the forum on Hadoop Dev at https://developer.ibm.com/answers?community=hadoop.
In this exercise, you will launch the console and inspect its Welcome page.
__1. Launch the BigInsights web console. Direct your browser to http://bivm.ibm.com:8080 or click the Web Console icon on your desktop.
__2. Log in with your user name and password (biadmin / biadmin).
__3. Verify that your console appears similar to this:
__4. Briefly skim through the links provided in these sections to become familiar with resources available to you:
Tasks: Quick access to popular BigInsights tasks
Quick Links: Links to internal and external quick links and downloads to enhance your environment
Learn More: Online resources available to learn more about BigInsights
The web console allows administrators to inspect the overall health of the system as well as perform basic functions, such as starting and stopping specific servers or components, adding nodes to the cluster, and so on. You’ll explore a subset of these capabilities here.
__5. Click on the Cluster Status tab at the top of the page.
__6. Inspect the overall status of your cluster. The figure below was taken on a single-node cluster that had several services running. One service – Monitoring — was unavailable. Your display may differ somewhat. It’s not necessary for all BigInsights services to be running to complete the exercises in this lab.
__7. Click on the Hive service and note the detailed information provided for this service in the pane at right. For example, you can see the URL for Hive’s Web interface and its process ID. In addition, note that you can start and stop services (such as the Hive service) from the Cluster Status page of the console.
__8. Optionally, cut-and-paste the URL for Hive’s web interface into a new tab of your browser. You’ll see an open source tool provided with Hive for administration purposes, as shown below.
Other open source tools provided with Apache Hadoop are also available through IBM’s packaged distribution (BigInsights), as you’ll see shortly. Close this browser tab.
__9. Click on the Welcome page of your web console.
__10. Click on the Access secure cluster servers button in the Quick Links section at right.
If nothing appears, verify that the pop-up blocker of your browser is disabled; a prompt should appear at the top of the page if pop-ups are blocked.
__11. Inspect the list of server components for which there are additional web-based tools. The BigInsights console displays the URLs you can use to access each of these web sites directly. (This information will only appear if the pop-up blocker is disabled on browser.)
__12. Click on the jobtracker alias. The display should be familiar to you — it’s the same one you saw in the previous lab that introduced you to some basic Hadoop facilities.
In this section, you’ll learn how to use the console to create directories in HDFS, navigate the file system, and upload small files — tasks you performed earlier through a command-line interface. In addition, you’ll perform a few other file-related tasks as well. Many people find the console’s graphical interface to be easier to use than the command-line interface.
__1. Click on the Files tab at the top of the page.
__2. Expand the DFS directory tree in the left pane to display the contents of /user/biadmin. Note the presence of the /WordCount_output and /test subdirectories, which you created in an earlier lab. If desired, expand each directory and inspect its contents.
__3. Become familiar with the functions provided through the icons at the top of this pane, as we’ll refer to some of these in subsequent sections of this module. Simply position your cursor on each icon to learn its function. From left to right, the icons enable you to copy a file or directory, move a file, create a directory, rename a file or directory, upload a file to HDFS, download a file from HDFS to your local file system, remove a file or directory from HDFS, set permissions, open a command window to launch HDFS shell commands, and refresh the web console page.
__4. Delete the /user/biadmin/test directory and its contents. Position your cursor on this directory, click the red X icon, and click Yes when prompted.
__5. Create a new subdirectory in /user/biadmin. With your cursor positioned on /user/biadmin, click the create directory icon.
__6. When a pop-up window appears, specify test2 as the new directory’s name and click OK.
__7. Expand the directory hierarchy to verify that your new subdirectory was created.
__8. Upload a file into this directory from your local file system. Click the upload icon.
__9. When a pop-up window appears, click the Browse button to navigate through your local file system to /home/biadmin/licenses. Select the BIlicense_en.txt file and click Open.
__10. Expand the /user/biadmin/test2 directory and verify that the BIlicense_en.txt file was successfully copied into HDFS. Note that the right pane of the Web console previews the file’s contents.
The web console includes a catalog of ready-made applications that users can launch through a graphical interface. Each application’s status, execution history, and output are easy to monitor from this page as well. In this exercise, you’ll first manage the catalog’s contents, selecting one of more than 20 pre-built applications provided with BigInsights to deploy on your cluster. Once deployed, the application will be visible to all authorized users. You’ll then launch the application, monitor its execution status, and inspect its output.
As you might have guessed, the sample application used in this lab is Word Count — the same application you ran from a command line earlier.
__1. Click the Applications tab of the Web console. No applications are deployed on a new cluster, so there won’t be much to see yet.
__2. In the upper left corner, click Manage. A list of applications available for deployment are displayed.
__3. Expand the Test category and click on the Word Count application.
__4. Click Deploy.
__5. When a pop-up window appears, accept the defaults for all settings and click Deploy.
__6. After the application has been deployed, you’re ready to run it. Click Run in the upper left pane.
__7. Verify that the Word Count application appears in the catalog. (Any other applications that were previously deployed to the Web catalog will also appear.)
__8. Click on the Word Count icon. The pane at right prompts you to enter appropriate information. For this application, you need to specify an execution name for your application’s run, the HDFS directory containing the input document(s) for the Word Count application, and an output directory in HDFS.
__9. For the Execution name, enter My Test Run 1.
__10. For the Input path, click Browse and navigate to /user/biadmin/test2. Click OK.
__11. For the Output path, type /user/biadmin/WordCount_console_output. (Recall that the Word Count application creates this output directory at run time. If you specify an existing HDFS directory for the output, the application will fail.)
__12. Verify that your display appears similar to this and click Run.
__13. As your application executes, monitor its status through the Application History pane at lower right.
__14. When the application completes successfully, click the link provided in the Output column to see the application’s output.
__15. Optionally, return to the Applications page of the console and click on the link provided in the Details column for your application’s run.
__16. Note that the console displays the Application Status page, which contains information about the Oozie workflow for your application as well as the application itself. If desired, click on one or more available links to explore details available for your review.
To find the other tutorials in this series, go to Overview tutorial.