Moving to HDP: An Overview
Hortonworks and IBM announced an expanded partnership at 2017 DataWorks Summit in San Jose. The expanded partnership focused on extending Data Science and Machine Learning to more developers across the Apache Hadoop ecosystem and adopting HDP for IBM’s Hadoop distribution. In addition to offering a fully integrated Data Science and Machine Learning experience, Hortonworks and IBM are combining HDP with Big SQL (IBM’s SQL engine for Hadoop) for improved query optimization and support of complex analytics with SQL on Big Data.
Since the announcement, Hortonworks and IBM have been working closely to pave a smooth migration path from IBM IOP to HDP. The goal of this post is to review the high-level migration process and outline the differences between the IOP and HDP stacks.
IOP to HDP Migration
The migration process supports moving customers from IOP 4.2.0 and IOP 4.2.5 to HDP 2.6.2. Apache Ambari 2.5.2 is used to automate the upgrade and only requires a few manual steps to complete the migration. The diagram below highlights the steps involved in this process:
Prepare
To make sure the upgrade is as smooth as possible, we have a few specific recommendations on how to perform configuration, and database backups, and how to ensure the cluster is in a healthy state before upgrading. The migration documentation will walk customers through how backups should be taken and details how to confirm services are running as expected before the upgrade.
It’s important to note that besides Ambari configuration, and Ambari, Hive, Ranger, Oozie and Big SQL database backups, no other data backups are required for the migration. The data stored in HDFS will be persisted and all metadata will be migrated as part of the upgrade.
Upgrade Ambari
Ambari 2.5.2 has been configured to manage both the IOP and HDP stacks, and handle migrating customers from IOP to HDP. Once the existing Ambari Server and Agents are upgraded, they will be used to manage and automate the rest of the migration process.
Remove Services & Components
The tables below highlight the specific services and components that are part of IOP that will not be supported or migrated when moving to HDP. These services and components will need to be removed before starting the upgrade. The version of Ambari that is used for automating the upgrade has been specially configured to make the removal of these unsupported services easy and convenient.
Register and Install HDP Version & Binaries
Using Ambari, the target HDP version will be registered, and the HDP packages will be installed side-by-side with the existing IOP packages. When all packages have been successfully installed, and all prerequisites have been met, the Ambari Express Upgrade wizard will be used to perform the in-place IOP to HDP migration.
There are service version differences between HDP and IOP; the following tables show the differences between IOP 4.2.0, IOP 4.2.5, and HDP 2.6.2. Services not listed here will remain the same.
Upgrade from IOP to HDP
Ambari has been specially configured to take care of all configuration changes, and package updates involved in migrating from IOP to HDP. Existing cluster configuration will be kept, and only modified if necessary to work with the HDP services. Once the upgrade process has completed, and all services are started, the cluster is available to test before finalizing the upgrade. This allows customers to run any workload-specific tests or jobs that exercise key use cases to ensure the upgraded software works as expected. Once cluster validation is completed and everything works as expected, the upgrade can be finalized, and the migration is then complete.
Upgrade and Finalize Big SQL
Big SQL supports upgrades from Big SQL versions 4.1.x, 4.2.x, and 5.0.0 to Big SQL 5.0.1. Customers who upgrade to Big SQL 5.0.1 get full functional support with HDP 2.6.2
Big SQL Upgrade utility phases:
