Rebuilding Python for improved performance on IBM Power Systems

Introduction

The Python interpreter is an important part of many applications, and as such, it’s runtime performance is gaining focus. The Python interpreter that is built and included in Red Hat Enterprise Linux (RHEL) systems (RHEL 8.1 and prior) for IBM® Power® is not fully optimized, resulting in lower performance. Rebuilding the Python interpreter with the latest gcc and correct options result in significant performance improvements. This tutorial provides instructions on how to rebuild the Python interpreter for improved performance on IBM Power Systems™.

Tests with Python 3.6.6 and IBM Advance Toolchain for Linux on Power 12 show average performance gains with the pyperformance test suite of 30%. Tests with Python 2.7.5 and Advance Toolchain 12 shows average performance gains with the pyperformance test suite of 40% over the default installation.

Estimated time

Users should allocate around 1 to 2 hours to rebuild the Python interpreter.

Prerequisites

Python needs several developer packages installed. Here is a partial list (individual systems might require more to be installed).

Run the following command to install the required packages:

# yum install openssl-devel readline-devel ncurses-devel bzip2-devel gdbm-devel libsqlite3x-devel zlib-devel lzma-sdk-devel tk-devel libffi-devel sqlite-devel xz-devel

Steps

The version of Python included with RHEL is built without some optimization flags that are provided by the Python build environment. Building the Python interpreter with those flags and with the latest gcc version available in IBM Advance Toolchain for Linux on Power provide improved performance for workloads on the IBM Power platform.

RHEL 8.2 includes a Python interpreter that was built with optimization flags. But previous releases, such as RHEL 8 and RHEL 7 did not include.Note that the 3.4.x versions of Python do not have the enable-optimizationbuild option. Building with the newer gcc will result in a gain, but not as much as will be seen on later versions of Python.

Step 1. Install Advance Toolchain for Linux on Power

First, make sure that you have the latest version of Advance Toolchain for Linux on Power installed on your system. See https://developer.ibm.com/linuxonpower/advance-toolchain/advtool-installation/ for instructions. At the time of writing this tutorial, Advance Toolchain 12 is the latest version for RHEL 7 releases and Advance Toolchain 13 is the latest for RHEL 8 releases.

Step 2. Download the Python source

In order to rebuild the Python interpreter, download the appropriate source level from the main python.org website. The source is listed on the python.org page at https://www.python.org/downloads/. Download the required version, either the latest of a particular level or the level to match the installed version. To determine the current level that you are using, invoke Python with the -V flag:

# python -V
Python 2.7.5

Or

# python3 -V
Python 3.6.6

Download and unpack the required Python level:

# wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tgz
# tar -xf Python-3.6.6.tgz

Step 3. Rebuild the Python source

Perform the following steps to configure and build the Python source, and install the newly built Python to a new directory. In this tutorial, we use Python version 3.6.6 as an example. You may need to update the steps based on your situation.

The PYTHON_LOCAL environment variable is where you would like the built files to be installed.

# PYTHON_LOCAL=/opt/python366-at12
# cd Python-3.6.6
# ./configure CC="/opt/at12.0/bin/gcc" CXX="/opt/at12.0/bin/g++" --with-lto \
--enable-optimizations --enable-ipv6 --enable-loadable-sqlite-extensions \
--prefix=${PYTHON_LOCAL} --exec-prefix=${PYTHON_LOCAL}
# make clean profile-removal
# make -j 40
# make install
# ln -s python3 ${PYTHON_LOCAL}/bin/python

If there are additional prerequisites needed based on error messages in the configure or make step, install and repeat the above steps.

You can find more information about the configure flags using the help option:

# ./configure --help

Step 4. Using the new build

After the build is complete, you can export the directories for your environment and run the new version, confirming that you are running the Advance Toolchain-built code. This relies on the $PYTHON_LOCAL environment variable that was used in the build above.

# export LD_LIBRARY_PATH=${PYTHON_LOCAL}/lib:${LD_LIBRARY_PATH}
# export PATH=${PYTHON_LOCAL}/bin:${PATH}
# python3 -VVV
Python 3.6.6 (default, Apr 6 2020, 09:01:50) 
[GCC 8.3.1 20191125 (Advance-Toolchain-at12.0) [ibm/gcc-8-branch revision 27869

Summary

Rebuilding Python on your Power hardware with the latest gcc from the Advance Toolchain for Linux on Power and using the optimization flags in the Python build environment results in a Python interpreter that has higher performance than the installed version.