We’re giving away 1,500 DJI Tello drones. Enter to win ›
Tutorial
By Paul Clarke | Published May 16, 2018 - Updated August 17, 2018
LinuxSoftware DevelopmentSystems
I had the good fortune to be selected to present at the recent OpenPOWER Summit on the topic of tools for porting and tuning for Linux on Power. The time slots for the presentations were fairly short (maximum of 30 minutes), but there was a lot I wanted to cover. So, I did my best to highlight the tools I felt had the most value, and perhaps lacked general awareness. A video of the presentation can be found at https://www.youtube.com/watch?v=PJwnfDSHOLI. (There are lots of great presentations from the OpenPOWER Summit US 2018 appearing on the OpenPOWER Foundation’s YouTube channel at: https://www.youtube.com/channel/UCNVcHm09eXVbvUzZkQs0_Sg.)
Instead of listening to me drone on for 30+ minutes, what follows is the basic content of the presentation in textual form, not verbatim and slightly more detailed – a better reference.
As a general note, I classify the use-cases for the tools in the following three ways:
The IBM® Advance Toolchain is a software suite containing the latest releases of compilers, various libraries, and various tools related to application porting, tuning, and debugging. Recent releases of these components include support for the latest features and the latest optimizations for OpenPOWER and IBM POWER® processor-based platforms. The purpose of the Advance Toolchain is to make these more modern software components available on distributions which only provide significantly older releases. The distributions, justifiably, are reluctant to change major components of the operating system like compilers and system libraries as the risk to stability is not worth the opportunity for better performance. Some distributions have made strides in providing developers with much more recent components as a developer toolset that provides a later compiler and its prerequisites. The Advance Toolchain goes farther in providing not only the latest compilers but also the latest releases of many system libraries. In addition, those system libraries are built with the latest compilers. In this way, applications built with the Advance Toolchain benefit not only from new optimizations in the latest software, but also by having that software compiled with the new compiler.
Further, the Advance Toolchain is where new compatibility features may appear first. For example, there is an ongoing effort under the auspices of the GCC project to provide compatible implementations of the Intel vector intrinsics. Those will not appear in major Linux distributions for a year or more, but are already appearing in the Advance Toolchain.
The Advance Toolchain is supportable through IBM Support Line. Updates with bug fixes and security-related fixes are released often. It is available for free download, and is entirely open source at https://github.com/advancetoolchain/advance-toolchain.
One caveat is that when an application is built with the Advance Toolchain, it then has a dependency on the Advance Toolchain runtime. So, if that application is to be deployed elsewhere, the Advance Toolchain runtime package must be installed there as well. Because the runtime is free, this is not of significant concern, but something of which to be aware.
Best practices:
IBM XL compilers are IBM’s flagship proprietary compiler suite, used for reporting SPEC benchmark results on IBM AIX®, IBM z/OS®, and Linux on Power. The IBM XL compiler development team works closely with the IBM Research team to incorporate the very best optimization techniques for performance advantage. Recently, the IBM XL C/C++ compiler switched to use a source code parser (front end) based on Clang, allowing the IBM XL C/C++ compiler to significantly improve source code compatibility with GCC and LLVM. Also, most common command-line options for GCC are also supported by the XL C/C++ compilers.
There are two variants of IBM XL C/C++ for Linux:
The IBM XL compilers can very well make use of the Advance Toolchain libraries and tools. In a sense, you can get the best of both worlds by using IBM’s flagship proprietary compiler with the latest fully optimized libraries. (Note that this will impose a dependency on the Advance Toolchain runtime.)
Listing 1. XL compilation using IBM Advance Toolchain libraries
$ xlc a.c -F/opt/at11.0/scripts/xlC-16_1_0-AT11_0.dfp.cfg
Beyond standard compilation, the IBM XL compilers also offer several advanced features which can be used to performance advantage, one of which falls in the gray box category: automatic parallelization. If this is enabled using a command line option, code can be generated to automatically use the multithreading capabilities of the Power system for performance advantage.
The IBM XL compilers also include the following advanced features that can be used to further exploit the capabilities of the Power systems with source code changes:
Best practice:
IBM Feedback Directed Program Restructuring (IBM FDPR®) is an offering from IBM Research to optimize an existing binary without the need for source code changes, recompiling, or relinking. This is also known as post-link optimization. The tool uses a three-step process:
This tool is labeled gray box even though it doesn’t require recompiling or relinking because the binary must be linked such that relocation information is preserved. Because this is not the default behavior of the linker, it is usually the case that the binary needs to be relinked with -emit-relocs.
-emit-relocs
There are open source wrapper scripts in the fdpr_wrap package. Both FDPR and fdpr_wrap packages can be found at https://developer.ibm.com/linuxonpower/sdk-packages.
fdpr_wrap
An example of use:
Listing 2. FDPR post-link optimization
$ gcc -o load -O3 -mcpu=power8 load.c -Wl,--emit-relocs $ ./load Total run time for 10000 iterations was: 3.579205 seconds $ /opt/ibm/fdprpro/bin/fdpr_instr_prof_opt load FDPR profiling: /home/pc/load-2.1pc/load.instr ... Total run time for 10000 iterations was: 26.036891 seconds $ ./load.fdpr Total run time for 10000 iterations was: 2.130412 seconds
FDPR can also generate a journal file, which includes suggestions for source code changes or compiler flags that could be implemented for performance improvement. Implementing these changes makes the improvements permanent, in that FDPR would not be required to realize them in subsequent builds of the program. Generating the journal is similar to the optimization example, above. Note that in this case, the compilation step adds the -g flag to generate debugging information so that FDPR can report line numbers in the journal:
Listing 3. FDPR generating journal
$ gcc -o load -Wl,--emit-relocs -O3 -mcpu=power8 -g load.c $ /opt/ibm/fdprpro/bin/fdpr_instr_prof_jour load $ ls -1tr | tail -1 load_jour.xml
An example extract of journal contents follows:
Listing 4. FDPR journal
<operation name="Unroll loop"> <problem>High branch penalty in a small loop</problem> <solution>Compiler: the loop is unrolled N times so it fits within a cache line</solution> <site> <ip>100006a4</ip> <dir>/home/pc/load-2.1pc</dir> <file>load.c</file> <fn>main</fn> <line>15</line> <xcount>132900736</xcount> </site>
In this example, FDPR has found a place where unrolling a loop would be advantageous, which suggests adding a compiler option like -funroll-loops.
-funroll-loops
and
The IBM SDK for Linux on Power (the SDK) is a full-featured integrated development environment (IDE) based on Eclipse with IBM-developed plug-ins aimed at porting and tuning C and C++ applications for Linux on Power.
The SDK runs exclusively on Linux, and can run on x86 or Power systems. It can be made to run on Microsoft® Windows® or MacOS, or anywhere on which a sufficient Linux environment can be established with virtualization or emulation. It can run in several modes:
Information about some of the more commonly used plug-ins is given below. I also created a tutorial for how to make the best use of the SDK at: https://developer.ibm.com/linuxonpower/tutorials/sdk_linux_on_power.
In addition, there are several videos showing how to use many of the plug-ins at: https://www.youtube.com/playlist?list=PLXUHQs-GmUIRbCsNtjcTAIFpZdvmhEPjq.
The SDK is free to download and use at https://developer.ibm.com/linuxonpower/sdk.
The SDK Migration Advisor is an included plug-in for the SDK, which scans the imported source code of a project looking for potential portability issues. The list of issues that Migration Advisor seeks includes:
#ifdef x86
long double, Float128
char
sync
The Migration Advisor is very simple to run: after the source is imported into a new project, right-click the project and click Run Migration Advisor from the project’s context menu.
When the Migration Advisor completes, it produces a report with the issues found.
Figure 1. Migration Advisor results
Double-click an issue to go directly to the source code line on which the issue was detected. Further, when the cursor hovers over that line, a pop-up box appears describing the issue in more detail.
Figure 2. Migration Advisor: Pop-up window
Finally, if you click Built-in quick fix, a portable fix for the issue is implemented directly in the code!
While this is extremely convenient, it can be tedious for code in which a large number of issues are detected. There is a feature to automatically implement all fixes for which the Migration Advisor has high confidence.
Figure 3. Migration Advisor: Automate
The Apply basic fixes automatically option can save a significant amount of time, especially for large projects with many detected issues.
Figure 4. Migration Advisor: Automated results
The SDK Source Code Advisor uses IBM FDPR to perform run-time performance analysis and report potential issues that might be ameliorated with source code or makefile changes.
Figure 5. Source Code Advisor results
Double-clicking the source line in the report displays the source code. Hovering over the source line in the code displays a pop-up box with a similar description and a hyperlink to apply a fix to the code.
Figure 6. Migration Advisor: Pop-up window
Cycles per instruction (CPI) is a measurement of the average number of processor cycles used to execute each instruction of a program. This is the efficiency with which a program is running on a processor or a set of processors. This metric is certainly interesting, but in isolation does not convey any information about how to improve the efficiency of the program. Most modern processors have the ability to monitor a large and diverse set of hardware events which comprise the CPI value. Further, some processors support more specific events which are themselves components of more general events. In this way, a hierarchical breakdown of the CPI value can be generated. This provides valuable insight into what type of inefficiencies a program is encountering. The SDK CPI Breakdown tool will run a given representative performance scenario for the program and graphically display the CPI and associated hierarchical breakdown.
Figure 7. CPI Breakdown results
There is a lot of information displayed in a small space, and the image may be difficult to read (it is far more legible in the application). At the far left is a legend explaining which components are measured events (light gray), and which are calculated metrics (darker gray). Also, the raw run CPI and stall CPI measurements are shown. The r__un CPI measurements are shown for when instructions are proceeding and the stall CPI measurements are shown for when instructions have been held up for some reason. Moving to the right, the first column is the overall cycles (PM_RUN_CYC) for which the program was actively running (scheduled, whether instructions are progressing or stalled). The next column breaks the overall number of cycles into subsets for instructions that are progressing and stalling. These categories may vary by processor architecture. Continuing to the right, each of the preceding categories are further broken down into more specific events and metrics, which also vary by processor architecture.
PM_RUN_CYC
Hovering over any block displays a more detailed description of the event or metric. The top three events or metrics in each level are displayed in red (first), orange (second), and pink (third). The size of each block is scaled to fit in the hierarchy, not by the magnitude of the event or metric. For a better view of relative magnitudes, the data can be displayed as a radar chart by selecting that tab.
Figure 8. CPI Radar Chart view
On the CPI Breakdown tab, double-click any of the events and notice that a fresh run of the representative performance scenario is launched for profiling based on the selected event, allowing you to drill down the most significant events. When this profiled scenario has completed, the results are displayed by source and line number.
Figure 9. CPI Drill-down results
GUI-based tools can be incredibly powerful, but they are not always a good fit, and they can be difficult to get right. Usability, automation, and personal or organizational preference can preclude the use of GUI tools. For those reasons, there are ongoing efforts to create command-line versions of the most valuable tools in the SDK. These are described below.
Similar to the SDK Migration Advisor (described above), the command-line Migration Advisor scans the source code looking for likely portability issues. As of this writing, the list of checkers is slightly fewer for the command-line version:
Usage is very simple:
Listing 5. ma results
$ ma run src/. ================ Migration Report ================ Problem type: Non Portable Pthread Problem description: Reports occurrences of non-portable Pthreads API File: ma/many.c Line: 3 Problem: pthreadid_np_t tid Line: 4 Problem: pthread_getthreadid_np() File: ma/pthread.c Line: 3 Problem: pthread_id_np_t tid Line: 4 Problem: pthread_getthreadid_np() Problem type: Performance degradation Problem description: This preprocessor can contain code without Power optimization File: ma/performance.c Line: 3 Problem: #ifdef _x86 Problem type: Inline assembly Problem description: Possible arch specific assembly File: ma/asm.c Line: 2 Problem: asm("mov %ax, 0") Line: 3 Problem: __asm__("mov %ax, 0") Problem type: Long double usage Problem description: Potential migration issue due size of long double variables in Power architecture. File: ma/t0.c Line: 3 Problem: long double ld File: ma/double.c Line: 3 Problem: long double ld Problem type: Hardware Transactional Memory (HTM) Problem description: x86 specific HTM calls are not supported in Power Systems File: ma/htm.c Line: 1 Problem: include rtmintrin.h Solution: replace rtmintrin.h for htmintrin.h Line: 4 Problem: _xbegin() Solution: replace xbegin for __builtin_tbegin Problem type: Decimal Floating Point (DFP) API Problem description: x86 API not supported in Power File: ma/dfp.c Line: 1 Problem: include bid`functions.h Line: 6 Problem:_bid64`pow(dfp0,dfp0)
The command-line Migration Advisor is not yet able to implement any fixes in the source code.
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/migration-advisor.
Similar to the SDK Source Code Advisor (described above), the command-line Source Code Advisor uses FDPR (described above) to analyze a representative performance scenario and report performance issues which could not be identified during compilation. The results are displayed in a very readable format for inspection and further manual action.
Listing 6. sca results
$ sca ./command [Problem: UNROLL LOOP] [Description: High branch penalty in a small loop.] [Solution: Specify the GNU extension "attribute ((optimize ("unroll-loops")))" on the function containing the loop. Example: void attribute((optimize("unroll-loops"))) foo(void); Note: Unrolling loops can sometimes negatively impact performance, so validation of its impact is recommended. [Reference: /home/pc/load-2.1pc/command.c:15 | Function: main | Instruction Pointer: 10000694]
The command-line Source Code Advisor is not yet able to implement any fixes in the source code.
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/source-code-advisor.
Similar to the SDK CPI Breakdown tool (described above), the command-line CPI Breakdown tool will profile a representative performance scenario and report a hierarchical set of information about where the program is spending its time. Using the command-line CPI Breakdown tool is a two-step process:
record
display
Use of the command-line CPI Breakdown tool is simple. The first step is to record the hardware event counts.
Listing 7. cpi record
$ cpi record ./load [...] $ ls -tr | tail -1 load_20180416_215506.cpi
Note that the scenario (“./load” in this example) will be run several times in succession in order to collect all relevant hardware performance events, as only a handful are collected during each run.
./load
The second step is to display the CPI breakdown.
Listing 8. cpi display results
$ cpi display -f ./load_20180416_215506.cpi RUN_CPI: 4.957 (100.00 %) STALL_CPI: 2.270 (45.80 %) BRU_CRU_STALL_CPI: 0.277 (5.60 %) BRU_STALL_CPI: 0.277 (5.59 %) CRU_STALL_CPI: 0.000 (0.00 %) FXU_STALL_CPI: 0.707 (14.27 %) FXU_MULTI_CYC_CPI: 0.000 (0.00 %) FXU_STALL_OTHER_CPI: 0.707 (14.27 %) VSU_STALL_CPI: 0 (0.0 %) VSU_STALL_VECTOR_CPI: 0 (0.0 %) VSU_STALL_VECTOR_LONG_CPI: 0 (0.0 %) VSU_STALL_VECTOR_OTHER_CPI: 0 (0.0 %) VSU_STALL_SCALAR_CPI: 0 (0.0 %) VSU_STALL_SCALAR_LONG_CPI: 0 (0.0 %) VSU_STALL_SCALAR_OTHER_CPI: 0 (0.0 %) VSU_STALL_OTHER_CPI: 0 (0.0 %) LSU_STALL_CPI: 1.311 (26.46 %) LSU_STALL_DCACHE_MISS_CPI: 1.118 (22.56 %) LSU_STALL_DCACHE_MISS_L2L3_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_L2L3_CONFLICT_CPI: 0.876 (17.67 %) LSU_STALL_DCACHE_MISS_L2L3_NOCONFLICT_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_L3MISS_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_LMEM_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_L21L31_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_REMOTE_CPI: 0 (0.0 %) LSU_STALL_DCACHE_MISS_DISTANT_CPI: 0 (0.0 %) LSU_STALL_REJECT_CPI: 0.029 (0.58 %) LSU_STALL_LHS_CPI: 0.000 (0.00 %) LSU_STALL_ERAT_MISS_CPI: 0 (0.0 %) LSU_STALL_LMQ_FULL_CPI: 0 (0.0 %) LSU_STALL_REJECT_OTHER_CPI: 0.029 (0.58 %) LSU_STALL_STORE_CPI: 0.002 (0.04 %) LSU_STALL_LD_FIN_CPI: 0.470 (9.47 %) LSU_STALL_ST_FWD_CPI: 0 (0.0 %) LSU_STALL_OTHER_CPI: 0 (0.0 %) NTCG_FLUSH_CPI: 0 (0.0 %) NO_NTF_STALL_CPI: 0.012 (0.24 %) OTHER_STALL_CPI: 0 (0.0 %) NTCG_ALL_FIN_CPI: 0 (0.0 %) THREAD_BLOCK_STALL_CPI: 0.440 (8.87 %) LWSYNC_STALL_CPI: 0.001 (0.01 %) HWSYNC_STALL_CPI: 0.000 (0.00 %) MEM_ECC_DELAY_STALL_CPI: 0 (0.0 %) FLUSH_STALL_CPI: 0.442 (8.92 %) COQ_FULL_STALL_CPI: 0.000 (0.00 %) OTHER_BLOCK_STALL_CPI: 0 (0.0 %) GCT_EMPTY_CPI: 1.245 (25.13 %) GCT_EMPTY_IC_MISS_CPI: 0 (0.0 %) GCT_EMPTY_IC_MISS_L3MISS_CPI: 0 (0.0 %) GCT_EMPTY_IC_MISS_L2L3_CPI: 0 (0.0 %) GCT_EMPTY_BR_MPRED_CPI: 0 (0.0 %) GCT_EMPTY_BR_MPRED_IC_MISS_CPI: 0 (0.0 %) GCT_EMPTY_DISP_HELD_CPI: 0.000 (0.00 %) GCT_EMPTY_DISP_HELD_MAP_CPI: 0.000 (0.00 %) GCT_EMPTY_DISP_HELD_SRQ_CPI: 0 (0.0 %) GCT_EMPTY_DISP_HELD_ISSQ_CPI: 0 (0.0 %) GCT_EMPTY_DISP_HELD_OTHER_CPI: 0 (0.0 %) GCT_EMPTY_OTHER_CPI: 1.245 (25.12 %) COMPLETION_CPI: 0.384 (7.75 %) OTHER_CPI: 0.618 (12.46 %)
There is a lot of information displayed. For convenience, there are command flags which can limit the output.
Listing 9. cpi display with filters
$ cpi display -f ./load_20180416_215506.cpi --top-events 5 ================ Events Hot Spots ================ PM_RUN_CYC : 30007450676 PM_CMPLU_STALL : 13742052138 PM_CMPLU_STALL_LSU : 7939376272 PM_GCT_NOSLOT_CYC : 7540013513 PM_CMPLU_STALL_DCACHE_MISS : 6770379676 $ cpi display -f ./load_20180416_215506.cpi --top-metrics 5 ================ Metrics Hot Spots ================ RUN_CPI : 4.957 STALL_CPI : 2.270 LSU_STALL_CPI : 1.311 GCT_EMPTY_CPI : 1.245 GCT_EMPTY_OTHER_CPI : 1.245
Because the ultimate goal is to narrow down where in the code adverse events are happening, there is a further convenience function that can drill down on the most frequently occurring events. New profiling runs are launched in which those specific events are recorded and the profiling information, including source, line, and potentially instruction are included in the command output.
Listing 10. cpi drilldown results
$ cpi drilldown --auto 5 --threshold 0.25 ./load 2000 Recording CPI Events: 20/20 iterations (elapsed time: 29 seconds) Running drilldown with event: PM_RUN_CYC =============================== Drilldown for event: PM_RUN_CYC =============================== 99.51% in /home/pc/load-2.1pc/load 99.51% in main [/home/pc/load-2.1pc/load.c] =============================== Running drilldown with event: PM_CMPLU_STALL =================================== Drilldown for event: PM_CMPLU_STALL =================================== 99.07% in /home/pc/load-2.1pc/load 99.07% in main [/home/pc/load-2.1pc/load.c] 0.9% in /proc/kallsyms 0.63% in rfi_flush_fallback [??] =================================== Running drilldown with event: PM_CMPLU_STALL_LSU ======================================= Drilldown for event: PM_CMPLU_STALL_LSU ======================================= 98.3% in /home/pc/load-2.1pc/load 98.3% in main [/home/pc/load-2.1pc/load.c] 1.67% in /proc/kallsyms 0.92% in rfi_flush_fallback [??] ====================================== Running drilldown with event: PM_GCT_NOSLOT_CYC ====================================== Drilldown for event: PM_GCT_NOSLOT_CYC ====================================== 99.03% in /home/pc/load-2.1pc/load 99.03% in main [/home/pc/load-2.1pc/load.c] 0.94% in /proc/kallsyms 0.47% in rfi_flush_fallback [??] ====================================== Running drilldown with event: PM_CMPLU_STALL_DCACHE_MISS =============================================== Drilldown for event: PM_CMPLU_STALL_DCACHE_MISS =============================================== 99.55% in /home/pc/load-2.1pc/load 99.55% in main [/home/pc/load-2.1pc/load.c] ===============================================
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/cpi-breakdown
There is a tool called curt on AIX that displays statistics related to system utilization. A new tool for Linux, also called curt is inspired by the AIX tool (but is otherwise unrelated).
Statistics reported by the Linux curt tool include:
Use of the tool is a two-step process:
Use perf record to generate a recording of relevant events:
perf record
$ perf record -e '{raw_syscalls:*,sched:sched_switch,sched:sched_migrate_task, sched:sched_process_exec,sched:sched_process_fork,sched:sched_process_exit, sched:sched_stat_runtime,sched:sched_stat_wait,sched:sched_stat_sleep, sched:sched_stat_blocked,sched:sched_stat_iowait,powerpc:hcall_entry, powerpc:hcall_exit}' -a command --args
Use the curt script to process the data recorded by the perf command: $ ./curt.py perf.data
$ ./curt.py perf.data
With the most recent version of curt, both recording and reporting can be done in a single, simple step: $ ./curt.py --record allcommand
$ ./curt.py --record all
command
Sample output (heavily edited for brevity and clarity):
Listing 11. curt results
PID : 5020: [task] command cpu user sys irq hv busy idle [5092] imjournal 6 0.288924 0.154960 0.000000 0.000000 0.000000 5001.594250 [5092] imjournal ALL 0.288924 0.154960 0.000000 0.000000 0.000000 5001.594250 [task] command cpu runtime sleep wait blocked iowait unaccounted [5092] imjournal 6 0.461900 0.000000 0.000000 0.000000 0.000000 997.568960 [5092] imjournal ALL 0.461900 0.000000 0.000000 0.000000 0.000000 997.568960 [task] command cpu util% moves [5092] imjournal 6 0.0% [5092] imjournal ALL 0.0% 0 ( ID)name count elapsed pending average minimum maximum ( 3)read 6 0.041416 0.000000 0.006903 0.002252 0.022116 (167)poll 4 4004.103382 997.585996 1001.025845 1001.018766 1001.029046 (221)futex 1 0.011118 0.000000 0.011118 0.011118 0.011118 (106)stat 1 0.007298 0.000000 0.007298 0.007298 0.007298
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/curt.
, , and
The Power Functional Simulator is a full-system simulator for POWER. This very powerful tool provides a complete POWER environment when a POWER processor-based system is otherwise unavailable or impractical. Given an image of an installed file system, it will boot through firmware and operating system to a login prompt for a POWER processor-based development environment.
The Power Functional Simulator can be found at: https://developer.ibm.com/linuxonpower/sdk-packages.
Convenient wrapper scripts, which greatly simplify getting an environment established, can be found at: https://github.com/open-power-sdk/power-simulator.
Listing 12. Power Functional Simulator boot
[x86-laptop]$ mambo -s power9 #(edited for brevity...) You are starting the IBM POWER9 Functional Simulator When the boot process is complete, use the following credentials to access it via ssh: ssh root@172.19.98.109 password: mambo Licensed Materials - Property of IBM. (C) Copyright IBM Corporation 2001, 2017 All Rights Reserved. Using initial run script /opt/ibm/systemsim-p9/run/p9/linux/boot-linux-le-skiboot.tcl Starting mambo with command: /opt/ibm/systemsim-p9/bin/systemsim-p9 -W -f /opt/ibm/systemsim-p9/run/p9/linux/boot-linux-le-skiboot.tcl Found skiboot skiboot.lid in current directory Found kernel vmlinux in current directory Found disk image disk.img in current directory Booting with skiboot ./skiboot.lid..... Booting with kernel ./vmlinux..... root disk ./disk.img INFO: 0: (0): !!!!!! Simulator now in TURBO mode !!!!!! OPAL v5.7-107-g8fb78ae starting... [...] Linux version 4.13.0-rc4+ (pc@moose1.pok.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)) #2 SMP Fri Aug 18 17:01:57 EDT 2017 [...] Debian GNU/Linux 9 mambo ppc64le 172.19.98.109 mambo login:
The environment provided by the Power Functional Simulator is single core, single thread. There are simulators available for POWER8 and POWER9. A simulator is a great way to begin getting experience with Linux on Power, beginning a porting effort, and even for establishing a robust cross-compilation and runtime environment.
The Performance Simulator is a cycle-accurate POWER instruction stream reporting tool. It transforms a POWER instruction trace into a report where the various stages of each cycle of every instruction’s lifetime is reported. The resulting reports can be viewed with one of the viewers which is included with the Performance Simulator package.
Instruction traces can be captured using the itrace function of the Valgrind tool suite. Valgrind itrace is not included with the Valgrind that comes with Linux distributions. However, itrace is available with the Valgrind that comes with the IBM Advance Toolchain (see above).
itrace
Using the Performance Simulator is a three-step process:
Record instruction trace (.vgi file):
.vgi
$ valgrind -tool=itrace -binary-outfile=tracefile.vgi --num-K-insns-to-collect=100 -demangle=no command
Create a .qt format file from the .vgi file:
.qt
$ vgi2qt -f tracefile.vgi -o tracefile.qt
Run the Performance Simulator timer (.pipe file):
.pipe
$ /opt/ibm/sim_ppc/sim_p8/bin/run_timer tracefile.qt 100000 10000 1 tracefile -scroll_pipe 1 -scroll_begin 1 -scroll_end 100000
You can view the resulting instruction timing report with one of the two included viewers (scrollpv and jviewer).
scrollpv
jviewer
The scrollpv viewer displays the instruction cycle report.
Figure 10. scrollpv display
Each phase of each instruction’s lifetime is shown visually in the main pane, lower left. The instruction disassembly is two columns to the right. Hovering over any of the character mnemonics in the main pane will display an explanation in the text area near the top of the window. In the example, the cursor is over an ‘s’ near the center of the pane. The corresponding explanation is “cannot issue sources not ready”. This instruction is currently waiting for its operands to be made available from the processing of a previously executed instruction.
Like scrollpv, jviewer also displays the instruction cycle report.
Figure 11. jviewer display
jviewer is very similar to scrollpv. Which viewer to use is personal preference.
The pipestat tool operates on the output of the Performance Simulator cycle-accurate timer, and performs detailed analysis, reporting the following information:
pipestat
The pipestat tool produces a lot of useful output. Extracted below are a few snippets.
Listing 13. pipestat results
HOT execution count blocks: 0x000004025c98-0x000004025cac N:678 6 inst trace inst 297 HOT misaligned short loops: 0x00000400ea58-0x00000400ea74 N:170 8 inst short misalign32 Loop size summary data: Instrs loops total iter min iter max iter avg iter total inst % of trace 6 1 402 402 402 402.00 2412 2.49 HOT Loop constructs (5 total): Header blk IA arch inst static dynamic iter inst/iter nodes taken !taken BL BLR 0000000400eb40 112 6 2412 450 5.4 1 0.00 0.00 0 0 BkEdge: bc_tk bc_nt fallth Edge: bc_tk bc_nt br fallth 402 0 0 0 0 0 0 HOT long latency instruction blocks: 0x00000400ea7c-0x00000400ea90 N:48 6 inst badness 240 HOT redundant loads: intra+stack: 261 (0.27%) inter+stk: 552 (0.57%) intra: 0 (0.00%) inter: 0 (0.00%) 0x00000400e0d4-0x00000400e0f4 N:200 redundant loads 487 HOT bad branch hints: 0x000004025dc4 hint likely not taken but was taken 36.47% (450/1234) branch mispredict summary 16444 branches 1271 mispredict 1240 penalty samples avg penalty 24.7cy total penalty 30618cy HOT branch mispredict count: 0x000004025dc4 mispredict 450 (36.5% of 1234) pen 17.0cy avg over 450 LBE 0.0081/0.1153 HOT branch mispredict frequency: 0x00000400e0a4 mispredicted 100.0% of 1 pen 17.0cy avg over 1 0x00000400e824 mispredicted 47.3% of 110 pen 19.8cy avg over 52 HOT branches with high linear branch entropy and executed frequently 0x00000400e824 N:110 LBE 0.6909/0.0000 0x00000400e2b4 N:240 LBE 0.4333/0.0455 HOT load hit store separated by less than 100 instructions: of all exec: Pathlength Std. red other AGEN ST IA LD IA Count % count min count Avg max Dev. LDs store regs Store Values 400df64 400e0dc 200 100.0 200 13 87 34.2 68 358 87 2/2/2 st: G1 ld: G1 Total LHS events: 3282 15.1% of loads 20.0% of stores
The information displayed may be a bit cryptic, but there is comprehensive documentation that comes with pipestat. The pipestat tool can be found at https://developer.ibm.com/linuxonpower/sdk-packages.
No Instruction Set Architecture (ISA) can have every possible useful vector instruction and an associated compiler built-in. The pveclib project provides some well-crafted implementations of useful vector functions which are not part of the POWER ISA. A sampling of the functions includes:
udiv_qrnnd
fxu_bcdadd, fxu_bcd_sub
vec_BCD2DFP, vec_DFP2BCD
vec_bcdadd, vec_bcdsub, vec_bcdmul, vec_bcddiv
vec_shift_leftdo
vec_isalpha, vec_isalnum, vec_isdigit
vec_toupper, vec_tolower
vec_absdub
vec_revq, vec_revd, vec_revw, vec_revh
vec_clzq, vec_popcntq
vec_sldq, vec_srqi, vec_srq, vec_slqi, vec_slq
vec_pasted
vec_mulouw, vec_muleuw, vec_mulosw, vec_mulesw
vec_adduqm, ...
The pveclib project is open source and can be found at https://github.com/open-power-sdk/pveclib.
The Shared Persistent Heap Data Environment (SPHDE) provides some advanced, high-performance, cross-platform implementations of functionality useful in a multiprocess or multithreaded application environment:
SPHDE is open source and can be found at https://github.com/sphde/sphde.
The Linux Performance Customer Profiling Utility (LPCPU) is a free download which can be used to collect system information and system performance-related data. Use of LPCPU is a three-step process:
System-wide data is collected for a specified period of time. The information collected is gathered into a compressed .tar file.
# tar -xjf lpcpu.tar.bz2 # cd lpcpu # ./lpcpu.sh duration=150 extra_profilers="perf tcpdump" [...] Packaging data...data collected is in /tmp/lpcpu_data.hostname.default.2018-02-26_1000.tar.bz2
The .tar file can then be offloaded to another system for post-processing and analysis.
$ tar -xf ./lpcpu_data.hostname.default.2018-02-26_1000.tar.bz2 $ cd ./lpcpu_data.hostname.default.2018-02-26_1000 $ ./postprocess.sh
Point a browser at the resulting summary.html file.
Data collected includes output from the following common tools:
iostat
mpstat
vmstat
perf
meminfo
top
sar
/proc/interrupts
tcpdump
netstat
...
The post-processing steps create numerous charts and interactive graphs.
Figure 12. LPCPU CPU utilization graph 1
Figure 13. LPCPU CPU utilization graph 2
Figure 14. LPCPU Interrupts graph
Figure 15. LPCPU Dirty Memory graph
The LPCPU tool is open source and can be freely downloaded at http://ibm.co/download-lpcpu.
Formal support is available for the Advance Toolchain and the XL compilers.
The Linux on Power Developer Portal is a hub for all sorts of information about hardware and software components in the Linux on Power ecosystem. Refer to: https://developer.ibm.com/linuxonpower. (Note: This portal replaces the IBM developerWorks Linux on Power Community.) Within the Portal, informal support can be requested by asking questions on the dW Answers forum at: https://developer.ibm.com/answers/smartspace/linuxonpower/index.html).
Community support options are also available at the following well-known websites:
Given all the nifty tools mentioned, where should one start when embarking on an effort to port code to Linux on Power?
For interpreted code, including Java, Python, Perl, and shell scripts, just move the code to Linux on Power, and it should run without modification.
For compiled code, with little to no source code analysis:
For compiled code, with source code analysis, perform the same steps as above, but between steps (3) and (4):
3.1. Use common performance analysis tools such as perf to look for hot spots in the code for careful analysis, and see if any can be explained by architectural differences between prior platforms and POWER. See “Porting to Linux on Power: 5 tips that could turn a good port into a great port” at https://www.ibm.com/developerworks/library/l-port-x86-lop/. Apply mitigation if possible.
3.2. Consider using the SDK CPI Breakdown and Drill-down tool or the command-line version for deeper architecture-specific analysis. Drill-down on higher-frequency hazards. Apply mitigation if possible.
For very deep analysis, consider using the pipestat command for fine-grained instruction-level analysis.
As always, consider alternative approaches to common, important functionality that might be provided by SPHDE or pveclib.
For system-wide performance analysis, use LPCPU and/or curt.
Hopefully, the collection of tools explained in this article provide what’s needed to get the best results as quickly as possible. If there is a gap in a tool’s functionality or documentation, or a new tool would help, or reality isn’t meeting expectations, ask in one of the support channels! We’re here to help!
IBM Power SystemsLinux+
Back to top
