User’s of Application Performance Analyzer (APA) for z/OS (cited by The Times as “one of this generation’s unsung heroes”) have many informational resources at their disposal.  Even a quick web search returns results from IBM’s Knowledge Center, IBM Developer Answers, Stack Overflow, Reddit (r/COBOL, r/Mainframe), and a host of other mainframe forums.  But searching for answers is time-consuming, especially when you’re just trying to develop general understanding of a new tool.

So, as APA Offering Manager, I’ve got your back. I’m always keeping an eye out for anything APA-related. And recently I’ve sensed some angst surrounding how system programmers and developers feel they should approach their performance tuning.  This is unsurprising, because performance tuning isn’t always straightforward, and many are “volun-told” to improve the business’ performance having very little previous exposure or business context. 

“What’s your background? Do you have any measurement tools, like [Application Performance Analyzer]?
Do you have the ability to find people in your shop that DO have these tools? 
I will probably cry a little bit if you come back and say something like…
‘No tools, [I] just got picked at random to fix something I don’t understand…
That’s so frustrating to be in that position.”1

Ok, so let’s assume in your case you actually have APA and you’ve just performed some observations.  Maybe you had in mind a specific batch job or a suspect application, or periodically monitoring system performance is standard practice.  Regardless, consistent optimization can save a lot of money in the future, often negating any CPU overhead incurred from conducting performance analyses in the present.  Sure, maybe it is time to finally replace that aging S/360, but then again maybe not (the z15 folks are going to hate me for this).

A little preventative maintenance can go a long way when it comes to maximizing efficiency, productivity, and competitiveness.  But in order to continuously optimize, you must first consider the business’ and IT’s objectives.  

“The real challenge with COBOL is not the technology, but the complexity of the underlying business
and the lack of documentation of the systems/programs. Thus the real value of a lot of COBOL experts
is in fact not the actual COBOL knowledge, but the understanding of the business.”2

This really underscores the importance of understanding how performance tuning can touch the entire business, and it also highlights the importance of moving towards a more holistic application delivery model; like DevOps (Need help transitioning? Check out our Z DevOps Acceleration Program).  

That last quote reminded me of a thread I recently stumbled upon over at a favorite mainframe forum.  There, a member sought to optimize a problem COBOL application.  After reviewing the performance report, an “INSPECT” verb was found to be responsible for about 80% of the application’s CPU utilization.  Automatically you’d think, that’s a problem.  So the logic was to replace the INSPECT verb with a PERFORM loop.  But, we (the original poster included) didn’t really know enough about the application to determine if it made sense to make the change. In COBOL we have several INSPECT verbs to choose from, and just as many for PERFORM.  So, what do you do?

“You may [see] a verb using 85% of CPU, but out of how much [service] time?
If it’s 85% of two hours, then you can probably make a difference.
If it’s 85% of 1 second, you may be past the point of diminishing returns.”3

In our scenario, without knowing much about the problem application or the business goals, we can’t really provide any meaningful advice to the member.  The best we can do is ask clarifying questions (as seen in the above response). It’s entirely conceivable that a single change could actually result in decreased performance.  What we really need here is some situational awareness and better understanding.

Fortunately, APA’s S01-Measurement Profile presents exactly what you need. A general overview of your observation, S01 should be one of the first reports you review. There you’ll find information on CP, zIIP, and zAAP utilization, along with other important details.  It’s analogous to taking the mainframe’s temperature.

Ok, so consider this: what if you find your zIIP or zAAP utilization was at 80%?  Well, if it’s a Java application, maybe explore further. Generally, if a Java application is burning through those zIIP and zAAP processors, great!  Offloading zIIP- and zAAP-eligible workloads to these processors will ultimately free up capacity for your general processor. But without the S01 or S09 reports you’d be making some serious assumptions.  

I mention the S09-Measurement Analysis report because of it’s “problem areas” section.  There we are essentially saying, “Hey….you should strongly consider these areas for tuning.”  But even there, we subtly mention the importance of understanding the business and IT goals: 

Obviously, the S09 report isn’t sacrosanct, performance tuning still requires sound logic and rational thinking as to whether changes should be made and to what degree. There is a lot to consider when you’re tuning, it can be tough when you’re first beginning. And for those brand new to APA, it’s important to consider the following closing remarks. 

First, APA relies on statistical sampling for reporting.  When faced with a problem application, you might be tempted to thin-slice your sample size or observation duration.  Don’t!  Because doing so will increase the volatility & sensitivity of your results.  APA is a great tool when used intelligently, so let it work its magic.  Without it you’d otherwise be forced to resort to some highly time-intensive performance tuning strategies (i.e. manually parsing through SMF records).

Second, APA isn’t a cure-all, and we don’t market it as such.  APA does provide you with situational awareness, an azimuth for where you should head on your inquiry, and the source-code program mapping feature is pretty nice too!  But APA doesn’t know what your business or IT goals are; you should though. And once you do know the business and IT objectives, you can finally perform intelligent observations, quickly diagnose your most intensive applications, and save valuable time and money in the process.

If you’re new to performance tuning, let’s get you set up with APA today!
Email Chris!

1Goodman, Ed. (2011, June 28). Performance Enhancement [Msg 4]. Message posted to
2Weismat (2009, Novemeber 11). Does it make sense to study COBOL? [Msg 3] Message posted to
3Goodman, Ed. (2013, January 23). Strobe [Msg 7]. Message posted to

1 comment on"Where to begin with Application Performance Analyzer for z/OS"

  1. Hi Chris,

    Thanks for the article. Indeed, APA is a very powerful and underappreciated tool.

    I have been using it in our shop for a few years, mostly to analyze long running and CPU intensive batch JOBs.

    Your comment “When faced with a problem application, you might be tempted to thin-slice your sample size or observation duration. Don’t! ” made me wonder that perhaps I am using it wrong. My attitude has been to fine tune the APA statistics as much as I can so that the results will give me a better picture of where the performance bottle necks are. So I tend to take the maximum number of samples that APA will permit, usually 60000 sample for 1 minute duration or 100000 samples for 2 minutes.

    Am I doing it wrong? Can I get statistically better results by sampling differently? If so, what is the “best practice”?

    I appreciate your thoughts on the matter.


    Danny Amir
    Bank HaPoalim

Join The Discussion

Your email address will not be published. Required fields are marked *