User’s of Application Performance Analyzer (APA) for z/OS (cited by The Times as “one of this generation’s unsung heroes”) have many informational resources at their disposal.  Even a quick web search returns results from IBM’s Knowledge Center, IBM Developer Answers, Stack Overflow, Reddit (r/COBOL, r/Mainframe), and a host of other mainframe forums.  But searching for answers is time-consuming, especially when you’re just trying to develop general understanding of a new tool.

So, as APA Offering Manager, I’ve got your back. I’m always keeping an eye out for anything APA-related. And recently I’ve sensed some angst surrounding how system programmers and developers feel they should approach their performance tuning.  This is unsurprising, because performance tuning isn’t always straightforward, and many are “volun-told” to improve the business’ performance having very little previous exposure or business context. 

“What’s your background? Do you have any measurement tools, like [Application Performance Analyzer]?
Do you have the ability to find people in your shop that DO have these tools? 
I will probably cry a little bit if you come back and say something like…
‘No tools, [I] just got picked at random to fix something I don’t understand…
That’s so frustrating to be in that position.”1

Ok, so let’s assume in your case you actually have APA and you’ve just performed some observations.  Maybe you had in mind a specific batch job or a suspect application, or periodically monitoring system performance is standard practice.  Regardless, consistent optimization can save a lot of money in the future, often negating any CPU overhead incurred from conducting performance analyses in the present.  Sure, maybe it is time to finally replace that aging S/360, but then again maybe not (the z15 folks are going to hate me for this).

A little preventative maintenance can go a long way when it comes to maximizing efficiency, productivity, and competitiveness.  But in order to continuously optimize, you must first consider the business’ and IT’s objectives.  

“The real challenge with COBOL is not the technology, but the complexity of the underlying business
and the lack of documentation of the systems/programs. Thus the real value of a lot of COBOL experts
is in fact not the actual COBOL knowledge, but the understanding of the business.”2

This really underscores the importance of understanding how performance tuning can touch the entire business, and it also highlights the importance of moving towards a more holistic application delivery model; like DevOps (Need help transitioning? Check out our Z DevOps Acceleration Program).  

That last quote reminded me of a thread I recently stumbled upon over at a favorite mainframe forum.  There, a member sought to optimize a problem COBOL application.  After reviewing the performance report, an “INSPECT” verb was found to be responsible for about 80% of the application’s CPU utilization.  Automatically you’d think, that’s a problem.  So the logic was to replace the INSPECT verb with a PERFORM loop.  But, we (the original poster included) didn’t really know enough about the application to determine if it made sense to make the change. In COBOL we have several INSPECT verbs to choose from, and just as many for PERFORM.  So, what do you do?

“You may [see] a verb using 85% of CPU, but out of how much [service] time?
If it’s 85% of two hours, then you can probably make a difference.
If it’s 85% of 1 second, you may be past the point of diminishing returns.”3

In our scenario, without knowing much about the problem application or the business goals, we can’t really provide any meaningful advice to the member.  The best we can do is ask clarifying questions (as seen in the above response). It’s entirely conceivable that a single change could actually result in decreased performance.  What we really need here is some situational awareness and better understanding.

Fortunately, APA’s S01-Measurement Profile presents exactly what you need. A general overview of your observation, S01 should be one of the first reports you review. There you’ll find information on CP, zIIP, and zAAP utilization, along with other important details.  It’s analogous to taking the mainframe’s temperature.

Ok, so consider this: what if you find your zIIP or zAAP utilization was at 80%?  Well, if it’s a Java application, maybe explore further. Generally, if a Java application is burning through those zIIP and zAAP processors, great!  Offloading zIIP- and zAAP-eligible workloads to these processors will ultimately free up capacity for your general processor. But without the S01 or S09 reports you’d be making some serious assumptions.  

I mention the S09-Measurement Analysis report because of it’s “problem areas” section.  There we are essentially saying, “Hey….you should strongly consider these areas for tuning.”  But even there, we subtly mention the importance of understanding the business and IT goals: 

Obviously, the S09 report isn’t sacrosanct, performance tuning still requires sound logic and rational thinking as to whether changes should be made and to what degree. There is a lot to consider when you’re tuning, it can be tough when you’re first beginning. And for those brand new to APA, it’s important to consider the following closing remarks. 

First, APA relies on statistical sampling for reporting.  When faced with a problem application, you might be tempted to thin-slice your sample size or observation duration.  Don’t!  Because doing so will increase the volatility & sensitivity of your results.  APA is a great tool when used intelligently, so let it work its magic.  Without it you’d otherwise be forced to resort to some highly time-intensive performance tuning strategies (i.e. manually parsing through SMF records).

Second, APA isn’t a cure-all, and we don’t market it as such.  APA does provide you with situational awareness, an azimuth for where you should head on your inquiry, and the source-code program mapping feature is pretty nice too!  But APA doesn’t know what your business or IT goals are; you should though. And once you do know the business and IT objectives, you can finally perform intelligent observations, quickly diagnose your most intensive applications, and save valuable time and money in the process.

If you’re new to performance tuning, let’s get you set up with APA today!
Email Chris!

1Goodman, Ed. (2011, June 28). Performance Enhancement [Msg 4]. Message posted to
2Weismat (2009, Novemeber 11). Does it make sense to study COBOL? [Msg 3] Message posted to
3Goodman, Ed. (2013, January 23). Strobe [Msg 7]. Message posted to

4 comments on"Where to begin with Application Performance Analyzer for z/OS"

  1. Hi Chris,

    Thanks for the article. Indeed, APA is a very powerful and underappreciated tool.

    I have been using it in our shop for a few years, mostly to analyze long running and CPU intensive batch JOBs.

    Your comment “When faced with a problem application, you might be tempted to thin-slice your sample size or observation duration. Don’t! ” made me wonder that perhaps I am using it wrong. My attitude has been to fine tune the APA statistics as much as I can so that the results will give me a better picture of where the performance bottle necks are. So I tend to take the maximum number of samples that APA will permit, usually 60000 sample for 1 minute duration or 100000 samples for 2 minutes.

    Am I doing it wrong? Can I get statistically better results by sampling differently? If so, what is the “best practice”?

    I appreciate your thoughts on the matter.


    Danny Amir
    Bank HaPoalim

    • Danny,

      I’m so pleased that you are using APA regularly. It truly is a neat little tool. My apologies for the delayed response. I really like that term “thin-slicing”. I first learned about it from Nassim Taleb (a polarizing, if not thought-provoking personality). But Daniel Kahneman – one of the greats in behavioral economics (actually from your neck of the woods), discusses it along with heuristics (a procedure that aids one in acquiring suitable, but imperfect answers to complex questions). This is not relevant (yet), but interesting nonetheless! For APA, we don’t have to rely on thin-slicing, or heuristics. Mostly because we are in a relatively closed system. Our applications can be viewed as processes, each with process flows that have clearly defined begin- and end- points.

      I mention process flows because by their very nature, they are oftentimes predictable. Many of the workloads on Z exhibit these same characteristics. However, I believe this might be more the case for batch jobs (less variability). Regardless, it’s important to understand what you are measuring and recognize any random variability (aka stochastic variability) that exists (like in CICS, or IMS).

      For improved results, statistically speaking, if you consider the application you are sampling then you’re already doing great. Consideration should be a primary focus. Secondly, consider our variables:

      Observation Duration
      Sample Size
      Sampling Rate

      These variables are related to process management, lean principles, process flows and mapping, etc. And in other fields can be expressed in the ubiquitous “Little’s Law” or I = R x T. Where:

      I=Inventory (Sample Size)
      R=Throughput/Rate (Sampling Rate)
      T=Time (Observation Duration)

      Let’s consider your two scenarios. In the first example we have a Time/Duration of 1 minute (or 60 seconds) and a Sample Size of 60,000. To find the Throughput/Rate we just divide Sample Size by Time. That gives us a Sampling Rate of .001 or a Sample is taken every .001 seconds. Take the inverse of that (1/.001) and you’ll find 1000 Samples taken every second. If we look at the second example, we find a Sampling Rate of .0012 or a Sample is taken every .0012 seconds; the inverse being 833.33 Samples taken every second. So, less precise in the second case. A better option would be to take 100,000 samples in 100 seconds (or 1 min 40 seconds).

      This might be where a heuristic benefits the user of APA. You could say, for a batch job we want a sampling rate of XXX, whereas for a CICS, IMS, or Java application we want this XXX sampling rate. In all instances, we only need to focus on two variables at a time, because they are all interdependent. For instance, change Time and Sample Rate and so too will “Inventory” (remember in our case that is the Sampling Size).

      Ok, so this may have been super verbose, but I think it’s important to address. In summation…consider sampling rates in conjunction with the applications that you are sampling. For instance, you can probably have lower rates in batch jobs (where there is less variability), but probably want higher sampling rates (along with longer durations) for applications that are highly unpredictable, or in cases where a lot is occurring under the covers (Java applications are a good example…lot’s of moving parts). A focus on sampling rates, and documentation of the methodology is always recommended. And if you’ve made it this far, then I should mention that we are operating in a hypothetically closed system. As workloads increase, or applications evolve, performance tuning should be performed at regular intervals. Increased workloads and application changes can result in constraints in total application throughput; and without performing regular performance observations we’d really be doing ourselves a disservice. I hope this helps. Thanks for the question, I love the interaction!

      Thanks again,


      • Hi Chris,

        Thanks for the detailed answer and your advice. I’ll try to implement it next time I use APA.



Join The Discussion

Your email address will not be published. Required fields are marked *