Streams ManagementServer - huge CPU usage
hi, streams gurus, we have a strange metrics about streams ManagementServer on our domain:it's take about 200-700% of CPU usage. We have one domain, one host, 20 CPU, 120Gb RAM. The report screenshots attached here. thanks for the advices or shared experiences!
So glad you are back. I started another post because your post from April 4. Seemed to be lost somehow. I will point that issue to this one and we can discuss it here.
That process looks like jmx. The easiest way to correlate process id's is usually is to use streamtool getdomainstate -d <domain> --long for domain management services streamtool getresourcestate -d <domain> -i <instance> --long for instance management services and streamtool lspes -d <domain> -i <instance> for pes.
We have this document that talks about performance best practices and much of that advice pertains to load as well. We normally recommend putting the applications on a separate host from zookeeper. https://developer.ibm.com/streamsdev/docs/performance-best-practices/
Can you tell us about your zookeeper setup? Is zookeeper on separate host(s)? Are data and transaction log on separate local disks? What kind of disk is used for the zookeeper transaction log? What is the maximum latency with streamtoool getzkstate
Can you tell us a little about your usage? How many instances/jobs/pe's? Do you use the jmx or rest api? Does the CPU spike during specific activities or stay high all the time even when the jobs are in a steady running state? Have you considered upgrading to a later version? There have been many fixes and efficiency improvements in jmx. 4.2.0.3 is the latest but you should be aware of this issue. http://www-01.ibm.com/support/docview.wss?crawler=1&uid=swg1IT18807
Are you able to file a PMR? That might be best so we can collect specific detailed information and get you the fix for IT18807 if you are considering upgrade. You can open a ticket in the support portal at:
hi kathey ,thank for useful response. Now, our domain run well, we didn't restart domain,didn't restart any instance and ManagementServer by PID with huge CPU usage disappeared. Now, ManagementServer is running with another PID with low CPU usage.I suppose,/suppose/ that MS restarted by itself. We've also re-submited one job without threadedport config,it was deleted from spl code. We use RHEL server 7.0 and there were some references about the bad performance threaded model on RHEL 6.0 . So, I don't know what exactly happened but now we are well. thanks for good pdf performance-best-pracices.
Glad things are back to normal. If you get back in that state and want to restart jmx manually, you can run streamtool restartdomainservice -d <domain> jmx . If the problem, persists, do consider upgrade at somepoint, maybe on our next 4.2 fixpack or mod release to pick up many jmx fixes and improvements. If it still persists, definitely let us know.
ok, what about "config threadedport" on V.4.2? From knowledge center:"A performance problem exists when the pthread_yield subroutine is used on RHEL6. ". Does it support on Streams 4.2 and RHEL V.> 5.0 ?
Streams 4.2 requires a minimum OS version of RHEL 6.6. I will request a doc update to remove RHEL 5 from the doc[1]. The default behavior should prevent any problem related to this issue for the OS in use, but any related potential performance problem would be in the pe process, not jmx.
https://www.ibm.com/support/knowledgecenter/SSCRJU_4.2.0/com.ibm.streams.ref.doc/doc/sc.html