Digital Developer Conference: a FREE half-day online conference focused on AI & Cloud – North America: Nov 2 – India: Nov 9 – Europe: Nov 14 – Asia Nov 23 Register now

Close outline
  • United States
IBM?
  • Site map
IBM?
  • Marketplace

  • Close
    Search
  • Sign in
    • Sign in
    • Register
  • IBM Navigation
IBM Developer Answers
  • Spaces
    • Blockchain
    • IBM Cloud platform
    • Internet of Things
    • Predictive Analytics
    • Watson
    • See all spaces
  • Tags
  • Users
  • Badges
  • FAQ
  • Help
Close

Name

Community

  • Learn
  • Develop
  • Connect

Discover IBM

  • ConnectMarketplace
  • Products
  • Services
  • Industries
  • Careers
  • Partners
  • Support
10.190.13.195

Refine your search by using the following advanced search options.

Criteria Usage
Questions with keyword1 or keyword2 keyword1 keyword2
Questions with a mandatory word, e.g. keyword2 keyword1 +keyword2
Questions excluding a word, e.g. keyword2 keyword1 -keyword2
Questions with keyword(s) and a specific tag keyword1 [tag1]
Questions with keyword(s) and either of two or more specific tags keyword1 [tag1] [tag2]
To search for all posts by a user or all posts with a specific tag, start typing and choose from the suggestion list. Do not use a plus or minus sign with a tag, e.g., +[tag1].
  • Ask a question

Streams ManagementServer - huge CPU usage

270005426J gravatar image
Question by Alexandr_Semeshchenko  (48) | Apr 06, 2017 at 02:06 PM streamsdevstreamscpumanagementserver

hi, streams gurus, we have a strange metrics about streams ManagementServer on our domain:it's take about 200-700% of CPU usage. We have one domain, one host, 20 CPU, 120Gb RAM. The report screenshots attached here. thanks for the advices or shared experiences!

screen-shot-2017-04-04-at-44829-pm.png (131.4 kB)
screen-shot-2017-04-04-at-45153-pm.png (25.8 kB)
screen-shot-2017-04-04-at-45217-pm.png (25.6 kB)

People who like this

  0
Comment
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster

3 answers

  • Sort: 
120000KENK gravatar image

Answer by kathey (618) | Apr 06, 2017 at 02:39 PM

So glad you are back. I started another post because your post from April 4. Seemed to be lost somehow. I will point that issue to this one and we can discuss it here.

That process looks like jmx. The easiest way to correlate process id's is usually is to use streamtool getdomainstate -d <domain> --long for domain management services streamtool getresourcestate -d <domain> -i <instance> --long for instance management services and streamtool lspes -d <domain> -i <instance> for pes.

We have this document that talks about performance best practices and much of that advice pertains to load as well. We normally recommend putting the applications on a separate host from zookeeper. https://developer.ibm.com/streamsdev/docs/performance-best-practices/

Can you tell us about your zookeeper setup? Is zookeeper on separate host(s)? Are data and transaction log on separate local disks? What kind of disk is used for the zookeeper transaction log? What is the maximum latency with streamtoool getzkstate

Can you tell us a little about your usage? How many instances/jobs/pe's? Do you use the jmx or rest api? Does the CPU spike during specific activities or stay high all the time even when the jobs are in a steady running state? Have you considered upgrading to a later version? There have been many fixes and efficiency improvements in jmx. 4.2.0.3 is the latest but you should be aware of this issue. http://www-01.ibm.com/support/docview.wss?crawler=1&uid=swg1IT18807

Are you able to file a PMR? That might be best so we can collect specific detailed information and get you the fix for IT18807 if you are considering upgrade. You can open a ticket in the support portal at:

http://www-947.ibm.com/support/entry/portal/Overview/Software/Information_Management/InfoSphere_Streams

Comment

People who like this

  0   Show 1   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
270005426J gravatar image Alexandr_Semeshchenko (48)   Apr 07, 2017 at 07:57 AM 0
Share

hi kathey ,thank for useful response. Now, our domain run well, we didn't restart domain,didn't restart any instance and ManagementServer by PID with huge CPU usage disappeared. Now, ManagementServer is running with another PID with low CPU usage.I suppose,/suppose/ that MS restarted by itself. We've also re-submited one job without threadedport config,it was deleted from spl code. We use RHEL server 7.0 and there were some references about the bad performance threaded model on RHEL 6.0 . So, I don't know what exactly happened but now we are well. thanks for good pdf performance-best-pracices.

120000KENK gravatar image

Answer by kathey (618) | Apr 07, 2017 at 02:23 PM

Glad things are back to normal. If you get back in that state and want to restart jmx manually, you can run streamtool restartdomainservice -d <domain> jmx . If the problem, persists, do consider upgrade at somepoint, maybe on our next 4.2 fixpack or mod release to pick up many jmx fixes and improvements. If it still persists, definitely let us know.

Comment

People who like this

  0   Show 1   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
270005426J gravatar image Alexandr_Semeshchenko (48)   Apr 10, 2017 at 08:03 AM 0
Share

ok, what about "config threadedport" on V.4.2? From knowledge center:"A performance problem exists when the pthread_yield subroutine is used on RHEL6. ". Does it support on Streams 4.2 and RHEL V.> 5.0 ?

120000KENK gravatar image

Answer by kathey (618) | Apr 12, 2017 at 03:47 PM

Streams 4.2 requires a minimum OS version of RHEL 6.6. I will request a doc update to remove RHEL 5 from the doc[1]. The default behavior should prevent any problem related to this issue for the OS in use, but any related potential performance problem would be in the pe process, not jmx.

https://www.ibm.com/support/knowledgecenter/SSCRJU_4.2.0/com.ibm.streams.ref.doc/doc/sc.html

Comment

People who like this

  0   Share
10 |3000 characters needed characters left characters exceeded
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster

Follow this question

110 people are following this question.

Answers

Answers & comments

Related questions

I cant add resources to a enterprise domain in IBM Streams 6 Answers

How to stop SPSS modeler 15/16 and IBM Streams 3.2 from generating strange logs - modelerrun_logging.log 1 Answer

Streams congestion ,Explanation required. 1 Answer

JobControlPlane - Crashed - Dump 1 Answer

CDISP0046E ERROR: A type mismatch 1 Answer

  • Contact
  • Privacy
  • IBM Developer Terms of use
  • Accessibility
  • Report Abuse
  • Cookie Preferences

Powered by AnswerHub

Authentication check. Please ignore.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • API Connect
  • Analytic Hybrid Cloud Core
  • Application Performance Management
  • Appsecdev
  • BPM
  • Blockchain
  • Business Transaction Intelligence
  • CAPI
  • CAPI SNAP
  • CICS
  • Cloud Analytics
  • Cloud Automation
  • Cloud Object Storage
  • Cloud marketplace
  • Collaboration
  • Content Services (ECM)
  • Continuous Testing
  • Courses
  • Customer Experience Analytics
  • DB2 LUW
  • Data and AI
  • DataPower
  • Decision Optimization
  • DevOps Build
  • DevOps Services
  • Developers IBM MX
  • Digital Commerce
  • Digital Experience
  • Finance
  • Global Entrepreneur Program
  • Hadoop
  • Hybrid Cloud Core
  • Hyper Protect
  • IBM Cloud platform
  • IBM Design
  • IBM Forms Experience Builder
  • IBM Maximo Developer
  • IBM StoredIQ
  • IBM StoredIQ-Cartridges
  • IIDR
  • ITOA
  • InformationServer
  • Integration Bus
  • Internet of Things
  • Kenexa
  • Linux on Power
  • LinuxONE
  • MDM
  • Mainframe
  • Messaging
  • Node.js
  • ODM
  • Open
  • PartnerWorld Developer Support
  • PowerAI
  • PowerVC
  • Predictive Analytics
  • Product Insights
  • PureData for Analytics
  • Push
  • QRadar App Development
  • Run Book Automation
  • Search Insights
  • Security Core
  • Storage
  • Storage Core
  • Streamsdev
  • Supply Chain Business Network
  • Supply Chain Insights
  • Swift
  • UBX Capture
  • Universal Behavior Exchange
  • UrbanCode
  • WASdev
  • WSRR
  • Watson
  • Watson Campaign Automation
  • Watson Content Hub
  • Watson Marketing Insights
  • dW Answers Help
  • dW Premium
  • developerWorks Sandbox
  • developerWorks Team
  • Watson Health
  • More
  • Tags
  • Questions
  • Users
  • Badges