Spark 1.6.0 was released on Jan 4th, and we took it for a “test drive”. In our performance labs, we tested four workloads with varying data volumes on a bare metal Hadoop cluster and compared results to earlier Spark versions using the same workloads. Here is the preview.

Spark 1.6.0 performance preview




Some observations

  • We ran all workloads using spark-submit in yarn-client mode. We found that Spark 1.6.0 requires more driver memory than previous versions. For example, the TPCDS SQL workload used 8GB driver memory in 1.5.1 but needed 12GB for Spark 1.6.0 in order to run without failed stages. So the memory footprint of Spark 1.6.0 increased noticeably, and based on your application, you may also need to increase executor’s memory.
  • The TPCDS SQL workload improved by almost 10% in performance compared to 1.4.1. Almost all queries in the workload ran faster, making this a really solid release for Spark SQL. You can evaluate Spark SQL using this test kit on github.
  • The K-Means workload degraded by 20% or so from 561 seconds in Spark 1.5.1 to 680 seconds in 1.6.0. We have noticed higher CPU usage in 1.6.0. K-Means library updates may also affect this workload (i.e., code change may be required in our workload). We are looking into the cause.

More to come

You may ask where is Spark Streaming? We are currently evaluating how streaming applications and Parquet data formats are affected by Spark 1.6.0. Please stay tuned for more performance information.

22 comments on"Spark 1.6.0 Performance Sneak Peek"

  1. […] running these queries simply with text and Parquet storage formats in Spark 1.5.1 and in the just-released Spark 1.6.0. The total Spark working memory is 500GB. The TPC-DS scale factor is […]

  2. ロレックス レディース 選び方 100円 October 10, 2016

    贈り物 ブレゼント通販!
    ロレックス レディース 選び方 100円

  3. 大黒屋 シャネル時計 プルミエール October 11, 2016

    当店では信用第一を基に、スーパーコピーshoppingkopi 出荷した商品に
    大黒屋 シャネル時計 プルミエール

  4. bague cartier replique love November 13, 2016

    cartierlovejesduas I’d love to actually grow some real vegetables this year! Last year my gardening was a bit of a bust. There was only one flower that actually grew and my sisters and I ended up calling it the Voldemort flower because the bud looked like a really creepy red eye staring at you. It took forever to bloom, but when it finally did the name had already stuck! These muffins look really good–I definitely need a reminder of summer at this point. I would definitely make them to serve at a barbecue!
    bague cartier replique love

  5. Sherlyn Edeker February 03, 2017

    Pretty section of content. I just stumbled upon your site and in accession capital to assert that I acquire in fact enjoyed account your blog posts. Anyway I will be subscribing to your feeds and even I achievement you access consistently fast.

  6. Knockoff love bangle rose June 16, 2017

    What’s next…will people be removing the dates from their posts?
    Knockoff love bangle rose

  7. ルイヴィトンバッグコピー July 25, 2017

    以上 宜しくお願い致します。(^0^)
    広大な客を歓迎して買います!── (*^-^*)

  8. technology adviser September 01, 2017

    Any team would be lucky to have you on it. Found this on LinkedIn and I am happy I did I am bored to death at work so I decided to browse your blogs on my iphone during lunch break. I really appreciate your efforts and I will be waiting for your further blog post thank you once again.

  9. 世界的に有名スーパーブランドコピー専門店 September 02, 2017

    スーパーコピーブランド通販 ブランドスーパーコピー格安買取

    激安、卸売り、偽物、シャネル財布、バッグ、時計、代金引換、スーパーコピーブランド専門店販売送料 無料。


  10. Just wanted to say fantastic post! You should be incredibly proud of yourself. I have learned new things through your website.

  11. It is really magnificent you do not have a larger following. Oh my goodness! A excellent article.

  12. It’s a shame you don’t have a donate button! I’d definitely donate to this excellent blog! I guess for now i’ll settle for bookmarking and adding your RSS feed to my Google account. I look forward to new updates and will share this blog with my Facebook group. Chat soon!

  13. Hi! This is my first visit to your blog! We are a group of volunteers and starting a new project in a community in the same niche. Your blog provided us beneficial information to work on. You have done a outstanding job!

  14. If you would like to improve your familiarity only keep visiting this
    site and be updated with the most up-to-date news posted here.

  15. I believe this is among the most important information for me.
    And i am satisfied reading your article. But want to observation on some
    normal things, The site style is perfect, the articles is in point of fact
    great :D. Excellent activity, cheers.

  16. Appreciating the hard work you put into your blog and in depth information you offer. It’s great to come across a blog every once in a while that isn’t the same outdated rehashed information. Fantastic read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.

  17. Thanks for the marvelous posting! I truly enjoyed reading it, you might be a great author.I will remember to bookmark your blog and may come back in the foreseeable future. I want to encourage continue your great posts, have a nice weekend!

  18. 2019年最高ブランドコピー 服、財布.
    ブランドコピー 服 、ブランド コピーネットショップとしてずっと多くのお客様に信頼頂いております。 品質よくて、 激安 の大特価でご提供します。
    ★2019年新品バッグ、財布、手帳、腕時計、ブランド 靴、ファッション 服
    ■主要取扱商品 ブランドコピー 服 バッグ、財布、腕時計、マフラー!

  19. This blog post is so much marvelous and interesting.

Join The Discussion

Your email address will not be published. Required fields are marked *