[SOLVED] Degraded Performance on HP DL580 G7 on BIOS v. 7/01/2013

After a December 2013 patching downtime, I noticed an abrupt decrease of almost 50% in the performance of 2 SQL Servers.  Both servers were running on HP DL580 G7s.  After many hours of testing and data collection, we finally updated the BIOS firmware from 7/01/2013 to 10/31/2013.

The issue immediately resolved itself.

Throughout the troubleshooting process, I envisioned having the most amazing blog post resulting from the (hopefully) inevitable resolution.  But it turned out to be somewhat anti-climatic.

I would like to say that attempts to engage HP on possible hardware issues was most unhelpful.  Especially now that we know what the issue was, it should have been fairly easy for HP to realize they had a faulty BIOS release for this server model.  Instead, we wasted 2 months trying to figure out what the issue could be.

I hope that if there is anyone else out there with this issue, that this post can be of use.

Brandon has worked in IT for nearly 20 years, and currently serves as a SQL Server DBA for a healthcare company in California. In his spare time, he runs for miles and helps maintain RealCajunRecipes.com. Brandon is a certified SQL Server administrator.

Posted in SQL Server, Windows
10 comments on “[SOLVED] Degraded Performance on HP DL580 G7 on BIOS v. 7/01/2013
  1. Evan says:

    Huge thanks for this. Been banging our heads against the wall for two weeks because of this. HP was no help for us either.

    Again, big thanks!

  2. Younggun says:

    Hi, Brandon.
    We got a similar situation and had a tough time.
    Could you tell me your system memory setting?
    Is that more than 128 GB?

    Thanks in advance.

    • Brandon Abshire says:

      The server had 512GB of RAM. I did notice that allocation of RAM was slower. But it was because the CPU was throttled down by the BIOS firmware.

  3. Arun Sanghvi says:

    We are having similar issue. What was the fix?

    Thanks.

  4. Chrissy LeMaire says:

    Hey Arun,
    They updated the BIOS firmware from 7/01/2013 to 10/31/2013 to resolve the issue. Any BIOS firmware other than 7/1/2013 should work.

  5. Partha says:

    Hi, we are having the same issue and I just got to this page. Were you experiencing any issues with respect to FC and Network traffic being slower?

    • Brandon Abshire says:

      I didn’t observe any issues with slower I/O. The I/O seemed to be comparable to before the bad BIOS firmware was installed. There were no improvements after the fix was implemented. It’s purely CPU from my understanding.

  6. Paul Xu says:

    Thanks Brandon for sharing. This also solved our I/O issue with Linux on FC storage. dd tests showed i/o writes were in the range of 100-400 MB/s while DL380 G8s got 600-800MB/s. The BIOS upgrade to 10/1/2013 (note HP site and linux dmidecode showed the new BIOS release date is 10/1/2013, not 10/31/2013) improved the IO. BIOS download is at http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdHome/?sp4ts.oid=4142792&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3DswEnvOID%253D4049%257CswLang%253D%257Caction%253DlistDriver&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

  7. Tomaz Hrescak says:

    Thank you for sharing! We have 3 identical ld580g7 servers and we noticed that two of them are roughly 50% slower (cpu fp operations and memory r/w throughput).

    The OK server is on BIOS version P65 08/04/2012, and the two slow ones are on P65 07/01/2013!

  8. Rosemarie DeRonne says:

    Brandon, THANK YOU!!!!! so very much for your post. It resolved our issue. We were experiencing the same problem on our Proliant DL380 G9 and couldn’t find any resolution. There was an update for our server as well and the results were awesome. Your posted saved us many additional hours of troubleshooting. Thank you again for taking the time to post it.

Leave a Reply

Your email address will not be published. Required fields are marked *

*