[SOLVED] Degraded Performance on HP DL580 G7 on BIOS v. 7/01/2013

After a December 2013 patching downtime, I noticed an abrupt decrease of almost 50% in the performance of 2 SQL Servers.  Both servers were running on HP DL580 G7s.  After many hours of testing and data collection, we finally updated the BIOS firmware from 7/01/2013 to 10/31/2013.

The issue immediately resolved itself.

Throughout the troubleshooting process, I envisioned having the most amazing blog post resulting from the (hopefully) inevitable resolution.  But it turned out to be somewhat anti-climatic.

I would like to say that attempts to engage HP on possible hardware issues was most unhelpful.  Especially now that we know what the issue was, it should have been fairly easy for HP to realize they had a faulty BIOS release for this server model.  Instead, we wasted 2 months trying to figure out what the issue could be.

I hope that if there is anyone else out there with this issue, that this post can be of use.