The ramblings of a developer
Like any good bug or problem, we started to hear (and see) reports of slow downs in page load times on Friday night. We investigated this and found that the data center we host our server in had a slow down in their SAN disk storage. This caused increased IO, server load, and page load times.
The graph provided below shows the server load from Wednesday, July 11, 2012 to Sunday, July 15, 2012. The areas marked in green are normal server load. This is roughly where we shoot for to maintain optimal performance. Areas marked in yellow are server spikes due to increased traffic or other usage (such as backup scripts). These are normal usage patterns but we still monitor it so it doesn't get out of hand. The area in red is what happened when the data center started having disk IO issues. You notice that the load on the server, relatively speaking, is far from it's normal operating condition.
The issue has been worked out and a fix has been applied in the data center. The server is now safely back in the 'green zone'. We will continue to monitor this as we always do to make sure the server is running as fast as possible.