Message-ID: <9255776.1075862308236.JavaMail.evans@thyme> Date: Thu, 8 Nov 2001 05:50:00 -0800 (PST) From: m.hall@enron.com To: ramesh.rao@enron.com Subject: FW: Unify Status Report Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-From: Hall, Bob M X-To: Rao, Ramesh X-cc: X-bcc: X-Folder: \SBECK (Non-Privileged)\Beck, Sally\Inbox X-Origin: Beck-S X-FileName: SBECK (Non-Privileged).pst This is being to sound a lot like SYBASE. did we spend all this money and still have the same issues? can we talk some time about this. bob -----Original Message----- From: Warner, John Sent: Wednesday, November 07, 2001 6:05 PM To: Hall, Bob M; Superty, Robert; Wynne, Rita; Mcclure, Mark; Jaquet, Tammy; Pinion, Richard; Baxter, Bryce; Reeves, Leslie Cc: Rao, Ramesh; Pena, Matt; Valor, Tino; Ballmer, Charles; Vinson, Darryl; Ward, Bob Subject: Unify Status Report Summary of Bid-Week Performance On Monday, October 29 we experienced heavy system activity and significant performance problems in the morning. We were unable to fail-over, but we restarted SQL Server. Performance improved slightly. On Tuesday, October 30 we again experienced heavy usage and significant performance problems. We then re-started the standby server, and failed over. Overall reliability improved greatly, as the number of user calls related to performance issues dropped. Since the October 30 failover, we've experienced moderate to heavy system usage with just a few user calls related to performance. We have not failed over nor restarted SQL Server since October 30. Operational Improvement Initiatives It is very clear that current system performance and reliability is not meeting our expectations during heavy usage times. A Performance Improvement Team has been established to formally review the entire Unify system to identify the top performance offenders. To date the team has developed a methodology for identifying problematic procedures and tested that methodology to ensure metrics gathering won't impact production performance. Moving forward, the team will: Begin metrics gathering from our production systems Monday, Nov 12th. Analyze results and prioritize performance improvement opportunities beginning Monday, Nov 19th. Begin analyzing, tuning, and testing our biggest resource-consuming procedures on Monday, Nov 26th. Deploy changes to production by December 14th. In parallel with our efforts to optimize existing code, we will also be testing a new firmware version. Firmware governs processor and memory configuration on our servers. We believe that with Plateau 8.1 (the latest version of firmware for our database servers), we can realize performance gains. We will complete testing of Plateau 8.1 by late November/early December, and schedule the migration at that point. The dreaded "buffer latch" bug seems to have been eradicated as of October 19. Although we have encountered a second order problem with the fix provided by Microsoft, we are able to work around that problem. We have implemented new fail-over procedures whereby we restart our database server (the one which was experiencing problems) every time we fail-over. This does not impact our turnaround times, and will ensure that every time we fail over, we're moving to a freshly-restarted database server. We have implemented new database maintenance procedures which will reduce the liklihood of experiencing an index corruption, as we experienced on October 19th. Enhancement/Bug Fix Initiatives Team Leads have initiated regular meetings to review open work items. Our plan is to meet bi-weekly to discuss progress, current priorities and new change requests. We have added an "on hold" status to our SIR database. Putting an item on hold should be an explicit decision agreed upon by IT and business users. This new status will help ensure we're on the same page when priorities change and items are put on hold. Attached is a spreadsheet of our "Top 10" by business area. This list includes our major open items by priority, and includes bug fixes, enhancements, and some specific operational initiatives as well.