Subject: August End Of Month Close - A Summary Debrief
From: [email-masked]
To: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Date: Wed, 6 Sep 2000 06:04:00 -0700 (PDT)

The August end of month close is behind us.  Once again, we learned some 
things. We made progress.....but we also have issues to work. 

Summary On Improvements For This End Of Month Close

I would like to congratulate the users, the application team and the DBA team 
for a much improved end of month close for August gas. This would not have 
been possible without the significant commitment, focus and teamwork 
exhibited by all the folks involved in this effort. 

The most tangible measure of progress over the July close : 
We were able to keep all users in the system throughout the final week. No 
one was kicked off the system.
The gas accounting close process finished on time. Key users and IT folks 
didn't have to work significant overtime to finish the process. 

 Improvements arose from : 

Upgrading the Unify system to Sybase 12 with extra server memory. This 
reduced system 'locking and blocking' and made it possible to keep all users 
in the system throughout the week. Nightly settlements for gas invoicing also 
ran much faster. Many thanks to the DBAs - Jim Ogg and Kishroe Bharathi-- for 
many late nights and weekends. Many thanks also to the application team and 
the users for lots of testing before the production rollout.

Conducting a pre-planning session that identified new end of month techniques 
to improve efficiency. The users generated several new ideas including the 
idea to pre-process PMAs the night before running the current mass draft. 
This meeting provided several new approaches, a clear plan and cohesive 
communication throughout the end of month process. Many thanks to the all the 
users as well as Bob Superty, Bryce Baxter, Melissa Ratnala, Dave Nommensen 
and Chris Schomer for making this happen. Many of these ideas will now be 
part of a standard process. 

The performance monitoring and tuning team reacted quickly to address changes 
in system performance. Many thanks to Matt Pena and the rest of this team for 
keeping the system up and running as fast as possible during the day. 

Release of a new tuned version of the Accurals process that runs much faster. 

Issues, Challenges, Action Items

While we made progress this month, there are still several significant issues 
and opportunities for improvement. The following is a list of the things that 
'went bump in the night' and the action items we've identified. 

1. We experienced several Sybase 12.0 errors early in the week that impacted 
gas business. All of these problems are now fixed. This had the following 
business impacts: 
The bridgeback of deals to Sitara was delayed. These deals were bridged 
manually before the end of month valuation. We also fixed the code problem 
that generated th original problem.
Global Systems replication to Unify was down on Monday. This problem was 
fixed with a software patch from Sybase Monday evening. All affected deals 
were then uploaded. Nominations were completed for these deals on Monday 
through manual estimation by the schedulers. 
Communication between the DBA group and the application group was delayed on 
the Global Systems replication problem. This delayed our ability to notify 
users of the problem. Communication on problems will be be improved in the 
future. The default assumption will be to notify key application and 
infrastructure personnel as soon as any serious problem is identified. 

2.  We experienced a hardware issue with a bad server CPU at 3:00AM on 
Thursday.  As a result, database backups ran into the morning when the server 
came back up. This created slow system response and delayed logins for users 
until 10:00 AM. This problem is now fixed. 
 Action Item: Kishore will coordinate with Randy Matson from the 
infrastructure group to coordinate a stress test of all production server 
components. This will allow us to see if we can find any other marginal 
components.  This will be completed by September 18. 

3. All service PMAs for ENA and HPLC were not picked up by the Settlements 
process. HPLC PMAs were run manually by the users. For ENA, missing PMAs were 
immaterial and the users decided to close with these results. We have now 
found and fixed the error that caused this problem. 
 Action Item: Chris Schomer is having his team investigate this problem to 
see if this is just a Sybase 12.0 issue or whether this error could have 
impacted previous months. A determination should be available by September 
15. 

4. Enpower deal load did not load deals with referenced prices correctly. 
These deals have now been successfully loaded and are currently completed 
through the mass draft process for power. We have identified and fixed the 
software error that caused this problem. 

5. Power mass draft is now running very slow as compared to prior months. We 
are currently working to complete the Power mass draft process. It will be 
completed by the user deadline for close of business Wednesday Sept. 6. 
 Action Item: Chris Schomer's team is investigating why the Power mass draft 
is running so slow.  At present we do not have a definitive answer. We hope 
to develop a definitive answer and action plan by September 15. Our current 
hypothesis for why we have this problem includes : 
Data changes - we now have a much higher volume of hourly priced deals
Potential database index issues. We will re-build all power database indexes 
and re-compile the stored procedures
Sybase optimizer flakiness. We often see Sybase suddenly start to have 
processing problems for no apparent reason.  

Update On Future Tactical  And Strategic Improvements

We have several pending tactical Unify performance improvements that are in 
process. These include: 

1) Isolating Unify Gas on a server of it's own 
2) Delivery of a middle tier mass draft process to reduce locking and blocking
3) Upgrade of the middle tier mass draft process to allow for drafting at a 
deal level (rather than just a counter-party level)
4) Delivery of the nominations and balancing process on a middle tier 
architecture.

Ramesh Rao and his team have the strategic projects progressing on schedule.  
We are seeing significant performance gains in our testing of SQL Server over 
Sybase. We also have identified a couple of hardware vendors that could 
provide very large, scaleable computing environments for this type of Unify 
architecture. We plan to have all critical evaluation testing completed by 
the end of September as planned. We will present our recommendations along 
with a high-level strategic delivery plan by early October. 

Let me know if you have any questions or concerns --  x39694 or (713) 
503-4873. 

Thanks.