Subject: Re: Systems - ERMS
From: [email-masked]
To: [email-masked]
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Date: Wed, 18 Oct 2000 02:48:00 -0700 (PDT)

John,

These issues are being addressed by Steve Stock and Chris Hanz in my area as 
well as people on the infrastructure side (in Jenny Rub's organization).  We 
understand the significance of these problems and they have not gone 
unrecognised.  There are 3 components to this plan, the DBA fixes are noted 
below (I know it's a lot of technical terms, but this is a highly complex 
problem).  The other 2 pieces are the application and the hardware.  I will 
include these plans as soon as I receive them.

My team stayed last night with Jeff Gossett and his folks.  A number of items 
came out of observing what transpires in risk.  We feel that there are many 
performance enhancements that can be made to the macros to speed up non ERMS 
processing.   I'll keep you posted on what we find and the plans to fix them.

Beth

Database Observed Problems

? OS - Severe I/O contention (due to 6 years old disk storage device)
? OS - Max CPU utilization (due to wait I/O contention)
? DB - Redo Log Sync (synchronization of Redo information memory with disk)
? DB - Shared Pool fragmentation (due to unbound SQL )
? DB - Inefficient buffer cache utilization (due to inefficient SQL)

Recommendation

Short Term

? Alleviate Shared Pool contention
- increased Shared Pool size
- disabled MTS
- increased session_cached_cursors
Implemented on 10/15/2000

? Alleviate Redo Log Sync
- increased redo_log_buffer
Implemented on 10/15/2000

? Alleviate I/O contention for rollback segment tablespace
- added additional tablespace with 2/3more rollback segments
Implemented on 10/17/2000

? Alleviate I/O contention for redo logs
- reorg disk storage and implement striping for Redo logs
- turn on force direct I/O write for intensive volumes
Under investigation ) 2 days work

? Turn on Veritas tuning feature
- expect 5% improvement
- tested on Shemp today
Required system reboot 

? SQL Tuning 
Deliver Lists of expensive SQL with recommendation by 10/27/2000

? Reassess CPR migration to different UNIX host
Waiting for disk storage ) 2 to 3 weeks
       New 10K located in Ardmore

Long Term

? Latest storage technology

? Capacity planning

? ERMT and CPR application review

? Upgrade ERMT to 8I

	John J Lavorato@ENRON
	10/17/2000 01:31 PM
		 
		 To: Beth Perlman/HOU/ECT@ECT, Philippe A Bibi/HOU/ECT@ECT
		 cc: 
		 Subject: Systems - ERMS

---------------------- Forwarded by John J Lavorato/Corp/Enron on 10/17/2000 
01:25 PM ---------------------------

	
	From:  Jeffrey C Gossett @ ECT                           10/17/2000 09:45 AM

To: John J Lavorato/Corp/Enron@Enron
cc:  

Subject: Systems - ERMS

John -

Again,  we are having "big time" systems issues.  While we are having the 
usual length of time issues (a few people were here past midnight last 
night), the issues that occured last night are much more serious than in the 
past.  The system brought through totally inaccurate information on a few 
books.  (Positions,  value,  etc.)  Luckily, these errors were all caught 
last night,  but if a less experienced person had been on those books,  it 
could have been a disaster.  (Traders getting incorrect positions and p&l and 
interest rate and credit getting incorrect numbers.)  I have contacted those 
in ERMS that need to be involved,  but I would appreciate it if you could 
provide additional support to this effort.

Thanks,

Jeff