Message-ID: <11901723.1075862314091.JavaMail.evans@thyme> Date: Mon, 26 Nov 2001 07:50:45 -0800 (PST) From: beth.apollo@enron.com To: sally.beck@enron.com Subject: EA confirm problem (FAX server issues prevented the faxed confirms from coming thru to be processed) Cc: m.hall@enron.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: m.hall@enron.com X-From: Apollo, Beth </O=ENRON/OU=NA/CN=RECIPIENTS/CN=BAPOLLO> X-To: Beck, Sally </O=ENRON/OU=NA/CN=RECIPIENTS/CN=Sbeck> X-cc: Hall, Bob M </O=ENRON/OU=NA/CN=RECIPIENTS/CN=Rhall3> X-bcc: X-Folder: \SBECK (Non-Privileged)\Beck, Sally\Inbox X-Origin: Beck-S X-FileName: SBECK (Non-Privileged).pst Sally - Just so you know, this IT problem(see mesage below) has caused lots of late nights and work over the holiday weekend to catch up. Not totally confident that the cause has been diagnosed, but we will keep pushing IT. It went un-discovered for quite a while due to the ramp down in Enron Business that has occurred over the last few months(related to Enron's status.) I mentioned to Beth P. who was aware of the problem. I just hope that the procedures that they have outlined below are enough to ensure that this does not happen again. Pat and crew jumped on it - they worked this weekend to get rid of the back log. Everything is up to date in TAGG and showing confirmed-- They wanted to be ready for bid week -- assuming we have the normal high volume. ALL OK now Mainly just an FYI -- in case you hear anything. -----Original Message----- From: Weatherspoon, Patricia Sent: Wednesday, November 21, 2001 9:09 AM To: Apollo, Beth Cc: Theriot, Kim S.; Reeves, Leslie Subject: FW: Facsys Server Status Importance: High -----Original Message----- From: Pua, Cornelio Sent: Tuesday, November 20, 2001 7:21 PM To: Weatherspoon, Patricia; Gilley, Carolyn; Neves, Brenna Cc: Kachinski, Joe; Stevens, Wilford; Larkin, Brian; Paskin, John; Wong, Joe; Wang, N. Jessie; Wells, Rob; Behney, Chris Subject: Facsys Server Status Importance: High Pat, Carolyn: Status: As it stands right now, infrastructure has recovered approximately 14,000 fax messages from the 2 fax servers. 5,213 of the total number were recovered from backups last night. EDMS has been continuing to support records management in getting these fax messages processed and has been printing the faxes since last weekend. We will print the remaining 5,213 tonight. The question of total number of confirmations that were affected by this is still unknown until these documents are released to LiveLink. So far, 7,961 confirmations have been indexed. An additional 1,800 (approx.) will be indexed tonight. These documents will need go though the OCR process, routed to the LiveLink workflow process, and then filed in LiveLink. We currently do not have any indication on how many confirmations will be in the 5,213 fax messages still to be processed. Investigation To Date: The infrastructure team is continuing to investigate what went wrong with the facsys servers. Unfortunately, the servers did not give any indication that it was not properly routing fax messages until it completely stopped routing fax messages last Thursday, November 15, 2001. Apparently, some of the fax messages were being routed and the assumptions have been that the reduced volume over the past few weeks were attributed to Enron's current market position. The problem was compounded with the automated monitoring tool that supposed to alert administrators did not function as intended. Immediate/Future Actions: To ensure this does not happen again, continuous monitoring will be done. Infrastructure will continue to be responsible for the facsys servers. They will manually monitor the facsys servers until we have the necessary utilities in place for the EDMS team to monitor them. They will send e-mail confirmations of checks daily (9 am and 3 pm on weekdays, once during weekends/holidays). Brian Larkin has indicated that this tool will be available next week. Additionally, the infrastructure team is accelerating plans to ensure proper support of the facsys servers are established. Should any issues arise regarding the facsys servers, EDMS will utilize the following escalation process: 1. Contact the Emergency Hotline @ 713-345-4911 2. Joe Kachinski 3. Wilford Stevens 4. Brian Larkin If we are unable to contact any of the people indicated above, the problem will be escalated to the proper management reporting structure until it is addressed. Thank you very much. Cornelio