Spreadsheet: 96388d1295807133-condition-based-sumproduct-sumproduct
Sheet: Raw1

 |  |  |  | Company |  | LSA Affected
Sno | Month - PIR DUE | Month - Event Reported | Event Date | ZNA | FARMERS | Canada | Exchange | Management | ZNA | Change # | Exception? | Group | Change Type | Severity | Change Title | Event & Outage Detail | Customer Follow up -- Line item in red until CAB meeting then change color to black | Date Confirmed | PIR Due Date = Confirmed Date + 10 business days | PIR Complete Date | Client (Retained) Y/N | PIR Completion Status | Change Category
1 | Dec-09 | Dec-09 | 2009-12-03 00:00:00 |  | 1 |  |  |  |  | CR0193188 |  | ispna_ts_dbase-dist | BAU | 4 | BAU -  ispna_ts_dbase-dist - Bert Frichot | Event  3 :   (Farmers) HERO database server OKAPDB02 was unavailable. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Approx. 20 GR HelpPoint users received "Page Cannot Be Displayed"   errors within different facets of HERO. Corrective Action:  OKAPDB02 recycled itself. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:35 PT to 07:55 PT Event Duration: From 07:35 PT to 07:55 PT  Ticket Number: IM1702079 | [NAME]: Team was adding space and caused a recycle. However, server is in active-active pair so should have been no disruption.  [NAME]: PIR completed | 2009-12-03 00:00:00 | 2009-12-17 00:00:00 | 2009-12-10 00:00:00 |  | Met | isp
2 | Dec-09 | Dec-09 | 2009-12-04 00:00:00 | 1 |  |  |  |  |  | CR0190057 |  | ispna_firewall | Minor | 4 | Perseus Omaha Vendor Farm Firewall Change request - DMR 83455 | Event  1 :   (ZNA) Some CapGemini employees were unable to launch   Lotus Notes via Citrix. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Approx. 60 CapGemini employees received "Your roaming profile is   unavailable" messages when attempting to access Lotus Notes through   Citrix. Corrective Action:  Re-installed deleted IP routing rules on HM2NKFW11, SCHNKFW11,  (Immediate Solution) and OKCNKFW01. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 00:35 PT to 01:22 PT Event Duration: From 00:35 PT to 01:22 PT  Ticket Number: INC158219976 | [NAME] was a firewall issue caused by human error.  For the CapGemini DMZ Segment, a firewall rule had been deleted which removed an IP in error.  [NAME] added this back in to correct the issue.   [NAME]:  PIR Completed | 2009-12-07 00:00:00 | 2009-12-21 00:00:00 | 2010-01-14 00:00:00 |  | Not Met | isp
3 | Dec-09 | Dec-09 | 2009-12-06 00:00:00 |  | 1 |  |  |  |  | CR0186204 |  | ispna_distsvc_ops | Standard | 4 | Install  Office 2003 Service Pack 3  (please see attached document for affected servers) | Event  2 :   (Farmers) The Farmers Life website, www.farmerslife.com, was   unavailable. Severity Level: 4 Root Cause: CR0193809. Customer Impact: Users received "application error" messages when attempting to access   www.farmerslife.com. Corrective Action:  LifeNet support implemented changes to the Windows system registry  (Immediate Solution) on PAWEB0340 and recycled the Session State Server service. Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From (12/05) 22:00 PT to (12/06) 03:46 PT Event Duration: From (12/05) 22:00 PT to 03:46 PT  Ticket Number: IM1703325, INC158222433 | Per [NAME]:  The LifeNet site was functioning fine until the updates were made Saturday evening.   When we contacted OKC ops we were told that the Office updates were being applied to our servers and would not be done until midnight.   It did not make sense to us why Office updates would affect our web servers.  We then proceeded to contact Justin Meyer (Wintel West Ops) and he advised that Windows SP2 was also being applied to our servers which could affect our servers.  What we found was that a windows registry setting was changed after the server updates were made (most likely by CR0186204 listed above).   Per OKC ops, there were no system or network issues after the installs and so it must be an application problem.    Therefore, the LifeNet on-call staff spent time researching, diagnosing, validating and ultimately correcting the system problem.  We eventually found a knowledge base item that pointed us to the registry issue with HKLM\System\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection on PAWEB0340.  The value  had gotten changed to a 0 and it needed  to be a 1 to allow access by the other  LifeNet web farm servers.  PAWEB0340  is the server that manages session state for LifeNet on the web farm.  [NAME]: PIR complete. | 2009-12-07 00:00:00 | 2009-12-21 00:00:00 | 2010-01-19 00:00:00 |  | Not Met | isp
4 | Dec-09 | Dec-09 | 2009-12-06 00:00:00 |  | 1 |  |  |  |  | CR0189353 |  | ispna_ts_os390 | Standard | 4 | Add 32 gig of memory to PRG1and PRH1. | Event  3 :   (Farmers) Customer Self Service was not available. Severity Level: 4 Root Cause: Memory upgrade on the PRH1 system. CR0189353. Customer Impact: Farmers.com users were unable to access CSS. Corrective Action:  Memory upgrade completed. (Immediate Solution) Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 07:00 PT to 09:10 PT Event Duration: From 07:00 PT to 09:10 PT  Ticket Number: INC158222758 | Per [NAME]  >>  eAgent had not been added as an approver on this change and they were unaware of the memory upgrade.  Per [NAME]  >>  The PRH1 memory upgrade took longer than expected.      [NAME]:  PIR complete. | 2009-12-07 00:00:00 | 2009-12-21 00:00:00 | 2009-12-15 00:00:00 |  | Met | isp
5 | Dec-09 | Dec-09 | 2009-12-07 00:00:00 |  | 1 |  |  |  |  | CR0192796 |  | aspna_ade | Standard | 5 | Release 12, 2009 MICRO-4 IVR Implementation | Event  12 :   (Farmers) Some users are getting error messages when   attempting to save content via IVR FileNet. Severity Level: 5 Root Cause: Under investigation. Customer Impact: Affected users are receiving "System error" messages instead of a   confirmation indicating a successful save when uploading documents   via IVR or Image Center. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: None Event Duration: From 07:40 PT to Ongoing Ticket Number: IM1703414 | Per [NAME]  >>  This was a fake message and everything was still functioning properly.  The change was backed out and a fix will most likely be put in place with Rel12 PROD (12/19).    [NAME]:  PIR complete | 2009-12-08 00:00:00 | 2009-12-22 00:00:00 | 2009-12-21 00:00:00 |  | Met | asp
6 | Dec-09 | Dec-09 | 2009-12-07 00:00:00 | 1 |  |  |  |  |  | CR0194609 |  | ispna_netwksvc_support | Minor | 3 | OKC: [NAME] port on okcc6sw33 and 34 for sniffing. | (Farmers) OKC, KCY, and GR Help Point users experienced sporadic network performance issues. Root Cause: OKC Switch issue. Customer Impact:  500 users experienced network slowdowns when attempting to login or access network based applications. As a partial workaround, the GR and KCY Help Points were able to utilize the CRN fat client via their desktops. Help [NAME] was sporadically unavailable for all 3 Help Points. NWCS users experienced sporadic clocking or "white screens" when attempting to navigate various portions of the application. Corrective Action (Immediate solution): Network support changed the aging time on the MAC address tables and removed port monitoring on all switches located in the Oklahoma City Data Center.12/08 working with vendor CISCO. Corrective Action (Permanent solution): Under investigation. | [NAME]: Port Monitor change, 10 gig link input to a 1 gig port caused this event    [NAME]: PIR completed | 2009-12-08 00:00:00 | 2009-12-22 00:00:00 | 2010-01-06 00:00:00 |  | Not Met | isp
7 | Dec-09 | Dec-09 | 2009-12-13 00:00:00 |  | 1 |  |  |  |  | CR0194931 |  | ispna_distsvc_ops | Standard | 4 | Microsoft's Monthly Security Patch installation (Farmers OKDC Production Servers) | Event  2 :   (Farmers) OKC, KCY and GR HelpPoint users were unable to   utilize HERO and FNOL. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Approx 75 users received "Currently there is a connectivity issue"   messages when working within HERO and FNOL. Users were able to   utilize CRN to create claims as a workaround. Corrective Action:  Wintel support reinstalled Service Pack 2 for Windows Server 2003 on (Immediate Solution)  affected servers. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 11:30 PT to 15:23 PT Event Duration: From 11:30 PT to 15:23 PT  Ticket Number: IM1705840 | Per [NAME] was related to a particular patch that was installed. We received notification that there was a bug with this one patch after the installation was made.  [NAME]: PIR complete | 2009-12-14 00:00:00 | 2009-12-28 00:00:00 | 2010-01-18 00:00:00 |  | Not Met | isp
8 | Dec-09 | Dec-09 | 2009-12-13 00:00:00 |  | 1 |  |  |  |  | CR0194923 |  | ispna_netwksvc_telecom-w | Standard | 5 | Migrate [NAME] on OKCNTGN01 and KCYNTGN01 | Event  3 :   (Farmers) OKC HelpPoint CSA's received default-routed calls. Severity Level: 5 Root Cause: Under investigation. Customer Impact: Approximately 25 CSA's were not being placed into an 'ACW" status   via CTI. Workaround in place, users were able to manually enter   "Ready" status in order to receive calls. Corrective Action:  Telecom support backed out a portion of CR194923; they reverted  (Immediate Solution) ASAI protocol to ver. 3 from 4. Corrective Action: Under investigation. (Permanent Solution) Service Outage: None Event Duration: From 06:03 PT to 13:51 PT  Ticket Number: CR194923 | [NAME]: The protocol version was the issue with of this event. Corrective action was to revert back to protocol 3. The Avaya vendor will investigate further.  [NAME]: PIR completed | 2009-12-14 00:00:00 | 2009-12-28 00:00:00 | 2010-01-15 00:00:00 |  | Not Met | isp
9 | Dec-09 | Dec-09 | 2009-12-16 00:00:00 | 1 |  |  |  |  |  | CR0191456 |  | aspna_usams_zspire | Minor | 3 | Scheduled Z-Spire migration on  12/11/2009 | Event  1 :   (ZNA) Z-Spire was displaying degraded performance with GL   forms. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Affected users in multiple locations were receiving long page   generations when navigating between General Liability forms screens   which expanded to include Auto policy transactions. Corrective Action:  Support activated AVL & Rate File caching, replaced [NAME] and  (Immediate Solution) recycled environment under CR0196370. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:20 PT to 16:12 PT Event Duration: From 06:20 PT to 16:12 PT  Ticket Number: INC158243291 | [NAME]: Corrective action: Support activated AVL &amp; Rate File caching, replaced [NAME] and recycled environment under CR0196370. Vendor will investigate the Root Cause.  [NAME] completed | 2009-12-17 00:00:00 | 2009-12-31 00:00:00 | 2010-01-11 00:00:00 |  | Not Met | asp
10 | Dec-09 | Dec-09 | 2009-12-16 00:00:00 | 1 |  |  |  |  |  | CR0196164 |  | ispna_dcs_prod_inet | Emergency | 4 | Exception #102:  RI - Correct problem with SSL and IHS | Event  3 :   (ZNA) Risk Intelligence Release 3 was unavailable. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users were unable to login. A work around was in place. [NAME]had access to the newer version of RI. Corrective Action:  INET support redirected the SSL processing from the WebSphere to  (Immediate Solution) the F5. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:57 PT to 06:30 PT Event Duration: From 05:57 PT to 06:30 PT  Ticket Number: INC158245308 | [NAME]: This change was fully tested prior and post implementation and Risk Intelligence Portal worked as expected.  However, during the testing phase, Risk Intelligence Release 3 (old version) was not tested.  All users have access to Risk Intelligence Portal and could use that as a workaround instead of using Risk Intelligence Release 3.  To correct the problem, iNet support made changes to the JVMs that run the old version to redirect the SSL handling from Websphere to the F5.   [NAME] completed | 2009-12-17 00:00:00 | 2009-12-31 00:00:00 | 2010-01-07 00:00:00 |  | Not Met | isp
11 | Jan-10 | Dec-09 | 2009-12-17 00:00:00 | 1 |  |  |  |  |  | CR0184697 |  | aspna_usams_zcenter | Standard | 4 | zCenter Deployment Activity | Event ID:  13861  - (ZNA) Schaumburg, IL Windy Point users experienced slow response times in BPM. Root Cause: Under investigation. Customer Impact: Affected users reported slow response times between screens. Corrective Action (Immediate solution): Issue cleared without intervention while being investigated. 12/18 A tech call identified the need for OPNET software to identify the error. A DMR will be issued for installation Corrective Action (Permanent solution): Under investigation. | Windy Point users experienced slow response in BPM   Additional Findings:  Like the other occurrence this one cleared without intervention. We did manage to capture an http watch during the slowness and the same transaction with normal response. Performance management is looking at the traces.  This issue started early in the day and was not escalated till late in the day.    Working on getting OpNet installed on the [NAME] servers, SHAPZC01 and 02 so we can capture additional traces.   [NAME]:  PIR Complete | 2009-12-18 00:00:00 | 2010-01-01 00:00:00 | 2010-01-07 00:00:00 |  | Not Met | asp
12 | Jan-10 | Dec-09 | 2009-12-21 00:00:00 |  | 1 |  |  |  |  | CR0194645 |  | aspna_crn | Standard | 4 | HERO Rapid Release R9.12 Patch | Event  2 :   (Farmers) Some GR and OKC HelpPoint users were unable to   verify coverages in CRN. Severity Level: 4 Root Cause: HERO [NAME] CR0194645. Customer Impact: Approx. 15 Bristol West, GR, and OKC HelpPoint users were not   receiving "no coverage" alerts and were unable to verify coverages.   Work around was in place, coverages updated after users exited the   claim and re-entered. Corrective Action:  The FACTS team implemented CR0194650 to correct issues in  (Immediate Solution) production regions CICSLS21 and 22. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:32 PT to 14:00 PT Event Duration: From 06:32 PT to 14:00 PT  Ticket Number: IM1708485, IM1708514 | [NAME]: In FACTS system there was a code not put in pertaining to this change for Hero release, and that's what was identified and fixed with CR0194650.     [NAME] PIR Completed | 2009-12-22 00:00:00 | 2010-01-05 00:00:00 | 2009-12-24 00:00:00 |  | Met | asp
13 | Jan-10 | Dec-09 | 2009-12-22 00:00:00 |  | 1 |  |  |  |  | CR0197219 |  | aspna_crn | Emergency | 4 | Service Power Upgrade for Production Issue | Event  11 :   (Farmers) Some KCY and OKC HelpPoint users are unable to   view some reports in ServiceManager. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Approx. 75 users are receiving "Failed to retrieve data from the   database" or "Page Cannot be Displayed" messages when attempting to   view the Job Audit and Capacity reports in ServiceManager. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 11:43 PT to Ongoing Event Duration: From 11:43 PT to Ongoing Ticket Number: IM1709375 | [NAME]: With this change we had vendor side upgrade. This was an isolated issue not as part of service power upgrade.There were some DB connection issues that we figured out and crystal reports were reinstalled to make it functioning.    [NAME] completed | 2009-12-23 00:00:00 | 2010-01-06 00:00:00 | 2010-01-13 00:00:00 |  | Not Met | asp
14 | Jan-10 | Dec-09 | 2009-12-23 00:00:00 |  | 1 |  |  |  |  | CR0195673 |  | aspna_servicepoint | Standard | 4 | Migration of Release 4.12 for ServicePoint and MyFieldPoint to Production. | (farmers) AUS, KCY, and NW Service Point users are unable to convert an Express service request to a full Service request in ServicePoint Siebel. Root Cause: unknown Customer impact: 15 Servicepoint users is receiving "Transaction is a required field" message while trying to convert an Express SR into a full SR within ServicePoint Siebel. Service Outage: From 12:03 PT to 1/8/2010 20:00. | [NAME]  [NAME] completed | 2009-12-24 00:00:00 | 2010-01-07 00:00:00 | 2010-01-27 00:00:00 |  | Not Met | asp
15 | Jan-10 | Jan-10 | 2010-01-07 00:00:00 | 1 |  |  |  |  |  | CR0184697 |  | aspna_usams_zcenter | Standard | 4 | zCenter Deployment Activity | Event  6 :   (ZNA) [NAME], IL BPM Intake users are   experiencing slow response times. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are experiencing long page generations. Corrective Action:  Multiple support groups are working on the issue. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:21 PT to Ongoing Event Duration: From 08:21 PT to Ongoing Ticket Number: INC158282120 | This goes back to release 3.2 (or 3.1 for BPM) that was released in October.  Tables were added without indexes.  Queries against those tables had to search the entire table which took longer.  The first BPM tech call after that was on 12/14, followed by one on 12/15, 12/16, and 12/17 (Event ID:  13861).  There were some issues after that as well.  Once the holiday/vacation time hit at the end of December the load was lower, so the Intake team did not experience the slowness.  [NAME] added additional memory to the buffer pool on SHAPZC06 (DB server) via CR0198897.  That helped a little with the performance.  Back in December, [NAME] made the recommendation to add indexes for certain tables.  This was scheduled to be done with the release on Feb 14, 2010.  It was discovered on 1/7/10 that the tables not being indexed was the main issue.  CR0198910 was scheduled for 1/7/10 at 17:00 PT to create indexes for the recommended tables.  [NAME]: PIR Complete | 2010-01-08 00:00:00 | 2010-01-22 00:00:00 | 2010-01-14 00:00:00 |  | Met | asp
16 | Jan-10 | Jan-10 | 2010-01-11 00:00:00 |  | 1 |  |  |  |  | CR0198642 |  | aspna_e-agent | Standard | 3 | Release 12, 2009 eAuto/eAutoTwo/Express/ePartner off-cycle Deployment | Event  3 :   (Farmers) ServicePoint users experienced several errors  Severity Level: 3 Root Cause: Express 2.0 application deployment. CR198642. Customer Impact: Affected ServicePoint users are unable to make ACA payments, print   ID cards for non-express states and pull policy information for KS, MO and NM in eAgent. Corrective Action:  Work around has been provided for all issues except for Consumer  (Immediate Solution) Solutions credit card payment.  ACA fix has been identified and will be   implemented tonight through changed CR199509. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:30 PT to Ongoing Event Duration: From 06:30 PT to Ongoing Ticket Number: INC158288637, INC158288684, INC158288469 | Additional Findings: Express2 release which added a whole new app for 3 states has affected these three functions of eAgent. A fix was identified for ACA payments. An alternate test ID was identified as an alternate workaround for servicepoint users in consumer solutions to use for printing ID cards.  Root Cause: Express2 release for New Mexico, Missouri, and Kansas.  [NAME]: PIR complete | 2010-01-12 00:00:00 | 2010-01-26 00:00:00 | 2010-01-15 00:00:00 |  | Met | asp
17 | Jan-10 | Jan-10 | 2010-01-13 00:00:00 |  | 1 |  |  |  |  | CR0197975 |  | ispna_distsvc_notes | Minor | 4 | USWNH000 - Domino 8.5.1 Upgrade | Event  6 :   (Farmers) Lotus Notes hub server USWNH000 experienced   sporadic performance issues. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Some Farmers users experienced a delay when receiving or sending email. Corrective Action:  Lotus Notes support recycled host server PADLN2952 and resubmitted  (Immediate Solution) pending emails. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 16:08 PT to 19:19 PT Event Duration: From 16:08 PT to 19:19 PT  Ticket Number: IM1715907 | [NAME]  [NAME]: PIR completed | 2010-01-14 00:00:00 | 2010-01-28 00:00:00 | 2010-02-10 00:00:00 |  | Not Met | isp
18 | Feb-10 | Jan-10 | 2010-01-18 00:00:00 |  | 1 |  |  |  |  | CR0197738 |  | aspna_e-agent | Standard | 4 | [NAME] eAuto/eHome/Express/ePartner Micro-3 Deployment | Event  5 :   (Farmers)  eAgent users are reporting higher than usual   P4XD1 transaction errors. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are receiving the error" P4XD1 DB2 ERR INS NRML-  ACTMRTG" in eHome. These transactions will be recovered   automatically tonight. Corrective Action:  eHome application support has identified the problem and is working  (Immediate Solution) on a permanent  fix. Workaround in place.. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 13:00 PT to Ongoing Event Duration: From 13:00 PT to Ongoing Ticket Number: IM1717246 01/19/10: | Sampath Sivanesan:  What happened was an express1 fix went in last Friday that caused this. When an agent tried to enter a code, the system is triedto update with incorrect effective date. A fix for this has been identified and is currently being tested. Will migrate tonight or tomorrow. Workaround has been given to ASC.    [NAME]: PIR Complete | 2010-01-19 00:00:00 | 2010-02-02 00:00:00 | 2010-01-20 00:00:00 |  | Met | asp
19 | Feb-10 | Jan-10 | 2010-01-19 00:00:00 | 1 |  |  |  |  |  | CR0200075 |  | ispna_distsvc_notes | BAU | 4 | USZICD01 - Decommission the USZICD01 server. | Event  3 :   (ZNA) Lotus Notes server USZICD01 was unavailable. Severity Level: 4 Root Cause: CR0200075 to decommission USZICD01. Customer Impact: Less than 5 users were unable to add or delete forms on the Empire Fire   and Marine Insurance Company website. Corrective Action:  Support powered on USZICD01. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (01/16) 10:00 PT to (01/19) 11:21 PT Event Duration: From (01/16) 10:00 PT to 11:21 PT  Ticket Number: INC158310506 | Per [NAME]. The Application development OK'ed the decom. of server USZICD01 The Notes team followed the Change Control process, submitted and implemented CR0200075 to decom. server USZICD01After the server was decom., users started having issues using an applicationThe application team looked into the issue and determined that the still have applications which have links pointing to server USZICD01The Notes team brought up server USZICD01 online and users were able to access the application. The application team is working on a resolution to update the application to point to a new application server so that server USZICD01 could be decommissioned again in the future.  [NAME]: PIR completed | 2010-01-20 00:00:00 | 2010-02-03 00:00:00 | 2010-01-21 00:00:00 |  | Met | isp
20 | Feb-10 | Jan-10 | 2010-01-21 00:00:00 | 1 |  |  |  |  |  | CR0199222 |  | ispna_dcs_prod_inet | Standard | 4 | [NAME] contains performance related enhancements | Event  6 :   (ZNA) [NAME] users are receiving intermittent errors. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are intermittently unable to login. Corrective Action:  (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:00 PT to Ongoing Event Duration: From 06:00 PT to Ongoing Ticket Number: INC158310843 | There were multiple Crypt32 errors on PACLA1990, however those stopped at 13:40 CT on 1/19.  PACLA1990 is the active node for the Filenet Content Engine.  That server was also having DLL errors.  [NAME] wanted to rule out that the DLL errors were causing it or not.  However, to do this would require a reboot that would affect all users for 5-10 minutes.  After the reboot.  [NAME] did not see any DLL errors and Sajid ran load testing and everything ran normally  [NAME]: [NAME] | 2010-01-22 00:00:00 | 2010-02-05 00:00:00 | 2010-02-02 00:00:00 |  | Met | isp
21 | Feb-10 | Jan-10 | 2010-01-21 00:00:00 | 1 |  |  |  |  |  | CR0173409 |  | aspna_e-agent | Standard | 5 | Release 12, 2009 eAuto/eHome/Express implementation. | (ZNA)  eAgent users are experiencing problems submitting non-bound transactions in eHome. Root Cause: Due to Release 12 migration on 12/29. Customer Impact: User unable to submit non-bound transactions in eHome affecting (20-25) policies. Corrective Action (Immediate solution): 01/22; Workaround is in place and functioning correctly. WAS changes will need to be implemented, currently in testing and schedule to go into production on 01/27. Corrective Action (Permanent solution): Same as immediate. | Sampath Sivanesan:  Workaround is in place and functioning correctly. WAS changes will need to be implemented, currently in testing and schedule to go into production on 01/27.  [NAME]: PIR completed | 2010-01-22 00:00:00 | 2010-02-05 00:00:00 | 2010-02-02 00:00:00 |  | Met | asp
22 | Feb-10 | Jan-10 | 2010-01-22 00:00:00 |  | 1 |  |  |  |  | CR0201363 |  | aspna_facts | BAU | 4 | CMA-FACTS - BCO ADD/CLOSE | Event  7 :   (Farmers) FACTS online was unavailable to CRN and Hero   applications. Severity Level: 4 Root Cause: Application change (CR0201363). Customer Impact: CRN and Hero users were unable to access FACTS onlines. Corrective Action:  Application support backed out change. (Immediate Solution) Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 23:58 PT to (01/23)  02:50 PT Event Duration: From 23:58 PT to (01/23) 02:50 PT  Ticket Number: IM1719521, IM1719519 | Customer follow-up pending from [NAME].   [NAME]:  PIR Complete | 2010-01-25 00:00:00 | 2010-02-08 00:00:00 | 2010-02-17 00:00:00 |  | Not Met | asp
23 | Feb-10 | Jan-10 | 2010-01-23 00:00:00 |  | 1 |  |  |  |  | CR0201648 |  | ispna_distsvc_ops | Standard | 5 | OKC NICE - Application server is being rebuilt (PAVAP3026) | Event  3 :   (Farmers) The okcnice redirect was unavailable. Severity Level: 5 Root Cause: Under investigation. Customer Impact: Approx. 5 users received 'Page Under Construction' messages when   attempting to navigate to NICE using the okcnice redirect. [NAME]was in place; users were able to access the site by entering okcnice/nice. Corrective Action:  Telecom support rebuilt the redirect document and corrected IIS  (Immediate Solution) settings on NICE Perform server PAVAP3026. Corrective Action: Under investigation. (Permanent Solution) Service Outage: None Event Duration: From 06:40 PT to 07:00 PT  Ticket Number: IM1719526 | [NAME]: CR0201648 was opened to rebuild PAVAP3026 to correct an ongoing issue with the Application Server Service that has been plaguing the OKC HelpPoint (Event ID 14010).  [NAME] was rebuild and Nice webaccess was verified using http://pavap3026/nice and http://okcnice/nice, but it was not available for users navigating using okcnice only in their web browser.  The okcnice redirect was created to make things easier for the customer to navigate to the Nice web apps.  The okcnice redirect document and configs were not part of the original install document and therefore not moved by the vendor that performed the rebuild.   So although users were able to navigate to the OKC Nice Web App via the standard links the redirect okcnice shortcut was not available till [NAME] moved the redirect doc and configs.  [NAME]:  PIR complete. | 2010-01-25 00:00:00 | 2010-02-08 00:00:00 | 2010-01-28 00:00:00 |  | Met | isp
24 | Feb-10 | Feb-10 | 2010-02-05 00:00:00 | 1 |  |  |  |  |  | CR0194614 |  | aspna_usams_ddc | Minor | 3 | DDC: Exception #15 - Implement DDC / CC Release 9.3 -  Email Indexing | Event  1 :   (ZNA) DDC users were unable to process TPA claims. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Affected users received an error message. Corrective Action:  Change Control support reconfigured the DDC email account web page (Immediate Solution)  to ASP. NET 1.1. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:51 PT to 11:14 PT Event Duration: From 06:51 PT to 11:14 PT  Ticket Number: INC158352561 | Per Sudhakara Konduru:  This issue was related to the Release 9.3. The issue (Remedy# - 158352561) was DDC [NAME] could not access ASP.NET web services on PAVAP2564 because the DDCEmailAccount web page was configured to use ASP.NET 2.0 ( Release 9.3) and the existing web services use ASP.NET1.1.  When this web page was reconfigured this caused the existing web services to not be able to run as there can only be one version of ASP.NET in an IIS process.  The new web page, DDC [NAME], needs to be in it own process since it uses a different version of ASP.NET.  We have changed the configuration back to 1.1 (via Emergency Change CR0205082) so the existing web services are working fine.  [NAME]:  PIR Complete | 2010-02-08 00:00:00 | 2010-02-22 00:00:00 | 2010-02-19 00:00:00 |  | Met | asp
25 | Feb-10 | Feb-10 | 2010-02-05 00:00:00 |  | 1 |  |  |  |  | CR0202902 |  | ispna_ts_websphere | Major | 3 | For eAgent Production cell WZP,  implementation of WebSphere version 7 into Production | Event  2 :   (Farmers) Agents are receiving error message 'class   java.lang.NullPointerException' in eCMS. Severity Level: 3 Root Cause: Under investigation. Customer Impact: 15 agents are affected. Corrective Action:  eCMS and support group are working on the problem. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 15:06 PT to Ongoing Event Duration: From 15:06 PT to Ongoing Ticket Number: IM1725420 | Per [NAME]:  this issue was related to the WAS 7 upgrade.  A rollback to WAS 6.1 was completed until problem can be resolved.  Currently working with IBM on a resolution.   [NAME]:  PIR complete | 2010-02-08 00:00:00 | 2010-02-22 00:00:00 | 2010-02-24 00:00:00 |  | Not Met | isp
26 | Mar-10 | Feb-10 | 2010-02-14 00:00:00 |  | 1 |  |  |  |  | CR0203505 |  | ispna_distsvc_tech | Standard | 3 | Migrate CRN Siebel 6 Remote Servers OKCNTSB13 & W2SB27 to virtual servers | Event  1 :   (Farmers) The CRN Siebel startup script did not complete   successfully. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Over 50 OKC, KCY, and GR HelpPoint users were unable to utilize   CRN, FNOL, HERO, and the ServicePower Scheduler. Affected CSAs   entered claims via PaperLoss. Corrective Action:  Wintel support re-ran the CRN startup script and recycled  (Immediate Solution) OKCNTCL01 and OKCNTCL03. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:00 PT to 10:43 PT Event Duration: From 06:00 PT to 10:43 PT  Ticket Number: IM1728407 | This issues was triggered by change CR0203505, a migration of 2 Siebel servers (SB13 and SB27) to refreshed virtual platforms. This was triggered by our weekly login script being modified to minimize conflicts between the servers on the change and the others. This was made worse by poor communication and escalation by the production admin, and the issue is being addressed by myself, Wintel Tech, and CRN.  [NAME]:  PIR complete | 2010-02-15 00:00:00 | 2010-03-01 00:00:00 | 2010-02-15 00:00:00 |  | Met | isp
27 | Mar-10 | Feb-10 | 2010-02-14 00:00:00 |  | 1 |  |  |  |  | CR0203505 |  | ispna_distsvc_tech | Standard | 4 | Migrate CRN Siebel 6 Remote Servers OKCNTSB13 & W2SB27 to virtual servers | Event  2 :   (Farmers) [NAME] server OKCW2SC27 was  Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users were unable to Remote Sync to Siebel servers   OKCW2SB26 and 27. Corrective Action:  Wintel support virtualized OKCW2SB27 to VMware host server  (Immediate Solution) PAVMW3046. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:00 PT to (02/15)  03:45 PT Event Duration: From 06:00 PT to (02/15) 03:45 PT  Ticket Number: IM1728408 | Event 2 (Sunday) was due to issues with the original migration. [NAME] was successfully migrated but SB27 failed and the back out plan, going back to the physical server, failed. A workaround was done to bring SB27 back up on it's cluster partner, SB26. However, we did not feel both apps running on a single server would handle our Monday load. As such, since it was the quickest course and since we believed some SAN issues noticed after the failure contributed to the original migration failing, we decided to re-do the SB27 migration. This was successful at 3:30 am PST this morning (performed by Wintel Tech).  [NAME]: PIR completed | 2010-02-15 00:00:00 | 2010-03-01 00:00:00 | 2010-02-15 00:00:00 |  | Met | isp
28 | Mar-10 | Feb-10 | 2010-02-15 00:00:00 |  | 1 |  |  |  |  | CR0195225 |  | aspna_e-agent | Standard | 4 | [NAME] eAuto/eAutoTwo/Express implementation | Event  4 :   (Farmers) Some eAgent users are receiving [NAME]  errors   when creating eAuto VIP quotes. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are unable to complete eAuto VIP quotes. Corrective Action:  Application support is investigating. (Immediate Solution) Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 07:30 PT to Ongoing Event Duration: From 07:30 PT to Ongoing Ticket Number: IM1728519 | [NAME]: Express 1 was released to Texas this weekend. Some Texas users were  sporadically unable to create eAuto VIP quotes.  Also, this was a large release weekend where a number of applications received needed updates and refreshing. Permanent Fix went in on 2/17 8:00PM (PT)  [NAME]: PIR in progress | 2010-02-16 00:00:00 | 2010-03-02 00:00:00 | 2010-02-17 00:00:00 |  | Met | asp
29 | Mar-10 | Feb-10 | 2010-02-15 00:00:00 |  | 1 |  |  |  |  | CR0188105 |  | ispna_dcs_prod_inet | Standard | 4 | Update of WebSphere Plugins for eAgent web server instances in support of the Express Application | Event  5 :   (Farmers) Some eAgent users received 'getstatecode' error in   Auto Express. Severity Level: 4 Root Cause: Under investigation. Root Cause: Weekend release. Customer Impact: Affected users were unable to start new quotes in Auto Express. Number of Users impacted. 15 users are in the ticket. Corrective Action:  Websphere support made configuration changes to WZP08H3 server. (Immediate Solution) Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 07:30 PT to 08:30 PT Event Duration: From 07:30 PT to 08:30 PT  Ticket Number: IM1728500 | New servers were placed in production over the (2/13) weekend in support of an express release. One of the servers was misconfigured so [NAME] fixed the required settings. This was one of four servers, so only 1/4 of users in eAuto would have seen this issue. Additionally, if users got out and tried again, they would likely hit a different server and have no issues. Users that called the helpdesk had their IE settings changed to be inline with previously communicated settings, and then retried successfully.  [NAME]: PRI in progress | 2010-02-16 00:00:00 | 2010-03-02 00:00:00 | 2010-02-22 00:00:00 |  | Met | isp
30 | Mar-10 | Feb-10 | 2010-02-20 00:00:00 |  | 1 |  |  |  |  | CR0206210 |  | ispna_dist_security | BAU | 4 | [NAME] patches per [NAME] reports | Event  1 :   (Farmers) Telecom server PAAPP3063 was unavailable. Severity Level: 4 Root Cause: CR0204861 Customer Impact: Less than 5 OKC WFM users were unable to access CC Pulse. Corrective Action:  PAAPP3063 was recycled after CR0204861 was implemented. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 13:09 PT to 13:12 PT;  From 14:07 PT to 14:11 PT Event Duration: From 13:09 PT to 14:11 PT  Ticket Number: IM1731190 | [NAME]: Patches were applied during SLA hours due to crossing emails. Nothing was backed out and the system was fine after it rebooted.  Its turning out to be a documentation issue  [NAME]: PIR completed | 2010-02-22 00:00:00 | 2010-03-08 00:00:00 | 2010-03-03 00:00:00 |  | Met | isp
31 | Mar-10 | Feb-10 | 2010-02-20 00:00:00 |  | 1 |  |  |  |  | CR0205769 |  | aspna_e-agent | Standard | 5 | Release 4, 2010  Micro-1 eAuto/eAutoTwo/Express implementation | Event  2 :   (Farmers) Farmers customers were unable to make credit card   payments in FAB/eBPP due to a PCI turn on activities from   Release 2. Severity Level: 5 Root Cause: Under investigation. Customer Impact: Customers were unable to process credit card payments for FAB/eBPP.   Customers could call ServicePoints or their Farmers agent to make a   payment. Corrective Action:  Application support will back-out PCI change tonight 02/20 during the (Immediate Solution)  normal maintenance window. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:00 PT to 05:00 PT Event Duration: From 04:00 PT to 05:00 PT  Ticket Number: INC158385711 | [NAME]: The PCI compliance portion of this change had a problem with the SSL Certificate; No customer impacted    [NAME]: PRI completed | 2010-02-22 00:00:00 | 2010-03-08 00:00:00 | 2010-03-02 00:00:00 |  | Met | asp
32 | Mar-10 | Feb-10 | 2010-02-22 00:00:00 | 1 |  |  |  |  |  | CR0203834 |  | ispna_nt_terminal_server | BAU | 4 | SAP GUI v7.10 with Patch 14 | Event  5 :   (ZNA) SAP Print output via Citrix was unavailable. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users were unable to print SAP documents. Corrective Action:  Citrix support redirected the SAP Client system variable from  (Immediate Solution) SAP\FrontEnd62 to SAP\FrontEnd across the Production Citrix farms. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:05 PT to 07:40 PT Event Duration: From 06:05 PT to 07:40 PT  Ticket Number: INC158388737 | [NAME]: A step was missed in the first set of servers upgraded and corrected before any further servers were updated   [NAME]: PIR completed | 2010-02-23 00:00:00 | 2010-03-09 00:00:00 | 2010-03-03 00:00:00 |  | Met | isp
33 | Mar-10 | Feb-10 | 2010-02-22 00:00:00 |  | 1 |  |  |  |  | CR0205758 |  | ispna_ts_cics | Minor | 3 | Upgrade Foremost Production CICS Regions to CICS/TS 3.2. | Event  3 :   (Foremost) Users were unable to change STAR passwords. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Affected users were unable to use self service password reset utilities or   have the STAR Help Desk reset their passwords. Corrective Action:  Operations support recycled CICS region PRDFORA6, allowed backlog (Immediate Solution)  to process, and UNIX support ran the CTG script on GRAPFS03 and  Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:40 PT to 19:52 PT Event Duration: From 08:40 PT to 19:52 PT  Ticket Number: IM1731460 | [NAME]: CICS code change -- more details to follow    [NAME]: PIR completed | 2010-02-23 00:00:00 | 2010-03-09 00:00:00 | 2010-03-10 00:00:00 |  | Not Met | isp
34 | Mar-10 | Feb-10 | 2010-02-26 00:00:00 |  | 1 |  |  |  |  | CR0208232 |  | ispna_distsvc_ops | BAU | 3 | Security updates to production servers | Event  2 :   (Foremost) STAR was sporadically unavailable. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Affected users experienced clocking when attempting to login to   www.foremoststar.com. Corrective Action:  After security patch installation, several Foremost servers completed  (Immediate Solution) reboot cycle. Additionally, [NAME] was recycled. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 16:20 PT to 17:10 PT Event Duration: From 16:20 PT to 17:10 PT  Ticket Number: IM1734140 | [NAME] change was to apply patches to production servers. The servers that were involved were PAMPS1330-1335, which were rebooted as part of the patch process. The patches were pushed out with a implementation time of 1:00 am Saturday 2/27. Problem was, these servers are on GMT, not Eastern time. There fore, when they rebooted it was only 19:00 EST. This caused siteminder and authentication errors in STAR. Once they were back up, 3 of the web instances of STAR had difficulty getting connected with [NAME] again until we restarted them.     [NAME]:  PIR Complete | 2010-03-01 00:00:00 | 2010-03-15 00:00:00 | 2010-03-18 00:00:00 |  | Not Met | isp
35 | Mar-10 | Feb-10 | 2010-02-26 00:00:00 |  | 1 |  |  |  |  | CR0208020 |  | ispna_netwksvc_telecom-w | Minor | 4 | GR: Update IP settings related to Modular Messaging connection | Event  5 :   (Foremost) Some users were unable to access voicemail. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected Foremost users were unable to retrieve voicemail messages. Corrective Action:  Backed out changes introduced by vendor, Avaya, on PBX server  (Immediate Solution) MIGRR-CX003. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:00 PT to 06:00 PT Event Duration: From 05:00 PT to 06:00 PT  Ticket Number: IM1733605 | [NAME] problem was caused by changes that were made by vendor, Avaya.  Changes put in by vendor disabled connectivity to Modular Messaging System.  The changes were backed out.  [NAME]: PIR Complete | 2010-03-01 00:00:00 | 2010-03-15 00:00:00 | 2010-03-10 00:00:00 |  | Met | isp
36 | Mar-10 | Feb-10 | 2010-02-26 00:00:00 |  | 1 |  |  |  |  | CR0208247 |  | ispna_ts_cics | BAU | 4 | BAU for CICS for 2/26/10 thru 3/04/10 For Farmers | Event  8 :   (Farmers) eAgent users experienced slow response in Agency   Dashboard. Severity Level: 4 Root Cause: PLS CICS regions received multiple ABM3 abends. Customer Impact: Up to 13 users experienced delays navigating through screens in Personal   Lines and Express. Corrective Action:  CICS support investigating issue. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 15:45 PT to 15:55 PT Event Duration: From 15:45 PT to 15:55 PT  Ticket Number: IM1734109 | [NAME] - The team was in the process of building new regions at the request of Capacity Planning and one of the regions was inadvertantly started too soon.   [NAME]:   PIR Complete | 2010-03-01 00:00:00 | 2010-03-15 00:00:00 | 2010-03-10 00:00:00 |  | Met | isp
37 | Mar-10 | Mar-10 | 2010-03-02 00:00:00 |  | 1 |  |  |  |  | Bristol West Change # U6784 |  | Network (Bristol West) | No Change | 4 | Moving the Mellon VPN tunnel to the Davie ASA from the Ohio ASA. | Event  1 :   (Farmers) Bristol West experienced network issues. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Approx. 50 HelpPoint users were unable to access CUDA and PSP. Corrective Action:  Bristol West Network support corrected issue on the BW network  (Immediate Solution) infrastructure. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:45 PT to 08:10 PT Event Duration: From 07:45 PT to 08:10 PT  Ticket Number: IM1734975 | [NAME]:  Our network team was performing the planned change, U6784, on our WAN router.  The change was not successful and therefore was backed out.  In the process of backing out the change, the network technician inadvertently removed configuration necessary for the remote sites to connect to the WAN.  This configuration was corrected and sites were then able to communicate successfully.  [NAME]:  PIR Complete | 2010-03-03 00:00:00 | 2010-03-17 00:00:00 | 2010-03-17 00:00:00 |  | Met | Net
38 | Mar-10 | Mar-10 | 2010-03-03 00:00:00 | 1 |  |  |  |  |  | CR0203729 |  | ispna_dist_security | Standard | 2 | Approve Microsoft patches issued in Feb for Phase 4 to conform to G-TIR requirement - | Event  1 :   (ZNA) Risk Intelligence Express was intermittently                   unavailable. Severity Level: 2 Root Cause: Under investigation. Customer Impact: Users were sporadically receiving 'Service Unavailable' when logging in. Corrective Action:  Wintel support stopped IIS services on PAVAP0009. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:30 PT to 08:40 PT Event Duration: From 07:30 PT to 08:40 PT  Ticket Number: INC158412939 | [NAME] was a security patch delivered overnight Monday night / Tuesday morning to pavap0009 and during that process the application pool became corrupted on that server and IIS had stopped working. So when IIS was stopped on pavap0008 pavap0009 could not pick up the load causing the full outage. IIS has been disabled on pavap0009 which keeps it out of the F5 pool. Currently we are running on a single application server.  Fix to go in today with server being added back to F5 pool tonight.  [NAME]: PIR Complete | 2010-03-04 00:00:00 | 2010-03-18 00:00:00 | 2010-03-16 00:00:00 |  | Met | isp
39 | Mar-10 | Mar-10 | 2010-03-08 00:00:00 |  | 1 |  |  |  |  | CR0208500 |  | aspna_usams_zcenter | BAU | 3 | Deploy efile v4.0   on 03\05 at 7:00 PM CST | Event  2 :   (ZNA) issue with eFile EI cases not updating in eFile has been   resolved. Severity Level: 3 Root Cause:  CR0208500--Code issue was not discovered during testing of this weekends change Customer Impact: Affected users were unable to access documents associated with new   cases. Corrective Action:  eFile support applied code changes to  (Immediate Solution) PAWEB3152,PAWEB3153,PAWEB3437 and PAWEB3759. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:32 PT to 09:03 PT Event Duration: From 07:32 PT to 09:03 PT  Ticket Number: EU-INC158422092 | Follow-up:  Wintel - opened Microsoft Case # 110030368459572 to look at the debug logs -  Ramana [NAME] will open an Emergency change record with code change applied today.   [NAME]: PIR complete. | 2010-03-09 00:00:00 | 2010-03-23 00:00:00 | 2010-03-17 00:00:00 |  | Met | asp
40 | Mar-10 | Mar-10 | 2010-03-08 00:00:00 |  | 1 |  |  |  |  | CR0200561 |  | ispna_firewall | Major | 4 | Upgrade Grand Rapids Firewall Clusters to new Nokia IP560 | Event  10 :   (Farmers) Xactimate users experienced sporadic connection  Severity Level: 4 Root Cause: CR200561. Customer Impact: Less than 10 HelpPoint users received "A communication error has   occurred." messages when trying to send or receive with XactNet. Corrective Action:  The firewall policy was updated for the appropriate TCP ports for the  (Immediate Solution) connection. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:34 PT to 09:08 PT Event Duration: From 08:34 PT to 09:08 PT  Ticket Number: IM1737332 | [NAME]: The firewall policy was updated for the appropriate TCP ports for the connection.   [NAME]: PIR complete: | 2010-03-09 00:00:00 | 2010-03-23 00:00:00 | 2010-03-17 00:00:00 |  | Met | isp
41 | Mar-10 | Mar-10 | 2010-03-12 00:00:00 |  | 1 |  |  |  |  | CR0209997 |  | ispna_distsvc_storage-dist | BAU | 4 | Weekly BAU for Disributed Storage Open Systems -  03/08/2010 - 03/15/2010. | Event  6 :   (Farmers/Foremost) Lotus Notes server USWNS103 was     unavailable. Severity Level: 4 Root Cause: CR0209997. Customer Impact: [NAME] was in place; however, some users experienced issues accessing   their replica server. Corrective Action:  Wintel support scanned for SAN drives on host server PULTN0001 and  (Immediate Solution) restarted the Domino Server service. For the second occurrence, Wintel   support ran "chkdsk" on the SAN attached drives and then restarted   Domino on PULTN0001. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:37 PT to 08:12 PT;  From 09:20 PT to 09:50 PT Event Duration: From 07:37 PT to 09:50 PT  Ticket Number: IM1739395, INC158434810 | [NAME]: SAN team was doing SAN switch which caused the server to crash.We were alerted on our monitoring software that the Lotus Notes server had crashed and when we logged into the server to see what happened, we saw that the server had lost the SAN drives.  [NAME]: PIR in progress | 2010-03-15 00:00:00 | 2010-03-29 00:00:00 | 2010-03-16 00:00:00 |  | Met | isp
42 | Mar-10 | Mar-10 | 2010-03-15 00:00:00 |  | 1 |  |  |  |  | CR0209971 |  | ispna_dcs_prod_inet | Standard | 3 | eUmbrella February Release | Event  1 :   (Farmers) Kansas City ServicePoint users are receiving   javascript errors in eUmbrella application Severity Level: 3 Root Cause: Under investigation. Customer Impact: Agents and KCSU employees are not able to process new business in   eUmbrella. Corrective Action:  [NAME] release implemented over the weekend will be back out  (Immediate Solution) tonight CR0209971 at 20:00 PT. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (03/16) 08:00 PT to  Ongoing Event Duration: From (03/16) 08:00 PT to Ongoing Ticket Number: INC158438106 | Everything tested fine during the shake-out on Sunday. It was noted that the same EAR file was in QA and PROD, but QA did not have the issue.  After some time, Nerandra noticed that he was getting the same error in QA.    The decision was made to back out the [NAME] release (CR0209971) after hours.  The backout was complete at 20:24 PT and it was reported that the app was functioning properly  [NAME] in [NAME] | 2010-03-16 00:00:00 | 2010-03-30 00:00:00 | 2010-03-16 00:00:00 |  | Met | isp
43 | Mar-10 | Mar-10 | 2010-03-15 00:00:00 |  | 1 |  |  |  |  | CR0208588 |  | aspna_e-agent | Standard | 4 | Release 4, 2010  Micro-3 eAuto/eAutoTwo/Express implementation | Event  4 :   (Farmers) Farmers eAgent users are receiving a null pointer   exception error when trying to print eHome LOB documents. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are unable to print listed documents. Corrective Action:  Workaround has been provided and validated by ASC.. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:10 PT to Ongoing Event Duration: From 09:10 PT to Ongoing Ticket Number: IM1739920 | Sampath S - On 03/15/2010, eHome changes went into production as part of peregrine # CR0208588 (03/13 Home offcycle migration (Rel-4.0 Micro-3) caused this issue.  Currently we have identified the root cause and a permanent fix will be migrated tonight to production as part of peregrine # CR0211450.  We have provided a Workaround to ASC and Brett confirmed that our workaround is working fine without any issues.   [NAME] | 2010-03-16 00:00:00 | 2010-03-30 00:00:00 | 2010-03-16 00:00:00 |  | Met | asp
44 | Apr-10 | Mar-10 | 2010-03-22 00:00:00 |  | 1 |  |  |  |  | CR0212422 |  | ispna_netwksvc_telecom-w | BAU | 4 | Grand Rapids Weekly BAU | Event  2 :   (Foremost) The Star Helpdesk was unable to receive inbound calls. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Calls for approx. 23 Helpdesk agents were misrouted to a greeting   message. Corrective Action:  Telecom support changed the vector assignments for VDNs 20258 and  (Immediate Solution) 20528. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:00 PT to 06:31 PT Event Duration: From 06:00 PT to 06:31 PT  Ticket Number: IM1742707 | [NAME]: Vendor implemented bad code  [NAME]: PIR completed | 2010-03-23 00:00:00 | 2010-04-06 00:00:00 | 2010-03-29 00:00:00 |  | Met | isp
45 | Apr-10 | Apr-10 | 2010-04-01 00:00:00 |  | 1 |  |  |  |  | CR0212087 | N | ispna_ts_websphere | Major | 4 | For eAgent Production cell WZP,  re-introduce WebSphere version 7 into Production | Event  2 :   (Farmers) Agents were experiencing slow response in eAgent. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Up to 19 users reported slow response while navigating in ADE. Corrective Action:  Support performed some workload balancing and started another  (Immediate Solution) instance for server WZP02H4 on PRH1. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:05 PT to 09:50 PT Event Duration: From 09:05 PT to 09:50 PT  Ticket Number: IM1747343 | Per [NAME] - a configuration change made under WAS 6.1 was overlooked when we moved to WAS 7.0.  An attempt was made to correct the configuration, which involved bouncing 2 servers.  It was bouncing the servers that actually caused the slowdown issue. | 2010-04-02 00:00:00 | 2010-04-16 00:00:00 | 2010-04-06 00:00:00 | N | Met | isp
46 | Apr-10 | Apr-10 | 2010-04-02 00:00:00 | 1 |  |  |  |  |  | None |  | ispna_dcs_facilities_sch | No Change | 2 | n/a | Event  1 :   (ZNA) Dallas, [NAME] was unavailable. Severity Level: 2 Root Cause: Under investigation. Customer Impact: Up to 320 users were unable to utilize eZACCESS, ACCESS, Lotus   Notes, Internet, [NAME], or Network Shares and [NAME]. Farmers   users at the same address reported no issues. Corrective Action:  On-site support restored power to the UPS, and gracefully power  (Immediate Solution) cycled TXDAL1-AR001 and TXDAL1-AR002. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:17 PT to 10:52 PT Event Duration: From 10:17 PT to 10:52 PT  Ticket Number: INC158477810 | [NAME] [NAME] was cut to the UPS to install a monitoring card. This meant no city power was able to pass through the UPS, which took power away from the wintel servers, GAD server, and the network equipment. No CR was created | 2010-04-05 00:00:00 | 2010-04-19 00:00:00 | 2010-04-14 00:00:00 | Y | Met | isp
47 | Apr-10 | Apr-10 | 2010-04-04 00:00:00 |  | 1 |  |  |  |  | CR0214433 |  | ispna_distsvc_tech | Standard | 5 | Migrate CRN Siebel 6 [NAME] to virtual servers | Event  1 :   (Farmers) Remote Sync is delayed for some remote CRN users. Severity Level: 5 Root Cause: Under investigation. Customer Impact: None reported. Server-side CRN transactions routed to remote clients   are unavailable for some remote users. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: None Event Duration: From 06:00 PT to Ongoing Ticket Number: IM1748110 | [NAME] reported that this incident was change related. Servers were being migrated from pjysical to virtual.    [NAME]:  PIR complete | 2010-04-05 00:00:00 | 2010-04-19 00:00:00 | 2010-04-15 00:00:00 | N | Met | isp
48 | Apr-10 | Apr-10 | 2010-04-08 00:00:00 |  | 1 |  |  |  |  | None |  | aspna_foremost | No Change | 4 | PanAp change | Event  2 :   (Foremost) Issue with some Foremost Focus users receiving   abend messages and freezing of sessions. Severity Level: 4 Root Cause: Under investigation. Customer Impact: 5 users experienced abends in Focus. Corrective Action (Immediate solution): Application support performed a program newcopy in CICS region CIC02. 04/09; RQ01 application change #158313069 was backed out. (Permanent Solution)Under investigation. Service Outage: From 08:00 PT to 09:50 PT Event Duration: From 08:00 PT to 09:50 PT  Ticket Number: INC158488104 | [NAME] is working to identify if a CR was done, and if not, who the  responsible manager is.  According to [NAME], there were some groups at one point that utilized PanAp move numbers instead of CR records. This may have been one of those changes, which is why he is trying to locate the manager. 04/16/10: In today's CAB meeting [NAME] said he is still trying to find who performed the change and who the manager is. He will let us know as soon as he finds out. 04/26/10: Sent email to [NAME] for status. 05/03/10: Called [NAME] and left a message for status.  [NAME] indicated that this was a PanAp change which does not require a CR. The group responsible for this change was ASPNA_Formost, manager [NAME]. I will contact mgr to start PIR.  PIR Meeting invitation sent to [NAME] on 5/5/10. 05/10/10: [NAME] informed [NAME] that [NAME] is the Manager for Release and Change Management.  05/11/10: PIR email and PIR meeting invitation sent to [NAME]. [NAME] scheduled for 5/18.  [NAME]: PIR Completed 5/18/10 | 2010-04-09 00:00:00 | 2010-04-23 00:00:00 | 2020-05-18 00:00:00 |  | Not Met | asp
49 | May-10 | Apr-10 | 2010-04-14 00:00:00 |  | 1 |  |  |  |  | CR0215100 | N | ispna_distsvc_storage-dist | BAU | 3 | (BAU)  Weekly BAU for Disributed Storage Open Systems -  04/05/2010 - 04/12/2010. | Event  2 :   (Farmers) Some CRN transactions were delayed. Severity Level: 3 Root Cause: Under investigation. Customer Impact: KCY, GR and OKC HelpPoint users were sporadically experiencing   clocking, pending transactions, coverage stuck in pending, and delays   in pulling policy information. Users were required to take claims in   Paper Loss during CRN recycle from 02:15PT - 04:09PT. Corrective Action:  Recycled Siebel DB server OKCAXDB01 and the CRN environment. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 19:17 PT to (04/15)  04:09 PT Event Duration: From 19:17 PT to (04/15) 04:09 PT  Ticket Number: IM1752763 | OS Micro Code change was implemented and that is what triggered it.  This change had multiple come backs and CR was confirmed and received on 5/7.  PIR Completed - [NAME] | 2010-05-07 00:00:00 | 2010-05-21 00:00:00 | 2010-05-18 00:00:00 | N | Met | isp
50 | May-10 | Apr-10 | 2010-04-19 00:00:00 | 1 |  |  |  |  |  | CR0215984 | Y | ispna_distsvc_notes | Major | 2 | Upgrade USUNS800 to [NAME] 8.5.1 | Event  1 :   (ZNA) Direct Underwriters Lotus Notes is partially unvailable. Severity Level: 2 Root Cause: Under investigation. Customer Impact: Affected users being sporadically unable to access email and various   Notes application databases is tentatively believed to have been   mitigated as of 09:02 PT. Corrective Action:  Lotus Notes support deployed an IBM hotfix to both affected servers,  (Immediate Solution) completed at 09:02 PT. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:20 PT to 10:29 PT Event Duration: From 06:20 PT to 10:29 | [NAME]: Server upgrade to both servers (one server in each change (entries 5 & 6) caused a memory leak   [NAME]: PIR completed | 2010-04-20 00:00:00 | 2010-05-04 00:00:00 | 2010-04-26 00:00:00 | N | Met | isp
51 | May-10 | Apr-10 | 2010-04-19 00:00:00 | 1 |  |  |  |  |  | CR0216500 | Y | ispna_distsvc_notes | Minor | 2 | Upgrade USUNS802 to [NAME] 8.5.1 | Event  1 :   (ZNA) Direct Underwriters Lotus Notes is partially unvailable. Severity Level: 2 Root Cause: Under investigation. Customer Impact: Affected users being sporadically unable to access email and various   Notes application databases is tentatively believed to have been   mitigated as of 09:02 PT. Corrective Action:  Lotus Notes support deployed an IBM hotfix to both affected servers,  (Immediate Solution) completed at 09:02 PT. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:20 PT to 10:29 PT Event Duration: From 06:20 PT to 10:29 | [NAME]: Server upgrade to both servers (one server in each change (entries 5 & 6) caused a memory leak   [NAME]: PIR completed | 2010-04-20 00:00:00 | 2010-05-04 00:00:00 | 2010-04-26 00:00:00 | N | Met | isp
52 | May-10 | Apr-10 | 2010-04-21 00:00:00 | 1 |  |  |  |  |  | CR0217470 | N | ispna_distsvc_tech | Minor | 1 | PACLA1459  + PACLA1460Server remediation for new SAN | Event  1 :   (ZNA) VERITAS cluster PACLG1635 lost connectivity to SAN. Severity Level: 1 Root Cause: Outage exposed by CR217470 and flag not set on DMX. Customer Impact: Over 1000 users were unable to save or retrieve information from the   SAN. Issue affected processing with Lex, [NAME]'s Comp, BO,   Claims and Risk Intelligence. Corrective Action:  Support reset the SAN flag SPC2BIT then rebooted PACLA1459 and  (Immediate Solution) PACLA1460. CR0217470 has been created for a permanent fix over   the weekend.  04/22; Wintel/SAN are following up with documenting   procedures to set the flags when servers are refreshed. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 15:10 PT to 17:06 PT Event Duration: From 15:10 PT to 17:06 PT  Ticket Number: INC158516839 | [NAME]: Flag was not set after the update, causing high IO causes connectivity issues during peak volume.  Some documentation also out of date  [NAME]PIR in progress  04/26: Scheduled PIR date with [NAME] and [NAME] -- received an email  Scheduling conflicts cause this PIR to be completed on 05/04 | 2010-04-22 00:00:00 | 2010-05-06 00:00:00 | 2010-05-05 00:00:00 | N | Met | isp
53 | May-10 | Apr-10 | 2010-04-26 00:00:00 |  | 1 |  |  |  |  | CR0218032 | N | ispna_ts_os390 | Standard | 3 | Assign 65 gig of memory to large page support on PRG1 and PRH1 | Event  1 :   (Farmers) eAgent users experienced slow response. Severity Level: 3 Root Cause: CR0218032 to increase large page support from 36 GB to 65 GB. Customer Impact: Affected users experienced slow response and others could not access   the application. Corrective Action:  MVS support added a total of 12 DASD volumes as paging storage, 4  (Immediate Solution) volumes to the G1 system and 8 volumes to the H1 system.   Websphere support change server configuration to the H1 servers. Corrective Action: Support will IPL the G1 and H1 systems tonight to backout the change (Permanent Solution)  implemented over the weekend. CR219641. Service Outage: From 07:30 PT to 08:10 PT  The H1 system ran out of paging   volumes.;  From 09:10 PT to 10:00 PT  The G1 system ran out of   paging storage Event Duration: From 07:30 PT to ongoing Ticket Number: IM1756711 | [NAME]:   Implementation of large page support negatively impacted performance because there is an IBM bug as relates to large page support.  1 MB pages were not backed by 4k pages causing us to go short on storage and excessively page.  We were able to remediate the problem somewhat by adding additional paging volumes, thus limiting user impact.  IBM is aware of the problem and is working on a fix scheduled to be out next month.   [NAME]:  PIR completed | 2010-04-27 00:00:00 | 2010-05-11 00:00:00 | 2010-05-10 00:00:00 | N | Met | isp
54 | May-10 | Apr-10 | 2010-04-27 00:00:00 |  | 1 |  |  |  |  | CR0219361 | N | ispna_netwksvc_support | BAU | 3 | Meta IP [NAME] | Event  4 :   (ZNA)  SIG, Athena and Gracie applications were                    unavailable. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Users already in the applications were not affected, however users just   logging in received a '404' error when attempting to access the Web   sites. Up to 50 users affected. Corrective Action:  Network Operations reverted a DNS change (DMR93953). (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 13:40 PT to 15:10 PT Event Duration: From 13:40 PT to 15:10 PT  Ticket Number: INC158530590 | [NAME]:  A DNS change was completed that was time sensitive although the DMR did not indicate the timing of the DNS change and it should have been completed this Friday evening in coodination with application changes.   [NAME]:  PIR completed | 2010-04-28 00:00:00 | 2010-05-12 00:00:00 | 2010-05-10 00:00:00 | N | Met | isp
55 | May-10 | Apr-10 | 2010-04-27 00:00:00 |  | 1 |  |  |  |  | CR0219365 | N | ispna_netwksvc_telecom-w | BAU | 4 | OKC and KCY: Weekly BAU Changes (4/24/10 to 5/01/10) | Event  6 :   (Farmers) Some KCY and GR HelpPoint users experienced   sporadic issues when transferring calls. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Less than 20 users experienced hold music playing throughout calls   after a warm transfer had completed. Corrective Action:  Telecom support modified call forwarding parameters on the PBX. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 11:45 PT to 15:28 PT Event Duration: From 11:45 PT to 15:28 PT  Ticket Number: IM1757454 | [NAME]:  To correct an issue related to IM1757093 (Select OCR Supervisors call forwarding of their agent login was not forwarding to the call forward destination, but instead was covering to voicemail). a call forwarding change was made to allow an agent ID to forward to the call forward target destination of the physical phone the supervisors log into.  With this change the Supervisors agent ID would forward to the appropriate destination, but if the forwarded call did not get answered, the call would cover to the call coverage destination of the physical phone and not the call coverage destination of the agent ID.  The physical phones that these Supervisors log into are setup to cover to the ACD queue because they are not published numbers.  So the forwarded call ended up in the ACD queue when the call was not answered.  To correct this issue we changed the setting to allow the call forwarded call to cover to the "principal" coverage of the call.  This setting allowed the coverage to follow the coverage of the original agent ID that was called (voicemail).  The change that caused the issue was allowing these calls to be forwarded via physical extensions.  We normally do not allow agent IDs to be forwarded, but business had a need for the Supervisors to be able to have call forwarding turned on to allow other supervisors to cover for these Sups while they are unavailable.   [NAME]:  PIR completed | 2010-04-28 00:00:00 | 2010-05-12 00:00:00 | 2010-05-10 00:00:00 | Y | Met | isp
56 | May-10 | May-10 | 2010-05-09 00:00:00 |  | 1 |  | 1 |  |  | CR0218092 | N | ispna_distsvc_unix | Standard | 2 | OKC OS upgrades AIX 5.3.0.10 | Event  1 :   (Farmers) CRN was unavailable for the OKC, KCY, and GR   HelpPoints. Severity Level: 2 Root Cause: Under investigation. Customer Impact: Affected users were unable to utilize CRN. Claims were entered via   PaperLoss during outage; however, customer callbacks are required on   affected claims. ReportNet ETL jobs are delayed. Reserves, payments,   and other CRN-based transactions may be in a "Pending" status until   backlog clears. Corrective Action:  Applied APARs IZ63808, IZ65095, and IZ65325; recycled CRN  (Immediate Solution) database server OKCAXDB01, and brought CRN online. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:00 PT to 15:46 PT Event Duration: From 06:00 PT to 15:46 PT  Ticket Number: IM1761366 | The issue was due to upgrade in software - (code break) These upgrades were in preparation for the next months OKC move.  PIR Completed - [NAME] | 2010-05-10 00:00:00 | 2010-05-24 00:00:00 | 2010-05-13 00:00:00 | N | Met | isp
57 | May-10 | May-10 | 2010-05-10 00:00:00 |  | 1 |  |  | 1 |  | CR0222224 | N | ispna_firewall | Minor | 4 | Farmers OKC extranet firewall:Failover to backup firewall : | Event  3 :   (Farmers) XM was unavailable through CRN. Severity Level: 4 Root Cause: CR0179817 and CR0222224. Customer Impact: Approx. 25 users were unable to view injury evaluation via CRN. Corrective Action:  [NAME] support modified the firewall rule to allow Vatti-Manhattan IP (Immediate Solution)  access on OKCNKFW04. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:36 PT to 07:44 PT Event Duration: From 06:36 PT to 07:44 PT  Ticket Number: IM1761429 | Per [NAME] The original CR0179817 was a request to add IP's to the existing network connection.  This required firewall rules.  These rules as per this request were added to the firewall.  So, two rules were required, one to allow access through the firewall via the ports requested.  A second was also needed to prevent blocking on the firewall for spoofing (someone erroneously trying to use that IP to gain access to the internal network).    In  CR0222224, the backup or secondary firewall was made primary this past weekend.  When that was done, the spoofing rule was not in place causing access issues.  PIR completed | 2010-05-11 00:00:00 | 2010-05-25 00:00:00 | 2010-05-20 00:00:00 | N | Met | isp
58 | May-10 | May-10 | 2010-05-11 00:00:00 | 1 |  |  |  |  | 1 | CR0221117 | N | ispna_nt_terminal_server | BAU | 3 | Citrix - Install MS hotfixes to address vulnerabilities on Schaumburg Citrix servers | Event  1 :   ([NAME] was unavailable via Citrix. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Affected users received a "Runtime Errors MESSAGE_TYPE_X' error   message when logging in. Corrective Action:  Citrix support reapplied the SAP Gui and patches to the Citrix Farm. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (05/10) 05:18 PT to (05/10) 22:30 PT Event Duration: From (05/10) 05:18 PT to (05/10) 22:30 PT  Ticket Number: INC158557642 | Per [NAME] Citrix team installed several patches on Citrix Test\Dev\QA servers to address vulnerabilities identified by Qualsys scans.    PIR completed - [NAME] | 2010-05-12 00:00:00 | 2010-05-26 00:00:00 | 2010-05-17 00:00:00 | N | Met | isp
59 | Jun-10 | May-10 | 2010-05-17 00:00:00 | 1 |  |  |  |  | 1 | CR0223103 | N | aspna_usams_cms | Minor | 4 | Bulk Screener Program Changes for Access to External Users | Event  8 :   (ZNA) [NAME] was unavailable. Severity Level: 4 Root Cause:  CMS support is bringing work initiated under CR0223103, which is   running long, to completion for the production environment. Customer Impact: Affected users were unable to verify input on the Bulk Rating screen. Corrective Action:  Changes from CR0223103 were backed out by the WebSphere  (Immediate Solution) mainframe support. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:15 PT to 10:18 PT Event Duration: From 06:15 PT to 10:18 PT  Ticket Number: INC158573068 | Shreekan reports this event was caused by a code error which was backed out   [NAME]: PIR completed | 2010-05-18 00:00:00 | 2010-06-01 00:00:00 | 2010-05-25 00:00:00 | N | Met | asp
60 | Jun-10 | May-10 | 2010-05-18 00:00:00 | 1 |  |  |  | 1 |  | P6053 |  | Bristol West Network | No Change | 3 | No ServiceCenter change record | Event  1 :   (ZNA) Network was exhibiting limited connectivity between   North America and Europe. Severity Level: 3 Root Cause: Under investigation. Customer Impact: Mail routing to and from North and South American Lotus Notes   servers to European Servers was unavailable. A limited number of   Sametime users were unable to use chat and web collaboration features. Corrective Action:  Bristol West network support removed an independent statement in  (Immediate Solution) the BGP configuration, which was advertising to a legacy Bristol West   MPLS cloud segment. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:24 PT to 13:30 PT Event Duration: From 10:24 PT to 13:30 PT  Ticket Number: ORANGE 7994330 | [NAME]: The main issue appeared to be a bug on a cisco 7200 router that had an access list which had an implied permit " any any " at the end, but it should have been include a "denied" statement.    [NAME]: [NAME] | 2010-05-19 00:00:00 | 2010-06-02 00:00:00 | 2010-05-26 00:00:00 | Y | Met | [NAME]
61 | Jun-10 | May-10 | 2010-05-19 00:00:00 | 1 |  |  |  |  | 1 | CR0224139 | N | ispna_distsvc_ops | Emergency | 3 | Reboot USZZ1SCHAPP30 - due to hung MQ service | Event  1 :   (ZNA) USZZ1SCHAPP30 was unavailable. Severity Level: 3 Root Cause: Under investigation. Customer Impact: UQA, GCAPS, and D&B were unavailable, and Relay, GCPW, Quartz,   Phoenix, Neptune, Athena, DocPro and Zeuss had reduced  Corrective Action:  Wintel support rebooted USZZ1SCHAPP30 to restart the hung MQ  (Immediate Solution) General Request process. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:09 PT to 06:45 PT Event Duration: From 06:09 PT to 06:45 PT  Ticket Number: INC158569986, INC158579377 | [NAME] he way the change is related to the cause is the time of day the server was rebooted.   [NAME]: PIR completed | 2010-05-20 00:00:00 | 2010-06-03 00:00:00 | 2010-05-26 00:00:00 | N | Met | isp
62 | Jun-10 | May-10 | 2010-05-28 00:00:00 | 1 |  |  |  | 1 |  | None | N | Unknown Bristol West Change | No Change | 4 | Unknown Bristol  West | Event  4 :   (Farmers) Bristol West experienced network issues. Severity Level: 4 LSA Impacted: Root Cause: Under investigation. Customer Impact: Approx. 50 HelpPoint users were unable to access CUDA and PSP.   The BW Davie, FL office was without internet connectivity. Corrective Action:  Bristol West corrected an issue within their network. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:17 PT to 11:17 PT Event Duration: From 10:17 PT to 11:17 PT  Ticket Number: IM1768823 | 6/1: [NAME] indicated that Bristol West made a change over the weekend. [NAME] will follow up on this to provide what was changed and who perfromed the change. She indicated that the techs who handled this are busy this week with the data center move but will get someone to work with us on this hopefully no later than next week.  [NAME]: PIR completed | 2010-06-01 00:00:00 | 2010-06-15 00:00:00 | 2010-07-02 00:00:00 | Y | Not Met | Unk
63 | Jun-10 | Jun-10 | 2010-06-01 00:00:00 |  | 1 | 1 |  | 1 | 1 | CR0226565 | N | ispna_distsvc_tech | BAU | 4 | BAU - VMware activities week of 05/31/2010 | Event  4 :   (Foremost/Farmers) http://www.foremost.com was unavailable. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Agents received "Page cannot be displayed" messages when attempting   to launch the Specialty Dwelling URL within eAgent. Corrective Action:  Wintel support moved foremost.com to VM Host ZRVMW1987. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 17:38 PT to 18:29 PT Event Duration: From 17:38 PT to 18:29 PT  Ticket Number: IM1769640 | [NAME]: Server was vmotioned from one guest to another per change CR0226565.  [NAME] Completed | 2010-06-02 00:00:00 | 2010-06-16 00:00:00 | 2010-06-09 00:00:00 | N | Met | isp
64 | Jun-10 | Jun-10 | 2010-06-03 00:00:00 |  | 1 |  | 1 | 1 |  | CR0226529 | y | ispna_firewall | Major | 4 | LA & OKC firewall - Block unrestricted Internet access for http & https on the firewall per GITR request - HIGH LEVEL IMPACT | Event  3 :   (Farmers) Some HelpPoint and VPN users were unable to   access external internet web sites. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: An undetermined number of Farmers users were denied internet access   due to incorrect proxy settings. Corrective Action:  [NAME] support backed out CR0226529 that blocked unrestricted  (Immediate Solution) access for http and https traffic. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:49 PT to 05:55 PT Event Duration: From 04:49 PT to 05:55 PT  Ticket Number: IM1770163 | [NAME] indicated [NAME] support backed out CR0226529 that blocked unrestricted access.   [NAME]: PIR completed | 2010-06-04 00:00:00 | 2010-06-18 00:00:00 | 2010-06-10 00:00:00 | N | Met | isp
65 | Jun-10 | Jun-10 | 2010-06-06 00:00:00 |  | 1 |  | 1 |  |  | CR0223014              CR0227333 | Y              Y | [NAME]ispna_ts_mq | Standard | 4 | DB2 Database migration              FIG MF MQ & CICS - move CLAIMS functionality in CICSLS21 & CICSLS22 to the CMA regions CICSLA51 & CICSLA52, | Event  2 :   (Farmers) Some payments are not being submitted through  Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Affected users attempting to submit payments are receiving an "MQ   BACKOUT LOOP DETECTED" message in the transaction log. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 16:26 PT to Ongoing Event Duration: From 16:26 PT to Ongoing Ticket Number: IM1771136 | Moved workeload in ls21 and 22 to la51 and 52 among other changes. No connection between la51 and la52 regions is cause la51 and [NAME] caused the issue they set up a connection to resolve the issue. Combined some regions over the weekend.   [NAME] PIR completed 6/10- CR0227333 PIR completed 6/11 | 2010-06-07 00:00:00 | 2010-06-21 00:00:00 | 2010-06-10 00:00:00 | N | Met | asp
66 | Jun-10 | Jun-10 | 2010-06-06 00:00:00 |  | 1 |  | 1 |  |  | CR0223014              CR0227333 | Y              Y | [NAME]ispna_ts_mq | Standard | 4 | DB2 Database migration              FIG MF MQ & CICS - move CLAIMS functionality in CICSLS21 & CICSLS22 to the CMA regions CICSLA51 & CICSLA52, | Event  4 :   (Farmers) Some Bristol West coverages were erroring out. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Approx. 20 Bristol West policies showed an "MQ PUT INTO POINT   Q FAILED" message in the transaction log. Corrective Action:  Added the transmit queue CRNP3P2P to the MQJ1 channel. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 18:20 PT to 18:54 PT Event Duration: From 18:20 PT to 18:54 PT  Ticket Number: IM1771143 | Both MF MQ and CRN MQ made changes in support of the fixing the problem.   A parm change on both the MF MQ side and the CRN MQ side missed, causing messages to go  to the dead letter queue. When the parm was corrected on each side, the problem was resolved.   [NAME] PIR completed 6/10 - CR0227333 PIR completed 6/11 | 2010-06-07 00:00:00 | 2010-06-21 00:00:00 | 2010-06-10 00:00:00 | N | Met | asp
67 | Jun-10 | Jun-10 | 2010-06-07 00:00:00 |  | 1 |  |  | 1 |  | P6053 | N | Bristol West Network | No Change | 3 | No ServiceCenter change record | Event  3 :   ZNA and Farmers eZSB users were unable to reach   Choicepoint website. Severity Level: 3   LSA Impacted: Root Cause: Under investigation. Customer Impact: Farmers Small Business policy with Auto users requiring MVR were   unable to issue or finalize quotes. Corrective Action:  Bristol West Network support removed a specific BGP route being  (Immediate Solution) advertised into the ATT MPLS cloud. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:29 PT to 09:35 PT Event Duration: From 08:29 PT to 09:35 PT  Ticket Number: INC158611918 | While moving the Bristol West data center into the CSC network some routes needed to be added in order to give connectivity to the Bristol west users, which was the main cause of the problem.    Resolution: The Bristol West team remove the addition ( Injection ) from their ATT MPLS edge router and that cleared the specific advertise routes into the ZNA/Farmers network thus clearing the problem.   [NAME] completed | 2010-06-08 00:00:00 | 2010-06-22 00:00:00 | 2010-06-16 00:00:00 | Y | Met | [NAME]
68 | Jun-10 | Jun-10 | 2010-06-09 00:00:00 | 1 |  |  |  |  | 1 | CR0226444 | N | ispna_netwksvc_engineering | Minor | 3 | ZNA - SCH - Enable VACLs on ZNA switches for traffic capture to support Compuware Vantage proof of concept | Event  1 :   (ZNA) Chennai, India and Krakow, Poland applications within   the Schaumburg Citrix farm exhibited degraded performance. Severity Level: 4 LSA Impacted: ZNA Root Cause: Under investigation. CR0226444 [DMR95699] Customer Impact:  Affected users were experiencing long page generations and delays in launching new applications. Corrective Action:  Network support removed a cable from SCHIL_ZNA_DSR1 port 9/5. (Immediate Solution) Corrective Action: Same As Immediate (Permanent Solution) Service Outage: From 00:27 PT to 09:21 PT Event Duration: From 00:27 PT to 09:21 PT  Ticket Number: INC158619233 | The port that was set up to capture traffic is grouped via hardware to the port where the Perseus firewall connects to the same switch.  This caused the interface for the Perseus firewall to perform in a degraded state even though it should not have been a factor.    [NAME] completed | 2010-06-10 00:00:00 | 2010-06-24 00:00:00 | 2010-06-21 00:00:00 | Y | Met | isp
69 | Jul-10 | Jun-10 | 2010-06-10 00:00:00 | 1 |  |  |  |  | 1 | CR0223693 | Y | aspna_usams_zcenter | Standard | 4 | zCenter Deployment Activity | Event  4 :   (ZNA) Z-Center exhibited intermittent degraded functionality. Severity Level: 4 LSA Impacted: ZNA Root Cause: There was a parameter in the Pega Database properties causing a table mapping mismatch. Customer Impact: Affected users were unable to create new cases. Corrective Action:  For all occurrences, iNet support killed a Node 1 JVM that was hung  (Immediate Solution) with an 'Out of Memory' error, which the Node Agent process then automatically restarted.  The incidents are a reoccurrence of the issue   from (06/09).  Z-Center support corrected a parameter in the Pega   Database properties causing a table mapping mismatch. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:54 PT to 10:23 PT;  From 10:52 PT to 11:05 PT;  From   11:19 PT to 11:27 PT Event Duration: From 09:54 PT to 11:27 PT  Ticket Number: INC158624086 | [NAME] - the change that caused this error was our 4.1.2 release which was a planned/scheduled release that was implemented with CR0223693.    The issue was delayed because the process that ran to start causing this wasn't scheduled for 2 weeks after the deployment.  We added new functionality to batch process renewal cases.  The cases were queued up a few days after the release and things were fine, however when the SLA agents started running to process the cases there was an error in a table mapping that started creating large [NAME] log files (~1.5 [NAME]), which then cascaded into causing a JVM out of memory error.  [NAME] completed | 2010-06-23 00:00:00 | 2010-07-07 00:00:00 | 2010-07-01 00:00:00 | N | Met | asp
70 | Jun-10 | Jun-10 | 2010-06-14 00:00:00 |  | 1 |  |  | 1 |  | CR0206058 | N | aspna_auto | Standard | 3 | Rel 06/2010 Auto Online, Batch , Edge, RDR, EDI Elements | Event  1 :   (Farmers) eAgent users in New Mexico were receiving [NAME] error when trying to perform auto quotes. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: This issue was introduced in Release 6. Customer Impact: Users in New Mexico were unable to perform quotes in Auto express. Corrective Action:  Application support corrected the problem and a new copy of program (Immediate Solution)  [NAME] was implemented in CICS regions. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:00 PT to 08:50 PT Event Duration: From 07:00 PT to 08:50 PT  Ticket Number:        IM1773702 | Shrini: Code error introduced with Release 6     [NAME]: PIR completed | 2010-06-15 00:00:00 | 2010-06-29 00:00:00 | 2010-06-23 00:00:00 | N | Met | asp
71 | Jun-10 | Jun-10 | 2010-06-15 00:00:00 | 1 |  |  |  |  | 1 | CR0229099 | N | ispna_firewall | Minor | 4 | ZNA Bluecoat - Change external interface to [NAME] speed for increased performance. | Event  8 :   (ZNA) Users were unable to access external web sites. Severity Level: 4 LSA Impacted: ZNA Root Cause: CR229099 Customer Impact: Minimal users were affected due to time of day. Corrective Action:  [NAME] and [NAME] support changed the speed on port Gi9/29 on  (Immediate Solution) USZZ1SCHPROXY to match the internal interface. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 18:30 PT to 19:06 PT Event Duration: From 18:30 PT to 19:06 PT  Ticket Number: INC158633097 | [NAME]: Human error during the processing of this change.  After refresh, hit [NAME] button in error.  [NAME]: PIR completed | 2010-06-16 00:00:00 | 2010-06-30 00:00:00 | 2010-06-21 00:00:00 | N | Met | isp
72 | Jul-10 | Jun-10 | 2010-06-18 00:00:00 |  | 1 |  | 1 |  |  | CR0228977 | N | ispna_ngito_datacenter_okdc | Standard | 3 | OKDC Shepherd to Rockwell Move Wave 0 | Event  5 :   (Farmers) CRN was sporadically unavailable. Severity Level: 3 LSA Impacted: Root Cause: Under investigation. Customer Impact: Affected users experienced clocking when attempting to log into or   navigate within CRN. Corrective Action:  SAN support changed the SRDF replication from synchronous to  (Immediate Solution) adaptive copying for CRN database server OKCAXDB01 to the new   Rockwell Data Center site. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:45 PT to 11:30 PT Event Duration: From 10:45 PT to 11:30 PT  Ticket Number: IM1775741 | Per [NAME]:  The SRDF is a process to replicate the storage between sites. They are using this as part of the project to move OKC to the new Data Center. SRDF has been going for a few weeks but during this specific instance they were copying over storage for the CRN Data Base server, OKCAXDB01. Due to the setting on SRDF it caused the IO on the database server to drop due to resource contentions. At this point the CRN users started seeing a performance degradation because the database was unable to handle the requests as fast as normal. This was basically just an incorrect setting for the SRDF and SAN support was able to correct it immediately when they were made aware of the issue.  [NAME]:  PIR completed | 2010-06-24 00:00:00 | 2010-07-08 00:00:00 | 2010-07-02 00:00:00 | Y | Met | isp
73 | Jul-10 | Jul-10 | 2010-07-02 00:00:00 |  | 1 |  |  | 1 |  | CR0230216 | N | ispna_distsvc_gad | Standard | 4 | Update [NAME] logon script to update drive mappings for FNWL Users | Event  2 :   (Farmers) New World Life users were not automatically mapped to   share drives on login. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Users did not automatically map to their share drives. Users were able to   manually map drives as a partial workaround. Corrective Action:  [NAME] support reverted back to previous login script for NWL  (Immediate Solution) users. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:57 PT to 09:00 PT Event Duration: From 05:57 PT to 09:00 PT  Ticket Number: INC158669331 | [NAME] reported this was caused by human error in implementing the change (new person).  [NAME]: PIR completed | 2010-07-06 00:00:00 | 2010-07-20 00:00:00 | 2010-07-08 00:00:00 | N | Met | isp
74 | Jul-10 | Jul-10 | 2010-07-13 00:00:00 |  | 1 |  |  | 1 |  | CR0233140 | N | aspna_ade | Standard | 4 | Release 08, 2010 MICRO3 ADE/Farmers.com/EDB Production deployment | Event  11 :   (Farmers) Some OKC, KCY, and GR HelpPoint users are unable to   make rental assignments to vendors in CRN. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Less than 50 users are receiving "Switch To" messages when   attempting to create rental units. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 12:19 PT to Ongoing Event Duration: From 12:19 PT to Ongoing Ticket Number: IM1786557 | Uvin Kumar: Release related  PIR completed | 2010-07-15 00:00:00 | 2010-07-29 00:00:00 | 2010-07-20 00:00:00 | N | Met | asp
75 | Aug-10 | Jul-10 | 2010-07-16 00:00:00 |  | 1 |  |  | 1 |  | n/a DMR #98823 |  | n/a | n/a | 3 |  | Event  1 :   (Farmers) The LANFREE portion of the Personal Lines Data   Warehouse TSM is unavailable. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Weekly data loads are delay.  Greater impact in the morning is that   Personal Lines Data Warehouse will have outdated data. Corrective Action:  Tech call is in progress with multiple groups, IBM and EMC. (Immediate Solution) Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 19:00 PT to Ongoing Event Duration: From 19:00 PT to Ongoing Ticket Number: INC158699477 | This problem was caused by change activity to add a new node. LAPES09, to the LAN Free environment. This work was done under a DMR, and should have been done under a CR (change request).   Due to human error, the master password had been inadvertently reset during this  process which caused all other LAN Free agents to disconnect, which was the reason for this outage.  [NAME]:  PIR complete. | 2010-08-12 00:00:00 | 2010-08-26 00:00:00 | 2010-08-23 00:00:00 | N | Met | n/a
76 | Aug-10 | Jul-10 | 2010-07-25 00:00:00 |  | 1 |  |  | 1 |  | CR0235588 | Y | ispna_distsvc_storage-dist | Standard | 5 | ReIPing the centera6882 & setting the network parameters for Centera and resume the replication from OKC  the status. | Event  3 :   (Farmers) FileNet server OKCAXFN01 experienced a slow   image retrieval rate from the document repository hosted on   Centerra 6882. Severity Level: 5 LSA Impacted: Farmers Exchanges Root Cause: CR0235588. Customer Impact: None reported. Users may have experienced sporadic delays when   attempting to retrieve images via CRN FileNet. Corrective Action:  Backed out CR0235588, Re-IP of Centerra 6882. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: None Event Duration: From 08:58 PT to 19:32 PT  Ticket Number: IM1791715 | [NAME]: Storage group made a change to move the Centera to a different vlan.  This caused image retrieval to become slow.  Centera was moved back to original vlan & ip addresses.  [NAME]: PIR completed | 2010-07-26 00:00:00 | 2010-08-09 00:00:00 | 2010-08-04 00:00:00 | N | Met | isp
77 | Aug-10 | Jul-10 | 2010-07-25 00:00:00 |  | 1 |  | 1 |  |  | CR0235646 | Y | aspna_crn | Standard | 4 | CRN - Adding New Servers for Coverage Transactions (21st C Migration - Wintel Servers) | Event  2 :   (Farmers) Some CRN transactions were stuck in a "Pending"   status. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Less than 10 users experienced a status of "Pending" when attempting   to submit coverages and [NAME]. Payments and reserves were not   affected. Partial workaround was in place; pending transactions were   manually transferred from problematic server. Corrective Action:  Registered custom components on PAAPP4040. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:38 PT to 17:37 PT Event Duration: From 10:38 PT to 17:37 PT  Ticket Number: IM1791712 | Uvin Kumar: 2 MFI servers were added but some components were not registered on the new server.  [NAME]: PIR Pending | 2010-07-27 00:00:00 | 2010-08-10 00:00:00 | 2010-08-06 00:00:00 | N | Met | asp
78 | Aug-10 | Jul-10 | 2010-07-26 00:00:00 | 1 |  | 1 |  | 1 | 1 | CR0235556 | N | ispna_ewp_swd | BAU | 4 | Deploy PUSH - ZFS-MLT-AcrobatReader-9.3-GBL | Event  1 :   (ZNA) Andover, MA eZACCESS users were unable to view   documents within Forms and Letters. Severity Level: 4 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Up to 35 Affected users were unable to view documents when trying to   launch or view Adobe documents. Corrective Action:  eWP support cleared the cache and system memory for 3 users. 2  (Immediate Solution) more are in process. Remaining users will contact the Help Desk who   will walk them through resolving their issue. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:58 PT to 10:30 PT Event Duration: From 05:58 PT to 10:30 PT  Ticket Number: INC158716179 | [NAME]): There was a push of [NAME] that went over the weekend and it went to the wrong environment and did not work correctly. Fix was to manually clear the cache and memory and was able to work again.  [NAME] in progress that will review all ispna_ewp_swd changes (CR0235556, CR0235553, CR0235558, CR0235540 , CR0235551) implemented over the weekend  PIR completed | 2010-07-27 00:00:00 | 2010-08-10 00:00:00 | 2010-08-04 00:00:00 | N | Met | isp
79 | Aug-10 | Jul-10 | 2010-07-30 00:00:00 | 1 |  |  |  |  | 1 | CR0235860 | N | ispna_netwksvc_engineering | Minor | 3 | ZNA -  NJPAR1 (Parsippany NJ) adding an additional switch to an exsicting  stack to support new users | Event  1 :   (ZNA) Parsippany, NJ experienced a partial network outage. Severity Level: 3 LSA Impacted: ZNA Root Cause: A new switch added to the stack under CR0235860 was incorrectly   configured. Customer Impact: Up to 40 users were unable to access their network resources. Corrective Action:  Network support reloaded the configuration on  (Immediate Solution) PARNJ1F4_ZNA_AS1. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:28 PT to 06:35 PT Event Duration: From 04:28 PT to 06:35 PT  Ticket Number: INC158727044 | According to [NAME] this problem was caused by change CR0235860. Proceedures were not followed correctly. A new switch added to the stack  was incorrectly  configured. AN RCA was completed for this problem.  [NAME]:  PIR is complete | 2010-08-02 00:00:00 | 2010-08-16 00:00:00 | 2010-08-05 00:00:00 | Y | Met | isp
80 | Aug-10 | Aug-10 | 2010-08-02 00:00:00 |  | 1 |  |  |  |  | CR0235601 | N | ispna_firewall | Major | 2 | Farmers Los Angeles: Replace Cisco PIX eAgent production firewalls with Cisco ASA devices | Event  1 :   (Farmers) Image Center was unavailable. Severity Level: 2 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: As of (08/02) 08:00CT up to 1000 agents were unable to scan or view   documents through Image Center. Corrective Action:  Support added static routes to PRG1 and PRH1 to point to  (Immediate Solution) 10.182.211.1. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (08/01) 10:40 PT to (08/02) 12:58 PT Event Duration: From (08/01) 10:40 PT to 12:58 PT  Ticket Number: IM1794621, INC158730789 | [NAME] reported this was caused by a hardware change #CR0235601 which was to replace the Cisco PIX eAgent production firewalls with Cisco ASA devices.   [NAME]: PIR is [NAME] | 2010-08-03 00:00:00 | 2010-08-17 00:00:00 | 2010-08-17 00:00:00 | N | Met | isp
81 | Aug-10 | Aug-10 | 2010-08-02 00:00:00 |  | 1 |  |  |  |  | CR0228699 | N | ispna_dcs_prod_inet | Standard | 3 | [NAME] v6.1 on laapws08 and laapws15 | Event  3 :   (Farmers) Farmers Business Insurance Express is experiencing   performance degradation. Severity Level: 3 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Affected users are experiencing sporadic slowdowns when navigating   within FBIE. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:30 PT to Ongoing Event Duration: From 04:30 PT to Ongoing Ticket Number: IM1795015 | [NAME] reported this was caused by change #CR0228699 which was the Install Websphere v6.1 on laapws08 and [NAME]   [NAME]: PIR is [NAME] | 2010-08-03 00:00:00 | 2010-08-17 00:00:00 | 2010-08-12 00:00:00 | N | Met | isp
82 | Aug-10 | Aug-10 | 2010-08-03 00:00:00 |  | 1 |  |  |  |  | CR0228699 | N | ispna_dcs_prod_inet | Standard | 3 | [NAME] v6.1 on laapws08 and laapws15 | Event  1 :   (Farmers) Business Insurance Express was sporadically      unavailable. Severity Level: 3 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Approx. 500 users sporadically received "91 [NAME] application error"   messages when attempting to select quotes within FBIE and delays in   various other areas of the application. Corrective Action:  INET support regenerated the WebSphere plug-in configuration file  (Immediate Solution) for iPlanet servers LASPEW18 and 19. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:05 PT to 09:58 PT Event Duration: From 09:05 PT to 09:58 PT  Ticket Number: IM1795414 | [NAME] reported this was related to the remediation efforts to correct the problem initially caused by change #CR0228699.   [NAME]: PIR is [NAME] | 2010-08-04 00:00:00 | 2010-08-18 00:00:00 | 2010-08-12 00:00:00 | N | Met | isp
83 | Aug-10 | Aug-10 | 2010-08-06 00:00:00 |  | 1 |  | 1 |  |  | CR0237778 | Y | ispna_netwksvc_support | Standard | 4 | Farmers Agency & DM Websites IP Change to New Vendor DMR request from e-Business DMR 100904 | (Farmers) Farmersagent.com was unavailable for some  ServicePoint users. | [NAME]: It has not yet been confirmed that this change caused the event or was a trigger.  This will be based on whether the DMR included notification to the [NAME] team.  Meeting review has been scheduled and PIR will be done if it was determined that communication took place.  [NAME]: PIR completed | 2010-08-09 00:00:00 | 2010-08-23 00:00:00 | 2010-08-19 00:00:00 | Y | Met | isp
84 | Aug-10 | Aug-10 | 2010-08-09 00:00:00 | 1 |  |  | 1 |  |  | CR0236984 | N | ispna_distsvc_gad | Standard | 4 | Demote / Promote domain controller PAADC0005 -  in Oklahoma City data center move | Event  8 :   (ZNA) ZNA e-File sporadically generated authentication  Severity Level: 4 LSA Impacted: Root Cause: Unsuccessful re-promotion of PAADC0005 to Domain Controller   status after a physical relocation between data centers caused an IP   entry to be present in the eFile Global Active directory domain   controller pool for a server unable to perform authentication roles. Customer Impact: e-File sporadically generated authentication errors. Corrective Action:  DNS/IP services removed PAADC0005 from the  (Immediate Solution) zfsgaddc.amer.zurich.corp pool. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 12:47 PT to 13:45 PT Event Duration: From 12:47 PT to 13:45 PT  Ticket Number: INC158746425 | [NAME]: PAADC0005 was moved during CRN's Wave 7 server migration from OKC's Shepard's Mall to OKC's Rockwell Data Center. The change was CR0236848.     After the server was migrated and brought up in the new data center, it had replication issues and the GAD team demoted it as a valid domain controller to minimize conflicts with servers and/or users who were trying to authenticate with PAADC0005.   [NAME]: PIR completed | 2010-08-10 00:00:00 | 2010-08-24 00:00:00 | 2010-08-18 00:00:00 | N | Met | isp
85 | Aug-10 | Aug-10 | 2010-08-16 00:00:00 | 1 |  |  |  |  | 1 | CR0239533 | N | aspna_usams_workbench | BAU | 1 | workbench / RSP / RAD / PCR production deployment on 08/13/2010. | Event  1 :   (ZNA) Workbench exhibited degraded performance. Severity Level: 1 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Up to 500 users experienced long page generations and sporadically,   unusable delays logging in and between screens. Corrective Action:  [NAME] off of Scale-Out servers over to Microsoft session  (Immediate Solution) state management. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:45 PT to 14:15 PT Event Duration: From 05:45 PT to 14:15 PT  Ticket Number: INC158759772 ************************************************************* ** Event 4 was the result of remediation work being done for Event 1 and will be merged into Event 1..  Event  4 :   (ZNA) DocPro was unavailable. Severity Level: 3 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Up to 50 users were unable to utilize the application. Corrective Action:  Failed standby F5 back to primary F5. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 11:06 PT to 14:36 PT Event Duration: From 11:06 PT to 14:36 PT  Ticket Number: INC158761010 | [NAME]:  Scaleout session state manager was overwhelmed and slow to respond due to a portion of the Workbench Application release over the weekend which contributed to the increase load on Scaleout.  It was determined that ScaleOut was taking up double the memory that is normal, so we moved from ScaleOut to Microsoft session state management.  [NAME]:  PIR complete. | 2010-08-17 00:00:00 | 2010-08-31 00:00:00 | 2010-08-27 00:00:00 | N | Met | asp
86 | Aug-10 | Aug-10 | 2010-08-16 00:00:00 |  | 1 |  | 1 | 1 |  | CR0237064 | N | aspna_servicepoint | Standard | 4 | Implementation of ServicePoint and MyFieldPoint Release 8 | Event  9 :   (Farmers) Call whispers were unavailable at the AUS, KCY,   NW, and MW ServicePoints. Severity Level: 4 LSA Impacted: Farmers Mgmt, Farmers Exchange Root Cause: Under investigation. Customer Impact: Customer Advocates at the AUS, KCY, NW, and MW ServicePoints   were not receiving the beginning call announcement on inbound   customer calls. Corrective Action:  Telecom support backed out a portion of CR0237064. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:45 PT to 07:45 PT Event Duration: From 06:45 PT to 07:45 PT  Ticket Number: IM1800191 ************************************************ Event  11 :   (Farmers) iWD received duplicate transactions from [NAME]. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Approx. 150 AUBC and MWBC users received "The ownership of the SR has changed" messages when attempting to utilize iWD. Corrective Action:  Support backed out CR123156. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:30 PT to 16:46 PT Event Duration: From 08:30 PT to 16:46 PT  Ticket Number: INC158760782 | Per [NAME]:  Event 9 >>  CR1194 in Release 8.5. (Update of Siebel Workflows) was the root cause. CR1194 was implemented as one of several CR's(Business Change Requests). The workflow impacted   the whisper functionality in some call types that were received by the Customer Advocates.             Event 11 >>  Error Handling WorkFlow  was implemented as part of Release 8.5 impacted the way the IWD transactions were  assigned, duplicating the transaction to the same assignee.      [NAME]:  PIR complete. | 2010-08-17 00:00:00 | 2010-08-31 00:00:00 | 2010-08-27 00:00:00 | N | Met | asp
87 | Aug-10 | Aug-10 | 2010-08-16 00:00:00 |  | 1 |  | 1 |  |  | CR0238848 |  | aspna_crn | Minor | 4 | Compliance Deployment 10.2 Updates - Post Implementation | Event  12 :   (Farmers) Some CRN payments/reserves were stuck in a   "Pending" status. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Affected users experienced a status of "Waiting" when attempting to   submit payments/reserves. Corrective Action:  Siebel support recycled the Reserve component on OKCNTSB07 and  (Immediate Solution) re-submitted affected transactions. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 12:09 PT to (08/17)  02:00 PT Event Duration: From 12:09 PT to (08/17) 02:00 PT  Ticket Number: IM1800403 | Pending Customer Follow-up from Uvin Kumar.    [NAME]:  PIR complete. | 2010-08-17 00:00:00 | 2010-08-31 00:00:00 | 2010-08-27 00:00:00 | N | Met | asp
88 | Sep-10 | Aug-10 | 2010-08-19 00:00:00 | 1 |  |  |  |  | 1 | CR0240196 |  | ispna_netwksvc_engineering | Minor | 4 | ZNA - Create NAT on ILSCH1-LB005 | Event  6 :   (ZNA) Z-Spire produced sporadic error messages. Severity Level: 4 LSA Impacted: ZNA Root Cause: ZRWEB3525 was competing with legacy device T1-  5_DMZ_DRK_RM_SW1 for IP address 206.252.74.253. Customer Impact: Affected users had to periodically resubmit transactions. Corrective Action:  Network support removed ZRWEB3525 from the F5 application pool. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:00 PT to 07:30 PT Event Duration: From 07:00 PT to 07:30 PT  Ticket Number: INC158768743 | [NAME] indicated this was caused by the IP change - CR0240196   [NAME]: PIR is complete | 2010-08-20 00:00:00 | 2010-09-03 00:00:00 | 2010-08-25 00:00:00 | Y | Met | isp
89 | Sep-10 | Aug-10 | 2010-08-20 00:00:00 | 1 |  |  |  |  | 1 | CR0240236 |  | ispna_netwksvc_support | Minor | 4 | Replace [NAME] at WPBFL1-ZSL-AR1 | Event  5 :   (ZNA) [NAME], FL network resources were unavailable. Severity Level: 4 LSA Impacted: ZNA Root Cause: CR0240236 to replace router. Customer Impact: Affected users were unable to access network resources. Corrective Action:  Field tech removed network cable from the wrong port on the NIC and (Immediate Solution)  connected to the correct port. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:32 PT to 06:50 PT Event Duration: From 05:32 PT to 06:50 PT  Ticket Number: INC158771094 | Reported in today's IT Availability meeting - The Vendor failed to complete the work - cause was human error.   [NAME]: PIR complete | 2010-08-23 00:00:00 | 2010-09-06 00:00:00 | 2010-09-01 00:00:00 | Y | Met | isp
90 | Sep-10 | Aug-10 | 2010-08-21 00:00:00 | 1 |  |  |  |  | 1 | CR0240644 | Y | ispna_distsvc_unix | Standard | 2 | Reload shapdb03 - with  mksysb uimage of shapdb04 | Event  1 :   (ZNA) ZDW/eFile database on SHAPDB04 is unavailable due to   AIX file system corruption Severity Level: 2 LSA Impacted: Root Cause: Under investigation. Customer Impact: Minimal impact due to low Sunday volume. Users utilizing [NAME], e-  File, and applications receiving feeds from [NAME] are unable to   complete transactions. Corrective Action:  Multiple support teams are engaged. A Tech call is in progress.   (Immediate Solution) Support is implementing a plan to complete restoration of SHAPDB04. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 19:40 PT to Ongoing Event Duration: From 19:40 PT to Ongoing Ticket Number: INC158772451 | Reported in today's IT Availability meeting that the root cause was: CR# 240644 Reload SHAPDB03 - with MKSYSB.   [NAME]: PIR complete | 2010-08-23 00:00:00 | 2010-09-06 00:00:00 | 2010-09-01 00:00:00 | N | Met | isp
91 | Sep-10 | Aug-10 | 2010-08-27 00:00:00 | 1 |  |  |  |  | 1 | CR0241746 | N | ispna_distsvc_termsvr | BAU | 4 | MPhasis Underwriting:  Remove the groups and published apps associated with 32-bit silo | Event  7 :   (ZNA) Pune, India Mphasis Citrix Presentation Desktop was   missing published applications. Severity Level: 5 LSA Impacted: ZNA Root Cause: CR0241748 Customer Impact: Up to 20 users were unable to see their Lotus Notes icons. Corrective Action:  Citrix support re-displayed the Notes icon in the affected Citrix  (Immediate Solution) profiles. Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 05:54 PT to 06:38 PT Event Duration: From 05:54 PT to 06:38 PT  Ticket Number: INC158786059 | The users had 2  Notes icons and part of the change was to disable one icon, hide the other until change was complete, then reenable the hidden Notes icon which was done as soon as citrix was notified  [NAME]: PIR complete | 2010-08-30 00:00:00 | 2010-09-13 00:00:00 | 2010-09-09 00:00:00 | N | Met | isp
92 | Sep-10 | Aug-10 | 2010-08-30 00:00:00 |  | 1 |  |  | 1 |  | CR0240566 | N | aspna_e-agent | Standard | 3 | [NAME] eAuto/eHome/Express 1/Express 2/IVR implementation | Event  2 :   (Farmers)  Agent [NAME] is sporadically unavailable via   the Agency Dashboard. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Approx. 125 KCY ServicePoint marketing and accounting users are   receiving "Object is not defined" errors when attempting to scan or   attach documents within AFI. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 06:49 PT to 20:00 Event Duration: From 06:49 PT to 20:00 Ticket Number: INC158788889 | This was caused by faulty code introduced in IVR application,      [NAME]:  PIR complete | 2010-08-31 00:00:00 | 2010-09-14 00:00:00 | 2010-09-10 00:00:00 | N | Met | asp
93 | Sep-10 | Sep-10 | 2010-09-03 00:00:00 |  | 1 |  | 1 |  |  | CR0241016 | N | ispna_ngito_datacenter_okdc | Major | 3 | OKDC Shepherd to Rockwell Move Wave 11 - PROD | Event  2 :   (Farmers) CTI and NICE were unavailable for the OKC and   Lake Mary HelpPoints. Severity Level: 3 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: OKC and [NAME] users were unable to receive calls. As   of 12:19CT users logged into hard phones and were able to take calls.   Call recordings were unavailable for duration of the outage. Corrective Action:  Connected affected cables to allow the PBX and AES servers to regain  (Immediate Solution) connectivity to the network. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (09/02) 09:49 PT to (09/02) 11:16 PT Event Duration: From (09/02) 09:49 PT to (09/02) 11:16 PT  Ticket Number: IM1808467 | [NAME]: preactivity cabling in preparation for the weekend moves causes this error   PIR in progres - [NAME] | 2010-09-07 00:00:00 | 2010-09-21 00:00:00 | 2010-09-13 00:00:00 | Y | Met | isp
94 | Sep-10 | Sep-10 | 2010-09-03 00:00:00 | 1 |  |  |  |  | 1 | CR0243301 | N | aspna_usams_ezacce | Minor | 3 | Claim  Assignment Rules  for Aug 2010 | Event  3 :   (ZNA) eZACCESS and Care Canter displayed unexpected   behavior due to an overnight rules implementation. Severity Level: 3 LSA Impacted: ZNA Root Cause: An overnight rules implementation. Customer Impact: Affected users received inaccurate assignments. A small volume of   claims were delayed due to the introduced necessity of manual   processing. Corrective Action:  Care Center support redirected the Care Point service to PAVAP2984. (Immediate Solution) Corrective Action: Rollback of rules implemented under the previously implemented CR (Permanent Solution) Service Outage: From 05:00 PT to 08:27 PT Event Duration: From 05:00 PT to 08:27 PT  Ticket Number: INC158801341 | [NAME]: application rule change caused Care Center issues   [NAME] completed | 2010-09-07 00:00:00 | 2010-09-21 00:00:00 | 2010-09-17 00:00:00 | N | Met | asp
95 | Sep-10 | Sep-10 | 2010-09-09 00:00:00 | 1 |  |  |  |  | 1 | CR0242746 |  | ispna_distsvc_gad | Standard | 3 | GPO change: Implement [NAME] exception (PAC) file and IE settings for ZNA CWS users | (ZNA) Macro Express in eZACCESS displayed degraded   performance. Severity Level: 3 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Affected users were unable to use macros to expedite filling out   eZACCESS form fields. A work around of entering data manually was   in place. Corrective Action:  GAD support backed out CR0242746. User validation complete at  (Immediate Solution) 10:12 PT. RC: CR0242746. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From (09/07) 05:36 PT to (09/09) 09:54 PT Event Duration: From (09/07) 05:36 PT to 09:54 PT  Ticket Number: INC158806365 | [NAME] reported that they (ispna_distsvc_gad) created the change on behalf of the CSC EWP group that at the time did nott have a ServiceCenter         assignment group.  This change was to upgrade the PAC file which caused an error in an Internet Explorer Tool Bar.            [NAME] | 2010-09-09 00:00:00 | 2010-09-23 00:00:00 | 2010-09-16 00:00:00 | N | Met | isp
96 | Oct-10 | Sep-10 | 2010-09-17 00:00:00 |  | 1 |  | 1 | 1 |  | CR0244872 | N | ispna_distsvc_ops | Minor | 4 | Migrate it to virtual to retire the hardware. | Event  3 :   (Foremost) STAR was unavailable. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Users received "Sorry Foremost STAR is currently unavailable"   messages when attempting to access www.foremoststar.com. Corrective Action:  Recycled GRLPFS02 and GRLPFS03. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:00 PT to 04:30 PT Event Duration: From 04:00 PT to 04:30 PT  Ticket Number: IM1813992 | [NAME] reported this    incident was related to CR0244872. Two servers were migrated from    physical to become virtual servers. STAR should have been able to handle    this but there might have been an issue with [NAME]                                                                                                                [NAME] complete | 2010-09-20 00:00:00 | 2010-10-04 00:00:00 | 2010-10-01 00:00:00 | N | Met | isp
97 | Oct-10 | Sep-10 | 2010-09-23 00:00:00 | 1 |  |  |  |  | 1 | CR0246468 | N | ispna_distsvc_termsvr | BAU | 4 | replace the expiring certs for app.zurichna.com and csg.zurichna.com | Event  5 :   (Farmers) CRN was sporadically unavailable for the GR, KCY,   OKC, and [NAME]. Severity Level: 4 LSA Impacted: Farmers Exchanges Root Cause: Under investigation. Customer Impact: Approx. 30 users received "An error has occurred executing a Sql   statement" errors when attempting to utilize CRN. Corrective Action:  Under investigation. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:44 PT to 10:15 PT Event Duration: From 09:44 PT to 10:15 PT  Ticket Number: IM1816066 | Jerry from Citirix team informed that certificates were expiring , due which the change was performed. It is a vendor issue as reported by [NAME] in Progress by Satya/Ken on 10/06/2010 | 2010-09-23 00:00:00 | 2010-10-07 00:00:00 | 2010-10-06 00:00:00 | N | Met | isp
98 | Oct-10 | Sep-10 | 2010-09-27 00:00:00 |  | 1 |  |  | 1 |  | CR0238043 |  | ispna_dcs_prod_inet | Standard | 3 | [NAME] v6.1 on laapws08 and laapws15 | Event  4 :   (Farmers) Farmers Business Insurance Express was   sporadically unavailable. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Users received sporadic errors when attempting to open existing work   or creating new work within FBIE. Users were able to perform   additional attempts as a partial workaround. Corrective Action:  Support increased space on filesystem /usr/inet on LAAPWS08 and 15. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:00 PT to 07:45 PT Event Duration: From 07:00 PT to 07:45 PT  Ticket Number: INC158904109 | [NAME] as reported the issue and the incident was caused due to the change to  Upgrade from Websphere 6.0.2 to Webshpere 6.1 to take advantage of  performance enhancements that are included in this version.  Webshpere 6.0.2 has known memory leak issues.  PIR In progress Completed by Satya/Terry on 10/08/2010 | 2010-09-28 00:00:00 | 2010-10-12 00:00:00 | 2010-10-08 00:00:00 | N | Met | isp
99 | Oct-10 | Sep-10 | 2010-09-27 00:00:00 |  | 1 |  |  | 1 |  | CR0238043 | N | ispna_dcs_prod_inet | Standard | 3 | [NAME] v6.1 on laapws08 and laapws15 | Event  5 :   (Farmers) Farmers Business Insurance Express is sporadically   unavailable. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Approx. 500 Commercial users are experiencing latency when   attempting to utilize various portions of FBIE. Corrective Action:  [NAME] and memory upgrades were implemented on LAAPWS08  (Immediate Solution) and 15. Monitoring will continue throughout the night and final user   validation will take place in the morning. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:50 PT to Ongoing 1:45 PST. Event Duration: From 10:50 PT to Ongoing Ticket Number: INC158905080 | [NAME] as reported the issue and the incident was caused due to the change to  Upgrade from Websphere 6.0.2 to Webshpere 6.1 to take advantage of  performance enhancements that are included in this version.  Webshpere 6.0.2 has known memory leak issues.  [NAME] by Satya/Terry on 10/08/2010 | 2010-09-28 00:00:00 | 2010-10-12 00:00:00 | 2010-10-08 00:00:00 | N | Met | isp
100 | Oct-10 | Sep-10 | 2010-09-27 00:00:00 | 1 |  |  |  |  | 1 | CR0247187 | N | aspna_usams_rie | Emergency | 2 | To fix Risk image document functionality in production. | Event  1 :   (ZNA) RiskACCESS component [NAME] was unavailable. Severity Level: 2 LSA Impacted: ZNA Root Cause: Incompatible code synchronization configuration. Customer Impact: Risk Image documents were unavailable to internal and external users   via the shared eZDM FileNet system. eZACCESS and eZDM were   unaffected. Corrective Action:  WebSphere support restored a new web.xml file to WAP041 over a  (Immediate Solution) prior version that had been erroneously returned to the configuration   during a prior synchronization event. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:16 PT to 08:27 PT Event Duration: From 07:16 PT to 08:27 PT  Ticket Number: INC158904128 | Issue was caused while updating a code drop 6 and also while To fix Risk image document functionality in production.Change ticket was CR0247147 as updated by Prasad Maruvada  PIR Completed by Satya/terry on10/06/2010 | 2010-09-28 00:00:00 | 2010-10-12 00:00:00 | 2010-10-06 00:00:00 | N | Met | asp
101 | Oct-10 | Oct-10 | 2010-10-04 00:00:00 |  | 1 |  |  | 1 |  | CR0247675 | N | ispna_servicepoint_telecom | Minor | 3 | SP:  [NAME] reporting for ASC to AUSP Tserver | Event  2 :          (Farmers) Some [NAME] calls were sporadically          misrouted to incorrect departments.  Severity Level:        3        LSA Impacted:        Farmers Mgmt  Root Cause:        CR0247675.  Customer Impact:        Approx. 300 users received sporadic misrouted calls through the PBX.  Corrective Action:         Telecom support removed routing connections for the ASC site.  (Immediate Solution)  Corrective Action:        Under investigation.  (Permanent Solution)  Service Outage:        From 06:00 PT to 10:10 PT  Event Duration:        From 06:00 PT to 10:10 PT  Ticket Number:        IM1821393 | [NAME]:Cause Description : some of the configuration changes perfomed were not correct as per the implementation plan  PIR Completed on 10/13/2010 : Venu/Jo | 2010-10-05 00:00:00 | 2010-10-19 00:00:00 | 2010-10-13 00:00:00 | N | Met | isp
102 | Oct-10 | Oct-10 | 2010-10-05 00:00:00 | 1 |  |  | 1 | 1 |  | CR0245725 | N | ispna_distsvc_notes | Minor | 2 | USWNH000 - Domino 8.5.2 Upgrade | Event  1 :   (ZNA) Lotus Notes servers experienced issues with the Notes   Address Book. Severity Level: 2 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Affected users experienced a delay in processing e-File mail requests in   WFM; as a workaround, a manual download of email was in place.   Affected users with mail files hosted on USZNM101 and 102   experienced delays with messages routed to their BlackBerry devices.   Some server to server communication was unavailable. Corrective Action:  Lotus Notes support applied an IBM update patch to the affected  (Immediate Solution) servers. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:19 PT to 13:33 PT Event Duration: From 07:19 PT to 13:33 PT  Ticket Number: INC158924479 | Change number was confirmed and received by Situtation Management during Post Mortem.  Several comebacks from the original 10/5 date; change related confirmed on 10/15  PIR in progress - [NAME] | 2010-10-15 00:00:00 | 2010-10-29 00:00:00 | 2010-10-22 00:00:00 | N | Met | isp
103 | Oct-10 | Oct-10 | 2010-10-05 00:00:00 |  | 1 |  |  |  | 1 | CR0248763 |  | ispna_distsvc_termsvr | BAU | 4 | Add Bristolwest.com to DNS Suffix Search Order list | Event  3 :   (Farmers) Some users were unable to launch a few published   applications through Citrix. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Affected users received "Windows cannot connect to the domain"   errors when attempting to launch some published applications. Corrective Action:  Citrix support adjusted registry setting and rebooted affected servers. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 03:00 PT to 05:50 PT Event Duration: From 03:00 PT to 05:50 PT  Ticket Number: INC158922986 | [NAME]:  DSN suffix change; error in the script search order parameters  PIR completed | 2010-10-06 00:00:00 | 2010-10-20 00:00:00 | 2010-10-12 00:00:00 | N | Met | isp
104 | Nov-10 | Oct-10 | 2010-10-15 00:00:00 | 1 |  |  |  |  | 1 | CR0251678 |  | ispna_ts_notes | Emergency | 4 | AFTER THE FACT - Policy Signing with Correct ID | Event  2 :   (Farmers/ZNA) Some Zurich and 21st Century users' Lotus   Notes configuration was unwillingly changed. Severity Level: 4 LSA Impacted: Farmers Mgmt, Farmers Exchange Root Cause: Under investigation. Customer Impact: Approx 50 reported users experienced a change in their Lotus Notes   configuration preferences. Users were able to change the configuration   and restart the Lotus Notes client as a partial workaround. Corrective Action:  Pushed updated group policy to revert the Startup Options,  (Immediate Solution) Configuration, and Sending preferences to default. Affected users will   need to restart the Lotus Notes client in order for new policy to be   retrieved. Communication was sent to problematic users. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 04:22 PT to 13:59 PT Event Duration: From 04:22 PT to 13:59 PT  Ticket Number: IM1826552, INC158849081 | [NAME] :While helping  helpdesk to push the policies to update the version 7 of  Lotus to version 8, policy did not work and caused the event, In the event an Emergency Change was raised to fix the issue.                                     PIR in progess by Satya/Ken | 2010-10-18 00:00:00 | 2010-11-01 00:00:00 | 2010-10-26 00:00:00 | N | Met | isp
105 | Nov-10 | Oct-10 | 2010-10-18 00:00:00 |  | 1 |  |  | 1 |  | CR0250507 | N | aspna_servicepoint | Standard | 4 | ServicePoint - My FieldPoint Release 10 (October Major Release) | Event  6 :   (Farmers) Agent [NAME] (ASC) in Austin Texas    experienced problems transferring internal phone calls   following a conversion from Avaya to [NAME] routing system. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: CR0250507 Customer Impact: ASC was unable to transfer some internal calls. Corrective Action:  Support backed out change record CR0250507. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:00 PT to 07:43 PT Event Duration: From 07:00 PT to 07:43 PT  Ticket Number: IM1827292 | [NAME] reported that this issue was related to release 10 applied this past weekend.     PIR In progress: [NAME] PIR Meeting scheduled with [NAME] on 11/5/10 12:15 PT | 2010-10-19 00:00:00 | 2010-11-02 00:00:00 | 2010-11-15 00:00:00 | Y | Not Met | asp
106 | Nov-10 | Oct-10 | 2010-10-18 00:00:00 |  | 1 |  |  | 1 |  | CR0236085 | N | aspna_e-cms | Standard | 4 | eCMS Release 10, 2010 Migration | Event  8 :   (Farmers) eAgent users were receiving errors when trying to   retrieve Web eMail in eCMS. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: Release 10 implementation CR0236085. Customer Impact: Affected users were unable to retrieve Web eMail. Corrective Action:  Application support will provide a permanent fix tonight during code  (Immediate Solution) deployment. Work around in place. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 08:45 PT to 08:55 PT Event Duration: From 08:45 PT to 08:55 PT  Ticket Number: IM1827286 | [NAME] from Agent Support Services reported that this issue was related to CR0236085   PIR In progress: [NAME] | 2010-10-19 00:00:00 | 2010-11-02 00:00:00 | 2010-10-29 00:00:00 | N | Met | asp
107 | Nov-10 | Oct-10 | 2010-10-20 00:00:00 | 1 |  |  |  |  | 1 | CR0251373 | N | ispna_distsvc_notes | BAU | 4 | Standard Migration Window for Domino/Lotus Notes Enhancements | Event  3 :   (ZNA) Form and Rate Exemptions (FARE) Tool was unavailable. Severity Level: 4 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: Users received an ' ! An error has occurred ! ' message when attempting   to select a state. Corrective Action:  Channel Z support applied a data fix to eliminate a formula error. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 12:30 PT to 15:15 PT Event Duration: From 12:30 PT to 15:15 PT  Ticket Number: INC158960231, INC158956038 | [NAME] , from Incident management  as confirmed that this event was caused due to the Change : CR0251373, this was performed to fix a fomula on  Channel Z  application                            PIR in progress : Satya/Ken | 2010-10-21 00:00:00 | 2010-11-04 00:00:00 | 2010-11-03 00:00:00 | N | Met | isp
108 | Nov-10 | Oct-10 | 2010-10-25 00:00:00 | 1 |  |  |  |  | 1 | CR0252836 | Y | aspna_usams_pcr | Minor | 3 | SSL certificate installation on PCR production  server on 10/23/2010 | Event  1 :   (ZNA) PCR experienced limited connectivity. Severity Level: 3 LSA Impacted: ZNA Root Cause: CR0252836 Customer Impact: External users were unable to connected to PCR's new HTTPS URL   due to CR0252836. Corrective Action:  [NAME] support opened port 443 on Schaumburg external firewall. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:00 PT to 10:11 PT Event Duration: From 09:00 PT to 10:11 PT  Ticket Number: INC158969259 | Per Ramesh Gangavarapu we migrated PCR website from http to https and during that process, we had tested accessing PCR application for internal/external users  from Zurich NetworkWhen the external users tried to access PCR application outside the Zurich network, they were unable to login.  The reason for this is the firewall is blocking the external users to access PCR application from outside the Zurich network.  PIR completed  [NAME] | 2010-10-26 00:00:00 | 2010-11-09 00:00:00 | 2010-11-04 00:00:00 | N | Met | asp
109 | Nov-10 | Oct-10 | 2010-10-29 00:00:00 | 1 |  |  |  |  | 1 | CR0251933 | Y | ispna_netwkvoice_ops | Minor | 3 | ZNA CP Parsippany, [NAME]-fix /Patch/ PEP updates | Event  4 :   (ZNA) Parsippany, NJ office voicemail was unavailable. Severity Level: 4 LSA Impacted: ZNA Root Cause: Patching. Customer Impact: Affected users were unable to access the voicemail system. Corrective Action:  Telecom support removed patches that were applied overnight and  (Immediate Solution) restore the voicemail system from the back made immediately prior to   patching. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:30 PT to 05:45 PT Event Duration: From 05:30 PT to 05:45 PT  Ticket Number: INC158981395 | [NAME]: Vendor quality patches on voicemail had issues which required a restore.  Future patches on hold until issue is resolved.  PIR in progress: [NAME] | 2010-11-01 00:00:00 | 2010-11-15 00:00:00 | 2010-11-04 00:00:00 | Y | Met | isp
110 | Dec-10 | Nov-10 | 2010-11-22 00:00:00 | 1 |  | 1 |  |  | 1 | CR0257364 | N | ispna_ts_os390 | Standard | 3 | Upgrade CA-Jobtrac to Release 11.0 on the A972, A031 and A027 LPARs | Event  2 :   (ZNA) Relay application displayed 'Time Out' errors. Severity Level: 3 LSA Impacted: ZNA Root Cause: CR0257347 Customer Impact: Up to 40 affected users were receiving 'Time Out' errors when   attempting to run station manager and premium preview. Corrective Action:  MVS support modified PROC in MCC1.SOFTWARE.PROCLIB. MQ  (Immediate Solution) support reset the trigger job. Corrective Action: Same as immediate. (Permanent Solution) Service Outage: From 05:29 PT to 08:02 PT Event Duration: From 05:29 PT to 08:02 PT  Ticket Number: INC58034868 | Per Manikanta Bommisetty The upgrade to [NAME] requires the conversion of the previous release databases to CA-Datacom databases. Since there is an existing CA-Datacom environment on the A972 and A031, all [NAME] tasks require a specific STEPLIB to the controlling CA-Datacom environment. The proc that was executed used a Jobtrac service to post an event, but did not include the STEPLIB and, therefore, failed. I was unaware of the use of this function outside of the Jobtrac environment. This type of function should be handled internally, within Jobtrac.    PIR completed: [NAME] | 2010-11-23 00:00:00 | 2010-12-07 00:00:00 | 2010-12-01 00:00:00 | N | Met | isp
111 | Dec-10 | Dec-10 | 2010-12-04 00:00:00 | 1 |  |  |  |  | 1 | CR0261694 | N | ispna_ts_dbase-bal | Emergency | 3 | COS     MIGRATION- 11/03/2010   EMERGENCY   WC Pending Rate Change Hold for CA. and Specialty sirs for Phoenix Proper | Event  1 :   (ZNA) eZSB exhibited degraded performance. Severity Level: 3 LSA Impacted: ZNA Root Cause: Under investigation. Customer Impact: About 30 users receive VBScript runtime errors when they work in the   New Business - Rate, Underwriting, Issue portion of eZSB. Corrective Action:  Developers backed out CR0261694 to resolve the issue. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 05:01 PT to 07:00 PT Event Duration: From 05:01 PT to 07:00 PT  Ticket Number: INC159065202 | As per the update given by [NAME] , change was necessary for removing unnecessary menu options for Farmers but the module was referencing another component which belongs to Zurich applications.This module is in QA but not in Production.Hence,this has given the error.Eventually this module was rolled back which has fixed the problem.  PIR Completed : VenuMadhuri | 2010-12-08 00:00:00 | 2010-12-22 00:00:00 | 2010-12-09 00:00:00 | N | Met | isp
112 | Dec-10 | Dec-10 | 2010-12-06 00:00:00 |  | 1 |  |  | 1 |  | CR0257782 | Y | ispna_ts_dbase-bal | Minor | 2 | Scheduled Zurich COS/EPP migration and ULTRA SPLIT    on   11/20/2010 | Event  1 :   (Farmers) eZSB is showing degraded performance. Severity Level: 2 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Affected users could not rate policies due to receiving 'MQ response   not received' error messages. Corrective Action:  [NAME] increased the [NAME]. depth on  (Immediate Solution) USB.ETE0.MQNTP2153.WORKCOMP.DATA.TEST on   PACLG2155. eZSB app support restarted the trigger monitor service   on PAAPP3225. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 11:57 PT to 13:21 PT Event Duration: From 11:57 PT to 13:21 PT  Ticket Number: INC159068508 | [NAME] mentioned  the root cause as the workcomp queue was inadvertently missed out from flipping over to the new Ultra environment as part of the Ultra Split project. Ultra split initiative is to separate Ultra application components from eZSB infrastructure (which is moving to Farmers). There were many queues that were moved over to the Zurich side, but, unfortunately, this particular queue in question went unnoticed and was not caught during the project planning, testing and implementation.  PIR Completed : Venu/Radhika | 2010-12-09 00:00:00 | 2010-12-23 00:00:00 | 2010-12-15 00:00:00 | N | Met | isp
113 | Dec-10 | Dec-10 | 2010-12-13 00:00:00 |  | 1 |  |  | 1 |  | CR0238111 | N | aspna_auto | Standard | 4 | Rel 12/2010 Auto Online, Batch , Edge, RDR, EDI Elements | Event  9 :   (Farmers) eAgent users in the state of Virginia are receiving   P5XP2 errors when trying to replace a vehicle. Severity Level: 4 LSA Impacted: Farmers Mgmt Root Cause: Under investigation. Customer Impact: Affected users are unable to replace vehicles in the state of Virginia. Corrective Action:  eAuto support is working on a fix. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 10:15 PT to Ongoing Event Duration: From 10:15 PT to Ongoing Ticket Number: IM1850920 | Per [NAME]:  One of the change went thru Rel 12 implementation under CR 0238111, caused the P5XP2 Fatals Issue was Identified on Sunday during Production Shakeout and fix went in on Monday.  PIR Completed: [NAME] | 2010-12-14 00:00:00 | 2010-12-28 00:00:00 | 2010-12-21 00:00:00 | N | Met | asp
114 | Dec-10 | Dec-10 | 2010-12-13 00:00:00 |  | 1 |  |  | 1 |  | CR0260816 | N | aspna_qm | Standard | 3 | Release 12, 2010 eAuto/eHome/Express 1/Express 2 implementation | Event  1 :   (Farmers) Some Farmers agents and ServicePoint employees   were unable to process credit card payments. Severity Level: 3 LSA Impacted: Farmers Mgmt Root Cause: Release 12 implementation. Customer Impact: Some users were unable to make credit card payments. There were some   credit card payments processed during the incident. Corrective Action:  Application support backed out project numbers B-10-065 AND B-10- (Immediate Solution) 095. Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 07:35 PT to 10:54 PT Event Duration: From 07:35 PT to 10:54 PT  Ticket Number: IM1850749,INC159082742 | Per [NAME]:  Some Farmers and ServicePoint experienced credit card payment failures. Investigation pointed to REL 12 project releases. Support backed out these two tables to resolve the issue  PIR Completed: [NAME] | 2010-12-14 00:00:00 | 2010-12-28 00:00:00 | 2010-12-17 00:00:00 | N | Met | asp
115 | Dec-09 | Dec-10 | 2010-12-15 00:00:00 |  | 1 |  |  |  |  | CR0195616 |  | ispna_netwksvc_engineering | Minor | 4 | Multiple sites - Upgrade OS on Riverbed WAN optimization units as preventative maintenance | (Farmers) NWCS was sporadically unavailable. Root Cause: Under investigation. Customer Impact: For the occurrence from 06:00PT - 12:00PT, 10 users experienced clocking when attempting to login or navigate within NWCS. For the 12:00PT environment recycle, NWCS support coordinated with business in regards to application unavailability. During the second occurrence Santa Ana, CA users experienced clocking or extended periods of latency when attempting to access NWCS. Corrective Action (Immediate solution): For the first occurrence recycled the NWCS environment. 12/16 No representative present. Corrective Action (Permanent solution): Under investigation. | [NAME]:  This issue was caused by a bug in the version of code I had applied to all of the Riverbed units.   [NAME]: PIR completed | 2009-12-16 00:00:00 | 2009-12-30 00:00:00 | 2010-01-19 00:00:00 |  | Not Met | isp
116 | Jan-10 | Dec-10 | 2010-12-31 00:00:00 | 1 |  |  |  |  |  | CR0197607 |  | aspna_usams_ciid | Minor |  | Create New England Region     Freeze No - '105' | Event  2 :   (ZNA) Andover, MA eZACCESS users are unable to update the   state code on accident descriptions and acknowledgements for   5 states. Severity Level: 4 Root Cause: Under investigation. Customer Impact: Affected users are receiving an error message. Corrective Action:  eZACCESS support is investigating. (Immediate Solution) Corrective Action: Under investigation. (Permanent Solution) Service Outage: From 09:57 PT to Ongoing Event Duration: From 09:57 PT to Ongoing Ticket Number: INC158268486 | Prasad [NAME]: changed a table and ezaccess use of that table is not dynamic, but rather is hard coded.    [NAME]: PIR completed | 2010-01-04 00:00:00 | 2010-01-18 00:00:00 | 2010-01-13 00:00:00 |  | Met | asp