Description of Event:
At 1758 on Sunday June 10, 2018, while Alpha Pump Storage Plant (Alpha) had all three units offline, Unit 2 tripped on 230kV cable differential relay. Unit 2 was being brought up for evening load by Area Operations Center when a fault occurred, tripping and locking out Unit 2, 230kV Circuit Breaker (CB), CB280 prior to closing. Line tie breakers CB 250 and 260 tripped immediately following the event, and then reclosed as the Operator went to look at the Sequence of Events (SOE) Recorder. The operator reported that the event happened just prior to Unit 2 reaching synchronous speed and no indication of breaker closing was identified. Investigation confirmed that CB 280 had never closed and that any risk of damage would have been isolated to the generator side of the breaker.
It was not immediately apparent that the fault had occurred inside CB 280. Hundreds of alarms came into the control room due to two SOE cards being destroyed by the fault. Testing of all equipment from the generator to the breaker in the line of the fault revealed no damage or indications of the cause of the fault, nor the location of the fault. It was not until the SF6 gas sample report for CB280 came in that the location was determined. The report revealed high levels of CF4 within the SF6 interrupter tank, which is indicative of an internal arcing. CB 280 was degassed and arcing was indeed discovered between all phases and the grounded tank. CB 280 was replaced by a spare breaker located at Rocky Substation and the faulted CB 280 was shipped to OEM’s factory for root cause investigation and rebuild.
Actions Taken:
Immediate, Compensatory, or Interim Actions
1. Two SOE cards were immediately replaced and that system was restored.
2. Testing began into discovering the location/cause of the fault and if it had affected adjacent units.
3. Failed equipment was identified as breaker CB280 and it was replaced, restoring U2 to operation.
4. Failed CB 280 sent off to OEM for investigation.
5. Overhaul of failed CB 280 began.
6. Replacement of rebuilt CB 280 scheduled.
Method & Findings:
Problem Statement
Dielectric failure of 230kV Circuit Breaker CB 280 resulted in extended forced outage and equipment damages due to fault.
Requirement or Expectation: 230kV Generator Circuit Breakers should be capable of withstanding thousands of open and close cycles. The dielectric strength of the breaker is designed such that it can withstand the two times voltage peak value (180 degrees out of phase) that occurs just prior to the generator and line coming into synchronism and prior to paralleling. The breaker is also designed with a safety factor beyond the two times dielectric strength.
Deviation or Defect: Alpha 230kV Generator Circuit Breaker CB 280 failed in service by dielectric breakdown of internal component during generator synchronizing and prior to parallel.
Consequences of Deviation or Defect: Loss of generation to all three units until determination was made that non-involved units could be returned to service. Loss of generation to Unit 2 for eighteen days while a new breaker was installed. Major equipment damage and resource allocation. Two SOE cards failed because of the incident.
Significance of Deviation or Defect: Summertime extended forced outage of 18 days. A significant expense was incurred due to equipment damage and troubleshooting. Allocation of emergency resources to get the units back online. Refurbishment of the failed breaker as well as placing it back in service, essentially replacing the breaker twice since the original replacement breaker was only temporary.
Extent of Condition:
1. Alpha has three units and all three depend on the same OEM XZ style breaker with a common tank design (all three phases in single tank) used for generator synchronizing application. These breakers are overhauled every 2000-2500 cycles which usually falls within approximately every three years. These breakers see extreme duty because of pumped storage cycling application.
2. Beta Powerhouse (Beta) has the same breakers as Alpha (CB270, CB280, and CB290) used in the same application, however, Beta does not see the number of cycles that the Alpha breakers experience. Beta breakers only operate a few open and close cycles per year. The Beta breakers were refurbished/overhauled in 2015.
3. Alpha has two additional breakers for the intertie, but these breakers do not experience generator synchronizing or high cycling and have a dedicated tank for each phase interrupter.
4. Other powerhouses in the system also use high voltage breakers as generator synchronizing, but they fall in line with #2 above like Beta Powerhouse.
Cause Analysis
The protective relay that initiated the trip was 230kV cable differential. Technicians and engineers began analyzing the information from the relays. From the cable relay data, the “A”
phase had a 230kV fault current of 3720 Amps found on the U2 cable differential relays located in the switchyard. The max generator phase to phase (18kV) fault value from the relays was
30,035 amps.
Data found in the Cable Differential Relays in the switchyard shows that the A phase faulted to ground followed by C phase. This coincides with an A-B fault shown 180 degrees apart on the
XYZ (11GA-2) and (11GB-2) Generator Relays looking at the 18kV Generator side. The gen management relays also show the frequency was 59.4 hz and that the 230kV breaker (CB 280) never closed. The unit was Autosyncing and the synch check protection was enabled. The unit was in remote Auto being controlled from Area at the time of the event. The fault recordings show that the phase to ground fault developed into a 3-phase fault in about
350 Msec.
Supervisory Control and Data Acquisition (SCADA) records, generator management relays, and local Operator all agree there was no indication that the breaker ever closed. Operator reported that the unit was at 355 rpm (360 rpm is the sync speed). The unit had field and full voltage, preparing to parallel. The 230kV breakers located at Rocky Substation on the other end of the line did not trip, nor was any information present in the Rocky line relays that indicated an event. The SOE (located in the powerhouse) burned up two cards that referenced the switchyard alarming, and many
false alarm indications came in. The relevant alarms that came in was the U2 cable differential, and both U2 Gen Management relays (distance element as well as neutral overvoltage). These are consistent with the fault within CB 280.
After all testing had passed, the fault was narrowed down to two suspect locations.
1. In the switchyard, on the generator side of the breaker; indications of welding and an arcing were found on the structural steel that supports A phase “Switchyard Jack Bus”. Since A phase was utilizing the spare powerhouse cable routed for use in 2A at the time, it was possible this bus faulted to the steel structure, external to the breaker. At the time of discovery, it could not be determined if this indication area was from a previous event, or if it had occurred during this event, even after close inspection by personnel on a man lift. If this area had something come in contact between the bus and structure it could have burned up leaving no evidence behind and a passing test. This area was ruled out about two weeks after the event when a photo was found that showed the arcing indication existed on the structure in 2012. This section of the structure was painted over to avoid confusion during future inspections.
2. Generator Breaker CB 280 itself, on the generator side only. All standard testing on this breaker passed. Opening the breaker up to look inside is not an easy task as provisions need to be made to have a manufacturer representative onsite with SF6 gas extraction equipment available and this equipment was not immediately available. It was known that the fault started out as an A phase to ground fault (with CB 280 open), the cable differential picked up the 86E tripping relay in 3 cycles, however with the generator field active, the generator continued to feed the fault for 2+ seconds.
At approximately 20 cycles into the fault, the single phase to ground fault evolved into a multi-phase fault, later to be found within the breaker. Relay targets were discovered on the 230kV bus differential relay. CB 280 was found to have stayed open and no relay events were noted at Rocky Substation, the end of the line for Alpha. This information led to the generator side of CB280 as a possible fault location. It should be noted the 230kV bus differential is an electromechanical device with no event recording capability, and there were other relay targets at the switchyard that were not reset from previous operations or testing which caused some confusion. Once the SF6 gas sample came back positive for CF4, a by- product of arcing, it was apparent the fault had occurred inside the breaker.
Apparent Cause and Contributing Causes
AC-1 Contamination – Unseen contamination in the weld area of the bushing pocket of A-phase caused a flashover between ground and A-phase high voltage component.
Contributing Cause(s):
CC-1 High Cycling – Heavy use of SF6 230kV breakers at Alpha due to cycling of Pumped Storage Facility may have led to particulate contamination in the failure area that went un-noticed through its recent overhaul cycle and eventually accumulated or moved into a position allowing flashover.
Note: High Cycling not to be confused with amount of cycling since last overhaul which was 127 start/stop cycles, but with the number of cycles this breaker has experienced overall, which could be nearly 20,000 or more.
CC-2 Lack of detailed inspection procedure – OEM’s inspection procedure did not have the detail necessary to ensure the area that failed is thoroughly inspected each time an overhaul takes place.
Each breaker opening under load (pump mode) creates a small amount of SF6 by product that can eventually contaminate surfaces to the point of flashover. SF6 byproducts are cleaned and removed from the inside of the tank by specialized utility insulation and coating employees and inspected by OEM representative on each overhaul. This byproduct has been known to cause breaker failures in circuit breakers. Another pumped storage facility that has recently experienced this type of breaker failure was also 20kV class SF6 generator breaker.
Major overhauls are scheduled every 2500 open/close cycles or less per OEM manufacturer recommendations with utility Maintenance Personnel and oversight from an OEM vendor specialist. Once the tank is cleaned by utility insulation and coating specialists, the tank is inspected. The location where this flashover occurred is difficult to inspect due to its overhead location inside of the tank and visual obstruction from the shape of the corrugated material. It is
likely that a foreign particle went undetected in this case. This breaker was last overhauled less than two months prior to the event with 127 successful operations prior to failure.
Utility staff worked closely with OEM after the failure to determine why the breaker failed and how this area could have been missed. It should be noted that once the breaker was determined to be the source of the event, photos were taken to document the visual condition, however the immediate plan was to overhaul the breaker to place it back in service. The insulation and coatings team began to clean the failed breaker until damage to the steel tank was discovered at which time all overhaul attempts were halted and the breaker was sent to the OEM facility for investigation and refurbishment and a replacement breaker was installed. Without enough evidence to conclude with certainty what the contamination was or what it looked like, the investigation could not conclusively determine the size or type of particle or the cause with certainty.
Supporting Materials:
Corrective Actions:
Corrective Actions for Apparent and Contributing Causes
Corrective Actions:
CA-1 Closely inspect all areas of tank for foreign material at every overhaul. Close attention to inspection of the bushing pockets. Follow OEM inspection procedure with sign off for accountability.
CA-2 OEM to revise inspection checklist with possible addition of technology such as magnification to assist in inspecting hard to access locations.
CA-3 OEM to develop a process to document any particles/findings post clean-up and will be added to inspection checklist.
Sequence & Completion Dates for Actions Listed:
Corrective Action Matrix
CB 280 Failure -Corrective Action Matrix
Lessons Learned:
Recommendations:
1. Include inspection of all areas of the breaker tank for foreign material at every overhaul.
2. Include the OEM inspection procedure at every overhaul and include a sign off for accountability.
3. The routine inspection should include the use of technology for the inspection of hard to access areas.
4. The routine inspection should include documentation of any particles/findings.