Description of Event:
The unit 5 “LoLo Spring Energy – Block Trip” alarm annunciated at 19:06 indicating a loss of breaker trip functionality. Also at 19:06 the “Battery Ground Detected” alarm came in. The hydro operators verified the alarm and executed a plan to unload and isolate unit 5 by transferring station service loads, shutting down the adjacent unit (unit 6) that shares the river crossing and opening the river crossing (line) breaker. Unit 5 was then operating unloaded and isolated from the bulk electric system (BES) and station service equipment.
Once unit 5 was unloaded and isolated from the BES and station service equipment, the stop command was issued to the unit. The generator breaker failed to open, as expected, and the unit 5 breaker failure element operated (21:10). The breaker failure trip did not affect any generation, as all the affected equipment had been previously isolated in preparation to shutdown unit 5.
Unit 5 was isolated and placed under clearance (21:20) by opening the step-up transformer, high voltage disconnects. Unit 6 was then returned to service (21:36) while unit 5 was left shutdown for generator breaker investigation and repairs.
Actions Taken:
Maintenance personnel investigated the generator breaker and found that 33Tx, breaker lolo spring block trip dc breaker control relay had overheated and shorted to ground. The damaged 33Tx relay and the two other relays on either side which also had evidence of heating required replacement. The damaged relays were replaced and tested. The trip circuit functionality was also tested.
The generator breaker was not able to operate during the event due to the damaged control relays. The protection relays operated as expected.
Unit 5 was restored and made available on 11/12/2020 at 15:30.
Method & Findings:
The three relays that were overheated and damaged include the low low spring energy block trip relay (33Tx), low low SF6 density block close/trip (61Gx) and low spring energy block close, (33Cx). These relays are normally energized when the breaker is closed. When the 33Tx failed, it operated the block trip function as designed (fail safe). A review of records indicated that the 33Cx relay failed in the same way on unit 6 five years previously. These relays are located in the upper part of the control cabinet, in the center of the line of relays. They were installed when the breakers were converted from air-blast to SF6 roughly twenty five years ago.
The conclusion is that the relay is normally energized and generates heat when the breaker is closed, the unit on- line. Over time this heat causes the insulation to breakdown until the relay fails. In this case, the failure caused damage to two adjacent relays as well.
Supporting Materials:
Corrective Actions:
The damaged relays were replaced and the unit returned to service. The same vintage and design is used in the other nine units in the plant. New relays are being ordered for all breakers and will be replaced as unit outages allow. As part of the asset management process, replacement of these relays will be included in the 15 year actuator overhaul work.
Sequence & Completion Dates for Actions Listed:
Damaged relays replaced and the unit returned to service on 12 November.
Relay replacement on the other units will occur over the next year to two as outage schedules allow.
15 year replacement will be added to the Generator Breaker AM Strategy in December 2020.
Lessons Learned:
Industrial DC relays are extremely reliable. However over time heat can degrade them to point of failure. In many cases, this run to failure is the appropriate path. In the case of the generator breakers on critical units, the forced outage indicates that a proactive maintenance approach may be the better path.
Breaker failure protection is valuable on generating units. In this case, it was not needed as the operator recognized that the generator breaker trip was blocked and used an alternate path to shutdown the unit.
Good operator knowledge and training resulted in an orderly transfer of station service power and unit shutdown minimizing impact to the plant and energy production.
The Maintenance management system is not configured such that failures like this relay on unit 6 are tracked to inform maintenance and asset management. It is only through recollection of the plant engineer that the previous event was identified.
Spare parts are valuable. It was possible to return the unit to service quickly because spare relays were available in inventory.
Recommendations:
2. Look at maintenance management systems to create alerts or tracking of critical components so failure trends can be identified and mitigated.
3. Maintain adequate spare parts to mitigate risk of critical relays.