A presentation entitled “Does Your Backup System Work—Really?” during Week Three of the Combined Cycle Users Group (CCUG) 2021 virtual conference reminds why user groups can be worth their weight in gold. The presenter, asset manager for a 600-MW, 2 × 1 combined cycle with Siemens 501FD gas turbines and a KN steam turbine/generator, reviewed the root-cause analysis for a catastrophic ST/G failure.
Bottom line: The failure of a $400 relay “wiped out the ST/G” causing “a seven-month outage and tens of millions of dollars in lost margin and equipment repair costs.” Many cliches are represented in this analysis—sweat the small stuff, trust but verify, only the paranoid survive, etc—but perhaps the most important one is “we are all stronger [as an industry] when we learn from each other.”
Some background first: Plant is equipped with 230-kV main transformers. It was designed for auto transfer of ac power to dc, and therefore has limited ac backup from the alternative feed breaker. A 5-kV, 850-amp backup supply comes from an adjacent facility within an eight-second delay. The dc lube-oil pump is supplied with 125-V backup power and a test of the backup supply is included as a permissive in the start sequence.
Sequence of events. After accidental closure of the wrong breaker in the switchyard and loss of station power, the entire plant tripped while operating at baseload. The gas-turbine lube-oil pumps rolled over to dc power and shut down normally, the BOP equipment also came to controlled stops, but the ST/G slowed from 3600 to 0 rpm in about five minutes without the start of its dc lube-oil pump.
During the subsequent restart, the ac pumps—lube oil, boiler feed, circulating water, and aux cooling water—did not turn on, and the dc lube-oil pump did not receive a start signal. Backup from a second alternative source of power did start the ST/G lift-oil pumps, but resulted in a loss of primary containment and a minor lube-oil release in the ST/G building.
Digging deeper. Troubleshooting, including a hand-over-hand wiring check, revealed that an “on” relay was not burned up or damaged, but just failed to operate (a forensic analysis is being conducted to determine why). BOP ac pumps with hands-off auto switches set to “auto” attempted to restart after the trip, but overloaded the feeder, opening the breaker for the backup from the adjacent facility, and tripping the pumps. The second alternative power source, called “construction power,” was found “not fully utilized.”
A deeper review of critical systems revealed several single-point vulnerabilities (SPV) in the ST/G dc logic, but none in the gas-turbine logic. A review of pump load data showed that the total was higher than what the backup ac system was designed for. Learn more about single-point failure vulnerabilities from GTC Control Solutions’ recorded presentation from the 7F Users Group on the subject.
“Circuit heads” will want to take a close look at the two slides comparing the original emergency dc lube-oil pump logic and the revisions, addressing the SPVs. Briefly: A redundant pressure switch was added, coming off a separate existing heater tap (three-way isolation valve). The hot leg is split between two starting contactors with the neutral directly wired, and industrial control relays were replaced with solid-state relays and placed in parallel.
These additional changes also were implemented:
-
- Changed ac logic so only the ac lube-oil pumps and vapor extractors restart, and eliminated the auto restart of the boiler-feed, circulating water, and aux cooling-water pumps.
- Reviewed all facility schematics and modified for redundancy where appropriate.
- Added construction power backup to all lube-oil and lift-oil pumps, vapor extractors, and battery chargers on the ST/Gs and GT/Gs.
- Arranged to test both circuits automatically during each start sequence; a failure of either relay or pressure switch results in a failed start.
- Emphasize pressure-switch calibration and testing as a PM during outages.
- Determine whether dc lube-oil system can be tested online (in process at the time of the meeting).
Best practices. The most important recommendation to others is to validate control and automation logic, especially backup systems, in operation. Also, if your ac logic has auto-start, calculate total loads to make sure an overload won’t cause a trip, and test the load offline, Finally, review your dc lube-oil systems for SPVs and create, think-through scenarios where primary and secondary losses of power occur and identify contingencies.