You’re an experienced operator/technician, perhaps with 10, 15, 20, or more years of powerplant service. You’ve seen it all. Nothing would surprise you. So you think.

Here’s a case history likely to tame your unbridled confidence.

A GE F-class unit, equipped with a Mark V control system is humming along at base load, no issues, reliable unit, nothing much for operators to do. Next instant: All lube-oil pumps—AC- and DC-powered—are forced out of service. Bearings are destroyed in a heartbeat (last reading on bearing-metal thermocouples was 950F) and rotor grinds to a halt severely damaging the compressor and turbine sections.

Say what?

Abel Rochwarger, chief engineer at Gas Turbine Controls (GTC), consulted with the owner/operator. A team of customer and GTC personnel analyzed the root cause in this case and deemed it irrelevant. Instead, the team engineered a solution to prevent the control system from ever shutting the lube oil pumps again without operator control. CliffsNotes: The team learned that the network connecting the turbine control panel (TCP) and human/machine interface (HMI) malfunctioned, forcing logic (sidebar) without human intervention. Tens of signals were “forced” within the space of 0.1 sec.

As you read on, keep in mind that the 7F Users Group’s 2017 conference starts May 15 in San Antonio’s La Cantera Resort & Spa. That’s the ideal venue to discuss the findings of the accident profiled here with colleagues to determine if your plant might be at risk. Plus, GTC will be participating in the vendor fair on Tuesday evening (Booth 36) if you want to dig deeper into its solution.

Rochwarger told the editors that GTC was unaware of any other instance in which “this behavior of ‘self-inflicted logic forcing’ occurred and forced all lube-oil pumps out of service.” He continued, suggesting units that have not yet implemented a protection scheme for the lube-oil system against a TCP failure should consider the following recommendation:

If the TCP starts to do “strange things,” such as unexplained logic forcing, immediately put one lube-oil pump in “manual.” To protect against an AC failure, start the emergency pump, too (it should latch and stay on). Shut down the unit immediately. If the cooldown sequence does not engage, turn the unit manually.

The chief engineer cautioned against accepting the results of a root-cause analysis as a vaccine against all ills. He said, “Eliminating the proximate cause of this failure does not necessarily eliminate all other potential situations that may result in the same scenario. The fact that the TCP failed to protect in this instance suggests that there might be other—today unknown—sets of circumstances in which the TCP would not keep the lube-oil system running when needed.

Backgrounder

Logic forcing is a feature in modern (electronics based) turbine control panels allowing the operator to force the logic state of a digital (binary) variable to “0” or “1” independently of the following:

• • • The logic state mandated by the control algorithm (that is, even in contradiction), and

    • The status of the unit (online or offline).

This feature may pose a significant risk to personnel and property; therefore, OEMs restrict the access to logic forcing via password protection.

“The initial assessment indicated that the network connecting the HMI to the TCP was ‘overloaded with signal traffic’ beyond its design capacity. The underlying problem: One TCP version did it.”

Rochwarger challenged those who might say “problem resolved” with the following question: How can you be certain there are absolutely no other combinations of circumstances that would result in a similar condition? He pointed to the fact that later versions of the TCP, such as the Mark VI and Mark VIe, are based on their predecessors, as the OEM points out in its literature.

This raises a second question: Could this possibly mean all TCP generations that followed the Mark V may have carried over the design patterns that allowed the “self-inflicted logic forcing” to happen?

The point stressed by Rochwarger is: not to split hairs on what may or may not happen, or to rush towards an expensive and unnecessary upgrade, but to eliminate the possibility by changing the controls paradigm. Until the event described above, the TCP controlled the starting and stopping of the AC and DC lube oil pumps (and seal oil pumps if installed). The controls paradigm he suggests and the one implemented for the affected customer (refer to simplified conceptual diagram), who agreed that preventing further occurrence was much safer than upgrading to new unknowns:

AC lube-oil pumps

• • • Allow the TCP to start the AC pumps.

    • Do not allow the TCP to stop the AC pumps, but enable a manual stop.

    • Operator intervention is required to stop these pumps.

DC emergency lube-oil pump

• • • TCP enables pump to start; lube-oil pressure controls the start.

    • TCP cannot stop the pump, but enables manual stopping.

    • Operator intervention is required to stop the pump.

    • TCP is allowed to cycle the pump to cool bearings when required at zero speed.

The AC auxiliary and DC emergency seal-oil pumps (not shown in the diagram) require similar logic changes if installed.

Wrapping up, Rochwarger said the sequencing, hardware (external to the TCP), and wiring modifications required by the GTC alternative are not difficult to implement. The company says its solution offers a higher level of operational safety in case of a TCP malfunction, regardless of the TCP model. Plus, the same solution can be adapted to steam turbines and to synchronous condensers. Finally, a similar controls scheme has been developed by GTC for B/E-Class machines, like the Frame 5, 6B, 7EA, 9E, etc. with a mechanical main lube-oil pump.

You can contact GTC at www.gasturbinecontrols.com or 914-693-0830.

DIG DEEPER: Updating the Mark V communication interface overload issue