You want to believe the lifecycle care program purchased with your control systems transfers most of the risk to your OEM. After all, much like an insurance policy, you’re paying for protection against technological obsolescence and maintenance issues. But the reality, as illustrated here through one utility’s experience, is that successful implementation requires considerable owner/operator-OEM interaction and oversight.
What follows is derived from a presentation at the 2016 Ovation Users Group (OUG) Conference, held in Pittsburgh, July 25-28, plus an interview with a utility representative who recently concluded a program to upgrade systems at 13 separate power stations. However, it should be clear that the basic practices can be applied to any control-system lifecycle program.
Emerson Process Management calls its lifecycle care program “Evergreen.” In one sense, it is analogous to an LTSA or CSA for turbines and generators because it not only addresses basic maintenance issues but technology upgrades and new software/hardware releases as well. This is critical because digital and control-system technology has a lifecycle of five to 10 years—far shorter than the physical systems. Components include (depending on what is covered in each program) network switches and media converters, controllers, workstations, and power supplies.
On the advanced technology side, control systems are becoming “knowledge platforms” and so Evergreens also include the latest operator applications and algorithms, security programming, wireless capabilities, embedded process historians, model-based control, diagnostics, trending, and decision support. Some of these identified in the presentation are:
- Updated turbine, boiler, and excitation control schemes.
- Updated remote valve positioners (RVP).
- Ovation Security Center (a means of constant interface between user and OEM to attend to cybersecurity and patch management while the plant is online).
- Machinery health monitor and trending packages.
- HART and wireless HART for communication with field devices.
- Replacing old serial link controllers (RLC) with Ethernet link controllers (ELCs), for communication with virtually all other third-party devices and changing of communication protocols without upgrading.
From the OEM side alone, much preparation is required (figure). The scope has to be agreed on, systems have to be walked down and inspected in the field, the “system state” has to be reviewed, the factory acceptance test (FAT) has to be planned, hardware and software have to be tested, and a bill of materials has to be written up. Control systems interface with other digital assets so other party licenses have to be reviewed. According to Emerson’s Mike Brown and Chris McClellan, “there are at least 15 documents for every Evergreen upgrade, part of the internal audit, which must be signed off on by customer and OEM.”
Control-system alarms have to be carefully managed to prevent alarm flood to operators. Therefore, owner/operators need an alarm-management philosophy incorporated into the upgrade. If the upgrade includes replacement of the old Q-line I/O with Ovation I/O, then you need an accurate list of every Q card in every card frame in the system, including the attributes location, type, and group.
User perspective
According to Alliant Energy’s Wes Whitley, also OUG president, NERC’s Critical Infrastructure Protection Standards (CIPS) were the driving force to upgrade the utility’s entire powerplant fleet through Evergreen at a cost of millions, 5% to 10% of that for software upgrades. His experience was greatly assisted with checklists and other tools developed through the OUG executive board.
The scope of this program should not be under-estimated. “The only thing that remained the same was the physical I/O,” Whitley stressed, “and the screen graphics and control sheets.” Across the fleet, the scope included new Dell computers, replacement/conversion of older controllers with the latest versions, new controller software, new I/O software, new firmware to interface with the controllers, new monitors, new domain controller for NERC-CIPS compliance, anti-virus servers, and a new process data historian.
Let’s face it, it’s almost a new control system, and a testament to how fast the technology advances.
Identified below are a dozen areas Whitley suggests you consider for your best practices checklist. In his experience, controllers, system registrations, and other items were captured well, but power supplies were not. Seven of them were missed at one plant. Circuit breakers and fuses also are easy to overlook. He says you can’t be thorough enough when putting together the checklist.
1. Software freeze date. There could be three to four months between when the as-configured backup was done and when the new software was installed. Emerson has a tool to compare control sheets.
Recommendation: Run the compare tool just before you take the old system down; it will show every difference.
2. Controller consolidation. Ask questions like, “How much capacity is available in the current controllers?” and “What points come out of the controller for the upgrade?” There’s the potential to reduce the number of controllers (and corresponding power supplies).
Recommendation: Run the Emerson program which shows how every controller is connected to the one under consideration, or at least gain an intimate understanding of this in other ways. Good money is at stake when considering future upgrades, both in hardware and software costs.
3. Setpoints. According to Whitley, Ovation doesn’t store setpoint algorithm values, not even the most currently used. This is a critical configuration management issue. The more you automate, the less operators remember, even the name of the setpoint, much less the value.
Recommendation: Make sure you have a record of all your set-point values. Adding all setpoints to the historian is a good idea for both Evergreens and to trend changes in equipment operation.
4. Dress rehearsal. Part of Evergreen is to upgrade controllers. This may include media converters (copper to fiber, fiber to copper), power supplies, and other attributes. You should know what the OEM plans to do with each component of the system.
Recommendation: Go to the OEM’s field office and review where things are located, what will be done with each device, etc—basically a complete dress rehearsal. This can lead to overlooked cables or other equipment, time constraints that may affect other outage tasks, resource deficiencies, illogical installation order, etc. Be sure to understand the needs of other outage support such as equipment that needs to run, etc.
5. On-scan vs off-scan points. Some rationale needs to be imposed on which points will be scanned and which ones won’t. Otherwise, operators may suffer with nuisance alarms. Limit switches must be checked as well. Some may have failed without anyone (or the right people) knowing about it. Alliant found upwards of 50 suspect off-scan points at one unit.
Recommendation: Review every point for its value, if it is not being scanned; fix points/switches/field devices as necessary.
6. Reconciling control sheets. All tuning, timers, setpoints, and deadbands have to be reconciled in the controllers. Temporary changes have to be accommodated. Alliant found 200 points not reconciled at one plant.
Recommendation: Conduct a control-sheet audit onsite.
7. Ovation Control Builder (OCB) points. These can connect to the graphics on the HMI screens. The Evergreen upgrade process can generate new OCBs which can cause graphics errors. An Emerson tool is available that searches for every OCB point connected to graphics.
Recommendation: Create names for every OCB point used on graphics, rather than relying on, searching for, and replacing new OCB points.
8. Licensing. Users are entitled to a certain number of points in the historian; however, you can’t discover that on your own, Emerson has to tell you how many you are allotted and how many you have used. Licenses from others, like the PI database and Emerson AMS Device Manager (which provides intelligent diagnostics for individual field devices), have to be considered.
Recommendation: Review the Ovation licenses (especially the historian and ELC) as well as all third-party licenses.
9. IP addresses. These are critical for NERC CIPS compliance. Lots of work is involved if you enter an incorrect address.
Recommendation: Be overly cautious with IP addresses and provide any proposed changes to Emerson as soon as possible. At a minimum, notify Emerson of impending changes at the kickoff meeting.
10. Workstations. There may be redundant operator workstations and an opportunity to consolidate or reduce them. New graphics cards now available allow multi-window viewing on HMI screens.
Recommendation: Be careful not to put Network Interface Cards (Ethernet) in the full slot. Quite often the HMIs have only one empty full slot and using this for a NIC card will make it difficult to add a multi-window graphics card because the Ovation Highway Interface may need to be fully uninstalled, which can be very time consuming.
11. ELC cards. Users can’t generate mass database files for an ELC card (third-party devices, PLCs, etc). This could result in the user having to add hundreds of points.
Recommendation: Set up ELC cards prior to the FAT.
12. Network switch configurations. These may change because of NERC CIPS.
Recommendation. Fully discuss new network topologies at the kickoff meeting.
Once the upgrades are complete, retain all the results of the site walk-down for future Evergreens. Some components, such as power supplies, are not captured via software. Special or non-standard equipment may not be captured either. Keep a running log of changes/additions/subtractions, etc, to minimize the potential for errors next time. Include notes such as reduction of controllers, HMIs, I/O cards, ELCs, etc. This provides a history of what was changed and why.
These practices were very successful in upgrading Alliant’s entire fleet. At some plants, Evergreens were implemented with as little as four hours of downtime on a four-unit network.
