The need to recover quickly from plant equipment failure is often at odds with the patient, methodical steps required to conduct a root cause analysis (RCA) that will be accepted by all parties to a potential dispute among stakeholders down the road. Stakeholders may include plant staff, owner/operator, original equipment manufacturers (OEMs), insurance companies, component suppliers, third-party operation and maintenance (O&M) service providers, etc. Independent experts are often brought in following unanticipated events to assist with remediation efforts and root cause investigations.
At the recent 7F Users Group conference in Birmingham, AL, Dan Getsinger and Blake Whitley of Exponent Inc, a leading failure analysis firm, offered a sampling of best practices for site personnel who are often the “first responders” to an unanticipated event. These practices namely included identifying, collecting, and preserving evidence that will become critical to a successful RCA process.
Evidence, of course, includes more than the “broken stuff.” It can refer to all the pertinent information related to an event, the conditions under which it occurred, and historical facts leading up to an event. The preservation of evidence is an important part that enables an effective RCA process.
ASTM standards exist for collecting and handling evidence. Such standards outline processes such as evidence labeling and chain of custody protocols. At the plant level, it is very important to protect evidence from damage from handling, packaging, storing, and shipping.
- Tip #1: Document every movement of evidence material, not just location to location, but including photographs documenting condition prior to and on receipt of transfer. In this case, more documentation is usually better than less.
When it comes to expert analysis of broken components, there is a wealth of information contained at the microscopic level on a fracture surface. This type of evidence will likely become evident only with laboratory analysis and instrumentation (Fig 1), and is susceptible to being obscured or even lost if not handled properly. In addition, “spoliation of evidence” (loss, destruction or alteration of evidence resulting from a party’s failure to preserve) becomes a legal liability.
- Tip #2: Do not try to “fit” fractured surfaces back together, as this can damage or even destroy the microscopic features of interest to experts investigating failure mechanisms.
Broken pieces should be packaged in ways that avoid surface rubbing, impacts, or environmental exposure which may lead to corrosion, for example. Use gloves when handling evidence and bag all pieces separately.
- Tip #3: Don’t throw anything away that could be connected to the event. Even if a component’s involvement in an event is unknown, preserving and documenting may be important.
When disassembling equipment involved in the event, consider notifying stakeholders so they can be represented to witness the process. If a party cannot be present, they still can have an opportunity to provide input. Informing involved parties of RCA and rectification efforts can prove helpful in avoiding further disputes down the road. In keeping with the theme, more communication with stakeholders is probably better than less.
Documenting the disassembly process using a pre-planned agreed-upon procedure, notes, photographs, sketches, and/or video can be useful, especially in instances when another party may have questions (sometimes after-the-fact) about the process. Consider that disputes can transpire months or even years after the event occurs, and effort should be taken so that documentation is sufficient to clearly recall the equipment condition and disassembly process long after it takes place.
Photography skills is one area all plant staff can train for (Fig 2). Here, you can take a cue from your social media. Lighting is key! Include photos from multiple angles, far away, and up close. Again, more is better.
- Tip #4: Take photos in an order starting on a large scale and then zooming into smaller details in subsequent photos. This process will aid in understanding the locations and orientation of photographs when being reviewed down the road.
Check your photos before departing the scene to make sure they are clear and complete. Photograph as-found conditions and label relative positions prior to any disassembly. The goal is to provide clarity to others who will be viewing the photos in lieu of direct observation.
Collecting, transferring, and documenting evidence should be accomplished without bias towards the results or pre-conceived explanations. The RCA process involves proposing hypotheses about what happened and then seeking to disprove them using facts and reason (not seeking to prove them). Because bias often is subtle, independent RCA experts are often brought in to carry out objective, independent investigations of the evidence. Utilizing independent experts can improve the trust of the various stakeholders during the RCA process.
Similarly, notes should be around “facts” and direct observations only, not guesses, not assumptions, not emotional taint. Consider that communication, especially written-forms, like notes, emails, logs, and reports, should be thorough but ideally focused only on the facts.
Evidence can also mean O&M procedures, plant flow and piping and instrumentation diagrams (P&ID), and teardown and rebuild instructions. Disciplined and formal change management, document management, and version control processes at the plant will be helpful later (maybe even many years later) in RCA scenarios. The current state of the plant equipment should be captured and previous versions of the documentation retained. In situations in which sites do not record or maintain vital information about work performed (or not performed), it can detract from the RCA process.
Other examples of evidence include observations on, and the conditions of, spare or “twin” equipment, manufacturing records and certificates, maintenance and repair records, samples and analyses of fluids handled (before and after the event) by the equipment, and witness interviews.
Perhaps the most important takeaway is this: everything done following an incident will potentially come to light during an investigation, especially under a dispute situation, and could eventually be exposed to adversarial parties if the event develops into litigation. Imagine yourself in a courtroom proceeding concerning an event and consider your current processes and actions in light of such a possibility.
Sometimes it may not be obvious whether an event will become a dispute or litigation. Plant and owner/operator personnel must sometimes balance the demand to get back on-line with the value of proceeding with a detailed RCA process. Under this situation, more documentation (photographic, written, operational data, formal change management processes) is generally better than less, along with communication with the known stakeholders. Independent failure analysis experts can help owner/operators work through unique situations and various judgment calls.
The presentation slides from the 7FUG conference, accessible to registered users at www.powerusers.org, review the fundamentals of RCA processes, all based, in some fashion, on the scientific method that includes defining the problem, collecting the data, forming hypotheses, testing hypotheses (to disprove), and settling on the root cause. Brief case studies are also presented to highlight instances of bias, what happens when proper evidence handling procedures are not followed, and how to photograph, document, and analyze damaged components and systems.
For further reading, the slides also reference standards and guideline documents for conducting an RCA, evidence handling, labeling, and chain of custody — NFPA 921 (2024), API RP 585 Pressure Equipment Integrity Incident Investigation, DOE-NE-STD-1004-92 Root Cause Analysis Guidance, and ASM Handbook volumes 13C, 11A, and 11; and ASTM E1492 and E860.
Just as plant personnel are all trained in how to respond to emergencies like fires, chemical spills and explosions, owner/operators should consider formal training programs for “first responders” to the aftermath of an unanticipated event. This training would improve proper evidence collection, handling, documentation, etc., that can be used in a subsequent RCA or could be integral if the event becomes a dispute. CCJ
