Case Study: Palo Verde Nuclear Generating Station

The woman behind the chest-high glass partition at the Palo Verde Nuclear Generating Station dons a fresh pair of protective gloves before beginning her next work order. She gingerly moves a fragile molded shell into place onto a ceramic dish. After acknowledging the customer’s stated instructions, she precisely measures and alights each of the requested materials into the shell. A viscous dollop is gently positioned on top. It is then presented to the customer for full testing.

In the final analysis, it is determined that the plant’s cafeteria makes an outstanding taco salad.

“Everything is done to a high degree of precision at Palo Verde,” says Bryan Johnson, a lubrication engineer at this 4.2-gigawatt electrical power plant site, located 45 miles west of Phoenix in the town of Tonopah, Ariz. “That cafeteria worker was so incredibly precise. It’s an example of just how precise everything is here. That’s our culture. She was going to make the best taco salad in the world for you, and I’m sure she is that way for everyone who places an order at her lunch station.”

Employees at the Palo Verde nuclear plant admit that they apply an uncommon (perhaps over-the-top) attention to detail, and for good reason.

“The nuclear industry puts a premium on accuracy,” says Johnson. “We have a very detailed and precise design process. We are very careful and specific in our decision process. This results in operations and practices that are more precise and exact, and in a lower risk of a mistake occurring.”

That is important because, outside of taco salads and other cafeteria fare, this is a particularly serious business.

“The risk of not doing it right the first time can be significant,” says Brad Berles, the department leader for component performance

engineering. “We are protecting the health and safety of the public within a 10-mile radius and beyond. We have a community that relies on us to do it right the first time, every time.”

Attention to detail permeates processes, practices and procedures – in core operations, electrical distribution, safety systems ... and (the focus of this article) machinery lubrication.

In-house oil analysis that is as all-encompassing as most any vendor. Testing procedures that are put under the microscope as much as the oil and grease samples. A robust intranet site that can tell you exactly how many strokes of grease were pumped into a given zerk (or how many gallons of oil were added to a sump/reservoir) at any tracked moment in time.

“Some people may argue that what we do is excessive,” says Steve Lopez, the site’s section leader for predictive maintenance engineering. “Others will find that it is right on the mark.”

Read on and you can decide.

Lube Adds Life

Unlike some of the plants that have been featured in a cover story for this publication, Palo Verde is not new to experiencing the power and benefits of lubrication excellence. The Arizona site has been on the journey for more than two decades. In 2001, it was the inaugural winner of the International Council for Machinery Lubrication’s Augustus H. Gill Award for Oil Analysis Excellence. And for much of the past decade, elements of Palo Verde’s lubrication program have been recognized as best practices in the nuclear power industry.

Johnson, who has worked at the plant for 20 years, all of those in the role of lubrication program owner, rates the overall lubrication and oil analysis program as “an 8 or a 9” on a scale of 1 (low) to 10 (high).

“I believe that a really effective lubrication program can incrementally improve the mean time between failure or service life for all of the machines that it touches, be that through using the best lubricant available, the proper amount, the right monitoring,” says Johnson. “I believe that this program will add life to the machinery, which will add value to the plant.”

Adds Lopez, a site engineering leader since 1991, “Lubrication is one of those foundational things that can have the biggest impact on equipment health and site performance. Our management recognizes that.”

The lube program – along with fellow predictive maintenance technologies such as vibration analysis, infrared thermography and motor current analysis – allows the plant to attain high levels of runtime and maximize equipment life.

“We avoid taking a machine apart by employing various technologies,” says Johnson, who works closely with the engineers who oversee application of various predictive maintenance tools. “It is equally important to use PdM data to determine that a machine is running properly as it is to find a developing failure. We rely on sensory information from others to extend the value of PdM. Our skills and tools are often invited to determine the significance of extraneous input. We have many examples where we used, for instance, lube and vibe to look at a machine and decide that it is running perfectly fine. We are able to tell the plant that there is no reason to pull the string on this other information because the machine is functioning properly. That type of a save is more common than shutting down a machine before it comes apart at the seams.”

Says Berles: “PdM allows us to minimize how frequently we go into equipment for preventive maintenance. The fewer times we have to put our hands on equipment and do maintenance on it, the better off we are, especially when we have technologies that will show us if the equipment is performing as expected in its current condition.”

 

Predictive maintenance technician Mary Holtzclaw, who has worked 10.5 years in the Lubrication Lab, says: “We are accessible and have a quick turnaround time. If you had to rush a sample to an off-site lab, that could cost several hundred dollars, plus rush shipping.”

Predictive maintenance technician Darlene Lang is a 23-year veteran at Palo Verde. She has worked in the Lubrication Lab since May 2006.

Testing ... Testing

Palo Verde’s longstanding adherence to predictive maintenance and attention to detail is quite visible in its Lubrication Lab. In existence since 1992, it houses some of the most complex and intuitive analysis equipment on the market as well as a crew of trained and dedicated professionals.

“It is really something,” says Lopez. “I don’t know how many facilities have a lube lab like we do. We know it gives us an advantage.”

Laboratory instruments (including an Olympus CH2 Ferroscope IV, Stereo Star Zoom Microscope, Nicolet Magna 560 FTIR Spectrometer, Rheometric Scientific Universal Stress Rheometer SR5, Koehler K 19400 High-Temperature Dropping Point Apparatus, Standard Oil Dual Ferrograph Analyzer, Spectroil MC Spectrophotometer, Mettler DL25 Autotitrator, EM Science AquaStar C2000 Titrator, Pacific Scientific 80000A, Pacific Scientific AB52 Bottle Sampler, Koehler K70400 Oil Bath, micrOptix i-Lab S560, InfraCal Soot Meter, Koehler ASTM D-1404 and others) are used to perform the following tests:

• Acid number by color indication
• Water concentration
• Emission spectroscopy
• Rotrode emission spectroscopy
• Kinematic viscosity
• Fourier transform infrared spectrometry
• Flash point, Cleveland Open Cup
• Base number by autotitrator
• Grease dropping point
• Particle count
• Rotating pressure vessel oxidation test (RPVOT)
• Filter residue (patch test – weight/particle characterization)
• Wear particle analysis
• Rust-preventing characteristics
• Deleterious particle determination
• Thin film oxygen uptake test
• Debris identification (patch test – particle identification/distribution)
• Filter inspection
• Water-insoluble impurities extraction
• Congealing point of petroleum waxes, including petrolatum
• Rheometry (grease characterization and consistency)
• Oil content of grease
• Soot measurement in diesel oils
• Varnish potential rating

Mike Werley, material coordinator for mechanical maintenance, examines the procedures for dispensing oil from the lubricant shed.

Palo Verde has procedures for everything, from adding lubricants to taking oil samples.

Standard tests performed for new and in-service oils are acid number, water, metals by emission spectrometer, FTIR and viscosity. Standard tests for new and in-service greases are rheology, metals by emission spectrometer and FTIR. For oils and greases, protocols are in place to trigger other tests depending upon initial test data.

“I think other utilities have this level of dedication, but I think we are among the best from a capabilities standpoint, with the lube lab that we have got and the sheer amount of analysis that we can perform on-site,” says plant manager George Andrews.

Employing such extensive lab services is not for every plant or every company. The instruments listed represent a substantial financial investment. However, Palo Verde maintenance, engineering, operations and chemistry personnel state that offering and performing these tests in-house, as opposed to contracting such work to outside suppliers, saves time and money, and allows the site to push the envelope on asset care.

“We are accessible and have a quick turnaround time,” says Mary Holtzclaw, a PdM technician for the past 10.5 years in the Lubrication Lab. “If you had to rush a sample to an off-site lab, that could cost several hundred dollars, plus rush shipping.”

Holtzclaw is one of three full-time technicians in the lab. Geana Warner has worked here since 1998 and Darlene Lang has since 2006. They are considered part of the chemistry department, reporting to chemical work management section leader Paul Koss. They take samples that have been pulled by technicians in maintenance (see sidebar on Page 12), under the direction of reliability engineering, and the reminders of the planning and scheduling organizations. They test, review and analyze, and then work hand-in-glove with Johnson, a member of the engineering team. Johnson evaluates the results and related comments in order to ensure “situation normal” status and/or formulate action plans with these and other departments.

“It is a multi-layer process involving many organizations,” says Johnson.

The techs in the lube lab take particular care to unearth any abnormalities that have the potential to impact plant operations.

“We are making visible an issue that may not be apparent,” says Warner. “We look for evidence of physical breakdown so we can catch it before it gets to a point of need.”

“It’s kind of like crime scene investigation work,” adds Lang. “You are looking for clues and bringing those to light.”

Just like crime investigators, these are highly trained professionals with an eye for minute details. Each has gone through extensive initial instruction and subsequent annual training. All of their actions are under extreme scrutiny by Koss and other plant leaders.

“I do observations and compare what they do to what the procedure states. Our procedures are the tools we use to ensure the accuracy and precision of our testing,” says Koss.

He explains that lab techs must be formally qualified to operate each instrument and test.

“A qualification card must be signed off before the individual is able to perform each test as an independent worker,” he says. “We have an individual at the independent level for all of our tests. As we bring new people in or as they expand their tool box, they start out as a dependent worker – they can do the test under the direction of an independent worker. An independent worker can then sign off on their data. Our lab has a very extensive process to track who does each of the tests and what their qualifications are.”

Lab techs must log into a computer before each and every test. That provides full traceability and accountability.

Everyone in the plant knows what the techs do and have the ability to do.

“The first thing I get – especially from the guys in the plant – is, ‘Could I bring you an oil sample from my pickup truck?’” says Warner. “We do a lot, but, no, we don’t do that.”

Palo Verde has procedures for everything, from adding lubricants to taking oil samples.

Steve Lopez (left), section leader for predictive maintenance engineering, and lubrication engineer Bryan Johnson stand on a deck that overlooks the site’s cooling towers.

A Deep Dive into Lubricant

Every lubrication-related action is indeed documented and traceable. The repository for this information is the Lube Screen, a component of the site’s computerized maintenance management system. This Web-based platform, which took 1.5 years to create, is accessible to everyone within the facility.

“I think what we have is unique and, in some ways, leading the industry,” says Johnson. “The CMMS has different screens to manage our work orders, manage our documents, manage changes to how we define equipment. Screens were developed for lubrication as well as our other predictive maintenance efforts (vibration, thermography, etc.).”

One Lube Screen feature, called Lubed Index, lets users such as maintenance technicians see what specific lubricant has been designed for (and must be fed into) each of more than 20,000 identified lube points. The report can be printed out and clipped to a work order.

Another feature, called Lube Addition Process, denotes every time that grease or oil has been added into a given component, and lists the date, the person involved in the task and how many grease gun strokes or ounces of oil were installed. Entries are categorized as planned/preventive and corrective maintenance work.

Indeed, users can access an incredible amount of historical PdM test data as well as a condition status for each monitored machine.

“We set up a system where our customer could decide how deep they want to go into the data,” says Johnson. “The first level of lubrication includes microscopic image files, where available; iron, copper and viscosity tests for the last 12 samples. That covers most of what anybody would be interested in. We have customers that do want more. For them, the tool provides all other data available in the database. They can dive deeper. If they are really a power customer and they have to do their own Excel work, they can hit an extract feature. They can say, ‘Show me the last 50 records for this machine.’ It will export the last 50 data sets for whatever they picked on that big list and drop it into Excel for plotting and trending.”

This data is driving the actions of maintenance, operations and plant leaders.

“The turbine control oil is beginning to move on us,” says section leader Lopez. “When Bryan notices from the sampling that the acid number is going up, if they replace the filters more frequently, they can bring the acid number down before it runs away from us. That can save hundreds of thousands of dollars. When the acid number gets away, there would be a big oil change – approximately 1,000 gallons. Not only that, but you would have to downpower or shut down, so you would have some lost generation time. That could run a million dollars a day. There’s a lot of potential cost savings there by recognizing it early and then being responsive to that and then putting that onto the schedule.”

Decision-makers such as plant manager Andrews can request regular information related to the pieces of machinery that are most important to them. Whenever a condition report is written against that machinery, an automated e-mail is generated and sent to the subscriber.

“I love the online system,” says Andrews. “I can go on the Web site and all of my unit issues are listed. I can go in and see the work specs, when all of them are scheduled. I follow those. This system just makes the plant run better.”

Lube Screen is one way that communication is maximized across departments and across the 4,000-acre plant.

“We want to be sure that we have our program customers tightly integrated into the process so that they can help us come up with the smartest recommendations,” says Johnson.

Next Up? A Good, Hard Scrub

The Palo Verde Nuclear Generating Station excels at machinery lubrication and has done so for a long time. Attention to detail has gotten the plant to this point, and it is attention to detail that pushes them to continually improve.

“I don’t think anyone is ever really there,” says Johnson. “We are constantly challenging ourselves.”

Each year does indeed bring new opportunities for refinement and growth. Among the goals for 2011 are to examine the formation and impact of varnish.

“The varnish potential and creation of micro-varnish products is an area that we need to understand better, as it is a relatively new form of oil failure,” says Johnson. “We are beginning to test for this condition on site. There is some interesting work being conducted on this topic in industry. Certainly, additive depletion is an area which has contributed to the varnish potential issue in the Group II oils. I’d like to understand that correlation for our machinery.”

Also on the docket is the standardization of grease guns (brands, models, etc.) and the addition to Lube Screen of suggested greasing practices (amounts) for components such as couplings and non-bearing applications.

“We want Palo Verde to continue to be a leader in this area,” says Johnson. “In fact, in 2011, we are going to bring in the Electric Power Research Institute (EPRI) and give them an opportunity to take a really good scrub at us. They will review what we are doing and give us a critical self-assessment. We’ll take a hard look at ourselves and see where opportunities for improvement are.”

When they arrive, EPRI’s scientists and engineers will likely visit the Lubrication Lab, pore over Lube Screen data and interview the host of organizations that play a role in ensuring lubrication excellence. Who knows? They might even test out one of the cafeteria’s taco salads.