During early 2001, a lubrication oil supply system failure on unit No. 3 of the San Onofre Nuclear Generating Station (SONGS) resulted in substantial bearing, journal and steam path damage.
Figures 1, 2 and 3 illustrate the bearing and journal damage. The failure occurred when the primary and secondary AC-powered lube oil pumps failed due to a breaker failure in the switchgear, and the DC-powered emergency bearing lube oil pump failed due to a breaker trip.
The SONGS units lack the shaft-driven lube oil pumps common to some designs, so the turbine generator system coasted to a stop without any lube oil. Engineers at SONGS analyzed the event and have taken actions to avoid its recurrence.
Figure 1. Melted Bearing Babbitt
In 1983 and 1984, at the time that approximately 1,170 MW SONGS units went online, the prevailing design standard called for shaft-driven lube oil pumps. The decision to modify this standard and adopt the externally powered pumps is believed by the investigation team to have been motivated by the desire to enhance fire protection.
It was believed that the parallel design of two AC-driven and one DC-driven motors would provide reliable oil supply to the bearings, an assumption clearly brought into question by the 2001 failure. Upon further investigations, SONGS engineers learned that the design, which caused their problems, was not unique.
The design had caused similar failures in turbine generator sets from around the world and from many different original equipment manufacturers. It was clear that a redesign of the system was required.
The Root Cause Analysis
The lube oil system for each unit supplies 1,900 to 2,500 gpm at 15 to 17 psig to lubricate and cool the unit’s 12 bearings, one of which is a thrust bearing. On average, the system provides 2,100 gpm, as measured at the shaft centerline.
If the lube oil pressure drops below 12 psig, the secondary AC and DC motor-driven pumps are automatically engaged. If the pressure drops below 10 psig, the turbine generator is tripped and shutdown is automated.
After recovering and analyzing the failed components, SONGS engineers developed a fault tree by combining industry standard data and the data specific to SONGS. The team concluded that the lube oil supply systems were clearly vulnerable to the following failure modes:
-
Failure of electrical buses caused by seismic action and random occurrence
-
Spurious actuation of the fire-protection sprinkler system
-
Loss of cooling water
-
Fires in various locations where all three pump cables are routed together
Likewise, the engineers concluded that the mean-time-between-failure (MTBF) for the as-supplied system was only 10 to 15 years. Accounting for the limitations and conservatism factored into this estimate, the team was dissatisfied with this level of risk and undertook a project to improve the design of the lube oil system to achieve an acceptable level of reliability and risk.
Figure 2. Journal Bearing Damage
Redesigning the System
After carefully reviewing the SONGS failure data, general industry data and the designed reliability estimates, the team set out to improve reliability by modifying the system in two steps. The first step was to implement relatively high-impact, low-cost modifications, including:
-
Power source separation
-
Flood protection
-
Shielding from sprinkler protection
-
Operational enhancements
These changes alone, the team estimated, would increase the MTBF from the original 10 to 15 years for each system to 70 to 100 years, a significant improvement. However, the team concluded that given these changes, the likelihood that the station would experience a complete lube oil system failure, like the 2001 failure, in one of its units at some point in the expected life of the station exceeded 50 percent.
Their goal was to reduce the likelihood to less than 25 percent. So, step No. 2 of the modification plan was to add another independent oil pump to each system. The team estimated that step Nos. 1 and 2 combined would increase the MTBF to approximately 300 years and achieve their acceptable risk objective.
Figure 3. Thrust Bearing Damage
The design team set forth primary and secondary objectives. The primary design objective was to ensure reliable oil supply under all credible operating conditions, including the following:
-
Loss of one or both AC pumps and the emergency DC pump
-
Transient response to loss of running AC pump
-
Common mode failures such as fires, flooding and seismic events
Secondary design objectives included:
-
Modified system shall not rely on activation of electrical or hydraulic control signals, which reduce the overall reliability of the system.
-
Design shall rely on passive features, avoiding hydraulic, mechanical or electrical equipment state change, such as nonreturn valve opening, pump starts, pressure switch activation, etc.
-
Eliminate or reduce the time delay required for the new additional pump to achieve full speed.
-
Provide acceptable oil flows and pressure for all pump combinations during operation.
-
Technical solution used for the lube oil system upgrade should have provisions for a future generator hydrogen seal oil system retrofit, using economy of scale benefits.
The team evaluated the pros and cons of several design options for the new lube oil pump and settled on an AC motor-driven through a DC-AC inverter, which provides design flexibility and low maintenance. Primary power for the system is the AC bus, but backup power is provided by a large, independent battery bank (Figure 4).
The battery bank was designed to provide two hours of lube oil operation and six hours of seal oil operation, which will enable the turbine generator to be safely coasted downward.
Much consideration was given to lube oil and seal oil flow requirements during coast down and other considerations when selecting the pump and battery system designs. A conservative approach was taken.
Figure 4. Integrated Lubricating Oil and
Sealing Oil Battery Room
The main challenge of the retrofit was to be sure that the centrifugal pumps with dissimilar capacities and characteristic curves would provide stable oil flow while operating in parallel. This was accomplished by evaluating demand on the system and by selecting custom-design pumps and flow restriction devices.
For the lube oil system, the flow-restricting orifice on the main AC pumps was resized based upon the logic that one AC pump running alone can support the required flow to the bearings while maintaining pressure above 15 psig.
Flow from the new coast-down pump is added downstream of the orifice, so its flow was adjusted by reducing its speed to provide the minimum additional flow required with the pumps running in parallel. At the same time, the new coast-down protection pump is capable of supplying the required supply of lube oil during coast down.
During normal operation, both pumps work in parallel to supply lube oil to the bearings. System flow and pressure self-adjust to reflect system demand and pump characteristic curves. The result is a slight increase in pressure to approximately 20 psig and a corresponding increase in oil flow.
Figure 5. General Arrangement of
Retrofitted Lubricating Oil Systems
Special consideration was given to potential single failure modes, particularly to the failure of a pump discharge check valve. The engineers noted especially that an “open” failure of any of the four discharge check valves could result in a catastrophic bearing failure. In summary, the main design consideration for the new coast-down protection pump included:
-
Provide continuous contribution to the total oil flow to ensure that the pump does not overheat and that the discharge check valve is open during continuous operation.
-
To ensure that nominal coast-down protection pump flow is adequate to prevent its check valve from cycling.
-
To ensure that the nominal coast-down protection pump flow is low enough to prevent main AC pump discharge check valves from cycling.
-
Reassess check valves’ operational conditions and perform required modifications (valve disc machining to reduce flow required to fully open the check valves).
-
Maintain bearing supply pressure within the acceptable range of 14 to 26 psig for all pump combinations.
The designed lubrication oil system retrofits were implemented at both SONGS units from 2002 to 2003. Thus far, results have been positive and the system is operating as designed. However, the team found that the proximity of the plant adjacent to the Pacific Ocean creates special challenges.
The machines are subjected to a humid, saline mist/fog environment that promotes corrosion. After the first operational cycle, electronic equipment panels were slightly corroded. Required actions were taken to protect these components.
Editor’s Note:
This article was rewritten from the following technical paper: “Turbine Generator Lubrication Oil Supply Reliability Improvements at Southern California Edison’s San Onofre Nuclear Generating Station” by Fred Simma, Russel Chetwynd and Stuart Rowe. It appeared in the Proceedings of ASME Power Conference, Chicago, Ill., April 2005.
