After years of operating vacuum facilities with a schedule-based preventive maintenance program, engineers at the National Aeronautics and Space Administration's (NASA) John Glenn Research Center in Cleveland began considering reliability-centered maintenance (RCM) programs to increase facility uptime and decrease operational costs.
NASA's Electric Propulsion Lab Vacuum Facilities has implemented this type of program. These facilities test space hardware in simulated low-earth orbit or deep-space vacuum and temperature environments. Some of the test hardware ultimately is integrated into space flight hardware after testing at Glenn. As such, unscheduled failures of the facility vacuum equipment could impact important milestones in hardware development regarding performance, life and endurance testing, and could ultimately delay high-cost launches of the space shuttle.
These facilities utilize rotary piston vacuum pumps to achieve simulated space vacuum environments. During chamber pump-down, significant air and oil interaction takes place. The vacuum pump oil is used as both a lubrication fluid and as a sealing mechanism. Depending on the test requirements, the type of vacuum pump oil utilized may be either a high-quality, nonadditive petroleum base oil specifically designed for vacuum pump applications, or a fully synthetic hydrocarbon (PAO) where chemically active processes demand inert fluid characteristics. Both oil types require high oxidation resistance and thermal stability. Fluid viscosities are typically in the ISO 68 range.
Water vapor, which is naturally present in the ambient tank atmosphere, condenses in the pump-backing stage during the compression cycle. This free water then mixes with the pump oil and accumulates throughout the duration of the test.
Vacuum Facility No. 5 has four dedicated vacuum pumps with three actively operating during facility testing. The fourth pump was held back in an annual rotational rebuild cycle, which resulted in each pump being used for three years before being rotated for rebuild. The relative ages of the three remaining in-service pumps were then staggered to one, two and three years since being rebuilt.
The primary failure mechanisms of the pumps have historically been mechanical seal wear and oil leakage. This resulted in loss of oil, metal-to-metal adhesive wear and abnormal journal bearing wear from contaminant (water) ingression.
Initial oil samples identified sludge and varnish formation, as well as high acid levels and wear metals in the oil. These chemical changes and the wear metal catalysts promoting high acid formation were evident upon the addition of new oil after just one test sequence was completed. Continued use of the facility pumps accelerated damage to the pump as well as hastened deterioration of the oil. This continued even when the pumps were not running because catalysts were allowed to remain in contact with the oil between tests.
In the equipment operations and maintenance manuals, pump manufacturers state that there are "no fixed intervals for changing pump oil because all applications vary widely in the use of this equipment." This application was, therefore, an ideal candidate for a condition-based maintenance program to identify quantifiable contamination targets. It was also useful to establish when the end of fluid service life has been reached. Additionally, as water and other ingressed debris co-mingled in the rotary pump fluid, chemical changes were promoting oxidation of the fluid and abnormal wear of pump internals. By removing accumulated water and other metal catalysts, both fluid life and pump operational life could be extended.
To better characterize the typical ingression rate of water and other contaminants, the OilTrac™ program was brought on-line to measure and assess the contamination makeup, achieved contamination levels and trending behavior. Cost analysis data was also fed into OilTrac™ to enable financial assessments of return on investment (ROI) and cost savings. From this data, a contamination control program could be designed with proper sizing and application of filtration systems. Initial data collected on Vacuum Facility No. 5's rough pumps indicated an average water level of 10,506 ppm (one percent) in each operational pump and ISO particle counts in the 21/20 range. Chemical testing again confirmed high acid numbers and the presence of oxidation products. Iron and copper levels were also consistent with abnormal wear.
Performance Testing and Corrective Actions Subsequent filtration performance testing was performed concurrent with test chamber operations rather than allowing contaminants to accumulate and damage equipment. The next scheduled test used the same filter system, which operated simultaneously with a two-week vacuum chamber test. The filtration system was able to maintain an ISO 15/13/10 and an accumulated water level averaging 120 ppm throughout the entire vacuum chamber pump-down cycle. Particle counts at four micron were reduced by 98.9 percent and accumulated water was also controlled to 1.1 percent of the original baselined levels. Impact Assessment Based on a historical average of 3,400 hours per year of facility operation, the average savings for Vacuum Facility No. 5 alone were documented at $12,954 per year. Initial ROI of the filtration equipment is under one year. By extending oil and equipment life, other benefits include: Lower oil costs Less waste oil and reduced environmental impact Higher facility operational readiness and more flexible test scheduling Higher facility utilization by increasing available test hours in place of downtime previously reserved for maintaining the facility One of the pumps can be utilized as a redundant standby unit rather than being tied up in the rebuild cycle, effectively increasing overall vacuum train reliability. Additional Opportunities
Once a baseline trend was established for the vacuum pumps, an adequate filtration system was required to control the accumulation of contaminants. For this purpose, a COMO CRI-100 depth-type filter system was installed. Initial filtration was accomplished after completion of a scheduled vacuum chamber pump-down and test. Within 24 hours of filtration operation, the particle count was reduced to an ISO 18/17/13 representing a 90 percent reduction in accumulated particle contaminants. 
With the availability of baselined contamination data, filter performance data and cost assessment information, the operational impact of condition-based maintenance and contamination control could be quantified. Equipment life extension of oil-wetted components is estimated at three times the baseline. Oil life extension by controlling water and removal of oil oxidation by-products is estimated at four times the current baseline. This equates to a 67 percent reduction in pump failure rate. 
Utilizing Vacuum Facility No. 5 as the test case model, additional opportunities for the application of conditioned-based maintenance and contamination control programs have been extended to compressors in Vacuum Facilities No. 6 and No. 12 as well. Initial baselines and assessments have also been accomplished at NASA's Plumbrook Space Power Facility in Sandusky, Ohio, which is the world's largest space environmental simulation chamber, as well as the B-2 Spacecraft Propulsion Research Facility and Hypersonic Tunnel Facility, also in Sandusky. 
