The New Philosophy at Boeing - Minimizing Life-Cycle Operating Costs

Introduction
On June 8, 1998, a 6-axis Ingersoll MastermillTM, hereby known as the Portal Mill, was put in production at a Boeing manufacturing facility in Kent, Washington. Due to the value of the machine and the hardware it would eventually produce, implementation of predictive and proactive maintenance was considered imperative to minimize the life-cycle operating costs that would result from the machine’s anticipated 7-day per week, 24-hour per day work schedule. Tribology was identified as a key enabler to achieve this goal. When the machine was commissioned, Advanced Maintenance Practice (AMaP) condition-based maintenance (CBM) was relatively new to Kent and the use of tribology to monitor machine tool health was non-existent. Since oil analysis was unproven, it became necessary to show there were benefits over and above those provided by traditional preventive maintenance to justify the cost of implementation.

With the equipment and expertise now available, CBM can identify the health of a machine based on the condition of its oil and thus, minimize the “guess work” inherent in conventional maintenance based on pre-determined time cycles. This shift in philosophy extends the life of oil and filters, which equates to dollars saved due to reduction in repetitive, and often unnecessary, change-outs. In addition, particle count and elemental analyses can be used to predict the onset of mechanical failure long before other technologies (e.g., vibration analysis, infrared, etc.). If used proactively, this information can reduce the risk of lost production, premature component failure (parts and labor) and frequent fluid eplacement/disposal, thus leading to an overall reduction in machinery life cycle cost.

Preventive Maintenance - The Traditional Approach
The maintenance personnel involved with installation and commissioning of the machine established the Preventive Maintenance Instructions (PMIs) for the Portal Mill. To do this, the following sources were tapped:

• Original Equipment Manufacturer (OEM) Manuals
• Advice from Ingersoll installation personnel/engineering
• Formal training provided by the Ingersoll Training Department
• Past experience
• Consultation with Boeing Project Engineer

There were two tasks found in the annual “major” PMI that were of great interest to AMaP. They were:

• Drain hydraulic and hydrostatic tanks, and clean them. Fill with BAC 11.
• Change all hydraulic and hydrostatic filters (11 total).

The first item was taken directly from Ingersoll’s Mechanical Maintenance Manual and it is standard industry practice to replace all filters whenever tanks are emptied and cleaned. However, AMaP questioned whether it made sense to do this every year for the life of the machine without definitive proof it was even necessary. We were also concerned with the statement, “Fill with BAC 11.” This is because BAC 11 was a “generic” Boeing identifier for any ISO VG 46 anti-wear (AW) hydraulic oil (we have since adopted the standard ISO nomenclature). Since many companies produce BAC 11 “equivalent” oil, we were afraid this statement “opened the door” for mixing different products within the machine reservoirs.

This was a problem because Ingersoll had used a non-Boeing standard oil (Brand X) during factory acceptance testing (over a period of several months). At the time, Boeing (Puget Sound Region) had a contract with Brand Y Oil Company to supply all our BAC 11 equivalent oil. Therefore, we had to decide whether to continue using Brand X or purge the machine and refill it with Brand Y upon installation at Boeing. We compared the baseline characteristics of both oils on paper and found them to be similar, but could not get Ingersoll to commit to warranty coverage if we were to change the oil from their recommended standard. Without the ability to ensure complete removal of residual Brand X through purging, we were worried that mixing the two oils would create a “hybrid” with unknown and unpredictable qualities. This concern was valid since “home brewing” oil, even of the same viscosity, can cause the following:

• Additive depletion
• Loss of anti-wear performance
• Loss of demulsifying properties
• Reduction of oxidation stability
• Foaming

Our cost for Brand X was nearly three times that of our standard, and this differential can partly be explained by the “volume discount” we received from Brand Y. However, it must be noted that the Brand X oil was shipped in refinery pristine condition in new, sealed, 55-gallon drums, whereas the Brand Y product was transferred in bulk between the refinery and the Boeing end user (trucks, local distributor holding tank, recycled barrels, internal handling, etc.). Thus, potential contamination of Brand Y was much higher and further filtration would be necessary to clean it to acceptable levels before it could be added to the machine. Based on the potential cost increase for filtration and the risk associated with mixing oils, it was decided not to replace the Brand X oil.

Purpose of Oil Monitoring
In order to realize short and long term cost savings, the one-time and repetitive cost of implementing the oil monitoring program had to offset the cost of “doing business as usual,” i.e., traditional filter and oil changes. The following goals were identified to achieve these ends:

• The critical threshold limits for the hydraulic oil had to be established.
• OEM warranty compliance had to be maintained.
• The OEM tank baffle/filtration design had to effectively remove particulates to maintain recommended oil cleanliness levels.
• The Boeing deflector shield design and OEM conveyor return lines had to prevent coolant fouling (water and chlorine) of the oil.
• The existing PMIs needed revision to include predictive CBM techniques.
• The costs associated with oil analysis and traditional filter/oil change-out had to be quantified for direct comparison.
• Operating costs over the life of the machine had to be minimized.

Understanding The Oil
In order to use predictive techniques, it was imperative to understand the oil properties to enable us to monitor the machine’s condition and therefore maintain its health. Several sources were used to determine the critical trend limits for the Brand X oil as well as the cleanliness requirements for the machine and its components. Those relative to the Portal Mill can be found in Table 1. In addition, test samples of new oil were taken directly from a sealed 55-gallon drum. The lab results from these tests were compared with samples taken from the machine and it was found the working oil (ISO 14/9 and 12/8) was actually cleaner than that taken from the new barrel (ISO 15/9).

Brand X is a high VI base stock, AW oil developed specifically for severe hydraulic systems using high pressure, high output pumps and critical systems. According to the product Material Safety Data Sheet, it contains Zinc, Phosphorodithoic Acid, Alkyl Esters, Zinc Salts and Zinc dialkyldithiophosphate (ZDDP). Based on elemental analysis of new oil samples by the lab, the additive formula contains the elements Calcium, Phosphorous and Zinc in the following ratio: 1:3.6:5.4. It is believed that Calcium acts as a detergent to cleanse the oil and system components.

Phosphorodithoic Acid and Zinc act as AW agents that reduce friction and wear and prevent seizure. Zinc is also a rust and corrosion inhibitor that creates a “galvanic” protective film on machine component surfaces. The compound ZDDP is an anti-oxidant, which retards oxidative decomposition.

But how does this information apply to our application? The experience of designers and users of hydraulic and lube oil systems has proven the following fact: over 75% of all system failures are a direct result of contamination. Therefore, we had to identify the contaminants that would attack both the oil and the machine. Table 2 summarizes the types of contamination applicable to our situation.

Oil Sample Locations and Frequency
The machine is equipped with two 300-gallon tanks that are located below floor level. Each tank independently supplies oil to multiple sub-systems within the machine’s primary hydraulic and hydrostatic/lube systems. These systems are broken down in Table 3.

All filters and sample ports are located above floor level on the Fluids Service Panel (FSP). The hydraulic, or H Tank, components are all found on one side of the FSP and the hydrostatic/lube, or L Tank, components are all found on the other (See Figure 1). The H Tank sample port is located after the final stage hydrostatic/lube filter prior to the lines supporting the spindle unit/gearbox, Z-axis rail lubrication and Y-axis hydrostatics. The L Tank test port is installed after the secondary hydrostatic/lube filter upstream of the X-axis worm drive. Ingersoll had determined these were the best locations since they supplied oil to the most critical components. They did not feel additional test points were necessary since the oil sampled from these locations was representative of the oil flowing to any of the other sub-systems. This is reasonable since all eleven filters on the FSP are rated at 10 micron (BETA9 and BETA7.8 = 75) and the return oil is constantly run through independent recirculation filters on the dirty sides of each tank. This pre-filtered oil then flows across internal baffles to the clean side of the tanks before it is pumped through the last stage filters.

Ingersoll should be recognized for their design of the FSP. All filters, visual indicators and sample ports are located above ground for ease of maintenance. In addition, the critical filter housings have visual status indicators (clean, replace, bypass) as well as pressure differential sensors that automatically send an alarm to the machine control to notify the operator when a filter requires replacement. The alarm set points are fixed at 10 psi (69kPa)below the bypass pressure setting for each filter housing.

Monitoring oil cleanliness after the final-stage filters was of primary importance to ensure wear to critical components was being prevented (warranty compliance was a secondary concern). However, AMaP recommended the addition of two sensors upstream of both final-stage filters for the following reasons:

• To verify the cleaning efficiency.
• To provide an “absolute” method to show if the final stage filters had gone into bypass or suffered desorption by direct comparison of particle count test results (without having to rely on the visual indicators and delta pressure sensors).
• To obtain empirical data to either support or refute the need to replace the 10 mircon filter elements more efficient 5 micron elements.

The initial sampling frequency was dictated by OEM warranty. In the OEM Mechanical Maintenance Manual, particle and water count tests were required on a monthly basis. In addition, AMaP requested viscosity, specific gravity (SG), Total Acid Number (TAN) and elemental analysis tests. Lastly, the Boeing Project Engineer requested chlorine count tests between November 1998 and April 1999.

Test Data Analysis
Fluids Analysis Laboratory Reports generated by our internal lab were used by AMaP to identify variance from the baselines of each critical oil characteristic. CBM reviewed the test data and compared it to past trends for out-of-tolerance conditions. Feedback and recommendations in the form of exception reports were then distributed to the interested parties (i.e., Manufacturing, Maintenance, Reliability Engineering, etc.).

During Year 1, no major problems were identified. Particle counts increased over time, but the OEM-provided filtration kept particles greater than 15 micron well below the recommended cleanliness level. However, particles within the 5-15 micron range did rise above the threshold limit requiring us to take action on more than one occasion. When this occurred, a portable filter unit was used to “super-clean” the tanks. To facilitate use of the kidney-loop filter, AMaP had quick-coupling adapters installed to allow convenient connection and quick transfer between tanks. In addition, contamination of the oil by the previous method of breather cap and plug removal for insertion of suction wands and tubing was eliminated. We believe the modification has minimized external dirt ingression into the tanks.

All other critical oil characteristics remained relatively stable over the first year. We did not see a great variation in test results for the samples before and after the final-stage filters for either tank system. Therefore, we concluded the final-stage filters did not go into bypass or suffer desorption. Elemental analysis showed that metal particles were present but they were found to represent only a small percentage of the total particle count and were at levels to cause no concern.

Based on our findings, we decided to eliminate sampling on the upstream side of the final-stage filters. The scheduled sampling frequency was also reduced from monthly to quarterly. This change, coupled with the reduction from four samples to two, reduced the number of samples taken from 48 to eight per year. The amount of samples taken during the first year was knowingly aggressive and was justifiable due to the value of the machine and sensitivity of its hydrostatic design. Based on the trends established during Year 1, we believe the risk of missing something is small and that the reduction in sampling frequency will not jeopardize the machine’s health.

In addition, lab costs will be further reduced as chlorine count and SG tests will no longer be requested on a monthly basis. In the future, chlorine counts will only be obtained as needed if coolant fouling is suspected. SG will only be used if significant viscosity change is detected since this test can provide corroborative evidence that an incorrect fluid was added to the reservoir(s).

Cost Avoidance Calculations
To calculate the cost of the oil monitoring program, the actual labor hours charged against the machine for oil-related service between May 1, 1998 through May 31, 1999, were totaled and multiplied by the yearly maintenance labor rate (this includes hours charged by AMaP, lab and maintenance personnel). The cost avoidance for “skipping” the annual oil/filter change was determined by estimating the labor and materials that would have been required to replace filters, remove/replace the old oil and clean the tanks.

Although the initial cost for oil monitoring exceeded the estimated cost for the traditional change-out, this is not the case over the life of the machine. Many labor hours charged during the first year were due to program implementation and will not recur in subsequent years. In addition, sampling requirements (number of tests and frequency) will be significantly reduced in the future. If we adjust the first year’s cost accordingly, use the same cost avoidance figure for the annual change-out and adjust both for inflation (3%), the annual cost savings for the Year 2 is $7,822.

Assuming a 10-year life cycle for this machine, the cost differential between the two methods would continue to increase for Years 3-10 at a much greater rate. Adjusting for 3% inflation in each subsequent year, the estimated cost avoidance after Year 10 is $70,650 (see Table 4).

General Assumptions for Calculations
The following assumptions apply to calculations above:

• Engineering labor was not included in either cost calculation.Engineering typically charges time to overhead for technical support and therefore, labor cannot be extracted from our cost tracking system for comparative purposes.
• Sampling materials costs were not included in oil monitoring cost calculation. Materials costs for oil monitoring are considered minimal as there are few consumables used in the sampling process. For instance, glass bottles are cleaned and recycled in bulk by the lab. Although the working oil drawn from the machine is consumed, at only 250 ml/sample, the cost for make-up oil is negligible.
• Filter materials costs include both hydraulic and pneumatic filters. During the first major PMI for the machine, it was determined several airline filters did not need to be changed since they have visual indicators and pressure sensing capability. They, as with the hydraulic filters, only need to be changed when dictated by condition. Ingersoll’s Spare Parts List was used to calculate the cost of the replacement filters.
• Filter materials costs were included every 3rd year in oil monitoring cost calculation. The calculations were made assuming the life of filters would be extended for three years due to implementation of the oil monitoring program. Inflation was adjusted accordingly: 3rd year (6%), 6th year (15%) and 9th year (24%). Consider the following:
  • The existing filters have already lasted more than one year without replacement.
  • Make-up oil is filtered during transfer from barrel to machine down to 2 microns.
  • Re-generative wear caused by the abrasive action of small particles recirculating through the system is kept in check. Off-line “kidney loop” filtration down to 5 microns is used to polish the working oil within the reservoirs as needed.
  • Water contamination does not appear to be a concern. The chip/coolant deflector shields and conveyor return systems effectively prevent coolant ingression into the oil reservoirs. Condensation due to oil cooling also does not appear to be an issue.
  • The addition of capped, quick-disconnect fittings on the reservoirs will minimize contamination caused by intrusive maintenance activities (e.g., addition of make-up oil, off-line filtering, etc.).
• The working oil will last indefinitely. The primary oil characteristics have been stable over the first year, and there is no indication this should change under normal operating conditions. Although it is true the additives are depleted over time, we do not have a “fixed” volume of oil in the reservoirs. That is, through both design and normal leakage/boil off, we lose oil which has to be replenished to maintain proper fluid levels. The new oil boosts the amount of anti-oxidants, which keeps TAN in check.
• The life cycle for this machine was assumed to be 10 years.Machines of this class are typically kept in production for 25-30 years. For example, we have five gantry mills at Kent that range in age between 19-22 years. Therefore, the 10-year life expectancy is conservative and the real savings over the true life would be much more significant.
• Inflation at 3% per year was used through the 10-year span. Inflation rates can fluctuate greatly in the future, but most likely will not go down. The greater the rate the greater the variance in operating costs for the two methods. Higher inflation rates would work to generate greater savings over the life of the machine.

Summary and Conclusion
After one year’s time, we believe we have a good understanding of the oil characteristics as they apply to this machine. Although implementation of the Portal Mill’s oil monitoring program showed a net loss of $1,095 for the first year, this cost is more than recovered during the second year of operation. Over 10 years, the total cost avoidance projection shows more than $70K in savings per machine! It should be noted these calculations do not consider the potential cost savings that would result from improved machine availability, reduced component failure (parts and labor) and extended machine life (10+ years), all of which are long term benefits of CBM.

The lessons learned can be applied to new and existing machine tool installations. However, if we focus our resources on existing machine families rather than single assets, we can obtain a “bigger bang for the buck.” Thus, if similar results are achievable, cost savings will be significantly increased due to economy of scale. During Year 1, we accomplished the following:

• Maintained oil condition below baseline threshold limits.
• Complied with the OEM warranty (expired 7/10/99).
• Determined the effectiveness of the OEM tank baffle/filtration design and recommended improvements based on the data collected.
• Proved the Boeing deflector shield design and OEM conveyor return lines effectively prevent coolant fouling of the oil.
• Revised the existing PMIs to include predictive, CBM tasks. If remote-monitoring capability is added in the future, this will lead to more effective manpower utilization.
• Incorporated changes to the maintenance strategy that will save approximately $70K over the first 10 years of machine operation.

Future Recommendations
Wear metals discovered during elemental analysis can be traced back to the source, thereby identifying the defective component(s). However, the existing sample ports are not placed in the best areas since they are at the end of the filter chain. At this point most of the particles have been cleaned from the oil and thus, mask potential problem(s). The best locations for wear detection are directly downstream of the component of interest (e.g., gearbox, worm, drive, spindle bearings, etc.). Unfortunately, there are a host of problems that preclude the installation of test ports at this time; namely access, machine availability, safety and cost per unit. Therefore, we need to limit the amount of test ports and place them in locations where they will provide the most meaningful information. With this in mind, it is recommended that two ports be added to both reservoir return lines (one per tank). Sampling at these locations will provide a macro view of the entire hydrostatic/lube and hydraulic systems. By comparing oil samples with ones taken from the downstream ports, the true effectiveness of the tank/filtration systems can be determined.

We will continue to evaluate the OEM filtration design for improvement. We are certainly aware the 5 micron level particle count has risen above our baseline target at times, but have used off-line filtration to bring this under control. However, the question remains whether this would be necessary if we added 5 micron elements to the final stage filters for both tank systems. The X-axis worm hydrostatic circuit has two 10 micron filters in series. We could leave a 10 micron element in the primary housing and replace the secondary filter with a 5 micron element. The hydraulic system only has a single final-stage filter, but we could replace the original 10 micron element with one rated at 5 microns. Without further study, we cannot be sure these changes will reduce overall operating costs. That is, with 5 micron filters we can anticipate off-line filtration may no longer be required, but filter unit costs and number of change-outs will most likely increase. Downtime cost must also be considered since the machine must be shut off and there is the possibility external contamination can be introduced during filter change-out. The 5 micron particle count trend over the next year should provide us with the information we need to make a decision. If we can control contamination levels under the current plan, there will be no reason to make a change. However, if the machine begins to wear, or external contamination increases to the point the off-line cleanings cannot keep up, we may need to install the 5 micron elements.

Replacement of the conventional breather/filler caps with screw-on canister filters is also recommended. Two styles are available: one which removes airborne contaminants down to 3 microns (existing breathers are rated at 10 microns) and another which can also remove moisture. The latter elements are more expensive and would only be desirable if the water content in the oil begins to rise significantly. Otherwise, the standard 3 micron filters are quite acceptable. The quick connect fittings recently added (for the off-line filter) can be used to pump make-up oil into the tanks.

Finally, the long-range goal is for remote monitoring/data logging. AMaP is currently developing a prototype device, which can be installed on machines and tied into the Boeing Intranet. Although the initial intent is to support vibration analysis, it is hoped the new technology can be adapted for oil condition monitoring (e.g., particle count, viscosity, water content, etc.). With this capability, we can reduce the manpower currently necessary to take samples and run lab tests. Out-of-tolerance conditions can be reported to a central location, where AMaP personnel can then be dispatched to investigate the problem. Eventually, many more machines can be integrated into the system which today is limited by logistics and the shear number of machines versus available personnel.

Reference:
“Parker Hydraulics Products & Total Systems Engineering,” Catalog 0108-3, Section B, page 217, Parker Motion & Control.