Equipment Procurement

ANALYSIS OF PARLIAMENTARY BUDGET OFFICER COST ESTIMATION MODELS FOR THE JOINT STRIKE FIGHTER PROJECT

Download - (PDF - 330 Kb)

3554-1 (DMGOR AST)

18 March 2011

Andrée Fillion
Director General Major Projects Delivery (Air)
105 Hotel de Ville
Gatineau QC J8X 4H7

References:

A. Parliamentary Budget Officer Kevin Page, An Estimate of the Fiscal Impact of
Canada’s Proposed Acquisition of the F-35 Lightning II Joint Strike Fighter, 10
March 2011.
B. T.P. Wright, Factors Affecting the Cost of Airplanes, Journal of Aeronautical Sciences,
3(4) p.122-128 (1936).
C. Rodney Stewart Cost Estimating, 2nd Edition, John Wiley & Sons, Inc. (1991).
D. International Society of Parametric Analysts, Parametric Estimating Handbook, 4th
Edition (2008).
E. The Society of Cost Estimating and Analysis, Cost Estimating Body of Knowledge
(2010).
F. NATO Research & Technology Organization System Analysis and Studies Task
Group 054, Methods and Models for Life Cycle Costing, Technical Report RTOTR-
SAS-054 (2007).
G. The U.S. Government Accountability Office, Cost Estimating and Assessment
Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-
09-3SP (2009).
H. National Aeronautics and Space Administration, Cost Estimation Handbook (2002).

I. INTRODUCTION

1. On March 10 2011 the Parliamentary Budget Officer (PBO) of the Library of Parliament released the report titled An Estimate of the Fiscal Impact of Canada’s Proposed Acquisition of the F-35 Lightning II Joint Strike Fighter (Ref. A) documenting a program cost estimate for the Joint Strike Fighter (JSF).

2. Director General Major Project Delivery Air (DGMPD(Air)) tasked the Directorate Materiel Group Operational Research (DMGOR) Acquisition Support Team (AST) to review the cost estimating models developed by the PBO. The DMGOR AST identified the three main PBO models:

(1) a top-down parametric model estimating the average unit acquisition cost over all 2,478 JSF CTOL (conventional take-off and landing) aircraft;
(2) a learning curve model estimating the potential average unit cost that Canada will pay
for its 65 JSFs; and,
(3) a top-down parametric model estimating the operating and support (O&S) costs over
30 years.

3. This letter report is structured as follows. The next section discusses the use of
top-down parametric approaches for cost estimation in general terms, and analyzes the
specifics of the PBO parametric models. Section III analyzes the PBO learning curve
model. Conclusions and recommendations follow in Section IV.

II. PARAMETRIC MODELS FOR COST ESTIMATION

4. Parametric approaches to cost estimation use regression or other statistical methods to develop Cost Estimating Relationships (CERs). A strength of parametric approaches is their potential to capture major portions of an estimate quickly and with limited information. A major disadvantage of high-level (top-down) parametric approaches is that they do not provide low level visibility (cost breakdown), and changes in sub-systems are not reflected in the estimate if they are not quantified via an independent variable.

Suitability of Parametric Cost Models

5. It is widely accepted that top-down parametric models for cost estimation are more suitable during the design and early development phase of a program. As the program matures, entering the implementation phase and beginning initial production, bottom-up engineering approaches are more suitable as they incorporate known costs and more accurate uncertainty intervals. The production of JSFs has begun—the program is in its implementation phase. Top-down parametric cost estimating approaches are no longer recommended for the JSF program. The following quotes provide evidence as to the suitability of parametric approaches:

The Society of Cost Estimation and Analysis (SCEA) Cost Estimating Body of Knowledge (Ref. E, module 3, p.9):

     “...different costing techniques are used more prevalently during different stages of the acquisition cycle or new product development process... ...Early on, during concept and design stages, there is a heavy reliance on analogy and parametrics. As more design detail becomes available, build-up will begin to be used, though parametric estimating continues to be important. As Low-Rate Initial Production (LRIP) or full rate production begins, actual costs become available and are incorporated into cost estimates. Even before production actuals become available, actual cost experience on prototype units and early engineering development hardware can be used.”

NATO Systems Analysis and Studies Research & Technology Organization (Ref. F, p.4-5):

     “Parametrics rely on complex relationships and therefore require a considerable amount of data to accurately calibrate... ...Some of the commercially available cost estimating models do have historic public domain information attached and this enables the model to achieve reasonable results in the early phases of the procurement cycle when capability is known, but detailed requirements are poorly defined.”

The United States (U.S.) Government Accountability Office (Ref. G, p.116):

     “As a primary estimating method, parametric models are most appropriate
during the engineering concept phase when requirements are still somewhat
unclear and no bill of materials exists.”

The International Society of Parametric Analysts (Ref. D, p.3-23):

     “CERs may be too simple to be used to estimate certain costs. When detailed information is available, a detailed estimate may be more reliable than one based on a CER.”

Validation of Parametric Cost Models

6. Parametric models can be analyzed using standard mathematical tests of validity. The International Society of Parametric Analysts (ISPA) Parametric Cost Estimating Handbook (Ref. D) provides guidance on validating parametric cost estimating relationships (CERs). The handbook is paraphrased in what follows:

Validation of PBO Average Unit Acquisition Cost Model

7. The PBO uses a parametric model based on a cost estimating relationship to estimate the average unit acquisition cost of a JSF aircraft “derived by statistical analysis of the final costs of past fighter aircraft programs. The model employs a combination of design and performance-based approach in which information concerning performance requirements and technical characteristics of the proposed design solution are used as inputs to generate costing figures.” (Ref. A, p.26, footnote 54). Since the CER function and the historical data used are not provided, the standard mathematical tests cannot be applied to validate the CER. The model cannot be analyzed without further information.

Validation of PBO Operating and Support Cost Model

8. The PBO uses a parametric model based on a cost estimating relationship to estimate the yearly O&S costs of a JSF aircraft as a percentage of the acquisition cost. The PBO states (Ref. A, p.30): “Some relevant data exists indicating that average O&S costs for fighter/strike jets range from approximately 3-5% per annum. This data forms the basis for the development of CERs associated with particular inputs. These inputs include basic mass empty, delivery time, and flying hours per year.” Mathematical details and the historical data used are not provided. The standard mathematical tests cannot be applied to validate the CERs. The model cannot be analyzed without further information.

III. PBO LEARNING CURVE MODEL

9. The PBO report uses the well-known Wright learning curve (Ref. B) to estimate the potential average unit cost that Canada will pay for 65 JSFs. In general, the Wright learning curve is formulated as follows:

y = a • x^b    (1)

10. There are two accepted interpretations of the Wright learning curve (Ref. D): In one theory of learning, called the unit theory, y is the cost associated with unit number x in the production sequence, a is the cost associated with the first unit produced, and b is called the natural learning slope (in log-space). The value of b is almost always negative, reflecting the fact that unit costs decrease as production quantity increases. In the other theory, called the cumulative average theory, y represents the cumulative average cost of units 1 through x, a represents the first unit cost, and b is the natural learning slope. The learning slope b is commonly given as a percentage. The PBO report uses the latter cumulative average learning curve theory.

11. As input data, the PBO uses “known cost points” derived from various low-rate initial production (LRIP) contracts, and the U.S. Department of Defense Selected Acquisition Report (SAR) projected average unit production cost (Ref. A, p.28, footnote 60). Table 1 summarizes the PBO data used (all figures in this report are in millions (M) of 2009 U.S. dollars).

12. The PBO data is inconsistent (hence the asterisk symbols in Table 1): in reality LRIP 1 produced 2 aircraft, LRIP 2 produced 12 aircraft, LRIP 3 produced 17 aircraft, and 31 aircraft are planned for LRIP 4¹ . To facilitate analysis, it is assumed that the PBO intended to refer to LRIP 3 and LRIP 4 production—for the remainder of this report the true LRIP production numbers are used. The DMGOR AST could not validate the cost figures. In particular, the LRIP lots produced three variants of the JSF. It is unknown if the average costs per unit cited by the PBO are specific to the JSF CTOL version (the cheapest variant). In order to proceed with analyses, the PBO costs are given the benefit of doubt and assumed to be correct. In what follows, the PBO learning curve is reconstructed using their assumed data. The construction is shown to be flawed. It is then shown how the PBO misapplied their learning curve.

13. Using the data in Table 1, the PBO fitted the following learning curve:

y = 367,07 · x−0,134 . (2)

Figure 1 illustrates PBO’s cumulative unit average learning curve.

14. The DMGOR AST was able to reproduce the PBO learning curve from the input data points as follows. The system of two equations and two unknowns,

169 = a • (17)^b ,     (3)
86,2 = a • (2478)^b ,       (4)

was solved to determine the learning curve slope b = −0.134² . The PBO then decided to solve for a in the equation:

128,8 = a • (2478)−0,134 ,       (5)

using their (parametric) estimated cumulative average cost of $128.8M (in lieu of the input data). Solving equation (5) yields a = 367.07. The decision to solve for a using equation (5) is questionable at best. By construction, the derived learning curve only fits the PBO estimated $128.8M cumulative average unit cost over 2478 aircraft. The known cost points of $169M on average for 17 aircraft and $163M on average for 31 aircraft are discarded (PBO could have used equation (3) or (4) to solve for a).

15. Assuming that LRIP 1-4 produced 62 aircraft and LRIP 3 and 4 produced 17 and 31 aircraft respectively, using the PBO learning curve the average unit cost for the first 14 aircraft (LRIP 1 and 2) is calculated as $257.4M, the average unit cost for the first 31 aircraft (LRIP 1-3) is $231.3M, and the average unit cost for the first 62 aircraft is $210.7M. Computing,yields the average unit cost for aircrafts 15 to 31 (17 total) to be $209.8M. Computing , yields the average unit cost for aircrafts 32 to 62 (31 total) to be $190.1M. (The reader will note a minor discrepancy when computing the above values using equation (2) due to the limited number of significant digits published.) Table 2 highlights the observed discrepancy between the PBO learning curve and the input data (actual known costs) the PBO used but then discarded.

16. The PBO calculated Canada’s mid-point production (on the JSF production line) to be around aircraft number 848. The PBO learning curve formula was used to estimate that Canada will pay a unit average of $148.51M. However, the PBO applied the learning curve incorrectly: by simply plugging-in x = 848 into equation (2), PBO computed y = 148.51. This number represents the unit average cost of aircraft 1 to 848 (if one were to buy all of the first 848 aircraft). The correct computation is to also determine the average unit cost of the first 847 aircraft (148.26 = 367.07 · (847)−0.134), and then to compute 848 × 148.51 − 847 × 148.53 = 128.6 In other words, PBO has applied their own learning curve incorrectly and should have reported a result of $128.6M unit average cost for Canada. This represents a $19.9M error difference per aircraft, a $1.3B over estimation error in PBO’s published acquisition costs. The error is likely a result of confusion in computing the cost, on average, that Canada will pay for its aircraft vs. computing the cumulative average unit cost at a particular moment on the JSF production line.

17. The analysis of the PBO learning curve model shows that the PBO insisted on fitting a learning curve to their estimated $128.8M average unit cost at the expense of actual known unit average LRIP lot costs. Alternatively, one can use the known LRIP costs and the most common aeronautical Wright unit theory learning slope of b = −0.2345 (2^−0.2345 = 0.85, the cost of production of the unit 2x is 85% of cost of production of the first x unit) as per the National Aeronautics and Space Administration (NASA) Costing Handbook (Ref. H). Assuming an average unit cost of $169M for aircrafts number 15 to 31 (LRIP 3: 17 total), and an average unit cost of $163M for aircraft number 32 to 62 (LRIP 4: 31 total), two (unit) learning slope curves can be produced using the NASA learning slope:

y1 = 352,5 · x^-0,2345       (6)
et
y2 = 402,0 · x^-0,2345       (7)

Equation (6) was fitted using the LRIP 3 average unit cost—solving equation 169 = a1(23)^−0.2345 (aircraft number 23 is the midpoint of LRIP 3), and equation (7) was fitted using the LRIP 4 average unit cost—solving equation 163 = a2(47)^−0.2345 (aircraft number 47 is the midpoint of LRIP 4). Figure 2 illustrates the two learning curves. Using learning curve equation (6) the unit production cost of aircraft number 848 is $72.8M. Using learning curve equation (7) the unit production cost of aircraft number 848 is $83.0M. If the JSF production costs do indeed follow NASA’s learning slope, Canada could anticipate to pay under $90M per aircraft. (The curves should be updated as further LRIP data becomes available.)

IV. CONCLUSIONS AND RECOMMENDATIONS

18. Best practice guidelines in cost estimating indicate that top-down parametric models are suitable during early design phases of major program, but not during the implementation phase when known costs are obtainable.

19. The PBO parametric CER for the acquisition cost and O&S costs could not be validated using standard tests due to lack of mathematical details.

20. The PBO learning curve was reproduced from the data that the PBO used as input. However, the DMGOR AST noted a discrepancy in the model fit as it deviates from actual known input costs specified by the PBO. Instead of using the known costs to potentially validate the learning curve, the PBO fitted a curve which contradicts actual costs.

21. The DMGOR AST has determined that the PBO has made a computational error and incorrectly calculated the average unit cost for Canada’s JSF to be $148.51M. This mistake has lead the PBO to overestimate the unit acquisition cost by $19.9M, or by a total of $1.3B. Since their “initial logistic setup” costs are simply 18% of the acquisition total, this elevates the error total to $1.5B.

22. Using NASA’s learning slope, typical for aeronautical systems, the DMGOR AST fitted two learning curves to the known LRIP lot costs published in the PBO report. If the JSF production costs do indeed follow NASA’s learning slope, Canada could anticipate to pay under $90M per aircraft, however the DMGOR AST recommends that Project Management Office (PMO) JSF provides accurate LRIP costs so that further analysis can be undertaken.

23. The DMGOR AST recommends that PMO JSF requests the details and historical data used for the PBO’s parametric models. The DMGOR AST can then evaluate the CERs using standard mathematical tests of validity.

This Letter Report is a publication of Defence R & D Canada – Centre for Operational Research and Analysis and is for distribution only to the client for whom it was written.