Independent Review: 2014 Department of National Defence Annual Update on Next Generation Fighter Capability Life Cycle Costs

List of Acronyms

1. Executive Summary

Raymond Chabot Grant Thornton Consulting Inc. (RCGT) has been contracted by the National Fighter Procurement Secretariat (NFPS) to conduct an independent review of the 2014 Department of National Defence (DND) Annual Update on Next Generation Fighter Capability (NGFC) life cycle costs (LCC) to verify the assumptions and cost estimates, including their alignment to the LCC Framework (“the Framework”) developed in 2012.[1] This report presents the observations and recommendations resulting from the 2014 Independent Review.

The focus of the Independent Review was to assess whether the Framework was appropriately applied. This process included assessing whether the cost estimates and all changes to the underlying assumptions, validated in the 2014 Annual Update[2], were calculated and presented in a manner consistent with the Framework.  The review did not assess the appropriateness or accuracy of source data relied upon in the LCC estimation.

The Independent Review involved assessing DND’s LCC processes, procedures and documentation against the Framework. The Framework includes key principles that focus on the planning, sustainability and the continuous improvement of DND’s cost estimating processes and tools.

The review process included a:

• Review of the 2014 Annual Update;
• Review of the Model used to calculate the 2014 estimates; and
• Comparative review of the LCC estimation approach applied in the the 2013 and 2014 Models.

Our Independent Review of DND’s application of the Framework did not reveal any deviations from the Framework’s principles that would result in any material changes to the overall LCC estimate. However, we identified one (1) significant area of observation relating to the contingency for Acquisition costs.

As demonstrated in the table below, a total Acquisition contingency of $1,080M was estimated based on risk analysis. Of this contingency, only $76M was included in the LCC estimate so as to respect the frozen Acquisition envelope. As such, a contingency shortfall of $1,004M exists.

The 2014 Annual Update states that, “If full acquisition contingency was required, the remaining shortfall could be met by buying fewer aircraft.”[3] However, sensitivity analysis has not been undertaken to explore the impacts of decreasing the total number of aircraft acquired by Canada on the LCC Estimate. The following recommendation relates to this matter.

In addition to the recommendation pertaining to the LCC estimate’s alignment with the Framework principles, RCGT identified one (1) other opportunity for improvement, as described in the following table. This recommendation is not in response to any significant deviation from the Framework.

The overall assessment of the NGFC LCC process is that DND has made significant progress since the 2013 Independent Review to improve and refine its processes and methods. DND has taken substantive action towards addressing the recommendations made in the 2012 and 2013 Independent Reviews, and has fully addressed several recommendations. There are further opportunitites for improvements, but these are largely reflective of the maturity level of the NGFC LCC process, recognizing that the NGFC project remains in the options analysis phase. In the interim period between the 2014 Annual Update and the 2015 Annual Update, DND should continue its progress towards addressing the recommendations from the 2012 and 2013 independent reviews (See Section 4 and Appendix A) along with the recommendations provided through this 2014 Independent Review.

2. Background

2.1 Context

On April 3rd, 2012, the Auditor General of Canada presented his 2012 Spring Report to Parliament, identifying concerns with the way key information relating to Canada’s fighter capability was being developed and presented to Canadians. The report recommended that the F-35 fighter capability cost estimate be refined to include the full LCC and that the estimate be made public. The Government accepted the Auditor General's recommendation and launched a Seven-Point Plan[4] in response.

The Government, via the NFPS within Public Works and Government Services Canada (PWGSC), is committed to ensuring that due diligence is performed throughout the implementation of the Seven-Point Plan.

As part of the Seven-Point Plan, the Treasury Board Secretariat (TBS) commissioned an independent review[5], which included the development of a NGFC LCC Framework[6]. This Framework, completed in November 2012, was designed to provide DND with clear direction for the consistent estimation and reporting of NGFC LCC.

Beginning in 2012, the Department of National Defence will provide annual updates to Parliament. These will be tabled within a maximum of 60 days from receipt of annual costing forecasts from the Joint Strike Fighter program office.

DND’s first Annual Update[7] on NGFC LCC was completed in December 2012. An independent review of the 2012 Annual Update provided eight (8) recommendations for improvements to the NGFC LCC estimation approach. In August 2013, DND prepared its second Annual Update which was independently reviewed by RCGT, resulting in a further six (6) recommendations (See Appendix A).[8]

RCGT has now been contracted by the NFPS to conduct an independent review of the 2014 Annual Update on NGFC LCC to verify the assumptions and cost estimates, including their alignment to the Framework. This report presents the observations and recommendations resulting from the 2014 Independent Review.

2.2 Objective and Scope

The focus of the Independent Review was to review whether the Framework was appropriately applied.

This Independent Review includes assessing whether the cost estimates and all changes to the underlying assumptions validated in the 2014 Annual Update are calculated and presented in a manner consistent with the NGFC LCC Framework published in November 2012.

2.2.1 Limitation of Scope

While RCGT assessed whether data sources were appropriately documented and validated by DND, our review did not assess the appropriateness or accuracy of source data relied upon in the LCC estimation. For example, the costing information provided by the Joint Program Office (JPO) was not subject to validation (i.e. RCGT accepted the figures as provided and did not validate the accuracy of the costing assumptions underlying the information).

The evaluation of options conducted as part of the government’s Seven-Point plan  was not considered by the Independent Review.

2.3 Report’s Limitations

The purpose of this report is to inform decision-making within DND, NFPS and TBS and to support DND’s 2014 NGFC LCC Annual Update to Parliament. No party should act on the contents of this report without conducting further analysis. DND, TBS and the Government of Canada are responsible for decisions made relating to the NGFC Program. RCGT will not assume liability for the reliance on this report by any third parties. Final benefits and costs realized from implementing a plan to acquire a fleet of F-35 aircraft will be based on future events and government decisions, which may lead to material variances from the estimates included in this report.

The review consisted of specific activities agreed to by DND and the NFPS. The review assessed the alignment of the NGFC LCC estimate to the Framework, relying on information provided by DND. RCGT did not conduct an audit of the estimate and, therefore, does not provide assurance or express an opinion on cost estimates. Furthermore, the inability to review the Statement of Operational Requirement (SOR) limits our ability to conclude on whether the Model’s Cost Breakdown Structure (CBS) includes all project capability requirements.

Readers of this report should consider the document in its entirety. Selection of, or reliance on, certain elements of the report may result in misinterpretation of information provided. RCGT will not accept liability for such interpretations.

RCGT reserves the right, but will be under no obligation, to review and/or revise all findings, calculations and recommendations referred to herein, if we consider it necessary in light of further information that becomes known to us after the date of this report.

2.4 Key Framework Principles

The Framework sets out seven (7) key principles to guide DND in achieving the estimate outcomes required by multiple decision makers and stakeholders. The review criteria used by RCGT during the independent review are aligned to the following principles:

NGFC LCC Planning

Develop a plan to ensure the NGFC LCC model meets the needs of all prospective users and aligns with the LCC Framework.

Boundaries and Assumptions

Well-defined and agreed-upon boundaries are established. Key ground rules and assumptions are understood and agreed. Project documentation is readily available and forms the basis for costs.

Develop Model

A Cost Breakdown Structure (CBS) is developed representing the total Program. The model developed is in line with leading practices and supporting CBS and the range of decisions anticipated. Appropriate cost estimation methods are selected for each cost element.

Data, Populate and Document Model

Data is collected and normalized. The baseline estimate is developed and internal validation of model and results is conducted.

Review, Analyze and Update 

Undertake sensitivity, risk and uncertainty analyses and develop risk-adjusted cost estimates. Results are established and documented. Independent cost assurance activities are undertaken and necessary adjustments are made to the NGFC LCC Model.

Interpret and Report Results

Purpose-focused reports are developed for decision makers and stakeholders in accordance with prescribed guidelines.

People and Organization 

The NGFC estimator team is drawn from a professional costing organization, supported by standard tools, techniques and methods. The Cost Assurance role is integrated into the process with appropriate policies to ensure a non-advocacy approach.

2.5 Approach

The first step of the review was to understand the Framework, key changes made to the LCC Model since the 2013 Independent Review and the 2014 Annual Update, as well as any supporting documentation initially provided by DND. The RCGT team also attended a workshop organized by the Costing Team to walk through the 2014 LCC Model.

As information was made available by DND, RCGT reviewed the finalized Model, the 2014 Annual Update and related supporting documentation (listed in Appendix B) to:

• Examine cost data, assumptions and analyses to assess whether they were consistent with the principles outlined in the Framework;
• Examine cost data, assumptions and analyses for general appropriateness, reasonableness and accuracy;
• Examine whether cost data were developed using the most recent and up-to-date costing information provided by the JPO and other source data providers; and
• Analyse and summarize results.

The review of the LCC estimate considered whether:

• Each individual component of the estimate was traceable back to appropriate source documentation, aligned to project capability requirements and assumptions, and an appropriate calculation method was chosen;
• Cost estimates and related assumptions were documented, communicated and consistently applied;
• The estimates were derived from project capability requirements and a detailed CBS, appropriate for the stage of the NGFC project;
• All cost elements included in the estimates were aligned to the purpose and capability requirements as identified in DND’s NGFC Project Charter, CBS and ground rules and assumptions, and are neither omitted, nor double counted;
• Underlying data had been correctly normalized/adjusted for the technical baseline cost and for inflation using appropriate guidance; and
• The normalization/adjustments and time phasing of the cost estimate were logical, accurate and consistently applied.

The review of the Model included:[9]

• A detailed review of all cost data inputted into the model by reconciling all inputs to supporting documentation; and
• A sampling of all formulae within the model to verify that they are consistently applied within the model and the identification of potential errors.

In addition, a comparative review of the 2014 LCC estimation approach to the 2013 LCC estimation approach was conducted, which included:

• Reviewing ground rules and assumptions data to identify any changes in global assumptions between the 2013 and 2014 Annual Updates;
• Reviewing the Model change log to gain a broad understanding of changes in structure or methodology between 2013 and 2014 Annual Updates;
• Reviewing DND’s year-over-year LCC estimate reconciliation to confirm that all changes to the LCC estimate included in the reconciliation could be traced back to the Model change log; and
• Reviewing a sample of Model iterations to trace the LCC estimate reconciliation back to the actual changes in the Model.

3. Independent Review Observations

The independent review of the estimates and Model included an assessment of the scope, assumptions and calculations underlying the estimates. The independent review criteria were developed in 2012 based on relevant TBS policies, other relevant Government of Canada instruments and applicable leading practices and principles identified in the Framework. To facilitate year-over-year comparability, RCGT has applied the same review criteria used in the 2013 Independent Review.

The key observations associated with each review criterion are summarized below:

In addition to the Framework, our review included a comparative analysis of the estimation approach used in 2014 to the approach used in the previous year. This is discussed further in Section 3.8.

The table below summarizes DND’s 2014 estimates based on the application of the Framework. Actual costs will vary from the estimates over time, and these variances may be material.

Principle: Develop a plan to ensure the NGFC LCC model meets the needs of all prospective users and aligns with the LCC Framework.

Planning is an important step to ensure the Costing Team successfully achieves their tasks and work effectively toward objectives. Per the Framework, we expected the LCC plan to outline key elements such as scope, purpose, schedule, data, costing methods and quality assurance. The NFGC Cost Report[13] includes elements of an LCC Plan as defined in the Framework, and includes high-level background, methodology and costing estimate results. It also included the purpose of the Model, high level inputs and outputs, the CBS and costing methodology at the cost element level for each LCC phase.

Overall, the NGFC Cost Report is a well-developed document that acts as a preliminary findings report, explaining work completed and summarizing cost estimates, including key assumptions and cost details. However, the LCC plan does not include a master schedule with resource requirements and milestones. We observed evidence of DND work plan documents[14] which include tasks related to the NGFC LCC. In addition, configuration of planning documentation is formally controlled through the processes and procedures of DND’s D Cost S.

3.2 Boundaries and Assumptions

Principle: Well-defined and agreed-upon boundaries are established. Key ground rules and assumptions are understood and agreed. Project documentation is readily available and forms the basis for costs.

3.2.1 Cost Boundaries

The LCC Framework states that cost boundaries should include all costs from initiation through asset disposal at the Program level. The Framework defines Program level costs as costs “related to any group of resources and activities, and their related direct outputs, pursuing an objective or a set of objectives. A program may include various projects at various times”[15].

Through the review of the NGFC Cost Report and Model, we observed that cost boundaries are well defined and agreed-upon. The Model presents costs down to the Program level, from the Project Development stage through to the Disposal phase of the NGFC project. The Model assumes a lifespan of 30 years for each aircraft following its delivery[16].

Our review of the cost boundaries did not identify discrepancies in DND’s application of the Framework.

3.2.2 Ground Rules and Assumptions

Because the project is still at an early stage, the Framework states that “it is necessary to use a series of assumptions that constrain elements of the project in a meaningful way in order to allow the development of an LCC. These assumptions form a key element of the NGFC LCC as they define the basis on which the estimates are being developed”. The Costing Team should therefore maintain a separate document recording all assumptions and maintain relevant supporting information. Assumptions and any subsequent changes should continue to be reviewed and approved by the appropriate stakeholders.

Ground rules and assumptions have been documented through the LCC Plan supporting appendices, the Annual Update and separate documents dedicated to ground rules and assumptions.

In line with a recommendation from the 2013 Independent Review that “DND should maintain a dedicated and separate ground rules and assumptions document, containing all current and approved ground rules and assumptions”[17], DND has prepared a stand-alone Ground Rules and assumptions document that is approved by key stakeholders when changes are made. 

Our review of the ground rules and assumptions did not identify discrepancies in DND’s application of the Framework.

3.3 Develop Model

Principle: A Cost Breakdown Structure (CBS) is developed representing the total Program. The model developed is in line with leading practices and supporting CBS and the range of decisions anticipated. Appropriate cost estimation methods are selected for each cost element.

3.3.1 Cost Breakdown Structure

The Framework requires that “the Cost Breakdown Structure (CBS) provides a logical and complete breakdown of the NGFC Program”[18]. Per the Framework, we expected the CBS in the 2014 LCC Plan and Annual Update to be aligned to the elements defined in the SOR. We were not able to conduct this test, as DND was not able to provide the SOR because the document was not available.[19] In the absence of adequate documentation to directly compare against the CBS, RCGT compared the 2014 CBS with last year’s Annual Update to identify any gaps or changes. We identified the addition of one cost element to the CBS since 2013 and the renaming of four cost elements, which were the result of changes to the source data by the JPO.[20] These changes are appropriately reflected in the final 2014 cost estimates and Annual Update.

Without reviewing the SOR, RCGT cannot conclude on whether the Model’s CBS includes all project capability requirements. Should a revised SOR become available in future years, the Model should be assessed against it to ensure compliance with the Framework.

Until such time as a new SOR document is available, there will continue to be a limitation on the reviewer’s ability to assess compliance against this Framework principle (See Section 2.3 Report’s Limitations).

3.3.2 Model

According to the Framework, “the Model is the tool that captures all inputs, undertakes the necessary calculations to provide outputs suitable for consideration”.[21] It should be: Accurate; Comprehensive; Replicable; Auditable; Traceable; Flexible; Credible; and Timely. RCGT’s review of the Model included a detailed review of model inputs to identify any potential copy/paste or keying errors for data provided by JPO or from other DND sources, as well as a review of model formulae for consistency and reasonability. The observations arising from the review of the Model are summarized in the following table for each of the eight NGFC LCC Framework Principles listed previously:

In addition, the Framework indicates that the Model should be fully documented for any reader to understand what inputs are used and how they are manipulated to create outputs. In this regard, the Costing Team has developed a Configuration Management Plan which is meant to ensure that the manner in which costs are estimated through the Model remains consistent year-over-year and that any change is well considered and executed.  Through our review we found that the Configuration Management Plan included a listing of best practices used within the model, descriptions of the templates and structures used within the model and a description of the overall costing process and steps defined within the NGFC Framework. We identified the following elements which could be added in order to further improve the configuration management plan:

1. Roles and responsibilities within the Costing Team for the editing and updating of the Model to ensure that any changes made are performed by a team member with suitable experience and knowledge.
2. Process used for the editing and updating of the model to ensure a consistent and adequate methodology is used when editing the Model to ensure the proper application of formulae and accuracy of cost estimates. This should include the planning, implementation and review of the edits made to the Model.

3.3.3 Cost Methods

The Framework indicates that appropriate cost estimation techniques should be selected for each cost element included within the model. The appropriateness of the cost method is largely determined by the accuracy and adequacy of available information related to the given cost element, both of which are expected to improve as the project moves from the Development to Acquisition and then Operational stages.

There are a number of different costing methods that can be applied to different costing elements within the Model; these methods are discussed briefly below[22]:

• Engineering cost method – Estimation of a cost element by examining products component by component. This can also be characterized as a 'Bottom up’ Approach;
• Analogous cost method – Estimation based on experience with the same or similar products or technology;
• Parametric cost method – Uses significant parameters and variables to develop cost estimating relationships; and
• Extrapolation from actuals – Uses actual contract and project performance to estimate costs at completion including estimate of actual learning against projected learning curves and earned value management approaches.

The estimation of costs related to Development, such as the PMO, are derived from a combination of parametric data, obtained from a Memoranda of Understanding (MOU), and historical actual cost information related to the PMO from DND’s financial systems. There are some other costs related to initial source data and secondary studies which are derived using a parametric costing method.

The cost method applied for the Acquisition cost is primarily based on engineering cost, which is obtained via a component cost breakdown for the aircraft provided in the data generated by JPO. As with the JPO data used in the 2012 and 2013 Annual Updates, the JPO data used in the 2014 Annual Update continues to combine historical actual costs of production for F-35 units with forecasted production efficiencies through learning and economies of scale.[23]

With regard to Sustainment cost, the Model has continued to rely upon the JPO estimates as its primary source data. Sustainment costs are determined largely through use of parametric cost methods using the key drivers of number of aircraft and estimated flying hours to generate an estimate. However, some actual values have begun to be incorporated in the Sustainment cost estimates. As of this year, JPO Sustainment cost data has been independently confirmed by an independent CAPE assessment. It is anticipated that these cost estimates will continue to mature and JPO Sustainment cost estimates will eventually include the further integration of actual observed sustainment costs for F-35 aircraft in the US, which will be refined as aircrafts reach steady state flying hours.

Operating costs estimates within the model are generated from DND actual historical financial system information. At this point in the analysis, DND has opted to use an analogous approach to estimation using cost data related to existing CF-18 support units and bases. This approach will be the most appropriate until additional information can be obtained through the operation of an F-35 in Canada. As is the case with Sustainment costs, Operating costs are for the most part country-specific. Therefore, drawing comparisons with other JSF partner countries may not produce the most accurate estimation of Operating costs for an F-35 within Canada.

As part of the 2013 Independent Review, it was highlighted that there existed an opportunity to better align the Operating cost of fuel and lubricants consumption with the forecasted yearly flying hours rather than with the buy profile and the steady state estimated flying hours. This methodology was integrated in the 2014 version of the model and will provide more adaptability should changes in flying hour estimates occur.

Finally, the cost estimation of Disposal cost for the F-35 has been based on a parametric/analogous method, as it is derived from an estimate of Disposal costs related to the CF-18 fleet. When disposal of these aircraft is initiated, DND will then be able to update the disposal estimates accordingly.

Based on the review conducted, the cost methods currently employed are considered to be appropriate.

3.4 Data, Populate and Document Model

Principle: Data is collected and normalized. The baseline estimate is developed and internal validation of model and results is conducted.

3.4.1 Data Collection and Normalization

The Framework stipulates that DND should collect all data elements from appropriate sources and normalize / adjust these elements as appropriate within the Model. Inherent in this process is the review and analysis of key cost factors and drivers, in order to establish reasonability and assess sensitivity and risk related to those estimates.

As part of our review, we identified and traced the source of data within the Model and analyzed the method of normalization. These have been organized along each major Model cost element and discussed below.

There are two key indices required within the Model:

1. The Inflation Rate – costs, whether they be in US$ or CAN$, must be escalated in order to account for the forecasted inflation rates, which converts a “Current Year” estimate to a “Budget Year” estimate. Inflation information is derived from two sources:
   1. The DND economic model[24], which is developed within DND to estimate defence sector focused inflation figures across the department; and
   2. The US Economic model[25] which is provided via the JPO.
2. Foreign Currency Exchange – the foreign exchange rate is based on the Consensus Economic Inc. report[26] which is published on a monthly basis. The Model uses the August 2014 Average Annual rate.
   
   The Model uses the most recent forecast available at the time that JPO was projected to provide its data.

Based on our assessment of the data provided, the indices used are both current, consistent with methods used for the 2014 Annual Update and appear to be reasonable.

3.4.1.2 Development

The cost data for Development incorporates a number of cost items, including PMO costs for personnel, travel and other administrative costs, along with costs related to development of initial source data and secondary studies related to the project. The bulk of these costs are estimated based on a MOU, but also include some actual historical costs related to the PMO.

All material development cost information was able to be traced to its source. 

3.4.1.3 Acquisition

There are numerous cost items included in the Acquisition portion of the LCC estimate, including:

• The purchase of the aircraft itself (the Unit Recurring Flyaway (URF) cost);
• Costs related to ammunition and initial spare parts inventory;
• Aircraft modifications;
• Infrastructure and sustainment set-up for the aircraft;
• Initial training;
• Reprogramming lab costs;
• PMO Acquisition costs; and
• Other acquisition costs.

Over 90% of the estimated Acquisition costs of $8.9B are driven by JPO-provided data. This data has not been validated by the independent review team beyond ensuring that the data is appropriately entered into the model and whether the data provided reasonably estimates the expected Acquisition costs related to an F-35.

The JPO cost estimates are provided in 2012 year US$, meaning that they are adjusted for both inflation and foreign exchange in order to be translated into DND ‘Budget Year’ dollars. Infrastructure and other costs that are driven by DND sourced cost information (e.g. initial training and reprogramming lab costs) are adjusted for inflationary increases.

Acquisition cost estimates also integrate the use of the payment schedule which is described below:

• Long Lead (5%) - Due 1 year before order
• Full Funding (30%) - Due in year of order
• Payment (45%) - Due 1 year after order
• Delivery (20%) - Due 2 years after order

The data collected and input in the model is consistent with the data received from both JPO and internal DND sources. 

3.4.1.4 Sustainment

Sustainment cost data is sourced directly from JPO, and therefore is adjusted for both inflationary and foreign exchange impacts. Based on the review, Sustainment data has been collected and entered from JPO correctly.

3.4.1.5 Operating

The majority of Operating cost information is derived from actual historical results of DND’s CF-18 program. Information on personnel and most consumables is extracted from the Defence Resource Management Information System (DRMIS) and populated into the model. Other consumable items, such as ammunition and lubricants are based on DRMIS historical data and then pro-rated to the anticipated acquisition schedule for the F-35 fleet. Fuel usage in particular is estimated based upon:

• The steady state assumed flying hour estimate;
• The JPO estimated fuel burn rate for the F-35;
• The weighted average standing offer price for aviation fuel at the Cold Lake and Bagotville bases; and
• The number of F-35 aircraft in operation in a given year.

As all information for Operating costs are in Canadian dollars, the only normalization required for the data is to apply annual inflation to the estimate, as forecasted in the DND Economic Model.

3.4.1.6 Disposal

Disposal estimates are based on a DND-developed estimate for the disposal of the CF-18 fleet.[27] This information is provided in Canadian dollars. Therefore, the data is normalized for inflation using Canadian Economic Model rates.

3.4.1.7 Summary of Data Collection and Normalization

Based on the review of the Model and documentation provided there is sufficient evidence to support the tracing of data collection and all data has been normalized. As per the Framework, all cost elements that have a significant impact on the overall estimates are identified and related data is collected from a reliable source.  

3.4.2 Baseline Estimate

The LCC Framework outlines key expectations regarding the completeness of a baseline estimate that is derived from up-to-date and normalized cost information (as per Section 3.4.1) and utilized appropriate cost methods to create the estimate (as per Section 3.3.3). As part of the review, we have analyzed the components of each major cost element to review whether costs are appropriately estimated and that source data is appropriately documented. The following sub-sections identify the major sub-components of each cost element.

3.4.2.1 Development Cost

Development costs currently comprise approximately $550M (1.3%) of the unadjusted LCC estimate of $42.7B.[28] The cost estimate for Development includes costs related to:

• The PMO; and
• Contributions under the JSF Production, Sustainment and Follow-on Development MOU.

The estimated costs of Development include expenses already incurred related to Canada’s role in the JSF program. Remaining development costs include remaining development PMO and JSF MOU costs, as well as initial and secondary studies on the aircraft. 

Development costs for the most part are well-documented and traceable to their source information. However, documentation related to estimated costs related to Secondary Studies could be improved in future Model versions.

3.4.2.2 Acquisition Cost

Acquisition costs currently account for approximately $8.9B (20.8%) of the unadjusted LCC estimate of $42.7B. The main sub-components of the Acquisition cost estimate include:

• The unit recurring flyaway (URF) cost;
• Initial spare parts, training and ammunition;
• Infrastructure;
• Sustainment set-up and ancillary equipment;
• Acquisition PMO;
• Diminishing manufacturing sources and concurrency modifications; and
• Other potential acquisition costs.

Overall, unadjusted costs related to acquisition have increased from $8.6B to $8.9B, an increase of approximately 3.5% year over year. This section will briefly discuss the subcomponents of Acquisition, while the risks and contingencies will be explored further in Section 3.5.2.

The following subsections provide some additional details on individual Acquisition cost elements.

URF Cost

The URF cost is essentially the estimated unit price of an F-35 as estimated by the JPO. The 2014 URF cost estimates reflects JPO data from the latest actual production costs for the aircraft.[29] The current average cost of an aircraft is $97.1M CDN. Once adjusted for inflation and foreign exchange forecasts, the total estimated URF cost is approximately $6.3B or 70.8% of the total estimated Acquisition costs of $8.9B.

The URF cost categories and means of data collection have not changed from the 2012 Annual Update and still incorporate the five (5) major components of the aircraft:

• Airframe;
• Vehicle systems;
• Mission systems;
• Propulsion system; and
• Engineering change orders.

These cost elements are comprehensive of anticipated URF cost as indicated by current JPO production cost information.[30]

Initial Spare Parts, Training and Ammunition

As part of the acquisition process, there is a need to obtain initial complements of spare parts and ammunition to cover the training and initial operational period. Likewise, pilots and maintenance personnel must receive training to learn how to operate and maintain an F-35. Overall, these costs are estimated to be approximately $478M, or 5.4% of total estimated acquisition costs of $8.9B.

As part of the 2013 Independent Review it was noted that fuel and lubricant costs were included in two different areas of the Model, Acquisition (specifically in the training costs) and Operating. This resulted in the double counting of fuel and lubricant costs for the 2017-18 to 2020-21 periods. This issue has been resolved within the 2014 version of the Model.

The source data for cost estimates related to the ammunition costs for the fleet state that they include a foreign exchange and a contingency factor of 20%.[31] Other areas of the Model (such as infrastructure), remove all built in contingency factors in the source data in order to ensure that the Model itself is the only source of contingency modifiers. We would recommend that the initial ammunition calculation follow this same approach and be based upon the unadjusted estimate figure, which could result in an potential estimated reduction of up to $8.3M from the total LCC estimate. This does not represent a material reduction in the LCC estimate.

As with the fuel and lubricant costs, it was noted in the 2013 Independent Review that costs related to ammunition appeared in both the Operating and Acquisition (specifically in the training costs) sections resulting in possible double counting. As a result of this observation, the calculation of ammunition costs was aligned to yearly flying hours for aircrafts within Canada, eliminating the risk of double counting.

Infrastructure

The costs related to infrastructure are related to 23 construction and renovation projects needed to support an F-35 fleet. These costs are developed in a separate estimation template, whose outputs are then fed into the Model. Currently, the costs related to the infrastructure estimate are at an early stage of development and should be considered ‘rough order of magnitude’.[32] Overall infrastructure costs are estimated to be approximately $254M, or about 2.9% of total unadjusted Acquisition costs of $8.9B.

The cost estimate for infrastructure lays out comprehensive ground rules and assumptions that have been revised for the 2014 Annual Update. These include construction and renovation costs per square metre and other items such as design fees, travel and furniture costs. The original estimate provided to the DND Costing Team also includes a provision for contingency, which is not included in the Model in order to prevent duplication of contingency amounts.

Sustainment Set-up and Ancillary Equipment

Sustainment set-up and ancillary equipment mainly consists of costs related to training devices and support equipment for the F-35, along with the labour costs needed to install and set-up the equipment. These costs are currently estimated to be approximately $1.1B, or 12.4% representing the largest share of the estimated $8.9B in Acquisition costs after URF cost.

Costs related to sustainment set-up are inclusive of all the major components highlighted in JPO’s sustainment estimate.[33]

Acquisition PMO

Similar to Development PMO costs, Acquisition PMO costs are primarily related to personnel, overhead and travel costs for the PMO during the Acquisition phase. These costs are forecasted to be approximately $178M or 2.0% of total estimated Acquisition costs of $8.9B. Based on our review, these costs appear to be comprehensive of expected Acquisition PMO costs.

Diminishing Manufacturing Sources and Concurrency Modifications

Diminishing Manufacturing Sources (DMS) costs relate to the costs associated with losing a source of supply for parts or materials needed in the development, production or post-production of the F-35. Concurrency Modifications are costs associated with modifications to the F-35 resulting from design changes in the Acquisition phase.[34] Overall, these costs represent approximately $70.7M, or 0.8% of total estimated Acquisition costs of $8.9B.

Other Potential Acquisition Costs

Based on discussions with DND, none of the assumptions regarding Other Potential Acquisition Costs have changed since the 2012 or 2013 Annual Update; therefore no modifications to the Model are needed in this regard.[35]

3.4.2.3 Sustainment Costs

Sustainment costs are related to costs for sustaining the F-35 over its expected useful life of 30 years. Overall, Sustainment represents approximately 28.9% of the total LCC estimate. The cost components of Sustainment are described in the table below:

The costs included in the Sustainment estimate are comprehensive of what is included in JPO sustainment cost information.

3.4.2.4 Operating Costs

The Operating phase includes costs related to consumable items for the aircraft, such as fuel and ammunition, as well as costs related to personnel salaries and benefits, base repairs and ongoing training for pilots and maintainers. Overall, the estimated Operating costs are approximately $20.7B, comprising roughly 48.6% of the unadjusted LCC estimate of $42.7B. Within the Model, Operating costs are broken into two (2) sub-categories:

• Personnel costs, which include direct and support personnel salaries and benefits costs at the Cold Lake and Bagotville bases; and
• Operating costs, which include costs of fuel, unit level consumption and base support costs.

Given that Operating cost estimates are based on actual costs related to the CF-18 program, the costs appear to be comprehensive of all costs related to the operations and maintenance of a fighter aircraft fleet.

3.4.2.5 Disposal Costs

Disposal costs in the model reflect the costs to disarm and safely dispose of aircraft that are removed from service. Currently the costs related to Disposal are estimated to be around $137M, or 0.3% of the unadjusted LCC estimate of $42.7B. The current disposal estimate includes costs for:

• Initial salvage work to remove useful components from the aircraft;
• De-militarization of the aircraft; and
• Other miscellaneous disposal costs.

The CF-18 Fleet disposal estimate recognizes that there is potential revenue that could be obtained from salvaged equipment, but given the level of uncertainty around multiple variables related to aircraft disposal, such as demand for salvage components for the CF-18 in future (and likewise the F-35), geopolitical restrictions, and so forth, these revenues have not been estimated.[37] Should more reliable information on aircraft disposal revenues become available in the future, we would expect that DND would adjust their Disposal estimates accordingly.

3.4.2.6 Attrition Costs

Attrition costs are related to the cost of replacing aircraft that are lost during operations. These costs are not treated as part of the full program LCC estimate, but are instead stated on a separate line to respect the assumption that the Government will need to make a decision regarding the replacement of aircraft.[38] The approach to attrition costs in the model is consistent with the 2012 and 2013 Annual Updates, the attrition estimate continues to use the adjusted weighted average cost of an F-35 by the midpoint (9) of the estimated number of replacement aircraft required (7 to 11). Given the current level of information available to the Costing Team, the current approach remains valid.

3.5 Review, Analyze and Update

Principle: Undertake sensitivity, risk and uncertainty analyses and develop risk-adjusted cost estimates. Results are established and documented. Independent cost assurance activities are undertaken and necessary adjustments are made to the NGFC LCC Model.

3.5.1 Sensitivity Analysis

The Framework states that sensitivity analysis should be undertaken and that the results be well documented and communicated. Sensitivity analyses can be leveraged to quantitatively analyze the identified risk factors related to the full program LCC Estimate presented.

As part of the Model, DND has included sensitivity analysis calculations based upon DND and Defense Research Developmental Canada subject matter experts discussions. The Costing Team included details on the methodology behind its sensitivity analyses within its LCC Plan, which includes additional documentation on its risk analysis and contingency.[39] Currently the sensitivity analyses within the model analyze the potential variability in the Acquisition, Sustainment and Operating estimates due to a number of risk factors, including:

• Foreign exchange;
• Inflation;
• The learning and production curve of Lockheed Martin;
• Changes in the total number of aircraft ordered by all JSF partners;
• Shifting the buy profile by one (1) year; and
• Number of annual flying hours.

Based upon our review of the model and supporting documentation, there appears to be a good depth of risk factors considered in the sensitivity analyses. However, sensitivity analysis has not been undertaken to explore the impacts of decreasing the total number of aircraft acquired by Canada on the LCC Estimate. Given that the 2014 Annual Update states that, “If full acquisition contingency was required, the remaining shortfall could be met by buying fewer aircraft”,[40] it would be prudent to conduct sensitivity analysis on this factor’s impacts on the LCC estimate.

Recommendation F1:

Sensitivity analyses conducted were performed in line with Framework principles. However, it is recommended that DND conduct additional sensitivity analysis to explore the impacts on the LCC estimate, should Canada opt to decrease the number of aircraft purchased to respect the $9.0B frozen Acquisition envelope.

3.5.2 Risk and Uncertainty Analysis

Decision makers need to be informed of cost risks and uncertainties relevant to the cost estimates. The Framework sets the appropriate confidence levels range between 50% and 90%. Since the NGFC Project Charter states that the overall risk assessment for the project is “High”, we expected to observe that DND had documented and conducted a risk and uncertainty analysis as well as calculated a contingency budget to address identified risks and uncertainties. This would include stakeholder engagement and updates to the project risk log.

Through the review of the LCC Plan and supporting risk methodology documents, we were able to confirm that a standardized risk management process is in place. A risk and uncertainty analysis was completed to assess:

• The volatility of:
  • Inflation;
  • Canadian dollar vis-à-vis the US dollar; and
  • The flying hour rates.
• The fuel price;
• Potential production delays or reduced availability of aircrafts;
• Changes to the Production Sustainment and Follow-on Development MOU; and
• Whether or not Lockheed Martin realizes estimated production efficiency improvements.

In June 2014, a risk assessment was conducted with the participation of stakeholders’ subject matter experts drawn from across DND, including the Royal Canadian Air Force, the Project management Office, Chief Financial Officer Staff, and other relevant parties. The outcomes of this workshop are detailed in Annex E of the NGFC Cost Report.[41] The identified risks led to the calculation and application of the following 2014 contingencies. 2013 contingency amounts are also shown for comparison. 

The following sub-sections outline the contingencies that have been built into the Model by DND to address identified risks.

3.5.2.1 Acquisition Contingency

The approach to estimating the contingency on Acquisition is consistent with the approach used in the 2013 Annual Update.

An analysis of the current DND estimate of Acquisition contingency, based on the DND analysis of risks, should be $1.1B.[43] However, the full contingency estimated was not applied in order to respect the $9.0B frozen envelope for Acquisition. DND's current contingency provision of $76M does not fully cover the contingency estimated based on risk, and therefore a shortfall in the Acquisition contingency persists. The baseline Acquisition estimate has increased from $8.6B to $8.9B year over year (an increase of 3.5%) and the Acquisition contingency amount has decreased from $342M to $76M year over year (a decrease of 77.8%).[44] The reduction in the Acquisition contingency amount used in the risk adjusted LCC estimate means that the realization of the maximum impact of a single risk factor, such as foreign exchange risk, could potentially result in a significant increase in acquisition costs for the requisite number of F-35s.

The Crown is limited in its ability to appropriately account for identified risks in its LCC estimate since the full Acquisition contingency can not be applied due to the $9.0B frozen Acquisition envelope.

3.5.2.2 Sustainment Contingency

JPO Sustainment cost estimates are based on a parametric costing methodology meaning there is still a significant level of uncertainty and risk related to the estimates. Based on unadjusted Sustainment costs of approximately $12.3B and a contingency estimate of approximately $1.9B, the total risk adjusted Sustainment cost estimate of $14.3B equates to a contingency of roughly 15.4% This is reflective of decreased uncertainty related to the JPO provided Sustainment data since 2013, as a secondary review of the JPO data has been performed.[45] This is in line with Framework principles.

3.5.2.3 Operating Contingency

Currently, Operating cost estimates do not have a contingency amount included in the full program LCC. Given that the current Operating cost estimates are based on the costs related to the CF-18 program, which operates more aircraft than are forecasted for acquisition, it is possible that it might be feasible to save on Operating costs through potential reductions in personnel requirements and other related costs.

3.5.3 Document Results

Estimate results are documented in two ways: through the Model, and in the Annual Update. The Model documentation is discussed in Section 3.5.4 below. The review of the Annual Update is included in Section 3.6.

3.5.4 Model Documentation

To follow leading practices, documentation of the Model should be a continuous process undertaken at every stage of the LCC estimation. The Framework also lists two minimum criteria:

• “Document the model such that another cost analyst unfamiliar with the program could recreate it quickly and produce the same result; and”
• “Create an executive summary that provides sufficient explanation for a non-expert cost modeller to understand the costs and underlying assumptions.”

RCGT observed the Model contained documentation that would allow a reviewer with limited knowledge of the Model to gain an understanding of the inputs, outputs and mechanics behind the Model. As discussed in Section 3.3.2.2, a Configuration Management Plan was developed in line with recommendations made through the 2013 Independent Review.

The NGFC Cost Report developed by DND currently serves as an executive summary document to allow non-expert cost modellers to understand the costs and underlying assumptions.

3.5.5 LCC Assurance

In addition to conducting an internal review of the Model, the Framework requires that it be independently reviewed prior to any major milestone. “The primary purpose is to challenge the existing LCC estimate to ensure it is robust and reliable, taking into account the current life cycle of the project and knowledge of the system under investigation”[46].

The exercise conducted by RCGT and the subsequent results presented in this report constitute an independent review as defined by the Framework. This independent review is performed on an annual basis and aligns with practices in other JSF Partner nations.

There are no recommendations related to the conduct of the Model LCC assurance.

3.6 Interpret and Report Results

Principle: Purpose-focused reports are developed for decision makers and stakeholders in accordance with prescribed guidelines.

The Framework defines the report structure requirements as follows:

“[…] an LCC analysis report structure would bring out the key issues related to the costs presented in a concise, factual and easily understood manner. It includes details pertinent to the decision at hand, including the cost confidence level, risks and uncertainties, summaries of analysis such as sensitivity, risk and affordability analysis, recommendations and conclusions. The report does not assume the reader has a detailed understanding of LCC principles and careful attention should be paid to the expression of uncertainty.”

Per the Framework, cost estimates should be prepared at a minimum of 50% confidence. It is also recommended that baseline and contingency costs be presented against a range of confidence levels that would provide reference points for the decision maker.

The 2014 Annual Update[47] prepared by DND and supported by the Model adequately introduces the subject and decision at hand. It defines and details project phases, cost categories, assumptions and estimates. Cost risks and uncertainties, as well as contingencies, are calculated and presented by range (incremental cost decrease/increase), which follows best practices. Our review indicated that all confidence levels used within the risk analysis met or exceeded the minimum Framework requirements.

Overall, the Annual Update report appears complete and offers proper support to decision-makers.

3.7 People and Organization

Principle: The NGFC estimator team is drawn from a professional costing organization, supported by standard tools, techniques and methods. The Cost Assurance role is integrated into the process with appropriate policies to ensure a non-advocacy approach.

3.7.1 NGFC Estimator Team

The LCC framework identifies people and the organization as a foundational component to the development of robust and reliable costs. During our review, we expected to observe that cost modellers are drawn from a centralized cost organization, that quality assurance activities related to the Model and estimates were undertaken within DND, that employees preparing the LCC model are suitably experienced and that the team is multidisciplinary.

Through documentation review, we were able to confirm that DND has organizationally endorsed and standardized LCC tools/templates tailored to the specific program. DND’s D Cost S has centralized activities and efforts related to building and managing the Model.

The costing team is composed of financial analysts with good knowledge and experience with financial and cost accounting, including planning and budgeting to develop cost estimates. The team responsible for preparing the 2014 NGCF LCC estimate consists of the same individuals that were involved in 2013. As such, year-over-year continuity and experience within the team is strong. 

3.7.2 Cost Assurance Role

According to the Framework and leading practices, the cost assurance team members should be independent and therefore not involved with costing estimates. “To successfully manage these two functions (Estimation and Assurance), formal policy and organizational arrangements are established to guide LCC Estimation and Assurance activities and an NGFC LCC plan would capture the planned LCC approach to LCC Estimation and Assurance for this project”[48]. Per the Framework, the individual responsible for LCC cost assurance should be a recognized subject matter expert. As the cost assurance role is currently a process external to DND, via the Independent Review, DND’s role in cost assurance revolves mainly around internal quality assurance activities related to the Model and estimates.

Through the 2013 Independent Review, it was recommended that DND consider formalizing quality assurance activities through guidelines or policies to ensure roles are clear and structured. In accordance with that recommendation, D Cost S has prepared a draft standard operating procedure for the Testing Process for the Mechanics of Costing Spreadsheets. We reviewed this draft document as part of the 2014 Independent Review and have found that procedures relating to quality assurance have been well developed in line with a risk-based approach that is structured in accordance with DND’s risk framework and DND’s guide for writing financial procedures.[49]  

RCGT reviewed a quality assurance report[50] of the Model dated July 22, 2014. In general, we found the documentation provided was of reasonable quality and met the high-level Framework principles. However, the draft standard operating procedure and quality assurance report did not specify the process for selecting a Model reviewer, nor did they define the responsibilities, required expertise, program knowledge or level of experience of the reviewer.

Recommendation I1:

Cost Assurance activities are performed in line with high-level principles of the Framework, but opportunities for improvement exist. Building on its draft standard operating procedure, DND should detail roles and responsibilities, including required level of expertise, relating to cost assurance.

3.8 Comparative Review

RCGT conducted a comparative review of the 2014 version of the Model to the 2013 version of the Model to identify whether changes in the Model methodology resulted in any material changes in the LCC estimate from year to year.

DND prepared a cost reconciliation as part of the 2014 Annual Update to identify sources of variance related to the Model redesign. The variance sources that were identified and quantified by DND were:

• Foreign exchange rate;
• Inflation;
• Source data and methodology;
• Shifting of the buy profile; and
• Other minor changes.

The comparative review was conducted by randomly selecting 10 versions of the model that were saved between the 2013 Model and the 2014 Model and ensuring that the numbers found within these 10 intermediary versions were accurately captured within the reconciliation.  Additionally, we ensured that the changes occurring within each intermediary version of the model were accurately recorded within the Model change log[51].

Based on this review, it appears that all changes from the 2013 version of the Model to the 2014 version of the Model were accurately recorded in the Model’s change log and that the reconciliation prepared by D Cost S quantifying the impact of each change is materially correct.

Based on the summary of variances provided by DND, the 2013 and 2014 Annual Updates appear to be comparable and there have not been any material changes to the Model mechanics and methodology that would impact the LCC Estimate. DND has also documented amendments made to the Model along with impacts to individual cost elements within the Model itself and has included this information within the NGFC Cost Report.[52]

3.9 Variance Summary

The following table presents the overall variance between 2013 and 2014 estimates.

This variance can largely be attributed to changes in source data and methodology as well as changes to the buy profile. Many of these changes have been driven by changes to Canada’s ground rules and assumptions.

4. Conclusion

As part of the 2012 and 2013 Independent Reviews, a total of 14 recommendations were provided for improvements to the LCC, which are listed in Appendix A of this report. DND has made significant progress to address recommendations resulting from both Independent Reviews, which have led to improvements to the Model and documentation processes.

The 2014 Independent Review has provided one (1) recommendation to help better align the LCC process with the Framework, and one (1) recommendation for additional improvement to the LCC process.

Aside from the observations relating to the $1.0B contingency shortfall, the 2014 Independent Review of DND’s application of the Framework did not reveal any deviations from the Framework’s principles that would result in any material changes to the overall LCC estimate.

With regard to the 2014 Independent Review, the overall assessment of the NGFC LCC process is that DND has continued to improve and refine its processes and methods as the project continues to evolve. In the interim period between the 2014 Annual Update and the 2015 Annual Update, DND should remain focused on addressing any remaining recommendations from the 2012 and 2013 Independent Reviews, along with the recommendations provided within this report.

5. Appendix A | Recommendations from 2012 and 2013 Independent Reviews

5.1.1 Recommendations from 2013 Independent Review[54]

5.1.2 Recommendations from 2012 Independent Review[56]

6. Appendix B | List of Documentation Reviewed

• Consensus Economics. (August 2014). Long Term Consensus Forecasts.
• Department of National Defence. (2013). Next Generation Fighter Capability Annual Update 2013, Draft WME-410294 Ver 10B.
• Department of National Defence. (2014). NGFC Cost Model.
• Department of National Defence. (2014). NGFC Costing Model- Inflation Index Development and Application Review.
• Department of National Defence. (August 2011). PMO NGFC Weapons Cost- An Approach for Consideration.
• Department of National Defence. (2014). Next Generation Fighter Capability-  Annual Update.
• Department of National Defence. (n.d.). CF188 Fleet Long Term Disposal Cost Estimates, RDIMS #1049050.
• Department of National Defence. (December 2012). Next Generation Fighter Capability Annual Update.
• Department of National Defence. (July 2014). PMO Ground rules and assumption.
• Department of National Defence. (July 2014). Spreadsheet Findings Report.
• Department of National Defence. (June 2014). Project 2527: NGFC Infrastructure Program Costing Template.
• Department of National Defence. (May 10, 2014). Draft Annual Update- Work Plan.
• Department of National Defence. (September 2014). Cost Report in Support of the Next Generation Fighter Capability: F-35 Life Cycle Cost Estimate (Annual Update 2014).
• Government of Canada. (December 2012). Seven-Point Plan: Status Report National Fighter Procurement Secretariat.
• Joint Program Office. (February 2014). F-35 Sustainment Cost Ground Rules & Assumptions: 2013 Sustainment Cost Estimate v1.1.
• JPO. (August 2014). Production Drill August 2014-2020 Profile.
• JPO. (February 2014). 2013 Sustainment Cost Estimate: CAN Sustainment Cost Details.
• KPMG. (November 27, 2012). Next Generation Fighter Capability: Independent Review of Life Cycle Cost.
• KPMG. (November 27, 2012). Next Generation Fighter Capability: Life Cycle Cost Framework.
• Raymond Chabot Grant Thornton. (2013). 2013 Independent Review.