Internal Audit of Capital Project Cost Estimation

May 2013

7050-11-42 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

ADM(Fin CS)

Assistant Deputy Minister (Finance and Corporate Services)

ADM(IE)

Assistant Deputy Minister (Infrastructure and Environment)

ADM(Mat)

Assistant Deputy Minister (Materiel)

CFDS

Canada First Defence Strategy

CFO

Chief Financial Officer

CORA

Centre for Operational Research and Analysis

C Prog

Chief of Programme

CRS

Chief Review Services

CSC

Canadian Surface Combatant

D Cost S

Director Costing Services

Defn

Definition

DFC

Defence Finance Committee

DFPA

Director Financial Planning and Analysis

DGMSSC

Director General Materiel Systems and Supply Chain

DMPP

Director Material Policy and Procedures

DND

Department of National Defence

DRDC

Defence Research and Development Canada

FAA

Financial Administration Act

FY

Fiscal Year

ILS

Integrated Logistics Support

IM

Information Management

Impl

Implementation

LRPT

Long Range Planning Tool

MND

Minister of National Defence

O&M

Operations and Maintenance

OAG

Office of the Auditor General

OPI

Office of Primary Interest

PA

Project Approval

PAD

Project Approval Directive

PWGSC

Public Works and Government Services Canada

SS(ID)

Synopsis Sheet (Identification)

VCDS

Vice Chief of the Defence Staff

UK

United Kingdom

US

United States

Results in Brief

Overall Assessment

While project cost growth is not as high as that of some allies, improvements in standards for early project cost assumptions and changes to DND policy on estimate confidence levels are necessary for more reliable project estimates. The overall value of revised expenditure approvals has been relatively low for the capital program. However, within acceptable tolerance levels of operational risk, some capability trade-offs have been necessary for some projects to remain within budget ceilings.

Since 2008, Chief Review Services (CRS) has identified certain concerns regarding the accuracy of cost estimation in five capital acquisition internal audits. The Office of the Auditor General (OAG) has also raised concerns regarding project cost growth in a recent audit of DND acquisitions.1 Given the magnitude of the capital program in the 2008 Canada First Defence Strategy (CFDS), an audit of project cost estimation was included in the CRS Risk-based Audit Plan. The Departmental Audit Committee directed that this audit begin in fiscal year (FY) 2011/12. The objective of the audit was to assess the reliability and rigour of the costing methodology used to estimate capital project costs. The audit focussed on project value estimates and the independent challenges carried out at three main project decision points that occur at approximately two year intervals, to establish the project budget before a contract is awarded to the defence industry: a rough estimate at the project identification phase; an indicative2 estimate at the Project Approval (Definition) phase; and a substantive3 estimate at the Project Approval (Implementation) phase. Three major initiatives that should contribute to improved cost estimation were taking place concurrently with this audit: the Vice Chief of the Defence Staff Project Approval Process Redesign, related Defence Renewal Team initiatives, and a Treasury Board independent review of the life cycle cost framework for major military acquisitions. Lastly, as part of the Chief Financial Officer (CFO) transformation, it was planned that the cost estimation staff capacity would double in size.

Findings and Recommendations

Cost Estimation Accuracy. Insufficient rigour was applied to the cost estimation assumptions in the project identification phase due to insufficient standards for original cost estimates. The audit team’s review of seven identification phase project estimates found that long-term escalation rates were not standardized and contingency was not included in four of the projects. Although the DND Economic Model escalation rates are not intended to include technology improvements, some project sponsors took into account the technology enhancement costs for replacing equipment while others did not. The new mandate of the Defence Capability Board to ensure project capabilities are necessary and affordable may resolve this issue. Additionally, the original cost estimates will be challenged by the Director Costing Services before identification phase projects are included in the 20-year Long Range Planning Tool (LRPT).

In subsequent project phases, schedule delays, scope increases, unpredictable inflation and delays in engaging industry have contributed to cost growth. Over the last 7.5 years, the value of projects in the definition phase increased by 3.6 percent and the value of projects in the implementation phase increased by 1.6 percent. Cost growth associated with project delays was due to shortfalls in project management staff and the lengthy project approval process.4 The VCDS Project Approval Process Redesign project was on-going at the time of the audit, with the intention of streamlining the capital acquisition cycle time by 50 percent and doubling the throughput of project approvals. While recent cost validations in the options analysis and definition phases have not extended the project approval process, too much validation effort was found to be devoted to low-value items.

Higher confidence estimates would have been possible if industry had been engaged earlier for price and availability quotes before Project Approval (Definition) and if contracts had been tendered earlier. Unlike Major Crown Projects,5 lower-value projects were tendered to industry after Project Approval (Implementation). Delays in tendering contracts were due to a misunderstanding of Section 32 of the Financial Administration Act (FAA), which only requires certification of the availability of funds prior to contract award, not at the initiation of the tendering process.

It is recommended that clear standards for escalation rates, capability improvement factors, infrastructure estimates and contingency funding be developed for original project cost assumptions in the identification phase. In addition, cost validation activities should be risk-based to focus limited resources in areas of greatest benefit. The Project Approval Directive (PAD) should be revised to advance the timing of price and availability activities. It is also necessary to verify with Public Works and Government Services Canada (PWGSC) that lower-value projects can be tendered before Project Approval (Implementation) without FAA Section 32 certification.

Expected Confidence Level of Estimates. For complex military acquisitions, accurate project estimates early in a project life cycle are rarely possible given the insufficient maturity level of requirements and design. DND policy requires identification phase life cycle indicative cost estimates to have a high confidence level of +/- 25 percent even though the range could be +/-50 percent according to industry guidelines.6 Therefore, some project budgets were based on low confidence estimates that resulted in the use of capability trade-offs to offset cost increases and to remain within budget. There are project briefs required at the three main decision gates that have the option of discussing the confidence level of the cost information. This option was not exercised in a sample of recent project briefs. DND plans to adapt its existing costing tools and procedures once an updated Treasury Board procurement and project approval policy is promulgated.

It is recommended that the DND Costing Handbook and PAD be revised to address a sufficient range of cost variance in each project phase in accordance with industry standards and that the discussion of the confidence level of cost information in project briefs be mandatory.

Project approval policy requires precise cost estimates too early in a project life cycle. By not engaging industry earlier, project estimates have lower confidence levels. This results in some capital projects with budget ceilings that require capability trade-offs to control costs while remaining within acceptable tolerance levels of operational risk.

Note: For a more detailed list of CRS recommendations and management response, please refer to Annex A—Management Action Plan.

Introduction

Rationale for Audit

Since 2008, CRS has completed five audits of capital acquisition projects that identified certain concerns regarding the accuracy of cost estimates. A recent OAG report has also raised concerns regarding the cost growth in original project estimates. Given the magnitude of the capital program outlined in the 2008 CFDS, these audit observations warranted the inclusion of this audit in the CRS Risk-based Audit Plan for FY 2011/12 to FY 2013/14. The Departmental Audit Committee directed that this audit commence in FY 2011/12.

Background

Recent Initiatives. During the audit conduct phase, the following three initiatives were under way, and this should positively influence the DND cost estimation framework:

  • In June 2012, the VCDS initiated the Project Approval Process Redesign project. The desired outcome of the project is the effective management of DND expenditures with project approval processes that are proportional to complexity, cost, urgency, importance and risk. The aim is to reduce project acquisition cycle time by 50 percent while doubling the throughput of projects. Co-chaired by the VCDS, the Project Approval Process Redesign council includes Canadian Armed Forces/DND stakeholders and representatives from the Treasury Board Secretariat, PWGSC, and Defence Construction Canada. Given the extensive work that CRS has done in the past7 on streamlining the project approval process, it was agreed that CRS would act as an observer on the Project Approval Process Redesign working group. In this capacity, the audit team provided information to the Project Approval Process Redesign project staff on capital program slippage.
  • The Defence Renewal Team was established in August 2012 to support key activities relating to spending reduction and process reform by serving as a central point of contact on change initiatives for a three-year period. During the audit reporting phase, the cost avoidance opportunity associated with project approval streamlining was briefed to members of the Defence Renewal Team.8
  • An independent review of the life cycle cost framework for large military acquisition projects was initiated by Treasury Board and completed in January 2013. The report highlighted the challenges of estimating precise life cycle costs in the early phases of a project. It stated that the “Government and the public often seek a higher degree of precision than can be justified with the information available.”9 A related report stated that the April 2006 DND Costing Handbook life cycle costing policy was comparable to the leading international best practices.10 Based on the independent review recommendations, it is the intention of Treasury Board to promulgate a new policy on procurement and project approval policy that will encompass a cost estimation framework.

Cost Estimation Framework. At the three major decision gates in a major capital project’s11 nine-year life cycle, project costs are estimated then followed by an independent cost validation by the Director Costing Services as portrayed in Figure 1. The process details are as follows:

  • At the outset of a project, known as the identification phase, a rough life cycle cost estimate is proposed by a Level 1 project sponsor. Prior to inclusion in the 20-year LRPT and the five-year Investment Plan, a review is performed by Director Costing Services and the project is approved by the Defence Finance Committee (DFC). All projects that require definition or implementation phase funding are to be included in the Investment Plan. Therefore, an identification phase project estimate could be required four years in advance of the project definition phase that will be approved in the fifth year of the Investment Plan.

Figure 1. Project Approval Process. To reduce the life cycle time between the identification approval and the close-out phase from thirteen years to nine, the milestones for each phase were set by the Project Management Board in December 2003. Normal two-year definition phases prescribed in the PAD may be extended for complex developmental projects and the implementation phase may also be longer. For construction projects, the implementation phase is usually less than three years.

Text description for Figure 1:

This flowchart demonstrates the target duration of and sequence for the project approval process in DND. It has three different starting points, representing the project approvals, the five-year investment plan line and the various project approval phases as follows:

Starting Point A (Project Approvals):

  • Moves from left to right and includes the three typical approvals required as part of the project approval process: identification approval, project approval (Definition) and project approval (Implementation).
  • For each approval, the end of a phase and the beginning of a new phase is indicated. The identification phase ends when identification approval has been granted, which then begins the options analysis phase. The options analysis phase ends upon receipt of project approval (Definition), which then begins the definition phase. Lastly, the definition phase ends upon receipt of project approval (Implementation) which then begins the implementation phase. After the completion of the implementation phase, the close-out phase marks the end of the project.

Starting Point B (Five-year Investment Plan):

  • Starts from the middle of the definition phase and is assumed to be a realistic date where Vote 5 money is spent and moves backwards to show the approximate time of the first estimate.
  • This line signifies that the very first cost estimate must be made five years prior to the point at which Vote 5 money is expected to be spent—or four years prior to project approval (Definition).

Starting Point C (Project Phases):

  • Starting from the left and moving right, the five stages to the project approval process are indicated.
  • The identification phase and close-out phase do not require approvals.
  • The three middle phases—options analysis, definition and implementation—require approvals.
  • After a two-year options analysis phase, and depending on the risk profile of the project, expenditure approval by the Minister of National Defence (MND) or the Treasury Board will be required for the definition phase costs based on a substantive estimate. An indicative cost of the total project acquisition value is also estimated. Known as Project Approval (Definition), this two-year phase amounts to an average of 6.9 percent of the total project value. During this phase, the project office provides greater refinement to the indicative project estimate through increased engagement with industry.
  • Once the two-year definition phase is complete, expenditure authority, known as Project Approval (Implementation), is sought for the five-year implementation phase – the complete project acquisition budget based on a substantive estimate. Life cycle support costs are acknowledged, but only the acquisition costs are approved.
  • Should the expenditure authority for the definition or implementation phase be exceeded, the project must seek re-approval by the MND or Treasury Board. In these situations, all revised expenditure approvals greater than 10 percent or $10 million must be reviewed by the DFC.12

Objective

The objective of this audit was to assess the reliability and rigour of costing methodology used to estimate capital project costs.

Scope

The scope of the audit included the following:

  • major capital equipment, infrastructure and Information Management (IM) projects only – minor projects less than $5 million were excluded;
  • estimates in all phases of a project’s life cycle: identification, options analysis, definition and implementation.

The audit had the following scope limitations:

  • Given the ongoing work of the VCDS Project Approval Process Redesign project taking place concurrently with the audit, a detailed process analysis to determine the root causes of schedule delay was excluded.
  • Although it is standard practice in DND to consider life cycle costs during the options analysis phase of a project, it was decided at the outset of the audit that life cycle support costs would be excluded from the scope and be treated as a separate audit.13

Methodology

The audit results are based on evidence from the following sources:

  • interviews with key DND, PWGSC, and Treasury Board staff;
  • an examination of policies, project documentation, the Capability Investment Database, the corporate submission database, and the Investment Plan;
  • a comparison with United Kingdom (UK) and United States (US) defence project cost estimation results for similar types of combat systems;
  • a sample of 14 current IM, infrastructure, and equipment projects in different phases representing 33 percent of the major capital program’s value. (Each project office completed one questionnaire and participated in focus group discussions. Annex B contains the list of the projects reviewed.);
  • from January 2005 to July 2012, 348 projects endorsed by the Project Management Board for definition or implementation phase funding; and
  • a case study of the Canadian Surface Combatant (CSC) project cost estimates in the identification phase (See Annex C for detailed results)

Criteria

The audit criteria are outlined in Annex D.

Statement of Conformance

The audit findings and conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit thus conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the entity examined.

Findings and Recommendations

Cost Estimation Accuracy

Shortfalls exist in the cost estimation assumptions for projects in the identification phase. Project delay, scope increase, unpredictable inflation and delayed engagement with industry contributed to cost growth from the options analysis phase to the implementation phase.

Original Project Estimates

Although projects in the identification phase have limited information to create high quality estimates, shortfalls were evident in some cost assumptions made by project sponsors for long-term projects. As the approval authority for project briefs in the identification phase, the Defence Capability Board was recently mandated in December 2012 to challenge the capability and affordability of all future projects. This new mandate may address the audit observations on a sample of seven projects in the identification phase that aim to replace or upgrade major combat systems beyond the year 2022. Although these projects have not yet been subject to review by Director Costing Services prior to inclusion in the LRPT, the following observations were made:

  • Contingency was not included in four project estimates, which normally range from 10 to 15 percent for Major Crown Projects and may be as high as 20 percent for complex projects.
  • Long-term cost escalation factors were not in accordance with the DND Economic Model. Tailored escalation models are only applied to Major Crown Projects once they reach the options analysis phase.
  • In some cases, cost escalation assumptions were based on replacing the same combat system without technology improvements. According to a firm that specializes in research and development, the technology/capability improvements alone for military aircraft amounted to 4.6 percent per year over a 30-year period.14 These escalation rates account for capability/technology enhancements driven by such factors as interoperability requirements and current defence industry norms.

Construction in Support of Equipment Projects. Defence policy requires equipment projects to include the associated infrastructure such as jetties, training facilities, and hangars. Initial infrastructure costs were underestimated for an audit sample of 12 equipment Major Crown Projects. A shortfall of construction engineering staff to provide early infrastructure cost estimates is a major cause. While the initial infrastructure estimates were 1.6 percent of the indicative costs of the 12 equipment projects, these estimates grew to an average of 6.2 percent over time. The new challenge function provided by the Defence Capability Board and the Director Costing Services, before inclusion in the LRPT, should alleviate this issue that has been observed by CRS in a past audit.15

Good Practices

The Family of Land Combat Vehicle projects were authorized 3.0 percent of the project costs for project definition, to avoid the project delay associated with substantive estimates.

Project Cost Growth. There are a number of cost growth factors in the project phases that follow the first identification phase estimate. From the review of 165 projects that began the definition phase since January 2005, the value of these projects increased by 3.6 percent. In the same time frame, the value of 183 projects in the implementation phase increased by 1.6 percent. Although some projects were de-scoped to remain within budget, our analysis of the 63 revised expenditure approvals identified the following root causes for cost growth: project delay, scope increase, inflation, and delayed engagement with industry.

Project Delay. Although the project delay costs related to the lengthy project approval process may be addressed by the VCDS Project Approval Process Redesign project, shortfalls in project management capacity16 are also a significant cause for project delay. Over the last three fiscal years, 17 percent of the capital acquisition budget allocations were postponed to future years. Reviews of the capital program17 indicate most delays occur in the options analysis and definition phases.

One element of the lengthy capital acqusition cycle time is the 170-working-day corporate submissions process currently required for both the definition and implementation phase expenditure approvals. Although a target submission cycle time of 113 days was set by the VCDS in July 2011, there have been significant obstacles in achieving this goal.18 The VCDS Project Approval Process Redesign project intends to streamline the capital acquisition process for lower-risk projects, which represented 89 percent of all projects at the time of the audit.

Focus groups with project management personnel from the audit sample of 14 projects have indicated that too much cost validation time was spent on low-value items such as travel and rounding errors. Although the DND Costing Handbook states that items under three percent of the project value do not require as much cost validation effort, all cost information is routinely validated.

Scope Increase. Known as the “triple constraint” in project management, project cost is interdependent with the schedule and the scope of the requirements. Since January 2005, there were 19 revised expenditure approvals for projects that involved an increase in scope. For construction projects, this was due to changes in the national building code and understated requirements in preliminary design. For equipment projects, the number of deliverables increased due to the new capabilities required in the Canadian Forces Transformation.

Inflation and Foreign Exchange. Volatile construction costs, particularly in Western Canada, have affected 16 construction projects. Unpredictable foreign exchange was also a factor in some off-shore procurements. Some countries manage foreign exchange risk at the national level rather than having their defence departments absorb this pressure.

Early Engagement with Industry. Projects are not obtaining price and availability estimates from industry sufficiently early to provide higher confidence levels. The PAD states that price and availability activity should take place in the definition phase. This has occurred in six of the seven projects reviewed. Often, however, this activity takes place too late to obtain indicative project costs for Project Approval (Definition). As well, if contract terms and conditions are communicated early to industry, then price and availability quotes would include the risk premiums associated with contractor liability or liquidated damages.

In accordance with the PAD, tendering prior to the Project Approval (Implementation) is the most concrete means of obtaining substantive costs for capital equipment acquisitions and is the normal practice for high-valueMajor Crown Projects.19 However, an audit sample of 59 lower-value projects found that 46 projects were tendered after Project Approval (Implementation). Due to the lower confidence in these implementation phase estimates, the projects included more contingency funding that is often not spent. In a sample of 63 completed projects, it was determined that 68 percent of the projects that were lower in value did not spend any contingency funds. Some commodities have a bid price that expires before expenditure approval can be obtained. Whenever possible, requests for proposals should include a bid validity period that is sufficient for the duration of the approval process.

The intent of FAA Section 32 certification is to provide assurance that funds are available prior to a contract award, not at the initiation of the tendering process. Substantive project costs are not usually known until after the tendering process. However, with the exception of Major Crown Projects, PWGSC requires Section 32 certification to initiate the tendering process as a convenience to make it unnecessary to verify the availability of funds with DND just prior to contract award. This Section 32 requirement acts as an obstacle for DND staff to initiate the tendering process without expenditure authority. Although there is a risk of tendering a contract that may ultimately not be awarded, the projects are in the DND five-year Investment Plan and the project offices are already established in the definition phase. Following a tendering process similar to Major Crown Projects for lower-value projects would add certainty to cost estimates prior to implementation expenditure approval.

Summary. Recent controls have been put in place to challenge original project sponsor estimates in the identification phase and provide a consistent approach to estimating project cost escalation rates, technology improvement factors, infrastructure requirements, and contingency funding. Clear criteria for cost assumptions in the identification phase would reduce the amount of validation effort needed before a project is endorsed for inclusion in the LRPT. Engagement with industry for price and availability did not occur for some high and low-value projects until the definition phase. The tendering process did not take place for most lower-value projects until after expenditure approval for the implementation phase. Therefore, the estimates were lower in confidence and required more project contingency funds, and were later found to be unnecessary for those lower-value projects.

Recommendations

1.        The Assistant Deputy Minister (Finance and Corporate Services) (ADM(Fin CS)) should, in conjunction with the VCDS, develop clear standards for project sponsors in the area of escalation rates, capability improvement factors, infrastructure estimates, and contingency funding for identification phase project cost assumptions before projects are included in the LRPT. In addition, cost validation activities should be risk-based to focus limited resources in areas of greatest benefit.

OPI: ADM(Fin CS)

2.        VCDS should revise the PAD to advance the timing of price and availability activity.

OPI: VCDS

3.        To obtain greater confidence in cost estimates, Assistant Deputiy Minister (Materiel) (ADM(Mat)), in conjunction with ADM(Fin CS), should verify with PWGSC that tendering of contracts can proceed without expenditure authority and FAA Section 32 certification.

OPI: ADM(Mat)

Confidence Level of Estimates

High confidence project estimates early in the project life-cycle are rarely possible due to the insufficient maturity level of requirements and design.

Confidence Level Policy. Projects in the identification phase that require development or significant modification of existing products are expected to estimate complete life cycle indicative costs up to six or more years before a contract award. In this phase, project requirements have not been refined and a design does not even exist. Although the industry guidelines relating to the Project Management Body of Knowledge for projects in the identification phase could be a variance estimate of +/- 50 percent, the PAD requires identification phase life cycle indicative cost estimates to have a range of +/- 25 percent, as defined in the DND Costing Handbook. According to nine of the fourteen focus groups, there is insufficient staff in the identification phase to support the costing exercise. As an example, the estimation challenges faced by the CSC project are in Annex C.

For approval of identification, definition, and implementation phases, the PAD requires a project brief to discuss how the scope, schedule and costs are established as a performance baseline. It is optional to discuss the strength and confidence of cost data, an option that was not implemented in a sample of 10 recent project briefs.

Identification Phase Requirements. Project requirements are not sufficiently developed in the identification phase to enable high confidence estimates. High-level mandatory capabilities are a general description of an operational requirement, but they do not take into account the type of equipment or a specific solution. Even if a preliminary statement of requirement was completed, the confidence level of the estimate would still be less than 60 percent according to the 14 questionnaires received from the audit sample of projects.

Summary. The DND policy of a +/- 25 percent variance for indicative costs in a project’s identification phase does not recognize the wide range of variance associated with developmental projects. As well, the confidence level of cost information is not mandatory in project briefs. This policy has led to some constrained project budgets that require capability trade-offs with acceptable operational risk to remain within expenditure approval ceilings.

Recommendations

4.        ADM(Fin CS) should revise the Costing Handbook to provide a sufficient range of cost variance for developmental projects for each phase to reflect industry guidelines.

OPI: ADM(Fin CS)

5.        VCDS should revise the PAD to reflect the confidence levels in the Costing Handbook and require mandatory discussion of cost data confidence in project briefs.

OPI: VCDS

Cost Estimation Training, Tools and Staff

Current shortfalls in training, tools, and staff will hamper the CFO’s new expanded cost estimation role if left unaddressed.

Good Practice

Standard cost templates are used by Assistant Deputy Minister (Infrastructure and Environment (ADM(IE)) and Director General Maritime Equipment Project Management staff.

CFO Transformation. Since June 2011, ADM(Fin CS), as the CFO, has had an expanded responsibility for financial resource allocation, costing, in-year financial management, maintenance, and reporting of the Investment Plan, as well as for Treasury Board submissions. To increase the capacity of the 12 staff in the Director Costing Services cost validation section, six personnel were recently hired to commence cost estimation training. As the Director Costing Services staff will more than double in size by September 2013, it is intended to have their cost estimators embedded in high-value project offices and provide a centralized resource to assist smaller projects.

Cost Estimation Training Certification. While Director Costing Services cost validators have extensive experience, educational backgrounds and accounting designations, there are few certified cost estimators at DND. The Society of Cost Estimation and Analysis and the Association for the Advancement of Cost Engineering International are two organizations that provide certification for which the added value may outweigh the associated costs. With a four-year university degree, the Society’s core certifications consist of two exams and three years of experience; whereas the Association’s Cost Engineer certification requires four years of experience and the completion of a research paper. DND currently relies on informal on-the-job training and does not offer training related to cost estimation. Director Costing Services is currently evaluating costing capability requirements for the purposes of introducing life cycle costing with existing certification and training available in Canada and other countries.

Cost Templates. With the exception of Royal Canadian Navy and construction projects, there are no standard detailed cost templates for project office estimates. However, from our 14 focus groups, the main needs identified were as follows:

  • a comprehensive listing of all potential cost items to be included in an estimate;
  • flowcharts describing the numerous steps and considerations in a cost estimate as templates change frequently; and
  • a central repository of best practices or current examples available to facilitate projects in developing quality cost estimates.

Tools. There are requirements for improved DND costing tools. Since 2008, ADM(Mat) has had a requirement for a costing tool to make costing data more easily replicated. However, this requirement is currently deemed a low priority – number 18 of 20 on the list of Materiel, Acquisition and Support priorities. As part of the CFO transformation, ADM(Fin CS) has determined that a requirement exists for costing tools, such as the Automated Cost Estimating Integrated Tools, to produce better cost estimates. A review of tools that projects have used in the past is planned by Director Costing Services.

The DND Costing Handbook could be improved with additional examples and guidelines. For example, the current cost model for Integrated Logistics Support (ILS)20 acquisition costs does not take into account various procurement types. Annex E is a CRS analysis of past ship, vehicle, and aircraft projects performed to assess the different ILS breakouts in acquisition and operations and maintenance (O&M) costs. The audit focus groups also suggested a greater need for increased use of tailored escalation models for individual projects. Currently, there is only one tailored model for a Navy project in the DND Economic Model.

Shortfalls in Cost Estimation Staff. There are shortfalls in the capacity of ADM(Fin CS) to fulfill the new role of project cost estimation. The historical average of completed cost validations was 46 per year; however, the demand for cost estimation/validation will likely triple in future years. As well, for some Major Crown Projects, there will be an increase in cost estimation workload associated with the new Memorandum to Cabinet format that requires life cycle cost analysis for all alternative options.

Summary. There are plans to address the current limitations of the independent cost estimation capacity in the Department to accommodate the expanded role of ADM(Fin CS) in project cost estimation. The shortfall in standard cost templates, tailored escalation models, and forecasting tools reduces the confidence level of project estimates.

Recommendation

6.        ADM(Fin CS) should continue to address the capacity and skills of cost estimation staff, standardize project cost templates with a comprehensive cost breakdown list, and acquire improved tools to facilitate cost estimation.

OPI: ADM(Fin CS)

Performance Measurement

Measurement of cost estimation performance is not taking place to compare actual costs to original estimates and help improve cost estimation accuracy.

Good Practices

ADM(IE) projects have a standard cost breakdown structure for construction projects and a project cost history database to track cost estimation performance that enables the analysis of cost drivers.

Historical Cost Comparisons. There are no standard project cost breakdowns for all types of projects seeking expenditure approval. Although every project is required to complete a post-completion report, there is no standard cost breakdown template. Therefore, it is not possible to compare actual costs to original estimates. From our examination of 95 completed projects, 72 percent of the post-completion report cost breakdowns could not be matched to the implementation expenditure approval cost breakdowns. As well, it was not possible to determine the original identification and definition phase estimates due to the nine-year life cycle of projects.

Benchmarking with Allies. Currently, DND does not compare cost estimation performance with our allies. Todetermine if DND’s cost estimation accuracy resembled that of allies, CRS compared 20 DND high-value projects in the implementation phase to the cost growth of similar projects reported in US and UK audit reports. While the median unit cost increase of deliverables of the Canadian projects from Project Approval (Definition) was 12.4 percent,21 the cost growth was higher for both the US projects at 15.1 percent and the UK projects at 13.5 percent.

Summary. With the exception of construction projects, expenditure approval documents and post-completion reports do not have a standard cost breakdown format to enable performance measurement to improve cost estimation. Without cost estimation benchmarking, it is difficult to establish reasonable cost estimation performance targets.

Recommendations

7.        For all types of projects, ADM(Fin CS) should develop standard cost breakdown formats for expenditure approval and the post-completion report, and conduct performance measurement of cost estimation at each phase of similar projects to generate cost breakdown models and benchmark cost estimation performance with allies.

OPI: ADM(Fin CS)

8.        VCDS should reflect standard cost breakdowns for expenditure approval and post-completion reports in the PAD.

OPI: VCDS

Conclusion

While some improvements to the cost estimation framework are necessary, efforts currently being made by senior management to improve the reliability and rigour of costing methodology should move DND forward in its ability to estimate capital project costs. It is clear that, as stated in a Treasury Board-funded review of the life cycle cost framework, the government and the public often expect greater cost estimation precision than is possible with the available information. Within this government context, the Treasury Board is currently revisiting its policy framework on procurement and project approvals, an initiative that is intended to address some of the system-wide issues that have surfaced in recent years.

Annex A—Management Action Plan

CRS uses recommendation significance criteria as follows:

High—Controls are not in place or are inadequate. Important issues are identified that could negatively impact the achievement of program/operational objectives.

Moderate—Controls are in place but are not being sufficiently complied with. Issues are identified that could negatively impact the efficiency and effectiveness of operations.

Low—Controls are in place but the level of compliance varies.

Cost Estimation Accuracy

CRS Recommendation (High Significance)

1.        ADM(Fin CS) should, in conjunction with the VCDS, develop clear standards for project sponsors in the area of escalation rates, capability improvement factors, infrastructure estimates, and contingency funding for identification phase project cost assumptions before projects are included in the LRPT. In addition, cost validation activities should be risk-based to focus limited resources in areas of greatest benefit.

Management Action

Develop a standardized approach and cost estimating tools to cost and validate major capital projects before their inception into the Investment Plan.

ADM(Fin CS) is transitioning from its cost validation process to a full costing process, which includes having an embedded coster in the Project Management Office. To ensure a challenge function, a new standard and evaluation cell will be created within Director Costing Services.

OPI: ADM(Fin CS)/D Cost S/DFPA/DRDC/CORA

Target Date: December 2013

CRS Recommendation (High Significance)

2.        VCDS should revise the PAD to advance the timing of price and availability activity.

Management Action

Once ADM(Fin CS) has developed the new standards, VCDS/Chief of Programme (C Prog) will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new standards from ADM(Fin CS).

OPI: VCDS

Target Date: June 2014

CRS Recommendation (High Significance)

3.        To obtain greater confidence in cost estimates, ADM(Mat), in conjunction with ADM(Fin CS), should verify with PWGSC that tendering of contracts can proceed without expenditure authority and FAA Section 32 certification.

Management Action

Notwithstanding the constraint of the bid validity period stated in the tendering document, which may limit the time required to obtain approval that necessitates expenditure authority, ADM(Mat) will lead an initiative, in conjunction with ADM(Fin CS) and PWGSC, to validate whether tendering of contracts can proceed without expenditure authority and FAA Section 32 certification.

OPI: ADM(Mat)/DGMSSC/DMPPOCI: ADM(Fin CS), ADM(Mat)/DCOS/DMG Compt

Target Date: November 2013

Confidence Level of Estimates

CRS Recommendation (Moderate Significance)

4.        ADM(Fin CS) should revise the Costing Handbook to provide a sufficient range of cost variance for developmental projects for each phase to reflect industry guidelines.

Management Action

D Cost S will update the Costing Handbook and develop a costing methodology and framework for projects at the identification phase before inclusion into the LRPT.

OPI: ADM(Fin CS)/D Cost S/DRDC/CORA

Target Date: December 2013

CRS Recommendation (High Significance)

5.        VCDS should revise the PAD to reflect the estimation confidence levels in the Costing Handbook and require mandatory discussion of cost data confidence in project briefs.

Management Action

Once ADM(Fin CS) has revised the Costing Handbook, VCDS/C Prog will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new Costing Handbook from ADM(Fin CS).

OPI: VCDS

Target Date: June 2014

Cost Estimation Training, Tools and Staff

CRS Recommendation (High Significance)

6.        ADM(Fin CS) should continue to address the capacity and skills of cost estimation staff, standardize project cost templates with a comprehensive cost breakdown list, and acquire improved tools to facilitate cost estimation.

Management Action

Costing capability requirements will be assessed. Standard qualifications will be developed. A training program will be implemented for D Cost S personnel.

Standardized templates will be developed at the macro level in accordance with Treasury Board Secretariat guidelines.

D Cost S and the Centre for Operational Research and Analysis (CORA) are currently assessing different tools (software) to facilitate cost estimation.

OPI: ADM(Fin CS)/D Cost S

Target Date: March 2014

Performance Measurement

CRS Recommendation (Moderate Significance)

7.        For all types of projects, ADM(Fin CS) should develop standard cost breakdown formats for expenditure approval and the post-completion report, conduct performance measurement of cost estimation at each phase of similar projects to generate cost breakdown models and benchmark cost estimation performance with allies.

Management Action

As per the second bullet of 6, above, standardized templates (cost breakdown structure level 1) will be developed at the macro level in accordance with Treasury Board Secretariat guidelines.

D Cost S will institute a project closure benchmarking and reporting initiative in order to capture performance data.

OPI: ADM(Fin CS)/D Cost S

Target Date: March 2014

CRS Recommendation (Moderate Significance)

8.        VCDS should reflect standard cost breakdowns for expenditure approval and post-completion reports in the PAD.

Management Action

Once ADM(Fin CS) has developed the new standards, VCDS/C Prog will review the changes and revise the PAD to reflect the changes as appropriate. The PAD changes should be fully implemented within six months of receipt of the new standards from ADM(Fin CS).

OPI: VCDS

Target Date: September 2014

Annex B—Audit Sample of Current Projects

Table 1. CRS Audit Sample of Current Projects. A mixture of equipment, IM and construction projects in different phases was reviewed by way of focus group questions and survey.

Table Summary:

The project numbers are listed in the left-hand column. Read across the rows to determine the characteristics of the project, such as name, type, the project approval that was analyzed and the current phase of the project.

 

Project NumberProject NameProject TypeProject Approval AnalyzedCurrent Phase
C.001336 Canadian Surface Combatant Equipment SS(ID) Defn
C.002673 Joint Support Ship Equipment PA(Defn) Defn
C.001007 Medium to Heavy Lift Helicopter Equipment PA(Impl) Impl
C.001430 Tactical Armoured Patrol Vehicle Equipment PA(Impl) Impl
C.001035 Joint Unmanned Aircraft Surveillance Target Acquisition System Equipment SS(ID) Options Analysis
00002716 LAV Reconnaissance Surveillance System Equipment PA(Defn) Defn
C.002523 Mercury Global IM PA(Defn) Defn
C.002525 Small Arms Modernization Equipment SS(ID) Options Analysis
C.001802 Polar Epsilon II IM SS(ID) Options Analysis
C.000032 Tactical Integrated Command, Control and Communication Air Equipment PA(Defn) Defn
00000806 Marine Security Operations Centres IM PA(Impl) Impl
C.004601 Accommodate 4 Engineer Support Regiment (Gagetown) Construction

SS(ID)

Options Analysis

C.000875

Maintenance Facility Extension Wainwright

Construction

PA(Impl)

Impl

C.001490

Increase Academic and Training Aids Capabilities – St-Jean Garrison

Construction

PA(Defn)

Defn

Annex C—Canadian Surface Combatant Case Study

As the CSC project will be the most significant capital expenditure in DND, it was used as a case study to determine the cost estimate challenges facing a developmental project in the identification phase. As stated in the 2008 CFDS, the CSC project will replace the three Tribal-class destroyers and 12 Halifax-class frigates. During the four-year identification phase that commenced in 2005, the CSC project had five estimates ranging                                                                                                           22 using the DND Economic Model escalation rates. Revised estimates were necessary as benchmarks from our allies became available. In accordance with the capital acquisition process, the CSC project did not have the benefit of a preliminary statement of requirement which was delivered in the options analysis phase in November 2010.

These estimates were made at an early stage in the identification phase and resulted in a                                                                                                                                       It was acknowledged by the Department in March 2010 that there would                                                                                                                                                                                                                             23 Based on current Halifax-class annual Personnel, Operations and Maintenance costs, for                                                                                                                                                                                           24                                                                                                                                                                                                                                                                                                

A number of good practices were observed when reviewing the various cost estimates performed in the identification phase of the CSC project.

  • In a 2006 estimate, a range of costs were considered: +50 percent / -25 percent and, if applied,                                                                                                                        
  • Costs were benchmarked with allies.
  • Transferable modules for high-readiness ships were considered in 2006 to reduce costs.
  • Automation studies were done in 2006 for a number of ship subsystems that had the potential of reducing a ship crew by 50 percent.
  • In a 2008 estimate, sensitivity analysis was done with two annual escalation factors—1.8 percent and 3.5 percent. This resulted in an estimate range from                                                              
  • The 2008 estimate had 31 cost elements and 49 cost assumptions—significantly more than the earlier estimates.

A better project estimate would have resulted if not for the following practices:

  • In one of the estimates, the learning curve based on the Canadian Patrol Frigate project was applied to both the ship construction and program costs.
  • The incremental cost difference between a frigate and area air defence command ship was underestimated as 10.9 percent versus 46 percent.
  •                                                                                                                                                                                                                                                                                                                                                                                                                                                           This definition funding was split into two phases                                                                                                           25                                                                                                                                                                                                        

Annex D—Audit Criteria

Objective

To assess the reliability and rigour of costing methodology used to estimate capital project costs.

Criteria

  • Risk-based guidelines exist to assist with the preparation of project estimates.
  • A formal and effective process is in place to challenge the assumptions and related resource allocations in a project estimate.
  • A project estimate is developed at the appropriate level of detail with the assistance of costing models/applications.
  • Reporting of actual results compared to the project estimates is done on a periodic basis to facilitate decision making with explanations for significant variances.
  • Staff are provided with the necessary training, resources, and information to support their cost estimation responsibilities and have adequate experience to complete the tasks.

Source

Audit Criteria Related to the Management Accountability Framework: A Tool for Internal Auditors. Treasury Board of Canada Secretariat, March 2011.

Annex E—Integrated Logistics Support Cost Model

Table 2. Comparison of Life Cycle Cost Breakout for Past Projects to the Costing Handbook Model. There are significant acquisition, ILS and O&M cost differences between aircraft, ships and armoured vehicles.

Table Summary:

The project or reference title is listed in the left-hand column. Read across the rows to determine the percentage cost applicable to each characteristic. The percentages in columns two, three and five add up to one hundred percent of the life cycle cost. The exception is in row “costing handbook recommendation” whereby ADM(Fin CS) chose to use a range of values to describe ILS Percentage of Life Cycle Cost, and Percent O&M of Life Cycle Cost. However, the row adds up to one hundred percent in two ways:

  1. by adding the Percent Acquisition of Life Cycle Cost (28%), the lower limit of ILS Percentage of Life Cycle Cost (6%), to the upper limit of Percent O&M of Life Cycle Cost (66%); and
  2. by adding the Percent Acquisition of Life Cycle Cost (28%), the upper limit of ILS Percentage of Life Cycle Cost (12%), to the lower limit of Percent O&M of Life Cycle Cost (60%).

 

Project/ReferencePercent Acquisition
of Life Cycle Cost
ILS Percentage
of Life Cycle Cost
ILS Percentage
of Acquisition
Percent O&M
of Life Cycle Cost
Costing Handbook Recommendation 28% 6-12% Not applicable 60-66%
Cormorant Helicopter 20.6% 4.1% 19.9% 75.3%
Canadian Patrol Frigate 40.9% 10.2% 24.9% 48.90%
Light Armoured Vehicle III 55.1% 4.4% 8.0% 40.5%

 

___________________________________________________________________________________________________________________________

Footnote 1 Report of the Auditor General of Canada to the House of Commons. Chapter 6, Fall 2010.

Footnote 2 An indicative estimate has a variance of +/- 25 percent according to the DND Costing Handbook.

Footnote 3 substantive estimate has a variance of +/- 15 percent according to the DND Costing Handbook.

Footnote 4 In December 2003, the Program Management Board directed a 10-year life cycle from the time of the Synopsis Sheet (Identification) SS(ID) to project close-out in order to reduce the project life cycle from 13 years. The timeframe for the options analysis and definition phases would be four years. In September 2004, Assistant Deputy Minister (Materiel) (ADM(Mat)) adopted an acquisition cycle performance target of nine years, including a two-year options analysis phase, a two-year definition phase and a five-year implementation phase. The September 2011 PAD also provides for a maximum of two years for the options analysis phase (paragraph 1.1.23) and an average of two years for the definition phase (paragraph 2.6.5).

Footnote 5 Major Crown Projects are high-value or high-risk projects that require Treasury Board approval.

Footnote 6 Project Management Body of Knowledge, 4th Edition, Chapter 7.

Footnote 7 The CRS Treasury Board/Minister of National Defence Submission Process Review, July 2009, recommended a critical path to reduce the submission cycle time by 30 percent.

http://www.crs-csex.forces.gc.ca/reports-rapports/pdf/2009/134P0894-eng.pdf

Footnote 8 14 March 2013 CRS audit staff meeting with Defence Renewal Team staff.

Footnote 9 KPMG Large Military Acquisitions: Life Cycle Cost Framework, January 2013.

Footnote 10 KPMG Next Generation Fighter Capability: Life Cycle Cost Framework, November 2012.

Footnote 11 Major capital projects are projects greater than $5.0 million in value.

Footnote 12 The DFC is chaired by the Deputy Minister and supported by the CFO and the VCDS. These members sign off all expenditure approval submissions.

Footnote 13 Major audit stakeholders agreed on 23 March 12 that life cycle sustainment costs could be treated as a separate audit. The CRS Risk Based Audit Plan for FY 2013/14 to 2015/16 currently includes an audit of Capital Project Life Cycle Support Estimates.

Footnote 14 Why has the cost of fixed-wing aircraft risen? RAND Corporation, 2008. The lowest annual aircraft escalation rate of 6.7 percent was for an electronic warfare aircraft of which 2.1 percent accounted for material and labour cost escalation and 4.6 percent for technology/capability improvements.

Footnote 15 CRS Audit of Construction in Support of Capital Equipment Projects, September 2005.

Footnote 16 At the time of the audit, equipment project offices in ADM(Mat) had a vacancy rate of 23 percent that would increase to 29 percent in three years – minutes of Program Management Board, 5 July 2012.

Footnote 17 May 2011 DND Annual Performance Review and December 2010 VCDS review of 47 high-value projects both concluded that projects are on average four years behind schedule prior to the project implementation phase.

Footnote 18 The VCDS target was set based on the CRS 2009 Treasury Board/MND Submission Process Review.

Footnote 19 A Major Crown Project requires a Memo to Cabinet due to high-dollar value/risk in the options analysis.

Footnote 20 ILS includes training, spare parts, publications, simulators, technical data packages, special tools, and test equipment.

Footnote 21 Not accounting for fewer deliverables, the total net cost growth for the 20 DND projects was only 6.7 percent.

Footnote 22 All estimates reviewed were converted into a 15-ship estimate for comparison purposes.

Footnote 23 A briefing note to the Deputy Minister                                                                                                                                                                                 may be delivered with a common hull design.

Footnote 24 2011 annual SRB presentation.

Footnote 25                                                                                                                                                                                      

Date modified: