Follow-up on Internal Audit: Minor Warship Auxiliary Vessel In-Service Support Contract (MWAV ISSC)

 

March 2014

7050-28-4 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Caveat

The result of this work does not constitute an audit of the MWAV In-Service Support Contract project. Rather, this report was prepared to provide reasonable assurance that the Management Action Plans (MAP) that resulted from the 2010 MWAV In-Service Support Contract project audit were implemented as stated and as such have addressed the associated recommendations.

 

Acronyms and Abbreviations

 

 

 

ADM(Mat)

Assistant Deputy Minister (Materiel)

ASC

Audit Services Canada

CHI

Contractor Held Inventory

CRS

Chief Review Services

DID

Data Item Description

D Mar P

Director of Maritime Procurement

DMCM

Director of Maritime Class Management

DMG Compt

Director Materiel Group Comptrollership

AERMS

Audit and Evaluation Recommendation Management System

PWGSC

Public Works and Government Services Canada

DND

Department of National Defence

FFP

Firm Fixed Price

ISSC

In-Service Support Contract

MAP

Management Action Plan

MWAV

Minor Warship Auxiliary Vessel

OPI

Office of Primary Interest

PM

Preventative Maintenance

SOA

Standing Offer Agreement

DGMEPM

Director General Maritime Equipment Program Management

WEAF

Work Estimate Action Form

1.0 Introduction

In keeping with the Treasury Board Policy on Internal Audit,1 Chief Review Services (CRS) is required to undertake audit follow-ups to assess the implementation status of MAPs developed in response to previous CRS audit recommendations. In accordance with the CRS Risk-based Audit Plan for 2013/14 to 2015/16, this follow-up assessment of the MWAV ISSC II audit was selected to ensure that controls had improved in a follow-on contract to optimize value for money and contract administration. The services provided by the vendor include project management, maintenance, systems engineering, logistics, documentation, and inspection support.

In January 2010, CRS completed an audit of the $239 million MWAV II contract. The objective of the audit was to provide assurance that the MWAV project governance, risk and control processes were in place to administer the in-service support for MWAV II and follow-on contracts. The results of the 2010 audit are outlined in the following findings.

Value for Money Oversight. Additional measures to increase the oversight of project management fees and maintenance tasks should be implemented.

Materiel Management. Improvements in the oversight of vendor-held Department of National Defence (DND) inventory are needed to ensure it is managed in accordance with the contract and reported in financial statements.

Vendor Reporting. Additional measures will help improve oversight of vendor performance and ensure that vendor reporting content is specified by the DND and complied with.

Task Management Efficiency. Task management efficiency could be improved with a firm fixed price (FFP) basis of payment for preventative maintenance (PM), risk-based task approval thresholds, and staff augmentation.

Risk Management. Risk identification and management could be improved for the follow-on contract.

Certification of Payment. Additional supporting documentation will help to improve support of actual costs, labour hours and the receipt of deliverables.

In order to address the above issues, six recommendations were put forward (See Annex C). To address these recommendations, the Assistant Deputy Minister (Materiel) (ADM(Mat)) developed a set of MAPs to address the audit findings specific to their area of responsibility.

2.0 Objective

The purpose of this follow-up is to determine the progress made on the implementation of the MAPs in response to the 2010 Audit of the MWAV ISS contract.

3.0 Methodology

This audit follow-up is the outcome of a review of documentation and evidence to assess the progress made in implementing the MAPs, based on the assessment criteria in Annex A. The following methods were used:

  • interviews and inquiries with DND contract management staff in the ADM(Mat), in the Director General Maritime Equipment Program Management (DGMEPM), specifically the Technical Authority and the Requisitioning Authority in the Director of Maritime Procurement (D Mar P), and other headquarters staff;
  • interview with Public Works and Government Services Canada (PWGSC) Contracting Authority;
  • review of contract documentation that pertained to the MAPs; and
  • analysis of data from the Audit and Evaluation Recommendation Management System (AERMS).2

Statement of Conformance

The audit follow-up conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit follow-up thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit follow-up, and apply only to the entity examined.

4.0 Overall Assessment

Good Practices

A DND-initiated Corrective Action Report (when subcontractor work was performed without DND approval) helps ensure value for money for the Crown. When the prime contractor receives this report, they will absorb the cost of the unapproved work.

The follow-up on the January 2010 audit identified that there has been progress in some areas. For example, there have been improvements in controls that relate to certification of payments and value for money oversight.

At the time of follow-up, there were six recommendations and seven MAP items designated as completed by the Office of Primary Interest (OPI) in AERMS. Upon further examination, two out of seven MAP items require additional measures to fully address the issues. These two outstanding MAPs are planned for implementation in the near future.  Until all MAPs are fully implemented, there remain concerns in the following areas:

  • The most recent inventory report, including value of holdings, has not been reported from DGMEPM to Director Materiel Group Comptrollership (DMG Compt) for financial statement purposes.
  • A risk management plan has not been implemented that would provide the methodology on how to manage the risks with the MWAV III contract.

A scorecard of the MAP items can be found in Annex B, and a more detailed assessment of progress can be found at Annex C.

Annex A—Assessment Criteria

Line of Enquiry: Progress made on the 2010 Audit Recommendations

Progress has been made on the implementation of MAPs identified in the 2010 MWAV ISSC audit.

The following criteria were used to assess the level of completion for each MAP item.

  1. Obsolete or Superseded

Audit recommendations that are deemed to be obsolete or have been superseded by another recommendation.

  1. No Progress or Insignificant Progress (0-24% complete)

No action taken by management or insignificant progress. Actions such as striking a new committee, having meetings and generating informal plans are insignificant progress.

  1. Planning Stage (25-49% complete)

Formal plans for organizational changes have been created and approved by the appropriate level of management (at a sufficiently senior level, usually at the Executive Committee level or equivalent) with appropriate resources and a reasonable timetable.

  1. Preparation for Implementation (50-74% complete)

The entity has begun necessary preparation for implementation, such as hiring or training staff, or developing or acquiring the necessary resources to implement the recommendation.

  1. Substantial Implementation (75-99% complete)

Structures and processes are in place and integrated in some parts of the organization, and some achieved results have been identified. The entity has a short-term plan and timetable for full implementation.

  1. Full Implementation (100% complete)

Structures and processes are operating as intended and are implemented fully in all intended areas of the organization.

Annex B—Management Action Plan Scorecard

Table B-1. Management Action Plan Scorecard

Table Summary:

Six recommendation areas are listed in the left-hand column. For each recommendation, read across the row to determine the MAP item, the OPI and the CRS assessment of progress on the MAP item.

 

Recommendation #MAPsOPICRS Assessment of Progress on MAPs
1. Value for Money Oversight 1.a) PWGSC will review the monthly management fees with the contractor to correct them. ADM(Mat) Full Implementation
1.b) Consideration will be given to re-competing subcontract standing offer agreements yearly. ADM(Mat) Full Implementation
2. Certification of Payments Risk-based sample reviews of subcontractor backup documentation for invoices to be conducted. ADM(Mat) Full Implementation
3. Materiel Management Formal reporting of contractor held inventory (CHI) will form part of the follow-on MWAV contract.A requirement for an improved contractor materiel management plan will include additional inventory data. ADM(Mat) Preparation for ImplementationNew Target Date: May 2014
4. Task Management Efficiency The follow-on MWAV contract will have provisions for an FFP mechanism for PM work. ADM(Mat) Substantial Implementation
5. Vendor Reporting The adoption of (new performance) metrics will be considered for the follow-on contract. ADM(Mat) Substantial Implementation
6. Risk Management A DND risk management plan will be developed. ADM(Mat) Preparation for ImplementationNew Target Date: May 2014

Annex C—Summary of Management Action Plan Status

Table C-1. Status of the Implementation of the MAP Items for Recommendation 1

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 1—“It is recommended that a monthly management fee audit be undertaken by Audit Services Canada (ASC), and that the vendor be required to provide a monthly list of management positions filled. For the follow-on contract, the Contracting Authority should be requested to resolve the issue of higher costs associated with the subcontract Standing Offer Agreement (SOA) rates overriding the rate in the prime contract.”
MAPs OPI Target Date Progress to Date Status of Action Item
1.a) PWGSC recently spent extensive time with the In-Service Support contractor reviewing the monthly management fees for the various batches of vessels added during this contract to the original 12 Maritime Coastal Defence Vessels. The negotiations that took place during these meetings resulted in a correction of these fees for batch II and IV ships. This issue is now resolved. ADM(Mat) December 2010 Monthly management fees were corrected for the four relevant batches of ships in the MWAV II contract, as a result of a PWGSC audit based on the contractor’s costs. In addition, a credit for such corrected fees was provided to the Crown in 2006. The MWAV III monthly management fees were calculated using the same formula used in the PWGSC audit, and, as a result, the fees regarding the 2011 MWAV III are approximately                                                               than those of MWAV II. Full Implementation
1.b) To mitigate the risk of subcontract cost increases, consideration will be given to re-competing the SOAs on a yearly basis, where the competition exists. ADM(Mat) December 2010 The MWAV III contract states that subcontract SOAs must be re-competed every three years. Re-competing SOAs on a yearly basis for subcontracted work was considered; however, it would not create a competitive environment as smaller subcontractors would be unable to meet the demands of responding to yearly SOAs. Full Implementation

Table C-2. Status of the Implementation of the MAP Items for Recommendation 2

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 2—“It is recommended that risk-based sampling of subcontractor supporting documentation be performed to ensure actual charges are in accordance with contract terms and conditions.”
MAPs OPI Target Date Progress to Date Status of Action Item
DGMEMP concurs that risk-based sample reviews should be conducted on a more regular basis. That said, since the conduct of the audit, at least six such reviews have been conducted to date, and no major observations or concerns have been identified. To further mitigate risks, future reviews will be based on cost (either total value or unexpected higher costs), uniqueness of the work, or areas requiring managerial attention based on work performed to date. Of note, in conjunction with PWGSC, DGMEMP will strengthen the clause for MWAV III requiring sub-contractors to provide supporting documentation for their invoices, which will be reviewed more frequently. ADM(Mat) December 2010 Evidence has been provided that a risk-based sampling process called Work Estimate Action Form (WEAF) audits are executed up to four times per year by D Mar P and PWGSC concurrently. The MWAV III contract has a strengthened clause that requires vendors to provide supporting documentation of subcontractor invoices with their monthly claims. Full Implementation

Table C-3. Status of the Implementation of the MAP Items for Recommendation 3

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 3—“It is recommended that CHI oversight be improved, and the amount of holdings be reported in the DND financial statements.”
MAPs OPI Target Date Progress to Date Status of Action Item
CHI reporting was not required when the current contract was signed. However, the contractor does conduct regular audits of all Crown inventory, and this report is available to the Technical Authority and Procurement Officer. Formal reporting through to DMG Compt will form part of the new MWAV III contract. Of note, the requirement for a contractor's Material Management Plan will strengthen the MWAV III contract to include the need for setting up maximum/minimum inventory levels,3 and the Crown to approve the changes to those levels as well as to the inventory adjustments. ADM(Mat) December 2010 CHI reports have been provided by the contractor to the Technical Authority, as required, in the contract's Data Item Descriptions (DIDs).4 Evidence of regular (four times per year) audits of Crown inventory has also been provided. In addition, the contractor developed a materiel management plan that indicates that DND approves changes to maximum/minimum levels. Maximum/minimum-level settings are important to ensure that optimal levels of inventory are stocked to help avoid shortages or excess inventory levels. These levels have also been provided within the inventory reports. The objective of CHI oversight improvement has been met through inventory reports; however, the value of holdings has not been reported to DMG Compt since fiscal year 2010/11. By May 2014, the DID will be amended to ensure the contractor inventory reports will be in the format required by DMG Compt. Preparation for Implementation

Table C-4. Status of the Implementation of the MAP Items for Recommendation 4

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 4—“It is recommended that consideration be given to a firm price PM baseline for the follow-on contract where historical costs are available and best value is assured in subcontract tenders.”
MAPs OPI Target Date Progress to Date Status of Action Item
The next MWAV contract will provide provisions for the Crown to establish an FFP for PM. As historical data becomes available in the first two years of the new contract, PM routines intend to be set up on an FFP basis for each class where it makes sense. ADM(Mat) December 2010

A decision was made against including provisions to establish an FFP mechanism for PM in the MWAV III contract. Cost and operational availability were factors in deciding against using an FFP mechanism for PM in the MWAV III contract, outweighing the administrative efficiencies to be gained.

Although the quantity of tasks remains the same, an increase in task processing efficiency has been reported since the SOA process has been implemented. As well, there have been improvements to the approval process of PM tasks, such as increasing the DND Technical Authority’s approval threshold from $20,000 to $40,000. All work below $40,000 can be approved internally without waiting for the WEAF to go through PWGSC for approval. This increase in approval threshold saves time and increases overall efficiency as it allows approximately 98% of the WEAFs to be processed solely by DND.

CRS recommends that this MAP be closed.

Substantial Implementation

Table C-5. Status of the Implementation of the MAP Items for Recommendation 5

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 5—“It is recommended that performance metrics, when practical for some fleets, be included in the follow-on contract, and measures be taken to ensure that vendor report content is in compliance with the contract.”
MAPs OPI Target Date Progress to Date Status of Action Item
The adoption of metrics currently collected, along with other measures, will be considered for the next contract. Also, DIDs will be included in the follow-on support contract to ensure vendor reporting is improved and that it remains consistent with the initial intent. ADM(Mat) December 2010

Several improvements to the Crown-owned Maintenance Information System, accessible by DND, are leading to an upgrade of the System; these include an improvement on vendor reporting of performance.

Contractual-level performance metrics were considered for the MWAV III contract, but have not been implemented, as both PWGSC and DND were concerned about setting a Key Performance Indicator and not being able to provide the vendor with ships when required. Naval operations require flexibility to respond to unforeseen operational requirements, which prevents ships from being readily available for scheduled maintenance. However, the improved data from the vendor could help DND measure their performance, and could lead to baseline incentives for the vendor to meet performance targets.

The MWAV III contract now includes DIDs to ensure consistency in vendor reporting. Full compliance with the DID was observed for the five reports reviewed.

CRS recommends that this MAP be closed.

Substantial Implementation

Table C-6. Status of the Implementation of the MAP Items for Recommendation 6

Table Summary:

The MAPs are listed in the left-hand column. For each MAP item, read across the row to determine the OPI, the target date, the progress to date, and the status.

 

Recommendation 6—“It is recommended that a DND Risk Management Plan be developed for the follow-on contract.”
MAPs OPI Target Date Progress to Date Status of Action Item
A Risk Management Plan will be developed to address risks leading up to the award of the next contract. Additionally, the new MWAV III Contractor will be required to produce and maintain a Risk Management Plan related to the work to be performed under the contract. The contractor will also be required to include a risk register for current work in the monthly report. ADM(Mat) May 2010

The joint DND and PWGSC risk management plan was not provided; however, there is evidence of ongoing risk management being performed through updates to the risk register in the monthly progress reports. Unlike a risk register, the Risk Management Plan is necessary for providing a management plan for the identified risks. DND plans to develop the Risk Management Plan by May 2014. Improvements to the MWAV III contract include a DID for a Risk Management Plan from the vendor and updates to the risk register in the Monthly Progress Report. The vendor has developed and provided a Risk Management Plan to the Technical Authority.

Preparation for Implementation

______________________________________________________________________________________________________________________________

Footnote 1 Policy on Internal Audit http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/ia-vi/ia-vi_e.asp.

Footnote 2 AERMS is a system that tracks the progress of the MAP items.

Footnote 3 The maximum level is the maximum amount of inventory to be held, while the minimum level is the point at which certain inventory should be re-ordered, based on forecasted usage, availability, and lead time.

Footnote 4 DIDs are completed documents defining the data deliverables required of a defence contractor. A DID specifically defines the data content, format, and intended use of the data, with a primary objective of achieving standardization objectives.

Date modified: