Evaluation of the Canadian Association of Physician Assistants Contribution Program (CAPA)

May 2016

1258-230 (ADM(RS))

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Acronyms and Abbreviations

ADM(RS)

Assistant Deputy Minister (Review Services)

CAF

Canadian Armed Forces

CAPA

Canadian Association of Physician Assistants

CCPA

Canadian Certified Physician Assistant

CFA

Contribution Funding Agreement

CFHSG

Canadian Forces Health Services Training Centre

CMA

Canadian Medical Association

CMP

Chief Military Personnel

CPD

Continuing Professional Development

DND

Department of National Defence

DND/CAF

Department of National Defence / Canadian Armed Forces

FY

Fiscal Year

GC

Government of Canada

MOU

Memorandum of Understanding

OAG

Office of the Auditor General

OCI

Office of Collateral Interest

OPI

Office of Primary Interest

PA

Physician Assistant

PI

Performance Indicator

TBS

Treasury Board Secretariat

YOS

Years of Service

PACCC

Physician Assistant Certification Council of Canada

Executive Summary

This report presents the findings and recommendations of the Evaluation of the Canadian Association of Physician Assistants (CAPA) Contribution Program. The evaluation was conducted by Assistant Deputy Minister (Review Services) (ADM(RS)) between January and July 2015, in compliance with the Treasury Board Secretariat (TBS) Policy on Evaluation (2009). As per the TBS policy, the evaluation examined the relevance and the performance of the CAPA Contribution Program over a five-year period (fiscal year (FY) 2009/10 to FY 2013/14).

Overall Assessment

  • The CAPA Contribution Program contributed to the development of a viable PA profession in the CAF.
  • The CAPA Contribution Program did not achieve financial viability or self-sufficiency.
  • The development of the PA profession nationwide, funded through the CAPA Contribution Program, is outside of the Department of National Defence (DND) / Canadian Armed Forces (CAF) mandate.
  • The Contribution Program for the CAPA should be discontinued.
  • If the CAPA does not become self-sufficient, Canadian Forces Health Services Group (CFHSG) must initiate measures to ensure the PA capability remains integral to the CAF Health Services Program.

Background

The CAPA is a not-for-profit association founded in 1999 by the Canadian Forces Medical Services School to create, promote and advocate Physician Assistant (PA) as a recognized health occupation.1 In 2003, the Canadian Medical Association (CMA) officially recognized the PA as a credentialed health science profession in Canada. The CAPA represents and certifies all PAs in Canada, both military and civilian.

The CAPA Contribution Program agreement was for $5 million for five years (FY 2009/10 to FY 2013/14), which was then extended for two additional years in 2014 with no additional funding. The current terms and conditions of the CAPA Contribution Program are effective until March 31, 2016.

Program Description

The CAPA Contribution Program was initiated in 2009 to provide financial support to the CAPA. The CAPA, under the auspices of the CMA, supports the PA profession and sets professional standards through independent national governance and representation. For the Canadian Armed Forces (CAF), the CAPA provides certification for military PAs through a subsidiary certification council that authorizes military PAs to be employed in any clinical environment. The CAPA also provides the necessary National Competency Profile standards for the accreditation of the PA curriculum used at the Canadian Forces Health Services Training Centre (CFHSTC) and at Canadian universities.

Relevance

PAs provide an important role within the DND/CAF in support of the delivery of operational and domestic health care responsibilities. The CAPA provides the necessary independent national governance and representation of the PA profession, and without the contribution agreement, alternative funding would have to be found. However, some activities of the CAPA, such as the promotion of the PA profession across Canada, do not necessarily align with DND/CAF strategic outcomes.

Performance

The principal objective of the contribution agreement was to support the CAPA as it established its key objectives, including financial self-sufficiency. The CAPA directly contributed to the accreditation of the PA education program at the CFHSTC and the certification of PAs within the CAF. The CAPA has also been successful in implementing a National Competency Program for the PA profession, obtaining incorporation, increasing membership, establishing fees for certification, and developing professional programs. However, the CAPA had only limited success in promoting the PA profession nationally among provincial health care bodies, and it was not able to achieve the Contribution Program objective of reaching financial self-sufficiency.

Table 1 presents the CAPA Contribution Program objectives and an assessment of the degree of achievement. Although many objectives were achieved, the most critical CAPA objectives necessary to becoming a financially viable and fully self-sufficient organization were not achieved.

Table 1. Achievement of CAPA Contribution Program Objectives. This table provides an assessment of the level of achievement related to the CAPA Contribution Program objectives.

Table Summary

This table provides an assessment of the level of achievement related to the CAPA Contribution Program objectives. It has three columns. The serial numbers are listed in the left-hand column. Read across the row to know the CAPA objectives (as per the initiating documents) and the degree of achievement.

 

SerialCAPA Objectives (as per initiating documents)Degree of Achievement
1 Make the CAPA a financially viable and fully self-sufficient organization. Not Achieved
2 Obtain CAPA incorporation. Achieved
3 Enter into revenue-generating and cost-efficient agreements with other provincial, national and international health care organizations. Not Achieved
4 Obtain sponsorship support for events, research and awards, and lastly, revenues from donations and individuals, research agencies, global companies, and foundations. Not Achieved
5 Increase the membership. Achieved (within capabilities)
6 Ensure the establishment of fees from certification and licensing requirements. Achieved
7 Develop medical education requirements and programs. Achieved
8 Develop annual certification examination for new graduates and re-certification programs for experienced professionals. Achieved (through the PACCC)2
9 Develop professional development programs. Achieved
10 Initiate a professional and educational fees system. Achieved

Conclusion

The role of the PA is a critical component of the CAF Health Services program. Significant efforts have been expended by the CAPA to establish the PA profession and to set the national standards and certifications. The program has had limited success in establishing the PA profession outside of the CAF and has not achieved the goal of financial self-sufficiency.

One of the objectives of the CAPA is “to contribute to the development of a nationwide health care profession.” This objective may help ensure the long-term viability of the PA profession; however, the funding of this objective is outside the mandate of the DND/CAF.

Continued support to the CAPA, through a DND-sponsored Contribution Program, is not recommended. If the Association does not become self-sufficient, CFHSG must take action to ensure the PA profession remains integral to the CAF Health Services Program.

Key Findings and Recommendation

Relevance

Key Finding 1: The Contribution Program permitted the CAPA to address a specific, demonstrable and continued need, which contributed to the health care of CAF members.

Key Finding 2: By contributing to the improvement of the CAF health care system and the Canadian health care system, the CAPA Contribution Program aligns with federal roles and responsibilities.

Key Finding 3: The CAPA Contribution Program was well aligned with government priorities.

Performance (Effectiveness)

Key Finding 4: Key stakeholders confirmed the emergent confidence of CAF Health Services customers in military PAs. However, the survey administered by the CFHSG did not provide a specific measurement for the PA profession.

Key Finding 5: With the sound management of scarce CAF Canadian Certified Physician Assistant (CCPA) resources, CFHSG has met all operational health care requirements without negatively impacting in-garrison health care needs. However, the CAF PA career path, as it is now, remains a concern and could affect operations in the future if not addressed.

Performance (Efficiency and Economy)

Key Finding 6: The CAPA made progress in developing some revenue-generating activities, but permanent sources of funding are presently not able to ensure the financial self-sufficiency of the organization.

Key Finding 7:                                                                                                                                              

Key Finding 8: The CAPA could have generated other revenues without jeopardizing funding from the CAPA Contribution Program.

Key Finding 9: Progress was made towards the full achievement of Contribution Program outcomes.

Key Finding 10: The cost generated by the PA certification process is greater than certification process revenues.

Key Finding 11: The efficiency of delivering expected CAPA Contribution Program outputs was inconsistent.

Key Finding 12: The Contribution Program met many of the objectives required in the initiating documents, but did not succeed in achieving financial viability and self-sufficiency.

Key Finding 13: Limitations in policy restrict the ability of DND to reimburse CAF members for professional memberships in the CAPA

ADM(RS) Recommendation

1.         The Contribution Program for the CAPA should be discontinued as of March 31, 2016. If the CAPA does not become financially self-sufficient and ceases to exist as an association, determine the potential implications and way ahead for the CAF PA profession on the following issues:

  1. the continued certification of CAF PAs, coordinated by the PACCC, as recommended by the 2007 Office of the Auditor General (OAG) Report on Military Health Care;
  2. the preservation and renewal of accreditation for the CAF PA education program at the CFHSTC based on the competency profile developed by the CAPA;
  3. the requirement to meet CAF PA clinical training needs through the use of civilian health care facilities; and
  4. the maintenance of operational capabilities the PA profession provides to the CAF.

OPI: CMP

Note: Please refer to Annex A—Management Action Plan for the management responses to this recommendation.

1.0 Introduction

This evaluation report presents the findings and recommendations applicable to the Evaluation of the CAPA Contribution Program. The $5-million CAPA Contribution Program was initiated in 2009 for a five-year period. In 2014, TBS approved a two-year extension with no additional funding. This extension period will end on March 31, 2016.

The evaluation was conducted between January and July 2015 and was supported by an Evaluation Advisory Group comprised of key stakeholders. The evaluation was designed to meet the coverage required by the Financial Administration Act and the TBS policy applicable to contribution programs. In accordance with the Policy on Transfer Payments,3 this evaluation assesses the extent to which established CAPA Contribution Program objectives were achieved during the observation period (FY 2009/10 to FY 2013/14). Being an initial program, this is the first evaluation conducted of the CAPA Contribution Program. As such, there are no previous baselines to assess continuous improvement.

1.1 Context for the Evaluation

The 1999 Chief Review Services4 review of the Canadian Forces’ health care system identified significant deficiencies in the management and provision of health care in domestic and international operations. In addition, the closure of military hospitals in the 1990s necessitated that military health care providers obtain training in civilian health care facilities.

The Director General Health Services sponsored the establishment of the CAPA in 1999 to directly support and certify PAs, permitting them to work or train in civilian hospitals. In response to the 2007 OAG Report on Military Health Care, the Department and the Surgeon General established the CAPA as the professional certifying body5 for the PA profession. The CAPA also provides standards for the accreditation of the PA curriculum used at the CFHSTC and at Canadian universities.

From 1999 to 2001, the CAPA received some DND funding support and was staffed by member volunteers. In FY 2001/02, the CAPA started to receive funding from DND through the Rx 2000 Project to establish the Association as a national body through organizational and certification exam development. From FY 2004/05 to FY 2008/09, the Association received approximately $1.3 million from DND funding to cover salaries, development of the exam model, public relations, and marketing with the provinces and medical schools. The five-year $5-million CAPA Contribution Program commenced in FY 2009/10 with the primary objective of transitioning the CAPA from a DND-administered association to an independent national organization.

1.2 Program Profile

1.2.1 Program Description

The CAPA is a not-for-profit association founded in 1999 by the Canadian Forces Medical Services School to create, legitimize, and advocate PA as a recognized health occupation. The CAPA Contribution Program, initiated in 2009, aimed to transition the CAPA to become an independent, lead national association for the promotion of the PA profession in Canada.

1.2.2 Contribution Program Objectives

At the inception of the CAPA Contribution Program, the program objectives6 were, within a five-year period, the following:

  • Make the CAPA a financially viable and fully self-sufficient organization;
  • Obtain CAPA incorporation;
  • Enter into revenue-generating and cost-efficient agreements with other provincial, national and international health care organizations;
  • Obtain sponsorship support for events, research and awards, and lastly, revenues from donations and individuals, research agencies, global companies, and foundations;
  • Increase the membership;
  • Ensure the establishment of fees from certification and licensing requirements; and
  • Develop medical education requirements and programs, professional development programs, annual certification examinations for new graduates and re-certification programs for experienced professionals, and initiate a professional and educational fees system.

1.2.3 Stakeholders

Table 2 identifies the stakeholder organizations and groups having an interest in, or being affected by the CAPA Contribution Program. This list served the evaluation to determine: candidates for interviews; potential sources for data; relevant authorities for advice; and relevant membership for forming the Evaluation Advisory Group.

Table 2. List of CAPA Contribution Program Key Stakeholders. This list shows the key stakeholders of the CAPA Contribution Program.

Table Summary

This table shows the key stakeholders of the CAPA Contribution Program. It has two columns. The left-hand column lists the stakeholders internal to the DND/CAF. The right-hand column lists the stakeholders external to the DND/CAF.

 

Internal to the DND/CAFExternal to the DND/CAF
  • Canadian Army
  • Royal Canadian Air Force
  • Royal Canadian Navy
  • Assistant Deputy Minister (Finance) / Chief Financial Officer
  • Chief Military Personnel (CMP)
  • CAF Surgeon General
  • CFHSG
  • All CAF medical units
  • Military PAs
  • TBS
  • Health Canada
  • Canadian Intergovernmental Conference Secretariat
  • All provincial health authorities
  • All territorial health authorities
  • CMA
  • CAPA Board of Directors
  • CAPA staff
  • Civilian hospitals
  • Civilian PAs
  • Civilian physicians
  • Clinical assistants
  • Nurse practitioners
  • American Academy of Physician Assistants

1.3 Evaluation Scope

The scope of this evaluation is limited to assessing the relevance and performance of the CAPA Contribution Program and to addressing the extent to which the Contribution Program’s objectives (listed in Section 1.2.2) have been achieved during the period of observation (FY 2009/10 to FY 2013/14). While the evaluation also provides observations and suggestions regarding the PA profession in the CAF, the ADM(RS) recommendation is specific to the CAPA Contribution Program.

1.3.1 Coverage and Responsibilities

The CAPA Contribution Program is situated within Activity 4.0 (Defence Capability Element Production) of the Program Alignment Architecture, Sub-activity 4.1 (Military Personnel and Organization Lifecycle), and Sub-sub-activity 4.1.8 (Military Personnel – Healthcare). The sponsor of the CAPA Contribution Program is the CAF Surgeon General. The delegated Program Manager is the CFHSG Chief of Staff who, through the Surgeon General, is accountable to the CMP and the Chief of the Defence Staff.

1.3.2 Resources

During the evaluation period, the CAPA employed between 3.5 and 5 full-time equivalents, and was governed by a Board of Directors comprised of 14 volunteers.

Table 3 displays the Vote 10 funding provided to the CAPA through the Contribution Program from FY 2009/10 to FY 2013/14.

Table 3. DND/CAF Expenditures.

Table Summary

This table shows the amount released from the CAPA Contribution Program to the CAPA from FY 2009/10 to FY 2013/14. It has five functional columns that list the fiscal years. Read across the row to know the funding provided by the contribution program for each fiscal year.

 

 FY2009/10FY2010/11FY2011/12FY2012/13FY2013/14
Contribution Funding $67,703 $783,949 $889,119 $912,247 $835,204

Table 3. DND/CAF Expenditures. This table shows the amounts released from the CAPA Contribution Program to the CAPA from FY 2009/10 to FY 2013/14.

The two-year extension to the Contribution Program was approved in 2014 to make available the remaining $1,511,778 of the initial Contribution Program funding that had not been spent.

1.3.3 Evaluation Issues and Questions

In accordance with the TBS Directive on the Evaluation Function (2009),7 the evaluation addressed the five core issues related to relevance and performance. The methodology used to gather evidence in support of the evaluation questions can be found at Annex B. A logic model providing a theory of change for this contribution program is included at Annex C. An evaluation matrix listing each of the evaluation questions, with associated indicators and data sources, is provided at Annex D.

2.0 Findings

Sections 2.1 to 2.3 discuss the relevance of the CAPA Contribution Program; sections 2.4 and 2.5 will cover the performance aspects. Other issues not related to the relevance and performance aspects appear at section 2.6.

Relevance

In assessing relevance, the evaluation examined the extent to which the Contribution Program: addressed a demonstrable and continued need; was aligned with federal roles and responsibilities; and was aligned with Government of Canada (GC) priorities and DND/CAF strategic outcomes.

2.1 Relevance – Continued Need

To determine whether the CAPA Contribution Program addressed and continues to address a demonstrable need, the following key indicators were used:

  • evidence of a current and future need for the CAPA Contribution Program;
  • evidence that the CAPA Contribution Program is reactive to the needs of the DND/CAF and Canadians; and
  • extent to which the CAPA Contribution Program complements the services provided by other departments, agencies, and/or organizations (internal/external).

The observations and findings are based on evidence originating from literature reviews, document reviews, and on-line resources, in addition to information extracted from interviews with key informant stakeholders.

Key Finding 1 : The Contribution Program permitted the CAPA to address a specific, demonstrable and continued need which contributed to the health care of CAF members.

The CAPA directly supports the CAF health care system by providing the necessary certification for military PAs. The CAPA also provides the necessary National Competency Profile standards for the accreditation of the PA curriculum used at the CFHSTC, which authorizes the training centre to deliver the PA education program.

Should the CAPA cease to exist, the supervision and monitoring of a unique and single competency profile necessary for accreditation of PA education programs in Canada would be difficult to achieve through alternative means, as would the required certification process for PAs in Canada, both civilian and military. Individual memoranda of understanding (MOU) might be more difficult to establish to allow CAF PAs access to civilian health care facilities in order to acquire, practise and maintain their clinical competencies, which could lead to gaps in the provision of health care to CAF personnel and impede CAF operations.

2.2 Relevance – Alignment with Federal Roles and Responsibilities

This section examines the extent to which the CAPA Contribution Program aligns with federal roles and responsibilities.

The following indicator was used in the assessment of the alignment of the CAPA Contribution Program with federal roles and responsibilities:

  • evidence of alignment between federal roles and responsibilities (including legislative and policy obligations) and the delivery of the CAPA Contribution Program.

Observations and findings are based on evidence originating from official document reviews, in addition to on-line resources and to a validation of evidence through interviews with key informant stakeholders.

Key Finding 2: By contributing to the improvement of the CAF health care system and the Canadian health care system, the CAPA Contribution Program aligns with federal roles and responsibilities.

In Canada, responsibilities for “health” are shared between the federal government and the provinces and territories. Health Canada is accountable for the public health of Canadians in terms of concepts, policies and regulations, while the provinces and territories are responsible for the direct provision of health care to their citizens.8 As indicated on the Health Canada website: “The federal government, the ten provinces, and the three territories have key roles to play in the health care system in Canada. Health Canada’s role is to help Canadians maintain and improve their health.”9

2.3 Relevance – Alignment with Government Priorities and DND/CAF Strategic Outcomes

Section 2.3 examines the extent to which the CAPA Contribution Program’s objectives are consistent with federal government priorities and DND/CAF strategic outcomes.

The following indicators were used to make this determination:

  • evidence of alignment between the CAPA Contribution Program’s objectives and current federal government priorities; and
  • evidence of alignment between the CAPA Contribution Program’s objectives and DND/CAF strategic outcomes.

The observations and findings made in this section are based on evidence originating from official document reviews, in addition to a validation of evidence conducted through interviews with key informant stakeholders.

Key Finding 3: The CAPA Contribution Program was well aligned with government priorities.

The GC whole-of-government framework includes “Healthy Canadians” as one of the 16 outcome areas where the GC is aiming to create a responsible and accessible health system for Canadians. Support for this GC outcome area is reflected in the departmental Report of Plans and Priorities for Health Canada, DND, and Aboriginal Affairs and Northern Development Canada.10 It reinforces the concept that the health of Canadians represents a major priority for the federal government and many departments.

In addition, Defence Priorities and Elements 2014-18,11 Priority 4 – Strengthening the Defence Team, includes the commitment to “Provide enhanced support to the ill and the injured and to the families of CAF members.” The Canada First Defence Strategy12 underlines that “People are Defence’s most important resource. Both the Department and the Forces rely heavily on the work and expertise of dedicated personnel to ensure the operational effectiveness of the military.”

As such, a military health care program is provided for all members of the CAF. The certification of the military PA through the PACCC, a subsidiary of the CAPA, and the accreditation of the PA education program based on the competency profile established by the CAPA are important elements of this program. As an organization, the CAPA offers value to the PA profession and meets the needs of the CAF for military PAs, indirectly supports DND/CAF health care priorities, and indirectly contributes to DND/CAF strategic outcomes. However, one of the CAPA objectives is to become a nation-wide health care professional association through, inter alia, the accreditation of civilian university education programs, the certification of civilian PAs, and increased membership. However, this CAPA objective is outside of the DND/CAF mandate. The subsidization of a national organization supporting a new health care profession should be supported by, or shared with, other national health-related sponsors, such as Health Canada or Aboriginal Affairs and Northern Development Canada and facilitated through the Canadian Intergovernmental Conference Secretariat.

Performance

2.4 Performance – Achievement of Expected Outcomes (Effectiveness)

According to the TBS Policy on Evaluation, performance is defined as “the extent to which effectiveness, efficiency and economy are achieved by a program.”13 This section will review the effectiveness of the CAPA Contribution Program, while section 2.5 will cover the efficiency and economy of the Program.

The effectiveness of the CAPA Contribution Program was assessed by measuring the success of PA professional certification and PA program accreditation, and the extent to which the CAPA is financially viable and self-sufficient. These are identified on the logic model by immediate, intermediate, and ultimate program outcomes. The evaluation used two distinct approaches: the first one focused on the military PA profession in the CAF (2.4.1); the second reviewed the general PA profession in Canada (2.4.2).

The observations and findings for the Program’s effectiveness are based on evidence collected through: document and financial data reviews; questionnaires administered to PAs by the CAPA; and key informant interviews.

2.4.1 Outcomes Impacting the Military PA Profession in the CAF

The outcomes considered in this evaluation impacting the military PA profession in the CAF are as follows:

  • Immediate outcomes: CAF PAs have the necessary professional skills, and CAF PAs have increased access to civilian hospitals to complete residency requirements;
  • Intermediate outcomes: CAF members have confidence in military PA practitioners; and
  • Ultimate outcomes: Ensure sufficient capacities of certified CAF PAs are available to meet operational and in-garrison health care needs.

2.4.1.1 Immediate outcome: CAF PAs have the necessary professional skills

The following indicators were used to assess the achievement of this outcome:

  • evidence of improvement in the number of CAF PAs certified during the period FY 2009/10 to FY 2013/14; and
  • evidence that certified CAF PAs can meet the required competencies.

The majority of CAF PAs have the necessary professional skills to assume their clinical duties. PA certification is awarded by the PACCC,14 a subsidiary organization of the CAPA, which administers and maintains the PA certification process. Certifications ensure that PAs meet the standards set out in the National Competency Profile for the PA profession. To obtain certification, military PAs must successfully write a formal examination that is administered once a year. PAs who are granted certification can hold the title of “Canadian Certified Physician Assistant.”

Table 4 indicates the number of CAF PAs and CCPAs from FY 2009/10 to FY 2013/14.

Table 4. Certified military PAs. This table indicates the number of Certified CAF PAs as a percentage of total CAF PAs for the period FY 2009/10 to FY 2013/14.

Table Summary

This table indicates the number of certified CAF PAs as a percentage of total CAF PAs for the period FY 2009/10 to FY 2013/14. It has seven columns. The left-hand column states the type of PA. Read across the row to know the number of PAs by type in each fiscal year. The far right-hand column lists the variance in percentages. The bottom row lists the proportion of certified PAs in percentages.

 

 PAFY2009/10FY2010/11FY2011/12FY2012/13FY2013/14Variation
Military CCPAs 48 83 122 143 168 250%
Total Military PAs 150 142 201 219 231 54%
Proportion 32% 58% 61% 65% 73%  

According to Table 4, 73 percent of existing military PAs achieved certification as of FY 2013/14, which is an increase of 250 percent compared to FY 2009/10. As the certification level is an indicator of professional skills, the evaluation assessed that the majority of military PAs have the necessary skills to do clinical work.

2.4.1.2 Immediate outcome: CAF PAs have increased access to civilian hospitals to complete residency requirements

The following indicators were used to assess the achievement of this outcome:

  • evidence of increased access demonstrated by the number of CAF PAs having fulfilled their residency requirements for the period FY 2009/10 to FY 2013/14; and
  • number of CAF PAs who did not complete residency requirements during the period FY 2009/10 to FY 2013/14 due to lack of MOUs/contracts with hospitals.

All CAF PA students must complete clinical rotation requirements in civilian health care centres, which is a necessary requirement to fulfill the PA education program. The CFHSG works closely with the CFHSTC to identify medical clinics and hospitals where formal arrangements are made. These agreements allow the military PA students to complete the one-year clinical exposure required by the education program. It must be noted that the CAPA has no role in this activity. Finding and organizing this employment requirement remains the responsibility of each PA education centre. The certification of military PAs through the PACCC, however, facilitates military PA employment in the civilian health care centres.

Since the creation of the military PA education program, all CAF PA students have completed their clinical rotation requirements.

2.4.1.3 Intermediate outcome: CAF members have confidence in military PA practitioners

The following indicator was used to assess the achievement of this outcome:

  • The level of CAF members’ satisfaction with military PA practitioners (for the period FY 2009/10 to FY 2013/14).

Key Finding 4: Key stakeholders confirmed the emergent confidence of CAF Health Services customers in military PAs. However, the survey administered by the CFHSG did not provide a specific measurement for the PA profession.

The initiating documentation for the CAPA Contribution Program recommended that CFHSG administer an annual survey to CAF members to specifically measure client satisfaction with health services support provided by military PAs. Unfortunately, the survey was administered using a global and consolidated approach instead of isolating the health services support received from specific professionals within the CFHSG.

Interviews with key stakeholders indicated a high level of CAF members’ confidence in and satisfaction with the health care received from the CAF PA practitioners. Nevertheless, the CAF Health Services survey should be adapted to measure satisfaction of services received from specific clinical military health professions, such as the PA profession.

2.4.1.4 Ultimate outcome: Ensure a sufficient capacity of certified PA is available to meet operational and in-garrison health care needs

To assess the achievement of this outcome, relevant indicators focus on appointments, attraction, certification, and career development concerns of military PAs. The following indicators were used:

  • evolution of the number of PA Military Occupational Structure Identification requirements and the number of PA positions filled (for the period FY 2009/10 to FY 2013/14);
  • evidence of PA shortage in CAF deployed operations such as Afghanistan, Haiti, Alert and Vancouver (for the period FY 2009/10 to FY 2013/14);
  • evolution of the military PA retention rate (for the period FY 2009/10 to FY 2013/14); and
  • evidence of CAF ability to attract new military PAs (for the period FY 2009/10 to FY 2013/14).

Key Finding 5: With the sound management of scarce CAF CCPA resources, CFHSG has met all operational health care requirements without negatively impacting in-garrison health care needs. However, the CAF PA career path, as it is now, remains a concern and could affect operations in the future if not addressed.

Appointments of Military PAs. Interviews conducted with CFHSG staff involved in manning, tasking, and appointments of military CCPAs indicated that it has always been possible to fill operational CCPA positions. Even if some PA positions had to be kept vacant (e.g. 38 out of 198 positions (in 2013) and 53 out of the existing 198 position (in 2014)), the established priority system was not detrimental to any clinical military requirements.

Attraction of PAs. At present, CFHSG does not have many attractive elements to offer for the recruiting of new PAs. The actual career process is to join the CAF as a medical technician, spend up to eight years in this occupational specialty, then indicate interest in becoming a PA, be designated by a selection board, undergo the two-year education program in Borden, and be subject to a four-year obligatory service. However, CFHSG is conducting a study, with results expected to be delivered in 2016, to assess the proposal to commission military CCPAs. This could facilitate the attraction of new military PAs through the Direct Entry Officer Program.

Certification of Military PAs. In 2014, CFHSG determined that certification through the PACCC was compulsory for all military PAs in order to be selected to fill clinical appointments and to be promoted to the rank of Warrant Officer. This approach has contributed to the increase in the number of CCPAs shown in Table 4 (sub-section 2.4.1.1).

Development of CCPAs. As of 2014, no CAF PAs had less than 10 years of service (YOS). This situation is largely due to the present career progression process explained earlier. More than 63 percent of PAs have completed 20 YOS and are eligible for retirement programs, which places the viability of the CAF PA trade in jeopardy. The number of YOS has two main clusters: 11 to 20 YOS and 21 to 30 YOS with a peak of PAs in the 24 YOS area. The majority have between 21 and 30 YOS. The breakdown of the 164 serving military CCPAs as of October 31, 2014, showing their number of years of military service (as YOS) is presented in Table 5.

Table 5. YOS – Military CCPAs. This table presents the distribution of military CCPAs according to their years of military service, as of October 2014.

Table Summary

This table presents the distribution of military CCPAs according to their years of military service, as of October 2014. It has three columns. The left-hand column lists the YOS. Read across the row to know the number of military CCPAs for each level of YOS and the percentage thereof.

 

YOSNumber of Military CCPASPercent
Less than 10 0 0.0%
10 to 20 60 36.6%
21 to 30 83 50.6%
More than 30 21 12.8%
Total 164 100.0%

Attrition of Military PAs. The attrition rate of PAs can reach as high as 25 percent in some years. This seems to be a result of the attraction of available civilian PA jobs as the PA profession is expanding in the civilian health care system.  

In conclusion, CFHSG must address the attraction, appointment, certification, retention and development of military PAs to ensure CAF clinical requirements continue to be met.

2.4.2 Outcomes Impacting on the General PA Profession in Canada

The outcomes considered in this evaluation, which impact the general PA profession, are as follows:

  • Immediate outcomes: Academic programs and CFHSTC accreditations are renewed; number of university-based accredited programs for PAs has increased; and the CAPA is incorporated; and
  • Intermediate outcomes: Single national PA certification standards exist.

The evidence collected to evaluate the effectiveness of these outcomes is mainly supported by: interviews of key informants; document reviews; and the use of quantitative data compiled by CFHSG and the CAPA.

2.4.2.1 Immediate outcome: Academic programs and CFHSTC accreditations are renewed

The following indicators were used to assess the achievement of this outcome:

  • for the period FY 2009/10 to FY 2013/14, evidence that the accreditation of the CFHSTC program has been renewed; and
  • for the period FY 2009/10 to FY 2013/14, evidence that the accreditation of university-based programs has been renewed.

During the period FY 2009/10 to FY 2013/14, all existing PA education programs accreditations were granted or renewed. The accreditation is primarily conducted directly between PA education institutions and the CMA, as the CMA is the sole approving authority in Canada for medical-related education programs.15 The role of the CAPA in this process is to monitor, review, and maintain current the National Competency Profile defined for the PA profession. CMA approves the National Competency Profile and uses it as the standard for the accreditation of university-based programs.

For the period covered, the CFHSTC military PA education program was renewed in 2010 for a period of six years. CFHSG is in the process of supporting the CMA accreditation renewal for CFHSTC in 2016. The three other university-based PA education programs were granted accreditation in 2010 and 2011, all of them for a period of six years. They will be subject to the renewal of their accreditation by the CMA in 2016 and 2017. Table 6 illustrates the status of accreditation for PA education programs in Canada.

Table 6. Status of Accreditation in University-based PA Education Programs. This table presents the status of accreditation of the academic institutions offering the PA education program in Canada.

Table Summary

This table presents the distribution of military CCPAs according to their years of military service, as of October 2014. It has three columns. The left-hand column lists the YOS. Read across the row to know the number of military CCPAs for each level of YOS and the percentage thereof.

 

ProgramDiplomaYear of MostRecent AccreditationYearAccreditation tobe Renewed
CFHSTC Bachelor 2010 2016
McMaster University Bachelor 2010 2016
The Consortium of PA Education (Toronto) Bachelor 2011 2017
University of Manitoba Master 2010 2016

2.4.2.2 Immediate outcome: Number of university-based accredited programs for PAs has increased

The following indicators were used to assess the achievement of this outcome:

  • changes in the number of university-based accredited programs for the period
  • FY 2009/10 to FY 2013/14 due to the CAPA expansion program; and
  • evidence of effective marketing activities performed to gain professional endorsement on the accreditation of programs and development of the PA profession.

During the period FY 2009/10 to FY 2013/14, three new university-based PA education programs were implemented in two provinces. When the CAPA Contribution Program was initiated in 2009, only the CAF PA education program, created in 2002, was in place. Since then, the University of Manitoba, McMaster University, and the Consortium of Physician Assistant Education (University of Toronto, Northern Ontario School of Medicine, and The Michener Institute for Applied Health Sciences) have initiated PA education programs.

Based on interviews with key stakeholders, the implementation of relevant PA education programs by universities is part of the implementation of the PA profession in Canada. Once provinces or territories have put the legislation in place to recognize the profession, a university-based program can be established. Some universities interested in establishing the PA program are waiting for their provincial and territorial authorities to adopt the legislation. As a result, university-based PA programs cannot be implemented until provinces and territories adopt the legislation necessary to implement the PA model. Therefore, the CAPA continues to have its main efforts centered on advocacy and lobbying of provinces and territories to facilitate the adoption of this legislation. The CAPA, in its strategic plan for future activities, is preparing parallel marketing activities with universities in order to rapidly support the PA profession with academic programs.

2.4.2.3 Immediate outcome: The CAPA is incorporated

The following indicator was used to assess the achievement of this outcome:

  • evidence of actions leading to potential incorporation.

The CAPA was incorporated on November 27, 2009. As soon as the Contribution Program commenced in 2009, one of the first activities completed by the CAPA was to achieve incorporation.

2.4.2.4 Intermediate outcome: Single national PA certification standard exists

The following indicator was used to assess the achievement of this outcome:

  • evidence of the existence of a single national PA certification standard.

Since 2011, there exists only one national PA certification standard represented by the Scope of Practice and the National Competency Profile for PAs. Certification is granted by the PACCC, which develops certification exams based on the Scope of Practice and the National Competency Profile.16 Created in 2009 by the CAPA and approved in 2011 by the CMA, this standard constitutes the foundation of the clinical aspects of the PA profession. It also serves as the standard to determine accreditation of actual and incoming PA education programs in Canada.

2.5 Performance – Demonstration of Efficiency and Economy

This section will examine the performance of the CAPA Contribution Program related to efficiency 17 and economy.18 More particularly, it will address the extent to which the CAPA Contribution Program has used the resources in relation to the production of outputs and to the progress made towards the given outcomes, and the achievement of the objectives of the Contribution Program. In addition, outcomes identified in a program are usually measured in terms of effectiveness. In the present evaluation, since many outcomes were very closely related to the measurement of efficiency and economy, they are discussed in the present section.

The approach utilized to demonstrate efficiency and economy will cover the following outcomes and evaluation questions:

  • Immediate outcomes: Other revenue-generating activities are undertaken, and membership revenues have increased;
  • Intermediate outcome: The CAPA is financially viable and self-sufficient;
  • Unintended outcome: Limitation of self-sufficiency options due to stacking of grants and contributions; and
  • The specific efficiency and economy questions addressed are as follows:
    • What is the progress made by the Contribution Program towards the expected outcomes and the resources expended?
    • What is the trend of the output/input ratio towards achieving the aim of the Contribution Program?
    • How were the objectives established at the creation of the program met?

The evidence collected to assess the achievement of outcomes is supported by: document reviews; confirmation by interviews with key informant stakeholders; and the use of quantitative data compiled by CFHSG and by the CAPA.

2.5.1 Immediate outcome: Other revenue-generating activities are undertaken

The following indicators were used to assess the achievement of this outcome:

  • variations in other revenues (for the period FY 2009/10 to FY 2013/14);
  • evidence of specific activities aimed at generating permanent sources of funds (for the period FY 2009/10 to FY 2013/14);
  • evidence of growth in the number of sponsorships (for the period FY 2009/10 to FY 2013/14); and
  • evidence of growth in the revenues generated by sponsorship (for the period FY 2009/10 to FY 2013/14).

Key Finding 6: The CAPA made progress in developing some revenue-generating activities, but permanent sources of funding are presently not able to ensure the financial self-sufficiency of the organization.

The CAPA’s not-for-profit status allows the organization to generate revenue from various sources and activities. The only limitation to funding activities is where the CAPA cannot request or accept any funds for activities already covered by the Contribution Program and stated in the CFA. For all other activities not funded by the CFA, the CAPA can accept funds and declare the funds to the Minister of National Defence, who, in turn, will decide any impacts concerning funding received from the Contribution Program.

Typical revenue-generating activities include collecting membership dues, PA conference fees, examination fees for certification, advertising, sponsorship, and revenues from an exhibit during the conference. However, revenue can also be generated through commissions on sales from the CAPA boutique, advertising on the CAPA website, partnerships, and donations from supporters.

Table 7 shows the revenue reported for the period FY 2009/10 to 2013/14. As FY 2009/10 was the first year of operations for the CAPA, there were no other significant sources of revenue.

Table 7. Sources of Revenue. This table presents the revenues generated by CAPA activities for the period FY 2009/10 to FY 2013/14.

Table Summary

This table presents the revenues generated by CAPA activities for the period FY 2009/10 to FY 2013/14. It has six columns. The left-hand column lists the sources of revenue. Read across the row to know the amount per fiscal year. The second to last row shows the total amount, and the last row shows the variation vice the previous fiscal year in percentages.

 

Sources of RevenueFY 2009/10FY 2010/11FY 2011/12FY 2012/13FY 2013/14
Membership Dues

      

           

     

           

     

           

       

           

       

Conference Registration Fees

      

            

     

           

     

           

     

           

     

Certification Registration Fees

     

           

     

           

     

           

     

           

     

Advertising and Sponsorship

           

    

           

    

           

    

           

    

           

    

Exhibit

     

           

    

           

    

           

 

           

    

Total

           

    

           

       

           

       

           

    

           

       

Variation vice Previous Year   N/A                              

According to Table 7,                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

2.5.2 Immediate outcome: Membership revenues have increased

The following indicators were used to assess the achievement of this outcome:

  • evidence of growth in the number of CAPA members (for the period FY 2009/10 to FY 2013/14); and
  • evidence of growth in revenues generated by the different categories of membership (for the period FY 2009/10 to FY 2013/14).

The membership of the CAPA has increased. As shown in Table 8, the military and civilian PA membership of the CAPA progressed by 134 percent, increasing from 217 members in FY 2009/10 to 508 members in FY 2013/14. The growth of the number of members varied each year, maintaining an average progression of 26 percent per year during the last five years.

Table 8. CAPA Membership. This table presents the status of CAPA membership from FY 2009/10 to FY 2013/14.

Table Summary

This table presents the status of CAPA membership from FY 2009/10 to FY 2013/14. It has six functional columns that list the fiscal years. Read across the row to know the membership numbers and the variation in percentages per fiscal year. The far right-hand column, labelled “over years,” shows the total percent increase in number of members over the five fiscal years.

 

 FY 2009/10FY 2010/11FY 2011/12FY 2012/13FY 2013/14Over Years
Membership 217 312 395 488 508 134%
Variation N/A 44% 27% 24% 4%  

In terms of revenues, income generated by membership dues has also progressed positively, showing improvements for each year between FY 2010/11 to FY 2013/14,                                                                                                                                      Table 9 presents the situation in detail. FY 2009/10 does not appear in the table as there were no membership dues collected in that fiscal year.

Table 9. CAPA Membership Dues. This table presents the income generated by the membership dues between FY 2010/11 and FY 2013/14.

Table Summary

This table presents the income generated by the membership dues between FY 2010/11 and FY 2013/14. It has five functional columns that list the fiscal years. Read across the row to know the membership dues in dollars and the variation in percentages per fiscal year. The far right-hand column, labelled “over years,” shows the total percent increase in membership dues over the four fiscal years.

 

 FY 2010/11FY 2011/12FY 2012/13FY 2013/14Over Years
Membership

             

   

             

   

               

   

               

   

         
Variation N/A                          

2.5.3 Intermediate outcome: The CAPA is financially viable and self-sufficient

The following indicators were used to assess the achievement of this outcome:

  • the degree of financial autonomy and self-sufficiency of the CAPA at the end of FY 2013/14;
  • at the end of the extension period (FY 2015/16), the probability that the CAPA will become financially self-sufficient;
  • improvements shown in the sources of funds external to DND since the inception of the Contribution Program;
  • improvements shown in the number of sponsors; and
  • improvements/increases shown in the amount of funding from other sources.

Key Finding 7:                                                                                                                                                                                                                                                                                                                                                                          

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         19

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

2.5.4 Unintended outcomes: Limitation of self-sufficiency options due to stacking of grants and contributions

The following indicator was used to assess this outcome:

  • evidence of unintended elements having impacted on the results of the CAPA Contribution Program.

Key Finding 8: The CAPA could have generated other revenues without jeopardizing funding from the CAPA Contribution Program.

There exists a “stacking of grant and contribution” clause in the CFA. The CAPA staff understood that the disbursement of funds from the Contribution Program would have ceased if the CAPA had acquired grants or contributions from other sponsors or monetary contributions from partnerships. Based on an assessment made by the Officer in Charge of the Grants and Contributions at Assistant Deputy Minister (Finance / Chief Financial Officer), the clause in the CFA covering the “stacking of assistance” might have been misinterpreted. It should have been read in conjunction with one of the aims of the Contribution Program, which was for the CAPA to become self-sufficient and to ensure survivability within the period of time allocated by the Contribution Program.

Therefore, any additional funding received from activities not covered by the CFA or from external entities would have initiated a process of disclosure to the Minister of National Defence. This process could have resulted in a decision in funding adjustment to reduce the actual funds received by the CAPA from the Contribution Program. The official documentation creating this Contribution Program aimed to make the CAPA self-sufficient within the allocated contribution period. If additional funding was to be provided, the stacking clause should not be seen as a barrier to obtain other revenues and should be interpreted more widely.

2.5.5 Progress towards Expected Outcomes and Resources Expended

To assess the progress towards outcomes and the resources expended, the following indicators were used:

  • funds originating from other sources compared to the total expended for the period FY 2009/10 to FY 2013/14 to determine self-sufficiency and viability; and
  • for the period FY 2009/10 to FY 2013/14, evidence of positive impacts and of value for money in the achievement of the immediate outcomes.

Key Finding 9: Progress was made towards the full achievement of Contribution Program outcomes.

The CAPA staff and the Board of Directors members interviewed indicated that during the first five years, the Contribution Program focused more on developing the Association and increasing the opportunities for advocacy, lobbying activities, and contacts with a view to having the provinces and territories adopt the PA model. Contribution Program funding covered all approved activities, so the necessity of finding external sources of funding was not assessed as a priority.

Table 10. Sources of Funds. This table outlines the sources of funds generated by the CAPA during the period FY 2009/10 to FY 2013/14.

Table Summary

This table outlines the sources of funds generated by the CAPA during the period FY 2009/10 to 2013/14. It has seven columns. The left-hand column lists the CAPA sources of funds. For each source of funds, read across the row to know the amount generated per fiscal year, as well as the percent contributed by each source in the last fiscal year in the far right-hand column. The bottom row presents the sum of revenues from all sources per fiscal year.

 

CAPA – Sources of FundsFY 2009/10FY 2010/11FY 2011/12FY 2012/13FY 2013/14% last FY
Contribution Funding – DND

               

                 

                 

                 

                           
Membership Dues                                                                                    
Conference Registration Fees                                                                                
Certification Registration Fees                                                                              
Advertising and Sponsorship                                                                                  
Exhibit                                                                      
Contribution from Former CAPA                                                
Total                                                                                                            

Table 10 shows that DND                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      

As indicated earlier, most immediate outcomes not related to the generation of funds were achieved. The evaluation assessed that there was value for money for those achievements.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            

2.5.6 Trend of the Output/Input Ratio towards Achieving the Aim of this Contribution Program

To assess the trend of the output/input ratio towards achieving the aim of the Contribution Program, the following indicators were used:

  • perceptions of performance in consideration of resources/expenditures; and
  • for the period FY 2009/10 to FY 2013/14, evidence of evolution in the number of accreditations by province, number of certification exams, number of Continuing Professional Development (CPD) training activities, number of PA programs in universities, number of members and membership dues, number of CAPA sponsors, number of participants for CAPA events (including the annual conference), level of public awareness about the PA profession, and number of certified military and civilian PAs.

Due to the lack of a baseline or comparisons with other existing programs, it is difficult to assess whether the levels of resources utilized and of expenditures made were sufficient to achieve the aim of this program. Nevertheless, some data support relevant observations. For instance, one element was isolated to assess its specific efficiency. Certification fees versus certification costs was the element selected.

Key Finding 10: The cost generated by the PA certification process is greater than certification process revenues.

Table 11 presents the percentage of the costs covered by the fees charged to operate the certification process put in place by the CAPA.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

Table 11. Certification Fees versus Certification Expenses. This table shows the amount of expenses covered by the fees charged as a percentage of the cost of the certification process.

Table Summary

This table shows the amount of expenses covered by the fees charged as a percentage of the cost of the certification process. It has four functional columns that list the fiscal years. Read across the row to know the portion of certification expenses covered by certification fees per fiscal year.

 

 FY 2010/11FY 2011/12FY 2012/13FY 2013/14
Portion of Certification Expenses Covered by Certification Fees                                

 

Key Finding 11: The efficiency of delivering expected CAPA Contribution Program outputs was inconsistent.

It was expected that the outputs of the CAPA Contribution Program would show progressive improvement and positive results. Table 12 indicates the results applicable to the main outputs.

Table 12. Evaluation of Outputs and Deliverables. This table shows result-based information pertaining to the main outputs of the Contribution Program for the period FY 2009/10 to 2013/14.

Table Summary

This table shows result-based information pertaining to the main outputs of the Contribution Program for the period FY 2009/10 to FY 2013/14. It has seven columns. The left-hand column lists the types of outputs. For each type of output, read across the row to know the number, dollar amount or rate in each fiscal year. The far right-hand column gives an assessment for each output based on the analysis of the qualitative data.

 

 OutputsFY 2009/10FY 2010/11FY 2011/12FY 2012/13FY 2013/14Assessment
Implementation of the PA Model in Provinces and Territories 1 2 0 0 0 Stagnation
PA Programs Initiated in Universities 1 2 0 0 0 Stagnation
Certification Exams Administered by PACCC 38 78 90 99 90 Stable
Certified Military PAs 48 83 122 143 168 Growth
Certified Civilian PAs 248 324 392 465 545 Growth
CPD Training Credits Approved (Average/Member) 19.75 19.5 16.05 20.25 18.75 Stable
PAs Graduated from University-based Programs 31 54 46 51 60 Stable
Military PAs Graduated from CFHSTC 24 20 20 17 21 Stable
Number of Members 217 312 395 488 508 Growth
Membership Dues Received                                                                             Growth
Number of Sponsors 5 5 9 9 9 Stable
Revenues Generated from Sponsors                                                                                          
Incomes Generated by Annual Conference                                                                                          
Incomes Generated by Exhibits                                                                              
Incomes Generated by PA Boutique                                                
Number of Participants at Annual Conference 111 137 160 95 158 Unstable

The limited expansion of the PA model in provinces and territories was a significant factor in the development of the PA profession that impacted on: membership levels; PA position development; availability of universities to offer new PA education programs; accreditation of new PA education programs; and certification of PAs. The complexity of dealing with 10 provinces and three territories might have been underestimated when the CAPA Contribution Program was created.

2.5.7 Meeting the Objectives Established at the Creation of the Program

Key Finding 12: The Contribution Program met many of the objectives required in the initiating documents, but did not succeed in achieving financial viability and self-sufficiency.

According to the initiating documents for the CAPA Contribution Program, an evaluation must be conducted to report on how the Program has met the Program objectives. To determine the degree of achievement of the objectives stated in the initiating documents, the evaluation used the conclusions reached in sections 2.4 and 2.5 of this evaluation report. A compilation of the CAPA Contribution Program objectives that were either achieved or not achieved is provided in Table 1 in the Executive Summary.

2.6 Other Issues

CFHSG, as the sponsoring organization of the Contribution Program, raised the following parallel issue during the conduct of the evaluation.

Key Finding 13: Limitations in policy restrict the ability of DND to reimburse CAF members for professional memberships in the CAPA. This acts as a financial disincentive for personnel to maintain membership.

CAF PAs are an important part of the CAPA organization, comprising a large proportion of current membership and thus contributing to the financial viability of the CAPA Program. One barrier to membership is the financial impact on the individual CAF members to maintain CAPA membership. While the Surgeon General has directed that a CAF PA shall be certified and therefore must be a member of the CAPA, CAPA membership fees are currently not reimbursed to the military PAs, while most other CAF Health Care Professionals’ memberships are reimbursed20 (e.g. nurses, physicians). The current policy (source: CANFORGEN 125/13) limits reimbursement to CAF members of memberships that are legally required for the performance of their duties. In the case of most health care professions, the legal requirement is found in either federal law or provincial regulations. However, as the PA profession is not yet recognized and regulated in all provinces, this basic principle of the CAF membership reimbursement policy is not met. In contrast, the standard for reimbursement of public service employees does not require a legal authority but only membership to be considered by the employer as a requirement for the continuation of the performance of the duties of the employee’s position. The authority to make the determination for public service employees is delegated to Level 2 advisors (generally Brigadier-General rank or EX-02). CANFORGEN 125/13 is an interim policy until a new Compensation and Benefits Instruction is developed and issued. In relation to this approach, CMP should investigate mechanisms that would allow for the reimbursement of CAPA membership fees to military PAs as required for the performance of their duties.

3.0 Conclusion and Recommendation

In assessing the relevance and performance of the CAPA Contribution Program, the evaluation noted the importance of the CAPA as an independent organization that establishes and maintains the regulation and the curriculum of the PA profession both within the CAF and the provinces. However, evidence demonstrated that the CAPA failed to meet its main goal of financial self-sufficiency over the duration of the program.

The DND Contribution Program funding also supported the CAPA’s objective to become a nation-wide health care professional association and to promote the PA profession across Canada. Funding of this CAPA objective, however, is outside of the DND/CAF mandate and does not align with DND/CAF strategic outcomes. The subsidization of a professional association, which supports a national health care profession, should not be the sole responsibility of DND. Funding, if necessary, should be assumed by or shared with other national health-related sponsors.

Continued support to the CAPA, through a DND-sponsored Contribution Program, is not recommended. The termination of the Contribution Program at the end of the current terms and conditions as of March 31, 2016, necessitates that the CAPA become financially viable. If the CAPA does not become financially self-sufficient, it could lead to the Association’s eventual dissolution, which could impact the CAF PA profession. As such, the following recommendation is made:

ADM(RS) Recommendation

1.         The Contribution Program for the CAPA should be discontinued as of March31, 2016. If the CAPA does not become financially self-sufficient and ceases to exist as an association, determine the potential implications and way ahead for the CAF PA profession on the following issues:

  1. the continued certification of CAF PAs, coordinated by the PACCC, as recommended by the 2007 OAG Report on Military Health Care;
  2. the preservation and renewal of accreditation for the CAF PA education program at the CFHSTC based on the competency profile developed by the CAPA;
  3. the requirement to meet CAF PA clinical training needs through the use of civilian health care facilities; and
  4. the maintenance of operational capabilities the PA profession provides to the CAF.

OPI: CMP

Annex A—Management Action Plan

ADM(RS) Recommendation

1.         The Contribution Program for the CAPA should be discontinued as of March31, 2016. If the CAPA does not become financially self-sufficient and ceases to exist as an association, determine the potential implications and way ahead for the CAF PA profession on the following issues:

  1. the continued certification of CAF PAs, coordinated by the PACCC, as recommended by the 2007 OAG Report on Military Health Care;
  2. the preservation and renewal of accreditation for the CAF PA education program at the CFHSTC based on the competency profile developed by the CAPA;
  3. the requirement to meet CAF PA clinical training needs through the use of civilian health care facilities; and
  4. the maintenance of operational capabilities the PA profession provides to the CAF.

OPI: CMP

Management Action

The Surgeon General supports the ADM(RS) recommendation to not renew the DND-sponsored CAPA Contribution Program.

The CAPA was founded in 1999 by CFHSG with a view to create, legitimize, and advocate for PA as a recognized health occupation amongst other health care professions, as well as to heighten their already outstanding reputation within and outside the DND/CAF. The CAPA has now grown into a credible pan-Canadian association with members from various provinces, civilian stakeholders, government agencies, and other federal departments. The CAPA has successfully established professional development programs, certification/re-certification programs, medical education programs, and other initiatives.

During recent discussions with the CAPA, CFHSG was advised that the CAPA will address its long-term financial viability with a series of measures, including an increase in CAPA membership fees. The CAPA and CFHSG intend to continue their already well-established relationship via mutual presence and consultation on various committees and sponsorship events.

If the CAPA is not able to become financially self-sufficient and ceases to exist as an association, CFHSG has considered possible implications and mitigation strategies for the CAF PA profession.

The accreditation of the CAF PA education program should not be affected. The PA course curriculum taught at the CFHSTC was originally based on the University of Nebraska PA curriculum. If the CAPA ceased to exist as an organization, CFHSG would work with the provinces to align the competency profile of the PA profession.

  • The training success of CAF PAs is subsequently certified by the CAPA via a certification process (PACCC). Currently only New Brunswick and Manitoba recognize the CAPA certification, while the other two provinces that have a PA program (Ontario and soon Alberta) do not presently require PAs to be certified. The CAPA is working towards having their certification recognized by all provinces, thus allowing the PA certification to be “portable” across Canada, which would benefit the CAF PA profession with easier access to on-going CAF PA clinical training opportunities. If the CAPA ceased to exist as an organization, CFHSG would identify a collaborative approach to work with the provinces that have established PA programs.
  • Certification of CAF PAs by a recognized health association remains a main objective of the CAF Surgeon General. It is a sine qua non condition for any health system to have all of its health care professionals licensed/certified by a recognized professional body, to ensure patient safety, operational readiness, quality in health care delivery and mission success. Annual CAPA membership of each CAF PA is required to maintain professional certification status. Through CAPA membership, CAF PAs will contribute significantly to the CAPA’s financial stability. CFHSG will continue to work with the departmental office of primary interest (OPI) to obtain an appropriate financial authority for the reimbursement of CAPA membership for CAF PAs.
  • The Medical Technician-PA occupation restructure process has resulted in the identification and reassignment of 109 clinical PA positions to the new PA officer occupation and 94 Medical Technician-PA leadership positions to the Medical Technician occupation. With a reduction in the total number of positions required within the PA officer occupation, the number of certified personnel presently available is considered sufficient to maintain CAF operational capabilities.

OPI: CMP

OCI: CFHSG

Annex B—Evaluation Methodology, Validation and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The findings and associated recommendations of this evaluation are supported by multiple lines of evidence collected through qualitative and quantitative research methods. Information and data collected from relevant sources were analyzed to inform conclusions on the relevance and performance (effectiveness and efficiency/economy) of the program.

Key activities of the evaluation were grouped together to determine the expected outcomes of the program as defined in the logic model. Performance indicators (PI) were identified for each outcome to assess the performance of the CAPA Contribution Program. Outcomes and their associated PIs are listed in the Evaluation Matrix in Annex D.

The data for each PI was collected using the following research methods:

  • Literature review
  • Document review
  • Key informant interviews
  • Site visits
  • Administrative and financial data reviews

1.2 Details on Data Collection Methods

1.2.1 Literature Review

A literature review was undertaken to identify the existence of similar programs in other government departments or in other countries, to assess which type of national associations exist to support them, and to understand the evolution of the profession.

1.2.2 Document Review

A document review was completed to provide a thorough picture of the purpose, scope and mandate of the CAPA Contribution Program. Additionally, the document review enabled visibility into the larger organizational and operational context in which the program operates. Core program documents like the initiating documents were used to establish parameters regarding program delivery and as a source of data to support the performance evaluation of the economy and efficiency of the program.

The documents reviewed during the conduct phase of the evaluation included the following:

  • official initiating documents for this Contribution Program;
  • policy, legislation, and related accountability documents, such as: TBS Policy and Directive on Evaluation, Departmental Administration and Operation Directives, Departmental Performance Reports, Reports on Plans and Priorities applicable to the period covered by the evaluation (FY 2009/10 to FY 2013/14), and the DND Program Alignment Architecture;
  • strategic documents and plans, such as: CAPA Annual Activity Reports on the contribution, CAPA strategic plans, results of the yearly (2010 to 2014) PA members’ marketing survey, and results of the 2014 public opinion survey;
  • GC direction and related documents, such as: Canada First Defence Strategy, Speech from the Throne, Priorities of Defence, and GC whole-of-government framework;
  • program specific documents, such as: reporting documents, internal reports, and extractions from the CAPA website;
  • other government/military documents, such as: reports, service papers, and related documentation that enable or prevent comparisons between the CAPA Contribution Program and similar programs in allied militaries; and
  • other audits and evaluations conducted in the past by the OAG or by ADM(RS) relevant to the present evaluation.

1.2.3 Key Informant Interviews

Most of the key stakeholders who contributed to interviews were located within the vicinity of Ottawa-Gatineau or were contacted by phone. Interviews were conducted with key senior stakeholders from the CAPA (the Board of Directors and the management staff); from CFHSG (senior officials); from CMP (senior official); and from the CMA (accreditation and advocacy officials).

The interviewees were given an interview guide in advance. Interview guides were tailored to each individual and interview questions were aligned with the PIs indicated in the evaluation matrix. The evaluator transcribed the notes taken during the interviews in an evidence matrix with a view to regroup all of them according to related PIs and to make a conclusive analysis.

1.2.4 Site Visits

The CAPA Headquarters, the CMA Headquarters, and the CFHSG Headquarters were visited.

1.2.5 Administrative and Financial Data Reviews

Financial and administrative data for the CAPA Contribution Program, obtained from the CAPA itself, and those created and maintained by the CFHSG Comptroller office were analyzed with respect to the defined PI provided in the Evaluation Matrix.

2.0 Validation

Validity and credibility of data is a concern in all evaluations. To address these concerns, peer review, saturation, triangulation, and evaluation advisory group mechanisms were employed.

2.1 Peer Review

At each step of the evaluation where documents were created (i.e. work plan, logic model, evaluation matrix, evidence matrix, interview guides, portion of draft report, etc.), an internal peer review was conducted to validate the documents and to provide multiple reviews of the evaluation.

2.2 Saturation

The data collection was concluded once the evaluator assessed that the facts and evidence collected were consistent and led to the same conclusions for a given PI. The saturation point was then considered reached and the analysis could be initiated.

2.3 Triangulation

This triangulation was achieved by the diversity of methods to capture the same information. For instance, information that had been gathered in the initiating document and confirmed in financial statements was discussed in interviews. Another triangulation method was to ask the same question relating to the same PI to different stakeholders. Obtaining the same data from multiple stakeholders was considered data validation.

2.4 Evaluation Advisory Group

An Evaluation Advisory Group was established with a view to place the evaluation in a participative mode with stakeholders and to give them an opportunity to validate the tools, observations and findings as the evaluation process proceeded. Meetings were scheduled at different phases of the evaluation: initial meeting, review of the logic model and of the evaluation matrix, delivery of preliminary findings, review of the draft report, and discussions on the management action plan.

3.0 Limitations

In this section, limitations affecting the evaluation are identified and assessed. A mitigation strategy is also included to acknowledge how the evaluation attempted to overcome those limitations. The limitations and mitigation strategies are summarized in Table B-1.

Table B-1. Evaluation Limitations and Mitigation Strategies. This table identifies the main limitations of the evaluation process and the possible mitigation strategies.

Table Summary

This table identifies the main limitations of the evaluation process and the possible mitigations. It has two columns. The left-hand column lists the limitations, and the right-hand column lists the mitigation strategies.

 

LimitationsMitigation Strategies
Inconclusive benchmarking. None of the allies have been involved in the implementation and development of the PA profession in their health care systems. The evaluation took into consideration all facts and evidence collected from allies as background information only.
Non-existence of comparable programs. This contribution program is unique in terms of its mandate and objectives. There are 15 grant and contribution programs managed by the DND/CAF. The CAPA Contribution Program is the only one aimed at developing a new concept (the PA profession) which will, in the end, be more beneficial to the general Canadian population than to the sponsor of the contribution. The limitation raises the difficulty to compare relevance, effectiveness, efficiency and economy to other grant and contribution programs. Conscious of the lack of potential comparison with other programs and of the absence of a baseline, efforts were made to align the assessment of the worth and value of the program with universally recognized methods of evaluation. Mitigation was then ensured by continuously recognizing this fact during the assessment of the data and in increasing the sources and methods of data collection to ensure validity.
Interview bias. Interviews may lead to perceptions and interpretations of facts and evidence. Insights derived from interviews require corroboration from multiple sources: either from quantitative data or from agreement with other interviewees. Validation through saturation and triangulation was employed.
PA profession in Canada. There are a limited number of PAs in Canada, which may not provide a full appreciation of the PA profession’s achievements. The PA profession is clearly a domain in progress in other countries and related information had to be treated with a continuously open-minded approach in order to give fairness to the concept.
Period for data collection. The contribution program ran for seven years, but only the first five years of data were readily available. The five years of data were used to extrapolate the situation for the seven years. Validity is ensured when taking into consideration triangulation between methods.

Annex C—Logic Model

Figure C-1. Logic Model for the CAPA Contribution Program. This flow chart shows the relationship between the program’s main activities, outputs, and expected outcomes.

Text description for Figure C-1:

Figure Summary. This flowchart shows the relationship between the program’s main activities, outputs and expected outcomes.

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Tools and facilities exist and are used to certify CAP PAs.

Outputs: More CAF PAs are certified.

Immediate Outcomes: CAF PAs have the necessary professional skills.

Intermediate Outcomes: CAF members have confidence in PA practitioners.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point B:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Liaison and negotiation with civilian health care facilities are conducted.

Outputs: MOUs and contracts are signed with civilian health care facilities.

Immediate Outcomes: CAF PAs have increased access to civilian hospitals to complete residency requirements.

Intermediate Outcomes: CAF members have confidence in PA practitioners.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point C:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Specific MOUs/contracts for access to residency are prepared.

Outputs: MOUs and contracts are signed with civilian health care facilities.

Immediate Outcomes: CAF PAs have increased access to civilian hospitals to complete residency requirements.

Intermediate Outcomes: CAF members have confidence in PA practitioners.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point D:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Provinces and territories are assisted to establish credential standards and licensing for PAs.

Outputs: Effective accreditation program is delivered through the CM.

Immediate Outcomes: Academic programs and CFHSTC accreditations are renewed.

Intermediate Outcomes: Single national PA certification standards exist.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point E:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Develop national competencies profile.

Outputs: Effective accreditation program is delivered through the CM.

Immediate Outcomes: Number of university-based accredited programs for PAs has increased.

Intermediate Outcomes: Single national PA certification standards exist.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point F:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Organize the annual national conference and provide CPD opportunities.

Outputs: Certification exams, annual conferences, continuing medical education training produce results.

Immediate Outcomes: Other revenue- generating activities are undertaken.

Intermediate Outcomes: The CAPA is financially viable and self-sufficient.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point G:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Develop and administer certification exams and CPD tracking.

Outputs: Certification exams, annual conferences, continuing medical education training produce results.

Immediate Outcomes: Membership revenues have increased.

Intermediate Outcomes: The CAPA is financially viable and self-sufficient.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Starting Point H:

Inputs:

CFHSG: Funds made available by the seven-year DND contribution commitment ($5 million), professional support, management and oversight; andCAPA: Staff, resources, equipment, funds from other sources, networking and facilities.

Activities: Perform incorporation activities.

Outputs: Incorporation process has been undertaken and is successful.

Immediate Outcomes: The CAPA is incorporated.

Intermediate Outcomes: The CAPA is financially viable and self-sufficient.

Ultimate Outcome: Ensure sufficient capacity of certified CAF PAs to meet CAF operational and in-garrison health care needs.

Annex D—Evaluation Matrix

Note : According to the initiating documents for this contribution program, in addition to covering all issues determined by the TBS Policy on Evaluation (2009), this evaluation must specifically address the relevance, success, and cost-effectiveness of the contribution program in support of the CAPA.

Table D-1. Evaluation Matrix—Relevance – Continued Need. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the CAPA Contribution Program’s relevance pertaining to the continued need for this program.

Table Summary

This table indicates the data collection methods used to assess the evaluation question for determining the relevance of the CAPA Contribution program applicable to the continued need for the program. The table has seven columns and four rows. The left-hand column lists the evaluation questions. Read across the row to know the indicators applicable to the evaluation question. For each indicator, continue reading across the row to know the data collection methods used in the evaluation.

 

Relevance – Continued Need for ProgramIndicatorsLiterature ReviewDocument ReviewKey Informant InterviewsAdministrative/ Financial Data ReviewSurveys / Questionnaire / Focus Groups
1. Evaluation Question. To what extent (1) does the CAPA Contribution Program address a demonstrable need, and (2) is the Contribution Program responsive to the needs of Canadians? 1.1 Evidence of a current and future need for the CAPA Contribution Program. Yes21 Yes Yes No No
1.2 Evidence that the CAPA Contribution Program is reactive to the needs of the DND/CAF and Canadians. Yes Yes Yes No

Yes

Survey done by the CAPA

1.3 Extent to which the CAPA Contribution Program complements the services provided by other departments, agencies, and/or organizations (internal/external). Yes Yes Yes No No

Table D-2. Relevance – Alignment with Federal Roles and Responsibilities. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance (alignment with federal roles and responsibilities) of the CAPA Contribution Program.

Table Summary

This table indicates the data collection methods used to assess the evaluation question for determining the relevance pertaining to the CAPA Contribution Program’s alignment with federal roles and responsibilities. The table has seven columns and two rows. The left-hand column lists the evaluation question. Read across the row to know the indicator applicable to the evaluation question. For each indicator, continue reading across the row to know the data collection methods used in the evaluation.

 

Relevance – Alignment with Federal Roles and ResponsibilitiesIndicatorsLiterature ReviewDocument ReviewKey Informant InterviewsAdministrative/ Financial Data ReviewSurveys / Questionnaire / Focus Groups
2. Evaluation question.How does the delivery of the CAPA Contribution Program align with the roles and responsibilities of the federal government? 2.1 Evidence of alignment between federal roles and responsibilities (including legislative and policy obligations) and the delivery of the CAPA Contribution Program. Yes Yes Yes No No

Table D-3. Relevance – Alignment with Government Priorities. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance (alignment with government priorities and departmental strategic outcomes) of the CAPA Contribution Program.

Table Summary

This table indicates the data collection methods used to assess the evaluation question for determining the relevance pertaining to the CAPA Contribution Program’s alignment with government priorities and departmental strategic outcomes. The table has seven columns and three rows. The left-hand column lists the evaluation question. Read across the row to know the indicator applicable to the evaluation question. For each indicator, continue reading across the row to know the data collection methods used in the evaluation.

 

Relevance – Alignment with Government PrioritiesIndicatorsLiterature ReviewDocument ReviewKey Informant InterviewsAdministrative/ Financial Data ReviewSurveys / Questionnaire / Focus Groups
3. Evaluation question. How do the achievements of the CAPA Contribution Program align with federal government priorities and with departmental strategic outcomes? 3.1 Evidence of alignment between the CAPA Contribution Program’s objectives and current federal government priorities. Yes Yes Yes No No
3.2 Evidence of alignment between the CAPA Contribution Program’s objectives and the DND/CAF strategic outcomes. Yes Yes Yes No No

Table D-4. Evaluation Matrix—Performance (Effectiveness). This table indicates the data collection methods used to assess the evaluation issues/questions for determining the CAPA Contribution Program’s performance in terms of achievement of outcomes (effectiveness).

Table Summary

This table indicates the data collection methods used to assess the evaluation issues/questions for determining the CAPA Contribution Program’s performance in terms of achievement of outcomes (effectiveness). The table has seven columns and fifteen rows. The left-hand column lists the evaluation questions. Read across the row to know the indicator applicable to the evaluation questions. For each indicator, continue reading across the row to know the data collection methods used in the evaluation.

 

Relevance – Alignment with Government PrioritiesIndicatorsLiterature ReviewDocument ReviewKey Informant InterviewsAdministrative/ Financial Data ReviewSurveys / Questionnaire / Focus Groups
4.1 Immediate Outcome: CAF PAs have the necessary professional skills. Evaluation question. To what extent do the CAF PAs possess the necessary professional skills? 4.1.1 Evidence of improvement in the number of CAF PAs certified during the period FY 2009/10 to FY 2013/14. No Yes No Yes No
4.1.2 Evidence that certified CAF PAs can meet the required competencies. No Yes Yes No No

4.2 Immediate Outcome: CAF PAs have increased access to civilian hospitals to complete residency requirements.

Evaluation question. To what extent have the CAF PAs gained increased access to civilian health care facilities in order to complete their residency requirements?

4.2.1 Evidence of increased access demonstrated by the number of CAF PAs having filled their residency requirements for the period FY 2009/10 to FY 2013/14. No No Yes Yes No
4.2.2 Number of CAF PAs who did not complete residency requirements during the period FY 2009/10 to FY 2013/14 due to lack of MOUs/contracts with hospitals. No No Yes Yes No

4.3 Immediate Outcome: Academic programs and CFHSTC accreditations are renewed.

Evaluation question. To what extent have academic programs and the CFHSTC’s accreditations been renewed?

4.3.1 For the period FY 2009/10 to FY 2013/14, evidence that the accreditation of the CFHSTC program has been renewed. No Yes Yes Yes No
4.3.2 For the period FY 2009/10 to FY 2013/14, evidence that the accreditation of university-based programs has been renewed. No Yes Yes Yes No

4.4 Immediate Outcome: Number of university-based accredited programs for PAs has increased.

Evaluation questions. To what extent has the number of university-based accredited programs increased as a result of the CAPA Contribution Program?

4.4.1 Changes in the number of university-based accredited programs for the period FY 2009/10 to FY 2013/14 due to CAPA expansion programs. No Yes Yes Yes No
4.4.2 Evidence of effective marketing activities performed to gain professional endorsement on the accreditation programs and to develop the PA profession. No Yes Yes Yes No

4.5 Immediate outcome: The CAPA is incorporated.

Evaluation question. What action has been taken by the recipient of the Contribution Program to incorporate the CAPA?

4.5.1 Evidence of actions leading to potential incorporation. No Yes Yes No No

4.6 Intermediate outcomes: CAF members have confidence in military PA practitioners.

Evaluation question. Have CAF members developed confidence in PA practitioners as a result of the CAPA Contribution Program?

4.6.1 The level of CAF members’ satisfaction with military PA practitioners (for the period FY 2009/10 to FY 2013/14). No No Yes No

Yes

Annual survey of CAF members on CF Health Services

4.7 Intermediate outcomes: Single national PA certification standard exists.

Evaluation question. To what extent does a single national PA certification standard exist?

4.7.1 Evidence of the existence of a single national PA certification standard. No Yes Yes Yes No

4.8 Ultimate outcome: Ensure a sufficient capacity of certified CAF PAs is available to meet operational and in-garrison health care needs.

Evaluation question. To what extent is the number of certified PAs meeting CAF operational and in-garrison requirements?

4.8.1 Evolution of the number of PA Military Occupational Structure Identification requirements and the number of PA positions filled (for the period FY 2009/10 to FY 2013/14). No No Yes Yes No
4.8.2 Evidence of PA shortage in CAF deployed operations such as Afghanistan, Haiti, Alert and Vancouver (for the period FY 2009/10 to FY 2013/14). No No Yes No No
4.8.3 Evolution of the military PA retention rate (for the period FY 2009/10 to FY 2013/14). No No Yes Yes No
4.8.4 Evidence of CAF ability to attract new military PAs (for the period FY 2009/10 to FY 2013/14). No No Yes Yes No

Table D-5. Performance (Efficiency and Economy). This table indicates the data collection methods used to assess the evaluation issues/questions for determining the performance of the CAPA Contribution Program and the objectives stated in the initiating documents.

Table Summary

This table indicates the data collection methods used to assess the evaluation issues/questions for determining the CAPA Contribution Program’s performance in terms of achievement of outcomes (efficiency and economy). The table has seven columns and seventeen rows. The left-hand column lists the evaluation questions. Read across the row to know the indicator applicable to the evaluation questions. For each indicator, continue reading across the row to know the data collection methods used in the evaluation.

 

Relevance – Alignment with Government PrioritiesIndicatorsLiterature ReviewDocument ReviewKey Informant InterviewsAdministrative/ Financial Data ReviewSurveys / Questionnaire / Focus Groups

5.1 Immediate Outcome: Other revenue-generating activities are undertaken.

Evaluation question. To what extent have “other revenues” been generated as a result of the actions taken by the CAPA?

5.1.1 Variations in other revenues (for the period FY 2009/10 to FY 2013/14). No No Yes Yes No
5.1.2 Evidence of specific activities aimed at generating permanent source of funds (for the period FY 2009/10 to FY 2013/14). No Yes Yes No No
5.1.3 Evidence of growth in the number of sponsorships (for the period FY 2009/10 to FY 2013/14). No Yes Yes Yes No
5.1.4 Evidence of growth in the revenues generated by sponsorship (for the period FY 2009/10 to FY 2013/14). No Yes Yes Yes No

5.2 Immediate outcome: Membership revenues have increased.

Evaluation question. To what extent and in what proportion has the membership increased?

5.2.1 Evidence of growth in the number of CAPA members (for the period FY 2009/10 to FY 2013/14). No Yes Yes Yes No
5.2.2 Evidence of growth in revenues generated by the different categories of membership (for the period FY 2009/10 to FY 2013/14). No No Yes Yes No

5.3 Intermediate outcome: The CAPA is financially viable and self-sufficient.

Evaluation question. To what extent is the CAPA self-sufficient and has reached autonomy?

5.3.1 The degree of financial autonomy and self-sufficiency of the CAPA at the end of FY 2013/14. No No Yes Yes No
5.3.2 At the end of the extension period (FY 2015/16), probability that the CAPA will become financially self-sufficient. No No Yes Yes No
5.3.3 Improvements shown in the sources of funds external to DND since the inception of the contribution program. No Yes Yes Yes No
5.3.4 Improvements shown in the number of sponsors. No Yes Yes Yes No
5.3.5 Improvements/ increases shown in the amount of funding from other sources. No Yes Yes Yes No

5.4 Unintended outcomes: Limitation of self-sufficiency options due to stacking of grants and contributions.

Evaluation question. What are the impacts of this unintended outcome on the results of the CAPA Contribution Program?

5.4.1 Evidence of unintended positive or negative elements having impacted on the results of the CAPA Contribution Program. No Yes Yes No No
5.5 Evaluation question. Is the progress made toward expected outcomes adequate for the resources expended? 5.5.1 Funds originating from other sources compared to the total expended for the period FY 2009/10 to FY 2013/14 to determine self-sufficiency and viability. No No Yes Yes No
5.5.2 For the period FY 2009/10 to FY 2013/14, evidence of positive impacts and of value for money in the achievement of the immediate outcomes. Yes Yes Yes Yes No
5.6 Evaluation question. Is the trend of the output/input ratio sufficient for achieving the aim of this Contribution Program? 5.6.1 Perceptions of performance in consideration of resources/expenditures. Yes No Yes Yes No
5.6.2 For the period FY 2009/10 to FY 2013/14, evidence of evolution in the number of accreditations by province, number of certification exams, number of CPD training activities, number of PA programs in universities, number of members and membership dues, number of CAPA sponsors, number of participants at CAPA events, level of public awareness about the PA profession, and number of certified military and civilian PAs. Yes Yes Yes Yes No
5.7 Have the objectives established in the initiating documents been achieved? 5.7.1 Degree of achievement of the objectives stated in the initiating documents. Yes Yes Yes Yes No

 

Yes

___________________________________________________________________________________________________________________________

Footnote 1 The National Defence Act (Section 4) and Queen’s Regulations and Orders (Section 34.07(4)) – Medical Services Entitlements to Medical Care provide the authority for the creation of the CAPA.

Footnote 2 Physician Assistant Certification Council of Canada.

Footnote 3 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525, article 6.5.5. Last consulted on February 12, 2015.

Footnote 4 Chief Review Services was the former designation of ADM(RS). The current designation came into effect on May 13, 2015.

Footnote 5 Certification is provided through the PACCC, which is a subsidiary of the CAPA.

Footnote 6 Source: The Contribution Funding Agreement (CFA) and other initiating documents.

Footnote 7 TBS. Directive on the Evaluation Function, April 1, 2009. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681&section=text. Last consulted on February 1, 2015.

Footnote 8 Health Canada’s responsibilities are not oriented towards direct health care support. An exception to this is the provision of health care services to certain population groups for which it is constitutionally responsible (First Nations and Inuit people). Health Canada’s interest in health care is also reinforced by the creation, in 1999, of an Office of Nursing Policy embedded in the Health Canada Policy Directorate.

Footnote 9 http://www.hc-sc.gc.ca/hcs-sss/index-eng.php. Last consulted on March 15, 2015.

Footnote 10 Aboriginal Affairs and Northern Development Canada was the applied title of the department when this report was written; the name has been changed to Indigenous and Northern Affairs Canada, as of December 7, 2015.

Footnote 11 http://www.forces.gc.ca/en/about-reports-pubs-report-plan-priorities/2013-other-defence-priorities-corporate-risk-profile-paa-alignment-matrix.page. (Recommend removing the period at the end of this URL address, otherwise this link will not work – IM) Last consulted on March 15, 2015.

Footnote 12 http://www.forces.gc.ca/assets/FORCES_Internet/docs/en/about/CFDS-SDCD-eng.pdf. Last consulted on April 3, 2015.

Footnote 13 http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024. Last consulted on April 11, 2015.

Footnote 14 https://capa-acam.ca/paccc/. Last consulted on April 11, 2015.

Footnote 15 Discussions between the CFHSG and the CMA held at the beginning of 2016 revealed that, in the future, the CMA may no longer be involved in program accreditation. Another approving authority may eventually assume that responsibility.

Footnote 16 This document is available at: https://capa-acam.ca/wp-content/uploads/2012/12/NCP_en_sept20092.pdf.

Footnote 17 The term “efficiency” used in this evaluation is defined in the TBS Policy on Evaluation as “the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output.”

Footnote 18 The expression “economy” is defined in the TBS Policy on Evaluation as “minimizing the use of resources.” The policy adds that “. . . economy is achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes.”

Footnote 19                                                                                                                                                                                                                                                                                                                

Footnote 20 Extracts from the Public Service Alliance of Canada Collective Agreement: 1) Health Services collective agreement: Article 21 – Registration Fees, paragraph 21.01: The Employer shall reimburse an employee for the payment of membership, registration or other related fees to organizations or governing bodies when the Employer is satisfied that the payment of such fees is a requirement for the continuation of the performance of the duties of the employee’s position. 2) Operational Services Group collective agreement: Article 66 – Trade Certification Fees, paragraph 66.01: The Employer shall reimburse an employee for the payment of registration, licensing or certification fees to an organization, governing body or government agency when the payment of such fees is a requirement for the continuation of the performance of the duties of the employee’s position.

Footnote 21 In the following tables, the word “yes” indicates the method employed by the evaluation to measure the indicators. To facilitate its reading, all “yes” cases have been yellowed. For the last column, the method expected to be used is indicated in the box and shaded accordingly.

Date modified: