CORE COMPETENCY COMPLIANCE PROGRAM (CCCP)

PART I

AOA Core Competencies

Osteopathic Philosophy/Osteopathic Manipulative Medicine

Medical Knowledge

Patient Care

Professionalism

Interpersonal and Communication Skills

Practice-based Learning and Improvement

Systems Based Practice

In July 2003, the AOA Board of Trustees accepted and approved the Report of the Core Competency Task Force that created a new policy for AOA-accredited postdoctoral programs. As a result of this decision, additions were made to AOA intern and resident accreditation requirements to incorporate seven (7) core competencies into postdoctoral programs, the AOA inspection process, and into testing modalities for certification and re-certification of osteopathic physicians. Two of the core competencies, Medical Knowledge and Osteopathic Philosophy/Osteopathic Manipulative Medicine, are specialty specific. Each osteopathic specialty college is mandated to define and integrate these two core competencies into their respective training standards. The remaining five competencies are germane to all specialties.

Institutional Core Competency Plan

In order to assess effectiveness in the integration of core competencies, the AOA is requiring the development and implementation of an Institutional Core Competency Plan. Its purpose is to create an internal process that outlines the methods chosen by the institution to achieve compliance with implementation deadlines and evaluation activities. The Director of Medical Education (DME) is the institutional official recognized by the AOA as responsible for overseeing all aspects of osteopathic medical education within his/her respective facility, including the Institutional Core Competency Plan.

The written plan must address content issues that define responsibilities, goals, methodologies, and evaluation activities. The Director of Medical Education is charged to draft the plan that is to be approved by the Medical Education Committee. In addition, the Medical Education Committee is responsible to monitor the plan’s implementation and performance. At a minimum, the Institutional Core Competency Plan is to incorporate the following items:

Demonstrated institutional commitment to AOA Core Competencies
Assignment of responsibility to the DME in the formulation of the plan
Outline of the Medical Education Committee’s role for the approval and routine monitoring of the Institutional Core Competency Plan
Responsibility of the internship and residency directors and faculty for implementation of the plan.
Assigned responsibilities of support staff within the institution.
Integration of all AOA-accredited postdoctoral programs into the Institutional Core Competency Plan.
Determination of institutional resources, both personnel and financial, to successfully implement the plan.
Agreement on collaboration with the OPTI for assistance in the implementation of the plan.
Section of a method(s) to monitor the effectiveness and progress of the plan at the levels of the DME, Medical Education Committee, and OPTI.
Delineation of chosen teaching modalities for each core competency.
Determination of the evaluation modality selected for each core competency.
Formulation of anticipated outcomes for trainees in each program.
Inclusion of an expectation that traditional interns receive an adequate exposure to core competencies as an education goal.
Inclusion of a remediation plan for those who fail to meet performance expectations of the Institutional Core Competency Plan.
Creation of a continual quality improvement process incorporated into the Institutional Core Competency Plan.
Compilation of an annual written evaluation summary of overall programmatic effectiveness reviewed by the Medical Education Committee.

A written progress report of the Institutional Core Competency Plan must be reviewed and updated annually by the DME, with approval required of the Medical Education Committee, and a copy of the report sent the Osteopathic Graduate Medical Committee of the OPTI. The yearly report submitted to the Medical Education Committee and OPTI is to include the following:

Summary of progress made in implementation of the Institutional Core Competency Plan (until fully implemented).
Effectiveness of methods chosen by the institution to achieve compliance and evaluate Core Competencies.
Report on each trainee’s progress in learning and development in each Core Competency as reported on their respective annual report (e.g. Program Director’s Annual Evaluation Report).
Summary report for each trainee who is completing his/her respective program with an attestation that a minimal level of performance has been achieved in all competencies (e.g. Program Complete Summary-Final Resident Assessment as required by the specialty college).
Outcomes measures and report of success in achieving institutional goals.

GUIDELINES IN THE DESIGN OF THE INSTITUTIONAL CORE COMPETENCY PLAN

The AOA has developed a Core Competency Map and Program Director’s Annual Evaluation Report to assist DMEs in the design of an Institutional Core Competency Plan. These documents, described below, outline available instruments appropriate in assessing trainee performance. It is suggested that DMEs use these materials in the design of the plan and, with the concurrence of program directors, choose one recommended methodology to achieve compliance with each respective core competency. Likewise, the DME can select at least one recommended evaluation tool of that core competency. Starting in June 2006, all residents are to be assessed by at least two evaluation tools for each core competency to qualify for program complete status. This same process can be followed for each required core competency. Variability can be expected with Medical Knowledge and Osteopathic Philosophy and Osteopathic Manipulative Medicine that are unique to specialty training requirements as noted in AOA basic standards. The remaining five core competencies are common to all medical specialties and trainees can be assessed and evaluated in the same manner.

Each successive year, the Institutional Core Competency Plan document is to be reexamined and enhanced to build in additional metrics for evaluating each core competency. When all AOA core competencies are implemented in 2006, the Institutional Core Competency Plan is to be reviewed and updated to include best methods for institutional growth. The plan is to be designed as a dynamic document that serves as an institutional road map for continuous improvement in teaching and in evaluating competency-based medical education. The Institutional Core Competency Plan and the annual reports are to be made available to AOA evaluators when on-site accreditation reviews are conducted.

Institutions have flexibility in choosing the methods that work best for its postdoctoral programs. Consistency between the selected methods and those outlined in the Institutional Core Competency Plan is an expectation. Nothing prohibits a residency from developing additional core competency programs for a medical specialty; however, these supplemental programs are to operate within the methods already outlined in the Institutional Core Competency Plan. DME and residency program directors are expected to familiarize themselves with the Institutional Core Competency Plan and be held accountable for the integration of the plan into their respective programs.

CORE COMPETENCY COMPLIANCE PROGRAM (CCCP)

PART II

AOA Core Competency Development Task Force

AOA Core Competency MAP

The metrics chosen to assess a resident’s achievement of any competency must parallel the teaching process(es) employed by the residency program. As such, each institution will have flexibility when choosing methodologies to assess a resident’s performance.

Additionally, assessment tools vary in their reliability and validity. Therefore, it is advised than more than one assessment tool be used when assessing residents to improve the overall accuracy. The following tables present eight commonly used assessment metrics. A description of each metric is presented. Common uses, advantages and disadvantages for each are also presented.

360-Degree Evaluation

An assessment tool used to rate the performance of a resident. All individuals that have contact with the resident should

complete the evaluation. The information obtained may reveal trends (i.e. patterns of behavior) with certain groups (i.e., nursing staff, adolescent patients, geriatric patients, midlevel practitioners, etc.) that may be useful when providing feedback to the resident.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Communication skills
Interpersonal skills
Professionalism
Teamwork ability

Multiple raters

Improves validity
Improves reliability

Fosters self-reflection

Self-rating is a component

Peer pressure during process

Hesitation to be honest

Difficult to design

Standard set of items for all raters
Group-specific subset of items

Requires several raters

Not less than 20 attending physicians
Not less than 20 patients
At least two or more from other groups

Checklist

An assessment tool used to evaluate specific behaviors or tasks that are components of a more complex activity. The checklist records whether the action was performed or not. If performed, the checklist can be designed to identify if the action was performed accurately, partially correct, or unsatisfactorily/wrong.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Actions that are objective
- Specific
- Measurable

Examples:

Interviewing
Medical procedures

Useful information for feedback

What was not done
What was done

Correctly
Acceptably
Incorrectly

Difficult to design

What are the required actions
Requires expert opinion/consensus

Evaluator variability

Raters need to be trained

Only useful for fundamental skill assessment

Objective Structured Clinical Examination (OSCE)

An assessment tool that consists of multiple stations. The stations include various elements of clinical encounters. The stations include standardized patients (actors trained to portray illness in a standardized manner), actual patients, and/or components of clinical encounters (i.e., electrocardiograms for interpretation, radiographs for interpretation, etc.).

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Communication skills
Interpersonal skills
Professionalism
Psychomotor abilities

Multiple assessments

Improves validity
Improves reliability

Useful feedback information

What the resident does well
What needs improvement

Expensive

12 – 18 stations recommended
Selection/Creation of stations
Training of standardized patients (SPs)
Payment of SPs

Difficult to design

Scoring criteria
Passing thresholds

Monthly Service Rotation Evaluation

An assessment tool used to provide a global rating of performance. The forms usually record categories of behaviors, not specific actions, based on rating scales.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Global performance

Familiarity

Quick and simple to complete

Highly subjective

Rater biases

Leniency
Halo effect

Require direct observation of ratee

Second hand opinions reduce utility

Rater training required

Procedure/Case Logs

An assessment tool used to quantify patient encounters over a period of time.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Document training experience
- Scope
- Volume
- Variety

Can direct training

Fill exposure gaps
Increase procedural opportunities

Number of cases does not assure competence

Need to track patient outcomes

Assurance of data accuracy

Quality assurance

Time consuming activity

Data entry

Portfolios

An assessment tool used to document learning experiences. Usually a compilation of written documents (i.e., case logs, procedural logs, research activity, committee involvement, lectures and conferences attended, etc.).

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

A record of learning accomplishments

Useful for self-reflection on learning

Provide a global view of experiences

Time consuming to create

Difficult to assign a score

Written Examination

An assessment tool used to assess not only the examinee’s knowledge base, but also the ability to apply it to clinical situations. The most common written examination format uses multiple-choice questions.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Knowledge base
Level of understanding

Familiarity

Can cover many content areas quickly

Can be graded quickly

Can monitor progress over time

Use of anchor (repeated) questions

Require statistical analysis

Passing scores should be predetermined

Sampling error can occur

Use test blueprint

Chart Stimulated Oral Recall Examination

An assessment tool used to assess clinical problem-solving ability. Provides the ability to investigate the examinee’s rationale for requesting information (i.e., historical or physical examination data), interpretation of information provided, and management of selected cases, not evident by simply reviewing the chart.

Use(s)

Advantage(s)

Disadvantage(s)

Evaluates:

Problem-solving ability
- Ability to use information
- Ability to select the next step

Selected cases can be covered quickly

Can ask a series of related questions

Examiners must be trained

Scoring can be debated

Cases selection can be difficult

High anxiety level for some examinees

References

1. Frohna JG et al. Assessing Residents’ Competency in Care Management: Report of a Consensus Conference

Teaching and Learning in Medicine 2004; 16(1):77-84.

2. Swing SR. Assessing the ACGME General Competencies: General Considerations and Assessment Methods.

Academic Emergency Medicine 2002; 9:1278-1288.

3. ACGME Outcome Project. Accreditation Council for Graduate Medical Educations

http://www.acgme.org/Outcome/

4. Noel G et al. How well do Internal Medicine faculty members evaluate the clinical skills of residents?

Annals of Internal Medicine 1992; 117:757-765.

5. Newble D. Assessing clinical competence at the undergraduate level Medical Education 1992:26:504-511.

American Osteopathic Association

Program Director’s Annual Evaluation Report

Instructions for Program Director’s Annual Evaluation Report completion

The American Osteopathic Association requires DME’s and Program Directors to implement training, and Program Evaluators to assess, the AOA Core Competencies in all AOA training programs. Below is the timeline for the implementation and assessment process:

by July 2004, Program Directors must implement training in the first two competencies in a specialty specific manner:

· Osteopathic Philosophy & Osteopathic Manipulative Medicine, and

· Medical Knowledge

by January 2005, Program Evaluators must begin assessment through site review process of the first two competencies:

· Osteopathic Philosophy & Osteopathic Manipulative Medicine and,

· Medical Knowledge

by July 2005, Program Directors must implement training in the next three competencies:

· Patient-Care,

· Interpersonal and Communication Skills, and

· Professionalism

by January 2006, Program Evaluators must begin assessment through site review process of the next three competencies:

· Patient-Care,

· Interpersonal and Communication Skills, and

· Professionalism

by July 2006, Program Directors must implement training in the last two competencies:

· Practice-Based Learning Improvements and

· Systems-Based Practice Competencies

by January 2007, Program Evaluators must begin assessment through site review process of the last two competencies:

· Practice-Based Learning Improvements and

· Systems-Based Practice Competencies

As of January 2007, all seven core competencies may be evaluated as part of the on-site review process. On-site review evaluators must validate the presence of exposure to training in each competency within six months after it is to be implemented. However, some flexibility in the on-site evaluation must be expected during the first year of incorporation for each set of new competencies as included.

The attached instrument will assist you in this process; it was developed based on the AOA core competencies map and its associated references.

Specialty specific yearly reports and documents should be attached to this instrument.

Instructions: As part of its effort to monitor the educational progress of residents, the American Osteopathic Association asks that you complete a program director’s annual report on each osteopathic physician in your program. These reports will be reviewed by the specialty colleges and become part of the resident’s permanent file. Completion of a residency program requires an annual report from the resident and the program director for each year of training. Please evaluate the performance of the resident within thirty (30) days for the completion of the training year. The completed report should be sent directly to the appropriate specialty college listed on the attached sheet. This report includes observations of the resident in both in-hospital and ambulatory care settings. Please evaluate the resident based on each of the required elements of the following seven Core Competencies of the Osteopathic Profession.

Competency 1: Osteopathic Philosophy and Osteopathic Manipulative Medicine: Residents are expected to demonstrate and apply knowledge of accepted standards in Osteopathic Manipulative Treatment (OMT) appropriate to their specialty. The educational goal is to train a skilled and competent osteopathic practitioner who remains dedicated to life-long learning and to practice habits in osteopathic philosophy and manipulative medicine.

Competency 2: Medical Knowledge: Residents are expected to demonstrate and apply knowledge of accepted standards of clinical medicine in their respective specialty area, remain current with new developments in medicine, and participate in life-long learning activities, including research.

Competency 3: Patient Care: Residents must demonstrate the ability to effectively treat patients, provide medical care that incorporates the osteopathic philosophy, patient empathy, awareness of behavioral issues, the incorporation of preventive medicine, and health promotion.

Competency 4: Interpersonal and Communication Skills: Residents are expected to demonstrate interpersonal and communication skills that enable them to establish and maintain professional relationships with patients, families, and other members of health care teams.

Competency 5: Professionalism: Residents are expected to uphold the Osteopathic Oath in the conduct of their professional activities that promote advocacy of patient welfare, adherence to ethical principles, collaboration with health professionals, life-long learning, and sensitivity to a diverse patient population. Residents should be cognizant of their own physical and mental health in order to care effectively for patients.

Competency 6: Practice-Based Learning and Improvement: Residents must demonstrate the ability to critically evaluate their methods of clinical practice, integrate evidence-based medicine into patient care, show an understanding of research methods, and improve patient care practices.

Competency 7: System-Based Practice: Residents are expected to demonstrate an understanding of health care delivery systems, provide effective and qualitative patient care within the system, and practice cost-effective medicine.

1. Have you reviewed and approved the trainee’s research assignment (e.g., scientific paper, etc.)? FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A

2. Did the trainee participate in the annual resident in-service examination as required by the specialty college? FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A

3. Have you reviewed the results of the COMLEX III or the resident’s in-service examination with the trainee? FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A

4. Has the trainee met the requirement for the management of a panel of patients followed throughout the year in an ambulatory continuity setting? FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A

5. Has the trainee completed all other specialty specific requirements for this year?

Please comment on this trainee in terms of progress in the program, promise as a physician, and in other areas not specifically mentioned above. All comments will be treated confidentially.

This trainee has made satisfactory progress in this training program and is capable to proceed to the next year. FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A If no, please attach quarterly evaluations.

This confirms that this trainee has completed this year of training. FORMCHECKBOX Yes FORMCHECKBOX No FORMCHECKBOX N/A

The following signature verifies that the resident has had the opportunity to review this report.

This resident has been assessed with at least two evaluation tools for each required element of each of the seven competencies. FORMCHECKBOX Yes FORMCHECKBOX No

A document portfolio of this resident’s “best performance” evaluations for each competency is attached to this report. FORMCHECKBOX Yes FORMCHECKBOX No

I HEREBY ATTEST THAT THE GRADUATING RESIDENT HAS SUCCESSFULLY COMPLETED ALL THE REQUIREMENTS OF THE TRAINING PROGRAM, AND IS RECOMMENDED FOR PROGRAM COMPLETE STATUS. FORMCHECKBOX Yes FORMCHECKBOX No

The following signature verifies that the resident has had the opportunity to review this report.

CORE COMPETENCY COMPLIANCE PROGRAM (CCCP)

PART I

CORE COMPETENCY COMPLIANCE PROGRAM (CCCP)

PART II

Required Element #1: This resident demonstrated competency in his/her understanding and application

of OMT in (specialty field)

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #2: This resident integrated Osteopathic Concepts and OMT into the medical care he/she provided to patients as appropriate.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

FORMCHECKBOX completed OMT computer teaching modules

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #1: This resident demonstrated competency in the understanding and application of clinical medicine to patient care.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #3: This resident provided health care services consistent with osteopathic philosophy, including preventative medicine and health promotion based on current scientific evidence.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #2: This resident exhibited effective listening, written and oral communication skills in professional interactions with patients, families and other health professionals.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Comments:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #1: This resident understands national and local health care delivery systems and how they affect patient care and professional practice.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element:

Required Element #2: This resident advocated for quality health care on behalf of his/her patients and assisted them in their interactions with the complexities of the medical system.

Please check the box (es) for the methods, outcomes, or demonstrations of compliance that were utilized

Please check the appropriate rating box & comment on this resident’s performance for this element: