Continuum or Chasm: Using Assessment Strategies to Bridge the UME and GME Divide

This article is intended for clerkship directors, leaders in Undergraduate Medical Education (UME), and residency program directors who are interested in the current state and evolving innovations in assessing clinical skills of trainees during the transition between medical school and residency. This article seeks to:

• Describe Entrustable Program Activities (EPAs) as the emerging framework for longitudinal clinical skills training and assessment across the continuum of UME and Graduate Medical Education (GME)
• Review workplace-based assessments in general as a key clinical skills assessment methodology
• Summarize other existing and emerging strategies for assessing skills across the UME to GME continuum

The goal of surgical education is a safe, competent surgeon who is ready to begin a defined scope of independent practice upon graduation. Ideally, Undergraduate Medical Education (UME) and Graduate Medical Education (GME) programs would achieve this goal through a coordinated framework of training and assessment. This ideal framework would start with defining the knowledge, skills, and behaviors expected of a surgery graduate, and create a progressive curriculum across UME and GME. The foundational general medical knowledge and skills expected of all UME graduates would be augmented with a defined set of skills required for a surgical intern to perform expected duties on day one. Consistent educational and assessment practices for bridging the surgery continuum across the country will assure all residency programs can expect the same level of knowledge and skills of interns regardless of the medical school from which they come. This objective would include a process for medical schools to identify students who do not demonstrate the expected knowledge, skills, or behaviors so that remediation occurs prior to graduation. Thus, a key feature of this ideal system is a mechanism to provide accurate, actionable performance assessment data across the transition from UME to GME.

Many pieces of this ideal system are either in place or in development, starting with a framework to define the core essential activities that a graduating surgery resident should be able to perform, and the general attributes, or competencies, required to successfully perform these expected professional activities.^1,2 The Core Entrustable Professional Activities (EPA) for Entering Residency document describes the expected skills of a UME graduate, and the American College of Surgeons has described in more detail the specific knowledge and skills a surgical intern should possess prior to starting residency.^3,4 The EPA framework is now being developed for GME training in multiple specialties, creating a specialty-specific continuum of progressive, observable behaviors amenable to longitudinal assessment across UME and GME.^5–8 Pediatrics is furthest along with this effort; a group of institutions is studying the outcome of making advancement decisions from UME to GME based on assessments of EPAs.⁹ Orthopaedic surgery, drawing on the Canadian experience in a competency-based GME training program, is also exploring different frameworks for competency-based advancement bridging UME and GME.¹⁰

Currently, the American Board of Surgery is leading a pilot project around five core EPAs in General Surgery (see box 1). Using a common framework across UME and GME allows us to identify the gaps between medical school graduates’ performance and residency program expectations, and eventually tailor focused training to address deficiencies in individuals or in institutional curricula.¹¹

The choice of assessment tools depends on the construct being assessed and the purpose of the assessment. Using Miller’s pyramid (see Figure 1) as a conceptual framework, an EPA performed in the clinical environment represents the highest level of skills, and the assessment construct is the performance of that activity in the clinical context.¹² Thus, while knowledge assessments such as multiple choice exams, and skills assessments such as Objective Structured Clinical Exams, play a key role in assessing components of entrustable behavior and determining that a trainee is ready to perform effectively in the clinical environment, workplace-based assessments (WBA) allow additional assessment in real contexts.

In workplace-based assessment, the desired performance for a given clinical skill is clearly described, along with a checklist or global rating scale with anchoring descriptors for relevant aspects of the performance, which may also include the level of autonomy or supervision required, allowing for a shared mental model between rater and trainee. The trainee is observed providing clinical care, and feedback is given comparing the observed performance to the desired performance. Examples include the mini-clinical evaluation exercise (mini-CEX) used for evaluation and management skills, objective structured assessment of technical skill (OSATS) created for specific procedures, and the System for Improving and Measuring Procedural Learning (SIMPL), used in resident operative cases.^13–15

The ultimate purpose of assessment is to make an entrustment decision about what level of supervision a trainee requires while performing a given activity. However, this high-stakes assessment requires multiple low-stakes assessments of knowledge, skill, and attitude or behavioral performance across different contexts, with feedback provided to the trainee about how the observed performance compares with the expected.¹⁶ In the UME setting, these low-stakes assessments occur across all clinical rotations and the high-stakes assessments occur at the institutional level, either from structured assessment events or from the aggregated, low-stakes data, or both. This arrangement may not provide a program director with helpful information regarding a graduate’s skills specific to the care of surgical patients; however, there is significant interest in leveraging opportunities in the fourth year of medical school to achieve specialty-specific EPAs.¹⁷

For example, students going into surgery can take a capstone course in the fourth year of medical school, such as the ACS/APDS/ASE Resident Prep Curriculum. Modules developed for this curriculum cover technical and non-technical skills, and most include an assessment tool, the results of which can be fed forward program directors. These assessments are knowledge- and skills-based, corresponding to the “knows,” “knows how,” and “shows how” levels of Miller’s pyramid, and can help inform decisions around indirect supervision. Medical schools can also incorporate specialty-specific EPAs into sub-internships, including a process for tailored training or remediation during the fourth year to achieve all expected goals prior to graduation. Restructuring the Medical Student Performance Evaluation (MSPE) is one potential mechanism to feed performance information forward. However, in this process it is important to manage medical schools’ potential conflict of interest between providing accurate information, and at the same time optimizing students’ chances of matching into a residency program. One way this objective could be approached is to provide a structured report of core and specialty-specific EPA assessments after a student matches.

On the other side of the UME to GME transition, assessment of interns’ skills can be accomplished through a “boot camp” style program held at the beginning of residency training. The content of this activity is driven by local needs and resources, and may be a “massed” experience prior to starting clinical duties, or a more distributed training and assessment over the first months of internship.^18,19 The ACS/APDS Surgery Resident Skills Curriculum, Phase 1: Core Skills, is a useful resource for instruction and assessment tools for incoming interns.²⁰ This initial period of training and assessment can also incorporate workplace-based assessments to inform entrustment decisions and direct further tailored training. These intern training programs have arisen from concerns over the variability of medical school graduates’ readiness to perform expected duties as first-day interns. As EPA-like assessments at the medical school level become more robust and reliable, GME programs may be able to focus resources on more tailored training activities.

1. Accreditation Council for Graduate Medical Education. The General Surgery Milestones Project. Accessed December 30, 2017. Available at: https://www.acgme.org/Portals/0/PDFs/Milestones/SurgeryMilestones.pdf.
2. American Board of Surgery. General Surgery Training Requirements. Accessed December 30, 2017. Available at: http://www.absurgery.org/default.jsp?certgsqe_essentials.
3. Colleges, A.A.o.M., AAMC Core Entrustable Professional Activities for Medical Students, Curriculum Developers Guide. 2014.
4. American College of Surgeons. Successfully Navigating the First Year of Surgical Residency. Accessed December 30, 2017. 
5. Carraccio, C., et al., Building a Framework of Entrustable Professional Activities, Supported by Competencies and Milestones, to Bridge the Educational Continuum. Acad Med, 2017. 92(3): p. 324-330.
6. Hauer, K.E., et al., Developing entrustable professional activities as the basis for assessment of competence in an internal medicine residency: a feasibility study. J Gen Intern Med, 2013. 28(8): p. 1110-4.
7. Deitte, L.A., et al., Entrustable Professional Activities: Ten Things Radiologists Do. Acad Radiol, 2016. 23(3): p. 374-81.
8. Wisman-Zwarter, N., et al., Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: A Delphi study. Eur J Anaesthesiol, 2016. 33(8): p. 559-67.
9. Andrews, J.S., et al., Education in Pediatrics Across the Continuum (EPAC): First Steps Toward Realizing the Dream of Competency-Based Education. Acad Med, 2017.
10. Alman, B.A., et al., The Fourth Year of Medical School: Time for Reassessment: AOA Critical Issues. J Bone Joint Surg Am, 2017. 99(13): p. e72.
11. Lindeman, B.M., B.C. Sacks, and P.A. Lipsett, Graduating Students' and Surgery Program Directors' Views of the Association of American Medical Colleges Core Entrustable Professional Activities for Entering Residency: Where are the Gaps? J Surg Educ, 2015. 72(6): p. e184-92.
12. Miller, G.E., The assessment of clinical skills/competence/performance. Acad Med, 1990. 65(9 Suppl): p. S63-7.
13. Beard, J.D., et al., Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology. Health Technol Assess, 2011. 15(1): p. i-xxi, 1-162.
14. Bohnen, J.D., et al., The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial. J Surg Educ, 2016. 73(6): p. e118-e130.
15. Kogan, J.R., E.S. Holmboe, and K.E. Hauer, Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA, 2009. 302(12): p. 1316-26.
16. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency Curriculum Developers' Guide. 2014. Accessed December 30, 2017. Available at: https://members.aamc.org/eweb/upload/core epa curriculum dev guide.pdf.
17. Reddy, S.T., et al., Alliance for clinical education perspective paper: recommendations for redesigning the "final year" of medical school. Teach Learn Med, 2014. 26(4): p. 420-7.
18. Gollehon, N.S., et al., Assessing Residents' Competency at Baseline: How Much Does the Medical School Matter? J Grad Med Educ, 2017. 9(5): p. 616-621.
19. Fernandez, G.L., et al., Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship. J Surg Educ, 2012. 69(2): p. 242-8.
20. American College of Surgeons. ACS/APDS Surgery Resident Skills Curriculum. Accessed December 30, 2017. 

Kimberly M. Brown, MD, FACS, is an associate professor of surgery at the University of Texas at Austin Dell Medical School and serves as associate chair of education for the Department of Surgery and perioperative care.

Emil R. Petrusa, PhD, is an associate professor of surgery at Massachusetts General Hospital.