The education passport: connecting programmatic assessment across learning and practice

Eric J Warm; Carol  Carraccio; Matthew  Kelleher; Benjamin  Kinnear; Daniel J Schumacher; Sally Santen

doi:10.36834/cmej.73871

Authors

Eric J Warm University of Cincinnati https://orcid.org/0000-0002-6088-2434
Carol Carraccio University of Cincinnati https://orcid.org/0000-0001-5473-8914
Matthew Kelleher University of Cincinnati
Benjamin Kinnear University of Cincinnati https://orcid.org/0000-0003-0052-4130
Daniel J Schumacher Cincinnati Children's Hospital Medical Center https://orcid.org/0000-0002-3747-2410
Sally Santen University of Cincinnati https://orcid.org/0000-0002-8327-8002

DOI:

https://doi.org/10.36834/cmej.73871

Abstract

Competency-based medical education (CBME) shifts us from static assessment of learning to developmental assessment for learning. However, implementation challenges associated with CBME remain a major hurdle, especially after training and into practice. The full benefit of developmental assessment for learning over time requires collaboration, cooperation, and trust among learners, regulators, and the public that transcends each individual phase. The authors introduce the concept of an “Education Passport” that provides evidence of readiness to travel across the boundaries between undergraduate medical education, graduate medical education, and the expanse of practice.

The Education Passport uses programmatic assessment, a process of collecting numerous low stakes assessments from multiple sources over time, judging these data using criterion-referencing, and enhancing this with coaching and competency committees to understand, process, and accelerate growth without end. Information in the Passport is housed on a cloud-based server controlled by the student/physician over the course of training and practice. These data are mapped to various educational frameworks such Entrustable Professional Activities or milestones for ease of longitudinal performance tracking. At each stage of education and practice the student/physician grants Passport access to all entities that can provide data on performance. Database managers use learning analytics to connect and display information over time that are then used by the student/physician, their assigned or chosen coaches, and review committees to maintain or improve performance. Global information is also collected and analyzed to improve the entire system of learning and care.

Developing a true continuum that embraces performance and growth will be a long-term adaptive challenge across many organizations and jurisdictions and will require coordination from regulatory and national agencies. An Education Passport could also serve as an organizing tool and will require research and high-value communication strategies to maximize public trust in the work.

Metrics

Metrics Loading ...

Author Biographies

Matthew Kelleher, University of Cincinnati

Matthew Kelleher MD, MEd is Assistant Professor of Medicine and Pediatrics, and Associate Program Director, Department of Internal Medicine, University of Cincinnati, Cincinnati, Ohio

Benjamin Kinnear, University of Cincinnati

Benjamin. Kinnear MD, MEd is associate professor of medicine and pediatrics, and associate program director, Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio. ORCID

Daniel J Schumacher, Cincinnati Children's Hospital Medical Center

Daniel J. Schumacher MD, PhD, is associate professor of pediatrics, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center and the University of Cincinnati College of Medicine, Cincinnati, Ohio

Sally Santen, University of Cincinnati

Sally Santen MD, PhD, Senior Associate Dean, Virginia Commonwealth University, and Professor of Emergency Medicine and Medical Education, University of Cincinnati College of Medicine, Cincinnati Ohio

References

McGaghie WC, Miller GE, Sajid AW, Telder TV. Competency-based curriculum development on medical education: an introduction. Public Health Pap. 1978;(68):11-91.

Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638-45. https://doi.org/10.3109/0142159X.2010.501190

van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. 12 Tips for programmatic assessment. Med Teach. Nov 2014:1-6. https://doi.org/10.3109/0142159X.2014.973388

Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med teach. 2007;29(7):642-647. https://doi.org/10.1080/01421590701746983

Edgar L, Roberts S, Holmboe E. Milestones 2.0: a step forward. JGME. 2018;10(3):367. https://doi.org/10.4300/JGME-D-18-00372.1

Englander R, Flynn T, Call S, et al. Toward defining the foundation of the md degree: core Entrustable Professional Activities for entering residency. Acad Med. Oct 2016;91(10):1352-1358. https://doi.org/10.1097/acm.0000000000001204

Coalition for Physician Accountability. Initial summary report and preliminary recommendations of the undergraduate medical education to graduate medical education review committee (UGRC). https://physicianaccountability.org/wp-content/uploads/2021/04/UGRC-Initial-Summary-Report-and-Preliminary-Recommendations-1.pdf [Accessed May 15, 2021].

Santen SA, Foster KW, Hemphill RR, Christner J, Mejicano G. Simplifying the interconnected alphabet soup of medical education. Acad Med. 2022;97(5)

Gingerich A, Sebok-Syer SS, Larstone R, Watling CJ, Lingard L. Seeing but not believing: insights into the intractability of failure to fail. Med Educ. Dec 2020;54(12):1148-1158. https://doi.org/10.1111/medu.14271

Papadakis MA, Teherani A, Banach MA, et al. Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med. Dec 2005;353(25):2673-82. https://doi.org/10.1056/NEJMsa052596

Papadakis MA, Arnold GK, Blank LL, Holmboe ES, Lipner RS. Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards. Ann Intern Med. Jun 2008;148(11):869-76. https://doi.org/10.7326/0003-4819-148-11-200806030-00009

Jonker G, Ochtman A, Marty AP, Kalkman CJ, Ten Cate O, Hoff RG. Would you trust your loved ones to this trainee? Certification decisions in postgraduate anaesthesia training. Br J Anaesth. Nov 2020;125(5):e408-e410. https://doi.org/10.1016/j.bja.2020.07.009

Dweck CS. Mindset: The new psychology of success. Random House Digital, Inc; 2008.

Lucey CR, Hauer KE, Boatright D, Fernandez A. Medical education's wicked problem: achieving equity in assessment for medical learners. Acad Med. Dec 2020;95(12S Addressing harmful bias and eliminating discrimination in health professions learning environments):S98-s108. https://doi.org/10.1097/acm.0000000000003717

Low D, Pollack SW, Liao ZC, et al. Racial/ethnic disparities in clinical grading in medical school. Teach Learn Med. Oct-Dec 2019;31(5):487-496. https://doi.org/10.1080/10401334.2019.1597724

Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15(4):270-92. https://doi.org/10.1207/S15328015TLM1504_11

Warm EJ, Kinnear B, Pereira A, Hirsh DA. The residency match: escaping the prisoner's dilemma. J Grad Med Educ. Oct 2021;13(5):616-625. https://doi.org/10.4300/jgme-d-21-00477.1

Mendu ML, Kachalia A, Eappen S. Revisiting US news & world report's hospital rankings-moving beyond mortality to metrics that improve care. J Gen Intern Med. 2021:209-210. vol. 1. https://doi.org/10.1007/s11606-020-06002-x

Ried LD, Ried DB. Predicting the 2016 US News & world report rankings using a reputation and prestige model. Curr Pharm Teach Learn. Feb 2021;13(2):91-101. https://doi.org/10.1016/j.cptl.2020.09.011

Newsweek. Best Hospitals 2021 - Canada. 2021. https://www.newsweek.com/best-hospitals-2021/canada [Accessed May 9, 2021].

Fang DZ, Young CB, Golshan S, Moutier C, Zisook S. Burnout in premedical undergraduate students. Acad Psych. 2012;36(1):11-16.

Ishak W, Nikravesh R, Lederer S, Perry R, Ogunyemi D, Bernstein C. Burnout in medical students: a systematic review. Clin Teach. Aug 2013;10(4):242-5. https://doi.org/10.1111/tct.12014

Prins JT, Gazendam-Donofrio SM, Tubben BJ, van der Heijden FM, van de Wiel HB, Hoekstra-Weebers JE. Burnout in medical residents: a review. Med Educ. Aug 2007;41(8):788-800. https://doi.org/10.1111/j.1365-2923.2007.02797.x

West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med. Jun 2018;283(6):516-529. https://doi.org/10.1111/joim.12752

Bynum WEt, Artino AR, Jr., Uijtdehaage S, Webb AMB, Varpio L. Sentinel emotional events: the nature, triggers, and effects of shame experiences in medical residents. Acad Med. Jan 2019;94(1):85-93. https://doi.org/10.1097/acm.0000000000002479

Bynum WEt, Varpio L, Lagoo J, Teunissen PW. 'I'm unworthy of being in this space': the origins of shame in medical students. Med Educ. Feb 2021;55(2):185-197. https://doi.org/10.1111/medu.14354

Albuquerque J, Tulk S. Physician suicide. Cmaj. May 6 2019;191(18):E505. https://doi.org/10.1503/cmaj.181687

Schneider EC, Sarnak DO, Squires D, Shah A, Doty MM. Mirror Mirror 2017. Commonwealth Fund New York; 2017.

McDonald FS, Duhigg LM, Arnold GK, Hafer RM, Lipner RS. The American board of internal medicine maintenance of certification examination and state medical board disciplinary actions: a population cohort study. J Gen Intern Med. Aug 2018;33(8):1292-1298. https://doi.org/10.1007/s11606-018-4376-z

Naji L, Singh B, Shah A, et al. Global prevalence of burnout among postgraduate medical trainees: a systematic review and meta-regression. CMAJ Open. Jan-Mar 2021;9(1):E189-e200. https://doi.org/10.9778/cmajo.20200068

Dreyfus SE. The five-stage model of adult skill acquisition. Bulletin of science, technology & society. 2004;24(3):177-181. https://doi.org/10.1177/0270467604264992

Dunning D. The Dunning–Kruger effect: On being ignorant of one's own ignorance. Advances exper soc psychol. Elsevier; 2011:247-296. https://doi.org/10.1016/B978-0-12-385522-0.00005-6

Cutrer W, Pusic M, Gruppen LD, Hammoud MM, Santen SA. The Master Adaptive Learner. Elsevier Health Sciences; 2019.

Cutrer WB, Miller B, Pusic MV, et al. Fostering the development of master adaptive learners: a conceptual model to guide skill acquisition in medical education. Acad Med. Jan 2017;92(1):70-75. https://doi.org/10.1097/acm.0000000000001323

Andrews JS, Bale JF, Jr., Soep JB, et al. Education in pediatrics across the continuum (EPAC): first steps toward realizing the dream of competency-based education. Acad Med. Mar 2018;93(3):414-420. https://doi.org/10.1097/acm.0000000000002020

Warm EJ, Held JD, Hellmann M, et al. entrusting observable practice activities and milestones over the 36 months of an internal medicine residency. Acad Med. Oct 2016;91(10):1398-1405. https://doi.org/10.1097/acm.0000000000001292

Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment. Acad Med. Aug 2015;90(8):1084-92. https://doi.org/10.1097/acm.0000000000000736

Hauer KE, Edgar L, Hogan SO, Kinnear B, Warm E. The science of effective group process: lessons for clinical competency committees. J Grad Med Educ. Apr 2021;13(2 Suppl):59-64. https://doi.org/10.4300/jgme-d-20-00827.1

Thoma B, Warm E, Hamstra SJ, et al. Next steps in the implementation of learning analytics in medical education: consensus from an international cohort of medical educators. J Grad Med Educ. Jun 2020;12(3):303-311. https://doi.org/10.4300/jgme-d-19-00493.1

Warm EJ, Mathis BR, Held JD, et al. Entrustment and mapping of observable practice activities for resident assessment. J Gen Intern Med. Aug 2014;29(8):1177-82. https://doi.org/10.1007/s11606-014-2801-5

Schauer DP, Kinnear B, Kelleher M, Sall D, Schumacher DJ, Warm EJ. Developing the expected entrustment score: accounting for variation in resident assessment. J Gen Intern Med. Apr 4 2022; https://doi.org/10.1007/s11606-022-07492-7

Nousiainen MT, Mironova P, Hynes M, et al. Eight-year outcomes of a competency-based residency training program in orthopedic surgery. Med teach. 2018;40(10):1042-1054. https://doi.org/10.1080/0142159X.2017.1421751

Chan T, Sebok-Syer S, Thoma B, Wise A, Sherbino J, Pusic M. Learning analytics in medical education assessment: the past, the present, and the future. AEM Educ Train. Apr 2018;2(2):178-187. https://doi.org/10.1002/aet2.10087

Thoma B, Caretta-Weyer H, Schumacher DJ, et al. Becoming a deliberately developmental organization: using competency based assessment data for organizational development. Med Teach. May 25 2021:1-9. https://doi.org/10.1080/0142159x.2021.1925100

Ten Cate O, Graafmans L, Posthumus I, Welink L, van Dijk M. The EPA-based Utrecht undergraduate clinical curriculum: development and implementation. Med Teach. May 2018;40(5):506-513. https://doi.org/10.1080/0142159x.2018.1435856

Choe JH, Knight CL, Stiling R, Corning K, Lock K, Steinberg KP. Shortening the miles to the milestones: connecting epa-based evaluations to ACGME milestone reports for internal medicine residency programs. Acad Med. Jul 2016;91(7):943-50. https://doi.org/10.1097/acm.0000000000001161

Wagner LM, Dolansky MA, Englander R. Entrustable professional activities for quality and patient safety. Nurs Outlook. May - Jun 2018;66(3):237-243. https://doi.org/10.1016/j.outlook.2017.11.001

Kelleher M, Kinnear B, Wong SEP, O'Toole J, Warm E. Linking workplace-based assessment to ACGME milestones: a comparison of mapping strategies in two specialties. Teach Learn Med. Apr-May 2020;32(2):194-203. https://doi.org/10.1080/10401334.2019.1653764

Gerstle CR. Parallels in safety between aviation and healthcare. J Pediatr Surg. May 2018;53(5):875-878. https://doi.org/10.1016/j.jpedsurg.2018.02.002

Borrelli RA. Reflections on developing an identity for the third generation nuclear engineer in the post-Fukushima society. Reflections on the Fukushima Daiichi Nuclear Accident. Springer, Cham; 2015:353-365. https://doi.org/10.1007/978-3-319-12090-4_19

Jonker CM, Riemsdijk M, Vermeulen B. Shared mental models. Springer; 132-151. https://doi.org/10.1007/978-3-642-21268-0_8

Warm EJ, Englander R, Pereira A, Barach P. Improving Learner Handovers in Medical Education. Acad Med. Jul 2017;92(7):927-931. https://doi.org/10.1097/acm.0000000000001457

Eva KW, Regehr G. "I'll never play professional football" and other fallacies of self-assessment. J Contin Educ Health Prof. Winter 2008;28(1):14-9. https://doi.org/10.1002/chp.150

Ikizler TA, Berns JS. Transformation of ABIM and What the Changes Mean to Nephrologists. Clin J Am Soc Nephrol.2018;13(1):164-166. https://doi.org/10.2215/CJN.05700517

Teirstein PS. Boarded to death--why maintenance of certification is bad for doctors and patients. N Engl J Med. Jan 8 2015;372(2):106-8. https://doi.org/10.1056/NEJMp1407422

Boyd VA, Whitehead CR, Thille P, Ginsburg S, Brydges R, Kuper A. Competency-based medical education: the discourse of infallibility. Med Educ. Jan 2018;52(1):45-57. https://doi.org/10.1111/medu.13467

Murray KE, Lane JL, Carraccio C, et al. Crossing the gap: using competency-based assessment to determine whether learners are ready for the undergraduate-to-graduate transition. Acad Med. Mar 2019;94(3):338-345. https://doi.org/10.1097/acm.0000000000002535

Heath JK, Wang T, Santhosh L, et al. Longitudinal milestone assessment extending through subspecialty training: the relationship between acgme internal medicine residency milestones and subsequent pulmonary and critical care fellowship milestones. Acad Med. May 18 2021; https://doi.org/10.1097/acm.0000000000004165

Heifetz RA, Laurie DL. The work of leadership. Harvard business review. 1997;75:124-134.

Heifetz R, Linsky M. Leadership on the line, with a new preface: Staying alive through the dangers of change. Harvard Business Press; 2017.

Groupsmith. Technical Problems vs. Adaptive Challenges. 2021. https://www.sgaumc.org/files/files_library/technical_vs_adaptive_challenges.pdf [Accessed May 9, 2021].