Academic Medicine

Using Longitudinal Milestones Data and Learning Analytics to Facilitate the Professional Development of Residents: Early Lessons from Three Specialties Purpose: To investigate the effectiveness of using national, longitudinal milestones data to provide formative assessments to identify residents at risk of not achieving recommended competency milestone goals by residency completion. The investigators hypothesized that specific, lower milestone ratings at earlier time points in residency would be predictive of not achieving recommended Level (L) 4 milestones by graduation. Method: In 2018, the investigators conducted a longitudinal cohort study of emergency medicine (EM), family medicine (FM), and internal medicine (IM) residents who completed their residency programs from 2015 to 2018. They calculated predictive values (PVs) and odds ratios (ORs), adjusting for nesting within programs, for specific milestone rating thresholds at 6-month intervals for all subcompetencies within each specialty. They used final milestone ratings (May/June 2018) as the outcome variables, setting L4 as the ideal educational outcome. Results: The investigators included 1,386 (98.9%) EM residents, 3,276 (98.0%) FM residents, and 7,399 (98.0%) IM residents in their analysis. The percentage of residents not reaching Level 4 by graduation ranged from 11-31% in EM, 16-53% in FM, and 5-15% in IM. Using a milestone rating of Level 2.5 or lower at the end of PGY2, the predictive probability of not attaining the L4 milestone graduation goal ranged from 32-56% in EM, 32-67% in FM, and 15-36% in IM. Conclusions: Longitudinal milestones ratings may provide educationally useful, predictive information to help individual residents address potential competency gaps, but the predictive power of the milestones ratings varies by specialty and subcompetency within these three adult care specialties. Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A716. Funding/Support: The authors acknowledge the support of the Accreditation Council for Graduate Medical Education. Other disclosures: The authors are employees of the Accreditation Council for Graduate Medical Education. Dr. Holmboe receives royalties from Elsevier for a textbook on assessment. Ethical approval: The American Institutes for Research reviewed the research protocol reported herein and deemed it exempt. Previous presentations: Portions of this research were presented at the Association of Medical Education in Europe (AMEE), August 2018, Basel, Switzerland; and at the International Conference on Residency Education (ICRE), October 2018, Halifax, Nova Scotia, Canada. Data: No external data outside of the Accreditation Council for Graduate Medical Education was used for this study. Correspondence should be addressed to Eric S. Holmboe, ACGME, 401 North Michigan Avenue, Chicago, IL 60611; telephone (312) 755-7034; email: eholmboe@acgme.org; Twitter: @acgme. This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. © 2019 by the Association of American Medical Colleges

“A Friendly Place to Grow as an Educator”: A Qualitative Study of Community and Relationships Among Medical Student Coaches Purpose: The rise of coaching programs in medical education sparks questions about ways to support physician coaches in learning new educational practices specific to coaching. How coaches learn from one another is of particular interest considering the potential value of social learning. Using communities of practice (CoP) as a conceptual framework, the authors examine the sense of community and relationships among coaches in a new medical student coaching program, the value of this community, and the facilitators and barriers influencing community development. Method: In this qualitative study, investigators conducted 34 interviews with physician coaches at one institution over two years (2017-2018) and observed 36 coach meetings. Investigators analyzed interview transcripts using thematic analysis and utilized observation field notes for context and refinement of themes. Results: Coaches described a sense of community based on regular interactions, shared commitment to medical education, and new roles with similar experiences, joys, and challenges. They valued the sense of camaraderie and support, learning from one another, and opportunities for professional growth that strengthened their identities as educators and enhanced job satisfaction. Facilitators of community included regular meetings, leadership and administrative support, and informal opportunities to interact outside of meetings. Barriers included time constraints and geographic challenges for coaches at off-site locations. Conclusions: The sense of community among coaches was a valued and beneficial part of their coaching experience. Coaches’ interactions and relationships promoted skill acquisition, knowledge transfer, professional development, and career satisfaction. Thus, incorporating support for social learning in coaching programs promotes coach faculty development. Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A717. Acknowledgments: The authors wish to thank the coaching program staff, who helped notify us of meeting times for observations, and to coaches who participated in this study. Funding/Support: This research was supported by a Harnisch Grant from the Institute of Coaching. Other disclosures: None reported. Ethical approval: This study was reviewed by the institutional review board at the University of California, San Francisco and deemed exempt (July 29, 2016, IRB # 16-20142). Previous presentations: Preliminary findings from this study were presented as oral presentations at the Western Group on Educational Affairs Conference, Denver, CO, March 2018, and at the Institute of Coaching’s “Coaching in Leadership and Healthcare” Conference, Boston, MA, September 2018. Correspondence should be addressed to Leslie Sheu, University of California, San Francisco, Department of Medicine, 1545 Divisadero St., Box 0320, San Francisco, CA, 94143-0710; telephone: (415) 514-8642; e-mail: leslie.sheu@ucsf.edu; Twitter: @lesliesheu. © 2019 by the Association of American Medical Colleges

The Most Valuable Resource Is Time: Insights from a Novel National Program to Improve Retention of Physician-Scientists with Caregiving Responsibilities Purpose: To enhance understanding of challenges related to work-life integration in academic medicine and to inform the ongoing implementation of an existing program and the development of other interventions to promote success of physician-scientists. Method: This study is part of a prospective analysis of the effects of the Fund to Retain Clinical Scientists (FRCS), a national program launched by the Doris Duke Charitable Foundation at ten U.S. institutions, which provides financial support to physician-scientists facing caregiving challenges. In early 2018, 28 of 33 program awardees participated in semi-structured interviews. Questions were about challenges faced by physician-scientists as caregivers and their early perceptions of the FRCS. Multiple analysts reviewed de-identified transcripts, iteratively revised the coding scheme, and interpreted the data using qualitative thematic analysis. Results: Participants’ rich descriptions illuminated five interconnected themes: (1) Time is a critical and limited resource, (2) timing is key, (3) limited time resources and timing conflicts may have a particularly adverse effect on women’s careers, (4) flexible funds enable reclamation and repurposing of time resources, and (5) FRCS leaders should be cognizant of time and timing conflicts when developing program-related offerings Conclusions: Programs such as the FRCS are instrumental in supporting individuals to delegate time-consuming tasks and control how they spend their valuable time. Qualitative analysis suggests that access to and command of valuable time resources is crucial to career advancement, research productivity, and work-life flexibility, especially during critical time points along the physician-scientist trajectory. Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A718. Acknowledgments: The authors wish to thank the recipients of the Fund to Retain Clinical Scientists award who took the time to participate in this study. Funding/Support: This work was funded by the Doris Duke Charitable Foundation. The funders played no role in the analysis or interpretation of the results of this investigation or decision to submit for publication. Other disclosures: None reported. Ethical approval: This study was approved by the University of Michigan Institutional Review Board. Data: The data reported here are from outside sources and do not require permission for use. Correspondence should be addressed to Reshma Jagsi, Department of Radiation Oncology, University of Michigan, UHB2C490, SPC 5010, 1500 East Medical Center Drive, Ann Arbor, MI 48109-5010; telephone (734) 936-8700; email: rjagsi@med.umich.edu; Twitter: @reshmajagsi. © 2019 by the Association of American Medical Colleges

Broadening Medical Students’ Exposure to the Range of Illness Experiences: A Pilot Curriculum Focused on Depression Education Problem: Exposing medical students to a broad range of illness experiences is crucial for teaching them to practice patient-centered care, but students often have limited interaction with patients with diverse illness presentations. Approach: The authors developed, implemented, and evaluated a self-directed online curriculum followed by a small group discussion focused on depression education. The curriculum was based on a module created using the Database of Individual Patients’ Experiences methodology. Findings from 40 interviews with young adults across the United States about their diverse experiences with depression were summarized online, and the summaries were illustrated by video, audio, and text clips. From August 2016 to April 2017, third-year students completed either this online curriculum and the usual clerkship curriculum or just the usual clerkship curriculum. These intervention and control groups completed pre and post surveys. Outcomes: Students in the intervention group reported the online curriculum influenced their thinking about depression (51/56) nearly as often as they reported seeing patients in clinic did (53/56). They also reported greater decreases in personal stigmatizing attitudes towards depression than students in the control group as measured by the Depression Stigma Scale (5.75 to 4.02 intervention; 6.50 to 5.65 control; P = .004). In open-ended responses, students in the intervention group were 13 times more likely to describe key lessons from the curriculum that reflected patient heterogeneity. Next Steps: Future collaborations include implementing and evaluating this curriculum at other medical schools and developing additional versions based on other illness experiences. Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A714. Acknowledgements: The authors wish to thank the students involved in this intervention for their thoughtful participation and Madison Crowder for her editorial assistance. Funding/Support: This project was supported by the Josiah Macy Jr. Foundation, the University of Wisconsin School of Medicine and Public Health, the Wisconsin Partnership Program, the University of Wisconsin Medical Foundation, the University of Wisconsin Institute for Clinical and Translational Research which is supported by the Clinical and Translational Science Award program, the National Center for Advancing Translational Sciences (grant 1UL1TR002373), the Center for Patient Partnerships, the University of Wisconsin Department of Family Medicine and Community Health, and the Health Innovation Program. Other disclosures: None reported. Ethical approval: The University of Wisconsin-Madison institutional review board deemed this study exempt from review. Previous presentations: A portion of these findings were presented in a poster at the North American Primary Care Research Group Annual Meeting in Montreal, Quebec, Canada in November 2017. Correspondence should be addressed to Nancy Pandhi, Department of Family and Community Medicine, University of New Mexico School of Medicine, MSC 09 5040, 1 University of New Mexico, Albuquerque, NM 87131; telephone: (505) 272-0510; email: npandhi@salud.unm.edu. © 2019 by the Association of American Medical Colleges

Uncertainty in Decision-Making in Medicine: A Scoping Review and Thematic Analysis of Conceptual Models Purpose: The practice of medicine is rarely straightforward. Data used to facilitate medical decision-making may be conflicting, ambiguous, or scarce, and providing optimal care requires balancing clinicians’ expertise and available evidence with patients’ preferences. To explore uncertainty in decision-making across disciplines, the authors performed a scoping review and thematic analysis of the literature to formulate a model describing the decision-making process in medicine under uncertain conditions. Method: In 2016, the authors performed a comprehensive search of key databases using a combination of keywords and controlled vocabulary. They identified and reviewed 3,398 records. After applying their inclusion and exclusion criteria to the titles and abstracts then full texts, 19 articles were selected. The authors applied a qualitative thematic analysis to these articles, using codes to extract themes related to uncertainty in decision-making. Results: The 19 articles spanned 6 fields of study and 5 disciplines within the health sciences. The thematic analysis revealed 6 main themes: recognition of uncertainty, classification of uncertainty, stakeholder perspectives, knowledge acquisition, decision-making approach, and evaluation of the decision-making process. Conclusions: Based on the themes that emerged from their thematic analysis of the literature characterizing the effects of uncertainty and ambiguity on the decision-making process, the authors developed a framework depicting the interplay between these themes with a visual representation of the decision-making process under uncertain conditions. Future research includes further development and validation of this framework to inform medical school curricula. Acknowledgements: The authors wish to thank the mentors and faculty of the Educational Scholars Program of the Academic Pediatric Association for providing valuable feedback and instruction used to prepare this article. The authors also would like to acknowledge Dr. Sally Santen, who provided useful suggestions on all article drafts. Funding/Support: This work was partially supported by funding to Deborah DiazGranados from the National Institutes of Health National Center for Advancing Translational Science (UL1TR002649). Other disclosures: None reported. Ethical approval: Reported as not applicable. Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the views of the organizations with which they are affiliated or their sponsoring associations and agencies. Correspondence should be addressed to Marieka A. Helou, 1000 East Broad Street, Children’s Pavilion Building 5-1035, Box 980121, Richmond, VA 23298; telephone: (804) 828-9605; email: marieka.helou@vcuhealth.org. © 2019 by the Association of American Medical Colleges

Trends in Tenure Status in Academic Family Medicine, 1977–2017: Implications for Recruitment, Retention, and the Academic Mission Purpose: Tenure status has important implications for medical school faculty recruitment and retention and may affect educational quality, academic freedom, and collegiality. However, tenure trends in academic family medicine are unknown. This study aimed to describe trends in tenure status of family medicine faculty overall and by gender and status of minorities underrepresented in medicine (URM) in Liaison Committee on Medical Education-accredited medical schools. Method: This study used Association of American Medical Colleges Faculty Roster data to describe trends in tenure status of fulltime family medicine faculty from 1977 to 2017. Bivariate and trend analyses were conducted to assess associations and describe patterns between tenure status and faculty gender, race, and ethnicity. Interdepartmental variations in tenure trends over the years were also examined. Results: Among family medicine faculty, the proportions of faculty tenured or on a tenure track dropped more than three-fold from 1977 (46.6%; n=507/1,089) to 2017 (12.7%; n=729/5,752). Lower proportions of women and URM faculty were tenured or on a tenure track than male and non-URM faculty, respectively. But the gaps among them were converging. Compared to other clinical departments, family medicine had the highest proportion of faculty (74.6%, n = 4,291/5,752) not on a tenure track in 2017. Conclusions: Proportion of tenure positions significantly decreased among family medicine faculty in U.S. medical schools. While gaps between male and female faculty and among certain racial/ethnic groups remained for family medicine tenure status, they have decreased over time, mainly due to substantial increase of nontenured positions. Acknowledgments: The authors wish to thank anonymous reviewers and editors for their comments and suggestions for the early versions of the manuscript. Funding/Support: None reported. Ethical approval: As this study did not involve human subjects and was an educational research study, it was exempt from ethical review by the University of Texas Southwestern Medical Center Institutional Review Board. Other disclosures: The authors have permission from the Association of American Medical Colleges to use Faculty Roster data. Correspondence should be addressed to Imam M. Xierali, University of Texas Southwestern Medical Center, Department of Family and Community Medicine, 5959 Harry Hines Blvd., Suite 720, Dallas, TX 75390-9194; telephone: (214) 648-1362; email: imam.xierali@utsouthwestern.edu. © 2019 by the Association of American Medical Colleges

Developing and Implementing Curricular Objectives for Sexual Health in Undergraduate Medical Education: A Practical Approach Problem: In 2014, medical students at the Florida International University Herbert Wertheim College of Medicine (FIU HWCOM) first drew attention to perceived gaps in the sexual health curriculum. The authors used Kern et al’s model for curriculum development to review and update the existing curriculum. Approach: To develop sexual health curricular objectives for undergraduate medical education (UME), the authors reviewed existing specialty- and organization-specific objectives. Then an iterative process guided by clear criteria was used to develop feasible objectives, which were refined through stakeholder feedback. As these objectives were being finalized in 2017, UME sexual health competencies were first published; the authors mapped their objectives to these, as possible. The medical school’s course activities and assessments were mapped to the authors’ sexual health objectives to identify curricular gaps and unplanned redundancies. Outcomes: This process resulted in 12 sexual health curricular objectives that are adaptable by other institutions. In mapping the FIU HWCOM curriculum to these objectives, specific gaps and assessment weaknesses emerged. With stakeholder support obtained through a strong curricular management structure, the work of modifying content to address gaps has begun. Next Steps: New assessments specific to these sexual health objectives are needed, especially within the clinical curriculum. Assessment of these objectives may be strengthened through the development of new sexual-health-related entrustable professional activities (EPAs) nested within the broader Core EPAs for Entering Residency being piloted. When data on learning outcomes are available, the process of targeting curricular improvement will begin. Acknowledgments: The authors would like to recognize the students of Florida International University Herbert Wertheim College of Medicine for their advocacy regarding the creation of a sexual health curriculum at the medical school. Funding/Support: None reported. Other disclosures: None reported. Ethical approval: Reported as not applicable. Previous presentations: Presented at the Sex and Gender in Medical Education Summit; Salt Lake City, Utah; April 2018; and the Annual Spring Conference of the Society of Teachers of Family Medicine; Toronto, Ontario, Canada; May 2019. Correspondence should be addressed to Sarah E. Stumbar, Florida International University Herbert Wertheim College of Medicine, 11200 SW 8th Street, AHC2 584, Miami, FL 33199; telephone: 607-592-9587; e-mail: sstumbar@fiu.edu; Twitter: @FIUMedicine. © 2019 by the Association of American Medical Colleges

The Evolution of Resident Remedial Teaching at One Institution Residency training program directors and teaching faculty invest an enormous amount of time, energy, and resources in providing underperforming at-risk learners with remedial teaching. A remediation program was created at the University of Colorado School of Medicine in 2006 and centralized in 2012 that consolidated expertise in and resources for learner assessment and individualized teaching for struggling learners, particularly those placed on probation or receiving letters of warning (called focused review letters) from their residency programs. Since the implementation of the program, the authors have observed a decrease in the number of residents being placed on probation and, of those on probation, more are graduating and obtaining board certification. In this Article, the authors aim to describe the development and outcomes of the program and to explore possible reasons for the improved outcomes. Acknowledgments: The authors wish to thank Karen Potter for help collecting data. Funding/Support: The University of Colorado’s President’s Teaching Scholars Program provided funding to hire a statistician and to access the Federation of State Medical Boards’ Physician Data Center. Other disclosures: None reported. Ethical approval: This was deemed a program evaluation and thus exempt by the Colorado Multiple Institutional Review Board. Correspondence should be addressed to Jeannette Guerrasio, 4700 E Hale Pkwy., Suite 300/330, Denver, CO 80220; telephone: (303) 325-6515; email: jeannette@clinicalremediation.com. © 2019 by the Association of American Medical Colleges

Outcomes of a Longitudinal Quality Improvement and Patient Safety Preclerkship Curriculum Purpose: To evaluate whether quality improvement (QI) and patient safety (PS) training in preclerkship medical education resulted in students’ development and retention of knowledge, application-based and perceived skills, and attitudes throughout clerkships. Method: A longitudinal QI/PS curriculum with multi-modal curricular components was implemented in the preclerkship curriculum between 2015 and 2017 at the Warren Alpert Medical School of Brown University, Rhode Island. Assessments were administered at baseline (T1) end of year one (T2), Clinical Skills Clerkship (T3), and end of clerkships (T4) in the intervention cohort (n = 97). In 2018 T4 data for this cohort were compared to a prior control cohort (n = 97) at T4. Results: Results of knowledge-based multiple-choice questions (MCQs) (t(134) = -1.57, P < .001) and application-based skills (t(132) = -8.91, P < .001) demonstrated significant improvement from T1 to T2 (intervention cohort). Assessments of perceived skills showed significant growth from T1 to T2 (t(137) = -23.38, P < .001). Performance on application-based skills significantly improved from T2 to T3 (t(123) = -4.11, P < .001). Compared with the control cohort, the intervention cohort had significantly higher scores on MCQs (t(187.88) = 3.98, P < .001), application-based skills (t(72.69) = 6.40,P < .001), perceived skills (t(106.99) = 5.24, P < .001), and attitudes (t(152) = 5.86, P < .001). Conclusions: Incorporation of preclerkship QI/PS training resulted in improvements in knowledge, application-based and perceived skills, and attitudes that were retained throughout clerkships. Funding/support: None reported. Other disclosures: None reported. Ethical approval: This curriculum program evaluation did not require review by the Brown University institutional review board (supporting documentation available from authors upon request). Previous presentations: • Monteiro K, George P, Dumenco L. Overview and Longitudinal Evaluation of a Quality Improvement & Patient Safety Curriculum for Medical Students. Association of American Medical Colleges Integrating Quality Conference. June 8, 2018, Arlington, Virginia. • Monteiro K, George P, Dollase R, Dumenco L. Introduction and Evaluation of a Quality Improvement and Patient Safety Curriculum for Preclinical Medical Students. The Generalists in Medical Education Conference. November 3, 2017, Boston, Massachusetts. • Monteiro K, George P, Dollase R, Dumenco L. An Innovative Quality Improvement and Patient Safety Curriculum for First Year Medical Students. Association of American Medical Colleges Integrating Quality Conference. June 7, 2017, Chicago, Illinois. Correspondence should be addressed to Luba Dumenco, Office of Medical Education, Box GM 104, The Warren Alpert Medical School of Brown University, 222 Richmond Street, Providence, RI 02903; telephone: (401) 863-3992, email: Luba_dumenco@brown.edu. © 2019 by the Association of American Medical Colleges

Substance Use Disorder Education in Medical Schools: A Scoping Review of the Literature Purpose: As medical schools adapt their curricula to prepare future physicians for the opioid crisis and for treating patients with substance use disorders (SUDs), educators should refer to courses described in the literature. This scoping review aimed to (1) provide a comprehensive evaluation and summation of peer-reviewed literature reporting on SUD education in medical schools globally, and (2) to appraise the research quality and educational outcomes reported in SUD education studies in medical schools. Method: The authors searched 6 databases (3 Ovid MEDLINE databases, Embase, ERIC, and Web of Science) from inception through May 25, 2018. Original English-language research studies focusing on medical students and describing SUD education in medical schools were included. The Medical Education Research Study Quality Instrument (MERSQI) was used to grade included studies. Results: Of 3,178 articles identified, 43 met inclusion criteria. Studies were conducted in 9 countries. Most reported on educational interventions for tobacco (n = 20; 47%); others reported on interventions for SUDs broadly (n = 15; 35%), alcohol (n = 8; 19%), and opioids (n = 1; 2%). The mean MERSQI score was 12.27 (SD 2.30). Four studies (9%) reported on educational outcomes at the level of behaviors or patient or health care outcomes. The majority (n = 39; 91%) reported significant benefits. Conclusions: Educational interventions relating to SUDs were effective in improving medical students’ knowledge, skills, and attitudes. Educators should develop courses that achieve higher-level educational outcomes, increase education on opioid use disorders, and focus on the greatest public health concerns. Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A711. Funding/Support: None reported. Other disclosures: None reported. Ethical approval: Reported as not applicable. Correspondence should be addressed to Andrew J. Muzyk, P.O. Box 3089–Pharmacy, Durham, NC 27710; telephone: 919-681-3438; e-mail: Andrew.Muzyk@duke.edu. © 2019 by the Association of American Medical Colleges