Picking the Right Residents: Learn the Do's and Don'ts of Winning the Match Game

A number of methods are used to screen and select the best candidates for surgical residency, with no clear data supporting any one method. At the 2011 American College of Surgeons Clinical Congress in San Francisco, CA, a panel session on choosing residents entitled, “Picking the Right Residents: Do’s and Don’ts of Winning the Match Game,” was presented. Panelists representing general surgery, otolaryngology, and surgical education shared their ideas on residency selection. These panelists included moderator Rebecca C. Britt, MD, FACS; co-moderator Anna H. Messner, MD; Paul J. Schenarts, MD, FACS; Joseph B. Cofer, MD, FACS; and Kimberly Dean Schenarts, PhD.

The beginning of the busy interview and match season is an apt time to share the ideas presented at this panel.

In 2001, all residency applications became Electronic Residency Application Service (ERAS) based, resulting in a significant increase in the number of applications per program. In a recent survey of program directors in general surgery,¹ the United States Medical Licensing Exam (USMLE) Step 1 exam is the single most important factor in screening interview applicants (37%), followed by the USMLE Step 2 exam (24%). The most important factor given for actual resident selection on a 5-point Likert scale was the interview (4.69), followed by the USMLE Step 1 score (4.21) and letters of recommendation (4.02). The NRMP Program Director Survey results for 2010² show that for both otolaryngology and general surgery, the most important factors in selecting applicants for interview include USMLE Step 1 scores, letters of recommendation, and grades in clerkship in desired specialty. The most important factors for both specialties in ranking applicants include interactions with both the faculty and residents at the interview, evidence of professionalism at the interview, letters of recommendation in the specialty, USMLE Step 1 scores, and grades in the desired specialty.

The USMLE is designed to provide a common evaluation system for applicants for medical licensure but is often cited as an important factor for residency selection. A recent study to assess the correlation of USMLE scores with clinical skills, including central venous line insertion, temporary dialysis catheter insertion, and thoracentesis, found no correlation with clinical performance.³ The correlation of USMLE scores with cognitive skills such as the in-service training exam remains uncertain. In most research studies, the USMLE score has also not correlated with residency attrition.

Letters of recommendation are commonly cited as an important factor in residency selection, yet the large majority of letters “recommend” the applicant. The importance of writing a descriptive letter to highlight unique resident features and co-compare to peers, as well as using stand-out adjectives, was discussed at the panel session. The dean’s letter is highly variable in content and likely not predictive of future performance, although it may correlate with work ethic. A recent study³ evaluating letters of recommendation for Accreditation Council for Graduate Medical Education (ACGME) competency-based themes found that residents who were ranked by faculty as “most successful” had significantly more comments regarding excellence in patient care, medical knowledge, and interpersonal and communication skills in their letters. Several specialties, including otolaryngology and emergency medicine, are considering transitioning to a standardized letter format.

Despite the extensive application and residency screening processes, attrition from surgery programs remains high at approximately 20 percent, with many residents leaving for nonsurgical programs. Multiple studies have looked at the predictors of attrition based on residency application data, with no clear answers. The University of Texas Southwestern Medical Center⁴ found that academic variables were not associated significantly with attrition, with univariate analysis showing that predictors of attrition include age greater than 29 years at program entry, female sex, courses repeated, “C” grade on the transcript, lack of participation in team sports, and lack of superlative comments in the dean’s letter. The multivariate models showed age greater than 29 years, the dean’s letter, lack of participation in team sports, and merit scholarship in medical school all were significantly associated with attrition. In contrast, a recent study from the University of Tennessee at Knoxville⁵ found that USLME Step 1 scores, performance on the interview, and high-performance accomplishments outside of the medical field such as performing arts and college athletics were predictive of the successful completion of surgery residency. There was no correlation with attrition in this study for female sex, medical school grades, and Alpha Omega Alpha Honor Society status.

A recent national study⁶ looking at attrition for all categorical general surgery residents on the 2007–2008 resident rosters found a cumulative attrition rate of 19.5 percent, with the highest attrition in the PGY-1 year (5.9%), PGY-2 year (4.3%), and research years (3.9%). In multivariate analysis, the only predictor for attrition was postgraduate year. There was no correlation of attrition with any other programmatic or demographic characteristics. The residents most often pursued nonsurgical specialties, with anesthesia and radiology the two most common. As this study and others have shown, the 80-hour work week does not appear to have made an impact on the rate of attrition in general surgery.

The interview day is an opportunity to monitor the interactions of the candidates with both the faculty and residents and may provide insight into the future success of the candidate. The University of Pennsylvania recently assessed their residency selection strategy⁷ in response to their high attrition rate and worked with an organizational management expert to analyze the traditional selection criteria and design a modified screening and interview format. Modifications included asking the candidates to write a 500-word essay; a reduction in the number of faculty participants; and creation of a personalized, scripted interview. As a result, the program saw a reduction in attrition from an overall five-year rate of 27.3 percent to 3.2 percent. Additionally, the faculty evaluations improved for the residents selected with the new method, implying improved resident performance. The panel discussed novel ways to improve the interview process, including focusing on nonacademic accomplishments and performing multiple mini-interviews, where the interviewer is blinded to the candidates file. The importance of assessing the candidate interactions with the residents was also highlighted. The utility of assessing technical skills of the candidates was discussed, although no one on the panel felt this step was necessary and no clear data supports the use of it. While there are no clear answers on how to choose the best residents, programs must continue to refine their methods to ensure the production of successful trainees to maintain the surgical workforce.

Special thanks to the panelists: Anna Messner, MD, co-moderator; Paul J. Schenarts MD, FACS; Joseph Cofer MD, FACS; Kimberly Schenarts. PhD

1. Makdisi G, Takeuchi T, Rodriguez J, Wise L. How we select our residents—A survey of selection criteria in general surgery residents. J Surg Educ. 2011;68:67-72.
2. National Resident Matching Program. Results of the 2010 NRMP Program Director Survey. Available at: http://www.nrmp.org/data/programresultsbyspecialty2010v3.pdf. Accessed February 2, 2012.
3. Stohl H, Hueppchen N, Bienstock J. The utility of letters of recommendation in predicting resident success: Can the ACGME competencies help? J Grad Med Educ. 2011;3:387-390.
4. Naylor R, Reisch J, Valentine J. Factors related to attrition in surgery residency based on application data. Arch Surg. 2008;143:647-652.
5. Alterman D, Jones T, Heidel R, Daley B, Goldman M. The predictive value of general surgery application data for future resident performance. J Surg Educ. 2011;68:513-518.
6. Yeo H, Bucholz, E, Sosa JA, Curry L, Lewis FR Jr., Jones AT, Viola K, Lin Z, Bell RH Jr. A national study of attrition in general surgery training. Which residents leave and where do they go? Ann Surg. 2010;252:529-536.
7. Kelz R, Mullen J, Kaiser L, Pray LA, Shea GP, Drebin JA, Wirtalla CJ, Morris JB. Prevention of surgical resident attrition by a novel selection strategy. Ann Surg. 2010;252:537-543.