Underrepresented Applicants Post-USMLE Pass/Fail: A National Survey of Competitive Residency Directors (2024)

Associated Data

Supplementary Materials

Abstract

Keywords: underrepresented in medicine, diversity, USMLE: United States medical licensing examination, pass/fail, residency selection

Introduction

The United States Medical Licensing Examination (USMLE) recently changed their score reporting techniques for the Step 1 examination from a 3-digit number to pass/fail.^1–3 This change has resulted in uncertainty among future residency candidates, in particular those who are underrepresented in medicine (URM). The AMA definition of URM students includes “racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population”. Some believe this uncertainty, coupled with the recent US Supreme court ruling that higher education admissions are not to consider race, may place URM applicants at a considerable disadvantage.⁴

URM applicants face unique barriers when applying to residencies, as they often experience a disparity in their Step 1 scores and an overall achievement gap when compared to their non-URM peers due to systemic inequities throughout their education prior to medical school matriculation.⁵ It has been reported that the achievement gap has negative physical and psychological consequences on URM students, and after entering medical school they may internalize or externalize negative stereotypes which can impact standardized exam performance.⁶ In the past, when compared to non-URM candidates, URM residency applicants had lower mean Step 1 and Step 2CK scores.^7,8 Studies suggest that a reduction in the reliance on Step 1 scores may mitigate racial biases.⁹ Due to the weight that residency program directors (PDs) historically ascribed to Step 1 scores in choosing applicants for residency interviews, URM applicants may have been screened out of interview opportunities, a point that was emphasized by the Invitational Conference on USMLE Scoring when they recommended minimiz(ing) “racial demographic differences that exist in USMLE performance”.⁶

A few recent studies have also explored how the Step 1 scoring change may impact specific residencies’ application screening processes. In one study, about 50% of dermatology PDs believed that all medical students will have more difficulty matching into dermatology. Dermatology PDs anticipate focusing more on letters of recommendation (LOR), audition/away rotations, and Step 2CK scores.¹⁰ Another study surveyed medical student deans; when asked to rank the importance of various application factors, the most frequently selected first-choice was Step 2CK score. Collectively, medical school deans reported feeling that students who apply to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery would be most impacted.¹¹ In a survey of vascular surgery PDs, 42% of respondents reported their intent to replace the Step 1 score with the Step 2CK score.¹² Neurosurgery PDs believed similarly that Step 2CK will be of heightened importance and may effectively “replace” the role of Step 1 within the residency selection process.¹³ Similarly, a survey of plastic surgery PDs reported that LOR, Step 2 CK scores, and away rotations at their institution would be most important. Many stakeholders have expressed concerns about removing the standardized Step 1 score and posit additional issues that may arise from this scoring change.^14–16 Some have highlighted the potential impact of structural biases on URM applicants, given the existing socioeconomic inequities that lead to reduced financial and community resources among the URM population.¹⁷

A study by Makhoul et al¹⁶ reported valuable data on the perspectives of PDs from 30 specialties on the pass/fail Step 1 scoring change; overall, 77.2% of PD respondents believed that all applicants will have a more difficult time matching into their specialty. Although they discussed how this new policy may impact the overall process and specifically addressed the potential impact on International Medical Graduate (IMG) and, to a lesser extent, DO applicants, they did not explore the impact on URM applicants. To date, no study has explicitly surveyed PDs regarding how they believe URM applicants will be impacted by the Step 1 pass/fail policy.

Twelve competitive US residency specialties were selected for this study due to their relatively low rates of admitting URM applicants, osteopathic graduates (DO), and IMG.^18–20 Importantly, many who advocated for the Step 1 scoring change suggested that the change to pass/fail scoring will benefit URM candidates. We aimed to collect data to better understand the perspectives of PDs and included all three groups to ensure full inclusivity and to allow for a sort of control—as the difficulty of DO and IMG applicants matching into competitive specialties is well established. For the purposes of this study, URM included both allopathic and osteopathic grads, and IMG included both US and non-US grads.

We specifically focused on identifying what selection factors PDs from these twelve specialties consider to be most important now that Step 1 is pass/fail, and if they believe this change will impact the residency selection difficulty of URM, DO, and IMG applicants.

Methods

Results

A total of 433 responses were received (38%) out of the 1141 accredited residency programs contacted. Within the 433 responses, each specialty was represented as follows: 11% (n = 49) dermatology, 18% (n = 76) diagnostic radiology, 4% (n = 19), interventional radiology, 10% (n = 43) neurosurgery, 8% (n = 36) ophthalmology, 13% (n = 58) orthopedic surgery, 10% (n = 42) otolaryngology, 5% (n = 23) plastic surgery, 5% (n = 20) radiation oncology, 2% (n = 7) thoracic surgery, 10% (n = 43) urology, and 4% (n = 17) vascular surgery. Response rates within each specialty ranged from 24% to 48% (Table 1). A geographical breakdown of responses appears in Table 2. Notably, the highest response rate was from PDs in the northeast (30.7%), followed by the southeast (24.7%).

Across all 12 specialties, 24% (n = 433) reported believing that the Step 1 pass/fail policy will make it more difficult and 30% reported it will make it less difficult for URM applicants to match into their specialty, with a 6% net effect of less difficult (Table 3). None of the twelve specialties reported greater than 40% of PDs anticipating increased residency selection difficulty for URM applicants. However, 46% of PDs reported the difficulty would stay the same. Regionally, two trends were observed with respect to likelihood of URM applicants to be selected for residency: PDs in the west had a higher number of those reporting “less difficult” and a lower number of those reporting “more difficult” compared to other regions.

PDs were asked to report whether they felt the Step 1 pass/fail policy would impact the residency selection difficulty experienced by DO applicants (Table 3). Overall, 49.2% (n = 433) of all PDs reported it would be more difficult, while 6.2% reported it would be less difficult (43% net effect more difficult) and 44.6% reported it would be the same as in past years. Specialties in which greater than 50% of PDs reported increased difficulty for DO applicants included: plastic surgery (69.6%), neurosurgery (60.5%), otolaryngology (57.1%), and vascular surgery (53%). Notably, none of the neurosurgery, plastic surgery, and thoracic surgery PDs believed the pass/fail policy would decrease match difficulty for DO applicants. Regionally, two trends in the frequency of responses were observed: Fewer PDs in the west selected “more difficult,” while more PDs in the midwest responded “less difficult.”

Across all 12 specialties, 56.4% (n = 433) of PDs reported believing that the Step 1 pass/fail policy will make it more difficult for IMG applicants to be selected for residency, whereas 3.9% believed it would be less difficult (52% net effect more difficult), and 39.7% of PDs believed it would be the same as in past years (Table 3). Specialties in which greater than 50% of PDs believed it would increase difficulty for IMG included: Plastic surgery (78.3%), radiation oncology (75%), ophthalmology (69.4%), diagnostic radiology (61.8%), neurosurgery (58.1%), thoracic surgery (57.1%), vascular surgery (53%), and urology (51.6%). Notably, none of the orthopedic or plastic surgery PDs believed the pass/fail policy would decrease match difficulty for IMG applicants.

When PDs were asked to rank specific academic criteria (1-7) in the order of importance for interview selection (Options were: Step 2 CK score, Alpha Omega Alpha membership, class rank, tier of medical school, published research, research year/fellowship, and clinical grades), 61% of PDs ranked Step 2CK score as either #1 or #2 (Figure 1).

When PDs were asked to rank personal attribute criteria (1-6) in the order of importance (Options were: Away rotation at your site, strong clinical letter of recommendation, previous knowledge of the applicant, leadership in clubs, volunteer hours, and personal statement), 75% of PDs ranked an away rotation at their site as #1 or #2 (Figure 2).

Discussion

In the absence of a numerical Step 1 score, the majority of PDs from 12 competitive specialties anticipate the two most important residency interview selection factors will be Step 2CK score and participation in an away rotation at their site. While this is not surprising and supports the results of previous work, this study is unique in that we focused on twelve of the most competitive specialties with some of the lowest match rates for our three selected subgroups and explicitly asked PDs about the potential impact on these applicants. Notably, our study built on a previous study¹⁶ by specifically asking how the pass-fail scoring change would impact URM applicants. Overall, there was a stark contrast in PDs’ beliefs regarding the difficulty of matching between IMG and DO applicants (collectively) and URM applicants. The intention of changing the USMLE scoring to pass/fail was to mitigate barriers. However, data reported in this and other studies illuminate potential unintended consequences for all three student subgroups.

Historically, students would apply for residency with only a Step 1 score and would take Step 2CK after their residency application was already submitted. This often reduced the pressure felt by students when applying. However, given the results of this and other studies that PDs believe Step 2CK is one of the new most important factors when considering an applicant's residency selection, there is no pressure for students to opt to take this exam prior to their application submission date. To address this issue, many medical schools have shifted their recommendations and curricular timelines, requiring their students to complete Step 2CK prior to the beginning of their fourth year.²³

In this study, 46.0% of PDs believed that the match difficulty will remain the same for URM applicants and 30.0% of PDs reported it would be less difficult. Step 1 was previously reported by PDs as the most important factor in ranking applicants,¹³ and non-White examinees historically score lower on both Step 1 and Step −2CK.^7,8 Hence, to date, URM applicants have represented a disproportionately smaller percentage of those selected for interviews and are some of the least represented among these 12 competitive specialties.^24,25 The previous Step 1 scoring system placed URM applicants at a disadvantage, and some proponents of the scoring change feel the pass/fail system will mitigate this divide.⁶

While 76% of PDs from competitive specialties believed the score change will make it less difficult or stay the same for URM students, it should be noted that performance on Step 1 has been shown to be an indicator for Step 2CK performance, an exam that now has enhanced emphasis.^14,26 Indeed, previously published data demonstrate that URM residency applicants had a lower mean Step 1 and Step 2CK score when compared to non-URM candidates. This raises the concern that one standardized exam may simply replace another.^7,8 In addition, with Step 2CK being administered so late in training, it increases the amount of pressure on students to enhance their score,¹⁹ while often requiring them to secure away rotations prior to knowing their score that may impact where they can match.

In addition, we noted an interesting trend: a higher percentage of PDs from western programs believed it would be less difficult for URM students to match than PDs from other regions. Importantly, the western region had the lowest number of responses, so these results should be viewed with caution. Nevertheless, this trend aligns with results reported in an earlier study that surveyed vascular surgery PDs about their use of Step 1 scores to screen applicants. Those located in the western region were less likely to report using Step 1 as a screening tool than other regions, so it is possible that western regions may have less of a need to prioritize Step 2CK in the absence of Step 1 scores.¹² Studies have demonstrated that individuals who are URM are more likely to practice in underserved areas,²² and it is possible that the larger metropolitan areas in the western states are implementing more holistic admissions tools to diversify resident cohorts.

Our response rate by region made it clear that more PDs from the Eastern region participated. Among the Eastern PDs, we observed more PDs who reported using Step 1 as a screening tool and believe it will be more difficult for URM students to match. This might reflect the preferences of Eastern PD's as well as PDs from across the country for objective measures as an indicator of resident success. As mentioned before, such findings may reflect variations in how programs approach the admission process more objectively or holistically; however, it is important to consider differences in the number of applicants applying to western residency programs versus eastern residency programs. Given possible differences in the volume of applicants to programs each year, the Step 1 screening tool may be utilized more frequently in larger programs to limit applications to a number that can realistically be reviewed.

Participation in away rotations has always been an essential part of the medical student experience, especially for those interested in pursuing a competitive residency. Students obtain clinical experience among different patient populations and health systems while having the opportunity to network with residency PDs. However, away rotations can be expensive and impose financial constraints on students, which may particularly impact those from lower-income backgrounds.²⁷ In addition, URM students have faced barriers to receiving mentorship and making professional connections, which can limit opportunities to participate in away rotations.²⁸ Fortunately for URM students, numerous medical schools and professional organizations have recognized these disparities and developed programs to work toward providing more equitable access to some away rotations.²⁹ While it is encouraging that some funded away rotations exist for URM applicants, they are not universally available and have limited enrollment.³⁰

The current study has many limitations, including the variation in response rates across the regions and the twelve specialties, which prevented subspecialty analysis and thus limited specialty generalizations. There may have been a response bias toward PDs who feel strongly in favor of, or opposed to, the scoring change. While lower response rates were noted, it was encouraging to see that several of our results supported those of previous studies. It is also important to note a limitation with the questionnaire: while pilot testing was conducted, it was with a very small sample size (1%) and not of the target-audience. Limitations to this study include the inability to complete power analysis because there was not enough data (eg, no pre- and post-survey for comparison). An additional limitation is our speculation that URM and all students may feel pressured to complete away rotations thereby undertaking a greater financial burden. By including Step 2 scores as an option in our survey, we also implicitly made the assumption that given the Step 1 change in score reporting, students will choose to take Step 2CK prior to submitting their residency application. Finally, PDs from the 12 different specialties have varying experiences and might prioritize different factors in selecting residents beyond USMLE scores, making it challenging to generalize findings.

Conclusion

Our results suggest that the majority of PDs believe the USMLE Step 1 score change policy will not impact, or could even lessen, barriers for URM students to match into competitive specialties. Paradoxically, PDs believed that Step 2CK score and away rotation participation would become the two most important selection factors in the absence of Step 1 numerical scores. Hence, despite the beliefs of PDs that the Step 1 pass/fail scoring will mitigate barriers for URM students, potential unintended barriers need to be adequately considered and addressed.

While URM applicants have historically scored lower on Step 1 and Step 2CK than non-URM applicants,^7,8 residency programs are left with few if any, other external objective criteria with which to predict those applicants who will be successful. We encourage residency PDs to rectify this by determining a minimum score for which they can expect a future resident to be successful, rather than focusing on one resident outscoring another.

Concerns about the residency selection process are compounded by the recent US Supreme Court ruling that disallows higher education admissions to consider race and several state initiatives that are dismantling Diversity, Equity, and Inclusion initiatives. It is unclear what downstream effects may result from these recent events; therefore, medical education stakeholders need to strongly promote a holistic and inclusive residency application process.

Supplemental Material

Footnotes

References