Introduction

The availability and quality of facilities and human resources in healthcare is of utmost importance,¹ with physicians comprising one of the main human resources. Most medical doctors (MDs) seek to continue their training within a specialty. However, not all MDs pursuing specialty training can enter an official program.

Most Western countries have regulatory agencies that are responsible for evaluating, selecting, and placing MDs to continue training. These agencies use multiple high-stakes processes to select the best candidates.^2,3,4 For example, the selection process in the United States involves approving different components of the United States Medical Licensing Examination, interviewing, and participating in a national matching program. In France, students present a national ranking examination and choose the specialty they wish to pursue according to their rank.⁴ In Mexico, MDs must first register on the registry of the Examen Nacional para Aspirantes a Residencias Médicas (National Exam for Applicants to Medical Residencies, ENARM).⁵ This requires registration of an individual’s ID, the specialty for which they are competing (one of 27 options), and the preferred date to present the exam (one of five consecutive days on which the test is available yearly, for 2016 and 2017). An online registration platform is enabled at a predetermined day and time, and closes around 10 days later. As registration and date-selection occurs on a first-come, first-served basis, 36 000+ applicants typically register during the first 24 hours. The ENARM is a norm-referenced test,^6,7,8 and is the assessment tool used in Mexico to select those who will enter specialty training. There are approximately 8 000 official training positions available nationwide. These positions vary by specialty, and are determined by the certified training programs of the healthcare institutions that provide them. Applicants for each specialty are ranked from highest to lowest according to their total ENARM score. Ranked applicants receive a ‘pass’ certificate until the quota is met according to that specialty’s available positions, and present it to the institution of their choice to apply for an appointment.

The quality and performance of such selection and placement processes cannot be overemphasized, as they have a beneficial or harmful impact on the candidate, patients, healthcare systems, and ultimately, the public health of a given country. The Comité de Posgrado y Educación Continua (Committee of Postgraduate and Continuous Education [CPEC]), the agency in charge of the ENARM, confers the attributes of “equitable, transparent, objective, and valid” to the ENARM.^9,10 The CPEC declares that the five different test forms (one for each test day, each comprising an exclusive set of items) used for the ENARM in a particular year are equivalent because they are created with “the same objectivity, quality, balance, and academic level”.^9,10 However, our group recently reported that the test development processes and the theoretical basis under which the ENARM (2016 and 2017) was built do not support the stated attributes of validity, reliability, and equity.¹¹This contrasts with important advances that have resulted in a change in established standards for educational testing,^12,13,14 and supports the statement that, for some systems of educational assessment in medicine, tradition weighs more than evidence-based methods and best practices.¹⁵

The CPEC does not precisely define ‘equity’, but the sense of this attribute is close to that of ‘fairness’ (preferred in the English language and in the literature related to education), as defined by Lane and colleagues,¹⁶ “fairness in testing is achieved if a given test score has the same meaning for all examinees and is not substantially influenced by factors not relevant to the examinee’s performance.”

This study aimed to assess whether the ENARM met the attribute of ‘equity’ (i.e. fairness) as ascribed by the CPEC, by analyzing the databases of the results for all applicants in 2016 and 2017.

Materials and methods

This study analyzed the official databases of ENARM 2016 and 2017. The study took place in Guadalajara, Mexico from February to May of 2018.

Results

In the 2016 ENARM, 36 114 applicants from 112 medical schools applied for 27 specialties, mainly general surgery, internal medicine, gynecology-obstetrics, pediatrics, anesthesiology, and family medicine.Table I shows applicants’ demographics, NoCA by item difficulty, and ANoCA. Applicants’ age and sex distribution was similar across all ENARM days, except for day 5, in which applicants were on average 2 years older. Significant differences were found (according to the 95%CI) in scores for the low-, medium-, and high-difficulty questions; day 2 had higher scores (easier test form) and day 5 had lower scores (more difficult test form). These differences were also reflected in the ANoCA. Univariate linear regression showed that the ANoCA decreased significantly for each year of increase in age (−3.26 [95%CI −3.37 to −3.15], being female (−5.45 [95%CI −6.26 to −4.64]), being non-Mexican (−5.12 [95%CI −7.57 to −2.66]), and having studied in a private medical school (−3.36 [95%CI −4.3 to −2.42]).

In the 2017 ENARM, there were 6% more applicants and three more medical schools, but the same number of specialties. For most specialties, the number of applicants varied slightly. Table II shows demographics, NoCA by item difficulty, and the ANoCA. Applicants’ age and sex distribution was similar across all days. Significant differences were found (according to the 95%CI) in the scores for low-, medium-, and high-difficulty items. The NoCA and ANoCA showed patterns similar to 2016. Days 1 and 2 showed higher scores (easier test forms) and day 5 showed lower scores (more difficult test form). The univariate linear regression showed that ANoCA decreased significantly in similar way to the previous year: for each year of increase in age (−2.99 [95%CI −3.1 to −2.88]), being female (−5.29 [95%CI −4.47 to −6.1]), being non-Mexican (−4.97 [−7.29 to −2.65]), and having studied in a private medical school (−3.29 [95%CI −4.23 to −2.34]).

The identified inter-day differences influenced the number of applicants who achieved the SMS for the specialty for which they were competing. Table III highlights the number and proportion of ARSMS (overall and per day) for the 12 specialties with the greatest number of applicants (data on all specialties available upon request). Grouping days with similar proportions of ARSMS showed that applicants on day 2 were more likely to achieve their SMS when compared with days 1, 3, 4 (OR 1.55 [95%CI: 1.46 to 1.64]), and 5 (OR 3.8 [95%CI: 3.47 to 4.16]) for year 2016. The inter-day differences also had an effect for the 2017 ENARM. After grouping days with similar proportions of ARSMS, applicants on days 1 and 2 were more likely to reach their SMS when compared with days 3, 4 (OR 1.85 [95%CI 1.75 to 1.94]), and 5 (OR 4.04 [95%CI 3.72 to 4.38]).

The inter-day differences among item difficulty categories and the ANoCA were adjusted for age, gender, selected-specialty, and nationality using multivariable linear regression. The 2016 ENARM showed that days 1, 3, and 4 had similar coefficients (differences in the ANoCA), whereas day 2 showed an advantageous coefficient (e.g., 11.66 higher ANoCA than day 1) and day 5 showed a disadvantageous coefficient (16.18 lower ANoCA than day 1) (table IV). The 2017 ENARM showed days 3 and 4 had similar inter-day coefficients, days 1 and 2 had similar but higher (advantaged) coefficients, and day 5 had a lower coefficient (disadvantaged). Furthermore, the inter-day coefficients within the item difficulty categories also significantly differed (table IV).

Since the significant inter-day variability described above precludes the assumption of equity for the ENARM in 2016 and 2017, we estimated its impact as the number of applicants that could have been dubiously classified as not passing (secondary end-point). Table V shows that for 2016, i (day 2) presupposed a disadvantage for 2 070 (9.5%) applicants of R (days 1, 3, and 4) and for 1 495 (21.7%) applicants of j (day 5). In consequence, 3 565 (12.49%) applicants were affected by the lack of equity in the test forms. In other words, this last figure represents 44.64% of the 7 986 applicants that received a pass certificate that year.^23,24,25

Table V shows the estimate of affected applicants in 2017, where i (days 1 and 2) presupposed a disadvantage for 1 644 (13.6%) applicants of R (days 3 and 4) and 1 511 (19.79%) applicants of j (day 5), indicating 3 155 (13.64%) applicants were affected. This represents 37.2% of the 8 480 applicants that received a pass certificate in 2017.^26,27,28

A sensitivity analysis for the multivariable linear regression model that also adjusted to the status of ‘public/private’ medical school in addition to the other variables was performed (data not shown, available upon request), but excluded approximately 1 200 and 1 400 applicants (2016 and 2017, respectively) from foreign medical schools for which the “public/private” status could not be identified. It yielded similar statistically significant results and identical grouping of the days, but a higher estimate of applicants affected by inequity. Considering these findings, the authors decided to opt for the more discrete estimate.

Additional data on the ANoCA for each specialty for both years is available upon request.

Discussion

The effort required for an MD to meet the selection requirements is vast. The selection process is based on different constructs. A construct of particular importance is fairness. This ecological study evaluated the equity (i.e. ‘fairness’)¹⁶ of the ENARM over two consecutive years.

Our first important finding was that there were significant inter-day differences in scores for items classified as low-, medium-, and high-difficulty, and thus the ANoCA. These differences, although significant, may seem small. However, a difference of just one correct item has an impact on an applicant’s pass or no-pass status. Consequently, the variable ‘day on which an applicant presented the test’ in both years gave applicants an advantage or disadvantage. For 2016, day 2 applicants were 55% more likely to reach their SMS than applicants on days 1, 3, and 4, and almost four times more likely than on day 5. For 2017, day 1 and 2 applicants were 85% more likely to reach their SMS than applicants on days 3 and 4, and four times more likely than on day 5. Therefore, the stated premise of equity was not sustained.

The methods used to create the different test forms may explain these inter-day differences. The agency that creates the ENARM does not perform item analysis with established psychometric theories, such as classical test theory or item response theory. Instead, experts classify an item’s difficulty into three broad categories defined a priori, without performing field testing on the target population, whereas item discrimination is not calculated (data on applicants’ answers are erased after the test).^17,29The only consideration in assembling different test forms is including an exclusive set of items for each test form that accounts for the same number of items per area of knowledge and includes 25% low-difficulty, 50% medium-difficulty, and 25% high-difficulty items. Consequently, the problem is in the definition of difficulty. For example, the low-difficulty category has in itself a range of difficulty values; one test form may have low-difficulty items that are, on average, located at the top of this range, whereas another test form may contain items that are located, on average, at the bottom of the range. Furthermore, for a test where the score is computed as the sum of N dichotomously scored items, such as the ENARM, the test’s mean score (i.e., average ANoCA) directly relates to the average difficulty of item scores.³⁰

If research-supported methods in educational assessment already in use had been used for the ENARM, the attribution of equity would not have been compromised. Best practice involves performing item tryout and item analysis to assess item difficulty and discrimination, with the purpose (among others) of calibrating items to design alternate test forms as if they were on the same test score scale.³¹Standard 5.12 of the Educational Standards for Educational and Psychological Testing ¹² states that “a clear rationale and supporting evidence should be provided for any claim that scale score earned on alternate forms of a test may be used interchangeably.” However, in the ENARM, although ‘equivalence among different test forms’ is stated, no supporting evidence is provided and no test-equating methods are used.¹⁷

Our second finding concerns estimating the impact on applicants due to differences according to the day they took the exam. Since the ENARM is a selection exam for specialties and each specialty has a predetermined number of positions, it is expected that applicants with high scores will not be impacted. The expected impact would be on those scoring closely below and above each SMS. Our estimation showed that around four out of ten applicants that received a pass certificate might have not received a pass if the test forms were equitable. The impact might go beyond these estimates; that is, on applicants’ professional development, quality of healthcare provided by the training institutions, and ultimately, population health.

Regarding the variables associated with obtaining a lower ANoCA, we can only speculate. We have no explanation as to why female applicants obtained lower scores, but older applicants may most likely represent MDs that graduated years ago and may not be as updated as their recently-graduated counterparts. Regarding the private status of a medical school, a possible explanation is that acceptance into some private schools may not be as competitive and selective as that of public schools.

The strengths of this study that lend weight to the conclusions are that the official ENARM 2016 and 2017 databases were analyzed and the attribution of equity was assessed using different approaches. Nonetheless, several limitations should be noted. First, this was an ecological study with inherent limitations of that design. Second, differences in the average ANoCA among test forms do not necessarily prove a lack of equity, but do suggest that further review is needed.³² The ENARM is a norm-referenced test by which applicants are selected based on a quota. The inter-day differences in the proportions of ARSMS strongly suggest that the different test forms jeopardized the possibility that a better (or at least equally) prepared number of applicants entered specialty training. Third, the official databases did not include data on the correct/incorrect status for each answered item, which precluded calculating the standard error of measurement of each test form to assess its reliability.³³In addition, we could not use the method proposed by Raykov and Marcoulides³⁴ to assess whether the different ENARM test forms were consistent with the model of parallel tests in the classical test theory framework, as the way applicants’ scores are compared fits this model (i.e., no estimation of random error of measurement and no use of test-equating methods). Fourth, other relevant variables that might have influenced test scores were not available for analysis, such as applicants’ number of test attempts and the grade point average obtained during medical school. Fifth, it is assumed that applicants distributed in a more or less randomized manner among test days. This assumption is supported by the registration method. Test-registration occurs on a first-come, first-served basis; most applicants try to register immediately, which leads the web-server to saturate, delaying successful registration for applicants to a random fashion.

In many science-related areas, there is a concern that a gap between research and practice exists, even with the growing volume and quality of evidence along with the technological and organizational improvements in information management and synthesis.³⁵The field of educational assessment seems to be no exception, as attested by the example of the ENARM. Furthermore, although this study aimed to assess equity among ENARM test forms for 2016 and 2017, it has to be considered that the exam methods have been the same since at least the 2010 ENARM.¹⁸ Changes newly introduced for ENARM 2018 are likely to decrease the lack of equity by using the same test form for all applicants competing for the same specialty.³⁶ This may be the best immediate approach to improve the equity of the test. However, improvements in many other aspects of the ENARM are needed in the long term. Examples include: a) increasing transparency by creating a publicly-available test’s development technical report to provide evidence of validity, reliability and fairness; b) improving item-writing, as applicants’ opinions suggest that poorly-written items introduce construct-irrelevant variation (e.g., those written in English); c) improving item development by using item tryout and item analysis; d) assessing the effect on test scores of the nine different item arrangements used for each test form,¹⁸ as the location of items on a test form can affect item statistics, particularly in testing programs that report immediately or shortly after test administration;¹⁶ e) addressing sources of systematic errors (applicants’ opinions suggest that a 2017’s test form included an item that required applicants to identify a clinical sign in an image, but the image was veiled); and f) estimating the standard error of measurement, which requires the non-deletion of answered items after the test.

Evidence points toward the need to use a robust, evidence-based test-development process on the ENARM if the test’s stated objectives and characteristics are to be achieved. Estimates suggest that average items for high-stakes tests may be valued around 300 USD, with some items valued above 1 000 USD (e.g., those that undergo extensive statistical analysis or that need complex development processes such as simulations); overall, an item bank with a few thousand items may be valued around 1 000 000 USD.³⁷ The revenues obtained from the ENARM registration fees are around 110 000 000 MXP per year (approximately 6 000 000 USD). With these funds, it would be possible to redesign the test to improve its quality.

In summary, the ENARM has a paramount and multidimensional impact on Mexico’s healthcare system and on applicants wishing to undergo specialty training. However, the analysis of the official ENARM databases does not support the attribution of equity. For this reason, it is necessary to redesign the test using evidence-based test-development processes that support the fairness, validity, and reliability of the ENARM.