1 Introduction

In recent years, the landscape of education has been significantly influenced by the remarkable advancements in generative artificial intelligence and large language models (LLMs). These innovations have paved the way for many educational technology solutions, aiming to streamline the often cumbersome and time-consuming tasks associated with generating and analyzing textual content. These models, exemplified by Generative Pre-trained Transformer (GPT) [¹⁶], harness deep learning, reinforcement learning, and self-attention mechanisms to process and generate human-like text based on natural language inputs. Their capability to comprehend intricate patterns and relationships within textual content, encompassing semantic, contextual, and syntactic nuances, has revolutionized various sectors, including education [⁵, ³, ¹⁷].

LLMs such as GPT-3.5 [²], GPT-4 [¹⁶], Gemini [⁶], and Llama-2 [¹⁸] have been pre-trained on vast and diverse datasets across multiple domains. This pre-training equips them with the remarkable ability to perform natural language processing tasks with minimal or even zero additional training, thus lowering the technological barriers to creating innovative educational solutions. The recent intro-duction of ChatGPT and Google’s BARD marks a significant step towards user-friendly, LLM-based generative chatbots. These user-friendly interfaces enable a broader audience to harness the power of sophisticated language models, contributing to increased accessibility and engagement with artificial intelligence.

Researchers have measured the capability of LLMs to pass specific exams, but primarily to measure the LLMs’ power to mimic human intelligence. Mainly, GPTs and BARD models have been tested on a wide range of fields such as the United States Medical Licensing Exam (USMLE) [¹³], the American Board of Anesthesiology (ABA) exam [¹], and in a vast datasets in Medicine [¹⁷]; proficiency in reading comprehension [³], and various branches of knowledge, including subjects in the humanities, social sciences, physics, computer science, mathematics, and more [¹⁰], mainly in English language. Also, for the Spanish language, some studies have been conducted in the medical context, such as the Spanish Medical Intern Examination (MIR) [⁹], and Rheumatology-related questions from MIR [¹⁴].

While the potential benefits of integrating LLMs into education are evident, educators are concerned that the widespread use of LLMs may lead students to overly depend on technology for acquiring factual information and reasoning. They are concerned that students might stop developing their critical thinking skills if they become accustomed to relying solely on LLMs for answers without reasoning. Moreover, educators are apprehensive about the potential for cheating in online exams, where students could exploit LLMs to obtain answers, generate essays, or provide explanations [⁴].

This study centers on the evaluation of models that offer free accessibility to the majority of Mexican students. We specifically examine two LLMs, GPT-3.5 and BARD (supported by Gemini Pro). Our primary objective is to assess the general knowledge, problem-solving, and reasoning capabilities of GPT-3.5 and BARD. To achieve this, we analyze their performance on three sample exams for undergraduate ad-missions. Knowledge tests play a crucial role in selecting candidate students equipped with the necessary knowledge to pursue academic programs in biological and medical sciences, engineering, mathematical and physical sciences, and social and administrative sciences.

2 Material and Methods

The National Polytechnic Institute^fn (IPN, for its acronym in Spanish) is a public institution dedicated to advancing education, research, and innovation. As one of the leading educational institutions in Mexico, holding the estimated rank of the third-best university in the country^fn, the IPN plays a pivotal role in providing high-quality academic programs across various disciplines.

The IPN offers 69 academic programs in three main fields of study: 43 programs in engineering, including mathematics and physics; 14 programs in biological and medical sciences; and 12 programs in social sciences. The IPN publishes a study guide for the university admissions tests each year. The 2023 admissions tests were structured by areas of knowledge. This year, history and reading comprehension of the English language were included to enhance the comprehensive academic program [¹¹]. The admission exam comprises 140 questions covering subjects such as mathematics, history, writing, and reading skills in the Spanish language, biology, chemistry, physics, and reading comprehension of English as a foreign language.

The admission exams were prepared for three main groupings of fields of study: Engineering/Mathematical and Physical Sciences (E-MPS), Biological and Medical Sciences (BMS), and Social and Administrative Sciences (SAS). Each exam evaluates various vital skills and competencies of candidate students in their field of study. These exams aim to provide a standardized measure of a student’s readiness and ability to understand and analyze written passages in both Spanish and English with a deep understanding of Spanish, which includes comprehension, interpretation, and application of information. The physics, chemistry, and math sections assess student’s quantitative reasoning, problem-solving, and mathematical knowledge, while the questions also evaluate critical thinking and logical reasoning.

The distribution of exam questions by topics for undergraduate admissions to the three groups of fields of study offered by the IPN academic programs is presented in Table 1. The sample exams consist of 140 multiple-choice questions (indicated in the Q column of the table) with varying distribution based on the major chosen by the candidate student. For example, in the case of an engineering career, such as the E-MPS exam, mathematics and physics carry more weight, with 37 and 17 questions, respectively, in contrast to biological (BMS exam) or social (SAS exam) sciences.

On the one hand, LLMs demonstrate exceptional dexterity in processing and interpreting the text. However, not all LLMs used in our experiments can handle visual information. Therefore, we aim to minimize the inclusion of visual questions to ensure a fair comparison. In our experiments, we refrain from using visual information, which includes questions or options involving images such as sequences of figures, schemes, charts, and electrical diagrams. If a question originally included images and could be adequately described in text form, we included the question by providing a textual description of the image.

The distribution of questions prepared and adapted for the experiments is shown in Table 1. The column EQ represents the number of questions used in our experiments. The E-MPS and SAS exams consist of 126 questions each, and the BMS exam of 122 questions.

The language models employed for the ex-periments were GPT-3.5 and BARD. For the GPT-3.5 model, we utilized the ChatGPT web interface [¹⁵] with the version of November 2023, which includes support for the Spanish language. Similarly, to assess BARD, we employed the BARD web interface [⁸] with the updated version of December 2023, supporting the Spanish language and enhances introduced by the Gemini Pro model [⁷, ⁶].

We proceeded manually with the assessment of the models by entering questions along with the corresponding multiple-choice options in the models’ web interface. All questions have four response choices (a, b, c, and d). The responses generated by the models were compared to the correct answer sheet for each exam included in the study guide for admissions.

Sometimes, the model did not respond because the question was not understandable. In such cases, the question was paraphrased or provided with further clarification until the model obtained a response. In addition, we introduced one the following prompts followed by the question to push the model to select an option: “Seleccionar de las siguientes opciones ...” (“Select from the following options”) or “Seleccionar una opción de las siguientes opciones ...” (“Select an option from the following choices”). In the case of the reading and writing section, for text-dependent questions, the text is provided to the model; subsequently, a prompt is used such as “Dado el texto anterior,” (“Given the previous text,”) or ”Del texto anterior,” (“From the previous text,”) followed by the question along with its multiple choices. In the mathematics section, if a question or an option involves mathematical notation, it is represented with its Wolfram form [¹⁹]. This approach ensures that both models interpret formulas accurately. The repository containing the questions and their corresponding answers used in the experiments is available for download on GitHub^fn.

IPN admissions to an academic program is contingent upon achieving a minimum number of correct answers. The specific number required varies depending on the academic unit and program. Table 2 summarizes the minimum scores necessary for admitting a candidate student to the school and campus. The scores are provided by the IPN Office of Transparency and Access to Information [¹²]. The columns Estimated Minimum Score (2023) and Estimated Minimum Score represent the minimum score proportion relative to the minimum score in the year 2022. The admissions exams of the year 2022 comprise 130 questions. The table presents the highest, median, and lowest required minimum values for each of the fields of study.

For example, to be accepted into Aeronautical Engineering (Ingeniería Aeronáutica) at the ESIME Ticomán campus, a minimum score (2022) of 96 correct answers is required, and for the Geophysical Engineering (Ingeniería Geofísica) program at the ESIA Ticomán campus, a score of 70 correct answers is required. Both academic programs belong to the Engineering/Mathematical and Physical Sciences (E-MPS).

In our experiments, we considered the Estimated Minimum Score as the required minimum value for acceptance to the campus or the equivalent percentage of the minimum score.

3 Results

The overall results of the LLMs evaluation are presented in Table 3. For the Engineering/Mathematical and Physical Sciences exam (E-MPS), GPT-3.5 and BARD achieved an identical score of 57.93%

Regarding the Biological and Medical Sciences exam (BMS), BARD performed slightly better with a score of 59.83% compared to GPT-3.5. For the Social and Administrative Sciences exam (SAS), GPT-3.5 outperformed BARD with scores of 65.87% and 63.49%, respectively, achieving a more successful outcome in the examination. In summary, GPT-3.5 outperformed BARD marginally, securing scores of 60.94% and 60.42%, respectively. Considering the minimum acceptance score for the year 2022, which is a percent score of 53.85% for the E-MPS and SAS exams and a score of 55.38% for the BMS exam, both models demonstrated sufficient performance for IPN admission to an academic program.

Tables 4, 5, and 6 show the disaggregated responses of the exams by topic: Biology, Chem-istry, Foreign Language, History, Mathematics, Physics, Reading Comprehension, and Writing Comprehension.

Table 4 shows the results of the E-MPS exam. The exam consists of 126 questions, covering more questions in Mathematics and Physics. Both models performed well on most topics, and overall performance is identical score of 57.93%. In the topic-specific performance, GPT-3.5 outperforms BARD in Biology, History, Mathematics, and Writing Comprehension. On the other hand, BARD scored the same as GPT-3.5 in Chemistry and Physics; BARD is better in Foreign Language and Reading Comprehension.

Table 5 shows the topic-specific performance for the BMS exam, which comprises 122 questions. The exam covers more questions for Biology and Chemistry. Overall performance, BARD outperforms slightly GPT-3.5 on the BMS exam, with a score of 59.83% compared to GPT-3.5 of 59.01%. BARD performed better than GPT-3.5 on all topics except for Mathematics and Physics. The most significant difference in performance was in Physics, where BARD scored 36.36% compared to GPT-3.5, with a score of 72.73%.

Table 6 shows the topic-specific performance for the SAS exam, which comprises 126 questions. The exam covers more questions for History, Reading Comprehension, and Writing Comprehen-sion, which covers 49.21% of the exam. Overall performance, GPT-3.5 outperforms BARD on the SAS exam, with a score of 65.87% compared to BARD of 63.49%. GPT-3.5 performed better than BARD in Mathematics, Physics, Reading, and Writing Comprehension. BARD does better in Biology, Foreign Language, and History. Both models performed well in Chemistry.

Figure 1 illustrates the quartiles of required minimum percentage scores for academic pro-grams offered by the IPN, encompassing three main groups of fields of study. For the E-MPS exam, the quartiles are Q1 = 56.15, Q2 = 57.69, and Q3 = 60.77, with a minimum value of 53.85 and a maximum value of 73.85. Both GPT-3.5 and BARD perform slightly higher (57.93) than Q2. Consequently, the models have achieved admission to at least 50% of the schools/campuses offering academic programs in Engineering and Mathematical and Physical Sciences.

Fig. 1 Quartiles of required minimum percent scores for the academic programs offered by the IPN covering the three main groups of fields of study 

For the BMS exam, the quartiles are Q1 = 60.39, Q2 = 63.85, and Q3 = 69.23, with a minimum score of 55.38 and a maximum score of 75.38.

GPT-3.5 and BARD, with respective scores of 59.01 and 59.83, meet the criteria for admission to 25% of schools offering academic programs in Biological and Medical Sciences.

In case of the SAS exam, the quartiles are Q1 = 55.77, Q2 = 60.0, and Q3 = 65.39, with a minimum score of 53.85 and a maximum score of 75.38.

GPT-3.5 slightly exceeds Q3, representing the minimum score required for admission to 75% of academic programs in the Social and Administrative Sciences field.

BARD scored 63.49, placing it between the 50%-75% acceptance range in the same field of study.

4 Discussion

Although the required minimum score varies from year to year, the percentage of the minimum score is an excellent measure to estimate the proficiency of the models. According to the results, the evaluated models have exhibited proficiency in successfully passing all three admission exams.

GPT-3.5 consistently outperforms BARD in Mathematics across all exams. However, the overall performance in this subject remains relatively low, less than 63.33%. Furthermore, GPT-3.5 outperforms BARD in Physics in two exams (E-MPS and BMS). These subjects involve tasks that require comprehension, reasoning, problem-solving, and calculation. Exam questions cover diverse topics such as numerical series, geometry problems, systems of linear equations, trigonometry problems, and differential and integral calculus.

The models excel in solving specific and well-known academic problems where an apparent problem is presented, and a formula can be applied to find a solution, such as questions related to calculus or systems of linear equations. However, the models encounter challenges when faced with math word problems presented in a textual format requiring students to interpret and solve the problem. In such situations, GPT-3.5 and BARD may need help because the problem statement and sequence of explanations are well-defined; calculating or substituting values may pose difficulties. In these cases, a second interaction with the model was initiated solely for examination purposes to indicate the error in the model’s response and provide the correct information. The model then adjusted its selected choice and explained the solution but often encountered further inaccuracies.

Evaluating the overall performance in Spanish language-related questions, encompassing reading and writing comprehension topics across all exams, GPT-3.5 achieved a percentage score of 56.36%, while BARD attained a score of 55.45%. These scores correspond to raw scores of 62/110 and 61/110 questions for GPT-3.5 and BARD, respectively.

Both models encountered challenges in identi-fying the text’s central ideas, determining point of view and tone, and engaging in textual entailment to infer information.

Notably, BARD exhibited proficiency in tasks re-quiring factual information, such as history-related or conceptual problems.

5 Conclusions

LLMs have demonstrated proficiency in successfully passing all three exams required for IPN un-dergraduate admissions in Spanish language. The models’ notable achievements enable admission to up to 75% for some academic programs.

However, the most sought-after academic pro-grams, representing the top 25%, such as Medical Doctor and Obstetrician, Aeronautical Engineering, Business Administration and Development, Artificial Intelligence Engineering, and Bachelor’s in Physics and Mathematics, among others, currently fall beyond the scope of these models.

Due to LLMs becoming widely used, it may be necessary to modify the format and content of exams to ensure fair and reliable assessments for all students.

The widespread availability of advanced LLMs could create an unfair advantage for some students, exacerbating existing educational in-equalities and placing underprivileged students at a further disadvantage.

Despite the challenges LLMs pose, they also present promising opportunities for education. These models have demonstrated strong capabilities in supporting the learning process through detailed explanations during problem-solving and the ability to refine answers through interactions.

However, it is crucial to note that, for now, these models are not entirely reliable.