texto en 
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
Introduction
The essential purpose of assessing applicants for a postgraduate program is generally to assess the level of satisfaction of the admission profile and to select those who possess characteristics compatible with the program and ensure the success of the program and the individual. “In the strict sense, the aptitude assessments are a diagnostic nature of capacities or potentialities tending to predict educational success” (Ibañez, 2009). The procedures and criteria for admission and selection of students are critical for measuring the organizational quality of the postgraduate program and also have a direct influence on academic quality. “We cannot forget that the quality and credit of a master's degree are intimately related to the quality of the students selected” (Bengoetxea-Castro & Arteaga-Ortiz, 2009).
At the global level, there is a broad agreement in the educational field about the importance of the student selection process for admission for postgraduate studies. In the United States of America, there are many analysis and meta-analysis of admission processes (Kuncel, Credé, & Thomas, 2007; Kuncel, Wee, Serafin, & Hazlett, 2010), and among the most important results the next test predict in an acceptable manner the level of the students, scientific productivity and number of citations: Graduate Record Examinations (GRE-T), Graduate Management Admission Test (GMAT) and Miller Analogies Test (MAT). Similarly, most of the countries of the European Union have made important efforts to homogenize the selection criteria in graduate programs (Davies, 2009; European University Institute [EUI], 2014; Kehm, 2006); standing out themes of equity, quality and capacity to promote mobility. Most European countries operate with a dual system in which universities have some control over admission, but within the framework of government guidelines on selection criteria (McGrath et al., 2014).
In Latin America efforts that express the attention for admission, demanded by the process of selection of applicants for postgraduate studies, have also been made. In Colombia, Colonia-Duque (2010) characterized the profile of graduate students and concluded, among other things, that there is a significant correlation between the average grades obtained in the master's program with variables such as curriculum, average undergraduate grades, interview and skills test. In Chile, they continue testing instruments capable of measuring attributes that complement indicators already used for postgraduate admission processes (Santelices et al., 2010).
In Mexico, a large number of universities use the test known as Examen Nacional de Ingreso (EXANI-III), proposed in 1996 by the Centro Nacional de Evaluación para la Educación Superior (CENEVAL), for the selection of postgraduate students. The thematic structure of the test includes: logical-mathematical reasoning, verbal reasoning, research methodology and skills, information and communication technology (ICT) and English. However, although few studies have been carried out on the subject, it has been found that the predictive capacity of the test to obtain the master’s degree in the desirable times is very low (Álvarez-Montero, Mojardín-Heráldez, & Audelo-López, 2014).
García (1995) emphasized that a characteristic of postgraduate studies in Mexico is that public institutions are much more selective than the private sector. On the other hand, there are a large number of universities that have designed their own test and selection procedure, which integrate the relevant information to select the students that meet the requirements of the admission profile; this is the case of the Master's program in Forest Sciences of the Universidad Autónoma Chapingo (UACh).
The current state of the Mexican postgraduate study system is largely explained by the national policy conducted by the Consejo Nacional de Ciencia y Tecnología (CONACYT) through the program known as Programa Nacional de Posgrados de Calidad (PNPC), which has been executed since 1991 (Adalid & de Urdanivia, 2011). Admission to this program is an important condition for the existence and development of postgraduate programs, especially those in public education. Admission is achieved through participation and approval of a set of indicators included in the three stages of the established evaluation model: self-evaluation, peer evaluation, and results and impact assessment (Consejo Nacional de Ciencia y Tecnología [CONACYT], 2013). In this process, the selection of applicants for postgraduate programs is a stage that can facilitate the achievement of quality indicators if done properly, which are vital to enter or remain in the PNPC
The objective of this research was to determine the combination weighting-aggregation that generates an index for the selection of applicants and is the most compatible with the criteria of the master’s program. For this purpose, we used information from 19 applicants in the year 2015. The combination of different weighting methods with different aggregation methods has been addressed in a very limited way. The literature reports few cases, specifically, within the framework of composite indices (Chakrabarty & Bhattacharjee, 2012). On the other hand, although the issue of selection in decision-making is quite broad and has been approached from different angles, no specific studies are known that analyzes combinations of weighting methods with aggregation methods in decisions related to selection.
Materials and methods
We used two methods of weighting of the criteria and sub-criteria: Point Allocation Method (PAM) and Analytical Hierarchical Process (AHP). The results of both subjective weightings were used to estimate the classification of the 19 students using the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and the Weighted Summarize Method (WSM), resulting in four combinations: TOPSIS-PAM, TOPSIS-AHP, WSM-PAM and WSM-AHP. Subsequently, the Pareto Analysis was used to determine the most compatible combination with the criteria of the program and, consequently, with the best selection of applicants.
Evaluation and selection model for applicants for the Master’s program in Forest Science at the Universidad Autónoma Chapingo
The selection of applicants is conducted by the Coordination of Postgraduate Studies of the program, which is composed by a Coordinator and four members, all of the faculty members. Accepted applicants are those whose overall score is equal to or greater than 80. The selection process looks at three criteria: knowledge, experience and personality. Each criterion comprises different sub-criteria (Figure 1) evaluated on a scale from 0 to 100.
Figure 1 Components of the model of selection for applicants for the Master's Degree in Forest Sciences at Universidad Autónoma Chapingo. JCR: Journal Citation Reports
1. Knowledge test. It is a written test consisting of 20 questions; 15 are related to Forest Sciences and five with other areas of interest
2. Skills test. Written test to identify aspiring skills in the areas of math, logic, and research. This test has four sections: the first relates to questions related to verbal reasoning, the second deals with mathematical aspects, the third relates to the approach of solving problems and the latter concerns research skills. Each section is composed of 10 questions.
3. English language diagnostic test. It is applied to know the English level of the aspirants, in order to, after being accepted, identify the needs of courses for the good performance in the reading and understanding of the literature of support in courses and research. Generally, this test is a version of the Test of English as a Foreign Language (TOEFL) applied by the University's Language Department. The raw score of each section is converted by statistical means into a number; the total score of the test is reported on a scale from 310 to 677. Grades were converted on the scale from 0 to 100, using the technique of rank normalization.
Experience includes the following sub-criteria:
4. Curriculum. The background and professional experience of the applicant, are qualified through the documentary evidence.
5. Average of the previous grade. The minimum average should be 80 points on the scale from 0 to 100.
6. Practical skills for research and exercise of synthesis and understanding. This task consists of searching for five scientific articles published in indexed journals, preferably included in JCR (Journal Citation Reports). Articles must be related to the research topic proposed by the applicant; two in Spanish and three in English. Applicants should summarize on a page the content of each article correlatively with their research topic and deliver within a period of five business days.
Personality groups the two remaining sub-criteria:
7. Interview with the Coordinator of the program. The interview is conducted in two phases. In the first one, a questionnaire is given to each applicant, containing 16 questions related to the following subjects: personal objectives to study this Master’s program, research and teaching concerns, knowledge of the academic staff of the program and knowledge of other similar options of study. The second phase consists of direct interview with the members of the Coordination, who ask about aspects addressed in the questionnaire of the first stage and other pertinent questions, to obtain good results in the program.
8. Psychometric test. This test is applied by psychology professionals at the university. The results are confidential and are expressed in three categories: fit, moderately fit and unfit, to which the grades of 100, 80 and 66, respectively, are assigned.
Weighting methods
The PAM consists in giving the set of criteria to be weighed to a group of experts and they are asked to distribute 100 points between the set of criteria and subcriterios considered. All this is based on experience and subjective judgment on the importance of each criterion (Organisation for Economic Co-operation and Development [OECD], 2008).
The method AHP uses paired comparisons to analyze alternatives, but also, as in this case, is used to estimate the weightings of the criteria. Comparisons are based on expert judgment to derive priority scales (Saaty & Vargas, 2012). One of the disadvantages is that, due to paired comparisons, inconsistencies can occur in the judgments expressed by the experts when classifying the criteria, especially when they are many criteria (Velásquez & Hester, 2013); however, an ad hoc modification of the Saaty and Vargas method (2012) can avoid them. On the other hand, one of the advantages of the method is the easy use to derive weightings of coefficients and compare the alternatives. Comparisons are made by pairs of criteria, proposing the following questions: Which of the two criteria is the most important? and How many times? The strength of the preference is expressed on a scale of 1 to 9, which allows measurement within the same order of magnitude. The preference of 1 indicates equality of importance between two criteria, while a preference of 9 indicates that a criterion is extremely more important than that with which it is compared (Table 1). The information obtained was processed using the program Expert Choice Desktop 11.5.1860 (Expert Choice Inc., 2014).
Table 1 Saaty and Vargas Scale (2012) to estimate the priority of the criteria used in the selection of postgraduate candidates.
| Value | Definition | Comment |
|---|---|---|
| 1 | Equal importance | Criterion 1 is just an important as criterion 2. |
| 3 | Moderate importance | Experience and judgment slightly favor criterion 1 over 2. |
| 5 | Strong importance | Experience and judgment strongly favor criterion 1 over 2. |
| 7 | Very strong importance | Criterion 1 is more important than criterion 2. |
| 9 | Extremely important | The greater importance of criterion 1 over criterion 2 is beyond doubt. |
| 2, 4, 6 and 8 | Intermediate values between the previous ones, when it is necessary to qualify. | |
Aggregation methods
The TOPSIS method is commonly used in multi-criteria decision analysis. In general, the method consists of estimating a positive ideal point and a negative ideal point. The positive ideal point corresponds to a supposed alternative, from the best values reached in each one of the attributes (criteria) of the set of alternatives analyzed. The negative ideal point, on the other hand, corresponds to another supposed alternative containing the worst values obtained in each of the same criteria. Once these references are developed, the weighted distances of each alternative to the best and worst alternative are estimated and their classification is defined (Pavić & Novoselac, 2013; Wang & Xiao, 2011). The basic principle of the method is that the selected alternative must have the shortest distance of the positive ideal solution and the farthest distance of the non-ideal solution, in a geometrical sense.
The method assumes that each attribute has monotonically increasing or decreasing utility. This facilitates the task of locating ideal solutions (positive and negative). Thus, the preference order of the alternatives is produced by comparing Euclidean distances (San Cristóbal, 2012).
Assuming that there is a set of alternatives
1. The normalized matrix is constructed by replacing each x ij value by its normalized value (r ij ) using the following formula:
2. The weighted matrix is constructed by replacing each r ij by its weighted value (a ij ) using the corresponding weightings established (w j ) using the following formula:
3. The best
The minimum components of this positive ideal alternative correspond to criteria to be minimized.
4. Similarly, the negative ideal alternative
The maximum components in this alternative correspond to criteria to be maximized.
5. The vector of distances
6. Similarly, the distances
7. Finally, the relative distances
The highest
The WSM, also known as a weighted linear combination or classification, shows how the values of the multiple criteria options can be combined into a global value. This is done by multiplying the score of the value of each criterion by the weight of that criterion and then adding all those weighted grades; that is to say:
where x ij is the recorded value of alternative i with respect to criterion j, and w ij is the criterion weighting (Afshari, Mojahed, & Yuseff, 2010). This method clearly demonstrates the main concept of multi-criteria evaluation methods: the integration of the values and weightings of the criteria into a single magnitude (Podvezko, 2011). The condition of application of the method is that the criteria must be independent of each other. This method is one of the most frequently used in the multi-criteria approach (Paracchini, Pacini, Calvo, & Vogth, 2007).
Standardization
The range standardization technique was used in order to convert the data generated by each weighting-aggregation combination to the 0-1 scale using the following equation (Aznar-Bellver & Guijarro-Martínez, 2012):
The interval of the standardized values is 0 ≤ x ij ≤ 1. In this standardization procedure, the element of minimum qualification always takes value 0 and the one of maximum takes value 1.
Pareto order
The use of Pareto order in the classification of results, via indexes (classifiers), was proposed by Chakrabarty and Bhattacharjee (2012). We consider p = 4 classifiers, which result from the two weighting methods and the two methods of aggregation. In this study it was classified by the estimation of the following expressions:
where,
The method that has the lowest mean distance value is the one that is identified as the most compatible for the criteria used.
Results and discussion
The grades of the applicants in the selection and evaluation process, per subcriterion, are shown in Table 2. As expected, the resulting grades express conflict between the different sub-criteria considered; namely, generally, each criterion points out different alternatives (applicants) as the best. Likewise, it is observed that practically all the applicants obtained acceptable grades on the knowledge test, while the lower grades of some applicants were obtained on the skills and English tests.
Table 2 Grades obtained by the applicants for the Master program in Forest Science at Universidad Autónoma Chapingo.
| Applicant | Knowledge | Experience | Personality | |||||
|---|---|---|---|---|---|---|---|---|
| EC | EH | EI | Curr | Average | JCR | Interv | Psychom | |
| 1 | 85 | 93 | 0 | 78 | 80 | 60 | 65 | 80 |
| 2 | 90 | 60 | 9 | 82 | 80 | 90 | 90 | 100 |
| 3 | 70 | 75 | 50 | 84 | 85 | 85 | 70 | 100 |
| 4 | 85 | 48 | 25 | 85 | 88 | 80 | 89 | 80 |
| 5 | 85 | 78 | 17 | 86 | 89 | 70 | 79 | 80 |
| 6 | 80 | 73 | 26 | 82 | 86 | 50 | 73 | 80 |
| 7 | 90 | 63 | 71 | 78 | 82 | 65 | 73 | 100 |
| 8 | 85 | 53 | 79 | 83 | 81 | 92 | 69 | 80 |
| 9 | 70 | 78 | 39 | 84 | 86 | 74 | 68 | 66 |
| 10 | 85 | 70 | 99 | 82 | 85 | 88 | 88 | 66 |
| 11 | 70 | 63 | 70 | 80 | 86 | 90 | 90 | 80 |
| 12 | 75 | 73 | 100 | 88 | 90 | 60 | 83 | 80 |
| 13 | 90 | 88 | 50 | 80 | 84 | 76 | 76 | 100 |
| 14 | 80 | 58 | 100 | 84 | 88 | 75 | 84 | 80 |
| 15 | 85 | 78 | 91 | 80 | 81 | 88 | 92 | 80 |
| 16 | 90 | 80 | 57 | 86 | 84 | 40 | 85 | 80 |
| 17 | 80 | 65 | 82 | 82 | 82 | 70 | 89 | 80 |
| 18 | 80 | 73 | 97 | 90 | 95 | 78 | 78 | 80 |
| 19 | 80 | 50 | 100 | 84 | 87 | 90 | 75 | 80 |
EC = knowledge test, EH = skills test, EI = English test, Curr = curriculum, Average = average, JCR = review of article published in a journal recognized in the Journal Citation Reports, Interv = interview, Psicom = psychometric test.
The weighting of criteria and sub-criteria was collected through a survey designed especially for the experts, who have been or are members of the Coordination of Postgraduate Studies in which the selection of applicants was made, which guarantees the experience in the selection process. The results obtained by the AHP and PAM method (Table 3) show significant disparities in the weighting of the criteria and sub-criteria. The case of the sub-criteria knowledge test, to which the weighting method of AHP assigned twice the weight compared to the PAM method. Another case that calls attention is the sub-criteria interview of the Coordination, to which PAM assigns a weight 3.5 times greater than AHP.
Table 3 Weightings of the analytical hierarchical process (AHP) and the point allocation method (PAM) in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.
| Criteria | Sub-criteria | Weighing AHP | Weighing PAM | ||||
|---|---|---|---|---|---|---|---|
| Criteria | Sub-criteria | Final | Criteria | Sub-criteria | Final | ||
| Knowlegde | Knowledge test | 0.76 | 56.30 | 42.49 | 0.47 | 45.00 | 21.38 |
| Skills test | 30.70 | 23.16 | 31.25 | 14.84 | |||
| English test | 13.10 | 9.86 | 23.75 | 11.28 | |||
| Experience | Curriculum vitae | 0.18 | 22.50 | 4.00 | 0 .25 | 35.00 | 8.75 |
| Average grades | 65.90 | 11.72 | 43.75 | 10.94 | |||
| JCR exercise | 11.70 | 2.08 | 21.25 | 5.31 | |||
| Personality | Cordination interview | 0.07 | 86.30 | 5.78 | 0.27 | 73.75 | 20.28 |
| Psychometric test | 13.70 | 0.92 | 26.25 | 7.22 | |||
JCR: Journal Citation Reports
The application of the two aggregation methods, TOPSIS and WSM, using in each case the two weightings, PAM and AHP, resulted in four weighting-aggregation combinations. Table 4 shows the values for the 19 candidates evaluated.
Table 4 Values obtained in the four combinations of weighting-aggregation methods used in the selection of applicants for the Master's Program in Forest Sciences of the Universidad Autónoma Chapingo.
| Applicant | TOPSIS-AHP | TOPSIS-PAM | WSM-AHP | WSM-PAM |
|---|---|---|---|---|
| 1 | 0.5323(48) | 0.3670(4) | 0.0511(36) | 0.0465(0) |
| 2 | 0.4030(18) | 0.3740(6) | 0.0491(17) | 0.0499(29) |
| 3 | 0.4539(30) | 0.4814(29) | 0.0494(20) | 0.0505(33) |
| 4 | 0.3239(0) | 0.3625(3) | 0.0473(0) | 0.0487(18) |
| 5 | 0.4973(40) | 0.3873(9) | 0.0515(39) | 0.0497(27) |
| 6 | 0.4337(25) | 0.3482(0) | 0.0491(17) | 0.0473(7) |
| 7 | 0.5831(60) | 0.5871(53) | 0.0539(62) | 0.0530(54) |
| 8 | 0.4953(40) | 0.5544(45) | 0.0516(40) | 0.0517(43) |
| 9 | 0.4405(27) | 0.4095(14) | 0.0485(12) | 0.0475(9) |
| 10 | 0.6894(85) | 0.7530(89) | 0.0572(93) | 0.0575(92) |
| 11 | 0.4307(25) | 0.5856(52) | 0.0495(21) | 0.0528(52) |
| 12 | 0.6196(68) | 0.7234(83) | 0.0550(73) | 0.0562(80) |
| 13 | 0.7098(89) | 0.5954(54) | 0.0570(92) | 0.0549(70) |
| 14 | 0.5541(53) | 0.6773(72) | 0.0538(62) | 0.0556(75) |
| 15 | 0.7556(100) | 0.8022(100) | 0.0579(100) | 0.0585(100) |
| 16 | 0.6900(85) | 0.6103(58) | 0.0561(83) | 0.0539(62) |
| 17 | 0.5640(56) | 0.6701(71) | 0.0532(56) | 0.0546(67) |
| 18 | 0.6727(81) | 0.7342(85) | 0.0567(89) | 0.0571(88) |
| 19 | 0.5031(42) | 0.6190(60) | 0.0522(47) | 0.0540(62) |
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method. Values in parentheses converted to scale 0-100.
The values generated by each combination of methods were converted to scale 0-100, using the standardization technique by ranking (Table 4, values in parentheses), which allowed to identify the applicants that reached or surpassed 80 points in the global qualification. In this way, we find that combinations TOPSIS-AHP and WSM-AHP assign to five applicants a score greater than or equal to 80, while with TOPSIS-PAM and WSM-PAM only to four applicants were assigned (Table 5).
In this analysis it is emphasized that the four combinations include three applicants in their selection; 10, 15 and 18. The applicant number 15 is the one that best satisfies the admission profile since he appears in the first place of the four selections. The second applicant is the number 10, as he appears in second place in three selections and fourth in the other, while applicant number 18 is included in different orders (Table 5). Another important result is the fact that combinations TOPSIS-AHP and WSM-AHP select the same applicants, but in different order. Similarly, combinations TOPSIS-PAM and WSM-PAM select the same applicants, but in exactly the same order.
Table 5 Selection of the best candidates for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo, according to a combination of weighting-aggregation methods.
| Classification | Applicant/method | |||
|---|---|---|---|---|
| TOPSIS-AHP | TOPSIS-PAM | WSM-AHP | WSM-PAM | |
| 1 | 15 | 15 | 15 | 15 |
| 2 | 13 | 10 | 10 | 10 |
| 3 | 16 | 18 | 13 | 18 |
| 4 | 10 | 12 | 18 | 12 |
| 5 | 18 | - | 16 | - |
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.
Pareto order as a technique to identify the weighting-aggregation combination most compatible with the criteria used, according to the total distance or calculated average distance, shows that the aggregation method WSM-AHP has the minimum average distance (Table 6). This method should be considered as the most compatible with the other three built based on the criteria included.
Table 6 Total distances and averages, according to the weighting-aggregation method, for the identification of the combination most compatible with the criteria evaluated in the selection of applicants for the Master’s program in Forest Sciences of the Universidad Autónoma Chapingo.
| Combination-Methods | Total distance | Averge distance |
|---|---|---|
| TOPSIS-AHP | 548 | 182.66 |
| TOPSIS-PAM | 428 | 142.66 |
| WSM-AHP | 336 | 112 |
| WSM-PAM | 432 | 144 |
TOPSIS: Technique for order of preference by similarity to ideal solution, AHP: Analytical hierarchical process, PAM: Point allocation method, WSM: weighted sum method.
The fact that the resulting combination is WSM-AHP has the advantage that the aggregation method WSM is easy to apply. On the other hand, the weighting method AHP could present difficulties of operation if the number of criteria to be weighed is increased; however, this possible limitation has not affected the results obtained in the present study.
The order of Pareto determines that the index that generates an order whose distance is the minimum to the ordinances of the other indexes, is the one of greater compatibility with the criteria of the program. This is because each of the indices formed with different combinations, considers the criteria of the program differently, but they are always the same. The association of these ideas come from the following. On the one hand, the allocation of weights via AHP or PAM is done according to the interpretation that the experts have of the criteria of the program and, being participants of the same, they are the ones who know them best and establish through its academy. On the other hand, the aggregation methods, incorporates the assessments of the criteria used in the selection, so that, in this respect, the index also includes them. In other words, each index treats differently the criteria it uses and the difference is given by how they are weighted and aggregated, taking into account the knowledge of the experts and the algebraic properties of the aggregation methods. In this way, each index represents a different way of treating the information, producing a different order.
When you select the index whose order has a minimum distance with the order of the rest of the indexes tested, it is certainly the most compatible among those tested with the criteria of the program. Of course, if more weighting and aggregation methods were tested, the more compatible index would be more robust; however, it is important to keep the options between the most frequent and simple, which are usually the ones that have produced better results, as well as limiting the number of them.
Conclusions
Multi-criteria methods represent a good option to properly consider the large amount of information generated in a selection process. The weighting method determined the applicants to choose, since regardless of the aggregation used, the selected students were the same if the weights of the criteria were the same. The combination of WSM-AHP methods was the most compatible with the other combinations used, as it induces a classification whose distance to other classifications is minimal. This indicates that the considerations of the experts to weight (AHP) and the technique used to add (WSM) are more compatible than the rest of the combinations. Since some of the criteria considered are qualitative and susceptible to be evaluated by linguistic tags, a possible direction in later studies is the use of fuzzy logic. During the last 10 years, students in the Master’s program in Forest Science were chosen using the point allocation method (PAM) in the criterion weighting phase, and the weighted sum method (WSM) in the aggregation. The results obtained through this selection were very good, so they add certainty and offer the opportunity to specify the procedure used so far.
