Multicriteria decision aid application to a student selection problem

Multicriteria decision aid application to a student selection problem

Juan Carlos Leyva López

Departamento de Posgrado, Universidad de Occidente, Sinaloa – México, [e-mail]

ABSTRACT

The selection of students is a complex decision making process, in which multiple selection criteria often need to be considered and where subjectiveness and imprecision are usually present, resulting that fuzzy and imprecise data should be used. This paper formulates the student selection process as a multicriteria decision analysis problem, concretely as a ranking problem, by using the ELECTRE III methodology to construct a fuzzy outranking relation, and then a genetic algorithm to exploit it and to obtain a ranking in decreasing order of preference. An empirical study of a real selection problem in the Universidad de Occidente in Mexico is presented. After performing calculations and obtaining the final ranking of compared alternatives, a sensitivity analysis was carried out. The study was supported with a new decision aid system for rank a finite set of multicriteria alternatives.

Keywords: multicriteria decision analysis; ranking problem; genetic algorithms; student selection.

1. Introduction

Selecting students from competing applicants is a complex decision making process, which often requires a comprehensive evaluation of the applicant's performance. Multiple selection criteria should be simultaneously considered, and subjective assessments are usually present, resulting in fuzzy and imprecise data.

Statistical procedures, such as discriminant analysis and regression analysis are traditionally used for predicting the potential academic success of the applicant (Graham, 1991). The predictive validity study may help make admission or selection procedures more efficient and effective (Powers & Lehman, 1983; Dobson et al., 1999; Lievens & Coetsier, 2002). However, the selection criteria used in higher education admission processes varies widely among programs and no consistent conclusions can be reached on the predictive values of these criteria (Wilson, 1999). This may partly be due to the fact that the predictive validity of the selection instruments is not in itself sufficient for an assessment of the validity of a selection, although it can be a critical factor (Wolming, 1999). In this paper, prediction is not the stated purpose for the student selection problem. The selection of applicants is made on the grounds of the candidates' merits (performance evaluation) assessed by an interview process and his/her academic background, based on a given set of criteria in accordance with the requirements of the academic program of Master in Science. The artificial neural networks have also attracted the attention the last years (Flitman, 1997). However, the effectiveness of these methods is sometimes questioned (Hardgrave et al., 1994), due to the sophistication of the decision process, the rough assumptions required, and the level of accuracy achieved. They are, in particular, unable to adequately handle the subjectiveness and imprecision of the decision process and often impose a high cognitive effort on the decision maker (DM).

Alternatively, multicriteria analysis (MA) is widely used for selecting or ranking alternatives in relation to multiple criteria (Roy, 1996; Vanderpooten, 1990). In line with the multi-dimensional characteristics of the student selection process, MA provides an effective framework for solving the problem and, particularly, the approach based on fuzzy outranking relations, is adequate for dealing with situations in which imprecision and subjectiveness are present (Rogers et al., 2000).

The application of fuzzy set theory in MA provides an effective means for modeling the subjectiveness and imprecision (Fodor & Roubens, 1994). When the Decision Maker is a group of experts, Carlsson et al. (1997) illustrate the applicability of Ordered Weighted Averaging (OWA) operators to a doctoral student selection problem. Tremendous efforts have been spent and significant advances have been made, leading to the development of diverse methodologies along the lines of the outranking relation and the utility or value theory (Ribeiro, 1996). However, no single approach is exempt from criticism about its overall performance and practical use in tackling real problems. The approaches based on the aggregation model of preferences may be complex and hard to use. For the approaches developed in the context of multi attribute value theory, the ranking of the fuzzy value remains a challenging issue, as it is not straightforward but when the value function is reached, the prescription is immediately deduced from the aggregation preferences process (Vincke, 1998). In this approach, Yeh (2003) introduced an interesting empirical validity procedure to deal with the selection of multiattribute decision making methods for scholarship student selection. When the aggregation model of preferences is based on the outranking approach, a special treatment is required, but some non-rational violations of the explicit global model of preferences could happen. In outranking methods, we can distinguish two phases: aggregation and exploitation. The aggregation process corresponds to the operation, which transforms the marginal evaluations of separate criteria into a global outranking relation between every pair of alternatives, which is generally not transitive nor complete. The exploitation process deals with the outranking relation in order to clarify the decision through a partial or total preordering reflecting some of the irreducible indifferences and incomparabilities (Fodor & Roubens, 1994).

To some extent fuzzy relations offer a compromise between value functions and preference relations; their power of expressiveness is far beyond the one of value functions since they are good models for such phenomena as non-transitivity and incomparability. ELECTRE III, PROMETHEE and other methods for decision aid (e.g. Roy, 1990; Fodor & Roubens, 1994) build and exploit a fuzzy outranking relation.

Let A be the set of alternatives or potential actions and let us consider a fuzzy outranking relation S_A^s defined on A X A; this means that we associate with each ordered pair (a, b) a real number s (a, b) (0 < s(a, b) < 1) reflecting the degree of strength of the arguments favoring the crisp outranking aSb. The exploitation phase transforms the global information included in S_A^s into a global ranking of the elements of A. Usually; three different ways are used (Fodor & Roubens, 1994):

a: transform S_A^s into another valued relation R which presents some interesting property needed for ranking purposes, i.e. transitivity,

b: determine a crisp binary relation close to S_A^s, which presents crisp properties needed for ordering,

c: use a ranking method to obtain a score function.

Point a includes the process of finding the transitive closure or the intersection of traces. Point c is most commonly used in classical procedures like ELECTRE III and PROMETHEE. But the main difficult consists in finding reasonable ways of dealing with the intransitivities without losing too much of the contents of the outranking relation. In this sense, the methods included in points a and b lose information coming from S_A^s when exploit a not so close transitive valued relation R, or a crisp binary relation with desirable properties for ranking purposes. On the other hand, the methods based in score functions do not perform well in presence of irrelevant alternatives or in case of complex graphs with many circuits. Non-rational situations could happen when the prescription is constructed. Most significant is the following: Suppose that a and b are two actions such that s (a, b) > l and s (b, a) < l-b (b>0); if l³c and b³t (c and t representing consensus and threshold levels respectively), we should accept that "a outranks b" (a S^lb) and "b does not outrank a" (b nS^la); in this case the global preference model captured in outranking relation is giving a presumed preference favoring a. However, a score function or other similar method (based on flow of outranking or "distillation" process) could lead to a final ordering in which b is ranked before. ELECTRE and PROMETHEE methods do not have a way to minimize this kind of irregularity. In Leyva & Fernandez (1999) and Fernandez & Leyva (2004) a new method based on genetic algorithms and multiobjective optimization, which allows to exploit a known fuzzy outranking relation is introduced, with the purpose of constructing a prescription for ranking problems. The problem of obtaining the final ranking is modeled with multiobjective combinatorial optimization and the genetic algorithm approach rests on the main idea of reducing differences between the global model of preferences and final ranking.

In this paper, is formulated the student selection process as a MA problem, and is used a fuzzy MA approach for solving a real student selection problem at the Master in Management Information Systems (MMIS) of the Universidad de Occidente (U de O) in Mexico. The ELECTRE III – Genetic Algorithm approach can effectively calculate the overall performance of individual applicants, construct an aggregation model of preferences and exploit it to recommend to the DM a ranking of the applicants.

Section 2 describes the ELECTRE III method and the genetic algorithm, followed in Section 3 by an explanation of the student selection situation faced by the MMIS and, also in this section, is provided an empirical study to illustrate, one more time, the applicability of the approach. In Section 4, is carried out a sensitivity analysis of the final result. On these backgrounds, in Section 5 are presented the results and a brief discussion of the selection process. Finally in Section 6 are presented the conclusions.

2. The (ELECTRE III – Genetic Algorithm) Methodology

2.1 The ELECTRE method

As part of a philosophy of decision aid, ELECTRE (in its various forms) was conceived by Bernard Roy (1990) in response to deficiencies of existing decision making solution methods. Roy's philosophy of decision aid is well exposed in Roy (1996) (see also Roy (1990) and Vanderpooten (1990) for basic introductions to these methods); moreover, of the different versions of ELECTRE which have been (I, II, III, IV, IS and TRI), only is used the method specifically referred to as ELECTRE III. All methods are based on the same fundamental concepts, as explained subsequently, but they differ both operationally and according to the type of decision problem. Specifically, ELECTRE I is designed for selection problems, ELECTRE TRI for assignment problems and ELECTRE II, III, and IV for ranking problems. ELECTRE II is an old version; ELECTRE III is used when it is possible and desirable to builds fuzzy outranking relationships and quantifies the relative importance of criteria and ELECTRE IV when this quantification is not possible (Roy & Bouyssou, 1993).

A number of factors influenced the specific selection of the ELECTRE III method for the Postgraduate student selection problem. Firstly, in Leyva (2000) is presented a genetic algorithm to exploit a fuzzy outranking relation and is interesting to show, one more time, and with a real world application the functionality between the pair (ELECTRE III, Genetic algorithms). Secondly, ELECTRE was originally developed by Roy to incorporate the fuzzy (imprecise and uncertain) nature of decision-making, by using thresholds of indifference and preference. This feature is appropriate for solving this problem. A further feature of ELECTRE, which distinguishes it from many multiple criteria solution methods, is that it is fundamentally non-compensatory. This means, in particular that, good scores on other criteria cannot compensate a very bad score on a criterion. Another feature is that ELECTRE models allow incomparability. Incomparability, which should not be confused with indifference, occurs between some alternatives a and b when there is no clear evidence in favour of some kind of preference or indifference. Finally, the choice of ELECTRE III was also influenced by successful applications of the approach (for example: Roger et al., 2000; Hokkanen & Salminen, 1997; Al-Kloub et al., 1997; Georgopoulou, 1997; Roger & Bruen, 1998).

Two important concepts that underline the ELECTRE approach, thresholds and outranking, will now be discussed. Assuming that there exist defined criteria, g_j, j = 1, 2,...r and a set of alternatives A. Traditional preference modeling assumes the following two relations holded for two alternatives (a,b) Î A :

aPb (a is preferred to b) Û g(a) > g(b)

aIb (a is preferred to b) Û g(a) = g(b)

In contrast, ELECTRE methods introduce the concept of an indifference threshold, q, and then the preference relations are redefined as follows:

aPb (a is preferred to b) Û g(a) > g(b) + q

aIb (a is preferred to b) Û ½g(a) — g(b)½< q

While the introduction of this threshold goes some way toward incorporating how a decision maker actually does feel about realistic comparisons, a problem remains. There is a point at which a decision maker changes from indifference to strict preference. Conceptually, there is a good reason to introduce a buffer zone between indifference and strict preference, an intermediary zone where a decision maker hesitates between preference and indifference. This zone of hesitation is referred to as a weak preference; it is also a binary relation like P and I above, and is modeled by introducing a preference threshold, p. Thus, we have a double threshold model, with an additional binary relation Q that measures weak preference. That is:

aPb (a is strongly preferred to b) Û g(a) – g(b) > p

aQb (a is weakly preferred to b) Û q < g(a) - g(b) < p

aIb (a is indifferent to b; and b to a) Û ½g(a) – g(b)½< q

The choice of thresholds intimately affects whether a particular binary relation holds. While the choice of appropriate threshold is not easy, in most realistic decision making situations there are good reasons for choosing non-zero values for p and q.

Note that we have only considered the simple case where thresholds p and q are constants, as opposed to being functions of the value of the criteria; that is, the case of variable thresholds. While this simplification of using constant thresholds aids the exposition of the ELECTRE method, it may be worth using variable thresholds, in case where the criterion having larger values may lead to larger indifference and preference thresholds.

Using thresholds, the ELECTRE method seeks to build an outranking relation S. aSb means that according to the global model of DM preferences, there are good reasons to consider that "a is at least as good as b" or "a is not worse than b." Each pair of alternatives a and b is then tested in order to check if the assertion aSb is valid or not. This give rise to one of the following four situations:

aSb and not(bSa); not(aSb) and bSa; aSb and bSa; not(aSb) and not(bSa).

Observe that the third situation corresponds to indifference, while the fourth corresponds to incomparability.

The test to accept the assertion aSb is implemented using two principles:

i) A concordance principle which requires that a majority of criteria, after considering their relative importance, is in favor of the assertion – the majority principle, and

ii) A non-discordance principle, which requires that within the minority of criteria, which do not support the assertion, none of them is strongly against the assertion – the respect of minorities' principle.

The operational implementation of these two principles is now discussed, assuming that all criteria are to be maximized. We first consider the outranking relation defined for each of the r criteria; that is,

aS_jb Means that "a is at least as good as b with respect to the j^th criterion," j=1,2, ,r

The j^th criterion is in concordance with the assertion aSb if and only if aS_jb. That is, if g_j(a) > g_j(b) - q_j. Thus, even if g_j(a)is less than g_j(b) by an amount up to q_j, it does not contravene the assertion aS_jb and therefore is in concordance.

The j^th criterion is in discordance with the assertion aSb if and only if bP_ja. That is, if gj(b)>(a) + p_j. That is, if b is strictly preferred to a for criterion j, then it is clearly not in concordance with the assertion that aSb.

With these concepts it is now possible to measure the strength of the assertion aSb. The first step is to develop a measure of concordance, as contained in the concordance index C (a, b), for every pair of alternatives (a,b) Î A . Let k_j be the importance coefficient or weight for criterion j. We define a valued outranking relation as follows:

Where

Thresholds and weights represent subjective input provided by the decision maker. Weights used in the non-compensatory ELECTRE model are quite different from weights used in others, compensatory, decision modeling approaches such as the decision analytic approach (SMART) of Edwards (1997). In the decision analytic models, for example, weights are substitution rates and assess relative preference among criteria. Weights in ELECTRE are "coefficients of importance" and, as Vincke (1992) points out, they are like votes given to each of the criterion "candidates." Roger et al. (2000) review existing weighting schemes for ELECTRE and provide a useful discussion of the weighting concept in ELECTRE. Care also needs to be taken in determining threshold values, which must relate specifically to each criterion and reflect the preferences of a decision maker. Procedures for choosing appropriate threshold values are addressed by Roger & Bruen (1998).

Thus far, no consideration has been given to the discordance principle. In the concordance index, we have, in a manner of speaking, a measure of the extent to which we are in harmony with the assertion that a is at least as good as b. But what disconfirming or "disharmonious" evidence do we have? In other words, is there any discordance associated with the assertion aSb? To calculate discordance, a further threshold called the veto threshold is defined. The veto threshold v_j, allows for the possibility of aSb to be refused totally if, for any one criterion j, g_j(b) > g_j(a) + n_j. The discordance index for each criterion j, d_j(a,b) is calculated as:

For each pair of alternatives (a,b) Î A, there are now a concordance and a discordance measure. The final step in the model building phase is to combine these two measures to produce a measure of the degree of outranking; that is, a credibility index which assesses the strength of the assertion that "a is at least as good as b." The credibility degree for each pair (a,b) Î A is defined as:

This formula assumes that, if the strength of the concordance exceeds that of the discordance, then the concordance value should not be modified. Otherwise, we are forced to question the assertion that aSb and modify C (a,b) according to the above equation. If the discordance is 1.0 for any (a,b) Î A and any criterion j, then we have no confidence that aSb; therefore, S (a,b) = 0.0. The credibility matrix for this application is explained in Table 6.

This concludes the construction of the outranking model. The next step in the outranking approach is to exploit the model and produce a ranking of alternatives from the fuzzy outranking relation. Our approach for exploitation is using a genetic algorithm-based heuristic method (Leyva & Fernandez, 1999), which is explained, briefly in the next section.

2.2 The Genetic Algorithm for Deriving Final Ranking

In this section are explained some elements of the genetic algorithm which allows us to exploit a known fuzzy outranking relation with the purpose of constructing a prescription for the multi criteria ranking problem.

2.2.1 Encoding the solutions and the fitness function

A potential solution of a ranking problem is represented as an ordinal representation. In general, a potential solution is a ranking of the set of actions by decreasing order of preference. These actions (known as genes in GA's) are joined together forming a string of values (known as chromosome). Any symbol in this string is refereed to as an allele (Goldberg, 1989; Michalewicz, 1996). The chromosome is represented as the string of n-ary alphabet where n is the number of actions into the decision problem. In such representation, each action is coded into n-ary form. Actions are then linked together to produce one long n-ary string or chromosome. An action coded with a_ki value in the i-th entry of the string means that the action coded with a_ki value is ranking in the i-th place of the ordering and aki is preferred to a_kj if i < j, where ak_iÎ A = {a₁, a₂, , a n}, i=1, 2, , n, and [k₁, k₂, , kn] is a permutation of [1, 2, , n]. Each individual is associated with a number l (0 < l < 1), which will be connected with the credibility level of a crisp outranking defined on the set of genes. The fitness of an individual with credibility level l is calculated according to a given fitness function. The approach for defining individual's fitness involves separating the single fitness measure into two, one is called fitness and the other is called unfitness. We chose the fitness function f of an individual p with credibility level l as follows:

Let p = a_k1 ak₂ ak_n be the schematic representation of an individual's chromosome and suppose that given ak_i and ak_j, two actions such that s (ak_i, ak_j) > l and s (ak_j, ak_i) < l-b (b>0, representing a threshold level), we accept that "ak_i outranks ak_j" (ak_i S^l ak_j) and "ak_j does not outrank ak_i" (ak_j nS^l ak_i). In this case, into the crisp outranking relation generated by l, SA^l, a presumed preference favoring ak_i, holds. Then:

f(p) = ç{ (ak_i, ak_j) :ak_i nS ak_j and ak_j nS ak_i i = 1,2, ..., n-1, j = 2,3, ..., n, i<j } ç where [k₁, k₂, ..., k_n] is a permutation of [1,2,...,n].

f(p) is the number of incomparabilities between pairs of actions (ak_i, ak_j) into the individual p = ak₁ ak₂ ak_n in the sense of the crisp relation S_A^l. Note that the quality of solution increases with decreasing fitness score.

The unfitness u of an individual p measures the amount of unfeasibility (in relative terms) and we chose to define it as:

u(p) = ç{ (ak_i, ak_j) : ak_i S ak_j and ak_j nS ak_i; i = 1,2, ..., n, j = 1,2, ..., n, i>j } ç.

u(p) is the number of preferences between actions into the individual p which are not "well-ordered" in the sense of S_A^l.

An individual P is feasible if u(p) = 0 and infeasible if u(p) > 0. Defining the unfitness taking the zero minimum value if and only if the solution is feasible seems a natural approach. Each individual p can then be represented by a triad of values f, u and l.

We are interested in:

i) individuals whose unfitness function value is equal to zero. This assures us that the ordering represented by the individual is transitive; this is one of two characteristics that should be exhibited by all prescription (solution) of ranking problems (Vanderpooten, 1990).

ii) individuals whose fitness function value is equal (or near) to zero. This objective improves the comparability of S on A.

iii) individuals whose credibility level l is near to 1. This indicates us that the ordering represented by the individual with credibility level l is more trusty whenever the fitness and unfitness function values are zero or near to zero. In practice, the requirement connected to fitness function does not permit that l values approach to 1 because in this case we could have many incomparable genes.

Then, we use a genetic search for solving the multiobjective optimization problem

Min u, f, Max l

R_s , l Î[0,1] l > l₀

Where R_s is a strict total order of A.

2.2.2 Crossover and mutation operators

The crossover operator takes genes from each parent string and "combines" them to create a child string. The main reason is that by creating new strings from fit parent strings, new and promising areas of the search space will be explored. Many crossover techniques exist in the literature (e.g. Ordoñez & Valenzuela, 1992), but, when working with ordinal (permutation) encoding, it is necessary to create both crossover and mutation operators that are specifics to this form of encoding. The main difficulty encountered when using non-standard representations is the design of a suitable crossover operator, which must combine relevant characteristics of the parent solutions into a valid offspring solution. In this paper we make use of the crossover operator UX2 (Union Crossover #2) first introduced in Poon & Carter (1995).

The mutation operator is applied to the child string generated after crossover operator is finished. It works by interchanging two pairs of randomly chosen genes (actions), at each iteration under certain rules, in an individual. Mutation is generally seen as a background operator, which provides a small amount of random search. It also helps to keep against loss of valuable genetic information by reintroducing information, which was lost due to premature convergence, and thereby, expanding the search space.

2.2.3 Parent selection method

Parent selection is the task of assigning reproductive opportunities to each individual in the population based on its relative fitness. A commonly used method is binary tournament selection. In a k-ary tournament selection, k individuals are chosen randomly from the population, and the most fitted individual is then allocated in a reproductive trial. In order to produce a child, two k-ary tournaments are held, each of which produces one parent string. These two parent strings are then combined to produce a child.

The tournament selection criterion may be based on either the fitness or the unfitness of the individuals as defined previously. The difficulty with these criteria is that the individuals who have lower fitness scores (more fit) generally have higher unfitness score (more infeasible) and vice versa at the early stages of the GA. Thus if we favor selection of more fitted individuals, we are likely choosing parents which are mostly infeasible. This will not help the GA in finding feasible solutions. On the other hand, if we favor selection of less infeasible individuals, we might select individual, which are less fitted. Whilst feasibility may improve, the solution quality could suffer. To avoid this, we developed a Complement Selection (CS) method for selecting parents that attempts to improve comparability as well as feasibility.

The CS method is designed specifically for our problem and it takes into account the credibility level l of the candidate parents. In a complement selection, a parent P_i (P_j) is first (second) selected using a k-ary tournament based on the unfitness (fitness) function and credibility level; the rule is as follow:

We selected the individual P_i (P_j) which has lowest unfitness (fitness) score and its credibility level lP_i(lP_j) is greater than lP or lK, where lP is the average credibility level of the population and lK is the average credibility level of the tournament. If i (j) is not unique, then we select the individual with higher credibility level score. If there is not such i (j) we tried the rule with the individual P₁ , which has next lower unfitness (fitness) score; continue until the rule is satisfied.

The logic here is that we would like the two parents together to cover as few amount of preference violations and incomparabilities between actions as possible, i.e. a low u(P_i) and f(P_j)) with as high credibility level values lP_i and lP_j as possible.

2.2.4 Population replacement scheme

This GA part defines how new chromosomes will be put into the existing population. In this algorithm the current population is updated continuously during the mating process. After that the child has been produced through the GA operators, it will replace the "less fitted" member of the population. The average unfitness and/or fitness of the population will improve if the child solution has lower unfitness and fitness scores than those of the solutions being replaced. In this algorithm, every new offspring is replacing the worst chromosome in the population. We utilize the following approach in order to decide which is the worst individual in the population: Firstly we sort the population, in the present generation, by decreasing order of unfitness value – Criterion: If the unfitness value of the individual posed in j place (u(P_r)) is less than the unfitness value of the individual posed in j-1 place (u(P_s)) or if u(P_r) = u(P_s) and f(P_r) < f(P_s) then we interchange the individuals, otherwise we do not interchange (in case of tie, the fitness value is used in order to decide); in this way last individual is the worst. Each time that we replace a new offspring by the worst individual, the new population is sorted with the same criterion.

3. Postgraduate Student Selection: A Mexican Case

This applicant ranking problem is, like many decision problems, challenging because there is no single criterion that adequately captures the performance of each applicant; in other words, it is a multicriteria problem.

The Master of Science in Management Information Systems (MMIS) is a small postgraduate program established by the U de O in Mexico. The MMIS program is offered as a part-time study mode. It means that the student may have another activities like working as a full-time or part-time worker. As an important part of the selection process, the MMIS program needs to know and have a dedication agreement with each one of its students.

Until the past generation at MMIS, the selection process had been carried out in an ad hoc manner. DM (which in this case is a committee formed by postgrade academics) made its decisions mainly based on the academic performance of the applicants on an introductory course on Programming Language C/C++, the DM used also its experience, intuition and knowledge with the information available. A structured approach capable of producing consistent decision outcomes through adequately handling the inherent imprecision and subjectiveness is obviously desirable.

For the postgraduate student selection problem, the decision alternatives are clearly defined as each one of the applicants. Each applicant or alternative is characterized by her attributes, which are then related to the criteria.

3.1 Defining the criteria family

According to Bouyssou (1990) the criteria family should be legible (containing sufficiently small number of criteria), operational, exhaustive (containing all points of view), monotonic and non-redundant (each criterion should be counted only once). These rules provide a coherent criterion family.

In our approach, in order to define the criteria, the analyst worked close to the DM. He had in mind that selecting a qualified applicant could only be the first parameter in the planning procedure. A series of other parameters should be taken into account, being the most important:

• The size of the academic staff

• The experience gained in past selection processes

• The correspondence between the number of research projects and the number of accepted applicant to the program

• The finance security

• The graduated index

• The developing program of the MMIS

• The existence of the necessary infrastructure

• Other national targets such as employment, scientific political, etc.

• The academic and social impacts in our environment

• The technical and financial risk.

This framework imposed the use of 6 criteria as presented in an analytical way in Table 1. The DM approved the criteria family. However, the lack of time on the selection process forced us – at this stage of the selection process – to ignore one criterion where evaluation was not possible for all applicants. As a result, 5 criteria were finally applied (marked with an * in Table 1).

Most criteria were decomposed into simpler and well-defined attribute measures, which were then combined to produce a score for each applicant for each criterion. The score for the intelligence criterion (C1), the academic performance criterion (C2), the English proficiency criterion (C4), and the responsibility performance criterion (C5) were scaled from 0 to 10, the time spent in studying (C3) was scaled from 0 to 50; the units of these criteria are not meaningful outside this case study. The actual scores for each criterion are defined by a number of attributes that together describe the performance of the applicant. For example, the intelligence introduced by a particular applicant may influence the grade on programming language and mathematical subjects. In each case, a logical o arithmetic formula is defined to produce the score for each criterion. This input, where each applicant is assessed across each criterion, produces a matrix of performances. Table 3, into the Subsection 3.2, provides the performance matrix, for twenty-one applicants and five criteria.

3.2 The real world application

The next instance of the ranking problem discusses an empirical study of the following real selection problem sufficiently described above.

The U de O will offer another generation of the Master of Science in Management Information Systems and a selection process of candidates will be held. The problem is to identify the best possible candidates. After that, the DM saw many interested persons to enroll in the program, they finally accepted to compete for a place 21 applicants labeled in this application as A1, A2, , A21. The study was supported with a new decision aid system for rank a finite set of multicriteria alternatives, developed by our working group and whose main window is presented in Figure 1.

The DM has made an adequately comprehensive description of each applicant available. For this application, the 5 following criteria and its scale are formulated in the Table 2.

As mentioned in Subsection 2.1, three are the main inputs of the ELECTRE method.

3.2.1 The performance matrix

All applicants were evaluated using the criteria and scale showed in the Table 2. All criteria were treated as quantitative ones. A 21x5 matrix was produced. Table 3 provides the performance matrix, for twenty-one applicants and five criteria. Figure 2 illustrates part of the performance matrix.

3.2.2 The thresholds

The DM was supported in the definition of its preferences and uncertainties through the q, p, and v thresholds for all criteria, with the following guidelines:

Agreeing with Rogers & Bruen (1998) we did not propose a specific relation between q and p values. As far as the veto threshold v is concerned, we suggested that veto should be the most important factor for the most important criteria. As a result, in the most important criteria (the ones with greater weight values) v should have been closer to p than in the case of the least important ones. In this way, we ensured that is difficult for a not important criterion to exercise veto against important criteria. It has been assumed that the thresholds values shall be constant for all criteria (a=0). Thus, the thresholds value reflects the value of b coefficient.

C1- Intelligence

This was the most important criterion. The DM wanted to accept mainly applicants consistent with the MMIS objectives. As a result, the indifference threshold q was small, with a value of 0.2 while the preference threshold was p=0.5. In a similar concept, v was only twice as p, since the DM did not want to accept an applicant not consistent with the MMIS objectives, in the place of a consistent one.

C2- Academic performance

In any case, academic performance of the applicant in their previous studies cannot help from being an important decision parameter. Alike criterion C1, the DM set q=0.2 and p=0.5. Considering that a distillation of applicants was made previously, veto was easy to happen requiring rather small differences. We set v=1.0.

C3- Time spent in studying

Since the program is offered as a part-time study mode, the DM wanted to assure a minimum dedication of applicant's time. This criterion was allowed to have rather small indifference and weak preference zones, we set q=4 and p=9. However, considering the intrinsic subjectivity of this criterion, it could not easily exercise a veto. Threshold v was set to 40.

C4- English proficiency

English proficiency is ultimately an important decision factor. However, we set the veto threshold to 6, a rather high value, in order not to easily exclude from our short-list of applicants. q=1, p=1.5, v=6.0.

C5- Responsibility performance

Responsibility performance is of great value in the selection process. As a result q=0.5, p=1.0. However, in any case this is difficult to evaluate. As a result the veto threshold was rather high, v=7.0.

The threshold values are summarized in Table 4.

Figure 3 shown the threshold values of a criterion.

3.2.3 The weights (relative importance of the criteria)

The DM was supported in the definition of the 5 criteria weights, as shown in Table 5. Personal Construct Theory – PCT as suggested by Rogers et al. (2000) was used for the weight definition.

3.3 Calculations and the final ranking

The input data used in calculations are the values presented in Table 3 (performances of the alternatives). All compared alternatives and criteria have been taken as the foundation for the calculation. Information about the preferences of the decision maker, namely, the values of indifference, preference and veto thresholds for each criterion (defining a and b coefficients for thresholds functions), and values of relative importance of criteria have been presented in Table 4 and Table 5. The decision maker's experience constituted a basis for evaluation of the alternatives at hand, and was implemented by providing such information about the decision maker's preferences, obligatory in the chosen computation method. It has been assumed that the thresholds values shall be constant for all criteria (a=0). Thus, the threshold value reflects the value of b coefficient. The values of relative importance of the criteria indicate that what is most important for the decision maker is: the intelligence criterion and the academic performance.

The computation has been made on the input data (Table 3), and on the information about preferences of the decision maker (Table 4 and Table 5), using the ELECTRE III method.

According to the additional information pointed out before, we applied ELECTRE III to construct a fuzzy outranking relation. Table 6 shows the credibility matrix obtained.

After that, we used the genetic algorithm presented in Leyva & Fernandez (1999), (see the Subsection 2.2) for exploiting the outranking relation and deriving a final ranking of the alternatives in decreasing order of preferences.

The computation in the genetic algorithm was realized with the following parameters: 100 trials of the GA heuristic (each one with a different random seed) were generated. We worked with groups of 25 trials, which finished when {400, 350, 300, 300} populations had been generated. The population size was set to {55, 50, 40, 60}. The crossover probability was chosen {0.85, 0.75, 0.75, 0.70} and the mutation probability was {0.50, 0.60, 0.65, 0.50} respectively in each case. The final ranking obtained using the genetic algorithm is shown in Figure 4. Figure 5 illustrates part of the final ranking window.

Figure 4 shows part of the final ranking.

4. Sensitivity Analysis of the Final Result

In most cases, arriving at the final ordering accepted by the decision maker does not conclude the decision aiding process. The analyst can additionally propose to perform a sensitivity analysis. Examples of employing the sensitivity analysis have been presented also in Briggs et al. (1990), Goicoechea et al. (1982) and Rios Insua & French (1991).

Sensitivity analysis is understood to be the influence of the change of values quoted with regard to parameters, which consist of information about the decision maker's preferences on the form of the final result (the various methods use different parameters reflecting the decision making's preferences). It is quite useful in the interpretation of results, which have been achieved at, in the course of modifying the values of the appropriate parameters reflecting the decision maker's preferences, and in estimating the influence of the modifications on the final result. The decision maker has quoted some changes in values, which he accepts, with relation to the chosen parameters reflecting his preferences.

On such a basis, the range of sensitivity analysis has been defined, and it comprised the following:

• Taking into consideration the changes in values of relative importance (w) of criteria for single criteria in the originally assumed arrangement of relative importance of values;

• Taking into consideration the changes in values of relative importance (w) of criteria for a number of criteria at the same time which, as a result, generates a change of the whole arrangement of values of relative importance;

• Taking into consideration the changes of values for threshold functions: for the thresholds of indifference (q), preference (p) and veto preference (v), for a single criterion;

• Taking into consideration the changes of values for thresholds functions: for the thresholds of indifference (q), preference (p), and veto preference (v), for a number of criteria at the same time.

The results of the sensitivity analysis, which has been performed, depending on the range of shifting values of selected parameters of the decision maker preferences, have been presented in Table 7 (the arrangement of originally agreed upon input values for all parameters can be found in Table 4 and Table 5).

The least influence on the final ordering form of alternatives had changes of values for thresholds q, p and v, introduced for a number of criteria simultaneously and for relative importance of a criterion w. In 22 cases of introducing changes altogether, the majority of the cases, the form of the final result preserved the first fifteen alternatives as the final ranking selected by the decision maker (not necessarily in the same rank). It can be said that in both ranges of changes in values of certain parameters suggested by the decision maker discussed above, the sensitivity of the final result (ranking) was considerably insignificant.

The form of the final ranking, as shown in Figure 4, has been achieved at for the changes in values of relative importance (w) introduced both in individual criteria and in a number of criteria simultaneously. Basing on the sensitivity analysis, it is possible to formulate the following conclusion: the decision maker is able to accept also a different form of the final ranking, it is, nonetheless, possible when the influence of the introduced changes on the final result can be justified, and when the form of this result changes only slightly, compared to the final ranking accepted by the decision maker before the sensitivity analysis has been performed.

Performing a sensitivity analysis ends the decision aiding process. It must be mentioned, though, that with this calculation method, it is the decision maker who has taken a final assessment and stated that such factors as interpretation of the final result, coherence between the final result and its preferences, availability and access to information which may influence the final result and the way the information is modified, are consistent with its expectations.

5. Results and Discussion

The goal of this work was to introduce a more objective (and structured) method for the biannual exercise of selecting postgraduate student in the Master in Management Information Systems of the U de O.

The process was as follows. The U de O promoted a call to all persons interested to study the Master in Management Information Systems – Generation 2003-2005. 25 applicants were firstly admitted to compete for a place. Of these persons only the best applicants must be admitted in the program. We filled out a paper form to collect the relevant data during an interview for each one of the applicants. Exactly 21 applicants were submitted and the data for each applicant consolidated into a single spreadsheet. Thus, the performance matrix was developed. To facilitate the computations of the ELECTRE III method and the genetic algorithm, a Visual Basic application called SADAGE (Sistema de Apoyo a la Decision con Algoritmos Geneticos y Electre III) was previously developed. Thresholds and weights were defined and from this a ranked list of applicants was produced. A meeting of the academic board (the DM) was then called and the ranked list was proposed as a starting point to identify the cut-off line. The CONACYT (The National Council of Science and Technology of Mexico) suggest a maximum number between 15 and 20, so a non-significant cut-off was required. Each applicant was then quickly reviewed to ensure that it had been properly represented. Applicants "below" the line were more thoroughly reviewed to ensure that a good applicant was not being dropped in place of another applicant with more quantifiable benefits. One outcome of the review was a revision of the performances, which lead to a revised ranking of applicants. To some extent, this is to be expected in the first use of a new method. The revised list of applicants was then submitted and accepted as students for the Master in Management Information Systems (generation 2003-2005). The DM elected the first fifteen applicants on the final ranking as the new group of students.

6. Conclusions

As a pilot study, the use of (ELECTRE III-genetic algorithm) method to rank applicants to the Master of Science in Management Information Systems was successful. It happened what it is referred to as "the common sense test." That is, the decision maker at Posgrate accepted the ranking process and the outcomes. One reason for the success is, in our view, the structuring of the postgraduate student selection problem. Various anecdotal evidence from the author suggests that the process of structuring a decision problem improves the decision-making process and finds favour with the decision makers. Decision makers tend to fully accept incorporating multicriteria analysis methodology into the process of solving decision problems, notwithstanding the fact that such methods are not fully formalized from the mathematical point of view. In the context of solving multicriteria decision problems, it is fully justified to perform a sensitivity analysis of the final result. This help convinces the decision maker, who accepts the form of the final result, due to its final result has appropriate credibility.

Recebido em 01/2004; aceito em 12/2004 após 1 revisão

Received January 2004; accepted December 2004 after one revision

• (1) Al-Kloub, B.; Al-Shemmeri, T. & Pearman, A. (1997). The role of weights in multi-criteria decision aid, and the ranking of water projects in Jordan. European Journal of Operational Research, 99, 278-288.
• (2) Bouyssou, D. (1990). Building criteria: A prerequisite for MCDA. In: Readings in Multiple Criteria Decision Aid [edited by C.A. Bana e Costa], Springer-Verlag, Berlin, 58-80.
• (3) Briggs, T.; Kunsch, P.L. & Mareschal, B. (1990). Nuclear waste management: An application of the multicriteria PROMETHEE methods. European Journal of Operational Research, 44, 1-10.
• (4) Buchanan, J.T. & Henig, M.J. (1997). Objectivity and Subjectivity in the Decision Making Process. Internal report 1997-1. University of Waikato, Department of Management Systems. <http://www.mngt.waikato.ac.nz/depts/mnss/john/subobj1.htm> .
• (5) Carlsson, C.; Fuller, R. & Fuller, S. (1997). OWA operators for doctoral student selection problem. In: The ordered weighted averaging operators: Theory, Methodology, and Applications [edited by R.R. Yager and J. Kacprzyk], Kluwer Academic Publishers, Boston, 167-178.
• (6) Dobson, P.; Krapljan-Barr, P. & Vielba, C. (1999). An evaluation of the validity and fairness of the Graduate Management Admissions Test (GMAT) used for MBA selection in a UK business school. International Journal of Selection and Assessment, 7, 196-202.
• (7) Edwards, W. (1997). How to use multiattribute utility measurement for social decision making. IEEE Transactions on Systems, Man and Cybernetics, 7, 326-340.
• (8) Fernández González, E. & Leyva López, J.C. (2004). A method based on multiobjective optimization for deriving a ranking from a fuzzy preference relation. European Journal of Operational Research, 154, 110-124.
• (9) Flitman, A.M. (1997). Towards analyzing student failure: neural networks compared with regression analysis and multiple discriminant analysis. Computers Operations Research, 24, 367-377.
• (10) Fodor, J. & Roubens, M. (1994). Fuzzy Preference Modeling and Multicriteria Decision Support Kluwer, Dordrecht.
• (11) Georgopoulou, E.; Lalas, D. & Papagiannakis, L. (1997). A Multicriteria Decision Aids Approach for Energy Planning Problems: The case of Renewable Energy option. European Journal of Operational Research, 103, 38-54.
• (12) Goicoechea, A.; Hansen, D.A., & Duckstein, L. (1982). Multiobjective Decision Analysis with Engineering and Business Applications J. Wiley, New York.
• (13) Goldberg, D. (1989). Genetic algorithms in search, optimization, and machine learning Addison-Wesley.
• (14) Graham, L.D. (1991). Predicating academic success of students in a master of business administration program. Educational Psychology Measure, 4, 721-727.
• (15) Hardgrave, B.C.; Wilson, R.L. & Walstrom, K.A. (1994). Predicting graduate student success: a comparison of neural networks and traditional techniques. Computers Operations Research, 21, 249-263.
• (16) Hokkanen, J. & Salminen, P. (1997). Choosing a solid waste management system using multi criteria decision analysis. European Journal of Operational Research, 98, 19-36.
• (17) Leyva López, J.C. & Fernández González, E. (1999). A Genetic algorithm for deriving final ranking from a fuzzy outranking relation. Foundations of Computing and Decision Sciences, 24, 33-47.
• (18) Leyva López, J.C. (2000). A genetic algorithm application for the individual and group multicriteria decision making: PhD Thesis resume. Computación y Sistemas, 4, 183-188.
• (19) Lievens, F. & Coetsier, P. (2002). Situational tests in student selection: an examination of predictive validity, adverse impact, and construct validity. International Journal of Selection and Assessment, 10, 245-257.
• (20) Michalewicz, Z. (1996). Genetic Algorithms + Data Structures = Evolution Programs Springer-Verlag.
• (21) Ordoñez Reinoso, G. & Valenzuela Rendón, M. (1992). Permutation optimization with genetic algorithms: The Traveling Salesman Problem (in Spanish). Proc. of the Third Latin-American Congress of Artificial Intelligence, 271-282.
• (22) Poon, P.W. & Carter, J.N. (1995). Genetic algorithm crossover operators for ordering applications. Computers & Operations Research, 22, 135-147.
• (23) Powers, D.E. & Lehman, J. (1983). GRE candidates' perceptions of the importance of graduate admission factors. Research in Higher Education, 19, 231-249.
• (24) Ribeiro, R.A. (1996). Fuzzy multiple attribute decision making: a review and new preference elicitation techniques. Fuzzy Sets Systems, 78, 155-181.
• (25) Rios Insua, D. & French, S. (1991). A framework for sensitivity analysis in discrete multiobjective decision-making. European Journal of Operational Research, 54, 176-190.
• (26) Roger, M.; Bruen, M. & Maystre, L. (2000). ELECTRE and DECISION SUPPORT Kluwer, Academic Publishers.
• (27) Roger, M. & Bruen, M. (1998). Choosing realistic values of indifference, preference and veto thresholds for use with environment criteria with ELECTRE. European Journal of Operational Research, 107, 542-551.
• (28) Roy, B. (1990). The outranking approach and the foundations of ELECTRE methods. In: Reading in Multiple Criteria Decision Aid [edited by C.A. Bana e Costa], Springer Verlag, Berlin, 155-183.
• (29) Roy, B. & Bouyssou, D. (1993). Aide multicritère à la décision: Méthodes et cas Paris, Economica, mai.
• (30) Roy, B. (1996). Multicriteria Methodology for Decision Aiding Kluwer.
• (31) Vanderpooten, D. (1990). The construction of prescriptions in outranking methods. In: Reading in Multiple Criteria Decision Aid [edited by C.A. Bana e Costa], Springer Verlag, Berlin, 184-215.
• (32) Vincke, Ph. (1998). Outranking Approach. Technical report IS-MG 98/08 Universite Libre de Bruxelles, Institute de Statistique et de Recherche Opérationnelle, Serie: Mathématiques de la Gestion.
• (33) Vincke, Ph. (1992). Multicriteria Decision Aid Wiley, Chichester.
• (34) Wilson, T. (1999). A student selection method and predictors of success in a graduate nursing program. Journal of Nursing Education, 38, 183-187.
• (35) Wolming, S. (1999). Validity issues in higher education selection: a Swedish example. Studies in Educational Evaluation, 25, 335-351.

Publication Dates

• Publication in this collection
  05 July 2005
• Date of issue
  Apr 2005

History

• Received
  Jan 2004
• Accepted
  Dec 2004