Composition of probabilistic preferences with an empirical approach in multi-criteria problems

Abstract

This paper aimed at presenting an Empirical Approach to the Composition of Probabilistic Preferences (CPP) method. The CPP method is used to choose, sort and classify alternatives in multi-criteria problems. The CPP has explored a wide variety of applications from continuous probability distributions. However, the use of empirical probabilities, without the need to know or assume a joint probability function that gives rise to preferences, can help to deal with certain types of problems with ordinal preference scales. The empirical approach is applied here in three cases, with the purpose of illustrating its main characteristics and limitations. The proposal proved to be satisfactory for treating appraisers' preferences, especially when the number of alternatives is reduced in relation to the options of the scale of assessment points and the number of appraisers, as they contribute to increase the discriminatory capacity of the results.

Keywords:
Composition of probabilistic preferences; Empirical probabilities; ISO 31010

Resumo

Este artigo teve por objetivo apresentar uma abordagem empírica ao método de Composição Probabilística de Preferências (CPP). O método CPP se destina a escolha, ordenação e classificação de alternativas em problemas multicritério. O CPP tem explorado ampla variedade de aplicações a partir de distribuições de probabilidades contínuas. Entretanto, o uso de probabilidades empíricas, sem a necessidade de conhecer ou assumir uma função de probabilidade conjunta que origina as preferências, pode contribuir para lidar com determinados tipos de problema com escalas ordinais de preferência. A abordagem empírica é aqui aplicada em três casos, com a finalidade de ilustrar suas principais características e limitações. A proposta se mostrou satisfatória para o tratamento de preferências de avaliadores, especialmente quando o número de alternativas é reduzido em relação às opções da escala de pontos de avaliação e ao número de avaliadores, por contribuir para elevar a capacidade discriminatória dos resultados.

Palavras-chave:
Composição probabilística de preferências; Probabilidades empíricas; ISO 31010

1 Introduction

The problem of aggregating individual preferences for a collective decision, which has been called “social choice” by Arrow (1963), has received the attention of researchers for centuries, as evidenced by the studies in voting processes by Borda and Condorcet, commented by Pomerol & Barba-Romero (2012). Currently, the ranking of alternatives still remains relevant in different situations, where it is necessary to order candidates for a new position or to determine the priority of solutions to a given problem, from the preference of voters, judges, specialists, interest groups, among others. According to Pomerol & Barba-Romero (2012), ranking methods are more robust than cardinal methods, as the assignment of cardinalities to alternatives is based on fragile and volatile utility functions, subject to contexts and appraisers' partialities, while the choice order by the same appraiser remains unchanged. This makes research on preference ranking methods of still relevant.

This article presents a variant of the Composition of Probabilistic Preferences (CPP) method, developed by Sant’Anna & Sant’Anna (2001), for the ranking of alternatives assessed under multiple criteria. The new proposal explores the properties of the discrete joint probability theory, described by Soong (2004) and Ross (2009). The preferences of appraisers in multi-criteria issues are modeled with empirical probabilities, without assuming a random behavior by continuous probability distributions as proposed by Sant’Anna (2015a) and Abbas (2006). A summary review of the literature indicates that CPP has modeled issues with uniform probability distributions (Sant’Anna & Conde, 2011; Sant’Anna et al., 2012a), normal (Sant’Anna, 2013; Sant’Anna et al., 2015a), triangular (Sant’Anna & Silva, 2011; Treinta et al., 2014), Pareto (Caillaux et al., 2011; Sant’Anna & Mello, 2012) and Beta (Maciel, 2015; Sant’Anna, 2015b; Sant’Anna et al., 2015b), with applications in the most diverse areas of knowledge. In this context, the use of empirical probabilities constitutes a contribution for applying the CPP method.

A positive feature of the use of empirical distributions is that they do not require aggregation of preferences. According to Arrow (1963), individual choices tend to be conflicting and even inconsistent, and a possible aggregation of the preference vectors of the alternatives by single parameters (namely average, mode, median, maximum, minimum, among others) would eliminate such differences. In practical terms, the Empirical Approach treats the preferences of each expert for a given set of alternatives as a vector, rather than aggregating the preferences of a given alternative, received from the experts, by any type of algorithm. CPP has treated preferences as continuous probability distributions, centered on a single value extracted from the multiple experts’ valuations vector. The method proposed in this article considers the discrete probability distribution determined by this vector, without the need to identify the distribution function that would originate it. The characteristics and limitations of the use of empirical probabilities are present in the cases studied in this article. Section II presents the properties and calculation procedures of the CPP method. Section III describes the approach of this method with empirical probabilities. Section IV presents two case studies with two alternatives, assessed in five-point scales by 50 appraisers. Section V expands the application to a case with multiple alternatives, also assessed on a five-point scale. Section VI presents an analysis of the applications made and, finally, section VII brings the article conclusions.

2 The CPP method

The CPP method is based on the use of joint probabilities in decision aid, initially proposed by Sant’Anna & Sant’Anna (2001) and recently expanded in Sant’Anna (2015a). The method applies a probabilistic approach to multi-criteria issues, being useful for the choice, ranking or classification of alternatives. The probabilistic nature of CPP is especially useful in the treatment of inaccurate data. Inaccuracy is inherent to subjectivity and to the measurement errors in individual or group decision-making processes. Therefore, it is natural to assume a random behavior for the assessment of each alternative under each criterion.

The CPP method assumes that the evaluation of the preference of an alternative can be given in the form of the probability of this alternative being chosen among others. To reach this form, when preference is given by the value of a performance attribute, it does not treat it as a spot-on and definitive measure of preference, but rather as the realization of a random variable.

The three stages of the CPP method, according to Sant’Anna (2015a), are described in Chart 1. In the first stage, the exact values of the database, in the form of a decision matrix with alternatives assessed by a set of criteria, are assumed to be location parameters of probability distributions. These exact values are seen as observations of random variables that behave according to probability density functions (pdf). The second stage refers to the choice of a probability distribution that is identified or even assumed as characteristic of the disturbances, having as location parameters the exact values. This procedure of “randomization” of the exact values can employ different distributions of probabilities (Sant’Anna & Conde, 2011; Sant’Anna & Mello, 2012; Sant’Anna, 2015b; Sant’Anna et al., 2015b).

Chart 1
CPP method stages and steps.

Stage		Description	Calculation Procedure
1st Stage		Use of the problem decision matrix database to choose the probability distribution parameters.	According to the characteristics of each type of distribution (i.e., parameters of a normal distribution are the data mean and standard deviation).
2nd Stage		Choice of probability distribution.	Probability density function (pdf) and cumulative density function (cdf) according to the type of distribution selected to the issue.
3rd Stage	Step 1	Calculation of joint probabilities of maximizing preferences (Mij)	$M_{ij}={\displaystyle {\int }_{D_{X_i}}^{}\left[{\displaystyle \prod _{j\ne i}{F}_{X_j}(x_j)}\right]}{f}_{X_i}(x_i)d{x}_i$ (1)
		Calculation of joint probabilities of minimizing preferences (mij)	$m_{ij}={\displaystyle {\int }_{D_{X_i}}^{}\left[{\displaystyle \prod _{j\ne i}\left(1-F_{X_j}(x_j)\right)}\right]}{f}_{X_i}(x_i)d{x}_i$ (2)
	Step 2	I. Composition from the Progressive-Pessimistic (PP) point of view by independence hypothesis	$P{P}_i={\displaystyle {\prod }_{}{M}_{ij}}$ (3)
		A. Composition from the Progressive-Pessimistic point of view (PP) by maximum dependency hypothesis	$P{P}_i=\min M_{ij}$ (4)
		1) Composition from the Progressive-Optimistic point of view (OP) by independence hypothesis	$P{O}_i=1-{\displaystyle {\prod }_{}\left(1-M_{ij}\right)}$ (5)
		a) Composition from the Progressive-Optimistic point of view (OP) by maximum dependency hypothesis	$P{O}_i=\max M_{ij}$ (6)
		(1) Composition from the Conservative-Pessimistic viewpoint (CP) by independence hypothesis	$C{P}_i={\displaystyle {\prod }_{}\left(1-m_{ij}\right)}$ (7)
		(a) Composition from the Conservative-Pessimistic (CP) point of view by maximum dependence hypothesis	$C{P}_i=1-\max m_{ij}$ (8)
		(1) Composition from the Conservative-Optimistic viewpoint (CO) by independence hypothesis	$C{O}_i=1-{\displaystyle {\prod }_{}{m}_{ij}}$ (9)
		(a) Composition from the Conservative-Optimistic (CO) point of view by maximum dependency hypotheses	$C{O}_i=1-\min m_{ij}$ (10)

• Source: Sant’Anna (2015a).

The third stage involves two steps. Initially, the joint probabilities of each alternative are calculated to present the maximum ($M_{ij}$) and minimum ($m_{ij}$) preference among the others, as per Equations 1 and 2, where F(Xj), f (Xi) and D (Xi) are, respectively, the cumulative distribution of the vector Xj, which represents the evaluations of the other alternatives with the exception of the j-th alternative, the density function and the support of the random variable xj (Sant’Anna et al., 2012b). Calculations are performed for each alternative under each criterion. In the last step, joint probabilities are compounded under different decision points of view.

The Progressive-Pessimistic (PP) point of view calculates the preference by the joint probability of an alternative presenting the highest preference in relation to all the others. This joint probability may be calculated under assumptions of independence and maximum dependence, to portray the extremes of the correlation between the variables, thus being calculated respectively with Equations 3 and 4. The progressive approach denotes the decision-makers' intention to “earn more,” focusing on alternatives near the excellence boundary, where Mjc is the relevant factor for decision-making. On the other hand, the conservative approach denotes the intention of “avoiding losses”, in which decision-makers aim to differentiate the alternatives near the worse performance boundary, being mjc the relevant factor for decision-making. The pessimistic approach considers preferences across all criteria simultaneously, while the optimistic approach is satisfied by preference according to at least one of them.

The Progressive-Optimistic (OP) viewpoint calculates the probability of an alternative to have a higher preference over the other alternatives, as per, at least, one of the criteria, being calculated according to Equations 5 and 6. The Conservative-Pessimistic (CP) viewpoint represents the joint probability of an alternative not having the worst preference over all other alternatives for any criteria, being calculated from Equations 7 and 8. Finally, the Conservative-Optimistic (CO) viewpoint represents the joint probability of an alternative not receiving the worst evaluation by at least one criterion, being calculated according to Equations 9 and 10.

3 CPP with empirical probabilities

The empirical probability, also referred to as relative frequency, or experimental probability, is the ratio between the number of results in which a specified event occurs and the total number of experiments performed, without considering a theoretical sample space (Graybill et al., 1974; Abbas, 2006). In a more general sense, the empirical approach derives probabilities from the experiment and the observation of its results. Given an event A in the sample space of an experiment E, the relative frequency of A is the ratio n(A)/n(E), where n(A) represents the number of favorable results for the occurrence of A and n(E) the total number of results of the experiment E. In the present case, A is a vector of preferences in a scale of points and the number of results is the amount of assessments, so that the empirical probability of the vector (A₁, ..., A_m) is n(A₁, ..., A_m)/n(Assessments).

The CPP with the empirical probability approach to the preference vectors requires an adaptation in the calculation procedures described in Chart 2 and Equations 11, 12 and 13. In the first stage, the preferences of each appraiser are transformed into a vector, where each element of the vector represents the score of an alternative. For example, if an expert assesses four alternatives on a seven-point scale, where “1” represents the lowest preference and “7” is the largest, a possible vector resulting from this assessment could be (2,4,6,3) where “2” represents the assessment of alternative A, “4” alternative B, “6” alternative C and “3” alternative D. In the second stage, the empirical probabilities of each preference vector are calculated, as defined by Graybill et al. (1974). For example, if that vector (2,4,6,3) represents the preference of 15 out of the 45 appraisers, then the observed frequency of this vector equals 0.33, that is, P(A=2,B=4,C=6,D=3)=15/45. In the first step of the third stage, the probabilities of each alternative receiving the highest (Mij) and lowest (mij) preference are calculated. It should be noted that the Mij and mij calculations for the empirical variant involve less computational complexity when compared to the continuous CPP approach. On the other hand, it is possible to infer that, in some problems, the empirical approach may have reduced discritionary power, especially when dealing with vectors with many alternatives assessed in scales with few points. In this case, several null Mij and mij are estimated, indicating that the use of the empirical variant is not adequate to the issue. At last, the last step of the third stage is identical to that of described in Chart 1, with the Mij and mij probabilities being used to calculate the different compositions.

4 Application in bivariate cases

This section presents two simplified case studies to illustrate the basic differences between the CPP continuous approach and the variant with empirical preference probabilities to preference vectors. Table 1 presents a database, exclusively illustrative, for the first bivariate case, where alternatives “A” and “B” receive the preference of 50 appraisers for a single criterion, both under ordinal scales of five points, where “1” indicates the worst performance and “5”the best performance. The last row and last column of Table 1 consolidate the marginal preferences for each point of the scale, the columns for alternative “A” and lines for alternative “B”. The main diagonal of the matrix is highlighted, it indicates the number of appraisers that did not establish differences between the alternatives. Thus, the matrix in Table 1 describes the number of appraisers who opted for each preference vector, represented by ordered pairs (A, B), assuming a joint behavior of alternatives A and B.

Table 1
First bivariated case.

B \ A	1	2	3	4	5	TOT
1	2	2	1	3	4	12
2	4	1	2	0	4	11
3	4	2	0	0	4	10
4	4	1	2	0	2	9
5	1	1	2	2	2	8
TOT	15	7	7	5	16	50

Chart 2
Stages and steps of the empirical variant.

Stage		Description	Calculation Procedure
1st Stage		Transformation of each expert evaluation into a preference vector	The preference vector of an expert takes the form (A1, ..., Am), where each alternative Ai receives a value in the point scale available for the assessment of preferences.
2nd Stage		Calculation of observed probabilities of each preference vector	$P(A_1\mathrm{,...,}{A}_m)=n(A_1\mathrm{,...,}{A}_m)/n(avaliações)$ (11)
3rd Stage	Step 1	Calculation of joint probabilities of maximizing preferences (Mij)	$M_{ij}={\displaystyle \sum _{i=1}^{i=m}P(A_i>}{A}_1\mathrm{,...,}{A}_i>A_m)$ (12)
		Calculation of joint probabilities of minimizing preferences (mij)	$m_{ij}={\displaystyle \sum _{i=1}^{i=m}P(A_i<}{A}_1\mathrm{,...,}{A}_i<A_m)$ (13)
	Step 2	As per Chart 1	As per Chart 1.

• Source: Adapted from Ross (2009), Sant’Anna (2015a) and Soong (2004).

From the data in Table 1, the first bivariate case was approached by three procedures to identify the highest preference of appraisers: marginal aggregation of choices, the CPP method and the empirical variant. In relation to the aggregation procedure, which does not consider the joint behavior of variables A and B, the aggregation measure may represent the mode, the median, the mean or some other location measure considered representative of the context. It can be seen in Chart 1 that the mode of alternative “A” is “5” (with 16 preferences), the median is “3” (ranked 25^th) and the mean is “3” (obtained weighting the assessments with weights given by the marginal preferences). For alternative “B”, the mode is “1” (with 12 preferences), the median is “3” (ranked 25^th) and the mean is “2.8”. These results indicate that the use of the mode or the mean e as aggregation parameters results in the preference of the appraisers for alternative “A”, while the choice by the median implies indifference between the alternatives.

In the case of approaching the issue with the CPP method, the results also indicate the preference for alternative “A”. The calculation procedure considered each appraiser as a distinct criterion, assuming each choice of the ordered pair (A, B) as the exact measure to be “randomized” to A and B. For example, supposing that an appraiser associated their choice with the ordered pair (A=2, B=4), “2” would be taken as the location parameter of the probability distribution for the alternative “A” and “4” would be taken as the location parameter of the distribution for the alternative “B”, for the same type of distribution. The preferences were normalized to the interval (0,1) and these parameters were assumed as modes of triangular distributions with range (0,1). The results obtained from the PP point of view were PP (A) = 8.56654E-16 and PP (B) = 2.15797E-17, thus indicating the preference for alternative A.

If the problem is approached with the empirical variant, since the probability of each vector observation is equal to 1/50, due to the number of 50 appraisers, it can be seen that the upper quadrant of the matrix represents the joint probabilities in which the preference for A is greater than the preference for B, where these alternatives are assumed to be random variables on a scale of 1 to 5. The lower quadrant indicates the joint probabilities in which the preference for B is greater than the preference for A. Thus, the application of Equation 12 indicates that P (A> B) is 22/50, while P (B> A) is 23/50. The Empirical Approach indicates the preference of the appraisers for alternative “B”.

In order to explore the sensitivity of the different approaches, a slight variation of the data in Table 1 was made. Table 2 presents a change of preferences in the upper quadrant, the lower quadrant kept unchanged. Three appraisers who initially opted for pair (5,1) changed their choices to pair (2,1) and one appraiser changed from pair (5,2) to pair (3,2). These translations in their evaluations of alternative A do not change the results of the Empirical Approach, as P (A> B) remains equal to 22/50 and P (B> A) remains equal to 23/50. However, the results for the marginal aggregation are changed, as the modes become equal to 1 for both alternatives, the median of B (namely 3) becomes greater than A (namely 2,5), and the weighted average of B (i.e. 2,8) becomes higher than that of A (i.e. 2,78), indicating a new preference for alternative “B”. The same pattern of inversion occurs in the application of the CPP method with the triangular distribution and the PP point of view.

Table 2
Second bivariated case.

B \ A	1	2	3	4	5	TOT
1	2	5	1	3	1	12
2	4	1	3	0	3	11
3	4	2	0	0	4	10
4	4	1	2	0	2	9
5	1	1	2	2	2	8
TOT	15	10	8	5	12	50

The inversion of probabilities for maximizing the preferences for the alternatives in these two bivariate cases, verified for the non-empirical procedures, stems from the cardinal influence of the point scale on aggregation and CPP. When translating preferences from “5” to “3” and to “2”, the parameters employed by these methods are also changed, whereas for the Empirical Approach, the combined probabilities of vectors (5,1), (5,2), (2,1) and (3,2) are equal and do not change the preference of the appraisers from A to B, since A remains preferable to B in these four ordered pairs.

The illustration of the two bivariate cases indicated a greater robustness of the Empirical Approach and greater adherence to explore problems of preference with ordinal scales, since the empirical model was invariant to the intensity of preference of some appraisers. The robustness of ordinal multi-criteria support methods in relation to the methods influenced by the cardinality of the scales was identified by Pomerol & Barba-Romero (2012).

Indeed, one should not attribute to the different results the finding that some procedure is more accurate than another. The nature of the problem should guide the choice of the method. The empirical case proved useful to treat preferences of multiple appraisers as vectors, composed of as many elements as possible, without involving the aggregation of results. However, the empirical approach to a problem with multiple alternatives and a scale limited to a few points can generate a high amount of draws in the comparison between alternatives. This behavior was observed in the following multivariate case study, indicating that the continuous distribution approach presented better results.

5 Application in a multivariate case

The third case study of this research refers to the assessment of the preference of 23 risk management specialists, as described in Maciel (2015) research, on a basket of techniques presented in PMI (2001) and IEC (2009). The experts used an ordinal scale of five points for the assessment, where “1” means the lowest preference and “5” the highest. The basket of alternatives was composed of 14 risk identification techniques, ten qualitative risk analysis techniques and seven risk assessment techniques, where each set was assessed under three criteria: efficiency, effectiveness and complexity. The CPP and empirical variant applications followed the principles of Arrow (1963) and Pomerol & Barba-Romero (2012) to ordinal methods in voting processes, where each choice reflects an appraiser's personal criteria. In this context, the preferences of the experts represented 23 subcriteria of the efficiency, effectiveness and complexity subcriteria for each basket of alternatives. The databases of the survey conducted with the experts are replicated in the ^{Appendix A} of this article, corresponding to the most recent survey conducted by Maciel (2015), as depicted in the. A1 - A9

In order to assess the behavior of the empirical approach in a multivariate case, the empirical data base was compared to the CPP method with Normal distributions and PO point of view, assuming independence. Normal modeling is appropriate to the nature of the problem as one of its parameters (namely, the standard deviation) allows to distinguish the preferences of specialists that may present the same mean in their assessments. The choice of the PO point of view is due to the assumption that decision makers seek techniques closer to the frontier of excellence (i.e. progressive point of view), taking an optimistic point of view choose techniques that receive higher preferences in at least one criterion. The high number of possible points in the scale is a characteristic of the problem that implies a high amount of null values for the calculations of M_ij and m_ij, mainly for the empirical variant, as multiple alternatives receive similar preference assessments. Tables 3, 4 and 5 present the ranking obtained applying the different methods, based on Equations 1, 5, 11 and 12, programmed in the software “R” (R Core Team, 2015).

Table 3
Risk identification.

Alt	Description of Techniques	CPP - Normal Distribution - PO - Independence Hypothesis
		Criterion 1	Criterion 2	Criterion 3	Product	Rank
Alt 1	Brainstorming	0.379	0.628	0.993	0.236	12
Alt 2	Delphi technique	0.414	0.417	0.535	0.092	13
Alt 3	Interviews	0.892	0.958	0.924	0.789	2
Alt 4	Root Cause Analysis	0.968	0.876	0.603	0.511	5
Alt 5	SWOT Analysis	0.893	0.930	0.914	0.759	3
Alt 6	Ishikawa Diagram	0.861	0.794	0.565	0.386	8
Alt 7	Flowchart	0.708	0.814	0.622	0.358	10
Alt 8	Influence Diagram	0.263	0.518	0.526	0.072	14
Alt 9	Expert Support	0.965	0.971	0.946	0.887	1
Alt 10	Critical Analysis of Histories	0.641	0.741	0.610	0.290	11
Alt 11	Scenario Analysis	0.920	0.778	0.679	0.486	6
Alt 12	Business Impact Analysis	0.914	0.912	0.539	0.449	7
Alt 13	Risk Indices	0.845	0.765	0.803	0.519	4
Alt 14	Checklist	0.550	0.683	0.970	0.364	9
Alt	Description of Techniques	Empirical Approach
		Criterion 1	Criterion 2	Criterion 3	PO	Rank
Alt 1	Brainstorming	0	0	0.087	0.087	2
Alt 2	Delphi technique	0	0	0	0	7
Alt 3	Interviews	0	0	0	0	7
Alt 4	Root Cause Analysis	0.043	0	0	0.043	4
Alt 5	SWOT Analysis	0.043	0	0	0.043	4
Alt 6	Ishikawa Diagram	0	0	0	0	7
Alt 7	Flowchart	0	0	0	0	7
Alt 8	Influence Diagram	0	0	0	0	7
Alt 9	Expert Support	0.130	0.087	0.043	0.241	1
Alt 10	Critical Analysis of Histories	0	0	0	0	7
Alt 11	Scenario Analysis	0.043	0	0	0.043	4
Alt 12	Business Impact Analysis	0	0	0	0	7
Alt 13	Risk Indices	0	0	0	0	7
Alt 14	Checklist	0	0	0.087	0.087	2

Table 4
Qualitative risk analysis.

Alt	Description of Techniques	CPP - Normal Distribution - PO - Independence Hypothesis
		Criterion 1	Criterion 2	Criterion 3	Product		Rank
Alt 1	Matrix of Probability and Impact	0.999	0.999	0.993	0.991		1
Alt 2	Assessment of Risk Urgency	0.801	0.913	0.963	0.705		5
Alt 3	GUT Matrix	0.951	0.776	0.825	0.608		6
Alt 4	Scenario Analysis	0.869	0.896	0.918	0.714		4
Alt 5	Business Impact Analysis	0.988	0.896	0.598	0.529		7
Alt 6	Root Cause Analysis	0.976	0.970	0.798	0.756		3
Alt 7	Ishikawa Diagram	0.843	0.589	0.953	0.473		8
Alt 8	Decision Tree	0.924	0.897	0.949	0.786		2
Alt 9	Bow-Tie Analysis	0.846	0.973	0.547	0.450		9
Alt 10	Risk Indices	0.621	0.492	0.949	0.290		10
Alt	Description of Techniques	Empirical Approach
		Criterion 1	Criterion 2	Criterion 3	PO	Rank
Alt 1	Matrix of Probability and Impact	0.130	0.174	0.087	0.344	1
Alt 2	Assessment of Risk Urgency	0.043	0.043	0.043	0.125	2
Alt 3	GUT Matrix	0.043	0	0	0.043	6
Alt 4	Scenario Analysis	0	0.043	0.043	0.085	3
Alt 5	Business Impact Analysis	0.043	0	0	0.043	6
Alt 6	Root Cause Analysis	0.043	0.043	0	0.085	3
Alt 7	Ishikawa Diagram	0	0	0	0	9
Alt 8	Decision Tree	0.043	0.043	0	0.085	3
Alt 9	Bow-Tie Analysis	0	0.043	0	0.043	6
Alt 10	Risk Indices	0	0	0	0	9

Table 5
Risk assessment.

Alt	Description of Techniques	CPP - Normal Distribution - PO - Independence Hypothesis
		Criterion 1	Criterion 2	Criterion 3	Product	Rank
Alt 1	Scenario Analysis	0.964	0.988	0.878	0.836	5
Alt 2	Business Impact Analysis	1	0.996	0.981	0.978	1
Alt 3	Root Cause Analysis	0.984	0.997	0.841	0.824	6
Alt 4	Decision Tree	0.965	0.960	0.956	0.886	4
Alt 5	Bow-Tie Analysis	0.956	0.999	0.994	0.949	2
Alt 6	FN curves	0.791	0.609	0.896	0.431	7
Alt 7	Risk Indices	0.965	0.931	1	0.898	3
Alt	Description of Techniques	Empirical Approach
		Criterion 1	Criterion 2	Criterion 3	PO	Rank
Alt 1	Scenario Analysis	0	0.043	0	0.043	6
Alt 2	Business Impact Analysis	0.130	0.043	0.043	0.204	2
Alt 3	Root Cause Analysis	0	0.130	0.043	0.168	3
Alt 4	Decision Tree	0.043	0.043	0	0.085	5
Alt 5	Bow-Tie Analysis	0.043	0.174	0.130	0.313	1
Alt 6	FN curves	0	0	0	0	7
Alt 7	Risk Indices	0.043	0	0.130	0.168	3

Table 3 presents the results of the application of the CPP methods with normal distribution and the empirical approach for the “Identification of Risks” dimension. In the first case, the preference for each criterion is given by the probability of maximizing the preference according to some expert assuming a normal distribution for each evaluation with a standard deviation estimated by the standard deviation observed in the sample of evaluations of the alternatives by the experts. In the composition of the three basic criteria in a global score, the PO approach was adopted, to increase the power of discrimination. In the second case, it is given by the proportion of experts who prefer the alternative. The results of the first approach indicate a significant discrimination capacity, whit absence of ties in the ranking. The empirical variant confirmed the limitation regarding the ratio between the number of alternatives to be assessed under a five-point scale. The 32 null values in the three criteria of the empirical approach indicate that the respective alternatives were not likely to maximize their preferences in relation to the others in a same criterion. The value “0.043”, for instance, indicates that one of the 23 experts selected a preference vector in which alternatives “4”, “5”, “9” and “11” received preferences greater than the others, and 0.043 is the empirical preference probability for these alternatives. Although the empirical variant has confirmed Alternative 9 as the best choice, there are several ties in the final ranking. This aspect is attenuated if the alternatives for evaluation are scaled down or the scales of points are enlarged. Tables 4 and 5, by using 10 and 7 alternatives, respectively, allowed for improved discrimination.

Table 4 presents the results of the application of CPP with normal distributions and the empirical approach for the “Qualitative Analysis of Risks” dimension. Both approaches indicate Alternative 1 as the preferred one, from the PO point of view. The reduction of the number of alternatives did not change the performance of the CPP method, but improved the results of the empirical approach. The amount of null values was lower than in the previous analysis, as much as the number of ties in the final ranking.

Table 5 presents the results of the application of the CPP with normal distribution and the Empirical approach to the “Risk Assessment” dimension. The results indicate a reversal of order between the first two alternatives when evaluated by the two methods. The reduction in the number of alternatives significantly improved the results of the empirical approach, confirming the sensitivity of this proposal to the ratio between the number of alternatives and the assessment scale.

6 Analysis of the case studies

The case studies have allowed to identify the characteristics and limitations of the empirical approach. The bivariate case studies allowed to verify the effect of its exclusively ordinal treatment of the appraisers' preference vectors. This feature was observed through the translation of preferences in the upper quadrant of the matrix of joint probabilities. This change in the data indicated the influence of the cardinality of the point scale on the parameters, as well as in the whole probability distributions, inverting the positions of alternatives A and B when CPP is applied. These bivariate cases provided a greater variety of preference vectors to appraisers, who had a five-point scale leading to 25 ordered pairs. In addition, the existence of 50 appraisers provided better conditions for the data matrix to present values with a disperse distribution, enabling the use of the empirical approach.

On the other hand, the third case study contrasted the previous ones, presenting a greater quantity of alternatives for the same scale of evaluation, and a smaller number of evaluators. Thus, the combinations of the possible preference vectors were considerably expanded and, thus, the ability of one alternative to receive higher or lower preference was reduced. This amplified the amount of null Mij and mij, compromising the calculations of the probabilistic compositions, as they depend on these indices in their equations. The alternative used to solve this problem was to assume a more benevolent point of view, the PO, which guaranteed a larger discriminatory power to the results. In summary, the case studies indicated that the context of the problem, mainly in terms of size and dispersion, interferes with the application of the empirical approach.

Another way to face this problem would be by softening the definitions of Mij and mij to compute weak preference probabilities, considering, instead of only maximum and minimum values, larger/smaller or equal values. This was not applied because it also generated several ties, now not with the value zero, but with higher values. It would also modify an important characteristic of the original method, while the approach taken resulted in similar results in the cases studied.

7 Conclusions

This research aimed to present an Empirical Approach to the CPP method. The literature review has indicated that this method has explored a variety of multicriteria problems by means of continuous probability distributions. However, the use of empirical probabilities, without the need to know the form of the probability function that originates them, can aid the treatment of certain types of ordinal multicriteria problems. The article then proposed a calculation methodology based on the procedures of the CPP method, exploring the properties of the theory of discrete joint probabilities.

The proposal was applied in three case studies, with the purpose of illustrating its main characteristics and limitations. The Empirical Approach proved to be satisfactory for the treatment of the appraiser preferences, based on ordinal scales, when the ratio between the numbers of alternatives is reduced compared to the options in the assessment scale points. A large number of appraisers can also contribute to raise the discriminatory capacity of the proposal.

The results of the proposed model were also applied in an unfavorable situation, in order to demonstrate its main limitations. In this case, the research of Maciel (2015) was explored, whose decision matrix involved three distinct sets of alternatives, the largest one with 14 options of risk identification techniques assessed by 23 experts on a five-point scale. The conditions of this case indicated the weaknesses of the Empirical Approach, compromising the discrimination of the results with several ties between preferences, even though the alternatives with the largest probability of choice corresponded to the results with the application of estimation of parameters of a continuous distribution.

For future studies, calculations in other cases may be suggested, to quantify the most favorable ratios between the number of alternatives, types of ordinal scales and quantity of appraisers. It is also suggested to compare this proposal with other ordinal methods like the Borda and Condorcet methods, in order to assess the sensitivity and quality the Empirical Approach to CPP. Lastly, a possible development might consider in the calculation of empirical preference probabilities the draws, whose probabilities were disregarded, to maintain adherence to the original equations of the CPP method.

Appendix A Expert assessments.

Table A1
Assessments of risk identification techniques - Efficiency criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	4	3	3	1	5	4	4	3	3	3	4	2	3	3	4	1	3	5	3	4	3	3
A-2	3	4	2	3	2	3	3	2	3	2	3	3	3	1	2	4	1	5	5	5	3	3	3
A-3	3	4	3	5	2	5	5	5	3	4	5	4	4	4	4	4	2	5	5	5	4	4	4
A-4	3	3	5	4	3	5	5	1	5	4	5	5	3	4	2	4	3	4	5	5	5	5	4
A-5	3	3	5	4	3	5	2	5	4	4	5	3	3	2	4	4	3	4	5	3	4	5	5
A-6	4	4	5	4	3	4	3	4	4	3	4	5	3	3	2	4	3	4	5	5	5	5	4
A-7	4	4	5	4	4	5	2	4	4	3	3	3	4	4	3	4	3	2	5	2	4	5	4
A-8	3	3	3	2	4	3	3	3	3	2	3	2	3	3	3	4	3	3	5	3	4	5	2
A-9	5	3	3	5	5	5	3	2	2	3	3	3	5	5	2	4	2	5	5	4	4	5	2
A-10	3	4	3	5	4	4	3	3	3	4	3	4	4	4	4	4	3	4	5	4	4	5	3
A-11	4	5	4	5	5	4	4	3	3	4	3	4	2	4	4	4	4	4	5	5	4	5	3
A-12	3	4	5	5	5	4	5	4	3	5	5	4	4	4	3	4	4	4	5	4	4	5	4
A-13	4	4	4	5	5	4	5	4	3	5	3	4	3	4	4	4	2	3	5	5	3	5	4
A-14	3	3	3	5	5	3	5	4	3	4	3	3	3	3	1	4	2	2	5	3	3	5	3

Table A2
Assessments of risk identification techniques - Efficacy criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	3	4	3	1	5	4	4	3	4	5	5	1	3	2	4	1	3	5	3	4	3	3
A-2	3	3	3	3	2	3	3	3	2	3	3	3	3	1	3	4	1	5	4	5	3	3	3
A-3	3	4	5	5	3	5	5	5	2	5	5	4	5	4	4	4	2	5	5	5	4	4	4
A-4	3	3	4	4	4	5	5	4	4	4	5	5	3	4	3	4	3	4	5	5	4	5	3
A-5	3	2	5	4	5	5	2	5	4	4	5	3	3	2	4	4	3	3	5	3	4	5	5
A-6	4	3	4	4	4	4	3	4	3	2	4	5	3	3	4	4	3	4	4	5	5	5	3
A-7	4	4	4	4	5	5	2	4	3	3	3	2	4	4	5	4	3	2	4	4	4	5	4
A-8	3	3	3	3	4	3	3	3	2	2	3	2	3	3	5	4	3	3	4	3	4	5	4
A-9	5	3	5	5	3	5	4	2	2	3	3	3	5	5	3	4	2	5	4	4	5	4	3
A-10	3	4	3	5	2	4	4	3	3	4	3	5	4	4	3	4	2	3	5	5	4	5	3
A-11	4	2	4	5	3	4	5	3	3	5	3	4	2	4	4	4	3	3	5	5	4	5	4
A-12	3	2	4	5	4	4	4	2	4	5	5	4	3	4	5	4	3	3	5	5	3	5	5
A-13	4	4	4	5	5	4	4	3	2	5	3	4	3	4	4	4	2	2	5	4	3	5	3
A-14	3	2	5	5	3	3	5	4	2	5	3	3	3	3	1	4	2	2	5	3	3	5	3

Table A3
Assessments of risk identification techniques - Complexity criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	5	3	5	4	5	5	4	5	4	4	5	5	4	5	4	5	4	5	5	4	4	4
A-2	3	3	2	5	3	3	4	3	4	3	3	3	3	3	4	4	5	3	4	2	1	3	3
A-3	3	3	2	4	4	5	4	5	4	4	4	4	4	4	5	4	5	3	5	5	3	4	4
A-4	3	2	3	3	4	5	2	4	3	4	4	4	4	3	4	4	3	3	4	3	3	3	4
A-5	3	4	5	4	3	4	5	4	4	5	3	4	5	3	5	4	4	3	4	4	3	4	3
A-6	4	2	4	3	2	3	3	3	3	2	4	4	4	4	3	4	4	4	4	3	3	4	2
A-7	4	3	4	3	2	5	4	2	2	4	5	3	3	4	4	4	3	4	4	4	3	3	2
A-8	3	3	3	3	3	3	3	4	2	3	3	3	4	3	3	4	3	4	5	3	3	4	2
A-9	5	3	4	1	3	5	3	3	5	5	5	4	3	4	4	4	5	3	5	5	3	3	3
A-10	3	2	4	2	4	3	3	3	4	5	5	1	3	3	3	4	4	4	5	4	3	3	3
A-11	4	2	5	1	1	3	3	2	4	5	5	1	3	4	2	4	3	2	4	3	1	3	4
A-12	3	3	4	1	1	3	4	4	4	5	5	4	3	3	3	4	5	3	3	2	2	3	2
A-13	4	4	4	1	4	4	4	5	4	4	5	4	3	3	3	4	5	4	3	2	4	3	2
A-14	3	4	5	3	5	4	3	4	5	5	5	4	3	4	5	4	5	5	4	5	4	3	3

Table A4
Assessments of qualitative risk analysis techniques - Efficiency criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	5	2	3	4	3	5	3	5	3	4	4	3	3	4	3	3	3	4	4	5	5	4	4
A-2	3	3	2	4	4	3	3	3	2	3	3	3	3	3	2	3	3	3	4	5	4	4	5
A-3	3	3	4	4	3	4	5	4	3	2	4	3	4	3	3	3	3	5	4	5	4	4	3
A-4	3	3	3	3	4	4	4	3	3	3	5	3	3	4	2	3	3	4	5	4	5	4	3
A-5	3	3	3	4	5	4	3	4	3	4	5	5	4	4	2	3	5	4	5	5	4	4	4
A-6	3	3	4	4	5	4	5	4	4	3	3	4	3	4	2	3	5	4	4	5	5	3	2
A-7	3	3	4	4	5	4	4	4	3	2	3	4	3	3	2	3	3	4	4	5	5	4	3
A-8	3	4	2	5	1	3	4	3	3	3	4	2	3	2	2	3	2	2	5	5	3	3	3
A-9	3	4	3	4	2	3	4	3	3	2	3	3	3	2	3	3	5	4	5	5	3	3	3
A-10	3	2	3	4	2	3	3	3	2	4	3	4	2	3	3	3	1	2	4	5	3	3	2

Table A5
Evaluations of qualitative risk analysis techniques - Efficacy criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	4	2	3	4	4	5	3	5	3	5	4	2	4	4	3	3	3	5	4	5	5	4	4
A-2	3	4	2	4	4	4	3	4	3	3	4	4	4	4	3	3	3	3	4	5	5	4	5
A-3	3	2	2	4	4	4	5	4	3	2	4	4	3	3	3	3	3	5	4	5	5	4	3
A-4	3	3	3	3	5	4	4	3	4	3	5	5	3	3	3	3	3	3	5	5	5	4	3
A-5	3	2	3	4	3	4	3	3	4	3	5	5	3	4	4	3	3	3	5	5	5	4	4
A-6	3	3	4	4	4	4	5	4	5	4	3	5	2	3	4	3	4	5	4	5	5	3	2
A-7	3	3	3	4	2	3	4	3	2	2	3	5	2	2	4	3	3	4	4	5	5	4	3
A-8	3	4	3	5	2	3	4	3	3	3	4	4	2	2	3	3	2	2	5	5	5	3	3
A-9	3	4	4	4	2	4	4	4	3	2	4	4	4	2	4	3	5	4	5	5	5	4	4
A-10	3	3	2	4	2	3	3	3	2	4	3	4	2	3	4	3	2	2	4	5	5	4	2

Table A6
Assessments of qualitative risk analysis techniques - Complexity. criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	4	5	3	4	5	3	5	3	4	4	3	3	4	4	4	5	2	4	2	3	4	3
A-2	3	3	3	3	4	3	3	3	4	5	3	2	3	4	4	4	3	3	4	2	3	4	4
A-3	3	4	5	3	4	3	4	4	3	2	3	2	3	3	3	4	4	2	4	2	3	3	2
A-4	3	3	5	3	2	3	3	3	3	4	5	1	3	3	2	4	4	2	3	3	3	3	3
A-5	3	3	5	3	1	3	3	3	3	3	5	2	3	3	3	4	3	3	3	1	3	3	2
A-6	3	3	5	4	3	3	3	3	3	3	3	3	4	2	3	4	3	4	3	2	3	3	3
A-7	3	2	5	4	3	3	4	3	4	2	3	3	4	4	3	4	3	4	4	2	3	3	2
A-8	3	3	5	3	5	4	3	4	4	3	3	4	4	3	4	4	4	4	3	2	3	3	4
A-9	3	3	3	3	5	1	3	3	3	2	3	3	3	3	4	4	2	2	3	2	3	3	3
A-10	3	4	5	3	5	4	3	4	2	4	3	4	4	3	3	4	5	4	4	2	3	3	3

Table A7
Assessment of risk assessment techniques - Efficiency criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	3	3	4	5	3	3	3	3	4	5	4	3	3	1	3	1	3	5	3	4	4	3
A-2	5	3	3	4	5	4	3	5	4	3	5	5	3	4	2	3	3	4	5	4	4	4	3
A-3	3	3	2	4	4	3	4	3	4	3	2	5	3	3	2	3	4	4	4	5	4	4	3
A-4	3	4	2	4	3	3	3	3	3	3	2	3	3	3	2	3	2	2	5	4	2	3	3
A-5	3	3	2	4	2	3	4	3	3	2	3	3	3	2	3	3	5	4	5	5	2	3	3
A-6	3	3	3	4	2	3	3	3	3	2	4	3	3	3	3	3	3	3	5	3	2	3	3
A-7	5	3	2	4	2	3	3	3	2	4	5	3	2	3	4	3	1	2	4	5	3	4	2

Table A8
Assessments of risk assessment techniques - Efficacy criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	3	3	4	5	4	4	3	3	5	5	3	3	4	4	3	3	3	5	3	4	4	2
A-2	5	3	3	4	4	4	4	5	3	4	5	4	3	4	4	3	3	4	5	5	4	4	3
A-3	3	3	2	4	4	4	5	4	4	3	2	5	3	3	3	3	4	4	4	5	4	3	3
A-4	3	4	2	4	3	3	4	3	3	3	2	3	3	3	3	3	2	2	5	4	4	3	3
A-5	3	3	4	4	3	4	4	4	3	2	4	4	4	2	3	3	5	4	5	5	4	4	4
A-6	3	3	3	4	3	3	3	3	3	2	4	3	3	3	3	3	3	2	5	3	4	3	3
A-7	5	3	3	3	3	3	4	3	2	5	5	3	2	3	4	3	2	2	4	5	4	4	2

Table A9
Assessments of risk techniques - Complexity criteria.

A\E	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23
A-1	3	3	4	3	1	3	3	3	3	3	5	1	3	3	1	3	4	3	5	3	3	3	3
A-2	5	3	5	3	1	3	3	5	2	3	5	2	3	3	3	3	3	3	5	1	3	3	3
A-3	3	3	5	3	3	3	2	3	2	4	2	3	3	2	3	3	2	3	4	2	3	3	3
A-4	3	3	3	3	4	3	3	3	4	3	2	4	4	2	4	3	4	4	4	3	3	3	3
A-5	3	3	3	4	5	3	3	3	4	2	3	5	3	3	5	3	2	2	3	2	3	3	3
A-6	3	3	3	3	5	3	3	3	4	2	4	3	3	3	3	3	3	4	3	3	3	3	3
A-7	5	4	5	3	5	4	3	4	3	2	5	4	4	3	4	3	4	3	5	3	3	3	4

Acknowledgements

The authors thank the CAPES and CNPq agencies for the financial support as postgraduate scholarships awarded during the research.

• How to cite: Gavião, L. O., Lima, G. B. A., Sant’Anna, A. P., & Maciel, G. F. S. V. (2019). Composition of probabilistic preferences with an empirical approach in multi-criteria problems. Gestão & Produção, 26(2), e2802. https://doi.org/10.1590/0104-530X-2802
• Financial support: None.

Referências

• Abbas, A. E. (2006). Entropy methods for joint distributions in decision analysis. IEEE Transactions on Engineering Management, 53(1), 146-159. http://dx.doi.org/10.1109/TEM.2005.861803
  » http://dx.doi.org/10.1109/TEM.2005.861803
• Arrow, K. J. (1963). Social choice and individual values (2nd ed.). New Haven: Yale University Press.
• Caillaux, M. A., Sant’Anna, A. P., Meza, L. A., & Soares de Mello, J. C. C. B. (2011). Container logistics in Mercosur: choice of a transhipment port using the ordinal Copeland method, data envelopment analysis and probabilistic composition. Maritime Economics & Logistics, 13(4), 355-370. http://dx.doi.org/10.1057/mel.2011.20
  » http://dx.doi.org/10.1057/mel.2011.20
• Graybill, F. A., Mood, A. M., & Boes, D. C. (1974). Introduction to the theory of statistics Tokio: McGraw.
• International Electrotechnical Commission – IEC. (2009). IEC/ISO 31010: 2009: risk management-risk assessment techniques Switzerland: International Electrotechnical Commission.
• Maciel, G. F. S. V. (2015). Análise e validação de um modelo de processo de avaliação de riscos baseado na norma ABNT NBR ISO 31000 (Dissertação de mestrado). Universidade Federal Fluminense, Niterói.
• Pomerol, J.-C., & Barba-Romero, S. (2012). Multicriterion decision in management: principles and practice New York: Springer.
• Project Management Institute – PMI. (2001). Project management body of knowledge (PMBOK® guide) Pensilvânia: PMI.
• R Core Team. (2015). R: a language and environment for statistical computing Vienna: R Foundation for Statistical Computing. Recuperado em 8 de dezembro de 2017, de http//www. R-project.org
• Ross, S. M. (2009). Introduction to probability and statistics for engineers and scientists (4th ed.). Berkeley: Academic Press.
• Sant’Anna, A. P. (2013). Detalhamento de uma metodologia de classificação baseda na composição probabilística de preferências. Relatórios de Pesquisa em Engenharia de Produção, 13, 12-21.
• Sant’Anna, A. P. (2015a). Probabilistic composition of preferences, theory and applications New York: Springer.
• Sant’Anna, A. P. (2015b). Probabilistic human development indices. Brazilian Journal of Operations & Production Management, 12(1), 136-146. http://dx.doi.org/10.14488/BJOPM.2015.v12.n1.a13
  » http://dx.doi.org/10.14488/BJOPM.2015.v12.n1.a13
• Sant’Anna, A. P., & Conde, F. Q. (2011). Probabilistic comparison of call centres in a group decision process. International Journal of Management and Decision Making, 11(5-6), 417-437. http://dx.doi.org/10.1504/IJMDM.2011.043398
  » http://dx.doi.org/10.1504/IJMDM.2011.043398
• Sant’Anna, A. P., & Mello, J. C. C. B. S. (2012). Validating rankings in soccer championships. Pesquisa Operacional, 32(2), 407-422. http://dx.doi.org/10.1590/S0101-74382012005000012
  » http://dx.doi.org/10.1590/S0101-74382012005000012
• Sant’Anna, A. P., & Sant’Anna, L. A. F. P. (2001). Randomization as a stage in criteria combining. In Proceedings of the VII International Conference on Industrial Engineering and Operations Management (ICIEOM) (pp. 248-256). Salvador: ICIEOM.
• Sant’Anna, A. P., & Silva, D. C. D. (2011). Análise multicritério de materiais para a síntese de catalisadores automotivos. Engevista, 13(3), 226-243. http://dx.doi.org/10.22409/engevista.v13i3.282
  » http://dx.doi.org/10.22409/engevista.v13i3.282
• Sant’Anna, A. P., Ferreira, M. H., & Duarte, S. R. A. (2012a). Avaliação do desempenho de empresas utilizando a Composição Probabilística de Índices Financeiros. Pesquisa Operacional para o Desenvolvimento, 4, 304-324.
• Sant’Anna, A. P., Gomes, L. F. A. M., Costa, F. F., Rangel, L. A. D., Faria, M. J. S., Ferreira, R. G., Moreira, R. M., Fo., Ribeiro, R. O. A., & Senna, V. (2012b). Análise multicritério baseada em probabilidades de preferência. In V. F. Oliveira, V. Cavenaghi & F. S. Másculo (Eds.), Tópicos emergentes e desafios metodológicos em engenharia de produção: casos, experiências e proposições (Vol. 5). Rio de Janeiro: ABEPRO.
• Sant’Anna, A. P., Costa, H. G., & Pereira, V. (2015a). CPP-TRI: a sorting method based on the probabilistic composition of preferences. International Journal of Information and Decision Sciences, 7(3), 193-212. http://dx.doi.org/10.1504/IJIDS.2015.071372
  » http://dx.doi.org/10.1504/IJIDS.2015.071372
• Sant’Anna, A. P., Martins, E. F., Lima, G. B. A., & Fonseca, R. A. (2015b). Beta distributed preferences in the comparison of failure modes. Procedia Computer Science, 55, 862-869. http://dx.doi.org/10.1016/j.procs.2015.07.149
  » http://dx.doi.org/10.1016/j.procs.2015.07.149
• Soong, T. T. (2004). Fundamentals of probability and statistics for engineers New York: John Wiley & Sons.
• Treinta, F. T., Farias, J. R., Fo., Sant’Anna, A. P., & Rabelo, L. M. (2014). Metodologia de pesquisa bibliográfica com a utilização de método multicritério de apoio à decisão. Production, 24(3), 508-520. http://dx.doi.org/10.1590/S0103-65132013005000078
  » http://dx.doi.org/10.1590/S0103-65132013005000078

Publication Dates

• Publication in this collection
  09 May 2019
• Date of issue
  2019

History

• Received
  08 Dec 2017
• Accepted
  31 Aug 2018