PREDICTION OF EFFICIENCY IN COLOMBIAN HIGHER EDUCATION INSTITUTIONS WITH DATA ENVELOPMENT ANALYSIS AND NEURAL NETWORKS

ABSTRACT

This paper shows the results of a research of the application of data envelopment analysis (DEA) together with artificial neural networks (ANN) of higher education institutions in Colombia during the years 2011-2013, for the purpose of evaluating the technical efficiency of Colombian higher education institutions and subsequently carry out predictions, based on a group of management indicators. Information provided by the Ministry of National Education was used as data source. The results show that this two-stage approach provides the DEA with the predictive potential that it otherwise lacks, enhancing its evaluative qualities; this is also evident in the various research papers consulted. The results also show that 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation datasets, respectively.

Keywords:
efficiency; classification; higher education

1 INTRODUCTION

Currently it is a priority to take into account principles of rational use of resources and economic efficiency in the administration of public education institutions at their different levels to strengthen their institutional duties (teaching, research and extension). For this reason, the different levels of governance and organization relative to the management of higher education develop strategies and plans to increase efficiency in universities and thus improve their performance. There is a very close relationship between the allocation and efficient use of public resources and for this reason, researchers have spent a lot of time evaluating the efficiency of different education systems in the world.

In this regard, it should be noted that there are quite noticeable differences in terms of the resources available to the different Higher Education Institutions (HEIs) to develop their institutional tasks, which in some way affects the results they obtain. Therefore, it is very possible to find universities presenting a better performance than other universities that have more resources. If we also consider that under the flagship program of the Ministry of National Education, Ser Pilo Paga (Being Smart Pays Off), 98% of the resources are received by private universities (^{Observatory of the Colombian University, 2016}) it would be interesting to review the efficiency of Colombian public universities as competitive economic units.

This research uses the CCR model (^{Charnes et al., 1978}) of the Data Envelopment Analysis (DEA), focused on outputs, to determine whether an HEI is efficient, and subsequently neural networks to predict whether a new observation (HEI) can be classified as efficient. The data used in this study correspond to the management indicators of Colombian public universities during the period 2011-2013 and are taken from the database of the Colombian Ministry of National Education. The efficiency is the ability of an HEI to obtain the maximum outputs indicators from a given set of inputs indicators.

2 THEORETICAL FRAMEWORK

Data Envelopment Analysis (DEA) is an approach to evaluate the performance of a set of homogeneous entities called Decision Units (DMUs) that transform multiple inputs into multiple outputs (^{Cooper et al., 2011}). Charnes, Cooper and Rhodes developed the initial DEA model originally based in an earlier work devised by Farrel in 1957 (^{Charnes et al., 1978}).

This technique uses linear programming models to compare production units that handle the same amount of resources and they produce the same amount of products. This generates a frontier efficient values and the efficiency indices within the group of production units studied. Uses of DEA have involved not only business firms but also government and non-profit agencies including schools, hospitals among others, the term “Decision Making Unit” (DMU) was therefore introduced to cover, in a flexible manner, any such entity. The performance of a DMU is efficient if and only if it is not possible to improve any input or output without worsening any other input or output (^{Cooper et al., 2011}). In this way, Colombian state universities, can be treated as Decision-Making Units (DMUs).

In the CCR model is determined the relative efficiency of n DMUs to be evaluated, where each unit has m inputs (resources) $x_1,x_2,\dots ,x_m$ and s outputs (products) $y_1,y_2,\dots ,y_s$, is required the solution of the following linear program model.

$\begin{array}{l}\begin{array}{cccc}& & & \end{array}\max \theta ={\mu }_1{y}_{10}+\cdots +{\mu }_s{y}_{s0}\\ \text{subject to}{\nu }_1{x}_{10}+\cdots +v_m{x}_{m0}=1\\ \begin{array}{cccc}& & & \end{array}{\mu }_1{y}_{1j}+\cdots +{\mu }_s{y}_{sj}\le {\nu }_1{x}_{1j}+\cdots +{\nu }_m{x}_{mj}\\ \begin{array}{cccc}& & & \end{array}j=\mathrm{1,2,}\dots ,n\\ \begin{array}{cccc}& & & \end{array}{\nu }_1,{\nu }_2,\dots ,{\nu }_m\ge 0\\ \begin{array}{cccc}& & & \end{array}{\mu }_1,{\mu }_2,\dots ,{\mu }_s\ge 0\end{array}$

The DEA methodology is one of the main techniques to evaluate the performance of productive units both in the public sector and in the private sector. Thus, this technique has been used in the financial sector (^{Sathye, 2001}; ^{Wang et al., 2014}; ^{Kwon & Lee, 2015}; ^{Fukuyama & Matousek, 2017}), in the energy sector (^{Sathye, 2001}; ^{Wang et al., 2014}; ^{Kwon & Lee, 2015}; ^{Fukuyama & Matousek, 2017}), in the operational performance of companies (^{Sathye, 2001}; ^{Wang et al., 2014}; ^{Kwon & Lee, 2015}; ^{Fukuyama & Matousek, 2017}), and in the assessment of research centers ^Souza2016, among others.

Artificial Neural Networks (ANNs) simulate the information processing of the human brain, as an interconnected set of simple processing elements, units or nodes, such as the ability to memorize and relate facts and circumstances, and are a powerful tool for modeling non-linear functions through an adaptation or learning process of a set of training patterns (^{Gurney 2003}). These units or nodes receive and transmit signals analogously to biological neural networks (^{Mehrotra et al., 1997}). Neural networks were developed based on the McCulloch and Pitts model, where their object of study is the computation made by neurons, and showed that the neural network system of the human being can be imitated by a mathematical algorithm (^{Shabanpour et al., 2017}). These computer algorithms consist of many simple elements with highly interconnected that produces different signals based on the weighted sum of the input signals they receive, with ability of modelling complex non-linear relations (^{Schalkoff, 1997}). The type of network used is defined by the architecture that represents the connection structure between the nodes, by its method of determining the connection weights and by the activation function it uses. The architecture of a network consists of the grouping of the neurons forming layers; there are monolayer and multilayer networks (^{Fausett, 1994}).

2.1 Data Envelopment Analysis and Neural Networks

Neural networks have been used in multiple applications and recently in combination with data envelopment analysis as predictive techniques in a second stage to evaluate the efficiency of productive units. Among these works we can highlight the following: (^{Çelebi & Bayraktar, 2008}; ^{Emrouznejad & Shale, 2009}; ^{Pendharkar, 2011}; ^{Azadeh et al., 2011}; I-Fei, 2011; ^{Liu, 2012}; ^{Tosun, 2012}; ^{Choobdari et al., 2013}; ^{Olanrewaju et al., 2013}; ^{Azadeh et al., 2015}; ^{Kwon & Lee, 2015}; ^{Park & Kim, 2016})

2.2 Data Envelopment Analysis and Higher Education

The evaluation of the efficiency of higher education institutions is an area of great interest. The DEA methodology is particularly useful in the evaluation of educational institutions since the economic value of many of the inputs and outputs is difficult to determine, so weighing them is considered appropriate for universities (^{Colbert et al., 2000}). Thanassoulis (2016) has provided insights into the richness of DEA in education. Within the literature, we can highlight recent studies from different parts of the world such as (2013); ^{Nazarko & Šaparauskas (2014}); ^{Ruiz et al. (2015}); ^{Andersson et al. (2017}); ^{Fernandez-Palacin et al. (2018}), among others.

3 METHODOLOGY

In this research, we make a quantitative study supported by Data Envelopment Analysis (DEA) of 32 universities within the Colombian Public University System during the years 2011 to 2013. The approach we propose using of a DEA-CCR-O model (CCR model oriented on outputs) with four inputs and six outputs. The inputs of the process are the total number of Academic Staff, Administrative Staff Spending, Financial Resources and Physical Resources. The performance indicators (output) are number of postgraduate and undergraduate degrees enrolled, Results Saber PRO, Indexed Journals, Research publications and Professor Mobility. An output approach is used, since reducing resources for an inefficient university would be unreasonable, because this implies reducing the academic and administrative staff, as well as a reduction of financial resources.

The DEA model is conceived for the purpose of determining the technical efficiency indexes of each Colombian public HEIs and thus identifying the institutions that present the best practices in their performance, later for the efficiency results obtained with the DEA model we using neural networks with the purpose of predicting the classification of a new IES as efficient or not, and the validation of the predictive power of the neural network. The architecture used in this research is a monolayer neuronal network.

3.1 HEIs included in the analysis

In this efficiency assessment of public HEIs, the 32 Colombian public universities belonging to the System of State Universities (SUE, for its Spanish acronym) are considered for three consecutive years (2011-2013). The HEIs in the analysis stage were randomly assigned from HEI-1 to HEI-32 in order to ensure the confidentiality of the results of the evaluation. The institutions that were considered in the study are shown in Table 1.

Table 1:
HEI considered in the study.

HEI	HEI
1. Universidad Colegio Mayor de Cundinamarca	17. Universidad del Quindío
2. Universidad de Antioquia	18. Universidad del Tolima
3. Universidad de Caldas	19. Universidad del Valle
4.Universidad de Cartagena	20. Universidad Distrital
5. Universidad de Córdoba	21.Universidad Francisco de Paula Sant. - Cúcuta
6. Universidad de Cundinamarca	22. Universidad Francisco de Paula Sant. - Ocaña
7. Universidad de La Amazonía	23. Universidad Industrial de Santander
8. Universidad de La Guajira	24. Universidad Militar Nueva Granada
9. Universidad de Los Llanos	25. Universidad Nacional Abierta y a Distancia
10. Universidad de Nariño	26. Universidad Nacional de Colombia
11. Universidad de Pamplona	27. Universidad Pedagógica Nacional
12. Universidad de Sucre	28. Universidad Pedagógica y Tecnológica de Colombia
13. Universidad del Atlántico	29. Universidad Popular del Cesar
14. Universidad del Cauca	30. Universidad Surcolombiana
15. Universidad del Magdalena	31. Universidad Tecnológica de Pereira
16. Universidad del Pacífico	32. Universidad Tecnológica del Chocó

• Source: Compiled by authors.

3.2 The data and variables

The data of the variables employed in the realization of this study is derived from data which are publicly available on the Colombian ^{Ministry of National Education website (2015}). The data envelopment analysis requires the identification of input variables, which correspond to the resources used to carry out the institutional duties, and output variables, which are identified as the products or objectives of the HEIs. We selected variables are used both to determine whether or not a given HEI is efficient, and for the subsequent prediction through neural networks, four inputs and six outputs are employed considering previous researches (^{Tsolas & Charles, 2015}; ^{Selim & Bursalolu, 2015}; ^{Ramzi & Ayadi, 2016}).

Input variables

The first input is the total number of Academic Staff (full-time professors equivalent), includes professors, associate professors, assistant professors, casual professors and lecturers. The second is Administrative Staff Spending, contains the expense of non-academic staff that administers the teaching and research process. The third input is Financial Resources, represents monetary resources provided by the State and the monetary resources produced by the university itself. The fourth input is Physical Resources or area of built physical spaces available for university activities (academic and administrative).

Output variables

The first and second output is the number of postgraduate and undergraduate degrees enrolled, measured in equivalent full-time students. The third output is Results Saber PRO, contains the total quantity of students who are in the upper quintile according to the State exam scores. The fourth output is Indexed Journals indicating the number of academic journals of the HEI classified in the different categories in the Publindex of Colciencias. The fifth result is Research publications that show the number of papers of the researchers in different journals. The sixth output is Professor Mobility or number of professors linked to mobility programme supported by from the HEI to which they belong.

4 RESULTS

4.1 Data Envelopment Analysis (DEA)

The efficiency results of the CCR-O model for the years 2011 to 2013 are shown in Table 2. Therefore, there are 96 observations available, 32 universities for 3 years.

Table 2:
Efficiency of HEIs years 2011 to 2013

HEI	2011	2012	2013
	Efficiency	Efficiency	Efficiency
HEI 1	Efficient	Inefficient	Efficient
HEI 2	Inefficient	Inefficient	Inefficient
HEI 3	Inefficient	Inefficient	Efficient
HEI 4	Inefficient	Inefficient	Inefficient
HEI 5	Efficient	Efficient	Efficient
HEI 6	Efficient	Inefficient	Inefficient
HEI 7	Inefficient	Inefficient	Inefficient
HEI 8	Inefficient	Efficient	Efficient
HEI 9	Inefficient	Efficient	Inefficient
HEI 10	Inefficient	Inefficient	Efficient
HEI 11	Efficient	Efficient	Efficient
HEI 12	Efficient	Efficient	Efficient
HEI 13	Efficient	Inefficient	Inefficient
HEI 14	Inefficient	Inefficient	Inefficient
HEI 15	Efficient	Efficient	Inefficient
HEI 16	Efficient	Efficient	Efficient
HEI 17	Inefficient	Inefficient	Inefficient
HEI 18	Inefficient	Inefficient	Inefficient
HEI 19	Efficient	Efficient	Efficient
HEI 20	Efficient	Inefficient	Efficient
HEI 21	Efficient	Efficient	Inefficient
HEI 22	Efficient	Efficient	Efficient
HEI 23	Efficient	Efficient	Inefficient
HEI 24	Inefficient	Inefficient	Inefficient
HEI 25	Efficient	Inefficient	Inefficient
HEI 26	Inefficient	Inefficient	Inefficient
HEI 27	Inefficient	Inefficient	Inefficient
HEI 28	Inefficient	Inefficient	Inefficient
HEI 29	Inefficient	Inefficient	Inefficient
HEI 30	Inefficient	Inefficient	Inefficient
HEI 31	Efficient	Inefficient	Inefficient
HEI 32	Efficient	Efficient	Efficient

• Source: Compiled by authors.

4.2 Neural networks

The results in Table 2 indicate that, from the 96 observations, 40 are considered efficient and 56 are inefficient. For the modeling of the neural network, the 96 observations were divided into 72, for training (train), and 24, for validation (test). The validation of the neural network will be carried out through repeated holdout, which consists in taking two sets of random samples for training and validation. This procedure is repeated k times, for which k models are constructed, which are also evaluated k times to improve the estimation of the performance of the classification model.

The modeling of the neural network was conducted through the caret and nnet packages of the R software. The best parameters for the neural network, Size and Decay, are determined through the caret package. The model obtained consists of an artificial neural network (ANN) (10, 3, 1), that is, of 10 inputs, 3 hidden neurons and 1 output, with 37 weights and with the decay parameter of 0.1, as shown in Figure 1. The inputs used for the ANN are the 10 variables of the DEA model mentioned before.

Figure 1:
Estimated neural network model.

Figure 1 shows the representation of the calculated neural network. The black color of the arcs indicates a positive weight, the gray one a negative weight, and the thickness the magnitude of the weight, where we can observe a positive weight of great relative magnitude between the hidden layer H3 and the response (categorical output variable that indicates whether or not the HEI is efficient).

The confusion matrix of the training data shown in Table 3 indicates a true positive (efficient) rate (sensitivity) of 81.25%, a true negative rate (specificity) of 97.50% and an overall correct classification rate of 90.28%.

Table 3:
Confusion matrix training data.

Real	Assigned
	Efficient	Inefficient
Efficient	26	1
Inefficient	6	39
Classification Rate	81.25%	97.50%
Overall Classification Rate	90.28%

The confusion matrix of the validation data shown in Table 4 indicates a positive (efficient) rate (sensitivity) of 75.00%, a true negative rate (specificity) of 68.75% and an overall correct classification rate of 70.83%.

Table 4:
Confusion matrix validation data.

Real	Assigned
	Efficient	Inefficient
Efficient	6	5
Inefficient	2	11
Classification Rate	75.00%	68.75%
Overall Classification Rate	70.83%

Figure 2 displays the ROC (Receiver Operating Characteristic) curve for the training data, indicating an AUC (Area Under ROC Curve) of 0.894, which shows that the neural network model has a good classification capacity.

Figure 2:
ROC curve of estimated neural network.

After carrying out the validation by repeated holdout with k = 100 (repeat holdout 100 times), a success rate was found for the training data that varies between a minimum of 34.72% and a maximum of 81.94%, with an average value of 61.76%, and a success rate for the validation data that varies between a minimum of 33.33% and a maximum of 83.33%, with an average value of 58.33%.

Figure 3 shows the Box Plots of the 100 correct classification rates when validating the neural network by means of repeated holdout 100 times, both for the training data and for the validation data, showing that in at least 75% of cases this rate is above 50%.

Figure 3:
Box plots.

The confidence intervals for the correct classification rates constructed by the quintile method shown in Table 5 indicate that 95% of the models built have correct classification rates between 36.77% and 79.17%, and between 33.33% and 79.17% for the training and validation data, respectively. 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation data, respectively.

Table 5:
Confidence interval.

Quintile	Training	Validation
2.50%	36.77	33.33
50%	64.58	58.33
97.50%	79.17	79.17

These differences in the success rates for both the training data and the validation data are due to the fact that every time a model is built, different samples are chosen for training and validation, thus modifying the parameters of the neural network model built, and therefore also its performance.

Also the modeling of the neural network using the 64 observations of the years 2011 and 2012 was also carried out for training and for validation the 32 observations of the year 2013 were used. The model obtained consists of an artificial neural network (ANN) (10, 5, 1), that is, 10 inputs, 5 hidden neurons and 1 output, with 61 weights and the decay parameter of 0.1, as shown in Figure 4. The inputs used for ANN again are the 10 variables of the DEA model.

Figure 4:
Estimated neural network model.

In Figure 4 we can observe a positive weight of great relative magnitude between the hidden layer H3 and the response (categorical output variable that indicates whether the IES is efficient or not).

The confusion matrices of the training data and validation data is show in Table 6 and Table 7, respectively

Table 6:
Confusion matrix training data.

Predicted class	True class
	Efficient	Inefficient
Efficient	23	5
Inefficient	5	31
Classification Rate	82.14%	86.11%
Overall Classification Rate	84.38%

Table 7:
Confusion matrix validation data.

Predicted class	True class
	Efficient	Inefficient
Efficient	9	3
Inefficient	3	17
Classification Rate	75.00%	85.00%
Overall Classification Rate	81.25%

In the Figure 5 displays the ROC (Receiver Operating Characteristic) curve for the training data, indicating an AUC (Area Under ROC Curve) of 0.841, which shows that the neural network model has a good classification capacity.

Figure 5:
ROC curve of estimated neural network.

5 CONCLUSION

The results show that the application of the DEA together with neural networks allows us to predict whether or not a HEI can be classified as efficient, with an adequate correct classification rate. Furthermore, this two-stage approach gives the DEA a predictive potential that it otherwise lacks, thus enhancing its evaluative qualities. Thus, from the point of view of state management, the proposed model can be of great benefit to decision makers because when analyzing the performance of universities by geographic area, they can confirm that the distribution of financial resources from the State should be redesigned to improve the overall efficiency of the system. In other words, this approach can lead to a more realistic distribution of resources based on the results obtained, showing that 50% of the models built have correct classification rates of 64.58% and 58.33% for training and validation data, respectively.

Further research is recommended to identify the most influential variables when predicting whether or not a HEI is efficient, and to include in the neural network variables that have not been considered in determining efficiency by means of the DEA, such as, for example, the transparency index of each of the HEIs, and evaluate the behavior of the neural network. As well as the use of different machine learning techniques instead of neural networks.

It is also recommended to model the neural network considering a predictive model of the number value of the efficiency index and not of the efficient or inefficient classification determined in this study.

REFERENCES

• ¹
  ANDERSSON C, ANTELIUS J, MÅNSSON J & SUND K. 2017. Technical efficiency and productivity for higher education institutions in Sweden. Scandinavian Journal of Educational Research, 61(2): 205-223.
• ²
  AZADEH A, MIANAEI HS, ASADZADEH SM, SABERI M & SHEIKHALISHAHI M. 2015. A flexible ANN-GA-multivariate algorithm for assessment and optimization of machinery productivity in complex production units. Journal of Manufacturing Systems, 35: 46-75.
• ³
  AZADEH A, SHEIKHALISHAHI M & ASADZADEH SM. 2011. A flexible neural networkfuzzy data envelopment analysis approach for location optimization of solar plants with uncertainty and complexity. Renewable Energy, 36(12): 3394-3401.
• ⁴
  BAYRAKTAR E, TATOGLU E & ZAIM S. 2013. Measuring the relative efficiency of quality management practices in Turkish public and private universities. Journal of the Operational Research Society, 64(12): 1810-1830.
• ⁵
  ÇELEBI D & BAYRAKTAR D. 2008. An integrated neural network and data envelopment analysis for supplier evaluation under incomplete information. Expert Systems with Applications, 35(4): 1698-1710.
• ⁶
  CHARNES A, COOPER WW & RHODES E. 1978. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6): 429-444.
• ⁷
  CHOOBDARI N, BARANDARAN B & ALLA KALVANDI H. 2013. Evaluation of Staff Efficiency Using the Combined Model of Neuro/DEA (Case Study: Operational Unit of Gilan Province Gas Company). Life Science Journal, 10: 185-191.
• ⁸
  COLBERT A, LEVARY RR & SHANER MC. 2000. Determining the relative efficiency of MBA programs using DEA. European Journal of Operational Research, 125(3): 656-669.
• ⁹
  COOPER WW, SEIFORD LM & ZHU J. 2011. Data envelopment analysis: History, models, and interpretations. In: International Series in Operations Research and Management Science. Springer US.
• ¹⁰
  EMROUZNEJAD A & SHALE E. 2009. A combined neural network and DEA for measuring efficiency of large scale datasets. Computers and Industrial Engineering, 56(1): 249-254.
• ¹¹
  FAUSETT L. 1994. Fundamentals of neural networks: architectures, algorithms, and applications. Prentice-Hall, Inc.
• ¹²
  FEI R & LIN B. 2016. Energy efficiency and production technology heterogeneity in China’s agricultural sector: A meta-frontier approach. Technological Forecasting and Social Change, 109: 25-34.
• ¹³
  FERNANDEZ-PALACIN F, LOPEZ-SANCHEZ MA & MUOZ-MRQUEZ M. 2018. Stepwise Selection of Variables in DEA Using Contribution Loads. Pesquisa Operacional, 38(1): 31-52.
• ¹⁴
  FUKUYAMA H & MATOUSEK R. 2017. Modelling bank performance: A network DEA approach. European Journal of Operational Research, 259(2): 721-732.
• ¹⁵
  GURNEY K. 2003. An introduction to neural networks. CRC press.
• ¹⁶
  HAN Y, GENG Z, ZHU Q & QU Y. 2015. Energy efficiency analysis method based on fuzzy DEA cross-model for ethylene production systems in chemical industry. Energy, 83: 685-695.
• ¹⁷
  I-FEI C. 2011. A two-stage cardholder behavioural scoring model using artificial neural networks and data envelopment analysis. International Journal of Advancements in Computing Technology, 3(2): 87-94.
• ¹⁸
  KWON HB & LEE J. 2015. Two-stage production modeling of large U.S. banks: A DEAneural network approach. Expert Systems with Applications, 42(19): 6758-6766.
• ¹⁹
  LIU Z. 2012. An intelligent WPICC evaluation model based on DEA-ANN hybrid algorithm. Journal of Computational Information Systems, 6(13): 4369-4378.
• ²⁰
  MALLIKARJUN S, LEWIS HF & SEXTON TR. 2014. Operational performance of U.S. public rail transit and implications for public policy. Socio-Economic Planning Sciences, 48(1): 74-88.
• ²¹
  MEHROTRA K, MOHAN CK & RANKA S. 1997. Elements of artificial neural networks. MIT press.
• ²²
  MINISTRY OF NATIONAL EDUCATION OF COLOMBIA. 2015. Available at: Available at: mineducacion.gov.co/sistemasdeinformacion/1735/articles-212353 modelo1.pdf last accessed 15 July 2017.
  » mineducacion.gov.co/sistemasdeinformacion/1735/articles-212353 modelo1.pdf
• ²³
  NAZARKO J & ŠAPARAUSKAS J. 2014. Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, 20(1): 25-44.
• ²⁴
  OBSERVATORY OF THE COLOMBIAN UNIVERSITY. 2016. Available at: Available at: http://www.universidad.edu.co/index.php/noticias/13838-98-4-de-recursos-de-ser-pilo-paga-se-ha-ido-a-las-universidades-privadas last accessed 10 May 2017.
  » http://www.universidad.edu.co/index.php/noticias/13838-98-4-de-recursos-de-ser-pilo-paga-se-ha-ido-a-las-universidades-privadas
• ²⁵
  OLANREWAJU OA, JIMOH AA & KHOLOPANE PA. 2013. Assessing the energy potential in the South African industry: Acombined IDA-ANN-DEA (Index Decomposition Analysis-Artificial Neural Network-Data Envelopment Analysis) model. Energy, 63: 225-232.
• ²⁶
  PARK S & KIM J. 2016. Energy efficiency in Korea: analysis using a hybrid DEA model. Geosystem Engineering, 19(3): 143-150.
• ²⁷
  PENDHARKAR PC. 2011. A hybrid radial basis function and data envelopment analysis neural network for classification. Computers and Operations Research, 38(1): 256-266.
• ²⁸
  RAMZI S & AYADI M. 2016. Assessment of Universities Efficiency Using Data Envelopment Analysis: Weights Restrictions and Super-Efficiency Measure. Journal of Applied Management and Investments, 5(1): 40-58.
• ²⁹
  RUIZ JL, SEGURA JV & SIRVENT I. 2015. Benchmarking and target setting with expert preferences: An application to the evaluation of educational performance of Spanish universities. European Journal of Operational Research, 242(2): 594-605.
• ³⁰
  SÁNCHEZ-GONZÁLEZ C, SARTO JL & VICENTE L. 2017. The efficiency of mutual fund companies: Evidence from an innovative network SBM approach. Omega(United Kingdom), 71: 114-128.
• ³¹
  SATHYE M. 2001. X-efficiency in Australian banking: An empirical investigation. Journal of Banking and Finance, 25(3): 613-630.
• ³²
  SCHALKOFF RJ. 1997. Artificial neural networks. McGraw-Hill Higher Education.
• ³³
  SELIM S & BURSALOLU SA. 2015. Efficiency of Higher Education in Turkey: A Bootstrapped Two-Stage DEA Approach. International Journal of Statistics and Applications, 5(2): 56-67.
• ³⁴
  SHABANPOUR H, YOUSEFI S & SAEN RF. 2017. Forecasting efficiency of green suppliers by dynamic data envelopment analysis and rtificial neural networks. Journal of Cleaner Production, 142: 1098-1107.
• ³⁵
  SOUZA GS & GOMES EG. 2016. A non-Archimedean DEA model to assess group comparisons. Pesquisa Operacional, 36(3): 533-546.
• ³⁶
  TOSUN O. 2012. Using data envelopment analysis-neural network model to evaluate hospital efficiency. International Journal of Productivity and Quality Management, 9(2): 245-257.
• ³⁷
  TSOLAS IE & CHARLES V. 2015. Incorporating risk into bank efficiency: A satisficing DEA approach to assess the Greek banking crisis. Expert Systems with Applications, 42(7): 3491-3500.
• ³⁸
  WANG K, HUANG W, WU J & LIU YN. 2014. Efficiency measures of the Chinese commercial banking system using an additive two-stage DEA. Omega(United Kingdom), 44:5-20.
• ³⁹
  XU B & OUENNICHE J. 2012. A data envelopment analysis-based framework for the relative performance evaluation of competing crude oil prices’ volatility forecasting models. Energy Economics, 34(2): 576-583.

Publication Dates

• Publication in this collection
  23 Sept 2019
• Date of issue
  May-Aug 2019

History

• Received
  01 Jan 2018
• Accepted
  14 May 2019