EVALUATING SUPPLY CHAIN MANAGEMENT PERFORMANCE IN PUBLIC HEALTH CARE: AN MCDA APPROACH

ABSTRACT

Rising demand for more efficient and higher-quality public services has challenged public managers to meet these expectations. Performance evaluation has emerged as a crucial management tool to enhance public services. Its application in specific functional areas, such as Supply Chain Management (SCM), holds the potential to mitigate the escalating healthcare costs. This study aimed to develop a performance evaluation framework for the supply chain management of a public teaching hospital, utilizing the Constructivist Multi Criteria Decision Aid methodology (MCDA-C). Through structured interviews with the hospital's administrative manager, 160 criteria across eleven domains were identified for assessing supply chain management performance. The resulting framework allowed for a comprehensive evaluation, yielding a score of 51.55%. Proposed enhancements are expected to boost the global performance from 51.55 to 76.25% when implemented. A literature comparison underscores the comprehensive and specific nature of this performance evaluation model, demonstrating its influence on the healthcare services provided.

Keywords: supply chain management (SCM); cognitive maps; health care; MCDA; MACBETH

1 INTRODUCTION

Effectively managing a supply chain is vital for any organization's success and given the anticipated persistence of substantial global supply chain challenges of late, it is a critical area of apprehension. Supply chain management is at the center of organizational operations, as it encompasses a range of key processes that must be run efficiently to achieve the organizational mission. Essentially, it is the integration of processes from the original suppliers who provide products, services, and information that add value for customers, all the way through to the final user (Cooper et al., 1997). The supply chain concept encompasses the entire logistic process of a particular product or service, from raw material procurement to delivery to the final consumer. This interconnected business network involves manufacturers, distributors, retailers, and consumers. In the public healthcare sector, the supply chain includes sourcing, procurement, transport, warehousing, and treatment of patients, besides exercising greater control over costs and ensuring the safety of procedures. It also involves organization and collaboration with channel collaborators such as funders, suppliers, intermediaries, third-party service providers, and customers.

In Brazil, public hospitals and the Unified Health System (SUS) are responsible for 70% of national hospital care (Brasil, 2017). Public hospitals include 193 certified teaching hospitals (THs), 47 of which belong to the federal government. They are responsible for high complexity health services and the training of physicians, and the uniqueness of the health process and their social and economic relevance implies an intense flow of materials, information, professionals and patients (Barata et al., 2010). THs are responsible for up to 10% of hospital admissions nationwide and for almost 50% of medium and high complexity procedures within the Unified Health System (SUS), indicating their significant importance.

However, the growing demand for health services, especially in public institutions, has created a need for better management of the effectiveness and complexity relationship in health care institutions, especially given the pressures of the COVID-19 outbreak, which disrupted supply chains globally and caused damage across entire chains in multiple industries, with travel restrictions enforced by multiple countries (Chatterjee et al., 2023; Chowdhury et al., 2021). It is concerning that about 20% of health spending contributes minimally or does not contribute to good health outcomes (OECD, 2017). The World Health Organization (WHO, 2010) reports that between 20% and 40% of all health spending is wasted due to inefficiency. Moreover, Miethke-Morais et al. (2021) point out the global concern that the economic impact caused by the COVID-19 pandemic has left on health systems, which increases the need for additional resources and financial investments (Miethke-Morais et al., 2021).

Given this context, there is a need for scientific instruments to aid in supply chain management, allowing for better management of public healthcare, controlling input use to reduce costs, avoid waste, eliminate product obsolescence, and meet the demand for excellent service. This is crucial at a time when consumption costs account for 50% of the total hospital expenditure in Brazil (Paschoal & Castilho, 2010), while in developed countries, such as the United States, this cost is about 30% (Minahan, 2007). Hospital logistics is an area where costs can be reduced, and efficiencies attained to deliver care at a reasonable cost. Estimates of the potential benefit of an efficiently managed health care supply chain have ranged from 2% to 8% of hospital operating costs (McKone-Sweet et al., 2005).

The present research is grounded in an environment that consists of a variety of health care stakeholders: caregivers, patients, regulatory agencies, physician groups, insurers and governments (municipal, state and federal) as well as a variety of other organizations (Ryan, 2005). The diversity of stakeholders in this environment presents a challenge to effective organization and management of healthcare delivery systems due to conflicting economic structures and objectives. Moreover, the uncertainty and complexity of both supply and demand for services makes it difficult to match supply to demand. To address this challenge, decision-making tools such as performance evaluation have been employed to enhance decision-makers' understanding of the decision context and its consequences. Performance evaluation is a continuous management process that involves measuring and managing organizational performance, taking into account the organization's environment and the decision-maker's perceptions and values (Bititci et al., 2012; Ensslin, L. et al., 2000; Roy, 1993).

Performance evaluation has been used to aid decision making in several contexts such as Finance (Ferreira et al., 2019), (Yilmaz et al., 2019), (Abdel-Basset et al., 2020; Aleskerov et al., 2004), health (Leonardo Ensslin et al., 2018), (Hsiao & Chen, 2019), (Ozcan & Tuysuz, 2016) agriculture (Taleb et al., 2023), waste management (Rodrigues et al., 2018) (Agarwal & Singh, 2022), urban services (Thiel et al., 2017; Zhou et al., 2022), human resource management (Rolim Ensslin et al., 2013), (Alshurideh et al., 2022), retail (Ozcan & Tuysuz, 2016), just to name a few. However, the sophisticated and dynamic feature of health care supply chain management makes performance evaluation more difficult (Burns et al., 2002).

In this regard, problem structuring methodologies and multiple criteria assessment methods are considered suitable instruments for performance assessment. The present study aims to develop an evaluation framework that assesses supply chain management in a public healthcare institution by integrating cognitive maps and the measuring attractiveness by a categorical-based evaluation technique (MACBETH) (Bana e Costa et al., 2012). Cognitive mapping is considered an instrument that contributes to reduce the number of omitted criteria in decision making contexts, promoting a better understanding of the criteria relationships (Eden, 2004), while MACBETH facilitates the estimation of trade-offs among evaluation criteria. The authors aim to contribute to the existing body of knowledge by designing and applying a multiple criteria decision support model constructed through an active learning technique, together with the supply chain management decision-maker in a public healthcare context. The proposed framework has the potential to aid decision-makers in improving the management of healthcare supply chains, thereby enhancing the quality and efficiency of healthcare delivery systems.

The rest of this paper is structured as follows. The next section provides a literature review on supply chain management and performance evaluation in the public sector and in healthcare, Section 3 describes the research methodology, while section 4 presents the application and results obtained. Section 5 concludes the study and points to directions for future research.

2 LITERATURE REVIEW

Public sector supply chain management consists of the handling of supply chains and multi-level networks. This ecosystem consists of private firms, which receive orders from public sector agents, accounting officers and policymakers. Public sector SCM not only concentrates institutions cooperating in goods and services, but it also shows how these institutions are involved with others operating at other levels. Therefore, both intra- and inter network collaborations are vital components (Migiro & Ambe, 2008).

Governments are the largest consumers of goods and services. To meet their objectives, their supply chain management (SCM) must ensure excellent quality, efficiency, and cost-effective delivery. Financial activity in the public sector may account for up to 30% of the GNP in the US and as much as 14-20% of the GDP in Europe (Angelo Mori & Doni, 2010; Callender & Matthews, 2000). Consequently, effective supply chain management can lead to significant cost savings. The entire government effort to improve the quality of current spending on goods and services used in the public sector must go through the modernization of the supply chain management (Tridapalli et al., 2011).

The global health care industry is one of the world’s largest and fastest growing industries, comprising various sectors: medical equipment and supplies, pharmaceutical, healthcare services, biotechnology, and alternative medicine sectors. Materials and logistics are, along with human resources and financial management, critical factors for the development of health care activities and operational excellence of the healthcare organization (Neil, 2004).

With extreme pricing pressures on today’s healthcare providers, delivering high-quality medical care while reducing costs is a top strategic priority. To achieve this objective, health care service provider efforts have been focused primarily on eliminating wastage in clinical operations. While these are valid and important ways to reduce healthcare costs, supply chain management (SCM) becomes a high management concern as it costs as much as 40 percent of a typical hospital’s operating budget, the second-largest expense for hospitals after labor (Darling & Wise, 2010). Moreover, by analyzing the coordination structure dimension of the supply chain management quality strategy level and the executive level, Wang and Shang (2023) have observed that there are several issues in the supply chain management that affect its quality improvement process, these include the incoming inspection department's failure to inspect as required, insufficient supplier control ability, and inadequate process and mechanism for managing supplier quality (Wang & Shang, 2023).

Actors in the health care supply chain comprise of i) producers who manufacture medical products such as surgical supplies, medical devices, and pharmaceuticals; ii) purchasers who include distributors, wholesalers, and GPOs. Distributors and wholesalers hold inventory for producers to facilitate delivery of products. GPOs sign purchasing contracts with producers in order to achieve economies of scale by aggregating the volume of member providers; iii) Health care providers who represent those at the end of the supply chain with the function to serve patients and include, among others, hospitals, integrated delivery networks (IDNs), physicians, clinics, nursing homes and pharmacies (Burns et al., 2002).

The public nature of hospital materials leads to lack of internal cost pressure and external efficiency motive, therefore many problems are evident in the public healthcare supply chain - the sluggish prediction of medical material demand as well as the patient-oriented response plan, ineffective information communication and coordination between supply and demand of materials and services, inadequate standardization of the management process, and the unreasonable external purchase, storage and distribution of materials (Chen, G.-s., 2010).

According to Neely et al. (1995), performance measurement can be defined as the quantification of the efficiency and effectiveness of an action, while a performance measure can be defined as a tool used to quantify the efficiency and/or effectiveness of an action. (Neely et al., 1995).

Melnyk et al. (2014) highlights that a performance evaluation system consists of two components: a performance measurement system and a performance management system. The former addresses the establishment of metrics as well as the collection and interpretation of data, while the latter seeks to evaluate the differences between actual and expected results; understand the reason for the irregularities; and introduce corrective measures to improve performance. These two systems form an “integrated system” for assessing organizational performance (Melnyk et al., 2014).

In this context, both the measurement and the management of organizational performance are shaped by the basic feelings, values and beliefs of the individuals and still have influence of the society in which they operate (Bititci et al., 2012). Therefore, the organization’s environment as well as the perceptions and values of the decision-maker should be considered in the development of a performance evaluation system. The performance evaluation process must thus consider the perception of the decision-maker; remain in alignment with the organization strategies; and be dynamic, since it is necessary to consider the environmental variables and periodically revise the strategies, objectives and metrics established for the adaptation to the environment (Bititci et al., 2000; Bourne et al., 2000).

Organizational performance measurement and metrics have received growing attention from researchers and practitioners. This is because these measures and metrics have been shown to affect strategic, tactical, and operational planning and control. Performance measurement and metrics have a significant role to play in setting objectives, evaluating performance, and determining future courses of action. Yet, performance measurement and metrics pertaining to SCM have not received similar attention from researchers or practitioners. (Gunasekaran et al., 2004).

Furthermore, there is a need to establish a more integrated Operations Management function across the supply chain, thus it becomes necessary to measure the performance of the various parts of the supply chain in various dimensions, in a consistent way. There is a need to define and measure performance for the supply chain as a whole and to be able to drill down to different measures and different levels of detail, in order to understand the causes of significant deviations of actual performance from planned performance (Lohman et al., 2004).

The main goal of SCM models and frameworks is to support management by measuring business performance as well as analyzing and improving business operational efficiency (Kurien & Qureshi, 2011), but with the little guidance available in the literature for the actual selection and implementation of Supply Chain Performance Measurement System, the process of choosing appropriate performance measures is difficult (Kurien & Qureshi, 2011). The hundreds of performance measures available for and used by different organizations in different industries can be broadly categorized into quality, financial, time, product flexibility, overall performance, and innovation (Elrod et al., 2013).

To manage this complex and dynamic environment, Supply Chain decision makers need the aid of a process that allows them to build knowledge on the decision-making context and thus promote due improvements to achieve the expected performance of the aspects considered most relevant by the final customers. This need can be satisfied through the Performance Evaluation carried out under the constructivist view (Ensslin, L. et al., 2010; Ensslin, L. & Vianna, 2008; Giffhorn et al., 2009; Tasca et al., 2010).

Therefore, it is necessary to evaluate the supply chain to make better use of resources in the face of a scarcity scenario. For this, it is important to search for various methods, performance indicator systems and technologies to evaluate, monitor, predict and optimize the supply chain performance. Soto Lopez et al. (2022) for example attempted to measure the internal hospital supply chain’s performance indicators to find their interdependencies with a view to understand the relationship among them and identify the key performance indicators for each of those aspects of the logistics process toward improvement. The authors used a rough group DEMATEL technique, by adapting DEMATEL effectively in an uncertain setting by using flexible rough intervals that reflect the subjective and vague decisions of the experts. (Soto Lopez et al., 2022).

In the work of Bhatti et al. (2015), a model was developed to analyze the performance of health care providers in Punjab, India using analytical modeling. The objective of this study was to develop a multidimensional quantitative performance evaluation model using Saaty’s analytical hierarchy process (AHP) approach (Bhatti et al., 2015; Saaty, 1980).

According to Fan & Zhang (2016), the performance evaluation procedure of a supply chain involves using the established performance indicators, employing an analytical method, following a particular procedure, performing quantitative or qualitative comparative analysis to provide an objective and accurate assessment of a supply chain. In the literature, several research propose systems and methods of performance indicators for the evaluation of the supply chain (Fan & Zhang, 2016).

Lega, Marsilio, & Villa (2013) affirm that a solid conceptual framework for the evaluation of supply chain management in public health organizations is not found in the literature. The authors then propose a model of evaluation of the performance of the supply chain in an Italian city. The authors start from the SCOR model and add some performance indicators that are more relevant to the health sector (e.g. safety in the category of organizational benefits and prices of supplies for financial benefit) (Lega et al., 2013).

Assessing the environmental performance of suppliers in the hospital supply chain in the United Arab Emirates, Malik et al. (2016) used the analytical hierarchical process to develop their model, transforming the qualitative assessment of a supply chain professional into a quantitative model (Malik et al., 2016). In addition, in hospital systems, Dotoli et al. (2015) proposes a model of performance evaluation of decision-making units on uncertainties. The authors used the technique of cross-efficiency fuzzy data envelopment analysis in a case study of the Apulian health system in Italy (Dotoli et al., 2015).

Supeekit et al. (2016) used an Analytical Network Process (ANP) to evaluate an internal supply chain of a hospital. The process was modified by the DEMATEL multiple criteria decision analysis method. The study uses the cited model to investigate the relationship between performance groups and calculates the weights of performance aspects (Supeekit et al., 2016).

Khaldi et al. (2017) investigate the feasibility of using the Adaptive Neurofuzzy Inference System (ANFIS) combined with DEA for performance evaluation of suppliers. Their model is aimed at modeling the performance measurement and forecasting of selected drug providers from a hospital (Khaldi et al., 2017).

In their studies, Chorfi et al. (2015) developed a classification and selection model of KPIs using MCDA. The model used AHP to select and compare relevant KPIs for the criteria in the pharmaceutical area. In Chorfi et al. (2016), they used a two-step DEA approach to evaluate the performance of the public pharmaceutical supply chain. The model sought to build a set of aggregate metrics that best characterize the performance of the public pharmaceutical supply chain, as well as to estimate relative efficiency and compare different public supply chains (Chorfi et al., 2016; Chorfi et al., 2015).

From data from 117 Greek public hospitals, Mitropoulos et al. (2015) combined stochastic DEA with Bayesian analysis to obtain statistical properties of efficiency scores. The generated model, Bayesian-CCDEA, was used to perform a sensitivity analysis to determine efficiencies under different probabilities of stochastic outputs, as well as to address the issue of DEA sensitivity for unreliable data, as in the case of a set of cross-sectional data (Mitropoulos et al., 2015).

Yazdani et al. (2020) integrated best worst method (BWM), decision-making model consisting of decision-making trial and evaluation laboratory (DEMATEL), and a modified version of evaluation based on distance from average solution (EDAS), to solve a supplier selection problem in a public procurement system with a view to meet sustainable development goals (Yazdani et al., 2020).

Performance evaluation in the public sector, particularly in a university hospital, has been previously addressed in studies conducted by (Longaray & Ensslin, 2014; Longaray et al., 2015; Longaray et al., 2018). These studies, including the present research, share a common hospital setting for their investigations. However, a key distinction lies in the approach and focus of the present study compared to the others. While the previous studies primarily concentrated on developing models to assist the management of Brazilian Federal University Hospitals in identifying, operating, and quantifying specific criteria essential for achieving or renewing the university hospital's certification with the Health and Education ministries, the present research takes a broader perspective. In contrast, the current study aimed to delve into a more comprehensive understanding of the indicators influencing supply chain management in a hospital setting. It seeks to shed light on how various critical aspects interact and contribute to the overall performance of the supply chain. These aspects extend beyond the traditional criteria emphasized in previous models.

In essence, the present study explores the multifaceted nature of supply chain management within a hospital, recognizing that it is not solely influenced by criteria directly related to operational efficiency indicators, material management efficiency, and financial performance. It takes into account factors from diverse areas that are intertwined with the supply chain, such as education, legal considerations, and human resources, to name a few. By doing so, this research offers a more holistic view of the complexities and interdependencies within the hospital's supply chain operations.

While (Longaray et al., 2015) seek to identify, organize and measure the aspects (criteria) that are considered relevant in the evaluation of the university hospital for the contractualization of the goals agreed with the SUS, (Longaray et al., 2018) describe the stages of developing a performance evaluation model for the internal logistics activities of a public teaching hospital to promote actions that could identify and propose alternatives to the logistics problems present in the institution. The focus of the latter is to aid decisions regarding resource allocation, planning, and supply management.

The comprehensive literature review carried out and published in (Longaray et al., 2023) has underscored the application of performance evaluation methods for assessing the overall performance of supply chain management in public healthcare. The review was instrumental in highlighting how MCDA models are applied, with most applications being concentrated in specific areas of the supply chain, such as logistics, procurement, supplier selection, pharmaceutical distribution, and internal operations. The review also noted that many studies derive their indicators from existing literature or through questionnaires administered in the workplace. These observations prompted the authors to opt for an MCDA model that accounts for the decision maker's context and reality.

Furthermore, based on the findings from the review, the authors observed a lack of integration between the supply chain and other areas of activity in a teaching hospital, which indirectly impact supply chain management, including education, legal, human resources, among other departments. This realization has driven the focus of the present study, emphasizing the significance of the other sectors in the effective management of the supply chain.

3 METHODOLOGICAL FRAMEWORK

The first step of the research was to conduct a literature review with the purpose of identifying the methodologies of performance evaluation of SCM in the public healthcare sector, the trends as well as pointing out research gaps to support the present research. The present study is grounded both on the Multiple Criteria Decision Analysis (MCDA) approach (Belton & Stewart, 2002; Roy, 1985), and cognitive mapping, an implementation of the Strategic Options Development and Analysis (SODA) methodology (Ackermann & Eden, 2001). Integrating MCDA with PSMs produces a deeper view of the decision context and enabling more effective support for distinct phases of the decision-making process. The combination of these two methodologies seeks to address the complex healthcare decision context by structuring the problem and assigning the weights according to decision-maker preferences and values.

To support the structuring of the decision problem, cognitive mapping was used. This is part of the SODA methodology originally conceived by Colin Eden for the structuring of complex decision problems (Ackermann, 2012; Belton & Hodgkin, 1999; Eden & Ackermann, 2004; Tegarden & Sheetz, 2003). Based on cognitive mapping techniques, the approach allows ideas to be structured, and that structure to be visualized, which facilitates dialogue and collaboration between decision agents, as well as the reorganization of different ideas and/or perspectives.

The cognitive maps used in the SODA methodology employ different techniques, including brainstorming and the listing of decision makers' goals and values (Longaray et al., 2018; Longaray et al, 2019). Cognitive maps are seen as structuring tools, the main advantages of which stem from the fact that they allow for a reduction in the number of omitted criteria, and promote learning through discussion and analysis of how these same evaluation criteria relate to each other (Eden & Ackermann, 2004). Indeed, cognitive maps' ability to deal with the complexity and subjectivity inherent in decision making is widely recognized. Human cognition is defined as a complex process that results from the interaction between the motor system and the neurological structures responsible for individuals' cognitive systems. In line with this, Eden (Eden, 2004) argues that a cognitive map is the representation of thinking about a problem that follows from the process of mapping. However, what truly characterizes a cognitive map is the aggregation of ideas, the ability to materialize thought and to allow cause-and-effect relationships to be identified. In practice, the problem description is a consequence of the coming together of the decision makers, and of the sharing of information which results from the development of cognitive maps.

The intervention instrument adopted to construct the model is the Constructivist Multi Criteria Decision Aid methodology (MCDA-C). This methodology is characterized as an evolution of traditional MCDA. The constructivist-based approach of MCDA-C methodology allows the decision maker to generate knowledge about their decision-making context and ensure their active participation in all stages of the methodology. The decision maker then identifies his/her consequences and impacts of future decisions, which are based on their values and preferences (Ensslin, L. et al., 2000; Ensslin, L., Dutra, et al., 2017; Ensslin, S. R. et al., 2013; Longaray et al., 2018; Tasca et al., 2012). The advantage of applying MCDA and measurement attractiveness by categorical-based evaluation technique (MACBETH) to supply chain management in public healthcare relates to a characteristic of our study-the development of an evaluation framework of strategic indicators from the manager’s point of view.

Drawing on the concept of constructivism, as proposed by Roy (1993), this study is based on the recognition that a decision maker must expand his or her understanding of the consequences of their decisions regarding the aspects that they deem to be important and, through constructed knowledge, evaluate these aspects and recommend improvements without imposing on the rationalism of objectivity (de Moraes et al., 2010; Ensslin, L. et al., 2010a; Lacerda et al., 2011; Roy, 1994).

Furthermore, in the constructivist paradigm, the decision is conceived, not as a moment, a pre-supposition of “decision making”, but rather as a process that is often long, conflicting, chaotic and uncertain (Ensslin, L. et al., 2010a). That is to say, the decision cannot be separated from the decision-making process and from the actors involved, because it is in the decision-maker that the necessary knowledge in a specific context is built on the consequences of the alternatives on their values and preferences, not on the values and preferences themselves. Thus, the decision-maker will select the most appropriate path to his or her needs and interests, contemplating the interaction with the environment and the people in it. Therefore, according to the constructivist paradigm, there is no “optimal solution” (Ensslin, L. et al., 2001; Keeney, 1996).

The main advantages of using MCDA-C are the possibility of addressing qualitative and quantitative information, the possibility of explicitly capturing and presenting the objectives and values of decision-makers, the possibility of allowing decision-makers to reflect on their objectives, priorities and preferences, and the possibility of developing a set of conditions and means to inform the decisions according to what the decision-maker considers most appropriate (Zamcopé et al., 2010) (Lacerda et al., 2011).

MCDA-C is operationalized in a systemic and systematic way through three sequential and interactive stages: structuring, evaluation and recommendation (Bana e Costa, 1993; Ensslin, L., Dezem, et al., 2017; Lacerda et al., 2014; Longaray et al., 2015), as shown in Figure 1. MCDA-C sets out to identify the objectives that the context actors wish to achieve, organizing these objectives hierarchically, measuring the objectives through ordinal and cardinal scales and aggregating the criteria for a global assessment from which improvement actions can be generated. Table 1 summarizes the steps taken along the three phases of the MCDA-C methodology.

Figure 1
Phases of the MCDA-C methodology.

Table 1
Description of the phases and stages of the MCDA-C methodology.

Phase	Stage	Description
Structuring	Soft approach to problem structuring	This step aims at explaining the context and achieving an understanding of the decision problem. The actors involved in the context are identified
	Family of Viewpoints	The Primary Assessment Elements (PAEs) are identified through brainstorming with the context actors. Through interviews with the actors, the PAEs are transformed into one or more action-oriented concepts. Finally, the concepts are grouped into Fundamental Points of View (FPV)
	Construction of descriptors	For each FPV the cognitive maps are developed together with the context actors. The cognitive maps seek to expand the knowledge about the means and ends relationships of the concepts built in the previous step and this process generates the rest of the Hierarchical Value Structure (HVS). For the concepts at the base of the hierarchical structure ordinal descriptors (scales) are constructed, establishing the performance levels ”good” and ”neutral”
Evaluation	Independence analysis	MCDA-C uses a compensatory model to build the global evaluation model. This model assumes that the compensation rates used in integration are constant. To achieve this condition, the criteria must be independent
	Construction of value functions and compensation rates	Ordinal scales are transformed into cardinal scales and compensation rates are identified using MACBETH software
	Identifying the impact profile of alternatives	Identifying the impact profile of alternatives: the status quo performance and the impact of actions are evaluated
	Sensitivity analysis	Sensitivity analysis is used to evaluate the robustness of the valuation model
Recommendations	Elaboration of recommendations	The recommendations phase is intended to support the decision-maker in identifying improvement actions, as well as understanding the consequences of these actions on the strategic objectives

• Source: Schlickmann & Bortoluzzi, 2023.

Due to space limitations, a detailed description of the application of the methodology cannot be provided here, however, the phases of the method are summarized in Table 1 below to illustrate the steps taken along the model creation. A detailed MCDA-C procedure can be found in (Ensslin, L. et al., 2020; Lacerda et al., 2011; Longaray et al., 2018; Martins et al., 2023; Rodrigues et al., 2018; Silva da Rosa et al., 2012).

4 APPLICATION AND RESULTS

The first step of the structuring stage aims at explaining the context and achieving an understanding of the decision problem. The actors involved in the context are also identified. The present study was developed at the Dr. Miguel Riet Corrêa Jr. Public University Hospital (HU-FURG), which treats patients exclusively through the Unified Health System (SUS) since 2011, aiming at the integration of teaching with assistance. Its unique role is to implement public policies of SUS as well as promoting and encouraging development of teaching, research, and outreach programs in the health area.

The actors with the authority in the process and those interested in the decisions to be taken were identified. The stakeholders involved were the administrative manager (decision-maker), superintendent, expenses authorizer, head of administration sector, and the head of supplies. The decision maker, the facilitator and the acted are always present in all MCDA-C applications, while the rest step in only when the decision maker deems necessary. Table 2 shows the stakeholders involved in the decision context and their roles.

Table 2
Context Actors.

Actor	Role
Decision maker	Administrative manager.
Interveners	Superintendent, Expenses Authorizer, Head of Administration Sector, Head of Supplies.
Facilitator	Authors of the study.
The acted	Community of the Southern Region of RS, other employees of the company, and suppliers.

• Source: Research data.

The Administrative Management (GAD) is the unit of Dr. Miguel Riet Corrêa Jr. University Hospital (HU/FURG-EBSERH) responsible for managing and implementing administrative, budgetary, financial, equity and accounting policies within the hospital; by signing the financial processes for payment, after the analysis by the Expenditure Settlement Unit, submitting them to the deliberation of the Expenditure Authorizing Officer; and for managing and implementing logistics, hospital infrastructure, and people management policies. It is the unit responsible for the entirety of the hospital supply chain.

The model developed in this research was constructed according to the perceptions of the administrative manager through information obtained in semi-structured interviews. The facilitator guided the dialogue to ensure its objectivity and adherence to the topics proposed by MCDA-C. During this step, the authors avoided forming conclusions and identifying alternatives since the problem was not yet clear. Based on the interviews conducted with the decision-maker in the study, it was possible to identify the main difficulties and the consequences in the management process of the hospital’s supply chain, as was indicating the aspects desirable for managers to ensure a satisfactory performance of the services provided.

After the definition of the actors and the description of the problem environment, the authors proceeded to define a label for the problem, a title empathically defined with the decision maker and that synthesizes the decision-maker's dissatisfaction and what he wishes to do in relation to this situation. In the present work, the label was established as: “A framework to aid the management of the supply chain of HU-FURG/EBSERH.”

The second step starts with the identification of the Primary Assessment Elements (PAEs) and is based on the interaction of the facilitator with the decision-maker, in which the facilitator is motivated to discuss the management process of the decision-maker, reporting on his concerns, desires and motivations (Ensslin, L. et al., 2000; Ensslin, L. et al., 2001). It is highlighted that, according to the decision maker, the management of a supply chain of a public health care institution, a teaching hospital to be specific, must encompass all factors that interfere and/or are affected. This means that the SCM must involve all major aspects of hospital management.

This process culminated in the identification of 98 PAEs. The PAEs deal only with initial information on the problem of the decision maker, making it necessary to expand their knowledge about each PAE identified. Thus, the 98 PAEs were expanded into 427 concepts, from which the decision maker validated 228 of these as relevant to his decision context, with the goal of representing the objective associated with each PAE. Initially, the PAEs were action-oriented having a present pole, that is, the direction of preference of the decision maker and its opposite pole, which corresponds to the minimum performance that the decision maker wishes to avoid (Ensslin, L. et al., 2001); Table 3 presents a sample of PAEs and their respective concepts.

Table 3
Sample of PAEs and Concepts of the model.

PAE	CONCEPT
Information system management	1. Reducing rework due to diverse data sources ... Find errors in the flow of information
University support	2. Receive support from the University ... Have difficulty solving problems
Competencies	3. Adapt professional expertise to the needs of the sector ... Ignore the employee’s professional training
Group order items	4. Make the group attractive ... Lack supplier for that group
Involvement and Responsibility	5. Stimulate the commitment of employees ... have unproductive employees at the workplace
Institution Growth	6. Lead uniform growth in all areas ... Focus only in some specific areas

• Source: Research data.

Due to similarities of some concepts, it was possible to group them in eleven areas of concern, called Fundamental Points of View (FPVs): “Administration”, “Budget-Finance”, Hospital Infrastructure, “Information technology”, “Teaching”, “Purchases”, “Logistics”, “Politics”, “Legislation”, “Planning”, and “Human Resources management”. With the identification of the areas of concern, cognitive maps were constructed through the organization and hierarchy of concepts and their influence relationships (Ensslin, L. et al., 2010a; Ensslin, L. et al., 2001). The structuring phase is a stage of constant learning and, for that reason, after each validation process the decision maker made changes and refinements in the model. Based on this activity, it is possible to increase the decision maker's understanding of each area of concern, so that the means concepts can be identified to the strategic concepts (Lacerda et al., 2011). Figure 2 shows the Cognitive Map constructed for the FPV “Logistics”.

Figure 2
Extract of the cognitive Map of FPV ”Logistics”.

Cognitive maps were constructed for all FPVs, and it was possible to identify similar concepts that make up the areas of concern of the decision maker. Thus, in order to facilitate the analysis process, the cognitive Maps were divided into smaller maps, thus forming clusters and the name given to each cluster is the result of the decision maker's focus on the objective (Ensslin, L. et al., 2010a; Ensslin, L. et al., 2001). After the identification of the clusters, these were transported to the hierarchical structure of value (HSV), as shown in Figures 3 and 4 for the clustering of concepts, and Figure 5 (HSV). After completing the HSV, the next step is to construct the descriptors.

Figure 3
An extract from the clustering of concepts in Areas of Concern.

Figure 4
Clustering of concepts into Areas of Concern.

Figure 5
Hierarchical Structure of Value for FPV ”Logistics”.

The construction of descriptors consists of establishing, along with the decision maker, the ordinal scales used to measure each point of view, respecting the principles of the Measurement Theory (legitimacy and validation), as highlighted by (Micheli & Mari, 2014). These authors report that the construction of scales must comply with the principles of legitimacy (ordinality) and validation (cardinality). According to the authors, legitimacy is shown by the requirement of objectivity, accuracy, and precision, while validity is explained by the observance of the type of operation performed by the scales against the degrees of freedom admissible by it. Legitimacy (objectivity, accuracy and precision) is operationalized by the observance of six properties (Keeney, 1992). The validation principle, within the perspectives established by Micheli and Mari (2014), is achieved by observing the operations to be performed with the indicators and the degree of information contained in the scales as proposed by Stevens (1946). Thus, the validity of the model was met, respecting the limits for the statistical tests of the scales (Stevens, 1946).

Once the model scales were defined, to allow the comparison between the descriptors, the decision maker set Good and Neutral reference levels for each. These levels establish three situations: Values above the Good level, indicate that the performance is excellent, above the expectations of the decision maker; those below the Neutral level, indicate that the performance is compromising, below the expectations of the decision maker; and between the Neutral and Good levels, indicate that the performance is equivalent to the expected (Ensslin, L. et al., 2001).

An example of the reference levels of a descriptor is shown in Table 4, while Figure 6 shows the descriptors of ‘Logistics’ area of concern.

Figure 6
Extract of Descriptors of FPV “Logistics”.

Table 4
Reference levels for the descriptor of EPV 1.1.2.1 - Collaborators.

Descriptor of EPV 1.1.2.1 - % of achievement of projected number of collaborators
REFERENCE LEVELS	LEVELS OF IMPACT	ORDINAL SCALE
	L5	100%
GOOD	L4	95%
	L3	90%
NEUTRAL	L2	85%
	L1	80%

Thus, the scales for each of the Points of View of the HSV were constructed, as exemplified for the “Logistics” FPV in Figure 6. The completion of construction of the scales of the model marked the start of the evaluation phase of MCDA-C. The evaluation process allows the assessment and global integration of the model and is divided into four stages: (i) construction of value functions; (ii) identification of compensation rates; (iii) development of the global evaluation; and (iv) identification of the profile impact of the alternatives.

After defining the descriptors to evaluate the potential actions of each Point of View, we now quantify their performance according to the value system of the decision maker. This process stems from the need to transform the ordinal scales of the descriptors into cardinal scales. The construction of value functions is necessary since ordinal scales are not considered numerical scales, even when they consist of numbers that represent only alphanumeric symbols and thus do not belong to the set of real numbers. Thus, the MCDA-C methodology recognizes the misconception of using ordinal scales for functions involving numeric operations (Ensslin, L. et al., 2010a). For this, one can use value functions, which are a tool that help decision makers to articulate their preferences, ordering them (differences in attractiveness) between pairs of impact levels or potential actions (Ensslin, L. et al., 2000).

Several methods are used to construct value functions: Direct Punctuation, Bissection and Semantic Judgment. In this case study, semantic judgment is used, where the value function is obtained by the Measuring Attractiveness by Categorical Based Evaluation Technique (MACBETH) method, through peer-to-peer comparison of the difference in attractiveness between potential actions (Bana e Costa & Vansnick, 1995).

From the constructed descriptors, the decision maker was asked to verbally express the difference in attractiveness between two potential actions. As an example, in descriptor of the EPV 8.1.1 Best Practices, the decision maker was asked: ”What is the difference of attractiveness between 100% dissemination of best practices (L5) and 95% dissemination (L4)? The answer to this question is based on seven categories: (i) null; (ii) very weak; (iii) weak; (iv) moderate; (v) strong; (vi) very strong and (vii) extreme. The same procedure was performed in the other levels of impact of the descriptor forming a matrix, which by the MACBETH method determines the value function corresponding to the semantic judgments of the decision maker. The values of 0 for the ”Neutral” impact level and 100 for the ”Good” impact levels were set for each descriptor. Figure 7 exemplifies the value function obtained for EPV 8.1.1 - Best Practices.

Figure 7
Construction of the Value Function for Best Practices.

Figure 8 presents the descriptors of the “Politics” FPV with their respective ordinal and cardinal scales. The descriptors of the other FPVs along with their respective scales can be found in the supplementary material.

Figure 8
Hierarchical Structure of Value and Scales of the “Politics” FPV.

From this point, the decision maker can cardinally measure every aspect he considers relevant, in an isolated form. However, to conduct global evaluation of the model, considering all the criteria simultaneously, it is necessary to identify the compensation rates for each EPV and FPV. In order to be able to aggregate local information and obtain a global assessment, it is necessary find out the substitution rates associated with the criteria (Ensslin, L. et al., 2001), also known as compensation rates. For this, the decision maker will define the order of his preference among the existing alternatives. The process, conducted for the entire model, is exemplified in Figure 11 for the eleven FPVs.

Figure 9
Extract from the identification of alternatives.

Figure 10
Roberts' Matrix.

Figure 11
Global Compensation rates of all FPVs

In the next step, the alternatives must be ordered according to the preference of the decision maker. For this step, Roberts’ Matrix is used (Roberts, 1985), and the options are compared in pairs, as can be observed in Figure 10.

It is concluded from this matrix that the order of preferences of the alternatives for the decision maker is:

$A2>A11>A1>A6>A7>A3>A4>A5>A10>A8>A9>A0$

From this information we have the ordinal scale of the alternatives. To convert them into a cardinal scale, the M-Macbeth software is again used and, by means of semantic judgment, it is possible to construct the compensation rates.

The procedure shown in Figures 9, 10 and 11 was repeated for the entire model, and the Hierarchical Structure of Value with the ordinal and cardinal scales and compensation rates for the entire model was constructed.

The last step of the evaluation phase entails the global assessment of the model and the identification of the alternatives. With compensation rates determined, local assessments of each action were aggregated into a single global evaluation. For this purpose, the additive aggregation formula is used (Bana e Costa, 1993; Bana e Costa et al., 2012).

Figure 12
FPV “Politics” with its compensation rates and impact profile.

The formula for calculating the overall performance of the model is:

$V_{PEHU}\left(a\right)=0.1101\times V_{FPV1}\left(a\right)+0.1222\times V_{FPV2}\left(a\right)+0.0919\times V_{FPV3}\left(a\right)+0.0858\times V_{FPV4}\left(a\right)+0.0797\times V_{FPV5}\left(a\right)+0.1040\times V_{FPV6}\left(a\right)+0.0979\times V_{FPV7}\left(a\right)+0.0641*V_{FPV8}\left(a\right)+0.0615\times V_{FPV9}\left(a\right)+0.0667\times V_{FPV10}\left(a\right)+0.1161\times V_{FPV11}\left(a\right)$ (1)

In this equation, V(a) corresponds to the global score of the performance evaluation model for the SCM of HU-FURG/EBSERH and (a) refers to the evaluation of the criteria linked to each FPV. For the calculation process of the FPVs, the following equation is used, as exemplified by the calculation of the “Politics” FPV:

$V_{FPV8}\left(a\right)=0.2286\times V_{Activities}\left(a\right)+0.4285\times V_{Succession}\left(a\right)+0.3429\times V_{Stability}\left(a\right)$ (2)

Where:

$\begin{array}{c}{V}_{Activities}\left(a\right)=0.6471\times V_{BestPractices}\left(a\right)+0.3521\times V_{Networks}\left(a\right)\\ V_{Succession}\left(a\right)=0.5385\times V_{TeamMaintenance}\left(a\right)+0.4615\times V_{ContinuityOfActions}\left(a\right)\\ V_{Stability}\left(a\right)=0.6250\times V_{Strengthening}\left(a\right)+0.3750\times V_{ExternalPressure}\left(a\right)\end{array}$ (3)

Therefore:

$V_{FPV8}\left(a\right)=0.2286\times 66.62+0.4285\times 100+0.3429\times 100$ (4)

$V_{FPV8}\left(a\right)=92.37$ (5)

With compensation rates determined, it was necessary to identify the impact profile of the evaluated criteria. Based on this identification, it is possible to get a picture of which elements of the model present opportunities for improvement, directing the strategic actions to increase the performance of the context. As a result of the construction of the compensation rates and the application of the additive aggregation formula in the model, the total value of 51.55 points was obtained. Figure 13 shows the scores obtained for all the FPVs of the model.

Figure 13
Impact profile of the alternatives in the model FPVs.

As demonstrated by the figure above, the ‘Legislation’ FPV presented the highest score (103.96), thus representing an area with a significant contribution to the global assessment. This result can show an area in which the decision maker is content with the current performance. The lowest score of -6.93 in the teaching area of concern highlights the fact that the decision maker has minimum control over the aspects that involve it, which makes it difficult to control and adopt corrective actions, when necessary, when compared to the aspects of the other FPVs of the model. The other areas of concern such as Human Resources Management point to areas where the decision maker encounters more management difficulties, mainly because he has little or no control over the analyzed aspects and his performance is conditioned to external events, such as approval from the federal government to hire more professionals or depends on staff from areas out of his reach.

The knowledge of compensation rates and the model score allowed the manager to have a broad view of his management process, identifying the criteria in which SCM presents a compromising level and that need to be improved to reach higher levels of performance. This holistic view also allows the manager to find out the impact of his management on the criteria he considers essential. After an analysis of the global evaluation, it is noted that the area that currently compromises the supply chain management most is ‘Teaching.’ In this case, a negative score is clearly a concern factor for the decision maker, since the area represents 7.97% of the overall evaluation, thus improvement actions are indicated. Another point of concern is Human resources management, which presented a score of 12.52 in the evaluation and is at a compromising level. Being one of the areas that contributes most to the global model (11.61%), it requires special attention from the decision maker. In this case, in addition to recommending improvements in the criteria at a compromising level, it is suggested to pay attention to those criteria that, even though not at the compromising level, can be improved at a low cost. This can be extended to the other most influential areas of concern according to the decision maker: Administration (11.01%), Budget and Finance (12.22%), Purchases (10.40%) and Logistics (9.79%).

Sensitivity analysis consists of verifying that the final model result of the alternatives varies by changing the parameter values, i.e. it allows to know if a change in a parameter, such as the compensation rates, will cause a great variation in the evaluation of potential actions (Ensslin, L. et al., 2001). The sensitivity analysis was conducted on compensation rates to illustrate how a change in a compensation rate can change the other compensation rates. An increase and decrease by 10% in the compensation rates of the EPVs did not generate expressive changes in the overall evaluation, since the order of preference of actions remained constant, confirming that the model shows little sensitivity to changes in compensation rates, confirming the robustness of the developed model.

After completing the evaluation phase of the model, the next step is to propose improvement actions that comprise the last phase of the model. This phase of the MCDA-C is referred to as the recommendations phase and aims to aid the decision maker in leveraging the constructed model, through suggestions for each of the descriptors identified with compromising performance in the decision evaluation context, that is, the determination of the impact profile of the current situation. It is important to emphasize that the recommendations phase does not aim to prescribe actions, but to aid the decision maker in identifying the areas that need improvement, through the elaboration of improvement actions and in the evaluation of its consequences (Azevedo et al., 2011; Ensslin, L., Dutra, et al., 2017; Longaray et al., 2015). The recommendations phase is comprised of formulation of recommendations.

From the identification of the impact profile of the model, the criteria that need improvement are analyzed, i.e., the descriptors that presented performance at a compromising level. It is observed that 21 descriptors presented a compromising level of performance. From the analysis of the underperforming indicators, improvement actions were elaborated for each descriptor, through goals and action plans. Table 5 presents the proposed action plan for EPV 4.3.1 Integration, which addresses the integration of information technology systems in the supply chain management. The improvement of this descriptor is important as it affects the entire supply chain process. Chen and Shu (2023) reported that advanced management technology, through information systems for example, improves actual management work (Chen, H. & Su, 2023).

Table 5
Improvement measure template for the descriptor for EPV 4.3.1 Integration.

Critical performance factors to be improved:	EPV 4.3.1 Integration
Descriptor:	% of integrated systems
Proposed Actions:	- Create a multidisciplinary working group (WG) to map all candidate Integration processes.
	- Establish a schedule for the working group findings and final report.
	- Request the IT sector to support the WG.
	- Implement the integration of all the organization processes in the system.
	- Encourage the staff to actively participate in the system tests;
Responsible:	Head of the IT Sector
Deadline:	First semester of following year
Supervision Frequency:	Monthly
How it will be supervised:	Monitoring the Schedule with the working group
Responsible for supervising:	General Manager
Expected result:	Change from ‘<80% Integration of systems’ to ‘80% Integration of systems’
Descriptor Impact:	Improving from L1 (-133.33) to L4 (100)

Based on the implementation of the proposed improvements, the performance of the supply chain management of HU-FURG/EBSERH is expected to increase. If the proposals are successfully implemented, they will result in an increase of 24.7 points, raising the model score from 51.55 to 76.25 points. Looking at the areas of concern individually, especially at the Budget and Finance FPV, which contributes the most to the global score, improvement actions on a single EPV, such as ‘2.2.3 Operational Security’, may impact the entire model by almost 2%, a total of 30.30 points on this FPV alone and 3.67 points on the global score), presenting a higher improvement potential of the model when compared to the other FPVs that compose the model.

After the improvement actions have been implemented, the monitoring of the impact on the respective performance indicators and on the overall evaluation of the model should be conducted continuously, updating the performance of each one indicated. Based on the improvements that are being made, it is necessary to change the target for each indicator, thus establishing a management process with continuous evaluation cycles and focused on results. The construction of the present Supply Chain Management Performance evaluation model was based on the values, beliefs, and world view of the General Manager, responsible for making all decisions related to the health care supply chain.

Some of the indicators listed herein were found in extant literature, however, although some indicators are very similar to those of the model presented in this research, none of them are exactly analogous to the SCM of HU-FURG/EBSERH. Table 6 shows indicators from the model constructed in the present study that were similar to those found in literature. It is observed that the rest of the indicators found in the present study were characteristic to the ad hoc model constructed and were thus not found in literature.

Table 6
Comparison of the model indicators with those found in literature.

Indicator in present study	Indicator in literature	Authors
Screening/stay	no delay in treatment	(Supeekit et al., 2016)
Installed capacity	overcrowding	(Supeekit et al., 2016)
Mapping costs	total cost per patient	(Supeekit et al., 2016)
screening	total waiting time of patient	(Supeekit et al., 2016)
stay	patient throughput or turnaround time	(Supeekit et al., 2016)
Access time and distance	Accessibility to services	(Supeekit et al., 2016)
Mapping costs	cost of inventory	(Supeekit et al., 2016)
Ideal Quantity	obsolete stock cost/overstocking cost	(Supeekit et al., 2016)
Standardization Commission	Standardization	(Lega et al., 2013)
	Staff specialization	(Lega et al., 2013)
	Safety	(Lega et al., 2013)
Investment in IT	Technological Investment	(Lega et al., 2013; Yazdani et al., 2020)
Supplier satisfaction	Supplier relationship management	(Lega et al., 2013)
Information flow	accessibility and flow of information	(Longaray et al., 2018)
Integration of systems	Computerization of stock control systems	(Longaray et al., 2018)
Suppliers with Best Practice Certification	Suppliers’ capabilities	(Hamid Abu Bakar et al., 2009)
Suppliers with Best Practice Certification	Supplier evaluation	(Kumar et al., 2005)
Resolvability	Solvability rate	(Kumar et al., 2005)
	Efficiency of IT system	(Kumar et al., 2005)
Effectiveness of Terminals	Effectiveness of IT system	(Kumar et al., 2005)
Trained users	Training utilization rate	(Kumar et al., 2005)

With the analysis of the indicators presented in the literature, it is observed that some of them present a similar idea to those constructed for the supply chain management of the HU-FURG/EBSERH but differ on what is to be measured or the scales to be used. On the scales used, it is observed that those extracted from literature present only one statement/objective, but not necessarily the direction of preference. These divergences are completely acceptable, since the constructivist approach of the MCDA-C methodology assumes that the model is customized and unique to the application context in which the health care institution is inserted, and its construction considers the specificities and characteristics of the institution and the decision maker's perception.

5 FINAL CONSIDERATIONS

The present study aimed to construct an individualized multicriteria model to assess the management performance of a public health care supply chain. To attain this goal, it was essential to identify the decision context and point out the actors involved, to structure the problem according to the values and preferences of the decision maker to identify the criteria that affect the supply chain management and to finally develop the structure of the multicriteria framework, analyzing the obtained results and identify improvement opportunities. The main criteria capable of generating the greatest positive impact on the SCM process were also highlighted, recommendations that, if implemented, have the potential to improve management performance by 47.92%, from the global value of 51.55 to 76.25. It was observed that specific areas: Teaching, Human Resources Management, Planning, Information Technology and Hospital infrastructure need extra managerial efforts to improve criteria in these areas of concern due to the low scores obtained.

The participation of the decision-maker occurs in all model construction stages, however, it is in the Structuring and Evaluation stages that their participation in the model gets clearer, not only aggregating their perceptions, beliefs and opinions to the model being developed, but also through knowledge acquired by structuring the data collected during these stages. According to the decision maker, it is difficult to manage the supply chain without considering all institutional aspects that affect health care demand and supply. This explains the amplitude and specificity of the performance evaluation model constructed through the 160 descriptors established.

Regarding future research, the authors recommend further studies on supply chains in the public sector context since literature is still scarce on efforts of SCM of different areas in the public sector, with special attention to the disruptions caused by the COVID-19 outbreak. Public administration is a broad field of study and according to the analysis of scientific literature, there are limited studies that relate the tools or methodologies of performance evaluation with the innumerous processes with public service. In addition, it is recommended to apply the MCDA-C methodology in the development of other models of performance evaluation in other processes, activities, and functions in the public sector.

ACKNOWLEDGEMENTS

The authors wish to express gratitude to the manager and staff at the HU-FURG/EBSERH hospital for their cooperation during the execution of the study.

REFERENCES

• ABDEL-BASSET M, DING W, MOHAMED R & METAWA N. 2020. An integrated plithogenic MCDM approach for financial performance evaluation of manufacturing industries. Risk Management, 22: 192-218.
• ACKERMANN F. 2012. Problem structuring methods ‘in the Dock’: Arguing the case for Soft OR. European journal of operational research, 219(3): 652-658.
• ACKERMANN F & EDEN C. 2001. SODA-journey making and mapping in practice.
• AGARWAL S, & SINGH AP. 2022. Performance evaluation of textile wastewater treatment techniques using sustainability index: An integrated fuzzy approach of assessment. Journal of Cleaner Production, 337: 130384.
• ALESKEROV F, ERSEL H & YOLALAN R. 2004. Multicriterial ranking approach for evaluating bank branch performance. International Journal of Information Technology & Decision Making, 3(02): 321-335.
• ALSHURIDEH MT, AL KURDI B, ALZOUBI HM, GHAZAL TM, SAID RA, ALHAMAD AQ & AL-KASSEM AH. 2022. Fuzzy assisted human resource management for supply chain management issues. Annals of Operations Research, 1-19.
• ANGELO MORI P & DONI N. 2010. The economics of procurement contract awarding: problems and solutions in theory and practice. Journal of public procurement, 10(1): 93-120.
• AZEVEDO RC, ENSSLIN L, LACERDA RTO, FRANÇA LA, GONZÁLEZ CJI, JUNGLES AE & ENSSLIN SR. 2011. Avaliação de desempenho do processo de orçamento: estudo de caso em uma obra de construção civil. Ambiente Construído, 11(1): 85-104.
• BANA E COSTA CA. 1993. Três convicções fundamentais na prática do apoio à decisão. Pesquisa operacional, 13(1): 9-20.
• BANA E COSTA CA, DE CORTE J-M & VANSNICK J-C. 2012. MACBETH. International Journal of Information Technology & Decision Making, 11(02): 359-387. doi: 10.1142/S0219622012400068
  » https://doi.org/10.1142/S0219622012400068
• BANA E COSTA CA, & VANSNICK J-C. 1995. General overview of the MACBETH approach. In Advances in multicriteria analysis (pp. 93-100): Springer.
• BARATA LRB, MENDES JDV & BITTAR OJNV. 2010. Hospitais de ensino e o Sistema Único de Saúde. Rev adm saude, 12(46): 7-14.
• BELTON V & HODGKIN J. 1999. Facilitators, decision makers, DIY, users: Is intelligent multicriteria decision support for all feasible or desirable? European journal of operational research, 113(2): 247-260.
• BELTON V & STEWART T. 2002. Multiple criteria decision analysis: an integrated approach: Springer Science & Business Media.
• BHATTI SK, SINGH RP & SINGH I. 2015. Analytical model for performance measurement in healthcare sector of Punjab. Journal of modelling in management, 10(2): 226-237.
• BITITCI U, GARENGO P, DÖRFLER V & NUDURUPATI S. 2012. Performance measurement: challenges for tomorrow. International journal of management reviews, 14(3): 305-327.
• BITITCI U, TURNER T & BEGEMANN C. 2000. Dynamics of performance measurement systems. International journal of operations & production management, 20(6): 692-704.
• BOURNE M, MILLS J, WILCOX M, NEELY A & PLATTS K. 2000. Designing, implementing and updating performance measurement systems. International journal of operations & production management.
• BRASIL. 2017. Cadastro Nacional de Estabelecimentos de Saúde (CNES). In: Datasus Brasília.
• BURNS LR, DEGRAAFF RA, DANZON PM, KIMBERLY JR, KISSICK WL & PAULY MV. 2002. The Wharton School study of the health care value chain. The health care value chain: producers, purchasers and providers. San Francisco: Jossey-Bass, 3-26.
• CALLENDER G & MATTHEWS D. 2000. Government purchasing: an evolving profession? Journal of public budgeting, accounting & financial management, 12(2): 272-290.
• CHATTERJEE S, CHAUDHURI R & VRONTIS D. 2023. Supply chain sustainability during an uncertain environment: examining the moderating role of leadership support International Journal of Technology Management. doi: 10.1504/IJTM.2022.10048431
  » https://doi.org/10.1504/IJTM.2022.10048431
• CHEN G-S. 2010. On the supply chain management of public hospital materials. Paper presented at the 2010 International Conference on E-Product E-Service and E-Entertainment.
• CHEN H & SU W. 2023. Innovation and application of enterprise's supply chain management mode under the background of Industry 4.0. International Journal of Technology Management, 91(1-2), 19-38.
• CHORFI Z, BENABBOU L & BERRADO A. 2016. A two stage DEA approach for evaluating the performance of public pharmaceutical products supply chains. Paper presented at the 2016 3rd International Conference on Logistics Operations Management (GOL).
• CHORFI Z, BERRADO A & BENABBOU L. 2015. Selection of key performance indicators for supply chain monitoring using MCDA. Paper presented at the 2015 10th International Conference on Intelligent Systems: Theories and Applications (SITA).
• CHOWDHURY P, PAUL SK, KAISAR S & MOKTADIR MA. 2021. COVID-19 pandemic related supply chain studies: A systematic review. Transportation Research Part E: Logistics and Transportation Review, 148, 102271. doi: https://doi.org/10.1016/j.tre.2021.102271
  » https://doi.org/10.1016/j.tre.2021.102271
• COOPER MC, LAMBERT DM & PAGH JD. 1997. Supply chain management: more than a new name for logistics. The international journal of logistics management, 8(1): 1-14.
• DA ROSA FS, ROLIM ENSSLIN S, ENSSLIN L & LUNKES RJ. 2012. Environmental disclosure management: a constructivist case. Management Decision, 50(6): 1117-1136.
• DARLING M & WISE S. 2010. Not your father's supply chain. Following best practices to manage inventory can help you save big. Materials management in health care, 19(4): 30-33.
• DE MORAES L, GARCIA R, ENSSLIN L, DA CONCEIÇÃO MJ & DE CARVALHO SM. 2010. The multicriteria analysis for construction of benchmarkers to support the Clinical Engineering in the Healthcare Technology Management. European journal of operational research, 200(2): 607-615.
• DOTOLI M, EPICOCO N, FALAGARIO M & SCIANCALEPORE F. 2015. A cross-efficiency fuzzy data envelopment analysis technique for performance evaluation of decision making units under uncertainty. Computers & Industrial Engineering, 79: 103-114.
• EDEN C. 2004. Analyzing cognitive maps to help structure issues or problems. European journal of operational research, 159(3): 673-686.
• EDEN C & ACKERMANN F. 2004. Cognitive mapping expert views for policy analysis in the public sector. European journal of operational research, 152(3): 615-630.
• ELROD C, MURRAY S & BANDE S. 2013. A review of performance metrics for supply chain management. Engineering Management Journal, 25(3): 39-50.
• ENSSLIN L, DEZEM V, DUTRA A, ENSSLIN SR & SOMENSI K. 2017. Management support for agricultural enterprises: a case study for a fruit-producing company. International Food and Agribusiness Management Review, 20(4): 493-510. doi: 10.22434/ifamr2016.0152
  » https://doi.org/10.22434/ifamr2016.0152
• ENSSLIN L, DUTRA A & ENSSLIN SR. 2000. MCDA: a constructivist approach to the management of human resources at a governmental agency. International transactions in operational Research, 7(1): 79-100.
• ENSSLIN L, DUTRA A, ENSSLIN SR, KRÜGER AC & GAVAZINI AA. 2017. Avaliação multi-critério de desempenho: o caso de um Tribunal de Justiça. Cadernos Gestão Pública e Cidadania, 22(71).
• ENSSLIN L, GIFFHORN E, ENSSLIN SR, PETRI SM & VIANNA WB. 2010. Avaliação do desempenho de empresas terceirizadas com o uso da metodologia multicritério de apoio à decisão-construtivista. Pesquisa Operacional, 30(1): 125-152.
• ENSSLIN L, MUSSI CC, DUTRA A, ENSSLIN SR & DEMETRIO SN. 2020. Management support model for information technology outsourcing. Journal of Global Information Management (JGIM), 28(3), 123-147.
• ENSSLIN L, NETO GM & NORONHA SM. 2001. Apoio à decisão: metodologias para estruturação de problemas e avaliação multicritério de alternativas: Insular.
• ENSSLIN L & VIANNA WB. 2008. O design na pesquisa quali-quantitativa em engenharia de produção-questões epistemológicas. Revista Produção Online, 8(1).
• ENSSLIN SR, ENSSLIN L, BACK F & LACERDA R. 2013. Improved decision aiding in human resource management: a case using constructivist multi-criteria decision aiding. International Journal of Productivity and Performance Management, 62(7): 735-757.
• FAN X & ZHANG S. 2016. Performance evaluation for the sustainable supply chain management. Sustainable Supply Chain Management, 141.
• FERREIRA FA, MEIDUTÉ-KAVALIAUSKIENÉ I, ZAVADSKAS EK, JALALI MS & CATARINO SM. 2019. A judgment-based risk assessment framework for consumer loans. International Journal of Information Technology & Decision Making, 18(01): 7-33.
• GIFFHORN E, ENSSLIN L, ENSSLIN SR & VIANNA WB. 2009. Aperfeiçoamento da gestão organizacional por meio da abordagem multicritério de apoio à decisão. Revista Gestão Industrial, 5(4).
• GUNASEKARAN A, PATEL C & MCGAUGHEY RE. 2004. A framework for supply chain performance measurement. International journal of production economics, 87(3): 333-347.
• HAMID ABU BAKAR A, LUKMAN HAKIM I, CHOY CHONG S & LIN B. 2009. Measuring supply chain performance among public hospital laboratories. International Journal of Productivity and Performance Management, 59(1): 75-97.
• HSIAO B & CHEN L-H. 2019. Performance evaluation for Taiwanese hospitals by multi-activity network data envelopment analysis. International Journal of Information Technology & Decision Making, 18(03): 1009-1043.
• KEENEY RL. 1992. Value-focused thinking: A path to creative decisionmaking: Harvard University Press.
• KEENEY RL. 1996. Value-focused thinking: Identifying decision opportunities and creating alternatives. European journal of operational research, 92(3): 537-549.
• KHALDI R, CHIHEB R, EL AFIA A, AKAABOUNE A & FAIZI R. 2017. Prediction of supplier performance: a novel DEA-ANFIS based approach. Paper presented at the Proceedings of the 2nd international Conference on Big Data, Cloud and Applications.
• KUMAR A, OZDAMAR L & PENG NGC. 2005. Procurement performance measurement system in the health care industry. International journal of health care quality assurance, 18(2): 152-166. doi: 10.1108/09526860510588179
  » https://doi.org/10.1108/09526860510588179
• KURIEN GP & QURESHI MN. 2011. Study of performance measurement practices in supply chain management. International Journal of Business, Management and Social Sciences, 2(4): 19-34.
• LACERDA RTO, ENSSLIN L & ENSSLIN SR. 2011. A performance measurement framework in portfolio management: A constructivist case. Management Decision, 49(4): 648-668.
• LACERDA RTO, ENSSLIN L, ENSSLIN SR & DUTRA A. 2014. A constructivist approach to manage business process as a dynamic capability. Knowledge and Process Management, 21(1): 54-66.
• LEGA F, MARSILIO M & VILLA S. 2013. An evaluation framework for measuring supply chain performance in the public healthcare sector: evidence from the Italian NHS. Production Planning & Control, 24(10-11), 931-947.
• LOHMAN C, FORTUIN L & WOUTERS M. 2004. Designing a performance measurement system: A case study. European journal of operational research, 156(2): 267-286.
• LONGARAY AA & ENSSLIN L. 2014. Uso da MCDA na identificação e mensuração da performance dos critérios para a certificação dos hospitais de ensino no âmbito do SUS. Production, 24, 41-56. https://doi.org/10.1590/S0103-65132013005000021
  » https://doi.org/10.1590/S0103-65132013005000021
• LONGARAY AA, ENSSLIN L, ENSSLIN S, ALVES G, DUTRA A & MUNHOZ P. 2018. Using MCDA to evaluate the performance of the logistics process in public hospitals: the case of a Brazilian teaching hospital. International transactions in operational Research, 25(1): 133-156. https://doi.org/10.1111/itor.12387
  » https://doi.org/10.1111/itor.12387
• LONGARAY AA, ENSSLIN L, ENSSLIN SR & DA ROSA IO. 2015. Assessment of a Brazilian public hospital’s performance for management purposes: A soft operations research case in action. Operations Research for Health Care, 5: 28-48. https://doi.org/10.1016/j.orhc.2015.05.001
  » https://doi.org/10.1016/j.orhc.2015.05.001
• LONGARAY AA, ENSSLIN L, DUTRA A, ENSSLIN S, BRASIL R & MUNHOZ P. 2019. Using MCDA-C to assess the organizational performance of industries operating at Brazilian maritime port terminals, Operations Research Perspectives, (6), 100109. https://doi.org/10.1016/j.orp.2019.100109
  » https://doi.org/10.1016/j.orp.2019.100109
• LONGARAY AA, MARUBE N, ENSSLIN L, DUTRA A & ENSSLIN S. 2023. Performance evaluation of public healthcare supply chain management: a critical literature review. International Journal of Healthcare Technology and Management, 20(2): 144-170. http://dx.doi.org/10.1504/IJHTM.2023.131518
  » http://dx.doi.org/10.1504/IJHTM.2023.131518
• LOPEZ DS, GARSHASBI M, KABIR G, BARI ABMM & ALI SM. 2022. Evaluating interaction between internal hospital supply chain performance indicators: a rough-DEMATEL-based approach. International Journal of Productivity and Performance Management, 71(6): 2087-2113.
• MALIK M, ABDALLAH S & HUSSAIN M. 2016. Assessing supplier environmental performance: applying analytical hierarchical process in the United Arab Emirates healthcare chain. Renewable and Sustainable Energy Reviews, 55: 1313-1321.
• MARTINS P, PINHEIRO DE LIMA E, BORTOLUZZI SC & GOUVEA DA COSTA SE. 2023. Performance assessment of the operations strategy of credit unions. International Journal of Productivity and Performance Management, 72(1): 109-136.
• MCKONE-SWEET KE, HAMILTON P & WILLIS SB. 2005. The ailing healthcare supply chain: a prescription for change. Journal of Supply Chain Management, 41(1): 4-17.
• MELNYK SA, BITITCI U, PLATTS K, TOBIAS J & ANDERSEN B. 2014. Is performance measurement and management fit for the future? Management accounting research, 25(2): 173-186.
• MICHELI P & MARI L. 2014. The theory and practice of performance measurement. Management accounting research, 25(2), 147-156.
• MIETHKE-MORAIS A, CASSENOTE A, PIVA H, TOKUNAGA E, COBELLO V, GONÇALVES FAR & HADDAD L. 2021. COVID-19-related hospital cost-outcome analysis: The impact of clinical and demographic factors. Brazilian Journal of Infectious Diseases, 25.
• MIGIRO S & AMBE I. 2008. Evaluation of the implementation of public sector supply chain management and challenges: A case study of the central district municipality, North west province, South Africa. African Journal of Business Management, 2(12): 230.
• MINAHAN T. 2007. Can supply management technology be the antidote to the healthcare crisis? Health management technology, 28(9): 52, 54-55.
• MITROPOULOS P, TALIAS MA & MITROPOULOS I. 2015. Combining stochastic DEA with Bayesian analysis to obtain statistical properties of the efficiency scores: An application to Greek public hospitals. European journal of operational research, 243(1): 302-311.
• NEELY A, GREGORY M & PLATTS K. 1995. Performance measurement system design: a literature review and research agenda. International journal of operations & production management, 15(4): 80-116.
• NEIL R. 2004. The ol'switcheroo. Using knowledge from other industries. Materials management in health care, 13(5): 14-18.
• OECD. 2017. Health at a Glance 2017: OECD Indicators. Paris: OECD Publishing.
• OZCAN T & TUYSUZ F. 2016. Modified grey relational analysis integrated with grey dematel approach for the performance evaluation of retail stores. International Journal of Information Technology & Decision Making, 15(02): 353-386.
• PASCHOAL MLH & CASTILHO V. 2010. Implementação do sistema de gestão de materiais informatizado do Hospital Universitário da Universidade de São Paulo. Revista da Escola de Enfermagem da USP, 44, 984-988.
• ROBERTS FS. 1985. Measurement theory. Cambridge: Cambridge University Press.
• RODRIGUES A, FERNANDES M, RODRIGUES M, BORTOLUZZI S, DA COSTA SG & DE LIMA EP. 2018. Developing criteria for performance assessment in municipal solid waste management. Journal of Cleaner Production, 186: 748-757.
• ROLIM ENSSLIN S, ENSSLIN L, BACK F & TADEU DE OLIVEIRA LACERDA R. 2013. Improved decision aiding in human resource management: a case using constructivist multi-criteria decision aiding. International Journal of Productivity and Performance Management, 62(7), 735-757.
• ROY B. 1985. Méthodologie multicritère d'aide à la décision, Paris: Economica.
• ROY B. 1993. Decision science or decision-aid science? European journal of operational research, 66(2): 184-203.
• ROY B. 1994. On operational research and decision aid. European journal of operational research, 73(1): 23-26.
• RYAN JK. 2005. Systems Engineering: Opportunities for Health Care. In P.P.C. Reid, W. Dale, Grossman, Jerome H., Fanjiang Gary (Ed.), Building a Better Delivery System: A New Engineering/Health Care Partnership (pp. 276). Washington (DC): National Academies Press (US).
• SAATY TL. 1980. The analytic process: planning, priority setting, resources allocation. New York: McGraw.
• SCHLICKMANN A & BORTOLUZZI SC. 2023. Environmental Education Performance Evaluation in a Higher Education Institution. Systemic Practice and Action Research, 1-31.
• STEVENS SS. 1946. On the theory of scales of measurement. Science, New Series, Vol. 103, No. 2684 (Jun. 7, 1946), pp. 677-680. https://www.jstor.org/stable/1671815
  » https://www.jstor.org/stable/1671815
• SUPEEKIT T, SOMBOONWIWAT T & KRITCHANCHAI D. 2016. DEMATEL-modified ANP to evaluate internal hospital supply chain performance. Computers & Industrial Engineering, 102: 318-330.
• TALEB M, KHALID R, RAMLI R & NAWAWI MKM. 2023. An integrated approach of discrete event simulation and a non-radial super efficiency data envelopment analysis for performance evaluation of an emergency department. Expert Systems with Applications, 220, 119653.
• TASCA JE, ENSSLIN L & ENSSLIN SR. 2012. A avaliação de programas de capacitação: um estudo de caso na administração pública. Revista de Administração Pública-RAP, 46(3).
• TASCA JE, ENSSLIN L, ENSSLIN SR & ALVES MBM. 2010. An approach for selecting a theoretical framework for the evaluation of training programs. Journal of European industrial training, 34(7): 631-655.
• TEGARDEN DP & SHEETZ SD. 2003. Group cognitive mapping: a methodology and system for capturing and evaluating managerial and organizational cognition. Omega, 31(2): 113-125.
• THIEL GG, ENSSLIN SR, & ENSSLIN L. 2017. Street lighting management and performance evaluation: opportunities and challenges. Lex Localis, 15(2): 303.
• TRIDAPALLI JP, FERNANDES E & MACHADO WV. 2011. Gestão da cadeia de suprimento do setor público: uma alternativa para controle de gastos correntes no Brasil. Revista de Administração Pública, 45: 401-433.
• WANG Q & SHANG J. 2023. Analysis of the quality improvement path of supply chain management under the background of Industry 4.0. International Journal of Technology Management, 91(1-2), 1-18.
• WHO. 2010. The world health report: health systems financing: the path to universal coverage. Switzerland: WHO Press
• YAZDANI M, TORKAYESH AE & CHATTERJEE P. 2020. An integrated decision-making model for supplier evaluation in public healthcare system: the case study of a Spanish hospital. Journal of Enterprise Information Management, 33(5): 965-989.
• YILMAZ MK, KUSAKCI AO, TATOGLU E, ICTEN O & YETGIN F. 2019. Performance evaluation of real estate investment trusts using a hybridized interval type-2 fuzzy AHP-DEA approach: The case of Borsa Istanbul. International Journal of Information Technology & Decision Making, 18(06): 1785-1820.
• ZAMCOPÉ FC, ENSSLIN L, ENSSLIN SR & DUTRA A. 2010. Modelo para avaliar o desempenho de operadores logísticos: um estudo de caso na indústria têxtil. Gestão & Produção, 17(4): 693-705.
• ZHOU T, WU W, PENG L, ZHANG M, LI Z, XIONG Y & BAI Y. 2022. Evaluation of urban bus service reliability on variable time horizons using a hybrid deep learning method. Reliability Engineering & System Safety, 217: 108090.

Publication Dates

• Publication in this collection
  24 June 2024
• Date of issue
  2024

History

• Received
  22 June 2023
• Accepted
  16 Nov 2023