Performance evaluation of an emergency department in Rio de Janeiro: a hybrid approach using Discrete Events Simulation and Data Envelopment Analysis

Avaliação de desempenho do atendimento em uma emergência hospitalar do Rio de Janeiro: uma abordagem híbrida por meio de Simulação de Eventos Discretos e Análise Envoltória de Dados

Abstracts

Abstract

The efficiency and quality of the emergency department are paramount to ensure that patients receive immediate and appropriate care. Issues such as lengthy waiting times, critical resource management and allocation, and patient scheduling are linked to increased morbidity and mortality, particularly among the elderly and vulnerable populations. This study aims to assess the performance of an emergency department hospital in Rio de Janeiro based on the analysis of resource utilization and queue performance. The methodology encompassed the development of the emergency macro-process, a preliminary statistical analysis of the collected data, and discrete event simulation under different demand conditions. The study found that the average length of stay in the emergency department was 58.12 minutes, potentially increasing to 104.58 minutes under a 15% demand stress. Improvement scenarios were tested, and their efficiencies were measured using data envelopment analysis in an output-oriented and constant return to scale model. The sensitivity analysis revealed that the proposed performance enhancements could make the hospital more responsive to demand peaks and emergencies, ensuring greater resilience and better resource utilization under adverse conditions.

Keywords:
Discrete Events Simulation; DES; Data Envelopment Analysis; DEA; Emergency

Resumo

A eficiência e a qualidade do setor de emergência são essenciais para garantir sua segurança e integridade dos pacientes. Problemas como espera prolongada, gestão e alocação de recursos críticos estão associados ao aumento da morbidade e mortalidade dos pacientes, especialmente entre a população em situações de risco. Este trabalho avalia o desempenho de uma emergência hospitalar localizado no Rio de Janeiro/RJ, baseando-se na utilização dos seus recursos e desempenho de filas. A metodologia inclui o desenvolvimento do macroprocesso da emergência, análise estatística preliminar dos dados e modelo de simulação de eventos discretos sobre diferentes condições de demandas. O tempo médio de permanência na emergência é de 58,12 minutos nos meses de maior demanda e passaria para 104,58 minutos, diante de um estresse de 15%. Diferentes cenários de melhorias foram testados e suas eficiências foram mensuradas por meio de análise envoltória de dados, num modelo que considera orientação ao output e retorno de escala constantes. A análise de sensibilidade realizada em um dos cenários eficientes revelou que o desempenho proposto pode tornar o hospital mais responsivo a picos de demanda e situações emergenciais. Isso garante ao hospital maior resiliência e um melhor aproveitamento dos recursos em condições adversas.

Palavras-chave:
Simulação de Eventos Discretos; SED; Análise Envoltória de Dados; DEA; Emergências

1 Introduction

The emergency department in a hospital is the designated area for providing immediate medical attention to patients facing imminent risk of death or experiencing intense suffering (Aringhieri et al., 2017). Patients seeking care in this sector typically encompass those involved in accidents, suffering from sudden illnesses, or encountering medical emergencies that need urgent intervention. This rapid access to diagnosis and treatment aims to minimize the adverse effects and symptoms (Jarvis, 2016).

In this sector, patients receive care from multidisciplinary teams comprising doctors, nurses, nursing technicians, and other healthcare professionals, prepared with specialized training to address a wide range of urgent situations (Doudareva & Carter, 2022). Challenges such as extended waiting times, critical resource management, and patient scheduling are associated with heightened morbidity and mortality rates, especially among the elderly and vulnerable populations (Boyle et al., 2023).

In this context, evaluating the performance of emergency departments in private hospitals is crucial to ensure quality, efficiency, and safety in patient care (Pham et al., 2023). The application of Discrete Events Simulation (DES) in assessing the performance of these departments facilitates operational enhancements. This methodology focuses on optimizing resource allocation and reducing waiting times, directly correlated with patient satisfaction levels (Vanbrabant et al., 2019; Ortíz-Barrios & Alfaro-Saíz, 2020).

Fone et al. (2003) highlights the existence of various techniques, such as optimization modeling, applicable to healthcare systems beyond simulation. Moreover, alternatives to DES, including system dynamics (SD) and agent-based modeling (ABM), also play important roles. As Günal & Pidd (2007) point out, most DES efforts in hospitals have been directed towards specific units like emergency departments (Marchesi et al., 2020), outpatient services (Peres et al., 2019), operating rooms, wards, and intensive care units. Ouda et al. (2023) explain that this emphasis on individual departments is expected due to the difficulties in representing an entire hospital environment within a single simulation model. This approach also serves to simplify reality, allowing for a focus on the critical aspects most relevant to the process under investigation. The level of detail chosen for these models is paramount. A well-specified model contributes to a more realistic representation and fosters greater trust among stakeholders (Vázquez-Serrano et al., 2021).

Similarly, Liu et al. (2020) report that the application of DES in hospitals primarily aims to achieve several key outcomes: (i) reducing patient length of stay and improving overall operational efficiency; (ii) optimizing costs and realizing financial savings through enhanced resource allocation and service scheduling; (iii) enhancing the quality-of-service delivery; and (iv) ensuring the health, integrity, and safety of patients. In summary, the authors underscore the pivotal role of DES in addressing the operational complexities encountered by emergency departments, notably in refining workflows, elevating the quality of care, and minimizing patient waiting times.

The primary objective of this study is to evaluate the system performance within the emergency department of a private hospital in Rio de Janeiro. Additionally, the study aims to assess new resource allocation scenarios using Discrete Events Simulation and Data Envelopment Analysis (DEA). Section 2 of this article discusses the theoretical framework underpinning this research, detailing the utilization of DES, DEA, and hybrid approaches combining these techniques. Section 3 outlines the methodological procedures employed in this study, encompassing data collection and analysis methods. Section 4 delves into preliminary statistical analysis and system modeling. Section 5 presents the results of the current performance and sensitivity analysis. Section 6 explores and evaluates potential new resource allocations within the hospital's emergency department. Finally, Section 7 discusses the conclusions and final considerations drawn from the study.

2 Theoretical frameworks

2.1 Discrete Event Simulation

DES is a modeling technique employed to understand the behavior and performance of complex dynamical systems. It has widespread applications in engineering, computer science, logistics, economics, and numerous others (Prado & Yamaguchi, 2019). According to Chwif & Medina (2014), systems modeled using DES are distinguished by events that happen at specific, discrete points in time, thereby influencing the system's state. Each event has the potential to trigger subsequent events, with the system's overall behavior being modeled through the interactions among these events (Brailsford et al., 2019).

Typically, a DES model consists of individual elements or users interacting with the system, called entities. These entities have attributes and engage with the system through discrete and sequential events (Mohiuddin et al., 2017). According to Nance (1981), a simulation can be conceptualized as a function f that maps out an output variable Y_i from an input variable X_i​ plus a random error factor ε_i. The main objective is to understand how the input data influences the output variables, acknowledging that these effects are stochastic and randomly distributed according to a known theoretical probability distribution (Figure 1).

Figure 1
Representation of the system at a specific point in time. Source: Adapted from Nance (1981).

Roberts & Pegden (2017) elucidate that users entering the DES system are processed according to predefined rules, which heavily rely on stochastic variables to represent the randomness inherent in events such as customer arrivals and service times. Typically, a random variable is employed to model the arrival process of these entities, characterized by the temporal distribution between consecutive arrivals. Similarly, another stochastic variable defines the service period, specifying the duration each entity requires from the assigned resource to receive care (Zeigler & Muzy, 2017).

Based on queuing theory, the primary output results (Y_i) obtained through DES encompass average waiting time, average length of stay in the system, average number of customers in the queue, average number of customers within the system, and average resource utilization. These metrics can be tailored to align with the specific simulation requirements and capabilities (Nance, 1981; Macal, 2016; Zeigler & Muzy, 2017).

These metrics are pivotal in evaluating processes across diverse contexts, empowering managers and developers to fine-tune systems for enhanced efficiency, minimized operational costs, and heightened end-user satisfaction. By evaluating various scenarios, testing strategies and policies, optimizing processes, and analyzing the behavior of complex systems, it becomes possible to predict performance and understand system dynamics without implementing real-world changes. This capability provides a powerful tool for anticipating outcomes and making informed decisions, ensuring benefits before implementation.

2.2 Data Envelopment Analysis

Based on linear programming, DEA is a non-parametric technique that estimates a piecewise linear efficiency frontier from a set of similar organizations, distinguished by their levels of input and output combinations (Charnes et al., 1978; Cook & Seiford, 2009). The primary advantage of the DEA approach is that it does not require an explicit functional form for the data (Liu et al., 2013; Cook et al., 2014).

Classical literature recognizes two models used in the evaluation of decision-making units (DMUs). The first model, introduced by Charnes et al. (1978), assumes constant returns to scale (CRS). This model measures productive efficiency by assuming that any input variation results in a proportional output variation. The second model, developed by Banker et al. (1984), presupposes variable returns to scale (VRS). This model assesses technical efficiency by considering that DMUs can adopt technologies with different scale yields: constant, increasing, or decreasing.

These models support two types of orientations: input-oriented and output-oriented. In the input-oriented approach, efficiency scores are obtained by maximizing the reduction of resources and inputs while keeping the production level constant. Conversely, the output-oriented approach aims to maximize outputs given a fixed amount of inputs (Cooper et al., 2007). The set of expressions in Model 1 defines the linear formulation used to calculate the productive efficiency of each organization under analysis, with a focus on minimizing productive resources.

Min	${\theta }_0-\epsilon \left(\sum _{j=1}^m{s}_i^{-}+\sum _{r=1}^s{s}_r^{+}\right)$		(1)
Subject to:
	$\sum _{j=1}^n{x}_{ij}\bullet {\lambda }_j+s_i^{-}=x_{i0}{\bullet \theta }_0$	i = 1, ..., m;
	$\sum _{j=1}^n{y}_{rj}\bullet {\lambda }_j-s_r^{+}=y_{r0}$	r = 1, ..., s;
	${\lambda }_j\ge 0;s_i^{-}\ge 0;s_r^{+}\ge 0;{\theta }_0\ge 0$	$\forall i,r,j$

The coefficients y_rj and x_ij are, respectively, the known outputs and inputs of the j DMUs under analysis (j = 1, ..., n), constant values obtained in previous surveys on the i allocated inputs (i = 1, ..., m), resulting in a set r of outputs (r = 1, ..., s). The total or productive efficiency score of the DMU 0 under analysis, output oriented, is represented by θ₀; x_i0 and y_r0 are the quantities of the i^th input and the r^th output, respectively, presented by the DMU 0 analyzed; s_i^- and s_r⁺ are the observed clearances in the inputs and outputs; and λ_j represents the contribution of the DMU j to project 0 at the efficient frontier (Cooper et al., 2007; Cook & Seiford, 2009; Cook et al., 2014).

The non-Archimedean term, ε > 0, is a positive element smaller than any real number. The efficiency measure θ₀ must be multiplied by all inputs to position DMU 0 on the efficient frontier by reducing input values. The first set of constraints ensures that the reduction in each input does not exceed the threshold defined by the efficient DMUs. The second set of constraints ensures that the input reduction does not affect the current output levels of DMU 0 (Cook et al., 2014).

The expression defined by (2) presents the linear model used to calculate the output-oriented efficiency of each organization under analysis.

Max	${\eta }_0-\epsilon \left(\sum _{j=1}^m{s}_i^{-}+\sum _{r=1}^s{s}_r^{+}\right)$		(2)
Subject to:
	$\sum _{j=1}^n{y}_{rj}\bullet {\lambda }_j-s_r^{+}=y_{r0}{\bullet \eta }_0$	r = 1, ..., s;
	$\sum _{j=1}^n{x}_{ij}\bullet {\lambda }_j+s_i^{-}=x_{i0}$	i = 1, ..., m;
	${\lambda }_j\ge 0;s_i^{-}\ge 0;s_r^{+}\ge 0;{\theta }_0\ge 0$	$\forall i,r,j$

The η₀ score represents the inverse of the productive efficiency of the DMU 0 under analysis. Therefore, the value of η₀ is greater than or equal to one and must be multiplied by all outputs to place DMU 0 on the efficient frontier, by increasing output values. The first set of constraints ensures that this increase in each output does not exceed the threshold defined by the efficient DMUs. The second set of constraints ensures that the increase in outputs does not alter the current level of inputs in DMU 0 (Cook et al., 2014).

If the convexity constraint, $\sum _{j=1}^n{\lambda }_j=1$, is added to Models 1 and 2, these models then consider variable returns to scale. This constraint ensures that both θ₀ and 1/η₀ ​ represents the technical efficiency of DMU 0 under analysis. It ensures that inefficient DMUs are compared only with DMUs of the same size or activity level, allowing for a more accurate assessment of efficiency (Banker et al., 1984).

2.3 Hybrid approaches between DES and DEA

DES is a valuable tool for testing multiple scenarios in a controlled virtual environment, eliminating the risk and cost associated with real-world experimentation. DES excels at modeling complex interactive systems and their inherent uncertainties using random variables to represent various system components. By simulating different scenarios, decision-makers can observe how changes in input variables – such as customer arrival rates, service times, resource configurations, and operation policies – impact the outcomes of the system (Chwif & Medina, 2014).

A hospital can leverage DES to assess the impact of different staffing allocations within its emergency rooms, as shown by Weng et al. (2011), Keshtkar et al. (2020), and Tavakoli et al. (2022). This approach can make clear how adjusting the numbers of physicians and nurses influences patient waiting times and resources utilization, ultimately facilitating conditions to select the arrange that optimize both patient satisfaction and system efficiency.

Additionally, simulation enables the evaluation of extreme or rare scenarios, which, although unlikely, can have significant consequences if they occur (Zeigler & Muzy, 2017). These “stress test” scenarios are crucial in financial engineering and disaster management, where understanding behavior under extreme conditions can help prevent catastrophic failures (Sargent et al., 2016).

In this context, the design methodology for simulation projects proposed by Egilmez et al. (2018) needs economic and financial evaluations to assess the viability of proposed solutions across various scenarios. In other words, more is needed to guarantee superior performance regarding queue indicators. Implementing a specific scenario within the actual system must also be substantiated by its economic feasibility.

Although the presented argument by the authors is coherent and necessary for real-world implementation, Hosseini et al. (2019) and Norouzian-Maleki et al. (2022) identify circumstances that can render this approach impractical. These include the absence of usable records and financial information, as well as the proliferation of proposed scenarios, which can make financial projections for an excessively large number of scenarios unfeasible.

Different authors argue that scenario performances can be compared in terms of productive efficiency rather than relying solely on financial aspects (Dev et al., 2014; Marlin & Sohn, 2016; Keshtkar et al., 2020; Taleb et al., 2023). This approach involves using DEA to compare the results of the proposed scenarios, identifying which one provides an optimal combination of resource consumption and output generation, thereby leading to productive efficiency in the evaluated systems. Some approaches that integrate DEA and DES are listed in Table 1.

Table 1
Hybrid assessment models between DES and DEA.

Authors	Area/Objective	Simulation	DEA
Azadeh et al. (2011)	Manufacture of plastic artifacts - Evaluate the performance of different layouts in a new manufacturing plant arrangement	Simulator: Visual SLAM Specifics: Simulator used to model the injection manufacturing process	Model: Non-radial Fuzzy-DEA Orientation: Output Inputs: - Outputs: average machine utilization (desirable output), average wait time (undesirable output) and average system time (undesirable output)
Weng et al. (2011)	Emergency Sector - Assess potential bottlenecks, maximize production flows, and identify solutions to reduce patient time in the emergency department (ED) while increasing patient satisfaction	Simulator: Arena Specificities: The model simulated and followed patients in the emergency department throughout their stay in the ED, from presentation to discharge	Model: DEA-VRS Orientation: Input Inputs: doctors, nurses, beds Outputs: number of patients seen (desired output)
van den Bergh et al. (2013)	Aircraft Line Maintenance - The work focuses on the scheduling of allocated human resources to minimize cost and maximize maintenance service and attendance at an airport	Simulator: Microsoft Visual Studio (C++) Specifics: Real-time flight arrival data for the winter season was used to identify the appropriate distributions	Model No.: CRS Orientation: Input Inputs: Labor costs and allocated labor per shift Outputs: success rate (desired), average number of solutions served (desired) and average delay time (unwanted)
Azadeh et al. (2014)	Injection Molding Production Line - Reduce bottlenecks in the production line by injection molding	Simulator: Visual SLAM Specifics: The performance of different priority rules for task dispatch was evaluated, comparing them by queue performance indicators	Model: Stochastic-DEA Orientation: Output Inputs:- Outputs: queue size (unwanted), average machine utilization (desired) and cycle time (unwanted)
Dev et al. (2014)	Supply Chain - Analyze the efficiency of the total supply chain in the context of average supply rate performance, considering the possibility of time delays due to changes in lead time and inventory review period	Simulator: Arena Specifics: Uses to obtain supply time in a model that includes multiple levels and links of a hypothetical supply chain	Model No.: VRS Orientation: Input Inputs: delay due to lead time and inventory review period Outputs: Average Supply Rate
Marlin & Sohn (2016)	National education system - Determine the critical relationships, potential futures, and unforeseen consequences of potential policies regarding primary and secondary education in Afghanistan	Simulator: NOLH Specifics: In the simulation model, entities such as students and teachers are created, which then transit through some part of the system and then are terminated, in which attributes of the entities contain decision rules (containing records about demographic and socioeconomic information), making them analogous to agents	Model No.: CRS Orientation: Input Inputs: student problem-solving ability, teacher-solving ability, teacher training capacity, community-based education capacity; Student Demand and Required Funds Outputs: dropout, literate, graduates, parity between provinces and quality of education
Hosseini et al. (2019)	Refrigerator injection process of a home appliance company - Improve the resiliency performance of the production line	Simulator: Visual SLAM Specifics: 24 different scenarios were generated to be evaluated in the simulation in terms of queue time and utilization rate	Model No.: Fuzzy DEA Orientation: Output Inputs: - Outputs: Wait time (unwanted), average utilization of flexible devices (desired), average utilization of other devices (desired), and average fault tolerance time (unwanted)
Keshtkar et al. (2020)	Hospitalization of patients in emergencies - Investigate the impact of the inpatient problem on patients' ability to access other units in the hospital	Simulator: AnyLogic Specificities: the integration between DES and DS makes it possible to monitor the patient flow at the micro and macro level	Model No.: VRS Orientation: Output Inputs: number of porters, number of stretchers for transport and number of consultants Outputs: length of stay (unwanted) and patients seen (desired)
Navazi et al. (2019)	Refrigerator injection process of a home appliance company - Evaluate the layout of a high-volume production line of single-queue installations and multiple products that requires many downtimes per setup	Simulator: Visual SLAM Specifics: To improve the facility layout of the injection modelling process, this study investigates the performance of each layout against the three factors of Resilience Engineering, which includes flexibility, teamwork, and fault tolerance	Model: Fuzzy DEA-VRS Orientation: Output Inputs: - Outputs: Average Wait Time (Unwanted), Average Device Utilization (Desired), Average Flexible Device Utilization (Desired), and Average Fault Tolerance Time (Unwanted)
Monazzam et al. (2022)	Manufacturing - Increase efficiency and determine methods to evaluate and optimize the performance of different parts of production systems	Simulator: Enterprise Dynamics Specificities: It models different sectors of the production system and integrates them with results on productivity	Model: Ratio Efficiency Dominant Orientation: - Inputs: production planning, non-mechanized inventories, and assembly shop cycle time Outputs: total production profit, production volume, average productivity of production halls, number of trucks transporting the body and number of semi-finished products during the process
Tavakoli et al. (2022)	Health (COVID-19) - Propose a decision support system for hospital managers, who are willing to be more efficient and who rely on data to support decision-making	Simulator: Arena Specifics: Simulate the process of patient flow and then predict the future entry of patients into a hospital as a case study and suggest some policies based on different likely scenarios	Model No.: CRS Orientation: Input Inputs: number of doctors on the COVID-19 emergency hotline, number of nurses on the COVID-19 emergency hotline, number of doctors in the ICU, number of nurses in the ICU, number of doctors in the ICU, number of nurses in the ICU, number of radiologists in a CT scan unit, Number of service providers in the laboratory, number of beds in the ICU, number of beds in the ICU and number of beds in the special COVID-19 emergency line Outputs: total time, average patient waiting time, rate of patients discharged, and cost of inputs
Taleb et al. (2023)	Emergency Sector - Investigate the impact of the allocation of human resources (receptionists, nurses, and doctors) in terms of the queue performance indicators	Simulator: Arena Specificities: Performs the simulation from arrival at the emergency room until discharge	Model No.: Super Efficiency SBM-VRS Guideline: Not applicable Inputs: number of receptionists, number of nurses and number of doctors Outputs: average length of stay in the emergency room (unwanted) and average length of stay in queues (unwanted)

Therefore, DEA is used to evaluate the scenarios generated by the simulation models. It is crucial to emphasize that the indicators produced by the simulation models are incorporated as outcomes of each evaluated scenario and modeled as output variables within the DEA framework. Indicators such as cycle time, length of stay, and queue waiting time are considered undesirable outputs (Azadeh et al., 2011, 2014; van den Bergh et al., 2013; Keshtkar et al., 2020; Navazi et al., 2019; Taleb et al., 2023), as they signify a lower level of service from the customer's perspective. Conversely, the resource utilization rate, a standard output in traditional simulators, is considered a desirable process outcome, reflecting efficient allocation of productive resources (Azadeh et al., 2011; Hosseini et al., 2019; Navazi et al., 2019).

The hybrid and multimethodological approach can integrate various techniques with the use of DES and DEA. The most recurrent techniques include the design of experiments (DOE) (Monazzam et al., 2022; Hosseini et al., 2019; Keshtkar et al., 2020; Marlin & Sohn, 2016; Tavakoli et al., 2022) and fuzzy logic (Azadeh et al., 2011, 2014; Hosseini et al., 2019; Navazi et al., 2019).

In essence, the Design of Experiments (DOE) is a systematic and efficient method that investigates the relationship between multiple input variables and key output variables through a structured data collection and discovery approach. Often, multiple proposed scenarios are generated from the DOE technique, subsequently these scenarios are tested and evaluated using DEA (Monazzam et al., 2022; Hosseini et al., 2019; Keshtkar et al., 2020; Marlin & Sohn, 2016; Tavakoli et al., 2022).

Fuzzy logic allows the modeling of uncertainties associated with input and output data in the DEA model, which is particularly useful in environments where measurements are imprecise or subject to fluctuation, enabling DEA models to account for these uncertainties when calculating efficiency. This is especially relevant for stochastic indicators, such as those obtained through DES. Therefore, combining DEA with fuzzy logic, using results generated by DES, can result in models capable of handling ambiguities, thereby providing more robust analyses (Azadeh et al., 2011, 2014; Hosseini et al., 2019; Navazi et al., 2019).

3 Materials and methods

3.1 Data collection and processing

The study was retrospective, and all data obtained were de-identified following the Brazilian General Data Protection Law (LGPD) (Brasil, 2018), primarily containing records of time and process types (non-sensitive information). The data were collected from a private hospital in Rio de Janeiro, Brazil, covering a period from February 1, 2022, to September 30, 2023. This dataset encompasses 19 months (excluding August 2023) or 576 days, including 130,525 attendances. Table 2 details the data collected from the hospital and indicates the percentage of missing data (identified as NA – Not Applicable or blank values) which stands for the relative frequency of unreported values for each step.

Table 2
List of information available in the hospital’s database.

Variable	Description	% Missing Data
a	Timestamp of the withdrawal of the password at the totem	16.86%
b	Timestamp of service at the front desk	0.00%
c	Timestamp of service in triage	52.36
d	Registration of the date and time of the appointment at the doctor's appointment	16.36%
e	Duration of care at triage (in minutes)	55.07%
f	Duration of consultation with a doctor (in minutes)	16.46%

The only valid records available for the time of care throughout the process are variables [e] and [f], which pertain to the duration of care in triage and the duration of the medical consultation, respectively, measured in minutes. Notably, the service time at the reception was not recorded even once. However, there is a record of the time difference between the service ticket's withdrawal from the totem (variable [a]) and its entry at the hospital reception (variable [b]). The difference between variables [a] and [b] can be interpreted as the length of stay for each patient at the reception. Furthermore, the date and time stamp associated with the totem in variable [a] records successive arrivals at the hospital reception. Due to the lack of precision in the remaining information or because it is beyond the scope of this study, the other stages of emergency care will be excluded from the analysis.

3.2 Research method

Egilmez et al. (2018) and Taleb et al. (2023) propose a methodology for studies and experiments involving DES, which will be adopted in the present study: (i) Definition of the problem to be analysed; (ii) System analysis; (iii) Collection, analysis and preparation of input data; (iv) Implementation of the simulation model; (v) Verification and validation of the simulation model; (vi) Analysis of the system performance; (vii) Proposal of new scenarios and improvements; and (viii) Evaluation and analysis of the scenarios efficiency.

Clarifying the parallels suggested by Egilmez et al. (2018) in their methodological approach, the present study analyzes resources and evaluates queue performance indicators within the emergency department of a hospital in Rio de Janeiro. The system analysis was conducted to develop a conceptual model of the process. This step entailed describing the macro-process using the Business Process Model and Notation (BPMN) principles and employing the Bizagi Modeler as the computational tool (Chinosi & Trombetta, 2012; Giacomo et al., 2015).

To validate the conceptual model, the Disco software, a Process Mining tool, was employed to sequence the business process of the analyzed emergency department. This validation involved analyzing event data recorded within the hospital's information systems. Aligning with the approach of Egilmez et al. (2018), using Process Mining is supported due to its ability to leverage event logs and transactional data to gain insights into the operation of business processes, as elucidated by De Roock & Martin (2022).

The analysis and preparation of the data initially involved the removal of extreme outliers from the sample. According to Tukey (1977), this was done using the interquartile method, which helps identify and exclude extreme values that could skew the results. The values beyond the interval in Expression 3 are considered outliers, in which Q1 and Q3 are quartiles 1 and 3 of the analyzed base and k is a non-negative constant. The suggestion k = 3 was used to consider only the removal of extreme outliers.

$\left[Q1-k\left(Q3-Q1\right),Q3+k\left(Q3-Q1\right)\right]$

(3)

We examined monthly, daily, shift-based, and hourly variations in patient arrivals and the service delivery process across different activities to identify seasonal patterns and effects on demand and service provision. Moreover, we used the calculation of confidence intervals for means (Student, 1908), the H test (Kruskal & Wallis, 1952), and the U test (Mann & Whitney, 1947; Hart, 2001) to measure the statistical differences between groups.

In the absence of information on the service time duration at the reception, it was necessary to estimate this period for implementation in the simulation of the care process within the hospital emergency department. Considering that the queue formation a at the reception adheres to the premises of a Markovian process, it is established that the times between successive patient arrivals follow an exponential distribution with a parameter of 1/λ ​. The reception has a known number of servers, denoted as c, each providing service to customers according to an exponential distribution with a parameter of 1/μ​. Considering the Markovian nature of the queue, the average length of stay for a patient (W) can be calculated using Expression 4, while the probability of the system being empty (P₀) can be figured out using Equation 5 (Fogliatti & Matos, 2007).

$W=\frac{1}{\mu }+\frac{{\left(\frac{\lambda }{\mu }\right)}^c\mu }{\left(c-1\right)!{\left(c\mu -\lambda \right)}^2}{P}_0$	(4)
$P_0={\left(\sum _{n=0}^{c-1}\frac{{\left(\frac{\lambda }{\mu }\right)}^n}{n!}+\frac{c{\left(\frac{\lambda }{\mu }\right)}^c}{c!\left(c-\frac{\lambda }{\mu }\right)}\right)}^{-1}$	(5)

The simulation was conducted using Arena software, with the proposed flowchart adhering to the assumptions established within the conceptual model. Data processing was facilitated by the Input Analyzer tool. The process modeling encompassed the inherent variability of hospital emergency room activities and the seasonal factors influencing resource demand and activity levels. A sensitivity analysis was performed, considering demand increases of 5%, 10%, and 15% above the current emergency demand levels.

To find the warm-up period of the simulated model, the graphical method was employed. This approach entails truncation of the simulated period and visual examination of the resulting output time series (Hoad et al., 2008, 2010). The duration of the warm-up period served as the basis for determining the replication duration, which was established at 10 times the warm-up time. This measure ensured the attainment of statistically significant and representative results for the developed models (Currie & Cheng, 2016).

The verification process adhered to the principles delineated by Kleijnen (1995), encompassing: (i) adherence to established good programming practices; (ii) verification of intermediate simulation results through tracking and the application of statistical tests on a per-module basis; and (iii) the execution of final statistical tests. Notably, this latter stage served as a mechanism for statistically validating the proposed model. The Mann-Whitney U test (1947) was employed to compare the simulation time results with those observed in the actual onsite emergency setting.

To propose improvements, various scenarios for resource allocation at reception, triage, and medical consultations were developed. The combination of these allocations resulted in a total of 48 distinct scenarios. Given the absence of cost information for comparison, we employed an approach suggested by Taleb et al. (2023), where DEA is utilized to evaluate the efficiency of resource allocations in each scenario.

In the current efficiency evaluation using DEA, we assumed that the DMUs being compared exhibit constant returns to scale, as it involves the same hospital operating under different resource allocations, as proposed by Charnes et al. (1978). The chosen model is output-oriented, aiming to maximize process outputs while maintaining fixed resources (Cook et al., 2014).

The evaluation includes three inputs: the number of resources allocated at reception (x_1j), triage (x_2j), and medical consultation (x_3j). The only output in the model was the length of stay in the emergency department (y_1j), although the additive inverse was translated from this indicator since it is considered an undesirable result of the hospital care process, as indicated by Seiford & Zhu (2002). Finally, a sensitivity analysis was also carried out on one of the efficient scenarios, considering an increase of 15% and 30% over the hospital's current demand.

4 Modelling and simulation of the hospital emergency process

4.1 System analysis

In the conceptual model outlining the stages of care within this emergency room (Figure 2), as per the hospital's established standard, the patient's journey begins by obtaining a service card or password from a self-service kiosk. This kiosk determines the sequencing and order of patients in the reception queue. Upon reaching the reception desk, patients must provide or update their individual registration details. The triage stage involves a nurse conducting an initial assessment, observing vital signs, documenting primary symptoms, and re-establishing a prioritization criterion for patients. Subsequently, patients are directed to medical care, where proper investigations and diagnosis are undertaken.

Figure 2
Global emergency response process.

Based on data provided by the company's management team, Table 3 illustrates the allocation of resources according to shift and relevant activity. It is important to note that shift 1 (T1) encompasses the hospital's activities from 7:00 a.m. to 7:00 p.m., while shift 2 (T2) encompasses the corresponding activities from 7:00 p.m. to 7:00 a.m.

Table 3
List of information available in the hospital's database.

Station	Function	T1	T2
Totem	---	2	1
Reception	Receptionist	4	3
Triage	Nurse	1	1
Consultation	Doctor	6 (Mon and Tue)	3 (Mon and Tue)
		5 (Wed to Sun)	2 (Wed to Sun)

The actual care process within the analyzed emergency room was conceptualized in contrast to the conceptual model designated by the hospital as the standard care process. Corroborating the work of Garcia et al. (2019), this step aims to verify and compare the process observed through recorded events against the model established by the organization's management. By doing so, inconsistencies between the standard and actual processes can be identified, paving the way for potential process improvements through enhanced understanding, modeling, and redesign of the project.

The process generated from the process mining was facilitated by the Disco tool, which considered the record of each activity performed and the corresponding date/time for each medical record. The resulting flowchart exhibits similarities to the hospital's proposed model, wherein patients progress sequentially through reception, triage, and medical care (Figure 3). However, discrepancies arise in the post-diagnosis stages: the possibility of concurrent laboratory and imaging tests exists, yet there needs to be recorded data on the duration of care in these stages. Notably, there's stringent control over antibiotic administration, but the processes stemming from outpatient care within this stream still need clarification. Additionally, the medical reassessment stage and referrals to specialists, when necessary, are not explicitly depicted.

Figure 3
Global Emergency Response Process (Process Mining).

4.2 Preliminary statistical analysis

4.2.1 Patient arrival

The analysis of the seasonal effect on hospital demand was based on patient arrivals (a, in Table 2) and measured according to month, day of the week, and time of arrival. The monthly analysis calculated the total volume of arrivals per month (Table 4). The months were then grouped into three demand patterns based on the monthly patient volume: group1 encompasses the six months with the lowest number of patients served, while group 3 includes the six months with the highest number of patients served. Group 2 consists of the remaining seven months with intermediate patient volumes.

Table 4
Total arrivals per month.

Month	Total	Group	Month	Total	Group
Feb/22	3,380	Group 1	Dec/22	5,072	Group 1
Mar/22	4,856	Group 1	Jan/23	5,554	Group 2
Apr/22	4,967	Group 1	Feb/23	5,530	Group 2
May/22	6,683	Group 3	Mar/23	6,920	Group 3
Jun/22	6,717	Group 3	Apr/23	6,334	Group 2
Jul/22	5,072	Group 1	May/23	6,729	Group 3
Augo/22	5,280	Group 2	Jun/23	5,901	Group 2
Sep/22	5,026	Group 1	Jul/23	5,517	Group 2
Oct/22	5,705	Group 2	Aug/23	-	-
Nov/22	6,656	Group 3	Sep/23	6,622	Group 3

• We employed the Kruskal-Wallis H test to assess the statistical difference between the three groups, yielding H = 16.025 (p-value < 0.05). Consequently, the results show a rejection of the null hypothesis, suggesting that the medians of groups 1, 2, and 3 are not equal with 95% confidence.

The analysis of arrivals by day of the week considered the total number of appointments per day, categorized from Monday to Sunday. Based on this data, descriptive statistics, the mean number of patients seen per day, and their corresponding confidence intervals were analyzed (Table 5). The average number of patients visiting the hospital on Mondays is notably higher than on other days of the week, with a 95% confidence margin. Conversely, Saturdays and Sundays exhibit the lowest demand levels throughout the week. However, there is no statistically significant difference between the average number of patients on these two days, considering a 5% significance level.

Table 5
Average number of patients per day.

Day week	Interval (α = 5%)	Average
Monday	(229.61; 249.59)	239.60
Tuesday	(205.68; 225.15)	215.41
Wednesday	(194.09; 213.67)	203.88
Thursday	(185.73; 203.39)	194.56
Friday	(170.76; 187.60)	179.18
Saturday	(136.16; 149.82)	142.99
Sunday	(138.49; 152.82)	145.65

Finally, the last analysis pertains to patient arrivals by hour of each day, differentiating between groups 1, 2, and 3. The average customer arrival rate was calculated based on time and day of the week, segmented by demand group (^{Appendix 1} and Figure 4). A certain similarity exists in the fluctuation of patient arrivals throughout the day across all three groups. Arrivals are minimal between midnight and 5:00 a.m. for all groups. Starting at 5:00 a.m., the volume of patients begins to increase, peaking between 10:00 a.m. and 11:00 a.m. From that point onward, arrivals gradually decline until the end of the day.

Figure 4
Average number of patients per hour according to day.

4.2.2 Service time

The initial reception service, patient triage, and medical consultation were considered core stages of care within the emergency process. The timestamps for when a patient retrieves their individual at reception (variable [a] in Table 2) and when their care concludes at the reception (variable [b] in Table 2) provide the data for calculating the time spent at reception. The triage procedure, performed by a nurse upon the patient's arrival at reception and prior to medical care, is recorded as variable [e] in the hospital's database. The final stage analyzed, medical care, is identified as variable [f] in the hospital's database (Table 2).

Given that the number of servers at the hospital's reception and medical consultation varies depending on the shift (Table 2), the average length of stay at reception was calculated for each shift, along with the duration of care at triage and medical consultation. Table 6 presents these results, which are valid with 95% confidence: the average time spent at reception during shift 1 is notably longer than in shift 2; the average duration of triage care is longer in shift 2 compared to shift 1; the duration of the doctor's appointment is longer in shift 1 than in shift 2. These differences can be confirmed using the Mann-Whitney U test.

Table 6
Average service time and U test for difference between shifts.

Shift	Average length of stay at reception	Average service time at triage	Average service time at emergency consultation
T1	13.90 (13.83; 13.97)	3.81 (3.76; 3.85)	11.38 (11.32; 11.44)
T2	10.35 (10.23; 10.47)	4.20 (4.12; 4.29)	10.99 (10.89; 11.10)
Test U	1,016,617,025	12,716,506	575,068,112
P-value	< 0.05	< 0.05	< 0.05

The Input Analyzer tool was used with the triage attendance data to determine the statistical distribution that best fit the collected information. This same procedure was applied to analyze the duration of medical care in both shift 1 and 2. The resulting findings are presented in ^{Appendix 2}, while ^{Appendix 3} displays the histograms of these data alongside their respective theoretical distributions.

The hospital reception lacks a record of its service time. However, if this process adheres to the premises of a Markovian process model, and the average arrival rate (λ), the number of servers in operation (c), and the average customer length of stay (W) are known, the average service rate (μ) at the reception can be estimated for each shift using Equations 4 and 5 (^{Appendix 4}). From this perspective, the reception service time will be considered exponentially distributed, with an average of 11.52 minutes for shift 1 and 10.29 minutes for shift 2.

4.3 Discrete event simulation model

By synthesizing the detailed information thus far, the simulation model was implemented within Arena (Figure 5 and Table 7). The “Create” module was employed in the “Arrival” stage as a Schedule pattern, where the patient flow varied hourly, following a Poisson distribution based on the values presented in ^{Appendix 1}.

Figure 5
Emergency Simulation Flow Chart.

Table 7
Simulation Details in Emergency.

Stage	Guy	Specification	Expression (unit)
Arrival	Create	Schedule	Poisson (Appendix 1) | hour
Queue Reception	Seize	Based on Schedule	T1: 4
			T2: 3
Reception Shift	Decide	2-way by condition	CalHour(TNOW) >= 7 && CalHour(TNOW) <19
T1 Reception	Process	Delay Release	EXPO(11.52) | minute
T2 Reception	Process	Delay Release	EXPO(10.29) | minute
Leave Reception	Leave	Route	Enter Triage
Enter Triage	Enter	-	-
Queue Triage	Seize	Fixed Capacity	1
Shift Triage	Decide	2-way by condition	CalHour(TNOW) >= 7 && CalHour(TNOW) <19
T1 Triage	Process	Delay Release	1.5 + LOGN(1.86, 1.32) | minute
T2 Triage	Process	Delay Release	1.5 + EXPO(1.7) | minute
Leave Reception	Leave	Route	Enter Medical
Enter Medical	Enter	-	-
Queue Medical	Seize	Based on Schedule	T1: 6 (2f e 3f), 5 (4f a dom)
			T2: 3 (2f e 3f), 2 (4f a dom)
Medical Appointment	Decide	2-way by condition	CalHour(TNOW) >= 7 && CalHour(TNOW) <19
Medical Care T1	Process	Delay Release	2.5 + GAMM(5.78, 1.29) | minute
T2 Medical Care	Process	Delay Release	4.5 + EXPO(6.49) | minute
Denouement	Has	-	-

As a non-terminal simulation model, an initial simulation period of 20 weeks (140 days) was considered, with each day comprising 24 hours, reflecting the hospital's actual operational hours. Using group 3 (higher monthly demand) as a reference, five replications were conducted in this configuration, and the mean number of patients seen per day in each simulated week was calculated (Figure 6). This procedure aimed to determine the warm-up period of the system configuration, using the graphical truncation method. Based on the observed graph behavior, the system stabilized after seven weeks of simulation. Therefore, 49 days were designated as the warm-up period. Given that the simulation duration should be at least ten times the warm-up period, a final replication spanning 540 days was adopted for the subsequent models.

Figure 6
Average number of patients per day in group 3 (initial setting).

5 Results

The model validation was conducted by comparing the length of stay in the simulated model with that of the real system. The hospital's database contains a record of the date and time of patient arrivals (variable [c] in Table 2), as well as the time when the patient's consultation concluded (variable [d]).

Within the simulation model, each of the five replications recorded the length of stay for every patient throughout the process, across the three models corresponding to the monthly groups, with a significance level of 5% (^{Appendix 5}). We compared the observed and simulated length of stay at reception, triage, and consultation. A comparison of the real and simulated confidence intervals revealed a statistically significant difference in the length of stay for models in groups 1, 2, and 3, with 95% confidence. These findings are corroborated by the Mann-Whitney U test results (^{Appendix 6}). This discrepancy may be attributed to the low quality of some process time records, as evident in Table 2.

In the analyzed emergency department, the indicators of average waiting time, average number of clients in the queue, and average resource utilization were recorded, considering the months of groups 1, 2, and 3 (Table 8). While there are notable differences in average queue time and the average number of patients in the queue between the month groups, these differences are not proportionally reflected in resource utilization. Resource utilization remains relatively homogeneous across the groups, ranging between 30% and 50%. However, it is important to highlight that triage resources experience higher utilization, and the performance indicators in this specific care segment are less favorable.

Table 8
Performance indicators on the activities of the months of group 1, 2 and 3.

Group	Stage	Queue time (in minutes)	Number of customers in the queue	Average resource utilization
1	Reception	2.16 (1.95; 2.37)	0.24 (0.22; 0.26)	32.5%
	Triage	4.63 (4.45; 4.81)	0.51 (0.49; 0.53)	36.6%
	Consultation	0.36 (0.3; 0.42)	0.04 (0.03; 0.05)	26.3%
2	Reception	5.08 (4.78; 5.38)	0.66 (0.62; 0.7)	38.9%
	Triage	9.91 (9.6; 10.22)	1.29 (1.24; 1.34)	43.5%
	Consultation	0.6 (0.52; 0.68)	0.08 (0.07; 0.09)	31.2%
3	Reception	12.13 (10.76; 13.5)	1.85 (1.64; 2.06)	45.6%
	Triage	19.37 (18.32; 20.42)	2.95 (2.78; 3.12)	50.7%
	Consultation	1.85 (1.56; 2.14)	0.28 (0.23; 0.33)	37.7%

A sensitivity analysis was conducted on the demand (arrival) within the model for the months of group 3. This analysis considered demand increases of 5%, 10%, and 15% compared to the current customer arrival rates, with the results presented in Table9. These results indicate a deterioration in queue performance indicators as arrival rates increase. It is particularly noteworthy that a 15% increase in emergency room demand during the months of group 3 led to a substantial 79.94% increase in the average length of stay in the emergency room, extending the average duration from 58.12 minutes to 104.58 minutes. At the reception, the queue time experiences an even more pronounced increase: the average queue time, currently 12.13 minutes, rises to 37.42 minutes, a significant increase of 208.49% for this indicator.

Table 9
Sensitivity analysis on the arrival of group 3.

Scenario		Current	5%	10%	15%
Average attendances per day		218.77	230.03	243.20	257.26
Average length of stay in the emergency department (minutes)		58.12	68.10	84.05	104.58
Lead Time	Reception	12.13	16.56	24.42	37.42
	Triage	19.37	24.53	31.23	37.00
	Consultation	1.85	2.25	3.65	5.38
Queue size	Reception	1.85	2.66	4.14	6.71
	Triage	2.95	3.93	5.29	6.63
	Consultation	0.28	0.36	0.62	0.96
Average occupancy	Reception	45.57%	47.99%	50.75%	54.07%
	Triage	50.68%	53.21%	56.25%	59.41%
	Consultation	37.65%	39.79%	42.86%	46.32%

It is crucial to acknowledge that significant stages of care within this sector, including outpatient care following medical consultation, imaging and laboratory tests, medication administration, and medical reassessment, could not be modeled due to the lack of records in the consulted database. Additionally, because there was an absence of records for service time at reception the use of queueing theory as an approximation for service times in this stage was necessary, being another limitation of this study. Collectively, these simplifications contribute to the non-validation of the model under analysis.

6 Proposed scenarios

The expectation that a 15% increase in arrival rates will significantly extend waiting times beyond current levels suggests possible improvements in the emergency sector under analysis. A set of alternative scenarios and resource configurations in reception, triage, and medical consultation was conceived as a means of prospecting improvements in this hospital.

The front desk has four employees on duty during shift 1 and three during shift 2 (referred to as 'R0'). In addition to the current setup, three other scenarios were considered: R1) five employees serving during shift 1 and two during shift 2; R2) six operators during shift 1 and two during shift 2; and R3) five employees during shift 1 and three during shift 2. Given that each shift lasts 12 hours and a week consists of seven working days, the equivalent number of man-hours (MH) was calculated for each scenario (Table 10).

Table 10
Configuration of resources in reception scenarios.

Reception	T1	T2	MH
R0	4	3	588
R1	5	2	588
R2	6	2	672
R3	5	3	756

Triage currently operates with a single employee available in both shifts (this configuration is referred to as 'T0'). Beyond the existing setup, three alternative scenarios were considered for triage: T1) two nurses performing the initial evaluation in shift 1 and one nurse in shift 2; T2) three nurses in shift 1 and two in shift 2; and T3) four nurses in shift 1 and two in shift 2. The equivalent MH were calculated for each of these scenarios (Table 11).

Table 11
Configuring Resources in Triage Scenarios.

Triage	T1	T2	MH
T0	1	1	168
T1	2	1	252
T2	3	2	420
T3	4	2	504

In the medical consultation section, the configuration of care resources varies according to the day of the week. In the current operation (labeled 'C0'), six doctors work during shift 1 on Mondays and Tuesdays, while three are present during shift 2. For the remainder of the week, five physicians work during shift 1, and two physicians are available during shift 2.

As physicians are critical financial resources in hospital management, the scenario development was guided by two principles: maintaining MH similar to current levels and allowing resource availability to fluctuate according to demand throughout the day (as previously detailed in Figure 2). For Wednesday through Sunday (referred to as '4d'), a consistent model of doctor availability was designed with 12-hour shifts. However, these shifts vary in start times, causing the total number of available doctors to fluctuate throughout the day to match the influx of patients during the emergency department’s operational hours. Further details about the shifts and the hourly availability of physicians can be found in the chart in ^{Appendix 7}.

Regarding the scenarios for allocating available physicians on Mondays and Tuesdays, these also adhere to the principle of maintaining MH similar to the current configuration, ensuring that the total number of physicians available per hour aligns with the arrival of clients throughout the day. For the first scenario on Mondays and Tuesdays, labeled '23f1', eight-hour shifts with varying start times were considered. In the second scenario, '23f2', 12-hour shifts were implemented, similar to the current operation in the emergency room. The start times of these shifts were also varied to ensure that the number of physicians matched the demand of the emergency room during its operational hours (see ^{Appendix 8} for details).

The combination of the '4d' proposal with the two proposals '23f1' and '23f2' resulted in the creation of two new scenarios, labeled 'C1' and 'C2', for the weekly operations in the medical consultation sector of this emergency department. The equivalent MH for these scenarios were also calculated and are detailed in Table 12.

Table 12
Configuration of resources in the scenarios of the medical consultation.

Scenario	Mondays and Tuesdays		Wednesday to Sunday		MH
	T1	T2	T1	T2
C0	12h: 6	12h: 3	12h: 5	12h: 2	636
C1	1h: 4	5h: 3	1h: 3	1h: 4	628
(23f1 + 4d)	1h: 5	6h: 2	1h: 4	1h: 3
	2h: 7	1h: 4	10h: 5	10h: 2
	3h: 8
	3h: 6
	2h: 4
C2	1h: 4	1h: 5	1h: 3	1h: 4	636
(23f2 + 4d)	1h: 6	1h: 3	1h: 4	1h: 3
	4h: 7	4h: 2	10h: 5	10h: 2
	6h: 6	6h: 3

From the different combinations of corresponding configurations in reception, triage, and medical consultation, 48 scenarios (Scn) were generated and evaluated using discrete event simulation, particularly focusing on the length of stay (LoS) in the emergency room under analysis. The configurations of staff allocated to reception (Rec), triage (Tri), and medical consultation (Med) for each scenario, along with the resulting LoS, are detailed in Table 13. As the scenarios were not compared through financial evaluations, the efficiency (Eff) of each scenario was calculated using output-oriented VRS-DEA model. This analysis considered three inputs (person-hours allocated to reception, triage, and medical consultation) and one output (average length of stay in the emergency room, as an undesirable output).

Table 13
Combinations between scenarios and results obtained.

Scn	Rec	Tri	Med	LoS	Eff	Scn	Rec	Tri	With	LoS	Eff
0	R0	T0	C0	58.12	57.5%	24	R1	T1	C1	44.71	82.4%
1	R1	T0	C0	76.57	0.5%	25	R1	T2	C1	44.38	82.4%
2	R2	T0	C0	75.70	3.2%	26	R1	T3	C1	44.64	81.4%
3	R3	T0	C0	53.15	72.9%	27	R2	T1	C1	41.29	81.1%
4	R0	T1	C0	39.25	96.3%	28	R2	T2	C1	39.58	83.7%
5	R0	T2	C0	38.72	96.8%	29	R2	T3	C1	41.38	79.6%
6	R0	T3	C0	38.71	96.4%	30	R3	T1	C1	29.28	99.0%
7	R1	T1	C0	43.08	86.5%	31	R3	T2	C1	29.21	97.2%
8	R1	T2	C0	41.90	88.7%	32	R3	T3	C1	28.90	97.8%
9	R1	T3	C0	41.02	90.5%	33	R1	T0	C2	76.72	0.0%
10	R2	T1	C0	39.40	85.4%	34	R2	T0	C2	73.93	8.6%
11	R2	T2	C0	38.93	85.2%	35	R3	T0	C2	51.68	77.4%
12	R2	T3	C0	37.51	88.1%	36	R0	T1	C2	37.82	100%
13	R3	T1	C0	29.69	96.9%	37	R0	T2	C2	37.45	100%
14	R3	T2	C0	29.54	95.3%	38	R0	T3	C2	37.28	100%
15	R3	T3	C0	29.37	95.6%	39	R1	T1	C2	42.34	88.4%
16	R0	T0	C1	56.25	63.3%	40	R1	T2	C2	39.95	93.6%
17	R0	T0	C2	56.48	62.6%	41	R1	T3	C2	42.47	86.8%
18	R1	T0	C1	76.16	1.7%	42	R2	T1	C2	38.50	87.4%
19	R2	T0	C1	75.13	4.9%	43	R2	T2	C2	38.00	87.2%
20	R3	T0	C1	52.31	75.5%	44	R2	T3	C2	38.30	86.4%
21	R0	T1	C1	39.04	97.0%	45	R3	T1	C2	28.19	100%
22	R0	T2	C1	38.39	97.7%	46	R3	T2	C2	27.23	100%
23	R0	T3	C1	38.02	98.2%	47	R3	T3	C2	27.19	100%

• Scn = Scenario; Rec = Combination of staff at the front desk; Tri = Combination of employees in triage; Med = Combination of medical consultation staff; LoS = Length of stay in the emergency room; Eff = Calculated Efficiency (DEA).

The current emergency configuration (Scenario 0) has an efficiency of 57.5%. In this scenario, the average length of stay is 58.12 minutes. However, by changing the configuration of medical care (as seen in Scenarios 16 and 17), the hospital's efficiency could increase to 63.3% and 62.6%, respectively. These improvements correspond to adopting the C1 and C2 standards in medical care, reducing the length of stay to 56.25 minutes and 56.48 minutes, respectively.

The most significant gains in efficiency occur when changes are made to triage care. For example, scenarios 4, 5, and 6, which involve only changes in the number of nurses in triage, achieve efficiencies of 96.3%, 96.8%, and 96.4%, respectively. Correspondingly, the average lengths of stay in the emergency room would be reduced to 39.25 minutes, 38.72 minutes, and 38.71 minutes.

Other scenarios that benefit the emergency care system include configurations 36, 37, and 38. These maintain the current reception service and adopt the C2 configuration for medical consultation. They also implement varying triage care combinations (T1, T2, and T3, respectively). Each has achieved an efficiency indicator of 100%, with emergency room stay times of 37.82 minutes, 37.45 minutes, and 37.28 minutes, respectively.

The best results are observed in combinations 45, 46, and 47, where the service configuration at the reception is R3, and at medical appointments, it is C2. The key difference among these scenarios lies in the triage profiles adopted. Notably, in all three scenarios, the efficiency reached 100%, with average lengths of stay in the hospital recorded at 28.19 minutes, 27.23 minutes, and 27.19 minutes, respectively.

It is important to note that DEA efficiency indicators compare units to each other to identify the most efficient ones but do not provide absolute information on optimal performance. As noted by Cook et al. (2014), a scenario can be classified as efficient even if there is potential for improvement. Therefore, while the efficiency indicator offers an optimal relationship between the use of human resources and the resulting length of stay in the emergency room, further analysis of the employees' utilization is also valuable. Table 14 displays the utilization rate of human resources in reception, triage, and medical consultation operations for each of the six scenarios classified as efficient in this analysis.

Table 14
Resource utilization in the efficient scenarios.

Scenario	Reception	Triage	Consultation
Current (0)	45.60%	50.70%	37.70%
36	45.49%	29.78%	27.69%
37	45.51%	18.48%	27.66%
38	45.48%	15.03%	27.68%
45	33.87%	29.79%	27.79%
46	33.63%	18.47%	27.70%
47	33.67%	14.96%	27.70%

As demonstrated, all the efficient scenarios reported a utilization rate below 50%, regardless of the service sector analyzed. Initially, a utilization rate below 50% might suggest that resources are not being used efficiently, potentially indicating oversized capacity. In a hospital context, this could mean that there is more staff than needed for the volume of patients treated, leading to higher operating costs without a corresponding improvement in the quality of care. However, it is important to consider that a low utilization rate also reduces strain on resources, enhancing the facility’s responsiveness during unexpected peaks in demand or emergencies.

Finally, in the DEA assessment, the average length of stay in the emergency room – an undesirable outcome – is an output indicator targeted for minimization. Notably, a correlation exists between low resource utilization and shorter patient stays. This relationship is evident when comparing scenarios with low resource usage and reduced dwell times, demonstrating that such resource allocations can effectively decrease waiting times for patients in this system.

It is crucial to perform a sensitivity analysis on the scenarios identified as efficient to stress test them and understand their response under different conditions. Specifically, scenario 47, which showed superior performance in terms of average length of stay in the emergency room, was selected for this analysis. A sensitivity analysis was conducted focusing on the arrival rate to the system, with an increase of 15% and 30% in the volume of patients seeking emergency care considered. The results are detailed in Table 15, which includes the average length of stay, total number of consultations during the simulated period, and the average occupancy rates in triage, reception, and medical consultation for each sensitivity scenario.

Table 15
Sensitivity analysis on scenario 47.

Sensitivity	LoS (in minutes)	Total attended	Occupation reception	Occupation triage	Occupation consultation
Current (0)	52.12	118,137	45.60%	50.70%	37.70%
Scenario 47	27.19	118,143	33.67%	14.96%	27.7%
Scenario 47 with 15%	31.61	138,948	39.81%	17.59%	32.4%
Scenario 47 with 30%	48.09	168,717	48.44%	21.39%	39.0%

Compared to the sensitivity analysis performed on current scenarios for the three demand groups (as detailed in Table 8), it is observed that in scenario 47, the average length of stay is 27.19 minutes. This duration would increase to 31.61 minutes with a 15% rise in the current demand of group 3, reflecting an increase of 16.25% in this indicator. Similarly, under current conditions, scenario 47 handles 118,143 patient visits during the simulation period. This number would rise to 138,948 with the increased demand, a 17.61% increase in the number of completed consultations.

In this further sensitivity analysis, a stress increase of 30% in the demand of group 3 was applied under the conditions of scenario 47. This increase in arrivals resulted in the average length of stay in the emergency room escalating to 48.09 minutes, a substantial rise of 76.87%. Concurrently, the total number of patients treated during this scenario would jump to 168,717, marking a 42.81% increase from baseline conditions.

Scenario 47, as demonstrated by the sensitivity analysis, not only proves to be efficient but also shows remarkable robustness under stress and during unexpected peaks in demand. Even with a significant 30% increase in demand, the length of stay in the emergency room - though higher - remains below the duration currently observed in the operations for demand group 3. This resilience is crucial for hospital management, as it indicates that the scenario can handle a substantially higher volume of patients without critically compromising the quality of care.

In addition, the occupancy rates for resources such as reception, triage, and medical consultation in scenario 47, even under the stress of increased demand, remain below those observed in the current operation. This indicates that scenario 47 not only optimizes the use of available resources but also maintains the hospital’s responsiveness in challenging conditions. These results demonstrate that the suggested configuration efficiently meets the daily operational needs of the hospital, while also ensuring a safety margin that allows for significant variations in demand without a noticeable degradation in service quality.

Therefore, the implementation of scenario 47 represents not only a strategic choice to enhance standard performance indicators but also a crucial investment in the resilience of the hospital system. This capacity to adapt and respond effectively to unexpected fluctuations is vital for the long-term sustainability of any emergency department. It ensures that, even under pressure, the hospital can continue delivering high-quality care to patients.

7 Conclusions

Agile and effective care in a hospital's emergency department is crucial for ensuring patient safety and health. Research by Doudareva & Carter (2022) and Boyle et al. (2023) demonstrates that prolonged waiting times in emergency departments are linked to increased morbidity and mortality among patients. Consequently, employing techniques that enhance both the speed and the effectiveness of care is essential to support high levels of patient safety and treatment integrity.

In this context, the current study proposes a hybrid approach to evaluate the performance of the emergency department of a hospital in Rio de Janeiro using DES and DEA. This approach integrates the methodologies proposed by Egilmez et al. (2018) and Taleb et al. (2023). Specifically, DES was employed to assess current operational performance under varying demand conditions, as observed onsite.

For the months of low demand, the average length of stay in the emergency room was 31.94 minutes. In months of intermediate demand, this duration increased to 40.45 minutes. During the months with the highest demand, the average stay extended to 58.12 minutes. Furthermore, a sensitivity analysis revealed that a 15% increase in demand under these high-demand conditions would result in a 79.93% increase in the length of stay in the emergency room and a 208.49% increase in waiting time at the reception.

The proposal to vary the resource allocation configurations in reception, triage and medical consultation led to the creation of 48 distinct scenarios. These scenarios were assessed based on the estimated length of stay and the number of resources allocated to these sectors, with efficiency indicators derived from DEA evaluations. Six of these scenarios were found to be efficient and are viable for implementation by the emergency department of the analyzed hospital, ensuring enhanced agility in patient care.

Although the scenarios identified as efficient might suggest an underutilization of human resources in reception, triage, and medical consultation, a sensitivity analysis revealed that such allocations significantly enhance the capacity to manage unexpected or emergency demand peaks. When scenario 47 was subjected to increases in demand of 15% and 30%, the average length of stay and the resource utilization indicators remained lower than those observed under the current, unstressed operation. This result indicates that a new configuration, like that proposed in scenario 47, not only elevates operational efficiency but also boosts the hospital's resilience by ensuring greater responsiveness and adaptability.

It is important to highlight that this study was limited to analyzing only the initial stages of emergency care, and future studies should consider all stages to ensure greater precision and a more accurate reflection of the actual system evaluated. Additionally, future research should explore replicating the methodology with real service times for all operations rather than relying solely on estimates derived from queuing theory. As part of future studies, it is planned to evaluate the current layout of the emergency department using the DES model, incorporating animation with advanced graphic resources. Such an analysis could significantly enhance the visualization of impacts on the hospital's layout and spatial efficiency.

Statement on Data Availability

The study dataset contains sensitive information, such as personal information about the hospital's patients. For this reason, they cannot be made available in the publication but can be accessed on demand.

Appendix 1  Average number of customers arriving per hour on each day of the week according to the group of months.

	Group 1							Group 2							Group 3
Timetable	Mon	Tue	Wed	Thu	Fri	Sat	Sun	Mon	Tue	Wed	Thu	Fri	Sat	Sun	Mon	Tue	Wed	Thu	Fri	Sat	Sun
0am-1am	0.71	0.84	0.84	1.11	1.00	0.85	1.00	1.50	1.03	0.86	1.31	0.97	1.00	0.78	1.81	1.74	1.79	1.50	1.54	1.44	1.44
1am-2am	0.13	0.28	0.24	0.11	0.32	0.37	0.38	0.34	0.30	0.38	0.10	0.21	0.58	0.66	0.62	0.41	0.54	0.35	0.81	0.80	0.40
2pm-3pm	0.08	0.12	0.16	0.11	0.14	0.04	0.13	0.19	0.13	0.21	0.17	0.07	0.19	0.16	0.19	0.48	0.25	0.38	0.42	0.36	0.60
3am-4am	0.13	0.16	0.04	0.07	0.07	0.04	0.13	0.13	0.07	0.21	0.21	0.03	0.16	0.16	0.27	0.26	0.07	0.31	0.12	0.24	0.16
4am-5am	0.00	0.04	0.04	0.07	0.04	0.04	0.13	0.06	0.20	0.21	0.28	0.10	0.13	0.06	0.23	0.15	0.14	0.31	0.27	0.36	0.36
5am-6am	0.21	0.28	0.16	0.11	0.25	0.00	0.25	0.31	0.63	0.31	0.52	0.38	0.32	0.03	0.88	0.48	0.86	0.85	0.92	0.32	0.60
6am-7am	2.21	1.72	1.32	1.44	1.75	0.81	0.83	2.91	2.50	2.52	1.48	1.79	1.16	1.16	2.69	2.93	2.93	2.65	3.19	1.72	1.72
7am-8am	7.92	6.96	6.00	6.52	6.36	3.59	3.63	8.59	9.77	8.31	7.66	6.90	3.74	3.47	11.65	11.48	11.07	9.58	9.23	5.80	5.12
8am-9am	13.46	11.96	10.04	11.48	9.54	6.22	6.67	14.59	14.00	14.41	12.59	10.62	7.74	6.78	21.38	16.70	17.46	15.73	14.88	9.52	9.20
9am-10am	16.79	14.16	12.64	13.96	12.46	8.56	9.42	20.53	17.73	15.52	16.55	14.83	10.39	9.78	25.12	21.11	20.11	19.88	16.77	14.00	13.32
10am-11am	21.04	17.56	14.52	14.89	13.96	11.11	11.29	20.53	21.00	20.24	18.17	16.83	13.77	13.72	26.04	21.00	22.68	21.77	21.31	16.36	16.48
11am-12pm	15.17	14.84	14.64	14.96	11.57	12.19	11.88	20.69	18.53	18.24	19.03	15.48	13.26	14.16	23.27	19.44	18.54	19.69	18.31	16.76	16.36
12pm-1pm	15.17	12.76	12.48	12.89	10.43	10.11	9.21	17.59	15.37	15.62	14.03	12.14	12.42	11.53	18.42	16.19	16.64	16.62	14.69	14.40	13.64
1pm-2pm	16.25	12.76	11.24	14.00	11.25	8.85	8.00	17.69	17.00	14.86	13.38	13.03	10.32	9.81	19.00	17.74	18.04	16.54	16.58	11.84	12.24
2pm-3pm	15.17	13.32	13.12	12.89	10.79	9.33	8.33	17.69	15.67	16.10	15.24	13.17	10.32	10.50	18.12	18.52	17.82	15.38	16.15	12.32	11.72
3pm-4pm	13.83	12.88	10.80	10.59	10.82	8.04	8.42	16.16	14.47	13.90	13.10	12.14	9.81	10.91	16.92	16.15	15.64	12.15	14.85	12.08	13.08
4pm-5pm	12.50	11.08	10.04	10.26	8.86	8.67	7.38	15.03	13.37	12.59	11.93	10.17	9.19	9.00	16.58	12.93	12.46	11.58	12.50	11.28	10.28
5pm-6pm	11.63	11.60	9.36	7.96	7.64	6.74	6.96	13.13	12.70	10.41	11.34	9.48	8.23	8.25	13.96	14.37	10.50	10.73	10.73	9.28	9.28
6pm-7pm	11.71	9.84	8.96	8.19	8.00	5.70	5.67	13.19	11.90	11.34	9.55	9.14	6.39	7.56	15.04	13.85	12.21	11.31	11.15	8.36	8.08
7pm-8pm	10.42	8.80	9.20	8.85	6.57	5.19	6.75	11.81	10.63	10.14	9.41	8.62	6.58	7.59	14.38	14.07	12.86	9.88	10.69	6.88	9.04
8pm-9pm	8.38	7.52	6.92	7.56	7.75	4.96	5.38	10.16	9.37	8.38	8.62	7.45	4.74	5.47	10.92	10.78	8.86	8.50	8.31	6.52	7.48
9pm-10pm	5.96	4.92	5.72	4.67	4.50	3.00	4.83	6.72	6.00	6.14	5.93	5.21	4.10	5.41	8.50	6.48	7.11	6.54	6.73	6.16	5.84
10pm-11pm	4.25	3.60	3.00	3.63	3.25	2.89	2.83	4.56	3.83	4.03	4.10	3.24	3.65	3.34	5.42	5.11	4.89	4.04	4.38	3.16	3.72
11pm-0am	2.29	1.60	2.24	2.07	1.93	1.85	2.13	2.75	2.30	2.41	2.28	2.28	2.19	2.06	3.15	2.78	2.68	2.73	2.35	2.00	2.84

Appendix 2  Distributions with greater fit for the triage and medical consultation processes per shift.

Station	Shift	Expression	Quadratic error	Kolmogorov-Smirnov Test	Chi-square Test
Triage	T1	1.5 + LOGN(1.05. 0.848)	0.002263	1.0	52.7
				(< 0.05)	(< 0.05)
	T2	1.5 + EXPO(1.7)	0.001118	0.998	34.3
				(< 0.05)	(< 0.05)
Consultation	T1	2.5 + GAMM(5.78. 1.29)	0.000236	1.0	2.3e+03
				(< 0.05)	(< 0.05)
	T2	4.5 + EXPO(6.49)	0.000185	1.0	882
				(< 0.05)	(< 0.05)

Appendix 3

Distributions with greater fit for the screening and consultation processes per shift.

Appendix 4  Queuing Theory and estimation of service time at the reception.

Shift	c	W (min)	λ (/h)	μ (/h)	1/μ (min)
T1	4	13.90	12.92	5.21	11.52
T2	3	10.35	2.80	5.83	10.29

Appendix 5  Comparison of means between groups 1. 2 and 3 in the simulated models (in minutes).

	Length of stay	G1	G2	G3
Simulated	Total	31.94	40.45	58.55
		(31.88 ; 31.99)	(40.38 ; 40.52)	(58.44 ; 58.66)
	Reception	13.43	16.46	23.67
		(13.39 ; 13.47)	(16.41 ; 16.51)	(23.61 ; 23.74)
	Triage	8.03	13.28	22.86
		(8.00 ; 8.06)	(13.24 ; 13.32)	(22.79 ; 22.93)
	Consultation	10.48	10.71	12.02
		(10.45 ; 10.5)	(10.69 ; 10.73)	(11.99 ; 12.04)
Real	Total	39.55	46.12	56.51
		(39.24 ; 39.85)	(45.8 ; 46.44)	(56.08 ; 56.93)
	Reception	11.54	12.16	16.03
		(11.43 ; 11.64)	(12.08 ; 12.25)	(15.89 ; 16.17)
	Triage	9.25	9.12	15.67
		(9.16 ; 9.35)	(9.02 ; 9.23)	(15.13 ; 16.21)
	Consultation	18.57	21.35	30.48
		(18.33 ; 18.8)	(21.04 ; 21.67)	(30.00 ; 30.95)

Appendix 6  U-test statistic (p-value) for the statistical difference between the length of stay in the emergency room and the simulated models of groups 1, 2 and 3.

Test U	Group 1	Group 2	Group 3
Total	5,419,381,638	8,297,378,868	9,177,173,481
	(<0.05)	(<0.05)	(<0.05)
Reception	5,419,381,638	8,256,692,361	9,239,373,379
	(<0.05)	(<0.05)	(<0.05)
Triage	4,702,939,975	3,780,667,418	5,496,865,545
	(<0.05)	(<0.05)	(<0.05)
Consultation	4,333,842,674	4,295,193,555	7,653,239,678
	(<0.05)	(<0.05)	(<0.05)

Appendix 7

Proposal for the number of physicians available from Wednesday to Sunday

Appendix 8

Proposals for the number of physicians available for Mondays and Tuesdays.

• Financial support: This work was supported by the Carlos Chagas Filho Foundation for Research Support in Rio de Janeiro State (FAPERJ) [E-26/210.858/2024 to I.T.P]; the Coordination for the Improvement of Higher Education Personnel (CAPES); and the Pontifical Catholic University of Rio de Janeiro.
• How to cite: Oliveira, L. F. A., Peres, I. T., & Araujo, B. M. (2024). Performance evaluation of an emergency department in Rio de Janeiro: a hybrid approach using Discrete Events Simulation and Data Envelopment Analysis. Gestão & Produção, 31, e1024. https://doi.org/10.1590/1806-9649-2024v31e1024

References

• Aringhieri, R., Bruni, M. E., Khodaparasti, S., & van Essen, J. T. (2017). Emergency medical services and beyond: addressing new challenges through a wide literature review. Computers & Operations Research, 78, 349-368. http://doi.org/10.1016/j.cor.2016.09.016
  » http://doi.org/10.1016/j.cor.2016.09.016
• Azadeh, A., Moghaddam, M., Asadzadeh, S. M., & Negahban, A. (2011). An integrated fuzzy simulation-fuzzy data envelopment analysis algorithm for job-shop layout optimization: the case of injection process with ambiguous data. European Journal of Operational Research, 214(3), 768-779. http://doi.org/10.1016/j.ejor.2011.05.015
  » http://doi.org/10.1016/j.ejor.2011.05.015
• Azadeh, A., Motevali Haghighi, S., & Asadzadeh, S. M. (2014). A novel algorithm for layout optimization of injection process with random demands and sequence dependent setup times. Journal of Manufacturing Systems, 33(2), 287-302. http://doi.org/10.1016/j.jmsy.2013.12.008
  » http://doi.org/10.1016/j.jmsy.2013.12.008
• Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis. Management Science, 30(9), 1078-1092. http://doi.org/10.1287/mnsc.30.9.1078
  » http://doi.org/10.1287/mnsc.30.9.1078
• Boyle, L. M., Currie, C., Fernandez, C. L., Nguyen, L., & Halpenny, C. (2023). A discrete event simulation model of a hospital for prediction of the impact of delayed discharge. In The OR Society 11th Simulation Workshop: SW23: Proceedings (pp. 240-249). Southampton: The OR Society. http://doi.org/10.36819/SW23.029
  » http://doi.org/10.36819/SW23.029
• Brailsford, S. C., Eldabi, T., Kunc, M., Mustafee, N., & Osorio, A. F. (2019). Hybrid simulation modelling in operational research: a state-of-the-art review. European Journal of Operational Research, 278(3), 721-737. http://doi.org/10.1016/j.ejor.2018.10.025
  » http://doi.org/10.1016/j.ejor.2018.10.025
• Brasil. (2018, 15 de agosto). Lei nº 13.709, de 14 de agosto de 2018. Lei Geral de Proteção de Dados Pessoais (LGPD). Brasília, DF: Diário Oficial da República Federativa do Brasil (seção 1, nº 157, pp. 55-58). Retrieved in 2024, May 22, from https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm
  » https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm
• Charnes, A., Cooper, W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429-444. http://doi.org/10.1016/0377-2217(78)90138-8
  » http://doi.org/10.1016/0377-2217(78)90138-8
• Chinosi, M., & Trombetta, A. (2012). BPMN: an introduction to the standard. Computer Standards & Interfaces, 34(1), 124-134. http://doi.org/10.1016/j.csi.2011.06.002
  » http://doi.org/10.1016/j.csi.2011.06.002
• Chwif, L., & Medina, A. C. (2014). Modelagem e simulação de eventos discretos: teoria e aplicações (4ª ed.). São Paulo: Elsevier Brasil.
• Cook, W. D., & Seiford, L. M. (2009). Data envelopment analysis (DEA): thirty years on. European Journal of Operational Research, 192(1), 1-17. http://doi.org/10.1016/j.ejor.2008.01.032
  » http://doi.org/10.1016/j.ejor.2008.01.032
• Cook, W. D., Tone, K., & Zhu, J. (2014). Data envelopment analysis: prior to choosing a model. Omega, 44, 1-4. http://doi.org/10.1016/j.omega.2013.09.004
  » http://doi.org/10.1016/j.omega.2013.09.004
• Cooper, W. W., Seiford, L. M., & Tone, K. (2007). Data Envelopment Analysis: a comprehensive text with models, applications, references and DEA-solver software (2nd ed.). New York: Springer.
• Currie, C. S. M., & Cheng, R. C. H. 2016. A practical introduction to analysis of simulation output data. In 2016 Winter Simulation Conference (pp. 118-132), Washington. New York: IEEE. http://doi.org/10.1109/WSC.2016.7822084
  » http://doi.org/10.1109/WSC.2016.7822084
• De Roock, E., & Martin, N. (2022). Process mining in healthcare: an updated perspective on the state of the art. Journal of Biomedical Informatics, 127, 103995. http://doi.org/10.1016/j.jbi.2022.103995 PMid:35077900.
  » http://doi.org/10.1016/j.jbi.2022.103995
• Dev, N. K., Shankar, R., & Debnath, R. M. (2014). Supply chain efficiency: a simulation cum DEA approach. International Journal of Advanced Manufacturing Technology, 72(9-12), 1537-1549. http://doi.org/10.1007/s00170-014-5779-6
  » http://doi.org/10.1007/s00170-014-5779-6
• Doudareva, E., & Carter, M. (2022). Discrete event simulation for emergency department modelling: a systematic review of validation methods. Operations Research for Health Care, 33, 100340. http://doi.org/10.1016/j.orhc.2022.100340
  » http://doi.org/10.1016/j.orhc.2022.100340
• Egilmez, G., Sormaz, D., & Gedik, R. (2018). A Project-based learning approach in teaching simulation to undergraduate and graduate students. In ASEE Annual Conference & Exposition (pp. 1-10), Salt Lake City. Washington, D.C.: ASEE. http://doi.org/10.18260/1-2--29716
  » http://doi.org/10.18260/1-2--29716
• Fogliatti, M. C., & Matos, N. M. C. (2007). Teoria de filas (1ª ed.). Rio de Janeiro: Interciência.
• Fone, D., Hollinghurst, S., Temple, M., Round, A., Lester, N., Weightman, A., Roberts, K., Coyle, E., Bevan, G., & Palmer, S.(2003). Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health Med., 25(4), p. 325-335. https://doi.org/10.1093/pubmed/fdg075
  » https://doi.org/10.1093/pubmed/fdg075
• Garcia, C. S., Meincheim, A., Faria, E. R., Jr., Dallagassa, M. R., Sato, D. M. V., Carvalho, D. R., Santos, E. A. P., & Scalabrin, E. E. (2019). Process mining techniques and applications: a systematic mapping study. Expert Systems with Applications, 133(1), 260-295. http://doi.org/10.1016/j.eswa.2019.05.003
  » http://doi.org/10.1016/j.eswa.2019.05.003
• Giacomo, G., Dumas, M., Maggi, F. M., & Montali, M. (2015). Declarative process modeling in BPMN. In J. Zdravkovic, M. Kirikova & P. Johannesson (Eds.), Advanced Information Systems Engineering (pp. 84-100). Cham: Springer. http://doi.org/10.1007/978-3-319-19069-3_6
  » http://doi.org/10.1007/978-3-319-19069-3_6
• Günal, M. M., & Pidd, M. (2007). Interconnected DES models of emergency, outpatient, and inpatient departments of a hospital. In 2007 Winter Simulation Conference (pp. 1461-1466). Washington, DC. http://doi.org/10.1109/WSC.2007.4419757
  » http://doi.org/10.1109/WSC.2007.4419757
• Hart, A. (2001). Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ, 323(7309), 391-393. http://doi.org/10.1136/bmj.323.7309.391 PMid:11509435.
  » http://doi.org/10.1136/bmj.323.7309.391
• Hoad, K., Robinson, S., & Davies, R. (2008). Automating warm-up length estimation. In Winter Simulation Conference (pp. 532-540), Miami. New York: IEEE. http://doi.org/10.1109/WSC.2008.4736110
  » http://doi.org/10.1109/WSC.2008.4736110
• Hoad, K., Robinson, S., & Davies, R. (2010). Automating warm-up length estimation. The Journal of the Operational Research Society, 61(9), 1389-1403. http://doi.org/10.1057/jors.2009.87
  » http://doi.org/10.1057/jors.2009.87
• Hosseini, Z., Navazi, F., Siadat, A., Memari, P., & Tavakkoli-Moghaddam, R. (2019). A tailored fuzzy simulation integrated with a fuzzy DEA method for a resilient facility layout problem: a case study of a refrigerator injection process. IFAC-PapersOnLine, 52(13), 541-546. http://doi.org/10.1016/j.ifacol.2019.11.214
  » http://doi.org/10.1016/j.ifacol.2019.11.214
• Jarvis, P. R. E. (2016). Improving emergency department patient flow. Clinical and Experimental Emergency Medicine, 3(2), 63-68. http://doi.org/10.15441/ceem.16.127 PMid:27752619.
  » http://doi.org/10.15441/ceem.16.127
• Keshtkar, L., Rashwan, W., Abo-Hamad, W., & Arisha, A. (2020). A hybrid system dynamics, discrete event simulation and data envelopment analysis to investigate boarding patients in acute hospitals. Operations Research for Health Care, 26, 100266. http://doi.org/10.1016/j.orhc.2020.100266
  » http://doi.org/10.1016/j.orhc.2020.100266
• Kleijnen, J. P. (1995). Verification and validation of simulation models. European Journal of Operational Research, 82(1), 145-162. http://doi.org/10.1016/0377-2217(94)00016-6
  » http://doi.org/10.1016/0377-2217(94)00016-6
• Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583-621. http://doi.org/10.1080/01621459.1952.10483441
  » http://doi.org/10.1080/01621459.1952.10483441
• Liu, S., Li, Y., Triantis, K. P., Xue, H., & Wang, Y. (2020). The diffusion of discrete event simulation approaches in health care management in the past four decades: a comprehensive review. MDM Policy & Practice, 5(1), 2381468320915242. http://doi.org/10.1177/2381468320915242 PMid:32551365.
  » http://doi.org/10.1177/2381468320915242
• Liu, J. S., Lu, L. Y. Y., Lu, W.-M., & Lin, B. J. Y. (2013). Data envelopment analysis 1978-2010: a citation-based literature survey. Omega, 41(1), 3-15. http://doi.org/10.1016/j.omega.2010.12.006
  » http://doi.org/10.1016/j.omega.2010.12.006
• Macal, C. (2016). Everything you need to know about agent-based modelling and simulation. Journal of Simulation, 10(2), 144-156. http://doi.org/10.1057/jos.2016.7
  » http://doi.org/10.1057/jos.2016.7
• Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50-60. http://doi.org/10.1214/aoms/1177730491
  » http://doi.org/10.1214/aoms/1177730491
• Marchesi, J. F., Hamacher, S., & Fleck, J. L. (2020). A stochastic programming approach to the physician staffing and scheduling problem. Computers & Industrial Engineering, 142, 106281. http://doi.org/10.1016/j.cie.2020.106281
  » http://doi.org/10.1016/j.cie.2020.106281
• Marlin, B., & Sohn, H. (2016). Using DEA in conjunction with designs of experiments: an approach to assess simulated futures in the Afghan educational system. Journal of Simulation, 10(4), 272-282. http://doi.org/10.1057/jos.2015.14
  » http://doi.org/10.1057/jos.2015.14
• Mohiuddin, S., Busby, J., Savović, J., Richards, A., Northstone, K., Hollingworth, W., Donovan, J. L., & Vasilakis, C. (2017). Patient flow within UK emergency departments: a systematic review of the use of computer simulation modelling methods. BMJ Open, 7(5), e015007. http://doi.org/10.1136/bmjopen-2016-015007 PMid:28487459.
  » http://doi.org/10.1136/bmjopen-2016-015007
• Monazzam, N., Alinezhad, A., & Adibi, M. A. (2022). Simulation-based optimization using DEA and DOE in production systems. Scientia Iranica, 29(6), 3470-3488. http://doi.org/10.24200/sci.2021.55499.4253
  » http://doi.org/10.24200/sci.2021.55499.4253
• Nance, R. E. (1981). The time and state relationships in simulation modelling. Communications of the ACM, 24(4), 173-179. http://doi.org/10.1145/358598.358601
  » http://doi.org/10.1145/358598.358601
• Navazi, F., Tavakkoli-Moghaddam, R., & Memari, P. (2019). Layout optimization of injection process by considering integrated resilience engineering: a fuzzy-DEA approach. International Journal of Modelling and Simulation, 41(1), 52–66. https://doi.org/10.1080/02286203.2019.1670325.
  » https://doi.org/10.1080/02286203.2019.1670325.
• Norouzian-Maleki, P., Izadbakhsh, H., Saberi, M., Hussain, O., Jahangoshai Rezaee, M., & GhanbarTehrani, N. (2022). An integrated approach to system dynamics and data envelopment analysis for determining efficient policies and forecasting travel demand in an urban transport system. Transportation Letters, 14(2), 157-173. http://doi.org/10.1080/19427867.2020.1839716
  » http://doi.org/10.1080/19427867.2020.1839716
• Ortíz-Barrios, M. A., & Alfaro-Saíz, J.-J. (2020). Methodological approaches to support process improvement in emergency departments: a systematic review. International Journal of Environmental Research and Public Health, 17(8), 2664. http://doi.org/10.3390/ijerph17082664 PMid:32294985.
  » http://doi.org/10.3390/ijerph17082664
• Ouda, E., Sleptchenko, A., & Simsekler, M. C. E. (2023). Comprehensive review and future research agenda on discrete-event simulation and agent-based simulation of emergency departments. Simulation Modelling Practice and Theory, 129, 102823. http://doi.org/10.1016/j.simpat.2023.102823
  » http://doi.org/10.1016/j.simpat.2023.102823
• Peres, I. T., Hamacher, S., Cyrino Oliveira, F. L., Barbosa, S. D. J., & Viegas, F. (2019). Simulation of appointment scheduling policies: a study in a bariatric clinic. Obesity Surgery, 29(9), 2824-2830. http://doi.org/10.1007/s11695-019-03898-1 PMid:31037596.
  » http://doi.org/10.1007/s11695-019-03898-1
• Pham, H. L., Phouratsamay, S.-L., Di Mascolo, M., & Nguyen, B. Q. (2023). Reducing outpatient waiting time: a case study in Vietnamese private hospital. In Proceedings of the International Conference on Control, Automation and Diagnosis (pp. 1-6), Rome. New York: IEEE. http://doi.org/10.1109/ICCAD57653.2023.10152381
  » http://doi.org/10.1109/ICCAD57653.2023.10152381
• Prado, D., & Yamaguchi, M. 2019. Usando o Arena em simulação (6ª ed.). São Paulo: Falconi.
• Roberts, S. D., & Pegden, D. (2017). The history of simulation modeling. In 2017 Winter Simulation Conference (WSC) (pp. 308-323), Las Vegas, NV. New York: IEEE. http://doi.org/10.1109/WSC.2017.8247795
  » http://doi.org/10.1109/WSC.2017.8247795
• Sargent, R. G., Goldsman, D. M., & Yaacoub, T. (2016). A tutorial on the operational validation of simulation models. In 2016 Winter Simulation Conference (WSC) (pp. 163-177). Washington, DC. http://doi.org/10.1109/WSC.2016.7822087
  » http://doi.org/10.1109/WSC.2016.7822087
• Seiford, L., & Zhu, J. (2002). Modeling Undesirable Factors in Efficiency Evaluation. European Journal of Operational Research, 142(1), 16-20. http://doi.org/10.1016/S0377-2217(01)00293-4
  » http://doi.org/10.1016/S0377-2217(01)00293-4
• Student. (1908). The probable error of a mean. Biometrika, 6(1), 1-25. http://doi.org/10.2307/2331554
  » http://doi.org/10.2307/2331554
• Taleb, M., Khalid, R., Ramli, R., & Nawawi, M. K. M. (2023). An integrated approach of discrete event simulation and a non-radial super efficiency data envelopment analysis for performance evaluation of an emergency department. Expert Systems with Applications, 220, 119653. http://doi.org/10.1016/j.eswa.2023.119653
  » http://doi.org/10.1016/j.eswa.2023.119653
• Tavakoli, M., Tavakkoli-Moghaddam, R., Mesbahi, R., Ghanavati-Nejad, M., & Tajally, A. (2022). Simulation of the COVID-19 patient flow and investigation of the future patient arrival using a time-series prediction model: a real-case study. Medical & Biological Engineering & Computing, 60(4), 969-990. http://doi.org/10.1007/s11517-022-02525-z PMid:35152366.
  » http://doi.org/10.1007/s11517-022-02525-z
• Tukey, J. W. (1977). Exploratory data analysis (Vol. 2, pp. 131-160). Reading, MA: Addison-wesley.
• van den Bergh, J. V., De Bruecker, P., Beliën, J., De Boeck, L., & Demeulemeester, E. (2013). A three-stage approach for aircraft line maintenance personnel rostering using MIP, discrete event simulation and DEA. Expert Systems with Applications, 40(7), 2659-2668. http://doi.org/10.1016/j.eswa.2012.11.009
  » http://doi.org/10.1016/j.eswa.2012.11.009
• Vanbrabant, L., Braekers, K., Ramaekers, K., & Van Nieuwenhuyse, I. (2019). Simulation of emergency department operations: a comprehensive review of KPIs and operational improvements. Computers & Industrial Engineering, 131, 356-381. http://doi.org/10.1016/j.cie.2019.03.025
  » http://doi.org/10.1016/j.cie.2019.03.025
• Vázquez-Serrano, J. I., Peimbert-García, R. E., & Cárdenas-Barrón, L. E. (2021). Discrete-event simulation modeling in healthcare: a comprehensive review. International Journal of Environmental Research and Public Health, 18(22), 12262. http://doi.org/10.3390/ijerph182212262 PMid:34832016.
  » http://doi.org/10.3390/ijerph182212262
• Weng, S. J., Tsai, B. S., Wang, L. M., Chang, C. Y., & Gotcher, D. (2011). Using simulation and Data Envelopment Analysis in optimal healthcare efficiency allocations. In Proceedings of the 2011 Winter Simulation Conference (WSC) (pp. 1295-1305), Phoenix, AZ, USA. New York: IEEE. http://doi.org/10.1109/WSC.2011.6147850
  » http://doi.org/10.1109/WSC.2011.6147850
• Zeigler, B. P., & Muzy, A. (2017). From Discrete Event Simulation to Discrete Event Specified Systems (DEVS). IFAC-PapersOnLine, 50(1), 3039-3044. http://doi.org/10.1016/j.ifacol.2017.08.672
  » http://doi.org/10.1016/j.ifacol.2017.08.672

Publication Dates

• Publication in this collection
  02 Sept 2024
• Date of issue
  2024

History

• Received
  04 June 2024
• Accepted
  10 June 2024