Development of performance indicators for Occupational Health and Safety: a constructivist multicriteria approach for PPEAbstract
Paper aims This study aims to develop indicators, using the Multi-Criteria Decision Aid - Constructivist methodology (MCDA-C), to assess the occupational prevention activities in a psychiatric hospital.
Originality The application to the health sector of MCDA-C, in a case study during the supply of medicines to patients.
Research method We use the ProKnow-C. Then, MCDA-C was applied in a case study.
Main findings Four interest areas were defined, then the critical situations were identified to construct 126 specific indicators and finally the intervention strategies were defined. 26 indicators focused on PPE, 4 lagging and 22 leading. The model enabled the mapping of performance indicator levels. For that, 48 recommendations, and intervention strategies for each situation were defined.
Implications for theory and practice The use of specific indicators, obtained through a participatory process, allowed the identification of accurate and reliable metrics to measure the actual performance of activities, critical situations, and strategies.
Keywords Leading indicators. Lagging indicators. Performance measurement. Occupational health and safety. Multi-Criteria Decision Aid - Constructivist (MCDA-C).
1. Introduction
Accidents and work-related diseases cost, in the world, at least 2,680 billion euros per year (Elsler et al., 2017) and it is estimated that 7,500 people die due to unsafe and unhealthy working conditions every day (International Labour Organization, 2019). To control and reduce this situation, occupational risk prevention activities are developed, which must be defined according to the context of each organization.
These activities are necessary to create safe work environments and contribute to the protection of workers' health (Niu, 2010). This is highlighted by the actions arising from the Occupational Health and Safety Management System (OHSMS) registered in ISO 45001:2018, which aims to prevent injuries and health problems for workers by providing healthy and safe workplaces (International Organization for Standardization, 2018).
The performance of this type of systems needs to be constantly evaluated, being necessary the use of indicators (metrics to quantify the processes performance), which allow the identification of weaknesses and potentialities, consequently assisting in decision making and definition of strategies to improve working conditions.
According to the scientific literature, generic indicators have been used (Lingard et al., 2011; Payne et al., 2009; Williams & Carnahan, 2013), considering in a partial and isolated way the organization context. This can be measured based on indicators, such as the number of accidents at work, training activities, among others. It can be complemented with an indicator associated with the absenteeism days generated by an accident or occupational disease (Sgourou et al., 2010). However, the fact that generic and isolated indicators are used, may not represent the truth of the organization.
Thus, it was identified as a gap in the scientific literature due to the lack of specific, integrated, consistent and accurate indicators, which allow measuring the performance of prevention activities and that are identified by participatory decision makers according to the organization context.
The organizations, although they may have the same economic activity, have different physical characteristics, particular context, that is, multiple criteria that influence them (Ensslin et al., 2001). Using the same indicators to assess them could restrict the possibility of identifying the origin problems in prevention activities.
Based on this information, the aim of this research was to define the indicators to assess the performance of prevention activities of occupational health and safety, required in a hospital ward during the dispensing of medications to patients, through a constructivist multicriteria approach, with specific focus on personal protective equipment (PPE).
The research was carried out in a state public hospital dedicated to psychiatric and chemical dependency treatments, which provides clinical, psychological, and dental care, integrated with occupational therapy, physiotherapy, and specialized programs. Given the broad context derived from the multiple activities carried out in the hospital, this research is limited to the risk prevention analysis of the nursing staff during the use of PPE in the care and delivery of medication to patients.
To assess the performance of prevention activities in the use of PPE, the Multi-Criteria Decision Aid - Constructivist methodology (MCDA-C) (Ensslin et al., 2000) was applied, since this methodology allows the construction of specific indicators in the context of organization independent of economic activity, number of workers or geographic location. We identified that the MCDA-C has a replicable logical structure that ensures the validity, legitimacy, and effectiveness of a decision support model in a practical environment. The research delimits the construction of indicators, developing the Structuring Phase of the MCDA-C.
2. Research methodology
The research was characterized as exploratory, applied and case study, divided into two stages. At first, an exploratory literature that allowed the development of the theoretical framework was carried out, expanding the understanding of the central themes, using the Knowledge Development Process - Constructivist (ProKnow-C).
Secondly, the Multi-Criteria Decision Aid Constructivist methodology (MCDA-C) was applied, and the nurses working in inpatient mental health care during the supply of medicines to patients were taken as reference. Data was collected with nursing manager and chief of maintenance, using interviews, document analysis, in-place visits, and observations, allowing the construction of indicators with greater accuracy and reliability. Afterwards, the performance areas were defined, with the purpose of identifying the critical situations and subsequently the intervention strategies.
2.1. ProKnow-C
As mentioned, the theoretical framework of the research used the Knowledge Development Process - Constructivist (ProKnow-C), which is based on a constructivist approach, structured in a sequence of steps that help in the construction of the researchers' knowledge and the subsequent use, according to their boundaries and interests. In this process, the researchers worked out the selection criteria for: databases, keywords, time filters and the parameters for inclusion and exclusion of articles in the Bibliographic Portfolio (Ensslin et al., 2017; Valmorbida et al., 2016).
ProKnow-C is structured in four stages: (i) bibliographic selection of the portfolio; (ii) Bibliometrics; (iii) systemic analysis; and (iv) formulation of research questions and objectives found. This research was limited to the first stage of ProKnow-C.
The selection of the Bibliographic Portfolio involves (i) definition of the keywords; (ii) definition of the databases; (iii) search for articles in the databases selected based on defined keywords; and, (iv) keyword adherence test.
Interest area defined in the MCDA-C, Individual Protection Elements, originates the identification of the research axes and definition of the keywords (i). The defined search command is ((“Personal protective equipment”) AND (“Healthcare” OR “Health care” OR “hospital personnel”)).
The second moment, the definition of the databases (ii), they were: ProQuest, Scopus, Web of Science, Compendex, and ScienceDirect, for their adherence to the theme, as well as their alignment with the engineering area and the subarea of production engineering, with occupational health and safety and ergonomics as a complement.
The search process (iii) had as criteria: (a) articles published in scientific journals; (b) articles published since 2000; (c) search with keywords in the title, abstract and keywords of these from the databases; and, (d) articles published in the English language. The EndNote® X9 software (Thomson Corporation, 2013) was used to organize and filter the collected data.
2.2. Multi-Criteria Decision Aid Constructivist methodology (MCDA-C)
In step two, the MCDA-C methodology was applied. It involves the people responsible for decision-making in complex, conflicting and uncertain contexts in a participatory way (Dutra, 1998; Ensslin et al., 2010).
The MCDA-C has been used in several productive areas of industry as well as in academia. For example, in the performance evaluation of effective outsourcing of information technology where the authors presented a structured evaluation organized by 8 strategic objectives and 107 performance indicators; the strengths and weaknesses of the status quo; and a process diagram for generating improvements (Ensslin et al., 2020). In another paper, Matos et al. (2018) presented the analysis made in the internal audit department of a Brazilian Federal University where they developed a model that allows the manager to make decisions with transparency and consistency to define improvement actions. Likewise, Longaray et al. (2019) presented a case study of a fertilizer industry located in a Brazilian maritime port terminal, where they developed a customized performance assessment model to help the management process unique aspects of the company and the environment, improving the organizational management.
Decision making is systematized in three phases: (a) structuring the problem, (b) evaluating potential actions and (c) recommendations (Dutra, 1998; Ensslin et al., 2000, 2010). This paper presents the development of the MCDA-C structuring phase, which basically consists of three stages (i) contextualization (the soft approach to structuring); (ii) building a family of Fundamental Points of View (FPVs); and (iii) the construction of indicators (called descriptors in the figure of the MCDA-C phases) (Ensslin et al., 2000), showing in Figure 1.
The structuring phase begins with a meeting to identify the actors, with whom the Primary Assessment Elements (PAEs) are defined. These PAEs make it possible to establish the action-oriented Concepts, as well as to define the preferred or desired direction. Likewise, indicating a psychological opposite (minimum acceptable result) (Marafon et al., 2015). These Elements are grouped by large areas of similarity, also called areas of strategic concern or interest (represented by cognitive maps). Then, the Strategic Objectives linked to each of these Concepts were identified. For Strategic Objectives, Elementary Points of View (EPV) are identified, giving rise to the indicators. The ordinal scale is defined for each one, which allows measuring the level of compliance, and two reference levels are defined, which divide the scale into three possible levels of performance: compromising, market, and excellence. At the top level are the indicators with excellent performance, in the center are those with good or neutral results, at the bottom the performance is compromising. The strips are included to facilitate its visualization: green represents “excellent”, yellow represents “normal”, and red denotes “compromising”. When the indicators are qualified, the status quo (diagnosis of the current situation) of the organization can be known. Finally, to improve the performance level of each indicator, recommendations are defined with their action plans. This research presents the construction of indicators for the use of PPE by the nursing staff.
3. Background
Personal protective equipment (PPE) is a primary strategy to protect health care workers (HCW) from infectious diseases. This paper first addresses the characteristics and prevention activities related to PPE that should be used by professionals in the medical sector in charge of patient care in hospital environments. Second, the importance of having evaluation indicators in the prevention of risks for health professionals is described.
3.1. Personal Protective Equipment (PPE) and Health Care Workers (HCW)
The decision of which and where to use PPE is influenced both by risk perception and organizational, cognitive, and physical factors of the activity. Preventing the spread of infectious organisms must not only depend on the use of PPE but must also be recognized as an organizational responsibility.
First, the organization assesses the risks and defines the activities necessary to eliminate it. When it is not possible to eliminate risks, control and mitigation actions are implemented (Chia et al., 2005). Occupational risk prevention activities comprise a series of controls, one of which is the use of collective and individual protection equipment with a view to preserving the health and well-being of workers, consequently with positive effects on the quality and productivity of the systems. From an occupational health perspective, PPE should only be used when engineering and administrative controls are insufficient to adequately reduce exposures (Jones et al., 2020).
Personal Protective Equipment are protective devices that comprise a set of barriers to prevent workers from coming into contact with infectious agents or health deteriorating agents. Isolation or elimination of the source of risk should be the first preventive action, the use of Collective Protection Equipment (CPE) the second, and lastly the PPE.
In the case of direct contact with patients, PPE is considered essential to protect workers from a possible infection during care (Casanova et al., 2008; Gosch et al., 2013). And its proper use provides a secure barrier between the patient, medications and the HCW, preventing physical contact or actively filtering infectious particles in the air (Beam et al., 2011).
Safe climate practices and infection control (IC) (use of disposable gloves, needle recap practice, waste disposal practices, and disinfectant use) have a significant relationship. Green-McKenzie et al. (2001) identified that a strong security climate was twice as likely to rigorously comply with IC practices. This study set out to examine both the relationship between the availability of PPE and the provision of engineering controls on IC practices and the effect of commitment organizational with safety in IC practices. This research suggests that IC practices improve with the ready availability of PPE and provision of engineering controls. For example, this study identified that the use of eye protection and face shields increased when this protective equipment was available in the workplace. Likewise, Torp et al. (2005) defend how the availability of PPE has an impact on workers using them, as well as their physical properties.
Some studies have used contamination monitoring to examine PPE. They propose that infection control programs need to focus on quality techniques for using PPE (Beam et al., 2011).
For the prevention of occupational diseases, using PPE is not enough. It is necessary to choose the appropriate PPE, depending on the type of risk and occupational activity. For example, the guidance on the use of respiratory and face protection equipment presented by (Coia et al., 2013). This guidance supports the person in selecting and wearing the appropriate respiratory and face protection to minimize the risk of acquiring workplace infection. Likewise, the PPE must be stored in a location that meets the needs to preserve its quality. In addition to having PPE in the necessary amount, the correct use and disposal have an impact on protection. It is also important to have an adequate process for doffing (removal) and disposing of contaminated PPE (Chughtai et al., 2018; Kang et al., 2021).
To do the proper doffing it is necessary to train the HCW in doffing according to the PPE and emphasize the importance of hand hygiene. Due to possible contamination of the skin and clothing of health personnel, pathogens that generate potentially fatal diseases can be transmitted (Tomas et al., 2015; Williams & Carnahan, 2013). Gloves can result in confusion about how to properly integrate contact precautions. Removal of dry contact gloves with proper hand hygiene before and after patient contact can improve hand hygiene compliance and reduce the risk of contamination of the patient and their area with used gloves (Jain et al., 2018).
In fact, other research has been published on the contamination of the skin and clothing of healthcare professionals that frequently occurs while wearing contaminated gloves or aprons, reviewing training in the correct PPE removal process and how to involve immediate visual feedback on skin and clothing contamination. The research showed the effectiveness of the educational intervention in reducing contamination during the disposal of PPE (Tomas et al., 2016).
Another training technique that can be used is video capture and a powdered fluorescent marker in a simulated patient care environment. This allows for system innovations in HCW education in connection with the use of PPE (Beam et al., 2011). Also, it is imperative that infection control education includes a structured and effective way in the ongoing professional development of healthcare professionals. For example, talking about contagious diseases such as COVID-19, the Job Hazard Analysis (JHA) indicated that contamination of the torso, arms and hands is likely. Hand and body coverings should be used because, although SARS-CoV does not infect through the dermis, contaminated clothing and skin can serve as an environmental reservoir (Jones et al., 2020).
Therefore, knowing the importance of evaluating PPE protocols to minimize exchange contamination and identifying the high risk of contagion by COVID-19, research has been carried out with this emphasis. For example, regarding the use of PPE to prevent coronavirus disease, researchers conducted a survey with 2,711 HCW of intensive care units. The researchers wanted to know the difficulties of HCW after using PPE for more than 4 continuous hours. HCWs reported heat, thirst, areas of pressure, headache, difficulty going to the bathroom, severe exhaustion. Researchers confirmed the need for regular PPE training to avoid carelessness or forgetfulness of the trained HCW in the details of PPE use (Kang et al., 2021). Other commonly reported problems with the use of PPE were difficulty breathing, suffocation, heat stress and cloudy glass (Chughtai et al., 2018).
In the process of exchanging PPEs, different levels of contamination can also occur. Style combinations, use and exchange techniques, knowledge deficits and behavioral failures are also associated with contamination during doffing (Kang et al., 2017).
In addition, the organization must verify the effectiveness of the training, the training period to be carried out and the presence of standard procedures to perform the work (Steege et al., 2014). For example, to verify the effectiveness of training, a survey identified some reasons for non-compliance with the proper use of PPE: workload and staff levels, availability, perceived risk for health professionals, knowledge or lack of knowledge and loss of dexterity (gloves interfered with nurses' ability to provide care) (Hinkin et al., 2008; John et al., 2016). Likewise, they analyze the effectiveness of the use of PPE, reaffirming its importance to protect workers and patients in healthcare environments (John et al., 2016). These authors showed how a positive social climate, committed management and a well-developed health and safety management system can improve the results of preventive work through information on health risk and provision of PPE. Likewise, some studies show that the physical properties of PPE, health competence and the availability of PPE have an impact on the fact that workers use PPE when necessary or not (Torp et al., 2005). In addition, it is important to assess how PPE sets work in the field, where human behavior and individual PPE pieces influence protection (Jones et al., 2020).
While it may be impossible to achieve zero violations in the exchange of HCW PPE, the need to refine PPE protocols based on additional scientific evidence is highlighted, reinforcing PPE training using innovative methods, improving, and standardizing PPE equipment to achieve optimal use by HCW (Kang et al., 2017).
Within the training, aspects related to judgments related to care activities in which it is not expected to be contaminated and organizational factors must be considered. Even when PPE is used, it can often be incorrectly placed or removed, creating opportunities for transmission as well as self-contamination. Preventing the spread of infectious organisms should not only depend on the use of PPE, but rather be recognized as an organizational responsibility (Harrod et al., 2020).
Therefore, healthcare facilities must take the necessary precautions and change working conditions during the COVID-19 pandemic (e.g., regular breaks, shorter shifts, adequate supply of PPE, air conditioning, prophylactic dressing, better material, masks that fit properly and reduce the time of using PPE) to prevent adverse events associated with the use of PPE and minimize harm to HCW. Creating a safe working environment for health professionals can lead to better management of the COVID-19 pandemic and an increase in professional performance (Galanis et al., 2021).
Because there are several protocols for the use and disposal of PPE in hospital environments, it is important to validate their application to the reality of the organization, activities, physical resources, and the pathology of patients (Chughtai et al., 2018).
The use of PPE is not an isolated activity. It is part of a set of activities to prevent and control occupational hazards. This set is called the Occupational Health and Safety Management System (OHSMS).
3.2. Performance Measurement (PM) and prevention activities
Performance Evaluation seeks to measure business effectiveness and efficiency (Cavalluzzo & Ittner, 2004; LeRoux & Wright, 2010), having two basic functions: (i) assist managers to validate the results expected by management; and (ii) provide the means to achieve the strategic objectives established by the organization (Behn, 2003; Neely, 2005; Tapinos et al., 2005).
PM starts from the principle of raising and understanding the real characteristics of the organization, considering its organizational and managerial complexity, organizational environment, culture, values, competition between units and sectors, as well as economic and political conditions, guaranteeing the information legitimacy. Thus, a legitimate assessment is one that seeks: (i) what will be assessed - knowing the assessment object; (ii) how it will be carried out - identifying how each objective will be assessed and how much it contributed to the overall assessment, allowing the performance profile identification of the assessed object; and (iii) how internal management will be conducted - based on the analysis of identified strengths and weaknesses, promoting the organizational performance improvement through improvement actions (Ensslin et al., 2000; Igarashi et al., 2007).
For this process to add organizations value, it is necessary to have the continuous support of managers, since they are the key to defining and enabling actions (LeRoux & Wright, 2010). The consideration of the distinct specific purposes (assess, control, invest, motivate, promote, celebrate, learn, improve) of the organization is also necessary for a correct performance measurement, since each organization has its strategic objectives, therefore, it needs to create specific measures appropriate for each purpose, and the generic indicators is not recommended (Behn, 2003). Performance indicator is the PM structural element, being the control form, evaluation, and verification of conformity, which can be considered as a tool to aid decision making (Dutra et al., 2015).
For example, some studies include leading and lagging indicators to measure aspects related to prevention activities carried out by workers to reduce accidents or illnesses at work (Payne et al., 2009). In turn, Podgórski (2015) selected a set of 20 main indicators and 14 alternative indicators, referring to the activities count carried out in the OHSMS (for example, Percentage of workers participating in OSH refresher courses, Percentage of workers trained in the OHSMS themes). As an example, some Leading and Lagging indicators are shown in Table 1.
It was found that, for the most part, systems are developed based on legislation and outside the organization, as shown in Steege et al. (2014), Hinkin et al. (2008) and John et al. (2016). In them, generic performance indicators were used to measure, for example, aspects referring to the use of PPE: number of PPE used, accident frequency rates, workers who received training, inspections performed, training activities performed. The research by Steege et al. (2014) used indicators to measure the specific training for hazards, the employer's safe handling procedures and the use of safe handling guidelines, among others.
In PM of organization, its mission, vision, culture and organizational strategy must be considered (Kaplan & Norton, 2000), communication (feedback) (Škrinjar et al., 2008), as well as the parties involved (Garengo et al., 2005), internals (employees, administrators, owners, etc.) (Jacobsen et al., 2005), or externals (suppliers, partners, society, government, customers, unions, financial groups, investors, competition, environment, distributors, marketing, suppliers, scientific and academic community, among others) (Chenhall & Langfield-Smith, 2007; Tuomela, 2005).
In addition, the scientific literature has addressed the importance of linking strategic objectives (set of specific actions and decisions) to operational objectives (desired situations that the company seeks to have), as well as linking individual objectives to the general objectives of organizations (Garengo et al., 2005; Kaplan & Norton, 2000). These characteristics make each organization unique, even though they may have the same product or service. For these reasons, its performance should be measured by indicators adjusted to the particularities and not just by generic indicators (for example, occurrence of incidents, activity, diseases number, frequency, hours of training, problems solved, problems identified).
The use of indicators to measure security activities has been gaining strength to inform the situation of the OHSMS and define strategies. In this context, Reiman & Pietikäinen (2012) indicated the need to define safety performance indicators depending on the current management focus or the orientation intended by the organization. That is, the indicators must be defined in the organization's own safety model, instead of being adopted from literature, competitors or auditors and consultants external to the organization.
To know the management system state, it is necessary to measure the activities that compose it. Thus, the implementation of safety performance assessment in an organization must begin with the identification of the current safety management practices premises and this includes the construction of indicators. Bearing in mind that an indicator seeks information on a subject of interest, it can be constructed considering quantitative or qualitative measures. Safety indicators show the current state or development of the main organizational functions, processes, and technical infrastructure of a system.
This PM paper focuses on risk prevention activities at work, specifically the use of PPE in the hospital environment. The criteria for constructing indicators applying the MCDA-C methodology will allow the organization to recognize and measure its specific criteria. Thus, the key aspects of PPE, addressed by different authors, will be confronted when measuring. The use of the MCDA-C in the construction of indicators for the use of PPE seeks to contribute to the control of occupational risks. When it comes to activities in patient health care area, the use of PPE generates well-being for both the worker and the patient. And to ensure that this protection is adequate, it is necessary to evaluate its performance. In the review of the scientific literature of the indicators used to measure PPE, the use of generic indicators that may not meet the specific needs of an organization was identified. Therefore, applying a constructivist methodology to build them can be an effective tool to measure the actual level of occupational risk control activities and define improvement actions.
4. Case study
The second phase of this research used the MCDA-C, delimiting itself to the structuring phase, as detailed in the research methodology item, characterized as a case study. This application was performed in the infirmary sector of a Psychiatric Hospital, located in the state of Santa Catarina.
It is a state public hospital, subordinated to the State Health Department of Santa Catarina (SES-SC), dedicated to psychiatric and chemical dependency treatments. It was inaugurated in 1941, over the years it has been improving its activities and organizational structure. In 1995, following the guidelines of the WHO, the Ministry of Health, and the SES, it was divided into two service areas: the Santana Living Center (SLC) and Psychiatric Inpatient Units (PIUs). The PIUs have the chemical detox area, provide care for patients with acute outbreaks and short-term hospitalization wards. The SLC, on the other hand, has a participatory management unit providing care to long-term dependent and semi-dependent patients. The Psychiatric Hospital also has outpatient care for consultation and laboratory services. Providing clinical, psychological, and dental care, integrated with occupational therapy, physiotherapy, and specialized programs. The hospital's structure is divided into nine integrated areas: institutional, exams, care/treatment, men's wards, women's wards, mixed wards, coexistence, general services, and food. In the work team, there are 347 active professional servants, of which 27 are nurses, 141 Nursing Technicians/Auxiliaries and a Nursing Attendant.
The research focus was defined in the supply of medicines to patients in the nursing staff, a sector in which the use of PPE is necessary. The workflow can be summarized in the following activities: transcribing the prescription issued in the system by the doctor; pick up medicines from the pharmacy (daily); check the medication records to administer during the day; take the drugs to the corresponding ward, perform temporal storage; prepare medications, and dispense them to patients. The nursing staff is responsible for carrying out other patient care activities in the different Hospital areas, but they were not considered in this research.
According to MCDA-C, in the first stage of the Structuring Phase, the actors in this process were identified, that is, those who participate, directly or indirectly, in the process of building the indicators. The actors are identified in Table 2.
In the sequence, the family of Fundamental Points of View (FPVs) or Hierarchical Value Structure was defined, with the participation of the decision maker in the Primary Assessment Elements (PAEs) identification, based on concerns, desired results, unwanted goals, limitations, restrictions, potential factors, among other aspects of the values inherent to management. Table 3 shows the 18 PAEs identified, with 5 referring to PPE, the focus of this research.
These 18 PEAs identified allowed us to understand that the activity of supplying medicines to patients had varied variables and influences, resulting in a total of 82 Concepts. Considering the scope of this research, Table 4 presents the 6 concepts referring to PPE.
Four Fundamental Points of View (FPV) were determined: Standard Work Procedures – SWPs, Personal Protection Equipment - PPEs, Occupational Health Control Programs – OHCP, and Occupational Health and Safety – OHS, enabling, with the information collected, the definition of the Hierarchical Value Structure, shown in Figure 2.
In addition, the concepts were grouped according to their relationships in the Fundamental Points of View (Figure 3).
After confirming the classification of all concepts in the four FPV, we proceed to define the cognitive maps. The Figure 4 presents the cognitive maps of the Fundamental Points of View, with their concepts.
In Figure 5, the PPE cognitive map is presented, relating each FPV to the previously defined concepts.
In this map, four clusters were identified: (1) related to aspects of the “Procedure”; (2) related to the “Training”; (3) related to “Replacement” and (4) related to “Effectiveness”, shown in Figure 6. For each one, the Elementary Points of View (PVE) that will give rise to the indicators were identified.
The cognitive map allowed to define the Hierarchical Value Structure (Figure 7), and with them the constructed indicators. Identifying the possible ergonomics and human factors that can intervene in prevention activities. Environmental, organizational and work issues that influence work behavior in ways that affect health and safety.
Finally, for each PVE one indicator was constructed including the objective, measurement scale and reference levels. As an example, a fragment of the indicators is shown in Figure 8.
Thus, the Structuring Phase was concluded, with the construction of indicators associated with PPE, to be measured in the next item.
5. Results and discussion
As part of the overall management strategy, performance measures should be used to assess, control, budget, motivate, promote, celebrate, learn, and improve, and should not be used as a single measure of performance. Instead, the managerial purposes to which performance measurement can contribute must be considered, and how, subsequently, define and implement as improvement measures (Behn, 2003).
Considering the criteria for using performance measures, when the causes that generated critical situations were identified, the indicator and ordinal scales for each PVE were defined, being numerical and percentage scales. For the nursing staff, 126 indicators were constructed, 26 of them related to PPE, recorded in Table 5.
The PVE analysis process allowed to identify the human factors that can interfere with the performance of the worker's activities. The presence of general and individual human factors and characteristics that influence behavior at work in such a way as to affect health and safety, the use of PPE or compliance with other standards. This identification allowed for the construction of indicators for follow-up on several aspects. For example, it was considered important to carry out activities to control, the prevention information registered in the SWPs. Validate the inclusion of risk identification, the necessary prevention activities, and the PPE at each stage of the activity.
Likewise, it was considered important to validate the quality of the use of the PPE, in which it must be verified that the employee carried out the fit testing or fit checking of PPE, at the time of wearing the PPE, to ensure its effectiveness in the prevent contagion risks.
On the other hand, in the accident investigation identified the need to analyze how the use of PPE influenced the contagion of some pathology. Seeking to deepen the investigation of the accident root cause, the possible presence of errors in the resolution of critical situations, validate the existence or not of prevention procedures, as well as the definition proposals for new solutions.
Other indicators were associated with the assessment of training quality and availability (to avoid absences both in the training journey and patient care). In evaluating the effectiveness of training, the decision maker considered it important to include the removal characteristics of PPE.
5.1. Measurement of indicators
The measurement of the constructed indicators to assess the use of PPE, mentioned in Table 5, is shown in Figure 9. The result of each indicator marks a point on the corresponding arrow. When joined by lines, it is possible to visualize the behavior of the indicators. The bottom part of the figure shows the qualification of the initial status (Status quo) of the PPE indicators.
Indicators are essential to clarify the meaning of PVEs, to make them more understandable and to avoid ambiguities. It was defined what the behavior of each indicator should be for the organization, establishing the level of efficiency good, acceptable, or bad for each one.
Finally, recommendations for improving the result of each indicator were identified. As a result of implementing the recommended actions, it will be possible to move from a compromising level to a competitive level and then reach the level considered to be of excellence.
The effective use of PPE is considered essential to protect workers and patients in health care settings (John et al., 2016). For example, to reinforce the organization's commitment in this regard, the indicator “Ensuring that employees make correct use of PPE” was built. When measured, this result was at a compromising level.
In total, 48 recommendations were defined, of which 27 correspond to actions so that the 12 indicators in the compromising range (red) improve the result until reaching the level of excellence (green). Another 12 indicators were in the yellow range (Figure 9) for having their value within the values considered normal. Likewise, 19 actions were defined to improve performance. Finally, for the 2 indicators which results were in the range of excellence (replacement policy and stocks), an action plan was defined to preserve their performance.
As an example, and because it is considered of high relevance by the decision maker, the characterization of the proposed recommendations for “employees who use PPE properly during occupational activities (validate the fit checking of PPE)” (indicator 7 on the list) is shown in Table 6.
The application of the methodology with the direct participation of the decision-maker allows it to be part of the critical aspect’s identification of prevention activities, their possible causes and the definition of strategies to improve performance. In some cases, the actions seem obvious, but it is important to consider that when people are immersed in their work environment, they often do not perceive them, and consequently these situations start to become routine and without real importance.
In addition, the organization can use performance indicators and their measurement to motivate and promote improvement measures for each PVE.
6. Conclusion
Based on the scientific literature selected, it was possible to conclude that there are limitations in the use of performance measurement systems for management and decision making in organizations. It was found that in most studies, organizations use generic indicators without considering the particularities of the organization.
With the MCDA-C methodology it was possible to construct specific indicators for the organization's context. Evidencing that a participative and articulated process with the actors of the organization, especially with the decision-maker, allows the understanding of the aspects related to the prevention activities in the use of Personal Protective Equipment in the sector studied. The Elementary Points of View, their generating causes were also identified, which consequently allowed constructing the specific indicators for the organizational context, and finally the definition of strategies through an action plan with recommendations so that the indicators with normal performance or compromising, could aspire to reach the level of excellence. The model constructions can be considered comprehensive in its development, but practical and with indicators built according to the organization's context, making easy to apply by managers.
Another relevant aspect identified in the analyzed research informs that most of them did not involve decision-makers in the process, unlike this survey, with the active participation of the organization's internal actor (manager-decision maker) it was possible to obtain accurate and real information, resulting in reliable indicators and aligned to the needs of the organizational context. Likewise, the decision-maker considered the process useful, helping to understand the problem, the consequent development of the steps that ended with the definition of strategies to improve the OHSMS, which, according to the nursing manager (decision maker), would not be possible if the measurement depended exclusively on generic performance indicators, such as the number of PPE used, accident frequency rates, number of workers who received training or how many inspections were carried out.
This study made it possible to strengthen scientific and practical knowledge according to the constructivist trajectory, with the adopted view meeting the scientific and practical requirements to ensure the validity, legitimacy, and effectiveness of a model to support decision-making in a practical environment.
This research generated different contributions, among them: (i) Theoretical, since the research provides a current and consistent review on the use of PPE in the health care sector and safety performance measurement; (ii) Practice, for the theme of the Occupational Health and Safety Management System (OHSMS), proposing a structured process for performance measurement; (iii) Applied to the health sector, through the MCDA-C in a case study carried out in the nurses working in inpatient mental health care during the supply of medicines to patients.
It is suggested for future research: (i) transform the ordinal scales into cardinal scales, to select the best option for advancement in the OHSMS, visualizing the consequences of the implementation of each action. The development of phase II of the MCDA-C will allow for a detailed analysis of possible improvement proposals applying the MACBETH software; (ii) Monitor the management activity provided by the built model. The managers of the psychiatric hospital will be able to periodically analyze the advances in the defined action plans and establish the relevant changes; (iii) Replicate the proposal to another organization, developing a model for the occupational health and safety management system. Given the flexibility of MCDA-C, it can be applied in several organizations, allowing for the strengthening of improvement decisions based on a structured methodology. (iv) Create a simplified model to build performance indicators.
Acknowledgements
The authors would like to thank the nurses who work in the care and supply of medicines to patients admitted to the Santa Catarina Institute of Psychiatry (IPq-SC). This study was funded by the Coordination for the Improvement of Higher Education Personnel (CAPES) Brazil - Financing Code 001.
References
-
Beam, E. L., Gibbs, S. G., Boulter, K. C., Beckerdite, M. E., & Smith, P. W. (2011). A method for evaluating health care workers’ personal protective equipment technique. American Journal of Infection Control, 39(5), 415-420. http://dx.doi.org/10.1016/j.ajic.2010.07.009 PMid:21255874.
» http://dx.doi.org/10.1016/j.ajic.2010.07.009 -
Behn, R. D. (2003). Why measure performance? Different purposes require different measures. Public Administration Review, 63(5), 586-606. http://dx.doi.org/10.1111/1540-6210.00322
» http://dx.doi.org/10.1111/1540-6210.00322 -
Casanova, L., Alfano-Sobsey, E., Rutala, W. A., Weber, D. J., & Sobsey, M. (2008). Virus transfer from personal protective equipment to healthcare employees’ skin and clothing. Emerging Infectious Diseases, 14(8), 1291-1293. http://dx.doi.org/10.3201/eid1408.080085 PMid:18680659.
» http://dx.doi.org/10.3201/eid1408.080085 -
Cavalluzzo, K. S., & Ittner, C. D. (2004). Implementing performance measurement innovations: evidence from government. Accounting, Organizations and Society, 29(3-4), 243-267. http://dx.doi.org/10.1016/S0361-3682(03)00013-8
» http://dx.doi.org/10.1016/S0361-3682(03)00013-8 -
Chenhall, R. H., & Langfield-Smith, K. (2007). Multiple perspectives of performance measures. European Management Journal, 25(4), 266-282. http://dx.doi.org/10.1016/j.emj.2007.06.001
» http://dx.doi.org/10.1016/j.emj.2007.06.001 -
Chia, S. E., Koh, D., Fones, C., Qian, F., Ng, V., Tan, B. H., Wong, K. S., Chew, W. M., Tang, H. K., Ng, W., Muttakin, Z., Emmanuel, S., Fong, N. P., Koh, G., & Lim, M. K. (2005). Appropriate use of personal protective equipment among healthcare workers in public sector hospitals and primary healthcare polyclinics during the SARS outbreak in Singapore. Occupational and Environmental Medicine, 62(7), 473-477. http://dx.doi.org/10.1136/oem.2004.015024
» http://dx.doi.org/10.1136/oem.2004.015024 -
Chughtai, A. A., Chen, X., & Macintyre, C. R. (2018). Risk of self-contamination during doffing of personal protective equipment. American Journal of Infection Control, 46(12), 1329-1334. http://dx.doi.org/10.1016/j.ajic.2018.06.003 PMid:30029796.
» http://dx.doi.org/10.1016/j.ajic.2018.06.003 -
Coia, J. E., Ritchie, L., Adisesh, A., Makison Booth, C., Bradley, C., Bunyan, D., Carson, G., Fry, C., Hoffman, P., Jenkins, D., Phin, N., Taylor, B., Nguyen-Van-Tam, J. S., & Zuckerman, M. (2013). Guidance on the use of respiratory and facial protection equipment. The Journal of Hospital Infection, 85(3), 170-182. http://dx.doi.org/10.1016/j.jhin.2013.06.020 PMid:24051190.
» http://dx.doi.org/10.1016/j.jhin.2013.06.020 - Dutra, A. (1998). Elaboração de um sistema de avaliação de desempenho dos recursos humanos da Secretaria de Estado da Administração - SEA à luz da metodologia multicritério de apoio à decisão (Dissertação de mestrado). Centro Tecnológico, Universidade Federal de Santa Catarina, Florianópolis.
-
Dutra, A., Ensslin, S. R., Ensslin, L., Ripoll-Feliu, V. M., & Fillol, A. G. (2015). The construction of knowledge from the scientific literature about the theme seaport performance evaluation. International Journal of Productivity and Performance Management, 64(2), 243-269. http://dx.doi.org/10.1108/IJPPM-01-2014-0015
» http://dx.doi.org/10.1108/IJPPM-01-2014-0015 -
Elsler, D., Takala, J., & Remes, J. (2017). An international comparison of the cost of work-related accidents and illnesses. Brussels: European Agency for Safety and Health at Work. Retrieved in 2020, October 21, from https://osha.europa.eu/en/tools-and-publications/publications/international-comparison-cost-work-related-accidents-and/view
» https://osha.europa.eu/en/tools-and-publications/publications/international-comparison-cost-work-related-accidents-and/view -
Ensslin, L., Dutra, A., & Ensslin, S. R. (2000). MCDA: a constructivist approach to the management of human resources at a governmental agency. International Transactions in Operational Research, 7(1), 79-100. http://dx.doi.org/10.1111/j.1475-3995.2000.tb00186.x
» http://dx.doi.org/10.1111/j.1475-3995.2000.tb00186.x -
Ensslin, L., Ensslin, S., Dutra, A., Nunes, N., & Reis, C. (2017). BPM governance: a literature analysis of performance evaluation. Business Process Management Journal, 23(1), 71-86. http://dx.doi.org/10.1108/BPMJ-11-2015-0159
» http://dx.doi.org/10.1108/BPMJ-11-2015-0159 -
Ensslin, L., Giffhorn, E., Ensslin, S., Petri, S., & Vianna, W. (2010). Avaliação do desempenho de empresas terceirizadas com o uso da metodologia multicritério de apoio à decisão-construtivista. Pesquisa Operacional, 30(1), 125-152. http://dx.doi.org/10.1590/S0101-74382010000100007
» http://dx.doi.org/10.1590/S0101-74382010000100007 - Ensslin, L., Montibeller Neto, G., & Noronha, S. M. (2001). Apoio à decisão: metodologia para estruturação de problemas e avaliação multicritério de alternativas (296 p.). Florianópolis: Insular.
-
Ensslin, L., Mussi, C. C., Dutra, A., Ensslin, S. R., & Demetrio, S. N. (2020). Management support model for information technology outsourcing. Journal of Global Information Management, 28(3), 123-147. http://dx.doi.org/10.4018/JGIM.2020070107
» http://dx.doi.org/10.4018/JGIM.2020070107 -
Galanis, P., Vraka, I., Fragkou, D., Bilali, A., & Kaitelidou, D. (2021). Impact of personal protective equipment use on health care workers’ physical health during the COVID-19 pandemic: a systematic review and meta-analysis. American Journal of Infection Control http://dx.doi.org/10.1016/j.ajic.2021.04.084 PMid:33965463.
» http://dx.doi.org/10.1016/j.ajic.2021.04.084 -
Garengo, P., Biazzo, S., & Bititci, U. S. (2005). Performance measurement systems in SMEs: a review for a research agenda. International Journal of Management Reviews, 7(1), 25-47. http://dx.doi.org/10.1111/j.1468-2370.2005.00105.x
» http://dx.doi.org/10.1111/j.1468-2370.2005.00105.x -
Gosch, M. E., Shaffer, R. E., Eagan, A. E., Roberge, R. J., Davey, V. J., & Radonovich, L. J. Jr (2013). B95: a new respirator for health care personnel. American Journal of Infection Control, 41(12), 1224-1230. http://dx.doi.org/10.1016/j.ajic.2013.03.293 PMid:23726655.
» http://dx.doi.org/10.1016/j.ajic.2013.03.293 -
Green-McKenzie, J., Gershon, R. R. M., & Karkashian, C. (2001). Infection control practices among correctional healthcare workers: effect of management attitudes and availability of protective equipment and engineering controls. Infection Control and Hospital Epidemiology, 22(9), 555-559. http://dx.doi.org/10.1086/501951 PMid:11732784.
» http://dx.doi.org/10.1086/501951 -
Harrod, M., Weston, L. E., Gregory, L., Petersen, L., Mayer, J., Drews, F. A., & Krein, S. L. (2020). A qualitative study of factors affecting personal protective equipment use among health care personnel. American Journal of Infection Control, 48(4), 410-415. http://dx.doi.org/10.1016/j.ajic.2019.08.031 PMid:31610895.
» http://dx.doi.org/10.1016/j.ajic.2019.08.031 -
Hinkin, J., Gammon, J., & Cutter, J. (2008). Review of personal protection equipment used in practice. British Journal of Community Nursing, 13(1), 14-19. http://dx.doi.org/10.12968/bjcn.2008.13.1.27978 PMid:18399366.
» http://dx.doi.org/10.12968/bjcn.2008.13.1.27978 -
Igarashi, D. C. C., Paladini, E. P., & Ensslin, S. R. (2007). A metodologia multicritério de apoio à decisão construtivista como subsídio para o gerenciamento interno: estudo de caso no PPGC/UFSC. Organizações & Sociedade, 14(42), 133-149. http://dx.doi.org/10.1590/S1984-92302007000300008
» http://dx.doi.org/10.1590/S1984-92302007000300008 - International Labour Organization – ILO. (2019). Safety and health at the heart of the future of work: building on 100 years of experience Geneva: ILO. Retrieved in 2020, October 21, from www.ilo.org/labadmin-osh
- International Organization for Standardization – ISO. (2018). ISO 45001:2018: occupational health and safety management systems: requirements with guidance for use (47 p.). Geneva: ISO.
-
Jacobsen, K., Hofman-Bang, P., & Nordby Junior, R. (2005). The IC Rating? model by Intellectual Capital Sweden. Journal of Intellectual Capital, 6(4), 570-587. http://dx.doi.org/10.1108/14691930510628834
» http://dx.doi.org/10.1108/14691930510628834 -
Jain, S., Clezy, K., & McLaws, M. L. (2018). Safe removal of gloves from contact precautions: the role of hand hygiene. American Journal of Infection Control, 46(7), 764-767. http://dx.doi.org/10.1016/j.ajic.2018.01.013 PMid:29519650.
» http://dx.doi.org/10.1016/j.ajic.2018.01.013 -
John, A., Tomas, M. E., Cadnum, J. L., Mana, T. S. C., Jencson, A., Shaikh, A., Zabarsky, T. F., Wilson, B. M., & Donskey, C. J. (2016). Are health care personnel trained in correct use of personal protective equipment? American Journal of Infection Control, 44(7), 840-842. http://dx.doi.org/10.1016/j.ajic.2016.03.031 PMid:27181222.
» http://dx.doi.org/10.1016/j.ajic.2016.03.031 -
Jones, R. M., Bleasdale, S. C., Maita, D., & Brosseau, L. M. (2020). A systematic risk-based strategy to select personal protective equipment for infectious diseases. American Journal of Infection Control, 48(1), 46-51. http://dx.doi.org/10.1016/j.ajic.2019.06.023 PMid:31358421.
» http://dx.doi.org/10.1016/j.ajic.2019.06.023 -
Kang, J., Kim, E. J., Choi, J. H., Hong, H. K., Han, S. H., Choi, I. S., Kim, J., Kim, J. Y., Park, E. S., & Choe, P. G. (2021). Minimizing contamination in the use of personal protective equipment: simulation results through tracking contamination and enhanced protocols. American Journal of Infection Control, 49(6), 713-720. http://dx.doi.org/10.1016/j.ajic.2020.11.002 PMid:33159995.
» http://dx.doi.org/10.1016/j.ajic.2020.11.002 -
Kang, J., O’Donnell, J. M., Colaianne, B., Bircher, N., Ren, D., & Smith, K. J. (2017). Use of personal protective equipment among health care personnel: results of clinical observations and simulations. American Journal of Infection Control, 45(1), 17-23. http://dx.doi.org/10.1016/j.ajic.2016.08.011 PMid:28065328.
» http://dx.doi.org/10.1016/j.ajic.2016.08.011 - Kaplan, R. S., & Norton, D. P. (2000). Having trouble with your strategy? Then map it. Harvard Business Review, 78(5), 167-176, 202. PMid:11143152.
-
LeRoux, K., & Wright, N. S. (2010). Does performance measurement improve strategic decision making? Findings from a national survey of nonprofit social service agencies. Nonprofit and Voluntary Sector Quarterly, 39(4), 571-587. http://dx.doi.org/10.1177/0899764009359942
» http://dx.doi.org/10.1177/0899764009359942 -
Lingard, H., Wakefield, R., & Cashin, P. (2011). The development and testing of a hierarchical measure of project OHS performance. Engineering, Construction, and Architectural Management, 18(1), 30-49. http://dx.doi.org/10.1108/09699981111098676
» http://dx.doi.org/10.1108/09699981111098676 -
Longaray, A. A., Ensslin, L., Dutra, A., Ensslin, S., Brasil, R., & Munhoz, P. (2019). Using MCDA-C to assess the organizational performance of industries operating at Brazilian maritime port terminals. Operations Research Perspectives, 6, 100109. http://dx.doi.org/10.1016/j.orp.2019.100109
» http://dx.doi.org/10.1016/j.orp.2019.100109 -
Marafon, A. D., Ensslin, L., Lacerda, R. T. de O., & Ensslin, S. R. (2015). The effectiveness of multi-criteria decision aid methodology. European Journal of Innovation Management, 18(1), 86-109. http://dx.doi.org/10.1108/EJIM-10-2013-0106
» http://dx.doi.org/10.1108/EJIM-10-2013-0106 -
Matos, L. S., Iesbik Valmorbida, S. M., & Rolim Ensslin, S. (2018). Gestión y evaluación de la monitorización en el sector de auditoría interna de una Universidad Federal Brasileña: una propuesta constructivista. Revista Electrónica Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 3(16), 67-85. http://dx.doi.org/10.15366/reice2018.16.3.004
» http://dx.doi.org/10.15366/reice2018.16.3.004 -
Neely, A. (2005). The evolution of performance measurement research: developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12), 1264-1277. http://dx.doi.org/10.1108/01443570510633648
» http://dx.doi.org/10.1108/01443570510633648 -
Niu, S. (2010). Ergonomics and occupational safety and health: an ILO perspective. Applied Ergonomics, 41(6), 744-753. http://dx.doi.org/10.1016/j.apergo.2010.03.004 PMid:20347066.
» http://dx.doi.org/10.1016/j.apergo.2010.03.004 -
Payne, S. C., Bergman, M. E., Beus, J. M., Rodríguez, J. M., & Henning, J. B. (2009). Safety climate: leading or lagging indicator of safety outcomes? Journal of Loss Prevention in the Process Industries, 22(6), 735-739. http://dx.doi.org/10.1016/j.jlp.2009.07.017
» http://dx.doi.org/10.1016/j.jlp.2009.07.017 -
Podgórski, D. (2015). Measuring operational performance of OSH management system: a demonstration of AHP-based selection of leading key performance indicators. Safety Science, 73, 146-166. http://dx.doi.org/10.1016/j.ssci.2014.11.018
» http://dx.doi.org/10.1016/j.ssci.2014.11.018 -
Reiman, T., & Pietikäinen, E. (2012). Leading indicators of system safety: monitoring and driving the organizational safety potential. Safety Science, 50(10), 1993-2000. http://dx.doi.org/10.1016/j.ssci.2011.07.015
» http://dx.doi.org/10.1016/j.ssci.2011.07.015 -
Sgourou, E., Katsakiori, P., Goutsos, S., & Manatakis, E. (2010). Assessment of selected safety performance evaluation methods in regards to their conceptual, methodological and practical characteristics. Safety Science, 48(8), 1019-1025. http://dx.doi.org/10.1016/j.ssci.2009.11.001
» http://dx.doi.org/10.1016/j.ssci.2009.11.001 -
Škrinjar, R., Bosilj-Vuksic, V., & Indiharstemberger, M. (2008). The impact of business process orientation on financial and non-financial performance. Business Process Management Journal, 14(5), 738-754. http://dx.doi.org/10.1108/14637150810903084
» http://dx.doi.org/10.1108/14637150810903084 -
Steege, A. L., Boiano, J. M., & Sweeney, M. H. (2014). NIOSH health and safety practices survey of healthcare workers: training and awareness of employer safety procedures. American Journal of Industrial Medicine, 57(6), 640-652. http://dx.doi.org/10.1002/ajim.22305 PMid:24549581.
» http://dx.doi.org/10.1002/ajim.22305 -
Tapinos, E., Dyson, R. G., & Meadows, M. (2005). The impact of the performance measurement systems in setting the “direction” in the University of Warwick. Production Planning and Control, 16(2), 189-198. http://dx.doi.org/10.1080/09537280512331333084
» http://dx.doi.org/10.1080/09537280512331333084 - Thomson Corporation. (2013). EndNote X7 Stamford: Thomson Corporation.
-
Tomas, M. E., Cadnum, J. L., Mana, T. S. C., Jencson, A. L., Koganti, S., Alhmidi, H., Kundrapu, S., Sunkesula, V. C. K., & Donskey, C. J. (2016). Utility of a novel reflective marker visualized by flash photography for assessment of personnel contamination during removal of personal protective equipment. Infection Control and Hospital Epidemiology, 37(6), 711-713. http://dx.doi.org/10.1017/ice.2016.44 PMid:26976219.
» http://dx.doi.org/10.1017/ice.2016.44 -
Tomas, M. E., Kundrapu, S., Thota, P., Sunkesula, V. C., Cadnum, J. L., Mana, T. S. C., Jencson, A., O’Donnell, M., Zabarsky, T. F., Hecker, M. T., Ray, A. J., Wilson, B. M., & Donskey, C. J. (2015). Contamination of health care personnel during removal of personal protective equipment. The Journal of Emergency Medicine, 175, 1904-1910. http://dx.doi.org/10.1016/j.jemermed.2016.01.009 PMid:26457544.
» http://dx.doi.org/10.1016/j.jemermed.2016.01.009 -
Torp, S., Grogaard, J. B., Moen, B. E., & Bratveit, M. (2005). The impact of social and organizational factors on workers’ use of personal protective equipment: a multilevel approach. Journal of Occupational and Environmental Medicine, 47(8), 829-837. http://dx.doi.org/10.1097/01.jom.0000167275.13079.8d PMid:16093933.
» http://dx.doi.org/10.1097/01.jom.0000167275.13079.8d -
Tuomela, T. S. (2005). The interplay of different levers of control: a case study of introducing a new performance measurement system. Management Accounting Research, 16(3), 293-320. http://dx.doi.org/10.1016/j.mar.2005.06.003
» http://dx.doi.org/10.1016/j.mar.2005.06.003 -
Valmorbida, S. M., Ensslin, S., Ensslin, L., & Ripoll-Feliu, V. (2016). Rankings universitários mundiais: que dizem os estudos internacionais? REICE. Revista Electrónica Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 14(2), 1-25. http://dx.doi.org/10.15366/reice2016.14.2.001
» http://dx.doi.org/10.15366/reice2016.14.2.001 -
Williams, C. K., & Carnahan, H. (2013). Development and validation of tools for assessing use of personal protective equipment in health care. American Journal of Infection Control, 41(1), 28-32. http://dx.doi.org/10.1016/j.ajic.2012.01.027 PMid:22704736.
» http://dx.doi.org/10.1016/j.ajic.2012.01.027
Publication Dates
-
Publication in this collection
01 Oct 2021 -
Date of issue
2021
History
-
Received
21 Oct 2020 -
Accepted
24 Aug 2021
