Usability perception of the health information systems in Brazil: the view of hospital health professionals on the electronic health record

Abstract

Purpose

The purpose of this paper is to validate and measure the overall evaluation of electronic health record (EHR) and identify the factors that influence the health information systems (HIS) assessment in Brazil.

Design/methodology/approach

From February to May 2020, this study surveyed 262 doctors and nurses who work in hospitals and use the EHR in their workplace. This study validated the National Usability-focused HIS Scale (NuHISS) to measure usability in the Brazilian context.

Findings

The results showed adequate validity and reliability, validating the NuHISS in the Brazilian context. The survey showed that 38.9% of users rated the system as high quality. Technical quality, ease of use and benefits explained 43.5% of the user’s overall system evaluation.

Research limitations/implications

This study validated the items that measure usability of health-care systems and identified that not all usability items impact the overall evaluation of the EHR.

Practical implications

NuHISS can be a valuable tool to measure HIS usability for doctors and nurses and monitor health systems’ long-term usability among health professionals. The results suggest dissatisfaction with the usability of HIS systems, specifically the EHR in hospital units. For this reason, those responsible for health systems must observe usability. This tool enables usability monitoring to highlight information system deficiencies for public managers. Furthermore, the government can create and develop actions to improve the existing tools to support health professionals.

Social implications

From the scale validation, public managers could monitor and develop actions to foster the system’s usability, especially the system’s technical qualities – the factor that impacted the overall system evaluation.

Originality/value

To the best of the authors’ knowledge, this study is the first to validate the usability scale of EHR systems in Brazil. The results showed dissatisfaction with HIS and identified the factors that most influence the system evaluation.

Keywords
Usability; Health information system; Electronic medical record; Electronic health record; National Usability-focused HIS Scale (NuHISS)

1. Introduction

The adoption of health information systems (HIS) is growing worldwide, but professionals’ satisfaction with the usability of these systems is not improving (Gomes & Ratwani, 2019). Nevertheless, despite the difficulties intrinsic to implementing HIS, predominantly electronic health record (EHR), the use of information technology represents advances in the quality of health and patient safety (Feldman et al., 2018; Kaipio et al., 2020).

The International Organization for Standardization (ISO) described that “usability refers to an extent to which a system, product or service can be used by specified users to achieve specific goals with effectiveness, efficiency, and satisfaction in a specified context of use” (ISO, 2019). As a result, usability is often remembered as a barrier to accepting these technologies after adoption (Holden, 2011; Walter & Lopez, 2008), even though it is fundamental to optimize the benefits (BE) of using EHR (Kaipio et al., 2017).

The ever-changing difficulties in using HIS are a significant source of work stress for doctors. They reported usability problems, system failures and lack of integration between systems that barely support patient documentation and data recovery (Heponiemi et al., 2018, 2019; Mazur et al., 2019; Melnick et al., 2020; Roman et al., 2017). Additionally, low levels of satisfaction related to usability result in physician dissatisfaction and exhaustion at work, reducing efficiency and having consequences for patient safety (Howe et al., 2018; Roman et al., 2017).

One way to improve the usability of systems is to involve doctors with new technologies to avoid human error, ensure data integrity (Lawrence et al., 2019) and improve the interoperability and stability of these systems (Vainiomäki et al., 2017). In addition, a greater focus of clinical end users during product design and development and optimized certification requirements is necessary to improve usability (Gomes & Ratwani, 2019). Also, it is crucial to provide physicians with sufficient time and support in their problems, learning and updating related to HIS (Heponiemi et al., 2018).

Few instruments measuring such systems’ usability were validated in the literature, such as Bundschuh et al. (2011) in Germany and Hyppönen et al. (2019a, 2019b) in Finland. Thus, this study aims to validate, measure the overall system evaluation and identify the impacts of HIS evaluation in Brazil. We hope the results will highlight aspects of the HIS that encourage greater engagement by professionals in the field. In addition, we seek to provide an instrument that allows more research in Brazil since the instrument has never been validated in the Brazilian context.

2. Related research

The National Usability-focused HIS Scale (NuHISS) is a scale developed and validated by Hyppönen et al. (2019a, 2019b) in Finland, which includes seven factors: technical quality (TQ), information quality (IQ), feedback (FB), ease of use (EoU), BE, internal collaboration (IC) and inter-organizational collaboration. The authors considered it a valuable tool to measure HIS usability; moreover, Kaipio et al. (2020) used four out of the seven scale factors (TQ, EoU, BE and collaboration) to compare the perception of doctors and nurses regarding usability, also in Finland.

Previous studies have also presented these factors with an impact on usability evaluation. EoU was associated with the easy, fast and practical entry of data into the system, resembling professionals’ routine tasks, without additional steps that may generate rework (Castillo et al., 2010; Miller & Sim, 2004) and complexity of the systems (Boonstra & Broekhuis, 2010; Singh et al., 2020). Thus, usability was negatively associated with many screens for navigation in the systems (Boonstra & Broekhuis, 2010; Hudson et al., 2018; Miller & Sim, 2004) and difficulties of use (Singh et al., 2020; Topaz et al., 2016).

Studies have also highlighted the TQ of the HIS, which reflects the response time and crashes of the systems (Hudson et al., 2018; Miller & Sim, 2004; Ratwani et al., 2018) and positively impacts usability. Furthermore, user-friendly systems are a characteristic of TQ and are recognized as beneficial for workers’ professional and personal well-being (Heponiemi et al., 2019). On the other hand, limited systems that offer nothing more than the routine of professionals (Boonstra & Broekhuis, 2010) negatively impact the usability of HIS.

IQ is another factor that influences the system’s usability, as it can make it more challenging to use and result in rework, manual release and translation of digital paper records (Miller & Sim, 2004; Viitanen et al., 2011). Also, the lack of a summary view of the patient’s health status, prevention of errors associated with the medication request and a list of patient medication, for example, are not presented smoothly to the user (Hudson et al., 2018; Kaipio et al., 2017). Moreover, the delay in entering data may take more time to attend to a patient (Boonstra & Broekhuis, 2010).

The BE the system offers to the user may influence the usability perception of HIS. For example, some studies have reported BE such as safety for the patient and professionals, quality in the service provided, efficiency and effectiveness in care and integration with other tools (Castillo et al., 2010; Fennelly et al., 2020; Singh et al., 2020).

FB is related to users’ acceptance and implementation of suggestions for HIS developers (Boonstra & Broekhuis, 2010; Heponiemi et al., 2019). In addition, although doctors and nurses are willing to participate in developing HIS, appropriate methods for effectively including them and their FB seem to be lacking or underused (Martikainen et al., 2020). Finally, inter-organizational collaboration and IC are recalled in studies when HIS do not integrate with other systems or equipment to facilitate automatic data entry (Boonstra & Broekhuis, 2010; Castillo et al., 2010; Miller & Sim, 2004; Topaz et al., 2016; Viitanen et al., 2011) or when they do not offer or facilitate collaboration between professionals (Castillo et al., 2010; Kaipio et al., 2017; Larsen et al., 2018; Viitanen et al., 2011).

The positive relationship between usability and the overall evaluation of information systems merge in the literature as long as the six described usability factors facilitate and benefit information systems users. However, which factors most influence the overall evaluation of the system? This question can be answered from the hypotheses presented in the proposed theoretical model illustrated in Figure 1.

Figure 1.
Proposed theoretical model

3. Context of the study

The unified national health system (SUS, in Portuguese) is provided free to the entire Brazilian population. Brazil has approximately 210 million inhabitants, 5,570 municipalities and is the fifth largest country globally [Instituto Brasileiro de Geografia e Estatística (IBGE), 2010]. The health system is divided into public (SUS), in which services are financed and provided by the government at the federal, state and municipal levels, and private (profit and non-profit), in which services are financed in various ways with public or private funds (Paim et al., 2011). The consequence is that there are numerous information systems concerning health, generating complexity and making it difficult to monitor their usability.

Implementing digital health systems in Brazil is recent. In 2018, the Federal Government of Brazil enacted the digital health record law. It provided the digitization and use of computerized systems for the safekeeping, storage and handling of patient records (Brasil, 2018). In 2019, a secretariat of primary health care in the ministry of health was created, with new challenges for the federal management of SUS, among them the expansion of computerization of health units and the use of electronic records (Sellera et al., 2020).

Studies involving Brazilian health professionals analyzed their perceptions about the contribution of EHR. In comparison with paper records, EHRs have higher quality and safety (Colleti Junior et al., 2018). Besides, EHR reduces the number of papers filed and conduct errors, stores data for longer and avoids the redundancy of procedures. Furthermore, it increases service productivity and user satisfaction, facilitates intercommunication at points of attention and eliminates duplicate records in the lists of registered users (Gonçalves et al., 2013; Pinto & Santos, 2020; Silva et al., 2019; Vaidotas et al., 2019). However, difficulties in using these systems in Brazil are also a challenge given the high cost of implementation and the need to train professionals to improve usability (Gonçalves et al., 2013).

4. Material and methods

This research has a quantitative approach, operationalized through a survey applied to Brazilian doctors and nurses who use the EHR in hospitals. To measure usability, we used the 28-item HIS scale (NuHISS) (Hyppönen et al., 2019a, 2019b), which include TQ (five items), IQ (five items), FB (three items), EoU (seven items), BE (six items) and IC (two items). Cross-organizational collaboration was excluded from this study because it practically does not exist in Brazilian associations. In other words, some systems have integration modules between institutions but no collaboration modules. In addition, we used a five-point Likert scale, ranging from “1 – totally dissatisfied” to “5 – totally satisfied”. Finally, the study’s dependent variable was the overall evaluation of the systems, measured on a scale from 0 to 10, being “1. I am totally dissatisfied” to “10. I am totally satisfied”.

We collected gender, age and experience data under the Finnish study, whose questionnaire we translated into Portuguese and validated with two health specialists. In addition, a pilot test was conducted with 35 professionals, presenting relevant results. ^{Appendix 1} presents this questionnaire.

Data collection occurred between February and May 2020. Participants were invited to the research through social networks and e-mails sent to postgraduate programs in health. After filtering the data through tests of univariate and multivariate outliers (Z and Mahalanobis scores), the sample resulted in 262 valid cases of doctors and nurses working in hospitals and using the EHR in their workplace.

We selected respondents from all regions of Brazil, mainly Southeast (42.0%) and South (33.2%). The respondents work in public hospitals (57.6%), private hospitals (11.1%) and both (31.3%). They use different brands of EHR. Also, 60.3% work only in one hospital, 27.1% in two hospitals and 12.6% in three or more hospitals. The level of education is high: 65.6% have a stricto sensu education (master, doctorate or post-doctorate). Most are female (64.5%), married (61.8%) and are over 30 years (76.7%). The respondents’ characteristics are presented in Table 1.

Table 1.
Respondents’ characteristics

Characteristics	n	(%)
Gender
Female	169	64.5
Male	93	35.5
Age group
< 30 years	61	23.3
31–40 years	79	30.2
41–50 years	63	24.0
51–60 years	42	16.0
> 60 years	17	6.5
Number of hospitals the professional works at
Only one	158	60.3
Two	71	27.1
Three or more	33	12.6
Type of hospital
Public	151	57.6
Private	29	11.1
Both	82	31.3
Experience of use of the EHR
< 10 years	109	41.6
10–20	60	22.9
20–30	53	20.2
>30 years	40	15.3

Source:

Statistical analyses were performed with programming language R (version 3.5.1)/R-studio. When applicable, the chi-square test or Fisher’s exact test were used to compare categorical variables and t-tests to compare groups. Multiple statistical R packages (corrplot, psych, lavaan) were used for the statistical analyses. Besides, we performed a confirmatory factor analysis to evaluate the quality of the instrument and multiple regression to examine the proposed hypotheses. The statistical significance was determined as p < 0.05.

Confirmatory factor analyses validated the relationships between measured variables and latent constructs through the model fit indexes. The goodness of fit of the SEM model was χ², evaluated based on the chi-square test (RMSEA, CFI, TLI). A non-significant chi-square value indicates that the model fits the data well. RMSEA value of less than 0.05 suggests a good fit and 0.08 suggests a reasonable fit. For CFI and TLI, values above 0.90 represent an acceptable fit (Kline, 2015). The structural model was tested following the model already validated by Hyppönen et al. (2019a, 2019b) in two steps. First, a complete model was estimated in which all items were loaded in the same underlying dimension (null model). In the second stage, items with low commonality or loadings were removed from the model (QT2, QT5 and the IQ factor), looking for an appropriate fit. The same fit indexes were used as in the general SEM test.

4.1 Correlations

Figure 2 presents a Pearson correlation matrix. We can observe the relationship between items that compose the same dimension. For example, FB and BE have the strongest correlations, considering the correlations of items in the same dimension. EoU items also have high correlations in the dimension. On the other hand, IQ items do not correlate with each other or with items of other dimensions. In general, the correlations in Brazil are lower than those obtained by Hyppönen et al. (2019a, 2019b) but have similarities in the most correlated items, with FB and BE being the strongest in both Finland and Brazil.

Figure 2.
Correlation matrix between items

5. Results

5.1 Evaluation of the structural model

The exploratory factor analysis with Varimax rotation is adequate for the data (KMO = 0.88). All items presented measure of sampling adequacy above 0.8, considered excellent (above 0.5 is already satisfactory) (Hair et al., 2010). Bartlett’s test, conducted in R, is significant (p < 0.001). The significance of this test attests that the correlation matrix is not an identity matrix. Hence, we assumed there is some relationship between the variables. However, the commonality analysis showed low values for the items TQ2 (system errors), TQ5 (missing info), IQ1 (medic list quality), IQ2 (summary view), IQ3 (patient-provided information), EoU4 (operating information) and EoU7 (nursing record).

We tested the model with the six dimensions validated by Hyppönen et al. (2019a, 2019b), applying confirmatory factor analysis. It is noteworthy that there were seven dimensions in the original study, but the cross-organizational collaboration dimension was not tested in Brazil. This model, called Model 1, showed a reasonable fit (χ²/df = 2.88, CFI = 0.832, TLI = 0.809, NFI = 0.766, IFI = 0.834, RMSEA = 0.085 and SRMR = 0.070).

The two strongest factors (measured by the loadings of items) were FB and IC – all the item loadings were over 0.8. All items in EoU and BE factors had factor loadings of over 0.5. We observed items with small factor loadings in IQ (0.38 for IQ3 – patient-provided information) and TQ (0.36 for TQ2 – system errors and 0.37 for TQ5 – missing information). These very items had low factor loadings in the original study (after removing these items, the fit parameters improved: χ²/df = 3.29, CFI = 0.837, TLI = 0.810, NFI = 0.783, IFI = 0.839, RMSEA = 0.093 and SRMR = 0.070). Although the values are still off-limits recommended by the literature, the results are better than those found by Hyppönen et al. (2019a, 2019b). Therefore, the model refinement is necessary, as shown below.

After removing the mentioned items, the internal reliability assessed with the final factors data alpha coefficients showed that the IQ factor presented the smallest internal reliability, below the one suggested by Hair et al. (2010), as observed in Table 2. This result is similar to Hyppönen et al. (2019a, 2019b), who found low alphas for this dimension in the two data collections. In Brazil, the factor with the highest loadings and alpha was IC, suggesting that, in this context, it is the strongest factor. This result differs from the Finnish study, which found this factor to be the last in the ranking presented by the authors, suggesting that it is the weakest.

Table 2.
Discriminant validity

	TechQual	InfoQual	FeedB	EaseU	Benef	InterCollab
Technical quality	0.519
Information quality	0.475	0.331
Feedback	0.360	0.700	0.690
Ease of use	0.648	0.758	0.578	0.434
Benefits	0.416	0.676	0.408	0.595	0.534
Internal collaboration	0.378	0.557	0.326	0.568	0.583	0.837

• Note: Cronbach’s alpha – TQ = 0.754; IQ = 0.592; FE = 0.869; EoU = 0.836; BE = 0.878; IC = 0.909.
Source:

Table 2 also shows the correlations between the dimensions in the primary cells. In the diagonal position, the table shows the AVE values. The IQ dimension presented discriminant validity issues, as the square root of the diagonal should be greater than the correlation between it and the other dimensions (Fornell & Larcker, 1981). Also, the AVE value for IQ was very low, indicating that the proportion of the items variance explained by the construct was deficient.

We excluded the IQ dimension from the final model and built the relationships based on the five resulting dimensions. The correlation of 0.84 between items BE1 (IS help improve quality of care) and BE2 (IS help ensure continuity of care) indicated a lack of discrimination between them. The same was observed between EoU1 (the arrangement of fields and functions is logical on the computer screen) and EoU2 (terminology on the screen is clear and understandable, e.g. titles and labels), with a 0.74 correlation. After these corrections, the final model showed a good fit (χ²/df = 2.57, CFI = 0.909, TLI = 0.892, NFI = 0.860, IFI = 0.910, RMSEA = 0.077 and SRMR = 0.070). Figure 3 presents the standardized loadings. The loadings of each measure indicate whether the relationships are stronger or weaker within each dimension.

Figure 3.
Confirmatory factor analysis

5.2 Overall system evaluation

Users assessed the overall system evaluation by assigning scores from 0 to 10. The scale was dichotomized into low (7 or less) and high (more than 7) quality estimates, in line with the study by Hyppönen et al. (2019a, 2019b). The results showed that the overall system evaluation was 6.7, with a standard deviation of 1.87. Half of the professionals evaluated the system with scores below 7, showing that part of the professionals is dissatisfied with EHR. Thus, the survey indicated that 38.9% of users rated the system they used as a high-quality one. The high standard deviation highlights differences in users’ perceptions.

Considering the dimensions assessment separately, professionals rated the systems using a five-point scale, ranging from very dissatisfied (one point) to very satisfied (five points). Based on the averages, three groups could be defined: the best-evaluated dimensions (group a), the intermediate ones (group b) and the low satisfaction ones (group c). Among these three groups, the differences are significant. IC and BE were the dimensions with the highest satisfaction, not differing from each other (paired t-test, p = 0.859). In the sequence, TQ and EoU showed no difference either (p = 0.526). In group c, there are aspects related to the system FB.

For comparison with the 1–10 scale, we converted the values using linear interpolation. The results indicated averages between 6.365 and 3.495 for the aspects, as seen in the last column in Table 3, highlighting the low ratings of the system usability factors (i.e. below 7).

Table 3.
Satisfaction with factors

	Mean (five-point scale)	SD	Median	Min	Max	Mean (0–10 scale)
Internal collaboration	3.546a	1.137	3.500	1	5	6.365
Benefits	3.534a	0.937	3.667	1	5	6.335
Technical quality	3.383b	0.793	3.400	1	5	5.957
Ease of use	3.352b	0.798	3.429	1	5	5.880
Feedback	2.398c	1.010	2.333	1	5	3.495

• Different letters indicate significant differences (p < 0.05, paired t-test)
Source:

5.3 Impact of dimensions on the overall system evaluation

We used the overall system evaluation information as a dependent variable in the relationship model. The results of the multiple regression indicated a significant effect of TQ (p < 0.001), EoU (p = 0.001) and BE (p = 0.018) on the overall system evaluation. On the other hand, FB and IC did not impact the overall system evaluation (Table 4).

Table 4.
Regression results

	Estimate	Std. err	t-value	p-value
Technical quality	0.724	0.111	6.540	<0.001
Ease of use	0.523	0.156	3.347	0.001
Benefits	0.278	0.117	2.381	0.018
Internal collaboration	0.136	0.095	1.425	0.155
Feedback	0.143	0.102	1.402	0.162

Source:

The significance values of TQ, EoU and BE indicated that these aspects positively and significantly affect the overall system evaluation. That is to say, the higher these items assessments, the greater the users’ satisfaction.

The coefficient of determination (adjusted R²) is used to observe how the model formed can explain the current conditions. Our model R² value is 0.435, which means that TQ, EoU and BE explained 43.5% of the users’ overall system evaluation. Figure 4 presents the final model.

Figure 4.
Final model

6. Discussion and conclusions

The results showed that some of the chosen scale (NuHISS) (Hyppönen et al., 2019a, 2019b) factors have been validated (now in Brazil), such as TQ, EoU, BE, FB and IC. However, IQ has not been validated either in Brazil or Finland. Given this, we consider that NuHISS can be a valuable tool to measure HIS usability for doctors and nurses and monitor the long-term usability of health systems among health professionals.

The scale validity represents the degree to which a test measures what it claims to measure. The correlation test revealed the grouping of items, although the correlations between some dimensions were stronger than others.

The discriminant validity analysis suggested eliminating the IQ dimension, reflecting the availability and format of crucial information types in the EHR system. As in the study by Hyppönen et al. (2019a, 2019b), the values associated with this dimension presented discrimination and internal consistency problems, with the lowest Cronbach’s alpha value. Considering that the scale had already gone through previous validations outside Brazil, we opted for confirmatory factor analysis, which presented a good fit for five factors: IC, BE, TQ, EoU and FB.

The results suggest that health-care professionals do not perceive the support of EHR information for decision-making regarding patient care, and even the users’ lack of understanding may impair trust in EHR information. Therefore, we suggest public managers develop actions to promote the use of the system and engage these professionals to raise awareness about IQ and the resources that the EHR offers to assist in decision-making.

Regarding the items, we observed high loadings for IC and FB. However, in general, all items had satisfactory loadings on dimensions, evidencing their reliability for replication in future research.

We evaluated the internal consistency of factors based on reliability (Cronbach’s alpha). The constructs with the highest reliability were IC (0.909), BE (0.878) and FB (0.869). Values above 0.6 are recommended in the literature (Hair et al., 2010). The IQ factor was the weakest (0.592), in the boundary zone stated in the literature. Furthermore, we excluded IQ from the final validation because we considered only the factors that presented scale reliability (over 0.60).

The final fit of the confirmatory model showed acceptable values, with CFI and IFI values higher than 0.9 and TLI and NFI very close to 0.9. As for the RMSEA and SRMR measures, both were below 0.08. These results suggest that the scale is appropriate for the Brazilian context in future studies.

The survey indicated that 38.9% of users rated the system they used as high quality. This result is similar to the Finnish study by Hyppönen et al. (2019a, 2019b), who identified that 33% classified the system they used as high quality. These results converge with studies that comment that professionals’ satisfaction with the usability of these systems is not improving (Gomes & Ratwani, 2019). Because of this dissatisfaction, physicians present exhaustion at work, which may reduce work efficiency and have consequences for patient safety (Howe et al., 2018; Roman et al., 2017). The problem is that if physicians do not have a system that enables them to achieve efficiency, effectiveness and satisfaction, they may seek alternative solutions, like using paper to document and transfer health information. Therefore, when the system’s BE are not perceived, the very decision of refusing to use the system can indicate this problem (Hyppönen et al., 2019a, 2019b).

Our results also identified that the TQ, EoU and BE impacted the overall system evaluation, whereas FB and IC did not. Specifically, TQ, EoU and BE explained 43.5% of users’ overall system evaluation. This finding indicates that TQ, response time and system crashes (Hudson et al., 2018; Miller & Sim, 2004; Ratwani et al., 2018), in addition to EoU, are factors that enhance the service provided and qualify the care provided by health professionals. Similarly, the BE of HIS also influence the overall system evaluation, whether by data safety for the patient and professional, quality in the service provided, efficiency and effectiveness in care and integration with other tools (Castillo et al., 2010; Fennelly et al., 2020; Singh et al., 2020).

When we observe that three factors explain 43.5% of the overall system evaluation, we realize that 56.5% of the other factors may influence this evaluation and were not considered in the model proposed in this study. Therefore, it would be interesting to investigate the other factors that explain the overall evaluation of the EHR – an opportunity for further studies.

FB and IC did not influence the overall system evaluation. These factors result from user participation, i.e. FB is related to the acceptance and implementation of suggestions by users for HIS developers (Boonstra & Broekhuis, 2010; Heponiemi et al., 2019), whereas IC occurs when there is ease of collaboration between professionals (Castillo et al., 2010; Kaipio et al., 2017; Larsen et al., 2018; Viitanen et al., 2011).

Perhaps this is why IQ did not validate in the sample surveyed since the insertion of the information in the HIS depends on the user’s active participation. If the user does not notice that everyone works the same way, this impacts usability, as it can result in rework, manual entry and translation of paper records into digital (Miller & Sim, 2004; Viitanen et al., 2011).

TQ was the aspect that influenced the overall system evaluation most. However, when presenting satisfaction with the systems, this aspect is in the intermediate position, indicating the need for improvement, especially about stability in terms of technical functionality (no crash, no downtime), quick response to data entry and no loss of information or documents.

Our results showed it is important to reevaluate the TQ of the EHR, as it is the most representative factor of satisfaction and may influence the employees’ engagement with the use of the system and, consequently, the quality of the information provided. We emphasize that the quality of the information provided is closely related to the quality of the information received by the physician. With the correct information, professionals may be more assertive in the diagnosis and treatment of patients. Therefore, the quality of the system demands investment by the federal government.

The IC between doctors and nurses is one of the best-evaluated aspects, but its impact on the overall satisfaction of the system is not significant. This may occur because respondents understand ICs as personal activities that are not significant for the success of the EHR system.

NuHISS can be a valuable tool to measure the usability of HIS for doctors and nurses and monitor the long-term usability of health systems among health professionals. In addition, this tool enables usability monitoring to highlight information system deficiencies for public managers. As a result, the government can create and develop actions to improve the existing tools to support health professionals.

Furthermore, the results suggest dissatisfaction with the usability of the HIS, specifically the EHR in hospital units. For this reason, usability is a factor that those responsible for health systems must observe. This study is the first to validate the usability scale of EHR systems in Brazil. The results showed dissatisfaction with HIS and identified the factors that influence their overall evaluation most.

The study presented limitations because of the territorial extension and the complexity of many information systems in Brazil. We identified this limitation when observing the model fit indexes. Although adequate, the fit indexes were not as good as expected, possibly because of the variability between different public and private systems. The sample size was also small when compared to the original Finnish study.

Given the limitations, we suggest replicating the research in other countries, with other information systems and other health structures, to confirm the usability of HIS and consolidate the NuHISS. As a result, researchers will be able to identify all the specificities of the scale and discriminate the items or factors influenced by the context than those that are not.

• This study was financed by National Council for Scientific and Technological Development (CNPq) and Foundation to Support Research in the State of Rio Grande do Sul (FAPERGS) (No. 88887.176213/2018-00).

References

• Boonstra, A., & Broekhuis, M. (2010). Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research, 10(1), 231, doi: 10.1186/1472-6963-10-231.
  » https://doi.org/10.1186/1472-6963-10-231
• Brasil. (2018). Lei n° 13.787, de 27 de dezembro de 2018. Retrieved from www.in.gov.br/materia/-/asset_publisher/Kujrw0TZC2Mb/content/id/57221806/do1-2018-12-28-lei-n-13-787-de-27-de-dezembro-de-2018-57221499
  » www.in.gov.br/materia/-/asset_publisher/Kujrw0TZC2Mb/content/id/57221806/do1-2018-12-28-lei-n-13-787-de-27-de-dezembro-de-2018-57221499
• Bundschuh, B. B., Majeed, R. W., Bürkle, T., Kuhn, K., Sax, U., Seggewies, C., & Röhrig, R. (2011). Quality of human-computer interaction – results of a national usability survey of hospital-IT in Germany. BMC Medical Informatics and Decision Making, 11(1), 69, doi: 10.1186/1472-6947-11-69.
  » https://doi.org/10.1186/1472-6947-11-69
• Castillo, V. H., Martínez-García, A. I., & Pulido, J. (2010). A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review. BMC Medical Informatics and Decision Making, 10(1), 60, doi: 10.1186/1472-6947-10-60.
  » https://doi.org/10.1186/1472-6947-10-60
• Colleti Junior, J., de Andrade, A. B., & Carvalho, W. B. D. (2018). Evaluation of the use of electronic medical record systems in Brazilian intensive care units. Revista Brasileira de Terapia Intensiva, 30(3), 338–346, doi: 10.5935/0103-507X.20180057.
  » https://doi.org/10.5935/0103-507X.20180057
• Feldman, S. S., Buchalter, S., & Hayes, L. W. (2018). Health information technology in healthcare quality and patient safety: literature review. JMIR Medical Informatics, 6(2), e10264, doi: 10.2196/10264.
  » https://doi.org/10.2196/10264
• Fennelly, O., Cunningham, C., Grogan, L., Cronin, H., O'Shea, C., Roche, M. & O'Hare, N. (2020). Successfully implementing a national electronic health record: a rapid umbrella review. International Journal of Medical Informatics, 144(1), 104281, doi: 10.1016/j.ijmedinf.2020.104281.
  » https://doi.org/10.1016/j.ijmedinf.2020.104281
• Fornell, C., & Larcker, D. F. (1981). Structural Equation Models with Unobservable Variables and Measurement Error: Algebra and Statistics, Los Angeles, CA: SAGE Publications.
• Gomes, K. M., & Ratwani, R. M. (2019). Evaluating improvements and shortcomings in clinician satisfaction with electronic health record usability. JAMA Network Open, 2(12), e1916651, doi: 10.1001/jamanetworkopen.2019.16651.
  » https://doi.org/10.1001/jamanetworkopen.2019.16651
• Gonçalves, J. P. P., Batista, L. R., Carvalho, L. M., Oliveira, M. P., Moreira, K. S., & Leite, M. T. D S. (2013). Prontuário Eletrônico: uma ferramenta que pode contribuir para a integração das Redes de Atenção à Saúde. Saúde Em Debate, 37(96), 43–50, doi: 10.1590/S0103-11042013000100006.
  » https://doi.org/10.1590/S0103-11042013000100006
• Hair, J. F., Jr, Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis, 7th ed., Englewood Cliffs, NJ: Prentice Hall.
• Heponiemi, T., Hyppönen, H., Kujala, S., Aalto, A.-M., Vehko, T., Vänskä, J., & Elovainio, M. (2018). Predictors of physicians’ stress related to information systems: a nine-year follow-up survey study. BMC Health Services Research, 18(1), 284, doi: 10.1186/s12913-018-3094-x.
  » https://doi.org/10.1186/s12913-018-3094-x
• Heponiemi, T., Kujala, S., Vainiomäki, S., Vehko, T., Lääveri, T., Vänskä, J., … Hyppönen, H. (2019). Usability factors associated with physicians’ distress and information system–related stress: cross-sectional survey. JMIR Medical Informatics, 7(4), doi: 10.2196/13466. e13466.
  » https://doi.org/10.2196/13466
• Holden, R. J. (2011). What stands in the way of technology-mediated patient safety improvements? A study of facilitators and barriers to physicians’ use of electronic health records. Journal of Patient Safety, 7(4), 193–203, doi: 10.1097/PTS.0b013e3182388cfa.
  » https://doi.org/10.1097/PTS.0b013e3182388cfa
• Howe, J. L., Adams, K. T., Hettinger, A. Z., & Ratwani, R. M. (2018). Electronic health record usability issues and potential contribution to patient harm. JAMA, 319(12), 1276, doi: 10.1001/jama.2018.1171.
  » https://doi.org/10.1001/jama.2018.1171
• Hudson, D., Kushniruk, A., Borycki, E., & Zuege, D. J. (2018). Physician satisfaction with a critical care clinical information system using a multimethod evaluation of usability. International Journal of Medical Informatics, 112(1), 131–136, doi: 10.1016/j.ijmedinf.2018.01.010.
  » https://doi.org/10.1016/j.ijmedinf.2018.01.010
• Hyppönen, H., Kaipio, J., Heponiemi, T., Lääveri, T., Aalto, A.-M., Vänskä, J., & Elovainio, M. (2019a). Developing the national usability-focused health information system scale for physicians: validation study. Journal of Medical Internet Research, 21(5), e12875, doi: 10.2196/12875.
  » https://doi.org/10.2196/12875
• Hyppönen, H., Lumme, S., Reponen, J., Vänskä, J., Kaipio, J., Heponiemi, T., & Lääveri, T. (2019b). Health information exchange in Finland: usage of different access types and predictors of paper use. International Journal of Medical Informatics, 122, 1–6, doi: 10.1016/j.ijmedinf.2018.11.005.
  » https://doi.org/10.1016/j.ijmedinf.2018.11.005
• Instituto Brasileiro de Geografia e Estatística (IBGE). (2010). Censo 2010. Retrieved from https://censo2010.ibge.gov.br/
  » https://censo2010.ibge.gov.br/
• ISO. (2019). ISO 9241-210:2019 - Ergonomics of human-system interaction – part 210: human-centred design for interactive systems. Retrieved from www.iso.org/standard/77520.html
  » www.iso.org/standard/77520.html
• Kaipio, J., Kuusisto, A., Hyppönen, H., Heponiemi, T., & Lääveri, T. (2020). Physicians’ and nurses’ experiences on EHR usability: comparison between the professional groups by employment sector and system brand. International Journal of Medical Informatics, 134(1), 104018, doi: 10.1016/j.ijmedinf.2019.104018.
  » https://doi.org/10.1016/j.ijmedinf.2019.104018
• Kaipio, J., Lääveri, T., Hyppönen, H., Vainiomäki, S., Reponen, J., Kushniruk, A. & Vänskä, J. (2017). Usability problems do not heal by themselves: national survey on physicians’ experiences with EHRs in Finland. International Journal of Medical Informatics, 97(1), 266–281, doi: 10.1016/j.ijmedinf.2016.10.010.
  » https://doi.org/10.1016/j.ijmedinf.2016.10.010
• Kline, R. B. (2015). Principles and Practice of Structural Equation Modeling, New York, NY: Guilford publications.
• Larsen, E., Fong, A., Wernz, C., & Ratwani, R. M. (2018). Implications of electronic health record downtime: an analysis of patient safety event reports. Journal of the American Medical Informatics Association, 25(2), 187–191, doi: 10.1093/jamia/ocx057.
  » https://doi.org/10.1093/jamia/ocx057
• Lawrence, J. E., Cundall-Curry, D., Stewart, M. E., Fountain, D. M., & Gooding, C. R. (2019). The use of an electronic health record system reduces errors in the National Hip Fracture Database. Age and Ageing, 48(2), 285–290, doi: 10.1093/ageing/afy177.
  » https://doi.org/10.1093/ageing/afy177
• Martikainen, S., Kaipio, J., & Lääveri, T. (2020). End-user participation in health information systems (HIS) development: physicians’ and nurses’ experiences. International Journal of Medical Informatics, 137(1), 104117, doi: 10.1016/j.ijmedinf.2020.104117.
  » https://doi.org/10.1016/j.ijmedinf.2020.104117
• Mazur, L. M., Mosaly, P. R., Moore, C., & Marks, L. (2019). Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Network Open, 2(4), doi: 10.1001/jamanetworkopen.2019.1709. e191709.
  » https://doi.org/10.1001/jamanetworkopen.2019.1709
• Melnick, E. R., Dyrbye, L. N., Sinsky, C. A., Trockel, M., West, C. P., Nedelec, L. & Shanafelt, T. (2020). The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clinic Proceedings, 95(3), 476–487, doi: 10.1016/j.mayocp.2019.09.024.
  » https://doi.org/10.1016/j.mayocp.2019.09.024
• Miller, R. H., & Sim, I. (2004). Physicians’ use of electronic medical records: barriers and solutions. Health Affairs, 23(2), 116–126, doi: 10.1377/hlthaff.23.2.116.
  » https://doi.org/10.1377/hlthaff.23.2.116
• Paim, J., Travassos, C., Almeida, C., Bahia, L., & Macinko, J. (2011). O sistema de saúde brasileiro: história, avanços e desafios. In Saúde No Brasil, 1, 11–31, doi: 10.1016/S0140- 6736(11)60054-8.
  » https://doi.org/10.1016/S0140- 6736(11)60054-8
• Pinto, L. F., & Santos, L. J. D. (2020). Prontuários eletrônicos na Atenção Primária: gestão de cadastros duplicados e contribuição para estudos epidemiológicos. Ciência & Saúde Coletiva, 25(4), 1305–1312, doi: 10.1590/1413-81232020254.34132019.
  » https://doi.org/10.1590/1413-81232020254.34132019
• Ratwani, R. M., Savage, E., Will, A., Arnold, R., Khairat, S., Miller, K. & Hettinger, A. Z. (2018). A usability and safety analysis of electronic health records: a multi-center study. Journal of the American Medical Informatics Association, 25(9), 1197–1201, doi: 10.1093/jamia/ocy088.
  » https://doi.org/10.1093/jamia/ocy088
• Roman, L. C., Ancker, J. S., Johnson, S. B., & Senathirajah, Y. (2017). Navigation in the electronic health record: a review of the safety and usability literature. Journal of Biomedical Informatics, 67(1), 69–79, doi: 10.1016/j.jbi.2017.01.005.
  » https://doi.org/10.1016/j.jbi.2017.01.005
• Sellera, P. E. G., Pedebos, L. A., Harzheim, E., de Medeiros, O. L., Ramos, L. G., Martins, C., & D’Avila, O. P. (2020). Monitoramento e avaliação dos atributos da Atenção Primária à Saúde em nível nacional: novos desafios. Ciência & Saúde Coletiva, 25(4), 1401–1412, doi: 10.1590/1413-81232020254.36942019.
  » https://doi.org/10.1590/1413-81232020254.36942019
• Silva, A. B., Guedes, A. C. C. M., Síndico, S. R. F., Vieira, E. T. R. C., & Filha, I. G. D A. (2019). Registro eletrônico de saúde em hospital de alta complexidade: um relato sobre o processo de implementação na perspectiva da telessaúde. Ciência & Saúde Coletiva, 24(3), 1133–1142, doi: 10.1590/1413-81232018243.05982017.
  » https://doi.org/10.1590/1413-81232018243.05982017
• Singh, A., Jadhav, S., & Roopashree, M. (2020). Factors to overcoming barriers affecting electronic medical record usage by physicians. Indian Journal of Community Medicine, 45(2), 168, doi: 10.4103/ijcm.IJCM_478_19.
  » https://doi.org/10.4103/ijcm.IJCM_478_19
• Topaz, M., Ronquillo, C., Peltonen, L. M., Pruinelli, L., Sarmiento, R. F., Badger, M. K. & Lee, Y. L. (2016). Nurse informaticians report low satisfaction and multi-level concerns with electronic health records: results from an international survey. Annual Symposium Proceedings. AMIA Symposium, 2016, 2016–2025. 28269961.
• Vaidotas, M., Yokota, P. K. O., Negrini, N. M. M., Leiderman, D. B. D., de Souza, V. P., Dos Santos, O. F. P., & Wolosker, N. (2019). Medication errors in emergency departments: is electronic medical record an effective barrier? Einstein (São Paulo), 17(4), 1–5, doi: 10.31744/einstein_journal/2019GS4282.
  » https://doi.org/10.31744/einstein_journal/2019GS4282
• Vainiomäki, S., Aalto, A.-M., Lääveri, T., Sinervo, T., Elovainio, M., Mäntyselkä, P., & Hyppönen, H. (2017). Better usability and technical stability could lead to better work-related well-being among physicians. Applied Clinical Informatics, 8(4), 1057–1067, doi: 10.4338/ACI-2017-06-RA-0094.
  » https://doi.org/10.4338/ACI-2017-06-RA-0094
• Viitanen, J., Hyppönen, H., Lääveri, T., Vänskä, J., Reponen, J., & Winblad, I. (2011). National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. International Journal of Medical Informatics, 80(10), 708–725, doi: 10.1016/j.ijmedinf.2011.06.010.
  » https://doi.org/10.1016/j.ijmedinf.2011.06.010
• Walter, Z., & Lopez, M. S. (2008). Physician acceptance of information technologies: role of perceived threat to professional autonomy. Decision Support Systems, 46(1), 206–215, doi: 10.1016/j.dss.2008.06.004.
  » https://doi.org/10.1016/j.dss.2008.06.004

Appendix 1 – scales

Overall evaluation of system quality:

Usability scale

Technical quality

TQ1. The systems are stable in terms of technical functionality (no crash, no downtime).

TQ2. Faulty system function has caused or has nearly caused a serious adverse event for the patient.

TQ3. The system responds quickly to inputs.

TQ4. In my view, the system frequently behaves in unexpected or strange ways.

TQ5. Information entered/documented occasionally disappears from the IS.

Information quality

IQ1. The patient’s current medication list is presented in a clear format.

IQ2. The EHR system generates a summary view (e.g. on a timeline) that helps develop an overall picture of the patient’s health status.

IQ3. The system monitors and notifies when the orders given to nurses have been completed.

IQ4. Measurement results provided electronically by the patient (e.g. via the patient portal) help improve the quality of care.

IQ5. EHR systems support cooperation and communication between physicians and patients.

Feedback

FB1. The system supplier implements suggested corrections and amendments as wished.

FB2. The system supplier is interested in feedback from users.

FB3. Suggestions for corrections and amendments are implemented sufficiently quickly.

Ease of use

EU1. The arrangement of fields and functions is logical on the computer screen.

EU2. Terminology on the screen is clear and understandable (e.g. titles and labels).

EU3. Entering and documenting patient data is quick, easy and smooth.

EU4. The systems keep me clearly informed about what it is doing (e.g. saving data).

EU5. Routine tasks can be performed straightforwardly without the need for extra steps using the system.

EU6. It is easy to obtain necessary patient information using the EHR system.

EU7. The information on the nursing record is in an easily readable format.

Benefits

BE1. IS help to improve quality of care.

BE2. IS help to ensure continuity of care.

BE3. IS support compliance and adherence to the treatment recommendations.

BE4. IS help preventing errors and mistakes associated with medications.

BE5. IS help to avoid duplicate tests and examinations.

BE6. The EHR system provides me with information about the need for and effectiveness of treatment of my patients.

Cross-organizational collaboration

Answer CO1, CO2, CO3 and CO4, only if the system communicates with other health-care organizations (e.g. branch offices or other hospitals).

CO1. Information on medications ordered in other organizations is easily available.

CO2. Obtaining patient information from another organization often takes too much time.

CO3. Patient data (also from other organizations) are comprehensive, up to date and reliable.

CO4. EHR systems support cooperation and communication between physicians working in different organizations.

Internal collaboration

IC1. EHR systems support cooperation and communication between physicians and nurses.

IC2. EHR systems support cooperation and communication between physicians in your organization.

Note: the original scale in Portuguese is available for interested researchers upon contacting the authors.

	1	2	3	4	5	6	7	8	9	10
I am totally dissatisfied	O	O	O	O	O	O	O	O	O	O	I am totally satisfied

Edited by

Associate editor: Flavio Hourneaux Junior

Publication Dates

• Publication in this collection
  29 Aug 2022
• Date of issue
  Jul-Sep 2022

History

• Received
  04 Feb 2021
• Reviewed
  01 June 2021
• Reviewed
  18 Aug 2021
• Reviewed
  03 Nov 2021
• Accepted
  26 Dec 2021