Usability of the ROBOVID mobile app for health education about COVID-19 ^*

Objective:

to evaluate the usability of the ROBOVID mobile application for health education about COVID-19.

Method:

methodological study with an applied quantitative approach, developed with 21 adults, using an electronic form. Usability was assessed using the System Usability Scale. The Statistical Package for the Social Sciences software was used to analyze the data.

Results:

the overall average score on the System Usability Scale questionnaire was 87.3, which classifies the usability of the ROBOVID application as the best achievable. Furthermore, the application has high scores for ease of memorization, user satisfaction and ease of getting to know the system, as well as excellent scores for absence of errors and speed in carrying out the tasks set.

Conclusion:

the ROBOVID application complies with the principles of usability in terms of efficiency, effectiveness and satisfaction in the evaluation of end users, showing that the application meets consumer needs in terms of its functionalities, demonstrating its viability in terms of the initial objectives.

Descriptors:
User-Centered Design; Health Education; Mobile Apps; Information Technology; Covid-19; Nursing

Objetivo:

evaluar la usabilidad de la aplicación móvil ROBOVID para educación en salud acerca de la COVID-19.

Método:

estudio metodológico con enfoque cuantitativo aplicado, desarrollado con 21 adultos, mediante formulario electrónico. La evaluación de la usabilidad se llevó a cabo mediante la System Usability Scale. Para el análisis de los datos se utilizó el software Statistical Package for the Social Sciences .

Resultados:

la media global por el cuestionario System Usability Scale fue de 87,3, lo que clasifica la usabilidad de la aplicación ROBOVID como mejor alcanzable. Además, la aplicación presenta índices elevados para facilidad de memorización, satisfacción del usuario y facilidad de conocimiento del sistema, además de excelentes índices para ausencia de errores y rapidez en la ejecución de las tareas establecidas.

Conclusión:

la aplicación ROBOVID posee conformidad con los principios de usabilidad en los criterios de eficiencia, eficacia y satisfacción en la evaluación de los usuarios finales, explicando que la aplicación satisface las necesidades de consumo frente a sus funcionalidades, demostrando su viabilidad en cuanto a los objetivos iniciales.

Descriptores:
Diseño Centrado en el Usuario; Educación para la Salud; Aplicaciones Móviles; Tecnología de la Información; Covid-19; Enfermería

Objetivo:

avaliar a usabilidade do aplicativo móvel ROBOVID para educação em saúde acerca da COVID-19.

Método:

estudo metodológico com abordagem quantitativa aplicado, desenvolvido com 21 adultos, por meio de formulário eletrônico. A avaliação da usabilidade ocorreu por meio da System Usability Scale . Para análise dos dados utilizou-se o software Statistical Package for the Social Sciences .

Resultados:

a média global pelo questionário System Usability Scale foi de 87,3, o que classifica a usabilidade do aplicativo ROBOVID como melhor alcançável. Ademais, o aplicativo apresenta índices elevados para facilidade de memorização, satisfação do usuário e facilidade de conhecimento do sistema, além de excelentes índices para ausência de erros e rapidez na execução das tarefas estabelecidas.

Conclusão:

o aplicativo ROBOVID possui conformidade com os princípios de usabilidade nos critérios de eficiência, eficácia e satisfação na avaliação dos usuários-finais, explicitando que o aplicativo atende às necessidades de consumo frente a suas funcionalidades, demostrando sua viabilidade no que tange aos objetivos iniciais.

Descritores:
Design Centrado no Usuário; Educação em Saúde; Aplicativos Móveis; Tecnologia da Informação; Covid-19; Enfermagem

Highlights:

(1) ROBOVID was considered better achievable on the SUS scale by the participants.

(2) The ROBOVID app meets the user’s needs in terms of its functions.

(3) ROBOVID helps individuals to take care of themselves and prevent COVID-19.

(4) The ROBOVID app was evaluated as a useful, didactic and functional tool.

Introduction

Corona Virus Disease 2019 (COVID-19) is an emerging infectious and contagious disease, with a high mortality rate attributed to Severe Acute Respiratory Syndrome Coronavirus-2 (SARS-CoV-2), with a potentially serious respiratory illness, which has occupied a prominent place among health emergencies since its first occurrence in December 2019, inherently affecting all aspects of daily life worldwide ^{( 1 )} .

Numerous efforts have been made in recent years to find solutions to effectively combat the spread and damage caused by COVID-19 ^{( 1 )} . In the meantime, information is a fundamental strategy for understanding and preventing diseases, given that one of the biggest challenges is to reduce health inequalities and promote changes in people’s lifestyles.

One of the solutions to this end has been to take advantage of the mobile internet, which is already widely used in people’s daily lives. In fact, the popularity of mobile devices used by the population and the growing advancement in the capabilities of this type of technology is shaping up to be a promising approach to detecting, predicting and monitoring COVID-19 ^{( 2 )} . In view of this, the use of mobile health apps can serve this purpose, since they can provide personalized knowledge about the disease and bring about positive changes in behavior for the development of self-care ^{( 3 )} .

Considering the pervasiveness of mobile technology in the health field, in addition to being a facilitating tool for autonomous decision-making by the population in dealing with this disease, the ROBOVID mobile application was built and validated by nurses in a previous study ^{( 4 )} , based on identifying the population’s real doubts about COVID-19. It is worth noting that, according to the study, there are a limited number of apps developed on COVID-19 from the perspective of health professionals ^{( 5 )} . It is therefore an innovative educational technology that facilitates care and is a transformative component for health education, with regard to safe and reliable guidelines aimed at reducing transmission rates, complications, hospitalizations and deaths from this disease.

However, for the best use and enjoyment of applications (APPs), their interfaces need to be able to offer functionalities that allow users to carry out their tasks with as few difficulties as possible. This way, they can be successful and satisfying to use, taking advantage of all their benefits, promoting easy use and a good user experience ^{( 6 )} . Thus, the end user’s participation in this evaluation process is essential, as only the real recipients can directly point out what is lacking for them to identify with the educational material, thus guaranteeing the refinement of the technology ^{( 7 )} .

Based on this context, the interest in investigating the usability of the ROBOVID mobile application is justified, as a way of evaluating efficiency, effectiveness and satisfaction as key points that determine the user experience involved with an application ^{( 8 )} . In addition, the incipient production of research on the usability of educational health technologies in mobile application format, aimed at the prevention of COVID-19 from the perspective of health professionals, ratifies the relevance of this study.

Considering the emerging phase of the use of mobile applications in the health area, in which people are increasingly seeking information online, the question is: what is the level of usability of a mobile application with guidance on the prevention of COVID-19? To answer this question, the aim of this study is to evaluate the usability of the ROBOVID mobile app for health education about COVID-19.

Method

Study design

This is an applied methodological study with a quantitative approach, evaluating the usability of the ROBOVID mobile application. The mobile application was built and validated in a previous study ^{( 4 )} and seeks to provide health education in the face of COVID-19.

Development and availability of the ROBOVID mobile application

The app was built with a simple, yet attractive and easy-to-use interface, with open user control functionality, which allows the population to have individualized access and quickly and securely obtain information about COVID-19.

The name of the ROBOVID mobile application is the result of a combination of the suffix “robot”, which refers to technology, plus the suffix “vid” associated with the term “COVID”, the disease in the current pandemic context (ROBÔ+VID = ROBOVID). The application was designed for mobile devices (smartphones), for online use, on the iOS (iPhone Operating System) and Android platforms. The app’s home screen displays information in the form of “themes” in the lower bar, including: characteristics of the novel coronavirus; transmission; diagnosis; prevention/protection; signs and symptoms; social isolation; mask use; hand, product and surface hygiene; risk groups; medication use; and vaccines. By clicking on the topic of interest, the user is led to information in the form of questions, as well as video resources, folders and self-explanatory illustrations.

In addition, ROBOVID contains a side menu to guide the use of the app via the browser, with a description of how to use it, privacy policy, terms of use, contact us, as well as the item “evaluate ROBOVID”, a space for evaluating the app’s usability using the System Usability Scale (SUS) questionnaire. The app is available to download from the Apple Store for iOS and can be accessed using the QR Code (Quick Response Code) ( Figure 1 ).

Figure 1
- QR Code* from the ROBOVID app for iOS ^† devices. Rio de Janeiro, RJ, Brazil, 2023

Participants

Adults aged 18 or over and living in the state of Rio de Janeiro took part in the usability evaluation process, and these were the inclusion criteria. Participants who did not send in the completed form by the deadline were excluded.

Participants were selected using the snowball sampling method ^{( 9 )} which consists of a form of non-probabilistic sampling, through reference chains of possible participants, who were initially invited by indication from the research team itself and later by participants who indicated other individuals, respecting the inclusion criteria mentioned above.

It is worth noting that the group initially consisted of 29 participants, with eight individuals losing out due to not returning the instrument within the previously determined time limit. In this way, a limit of 21 participants was set, in line with studies that indicate a range of between seven and thirty evaluators for the validation of instructional materials ^{( 10 )} .

Data collection

For data collection, the evaluators were invited by e-mail to take part in the study, by means of an invitation letter detailing the study and its objectives. Those who showed interest in taking part in the study and formally agreed to do so by signing the Informed Consent Form (ICF) received a link to access the semi-structured electronic form on the Google Forms Virtual Platform, which was sent out in September 2022. Invitations were sent out weekly, up to three times, with a reminder to take part and a deadline of fifteen days for the full evaluation to be returned.

Participants were sent a tutorial on how to download and use the app, for both Android and iOS devices.

The electronic form accessed by participants contained three parts: 1) Questions about the participant’s sociodemographic profile; 2) The System Usability Scale (SUS) and 3) A field for suggestions from participants.

Thus, the usability of the application was evaluated using the System Usability Scale (SUS) questionnaire ^{( 11 )} , simple, easy to apply, developed in 1986 in a version translated and validated into Portuguese by a study ^{( 12 )} .

This instrument ^{( 13 )} is used to check the level of usability of a technological system. It consists of 10 items in which participants indicate their degree of agreement using a Likert-type scale, with values ranging from one to five, classified respectively as: 1- strongly disagree; 2- disagree; 3- neither agree nor disagree; 4- agree; 5- strongly agree.

The ten SUS questions are: Q1- I think I would like to use this product often; Q2- I found the product more complex than necessary; Q3- I found the product easy to use; Q4- I think I would need help from a technician to be able to use this product; Q5- I thought the various functionalities of this product were well integrated; Q6- I thought this product had a lot of inconsistencies; Q7- I suppose most people would learn to use this product quickly; Q8- I found the product very complicated to use; Q9- I felt very confident using this product; Q10- I had to learn a lot before I could get to grips with this product. In addition to the ten questions, there was a field for participants to write suggestions/comments for improving the application, which was excluded from the calculation of the SUS score because it was not part of the original instrument.

Data analysis

The SUS usability score was calculated by adding up the individual collaborations for each question. For odd-numbered questions, one was subtracted from the scale marked by the user, while for even-numbered questions the score was five minus the scale marked by the user. Next, the participants’ individual scores were added together and multiplied by 2.5 to obtain the total usability scale (SUS Score), i.e. the satisfaction index, which ranges from 0 to 100 points ^{( 11 )} . Technological tools evaluated using the SUS Score questionnaire are classified: between 0 and 25 points - worst achievable; from 26 to 39 - bad; 40 to 52 - acceptable; 53 to 74 - good; 75 to 85 - excellent; and from 86 to 100 - best achievable ^{( 14 )} . The SUS is divided into five distinct domains that assess the main characteristics of usability, namely: 1) Ease of knowing the system - which is contained in questions 3, 4, 7 and 10; 2) System efficiency - through questions 5, 6 and 8; 3) Inconsistencies - based on question 6; 4) Ease of memorization - assessed through question 2; 5) User satisfaction - scored by questions 1, 4 and 9. Therefore, in order to score the individual usability characteristics, the answers obtained from each participant were multiplied by 25, in order to bring together the range of possible values from 0 to 100. The overall average of the scores obtained for each question was then calculated, followed by the average of the items referring to the characteristics ^{( 15 - 16 )} .

The data was tabulated in Microsoft Excel ^® and then exported to IBM ^® SPSS (Statistical Package for the Social Sciences) v.26 for statistical analysis. Numerical variables were expressed as a measure of central tendency (mean and median) and a measure of dispersion (standard deviation).

Ethical aspects

The study complied with Resolution No. 466 of 2012 of the National Health Council, with opinion approved under No. 4.132.385 and Certificate of Submission for Ethical Appraisal (CAAE) number 34338120.6.0000.8160. It is worth noting that all participants signed the Informed Consent Form (ICF), which was accepted online.

Results

The study included 21 adults (100%) aged between 18 and 61 years (average 37 years), living in the state of Rio de Janeiro. The majority (15 - 71.4%) were female, 10 (47.6%) self-declared white and seven (33.3%) black. About to schooling, 10 (47.6%) had completed higher education, followed by 6 (28.5%) with completed secondary education. As for occupation, professors accounted for the highest percentage, with 12 (57.1%).

The average score obtained from the answers to the SUS questionnaire was 87.3 standard deviations (SD=6.8, Median= 87.5), ranging from 75.0 to 95.7, indicating that the usability of the ROBOVID application was better, as shown in Table 1 .

The domains that assess usability characteristics obtained means ranging from 83.3 to 94.0 ( Table 2 ).

The application presented itself as a technological resource that was highly easy to memorize (94.0), user satisfaction (90.5) and easy to get to know the system (88.4), compared to the indices classified as best achievable. However, although the system’s efficiency (84.1) and inconsistencies (83.3) showed slightly lower values, these items still fall within the range of excellent usability.

Table 1
- Individual distribution of participants’ responses to the System Usability Scale (n ^* =21). Rio de Janeiro, RJ, Brazil, 2022

Participants	Q1 †	Q2 †	Q3 †	Q4 †	Q5 †	Q6 †	Q7 †	Q8 †	Q9 †	Q10 †	Score SUS ‡
P1 §	4	4	3	4	4	4	4	4	3	4	97.5
P2 §	4	4	3	4	2	4	4	4	3	2	85
P3 §	4	4	3	4	2	2	4	4	0	4	77.5
P4 §	4	4	3	4	2	4	4	4	4	4	92.5
P5 §	4	4	3	3	4	4	4	4	4	4	95
P6 §	4	4	3	4	4	4	4	4	4	2	92.5
P7 §	4	4	3	4	4	4	4	4	4	2	92.5
P8 §	4	4	3	4	4	4	4	4	4	4	97.5
P9 §	4	4	4	4	2	4	2	4	4	4	90
P10 §	4	1	3	3	2	3	4	4	4	4	80
P11 §	4	4	4	4	2	4	4	4	4	4	95
P12 §	4	4	4	4	0	2	4	4	4	3	82.5
P13 §	4	4	3	3	4	2	3	2	4	4	82.5
P14 §	4	4	3	3	4	4	2	4	4	2	85
P15 §	4	4	3	4	4	4	4	3	2	4	80
P16 §	4	4	4	4	4	2	4	4	3	4	92.5
P17 §	4	4	4	4	2	4	2	4	3	4	87.5
P18 §	4	3	4	4	4	2	2	3	2	3	75
P19 §	3	3	4	4	2	3	3	4	4	4	85
P20 §	4	4	3	3	3	4	4	3	0	4	80
P21 §	4	4	3	4	4	2	4	4	2	4	87.5
Overall average SUS ‡											87.3

• ^*
  n = Number of study participants;
• ^†
  Qn = System Usability Scale question;
• ^§
  Pn = Participant
• ^‡
  Score SUS = System Usability Scale questionnaire score;

Table 2
- Distribution of the mean scores of the domains of the System Usability Scale of the ROBOVID application. Rio de Janeiro, RJ, Brazil, 2022

Usability features	Mean number of questions among participants	Overall mean	Meaning
Ease of getting to know the system	Q3 * (83.3) Q4 * (94.0) Q7 * (88.1) Q10 * (88.1)	88.4	Easy-to-use system when used for the first time
System efficiency	Q5 * (75.0) Q6 * (83.3) Q8 * (94.0)	84.1	Speed in carrying out established tasks
Inconsistencies	Q6 * (83.3)	83.3	No errors
Ease of memorization	Q2 * (94.0)	94.0	Easy-to-use system even after a long period without using it
User satisfaction	Q1 * (98.8) Q4 * (94.0) Q9 * (78.6)	90.5	Pleasant design

• ^*
  Q = Question of System Usability Scale

It should be noted that, although the ROBOVID mobile app was classified by the majority of participants as a health education tool that was better attainable, the suggestions/comments for improving the app included: the inclusion of other respiratory diseases (themes) and media, such as a video tutorial on hand washing, on the action of the COVID-19 vaccine on the body; as well as the inclusion of information on the importance of immunizing the population against COVID-19, as well as against other morbidities. In addition, it was requested that the app be made available in the Android Play Store.

After evaluation by users, the research team, together with the app developer, summarized the qualitative analysis of the recommendations made by participants. It should be noted that only two of the user requests could not be met, namely: the inclusion of other respiratory diseases could not be met, since the app has been registered with the INPI and published in national and international journals with the identity that refers to COVID-19, which makes it impossible to include other diseases. As for making the mobile app available in the Play Store for Android, it has not yet been authorized, despite numerous requests sent to the competent body.

Discussion

The ROBOVID mobile app aims to provide health education in the face of COVID-19, and in this study, it was considered in the usability assessment to be a technological tool that is better accessible to end users, explaining that the app meets consumer needs in terms of its functionalities, which allows us to infer that ROBOVID is easy to use and can be incorporated into people’s daily lives. In addition, the app in question reached the highest level in terms of usability evaluation, as in a study that reports that the highest percentages are found among scores ranging from 75.0 to 95.7 on the categorization scale ^{( 17 )} .

In this sense, the use of mobile health apps can serve the purpose of health education, since they can provide specific knowledge on a given topic, positively impacting individuals’ lifestyles, making them co-participants in their own care, with a view to independent and effective management ^{( 3 )} . In view of this, the development and use of educational technologies in nursing has grown exponentially, becoming a contemporary didactic mechanism that favors the automation of processes, even serving to increase safety in the care decision-making process ^{( 18 )} .

These apps need to be analyzed for usability, especially when it comes to mobile devices for the health sector. Regardless of the method used for this type of analysis, the need to evaluate the use of these apps is emphasized, since they should be developed based on technical references of standardization and consensus protocols in health, as well as being updated, as is the case with ROBOVID ^{( 19 )} .

With the advent of the internet, initiatives for teaching/education through this platform have emerged, a fact that has corroborated the significant increase in the number of studies involving the alternative or complementary use of health-related apps. In this respect, a multicenter study carried out in Norway found that the use of smartphone apps and teleconsultations contributed to a reduction in the number of amputations in patients with diabetic foot, proving to be a relevant complementary alternative ^{( 20 )} with regard to health education. Another study reports that cancer patients believe that the mobile app developed is useful for managing health care, but points out that testing the usability of technologies created for this purpose is essential, since evaluating usability influences the reliability that patients have in relation to the content made available in the app ^{( 21 )} .

Still in this vein, a study carried out in Iran developed a mobile app to encourage pregnant women’s self-care against COVID-19, and rated it as good, since the app can be used to reduce anxiety and stress about COVID-19 in pregnant women, as well as providing access to reliable information to answer possible questions, identify high-risk locations and provide pregnant women with instant access to health facilities and information related to self-care processes in the face of COVID-19 ^{( 22 )} .

Usability testing is therefore becoming increasingly essential before making the app available to the end user. Prior to real-world applicability checks, this provides a technical baseline in which users have familiarized themselves with the potential of mobile technology, which allows users to provide richer feedback on functional requirements when using the app ^{( 3 )} .

In order to assess the usability of the ROBOVID application, a specific tool was used for this purpose, the SUS, which has been used in other studies to analyze applications for educational mobile devices ^{( 10 , 21 )} . Conceptually, usability is the ability of software to be understood, learned and operated by an individual when used for specific purposes ^{( 22 )} .

According to the literature, a SUS score above 68 indicates an acceptable degree of usability ^{( 10 )} . In an extensive analysis of the application of the SUS questionnaire, it was identified that a score of 85 would be associated with excellent acceptance of a piece of software or an application ^{( 14 )} . However, the higher the score, the higher the degree of quality and potential of the mobile application for handling. From this perspective, the average SUS score for assessing the usability of the ROBOVID app was 87.3, meeting the parameters described in the literature. Thus, it is understood that users agreed that ROBOVID is effective and can help with self-care, especially about the prevention of COVID-19.

In this way, usability measurement is particularly important for checking, among other things, the complexity of the interactions between the user, the objectives and the characteristics of the tasks carried out when using the application ^{( 23 )} . The importance of carrying out a usability evaluation in this study is emphasized, as a way of measuring whether ROBOVID is a tool that facilitates autonomous decision-making in dealing with this problem among the population for which it was intended, and in view of the evaluation it was classified as best achievable.

Similar to this research, a study evaluated the usability and usefulness of a mobile app called MoomMae with 21 pregnant women and found positive feedback, showing that it has great potential to be a useful self-management tool for breastfeeding mothers in Thailand ^{( 24 )} .

Thus, usability brings together a set of quality measures of how a technological tool can be used by specific users, to achieve objectives that facilitate learning, for its efficiency, identification of inconsistencies, ease of memorization and user satisfaction ^{( 15 )} .

In view of this, the average usability value scored by converting the possible value ranges from 0 to 100, as a way of quantifying and ratifying the quality components, the findings of this study reveal a range of 75.0 to 95.7. ROBOVID is therefore a technological resource that is very easy to learn and generates satisfaction among users. However, although the speed with which tasks are carried out, the efficiency of the system and inconsistencies were below the overall usability average, the application achieved a best achievable rating ^{( 15 )} . It can therefore be understood that, in view of the scores achieved, the evaluators reported that the ROBOVID mobile app is a useful tool, which uses colloquial language, is easy to use and understand, with well-integrated and objective information, and is didactic and functional.

It should be noted that feedback is fundamental for evaluating applications, as it allows information to be organized, complemented and updated simultaneously to help identify weaknesses and improvements to make the application suitable for use. In addition, the usability of the app makes an important contribution to expanding knowledge, it is innovative in that it expands the dynamics and capacity of qualifications aimed at society and nurses’ health actions, and thus manages to intervene as a tool to facilitate the timely identification of signs and symptoms of COVID-19 ^{( 16 )} .

Finally, with regard to the mobile technology used, in general users did not have any difficulties, as they are familiar with the mobile device and the suggestions made were met when possible. It is therefore understood that evaluating usability in particular with the target audience for which this technology is intended favors improving the application from their point of view, which will promote engagement and greater use/adherence by the users themselves.

In addition, it is understood that the assessment of the ROBOVID app as being the best reachable by users ratifies its important contribution as a health education tool, making it possible to obtain qualified and therefore safe information on the adoption of protective measures for the prevention of COVID-19, positively impacting on the timely recognition of symptoms and complications of the disease, reducing the occupation of health services. In addition, studies of this nature can motivate health professionals by corroborating the clinical practice of nursing with regard to the evaluation and use of technological tools available in society, in order to promote health education among the population.

The limitations of the research include the lack of studies evaluating the usability of mobile applications aimed at COVID-19, which limits comparison and dialogue between studies with the same purpose. In addition, the study was only carried out with individuals living in the state of Rio de Janeiro, since the analysis of the app’s usability may be different between states in Brazil or may vary depending on the level of education among the participants. Another limitation is that the app has not yet been made available for the Android operating system on the Google Play Store platform, making it impossible for a larger number of people seeking reliable information about COVID-19 as a form of prevention to access it.

Conclusion

The ROBOVID app was classified as a technological tool that achieved the best usability rating by end users, explaining that the app allows users to obtain safe and qualified information while browsing the mobile app through its functionalities, meeting their real knowledge needs in relation to COVID-19. This can be explained by the fact that the ROBOVID app was built based on identifying the population’s real doubts about COVID-19, making it possible to consolidate it as a teaching-learning tool for the individuals who make up the app’s target audience.

In this way, the SUS questionnaire made it possible to assess the efficiency, effectiveness and satisfaction of users, as well as the ease of knowing the system, system efficiency, inconsistencies and ease of memorization, when using ROBOVID, demonstrating the viability of the application with regard to its initial purposes, from the perspective of civil society and health professionals.

References

• ^1.
  Shen J, Ghatti S, Levkov NR, Shen H, Sen T, Rheuban K, et al. A survey of COVID-19 detection and prediction approaches using mobile devices, AI, and telemedicine. Front Artif Intell. 2022;5:1034732. https://doi.org/10.3389/frai.2022.1034732
  » https://doi.org/10.3389/frai.2022.1034732
• ^2.
  Seshadri DR, Davies EV, Harlow ER, Hsu JJ, Knighton SC, Walker TA, et al. Wearable sensors for COVID-19: a call to action to harness our digital infrastructure for remote patient monitoring and virtual assessments. Front Digit Health. 2020;2:8. https://doi.org/10.3389/fdgth.2020.00008
  » https://doi.org/10.3389/fdgth.2020.00008
• ^3.
  Marques ADB, Moreira TMM, Jorge TV, Rabelo SMS, Carvalho REFL, Felipe GF. Usability of a mobile application on diabetic foot self-care. Rev Bras Enferm. 2020;73(4):e20180862. https://doi.org/10.1590/0034-7167-2018-0862
  » https://doi.org/10.1590/0034-7167-2018-0862
• ^4.
  Silva ACSS, Góes FGB, Ávila FMVP, Goulart MCL, Pinto LF, Stipp MAC. Construction and validation of a mobile application for health education about COVID-19. Rev Gaúcha Enferm. 2022;43:e20210289. https://doi.org/10.1590/1983-1447.2022.20210289.en
  » https://doi.org/10.1590/1983-1447.2022.20210289.en
• ^5.
  Galindo NM Neto, Sá GGM, Barbosa LU, Pereira JCN, Henriques AHB, Barros LM. Covid-19 and digital technology: mobile applications available for download in smartphones. Texto Contexto Enferm. 2020;29. https://doi.org/10.1590/1980-265X-TCE-2020-0150
  » https://doi.org/10.1590/1980-265X-TCE-2020-0150
• ^6.
  Borges AP, Faria TCC, Moraes RV, Divino EA, Beltrame RCT, Corrêa ACP. Evaluation of the usability of the mobile application for pregnant women based on the System Usability Scale questionnaire (SUS). Res Soc Dev. 2021;10(12):e118101220086. https://doi.org/10.33448/rsd-v10i12.20086
  » https://doi.org/10.33448/rsd-v10i12.20086
• ^7.
  Lima SJ, Santos MG, Amaral GG, Andrade HS, Guimarães EAA, Oliveira VC. Development and validation of an educational software for teaching vaccination. Res Soc Dev. 2021;10(7):e17510716394. https://doi.org/10.33448/rsd-v10i7.16394
  » https://doi.org/10.33448/rsd-v10i7.16394
• ^8.
  Kumar DS, Purani K, Viswanathan SA. Influences of ‘appscape’ on mobile app adoption and m-loyalty. J Retail Cons Serv. 2018:132-41. https://doi.org/10.1016/j.jretconser.2018.08.012
  » https://doi.org/10.1016/j.jretconser.2018.08.012
• ^9.
  Naderifar M, Ghaljaei F, Goli H. Snowball sampling: a purposeful method of sampling in qualitative research. Strides Dev Med Educ. 2017;14(3):e67670. https://doi.org/10.5812/sdme.67670
  » https://doi.org/10.5812/sdme.67670
• ^10.
  Lima CJM, Coelho RA, Medeiros MS, Kubrusly M, Marçal E, Peixoto AA Junior. Development and validation of a mobile application for the teaching of electrocardiogram. Rev Bras Educ Med. 2019;43(1):157-65. https://doi.org/10.1590/1981-5271v43suplemento1-20190164.ING
  » https://doi.org/10.1590/1981-5271v43suplemento1-20190164.ING
• ^11.
  Brooke J. SUS: a quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, editors. Usability evaluation in industry. London: Taylor & Francis; 1996. p. 189-94.
• ^12.
  Tenório JM, Cohrs FM, Sdepanian VL, Pisa IT, Marin HF. Desenvolvimento e avaliação de protocolo eletrônico para atendimento e acompanhamento do paciente com doença celíaca. Rev Inform Teor Aplic. 2011;17(2):210-20. https://doi.org/10.22456/2175-2745.12119
  » https://doi.org/10.22456/2175-2745.12119
• ^13.
  Padrini-Andrade L, Balda RCX, Areco KCN, Bandiera-Paiva P, Nunes MV, Marba STM, et al. Evaluation of usability of a neonatal health information system according to the user’s perception. Rev Paul Pediatr. 2019;37(1):90-6. https://doi.org/10.1590/1984-0462/;2019;37;1;00019
  » https://doi.org/10.1590/1984-0462/;2019;37;1;00019
• ^14.
  Bangor A, Kortum P, Miller J. Determining what individual sus scores mean: adding an adjective rating scale. J Usabil Stud [Internet]. 2009 [cited 2023 May 12];4(3):114-23. Available from: https://uxpajournal.org/determining-what-individual-sus-scores-mean-adding-an-adjective-rating-scale/
  » https://uxpajournal.org/determining-what-individual-sus-scores-mean-adding-an-adjective-rating-scale/
• ^15.
  Melo CMCS, Delgado AJF Filho, Oliveira ER, Araújo AA, Cavalcanti HGO, Melo CMCS, et al. Development and assessment of an application for primary care for users with diabetes mellitus. Aquichan [Internet]. 2020;20(2):e2026. https://doi.org/10.5294/aqui.2020.20.2.6
  » https://doi.org/10.5294/aqui.2020.20.2.6
• ^16.
  Cavalcanti HGO, Bushatsky M, Barros MBSC, Melo CMCS, Delgado AJF Filho. Evaluation of the usability of a mobile application in early detection of pediatric cancer. Rev Gaúcha Enferm. 2021;42:e20190384. https://doi.org/10.1590/1983-1447.2021.20190384
  » https://doi.org/10.1590/1983-1447.2021.20190384
• ^17.
  Andreasen EM, Høigaard R, Berg H, Steinsbekk A, Haraldstad K. Usability Evaluation of the Preoperative ISBAR (Identification, Situation, Background, Assessment, and Recommendation) Desktop Virtual Reality Application: Qualitative Observational Study. JMIR Hum Factors. 2022;9(4):e40400. https://doi.org/10.2196/40400
  » https://doi.org/10.2196/40400
• ^18.
  Colodetti R, Prado TM, Bringuente ME, Bicudo SD. Mobile application for the management of diabetic foot ulcers. Acta Paul Enferm. 2021;34:eAPE00702. https://doi.org/10.37689/acta-ape/2021AO00702
  » https://doi.org/10.37689/acta-ape/2021AO00702
• ^19.
  Silva AP, Barbosa BJP, Hino P, Nichiata LYI. Usability of mobile applications for health professionals: Integrative review. J Health Inform [Internet]. 2021 [cited 2023 Jun 29];13(3):100-5. Available from: https://jhi.sbis.org.br/index.php/jhi-sbis/article/view/879
  » https://jhi.sbis.org.br/index.php/jhi-sbis/article/view/879
• ^20.
  Smith-Strøm H, Igland J, Østbye T, Tell GS, Hausken MF, Graue M, et al. The effect of telemedicine follow-up care on diabetes-related foot ulcers: a cluster-randomized controlled non-inferiority trial. Diabetes Care. 2018;41(1):96-103. https://doi.org/10.2337/dc17-1025
  » https://doi.org/10.2337/dc17-1025
• ^21.
  Rezaee R, Asadi S, Yazdani A, Rezvani A, Kazeroon AM. Development, usability and quality evaluation of the resilient mobile application for women with breast cancer. Health Sci Rep. 2022;5:e708. https://doi.org/10.1002/hsr2.708
  » https://doi.org/10.1002/hsr2.708
• ^22.
  Moulaei K, Sheikhtaheri A, Ghafaripour Z, Bahaadinbeigy K. The Development and Usability Assessment of an mHealth Application to Encourage Self-Care in Pregnant Women against COVID-19. J Healthc Eng. 2021:9968451. https://doi.org/10.1155/2021/9968451
  » https://doi.org/10.1155/2021/9968451
• ^23.
  Maciel LHA, Sereno MC, Viana AIS. Usability evaluation of a mobile application as a facilitator of pregnant woman access to health services in a maternity hospital in southern Maranhão. Rev Saúde Digital Tec Educ. 2021;6(1):1-14. https://doi.org/10.36517/resdite.v6.n1.2021.a5
  » https://doi.org/10.36517/resdite.v6.n1.2021.a5
• ^24.
  Wang C, Chaovalit P, Pongnumkul S. A breastfeed promoting mobile app intervention: usability and usefulness study. JMIR Mhealth Uhealth. 2018;6(1):e27. https://doi.org/10.2196/mhealth.8337
  » https://doi.org/10.2196/mhealth.8337

• ^*
  The publication of this article in the Thematic Series “Digital health: nursing contributions” is part of Activity 2.2 of Reference Term 2 of the PAHO/WHO Collaborating Centre for Nursing Research Development, Brazil.
• All authors approved the final version of the text.
• All authors approved the final version of the text.
  Correia GS, Silva ACSS, Pinto LF, Góes FGB, Goulart MCL, Pereira-Ávila FMV. Usability of the ROBOVID mobile app for health education about COVID-19. Rev. Latino-Am. Enfermagem. 2024;32:e4191 [cited year month day]. Available from: URL . https://doi.org/10.1590/1518-8345.6924.4191

Edited by

Associate Editor:

Regina Aparecida Garcia de Lima

Publication Dates

• Publication in this collection
  07 June 2024
• Date of issue
  2024

History

• Received
  29 June 2023
• Accepted
  06 Jan 2024