Evaluation of the usability of a mobile application in early detection of pediatric cancer

Evaluación de la usabilidad de una aplicación móvil en la detección precoz de cáncer pediátrico

ABSTRACT

Objective

To evaluate the usability of a mobile application for early detection of pediatric cancer.

Method

A descriptive study, with a quantitative approach, evaluating the usability of the application “Fique Atento, pode ser câncer” by 19 oncology nurses at the Oswaldo Cruz University Hospital, Pernambuco, Brazil, using the validated System Usability Scale questionnaire. For analysis, Microsoft Excel and the support of the Statistical Package for the Social Sciences (SPSS) software were used.

Results

Through the application of the SUS questionnaire, the total value of the averages of all scores was 91.58%, the satisfaction rate was 88.6% and the efficiency rate 91.2%.

Discussion

The data demonstrated that the application has agreement and compliance with the principles of usability in the criteria of effectiveness, efficiency and user satisfaction.

Final considerations

The application brings an important contribution to expand the construction and validation of technological tools that align both theoretical and practical knowledge.

Keywords
Primary health care; Medical oncology; Child health; Adolescent health; Information technology

RESUMEN

Objetivo

Evaluar la usabilidad de una aplicación móvil para la detección precoz del cáncer pediátrico.

Método

Estudio descriptivo, con enfoque cuantitativo, evaluando la usabilidad de la aplicación “Fique atento, pode ser câncer” por 19 enfermeras de oncología del Hospital Universitario Oswaldo Cruz, Pernambuco, Brasil, utilizando el cuestionario de la Escala de Usabilidad del Sistema validado. Para el análisis de los datos se utilizó Microsoft Excel y el soporte del software Statistical Package for the Social Sciences (SPSS).

Resultados

Al aplicar el cuestionario SUS, el valor totals de los promedios de todos los puntajes fue del 91,58%, la tasa de satisfacción fue del 88,6% y la tasa de eficiencia del 91,2%.

Discusión

Los datos demostraron que la aplicación tiene concordancia y cumplimiento con los principios de usabilidad en los criterios de efectividad, eficiencia y satisfacción del usuario.

Consideraciones finales

La aplicación aporta un importante aporte para expandir la construcción y validación de herramientas tecnológicas que alinean conocimientos teóricos y prácticos.

Palabras-clave
Atencíon primaria de salud; Oncología médica; Salud del niño; Salud del adolescente; Tecnología de la informacíon

RESUMO

Objetivo

Avaliar a usabilidade de um aplicativo móvel para detecção precoce do câncer pediátrico.

Método

Estudo descritivo, com abordagem quantitativa, de avaliação da usabilidade do aplicativo “Fique Atento, pode ser câncer” por 19 enfermeiros oncológicos do Hospital Universitário Oswaldo Cruz, Pernambuco, Brasil, utilizando o questionário validado System Usability Scale. Para análise dos dados utilizou-se o Microsoft Excel e o suporte do software Statistical Package for the Social Sciences (SPSS).

Resultados

Mediante a aplicação do questionário SUS, o valor total das médias de todos os scores foi de 91,58%, o índice de satisfação foi de 88,6% e o de eficiência 91,2%.

Discussão

Os dados demonstraram que o aplicativo possui concordância e conformidade com os princípios de usabilidade nos critérios de efetividade, eficiência e satisfação do usuário.

Considerações finais:

O aplicativo traz uma importante contribuição para ampliação da construção e validação de ferramentas tecnológicas que alinham o conhecimento teórico ao prático.

Palavras-chave
Atenção primária à saúde; Oncologia; Saúde da criança; Saúde do adolescente; Tecnologia da informação

INTRODUCTION

Pediatric cancer is considered the leading cause of death among children and adolescents aged 1 to 19 years in Brazil. Differentiating from adults because it has an embryonic character, rapid evolution and high lethality1.

Cancer in children and adolescents has several clinical presentations related to origin, risk factors, anatomical site and treatment responses1. Due to these characteristics, signs and symptoms can be easily confused with common childhood diseases2.

It raises a great challenge for health professionals in the investigation of the disease, consequently increasing the duration of the diagnostic interval and affects the patient's survival3.

Considering the incidence, hospital morbidity and mortality of neoplasms together with the challenges that the Unified Health System has been facing, according to the World Health Organization (WHO), the need and the relevant role of access to technology in health arise, where its use must be based on quality scientific evidence4.

Computerization in Primary Health Care (PHC) manifests itself in order to improve nursing care and assist in decision making by professionals, since PHC is characterized as the first preferential gateway to the Unified Health System, with high resolvability and joint offer of individual and collective health practices5. Since, at this level of care, nurses have broad autonomy in the exercise of their tasks, from the recognition of cancer to the rescue of the patient out of treatment6.

In this context, it is essential to be attentive to the early detection of pediatric cancer, considering that the prevention of diseases is essential to reduce sequelae and deaths. However, there is still a need for a better understanding of the factors that can leverage the issue of a cancer diagnosis and consequently the identification of nursing diagnoses for children and adolescents affected by this disease7.

New technological tools are emerging frequently and are being integrated into the health area. The insertion of Information and Communication Technologies (ICT) in the scope of health is not a substitute for professional know-how, but aims to inform, support, evaluate and intervene in clinical care for a better prognosis of the disease, in order to reduce the physical repercussions, psychic and social problems that cancer causes. In addition to contributing to the production of knowledge in networks and expanding communication channels for access to services8.

Taking into account that in the therapeutic itinerary of cancer patients, failures in the reference and counter-referral of the flow, functioning and use of health services occur, it is likely that the insertion of ICT, in the scope of Primary Care, may corroborate in the early identification of the diagnosis, alignment of care and immediate referral to a specialized unit, in order to provide a greater chance of curing the disease, better therapeutic response and satisfactory itinerary2.

The development of the application is justified as a strategy to bring health professionals closer to another tool that can be a strong ally for the early detection of childhood cancer. Bearing in mind that with the quick and easy access to smartphones, the use of these devices as assistance support, the phones serves as an aid instrument in the teaching-learning process that is becoming more and more frequent9.

In view of this scenario, the use of the application may assist PHC health professionals in assistance. To incorporate the application with a focus on effectiveness and efficiency, it is essential to evaluate it before making it available to the user, including some tests to analyze usability, quality of the interface, functionality of the resources, data quality and integration with external systems10.

Faced with this context, the following guiding question arises: What is the usability level of an application for healthcare professionals? To answer this question, the study aims to assess the usability of a mobile application in the early detection of pediatric cancer.

METHODS

This is a descriptive study with a quantitative approach, assessing the usability of the application named by the authors as "Fique Atento, pode ser câncer". The proposal to build the mobile application is done with the intention to help health professionals in the early detection of pediatric cancer, through the contents based on the Protocol for Early Diagnosis of Pediatric Cancer and the Early Diagnosis of Cancer in Children and Adolescents of the Ministry of Health, in order to offer a standardization in nursing care with an emphasis on a uniform language.

Descriptive studies describe the identification and analyze the variables found for a later determination of the findings that result from the product. The quantitative approach refers to the understanding of individual elements through explanations based on variables, meaning it seeks to apprehend phenomenal relationships and measure them quantitatively11.

The usability evaluation was performed with the participation of nurses that had worked for more than a year at the pediatric cancer reference treatment centers, voluntarily. Participants were requested to sign Free and Informed Consent Form (FICF) and fill out all the premises listed in the usability questionnaire, from June to August 2019, at Hospital University Oswaldo Cruz, located in Recife, Pernambuco, Brazil.

Availability of the application and assessment tool

The application, available only for smartphones with Android operating system, has a storage capacity of 39 Megabytes permeated by various functionalities, such as the concept of childhood cancer and its classifications; the general signs and symptoms; the procedures and exams to be performed for each specificity; diagnostics; types of treatments and the necessary care regarding toxicity. Listed by video lesson resources, folders, self-explanatory illustrations, and contact option for questions, compliments, suggestions and others.

As it contains various updated and substantiated functions and content, oncology specialist nurses first assessed the approach of the issues included, whether they were sufficient, complete and understandable to assist in the detection of cancer during a nursing consultation, and subsequently, the usability of the application. In addition to obtaining information about the length of time each professional in the oncology area has exercised their profession, to ensure the reliability of the answers.

In order to ensure a successful application for use as an aid tool in the care process, usability assessments were carried out. Several techniques are used for this, ranging from an informal analysis to the availability of validated tests in order to reduce errors, increase efficiency and safety during the user's interaction with the application, making the usability test reliable and reliable.

The System Usability Scale (SUS)12 was used as a usability instrument due to its simplicity, easy application. Developed by John Brooke (1986), the version used was translated and validated for Portuguese by study13 containing 10 statements where nurses indicated their respective degree of agreement on a Likert scale ranging from 1 to 5: 1- strongly disagree; 2- partially disagree; 3- neutral; 4- agree; 5- I completely agree. The items in the SUS questionnaire are listed below.

1. Item 1. I would use this application frequently.

2. Item 2. I found the application unnecessarily complex.

3. Item 3. I found the app easy to use.

4. Item 4. I think I would need the support of a technical support to be able to use this application.

5. Item 5. I found that the various functions of the application were well integrated.

6. Item 6. I thought there was a lot of inconsistency in this app.

7. Item 7. I think most people will learn to use this application quickly.

8. Item 8. I found the app too heavy.

9. Item 9. I felt very confident using the app.

10. Item 10. I needed to learn a number of things before I could continue using the application.

The result of the SUS questionnaire’s usability score was achieved by adding the individual collaboration of each statement. For odd items, one was subtracted from the scale marked by the user, however, for even items, the score was five minus the scale marked by the user. Then, the individual scores of the participants were added and the value was multiplied by 2.5 to obtain the total usability scale (SUS Score), that is, the satisfaction index varies between 0 to 100 points12.

Technological tools with SUS Score between 0 and 25 points are considered the worst achievable; 26 to 39, bad; 40 to 52, acceptable; 53 to 74, good; 75 to 85, excellent; and from 86 to 100, best achievable14.

Assessing how easy it was to memorize the system from item 2 of SUS was made possible through the sum of each statement contained in the SUS questionnaire, as was evaluating the ease of knowledge through items 3, 4, 7 and 10, verifying the efficiency of the system based on items 5, 6 and 8; assessing user satisfaction through items 1, 4 and 9; and identifying inconsistencies based on item 6 of the SUS12.

specialist nurses, participants of the study tested and analyzed the software giving feedbacks and suggestions for improvements, in order to answer the usability questionnaire, using the free publication of the application on the Google Play Store (http://bit.ly/2rLo1SW).

Data were tabulated in Microsoft Office Excel using descriptive analysis, and then exported to SPSS version 21 software for statistical analysis of the mean scores of each participant’s questionnaire.

The study complied with the ethical principles of research in human beings, established by Resolution 466/2012 CNS/MS, being approved by the Research Ethics Committee under CAAE No. 02226818.1.0000.5192 and opinion No. 3.374.375.

RESULTS

In the survey, 19 nurses specialized in oncology were interviewed to assess usability, with 19/19 (100%) being women with average ages ranging from 24 to 69 years. Regarding the length of experience in the oncology department, the median was 5 years (SD ± 4.95), ranging from 1 to 21 years of service.

According to the respective degree of agreement of the items in the SUS questionnaire, in order to verify the suitability of the application as an auxiliary tool in the early detection of cancer, that is, if it has essential information and structures for this purpose, specialist nurses considered the software as an easy-to-use, accessible, practical device, with very well-grounded content and very useful in the assistance-teaching-learning process (Figure 1).

Figure 1 -
“Fique Atento, pode ser câncer” application assessment regarding use appropriateness in cancer detection by oncology nurse specialists

Seeking to ensure the usability of the application and the quality and satisfaction of end users based on the statistical analysis of the responses obtained through the application of the SUS questionnaire, it was observed that the total value referring to the average of all scores were 91.58, as is and median 95, ranging from 50 to 100 (Table 1).

Table 1 -
Scores obtained from the users’ evaluation using the SUS usability instrument

USER	SUS
1	50
2	92.5
3	97.5
4	95
5	95
6	85
7	100
8	90
9	95
10	100
11	85
12	97.5
13	95
14	97.5
15	100
16	85
17	90
18	90
19	100

• Source: Research data, 2019.

The SUS questionnaire and the items contained have specific usability characteristics with relevant meanings scored by converting value ranges between 0 to 100 points, as a way of quantifying and ratifying the quality components of a software (Table 2).

Table 2 -
Usability characteristic of the “Fique Atento, pode ser câncer” application, evaluated by SUS. Pernambuco. Brazil, 2019.

Usability features	Mean	Meaning
Easy to learn	90.46	Easy to use system when used for the first time.
Efficiency	91.2	Speed in the execution of the established tasks.
Inconsistencies	90.79	Absence or low error rate.
Easy to memorize	96.05	Easy to execute system even after a long period without using it.
Satisfaction	88.6	Pleasant design

• Source: Research data, 2019.

The application obtained the most frequent categorization between the scale of 86-100 (Table 3) as a way to categorize and classify the average usability value proposed individually among the responses of the participants, on a scale from worst achievable to best achievable.

Table 3
Scale of categorization and classification of usability of the “Fique Atento, pode ser câncer” application. Pernambuco. Brazil, 2019

Categorization Scale	Frequency		Usability classification
	N	%
0 - 25	0	0	Worst achievable
26 - 39	0	0	Bad
40 - 52	1	5.3	Acceptable
53 - 74	0	0	Good
75 - 85	0	0	Excellent
86 - 100	18	94.7	Best achievable

• Source: Research data, 2019.

After completing the usability test, nurses proposed suggestions for the application's content, such as: adding information about chemotherapy drug treatment, focusing on care in administration through catheters and, as future projects, requested the availability of the application for the IOS operating system. All suggestions, except the last one, were checked and updated in new versions for the tool’s improvement.

DISCUSSION

Establishing the nurse's profile implies recognizing abilities, skills, experiences and knowledge, aiming at the continuity of patient care, from primary care to high complexity 15.

Primary care is considered to be the care coordinator and the communication center of the health clinics, therefore, the insertion of ICTs in this area improves the quality of care, longitudinality of care, comprehensive services, promotion, prevention, treatment, rehabilitation, disease surveillance, among others; so that the demands for illnesses are minimized16.

Given this fact, the implementation of health technologies, concomitant with the knowledge and experience of the health professional, favors an expanded clinical practice related to diagnosis, care planning and adequate treatment. Emphasizing the importance of a cooperative and articulated action between professionals, parents/guardians and health services17.

Diagnosing cancer in a short period of time requires knowledge on the symptom presentation, exam interpretation, and differential investigation. In order to reduce the frequency of consultations and obtain a therapeutic itinerary with a resolving outcome3.

The use of technologies, among them, mobile devices, are expanding in the health area, precisely to assist and expand knowledge, being made available both to patients and professionals in order to inform, instruct, record, display, guide, remember or alert and communicate10.

According to Brooke 12, the essential software items that indicate quality, contained in the SUS questionnaire, are essential for a good result of the final product, since the usability scale covers a set of aspects of the system such as complexity, need for support, interface, among others. It has a high level of validity to measure the use of the application, as it is a robust and reliable evaluation tool.

Regarding the application’s evaluation based on the SUS questionnaire, the findings of this work reflected a good usability and high user satisfaction, considering that the highest percentages are found in the odd numbered items.

The usability score of “Fique Atento, pode ser câncer” in the SUS questionnaire was superior to another study of development and evaluation of the “ECG Made Easy” application, which obtained a score of 85.318 and regarding the application on total knee autoplasty with an 85.8 score19.

Another application20 also made use of SUS, to assess the usability of an anesthesiology medical educational application, which returned an average score of 90.6. Thus, the application "Fique Atento, pode ser câncer" has a degree of usability on a better achievable scale.

In addition to attributing values, the evaluators reported that the application “Fique Atento, pode ser câncer” is a useful tool, with accessible language, that is easy to handle and understand, with well-integrated and objective, didactic and functional information.

It appears that an application’s level of usability must have a score range over 40 to be made available and considered acceptable. However, the higher the score, the greater the degree of quality and the mobile application’s handling potential.

It is worth mentioning that feedback is essential for evaluation, as it allows the organization, complementation and updating of information, simultaneously assisting weaknesses and improvements to make the application fit for use.

FINAL CONSIDERATIONS

The application "Fique Atento, pode ser câncer" was evaluated with a maximum degree of usability on a "best achievable" scale, with better percentages in terms of easy memorization, efficiency, easy learning and inconsistencies.

Thus, the usability of the application brings an important contribution to the expansion of knowledge, has an innovative character - as it expands the dynamics and the capacity of qualifications for nurses' health actions - and therefore manages to intervene as a tool in order to facilitate the early identification of cancer diagnosis.

The application in education enables the dissemination and updating of knowledge in the area of oncology for children and adolescents, and can be used by anyone, considering its accessible language. Due to the quality of the software, its use in management and care can help in the adoption of governmental, institutional and professional strategies in order to contribute to a lighter and faster therapeutic itinerary for children and adolescents in health, ensuring a less somber prognosis.

The main limitation of the study is the fact that the application is only available for smartphones with the Android operating system, so, in the future, researchers will carry out application procedures for other smartphone operating systems.

We recommend carrying out future work involving a review of the content covered in the application through the suggestions of the participating nurses so that the effect of the application on user assistance can be evaluated.

The aim of this research is to stimulate the interest of other professionals in the development of other applications that contribute theoretical and scientific information, in order to expand knowledge and increase the insertion of technology in the health area, professional updating and care management.

Acknowledgments

To the National Council for Scientific and Technological Development (CNPq) for granting resources and financially supporting research, through the grant awarded.

REFERENCES

• 1. Instituto Nacional do Câncer (BR). Rio de Janeiro: INCA; c2020- [cited 2020 Sep 10]. Câncer infantojuvenil; [about 1 screen]. Available from: https://www.inca.gov.br/tipos-de-cancer/cancer-infantojuvenil
  » https://www.inca.gov.br/tipos-de-cancer/cancer-infantojuvenil
• 2. Lima BC, Silva LF, Góes FGB, Ribeiro MTS, Alves LL. The therapeutic pathway of families of children with cancer: difficulties faced in this journey. Rev Gaúcha Enferm. 2018;39:e20180004. doi: https://doi.org/10.1590/1983-1447.2018.20180004
  » https://doi.org/10.1590/1983-1447.2018.20180004
• 3. Ahrensberg JM, Olesen F, Hansen RP, Schrøder H, Vedsted P. Childhood cancer and factors related to prolonged diagnostic intervals: a Danish population-based study. Br J Cancer. 2013;108(6):1280-7. doi: https://doi.org/10.1038/bjc.2013.88
  » https://doi.org/10.1038/bjc.2013.88
• 4. Silva HP, Elias FTS. Incorporação de tecnologias nos sistemas de saúde do Canadá e do Brasil: perspectivas para avanços nos processos de avaliação. Cad Saúde Pública. 2019;35 Supl. 2:e00071518. doi:https://doi.org/10.1590/0102-311x00071518
  » https://doi.org/10.1590/0102-311x00071518
• 5. Ministério da Saúde (BR). Portaria n° 2.436, de 21 de setembro de 2017. Aprova a Política Nacional de Atenção Básica, estabelecendo a revisão de diretrizes para a organização da Atenção Básica, no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União. 2017 set 22 [cited 2020 Sep 10];154(183 Seção 1):68-76. Available from: https://pesquisa.in.gov.br/imprensa/jsp/visualiza/index.jsp?jornal=1&data=22/09/2017&pagina=68
  » https://pesquisa.in.gov.br/imprensa/jsp/visualiza/index.jsp?jornal=1&data=22/09/2017&pagina=68
• 6. Silva IR, Lima AC, Lima TM. Competências profissionais em saúde: um olhar sobre o câncer infantil na atenção básica [monografia]. Caruauru (PE) Centro Universitário Tabosa de Almeida; 2017 [cited 2020 Sep 10]. Available from: http://repositorio.asces.edu.br/handle/123456789/1335
  » http://repositorio.asces.edu.br/handle/123456789/1335
• 7. Ahrensberg JM, Fenger-Grøn M, Vedsted P. Use of primary care during the year before childhood cancer diagnosis: a nationwide population-based matched comparative study. PLoSOne. 2013;8(3):e59098. doi: https://doi.org/10.1371/journal.pone.0059098
  » https://doi.org/10.1371/journal.pone.0059098
• 8. Pinto LF, Rocha CMF. Innovations in Primary Health Care: the use of communications technology and information tools to support local management. Ciênc Saúde Coletiva. 2016;21(5):1433-48. doi: https://doi.org/10.1590/1413-81232015215.26662015
  » https://doi.org/10.1590/1413-81232015215.26662015
• 9. Tibes CMDS, Dias JD, Zem-Mascarenhas SH. Mobile applications developed for the health sector in Brazil: an integrative literature review. REME: Rev Mineira Enferm. 2014;18(2):471-86. doi:https://doi.org/10.5935/1415-2762.20140035
  » https://doi.org/10.5935/1415-2762.20140035
• 10. Santos AF. Incorporação de Tecnologias de Informação e Comunicação e qualidade na atenção básica em saúde no Brasil. Cad Saúde Pública. 2017;33(5): e00172815. doi: https://doi.org/10.1590/0102-311x00172815
  » https://doi.org/10.1590/0102-311x00172815
• 11. Severino AJ. Metodologia do trabalho científico. São Paulo: Cortez; 2017.
• 12. Brooke J. SUS: a quick and dirty usability scale. 1996. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, editros. Usability evaluation in industry. London: Taylor & Francis; 1996. p. 189-94.
• 13. Tenório JM, Sdepanian VL, Pisa IT, Marin HF. Desenvolvimento e avaliação de protocolo eletrônico para atendimento e acompanhamento do paciente com doença celíaca. Rev Inform Teor Aplic, 2011;17(2):210-20. doi: https://doi.org/10.22456/2175-2745.12119
  » https://doi.org/10.22456/2175-2745.12119
• 14. Bangor A, Kortum P, Miller J. Determining what individual sus scores mean: adding an adjective rating scale. J Usabil Stud. 2009 [cited 2020 Aug 20];4(3):114-23. Available from: https://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_May2009.pdf
  » https://uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS_Bangor_May2009.pdf
• 15. Costa MFBNA, Andrade SR, Soares CF, Ballesteros Pérez EI, Tomás SC, Bernardino E. The continuity of hospital nursing care for Primary Health Care in Spain. Rev Esc Enferm USP. 2019;53:e03477. doi: https://doi.org/10.1590/s1980-220x2018017803477
  » https://doi.org/10.1590/s1980-220x2018017803477
• 16. Silva MEA, Reichert APS, Souza SAF, Pimenta EAG, Collet N. Chronic disease in childhood and adolescence: family bondsin the healthcare network. Texto Contexto Enferm. 2018;27(2):e4460016. doi:https://doi.org/10.1590/0104-070720180004460016
  » https://doi.org/10.1590/0104-070720180004460016
• 17. Alpirez LA, Lopes Neto D, Moisés MS, Dias VP. Content validation of an infant evaluation instrument. Acta Paul Enferm. 2018;31(2):123-9. doi:https://doi.org/10.1590/1982-0194201800019
  » https://doi.org/10.1590/1982-0194201800019
• 18. Lima CJM, Coleho RA, Medeiros MS, Kubrusly M, Marçal E. Development and validation of a mobile application for the teaching of electrocardiogram. Rev Bras Educ Med. 2019;43(1):157-65. doi: https://doi.org/10.1590/1981-5271v43suplemento1-20190164.ing
  » https://doi.org/10.1590/1981-5271v43suplemento1-20190164.ing
• 19. Nogueira, JBS. Desenvolvimento e avaliação de usabilidade de aplicativo para planejamento de artroplastias totais de joelho [dissertação]. Fortaleza (CE): Centro Universitário Christus; 2016 [cited 2020 Jul 15]. Available from: https://unichristus.edu.br/wp-content/uploads/2017/06/DESENVOLVIMENTO-E-AVALIAC%CC%A7A%CC%83O-DE-USABILIDADE-DE-APLICATIVO-PARA-PLANEJAMENTO-DE-ARTROPLASTIAS-TOTAIS-DE-JOELHO.pdf
  » https://unichristus.edu.br/wp-content/uploads/2017/06/DESENVOLVIMENTO-E-AVALIAC%CC%A7A%CC%83O-DE-USABILIDADE-DE-APLICATIVO-PARA-PLANEJAMENTO-DE-ARTROPLASTIAS-TOTAIS-DE-JOELHO.pdf
• 20. Pereira RVS, Kubrusly M, Marçal E. Desenvolvimento, utilização e avaliação de uma aplicação móvel para educação médica: um estudo de caso em anestesiologia. Renote. 2017;15(1) doi: https://doi.org/10.22456/1679-1916.75104
  » https://doi.org/10.22456/1679-1916.75104

Publication Dates

• Publication in this collection
  16 July 2021
• Date of issue
  2021

History

• Received
  15 Oct 2019
• Accepted
  12 Nov 2020