Mobile web application for use in the Extended Family Health and Primary Care Center: content and usability validation

ABSTRACT

Purpose:

to validate the content and usability of the “Network NASF” application, intended for the teams of the Extended Family Health and Primary Care Center (NASF-AB).

Methods:

eighteen specialists, researchers, and professionals from different fields of study participated to validate the content and usability of the application, carried out in four stages: adjustment of the instrument; administration of the Suitability Assessment of Materials (SAM); validation of the content by calculating the content validity index (CVI); and usability evaluation through the System Usability Scale (SUS), in this order.

Results:

the participants classified the material as valid regarding both its content and usability. The index achieved in the SAM was 83.5%, as four, out of the six topics in the instrument, had values over 0.78. Hence, these four were considered excellent, while the other two were considered good. The recommendations given by the specialized judges were accepted and the usability index (5.5%) was considered relevant.

Conclusion:

the application developed for NASF-AB professionals was considered valid regarding its content and usability.

Keywords:
Validation; Technology in Health; Primary Health Care

RESUMO

Objetivo:

validar por conteúdo e usabilidade o aplicativo “NASF em Rede” direcionado para as equipes dos Núcleos Ampliados de Saúde da Família e Atenção Básica (NASF-AB).

Métodos:

estudo metodológico, no qual participaram 18 especialistas, pesquisadores e profissionais, com diferentes áreas de formação, com vistas à validação de conteúdo e de usabilidade do APP. Realizado em quatro etapas: adaptação do instrumento; aplicação do instrumento de Avaliação de Adequação de Materiais (SAM); validação de conteúdo por meio do cálculo do Índice de Validade de Conteúdo (IVC) e avaliação da usabilidade por meio do System Usability Scale (SUS), respectivamente.

Resultados:

os participantes classificaram o material como válido, tanto no que concerne ao conteúdo quanto à usabilidade. Obteve-se índice de 83,5% para o instrumental SAM, pois 04 dos 06 tópicos constantes no instrumento apresentaram valores de 0,78, os quais foram considerados excelentes e dois deles foram avaliadas como bons, sendo acolhidas as recomendações dos juízes especialistas. O índice de usabilidade (5,5%) também foi considerado relevante.

Conclusão:

o aplicativo desenvolvido para profissionais do NASF- AB foi considerado válido quanto ao conteúdo e à usabilidade.

Descritores:
Validação; Tecnologia em Saúde; Atenção Primária em Saúde

Introduction

The Extended Family Health and Primary Care Center (NASF-AB, in Portuguese), emerged from the need to include other professional categories in the primary health care (PHC), The intention was to broaden the field of action and solutions in primary care, diminishing the demand for secondary and tertiary care. Based on the same primary care guidelines, the NASF-AB is dedicated to health promotion, prevention, treatment, and rehabilitation in the region under its responsibility. Hence, its relevance is due to multi-professional health care support. Its role includes also technical and training support to the Family Health Strategy (ESF, in Portuguese) teams, whose work process is guided by theoretical-methodological reference material of the matrix support.

However, more than a decade after NASF was implemented, there are still challenges to be overcome for it to offer comprehensive health care. Particularly, there are difficulties related to the organization of the work processes and the low degree of integration with the other local and regional service networks^1-3.

Hence, using information and communication technologies (ICTs) can help integrate the actions through technical-educative and technical-assistive support. Moreover, the ICTs enable knowledge in the field of health to be communicated, widely spread, and updated, which aids in decision-making. The ICTs can also help supply the health needs of different geographical regions, furnishing a wide health assistance coverage⁴. Nevertheless, despite all the advantages pointed out, the process of using ICTs in PHC in Brazil is still incipient, as few PHC teams have a high-level ICT, even in more developed regions, like the South and Southeast⁴.

Furthermore, there is a shortage of studies approaching the use of ICTs in the context of the NASF-AB. The few scientific productions conducted in PHC are limited to the family health teams. It is believed, though, that the use of technology in the field of health by the NASF-AB teams can enable a more integrated work, consequently leading to advances in the management of comprehensive care with more and better results.

Thus, the “Network NASF” mobile application was developed. Its purpose is to overcome the fragmented management of care at the NASF-AB, helping the professionals offer care and promote the integration between the health services in the municipality of Fortaleza.

It encompasses a mobile web application - i.e., an Android-compatible mobile application named “Network NASF”. It is a technology that potentializes the teams’ work and directs and guides the professionals, strengthening integrative practices in primary health care.

The “Network NASF” enables public and private health services to be visualized, according to the users’ needs and profiles. Hence, it provides the NASF professionals access to data of the service (the type of service offered, its link to the health system, and opening hours), which makes it easier to take steps within the health care system. Hence, this study aimed to validate the content and usability of the “Network NAF” application intended for the Extended Family Health and Primary Care Center (NASF-AB).

Methods

This is a methodological study, addressing the validation of the “Network NASF” application. This type of study investigates, organizes, and analyzes data to aid in the development, adjustment, and validation of instruments and techniques for research and/or practice that uses mixed methods centered on the development and employment of specific tools to improve these instruments’ reliability and validity⁵.

The validation determines the choice and/or employment of a measuring instrument and is itself measured by the extent or degree to which the datum represents the concept the instrument is meant to measure⁶.

The “Network NASF” application was developed based on an integrative review, in which scientific articles on the NASF were selected, besides normative material from the Ministry of Health and focus groups with professionals who worked at the NASF-AB.

The application is structured with an initial page to register the device, which directs the user to a registration form. In it, they can contribute to the project, providing data of the services they use or offer, besides logging in a password and username to access the available functions. The device-filtering page screens the devices and gives quick access to those that meet the user’s specific needs, such as age group, type of service offered, link to the health system, and opening hours. Multiple selections for the same group are an enabled option, which returns a list of devices that correspond to what was required.

The study took place in the municipality of Fortaleza, the capital city of Ceará, Brazil. The municipality is divided into seven administrative Regional Departments (RD); therefore, seven judges participated in the study for content validation, as well as 11 NASF-AB professionals who worked in RD II and IV, for usability validation.

The criteria to select the judges considered their professional experience and qualification. They encompassed: doctor’s degree - 3 points; master’s degree - 2 points; master’s degree whose dissertation is relevant to the field (public health or ICT) - 1 point; research published in the field (public health or ICT) - 2 points; articles published in the field (public health or ICT) - 2 points; professional experience in the NASF-AB or PHC (at least one year) - 1 point. Those who achieved at least five points were selected, according to an adapted Fehring Model scoring system⁷.

The judges were selected through a search on the Lattes Platform. A total of 20 judges were invited via e-mail to participate in the study; seven of these responded, indicating their willingness to take part in the validation. Then, they were sent a link to the application, along with the validation instruments and consent form.

As for the health professionals at the NASF-AB, the selection was made through snowball sampling; hence, 11 participants were chosen for the usability test.

Following their positive answer, the participants were sent the informed consent form (ICF). Also, the judges received a content validation questionnaire according to SAM (Suitability Assessment of Materials)^8,9; and the professionals, a usability validation instrument, according to SUS (System Usability Scale)¹⁰.

The content validity consists of an instrument analysis by a committee of specialists on a certain construct. This validation is based on the applicability of the checklist formed by six categories: content, text intelligibility, graphics, presentation, motivation, and cultural suitability - totaling 22 items.

The content validation is based on a questionnaire structured on a Likert-type scale. According to Polit and Beck⁵, this scale consists of various items expressing one’s standpoint on a given topic. It is an ordinal scale that considers the following items: 1- Partially adequate; 2- Adequate; 0- Inadequate.

In this approach, the respondents were asked to indicate their agreement in each dimension of the application, enabling the judges to evaluate all the items. Space was provided for them to make suggestions on each item. At the end of the questionnaire, the specialists made an overall evaluation of the instrument, using the same criteria, through an open-ended question, which allowed them to make suggestions regarding the whole instrument.

The usability evaluation, in its turn, refers to the quality of the user’s experience when interacting with products, systems, applications - a crucial aspect for the quality of applications. The validation process was carried out through consultation with judges specialized in the field. In the validation, the number and qualification of the judges were considered. Some studies recommend five to ten judges, while others mention six to twenty specialists¹¹.

The usability evaluation was conducted through the administration of the SUS, also structured on a Likert scale. It is an English language questionnaire translated into Portuguese in stages that are highlighted in the paper by Tenório, Cohrs, Sdepanian, Pisa, and Marin¹⁰. The method was created by John Brooke in 1986 and is useful to evaluate products, services, hardware, software, websites, applications, and others.

The content validity index was based on the answers from the groups, considering the sum of “3” and “4” answers divided by the total number of answers. The minimum value of 0.78 is required in the CVI⁵, which was also adopted in this study.

This investigation was approved by the Research Ethics Committee of the institution of origin, through the evaluation report no. 1.710.586. It also complied with all the ethical stipulations of Resolution no. 466, of 2012.

Results

Characterization of the Participants

The participants had been working in the field of health for 1 to 5 years (44.40%)¹², less than one year (22.20%)¹³, six to 10 years (16.68%)⁴, and more than 15 years (11.10%). Such indicators show the relevance of being acquainted with the region to plan articulated actions between primary care and psychosocial care.

The profile of the participants in this study is verified in Figure 1. Of the 18 participants, the judges include occupational therapists, social workers, and psychologists; each category was represented by 16.4%; there were also 5.6% of researchers in the field. As for the NASF professionals, there were physical educators, physical therapists, nutritionists, and pharmacists, each in a percentage of 11.3% (Figure 1).

Figure 1:
Characterization of the participants, according to their field of work/studies. Fortaleza, CE, Brazil, 2019

The judges with one to five years of professional training correspond to 38.90% of the total¹⁴; there were six professionals with more than 15 years, a percentage of 33.30%. These percentages are relevant as indicators of the promotion of knowledge to work in the field of mental health. Thus, continuing education is necessary, as well as the debate around the problems in the service itself, which opens a way for decision-making.

Content Validation

For this evaluation, a checklist was administered, which was formed by six categories (content, text intelligibility, graphics, presentation, motivation, and cultural suitability), totaling 22 items. Its score was based on a zero-to-two scale, as follows: 0 - inadequate; 1 - adequate; 2 - totally adequate. This was also applied after reading the material.

The total adjustment score was calculated from the sum of the scores, divided by the total scores, multiplied by 100 to transform it into a percentage. In all situations, the SAM estimate percentage was interpreted as follows: 70-100% (high-quality material), 40-69% (adequate material), and 0-39% (inadequate material), as proposed by Doak, Doak and Root⁸. The items given a score lower than or equal to 39% (inadequate material) by more than 50% of the judges were modified. As the calculation resulted in a percentage of 83.5%, it was considered in the 70-100% category (high-quality material) in the application’s usability evaluation in the NASF-AB.

Table 1:
Distribution of the content validity indexes for the application, according to the analysis by the specialized judges. Fortaleza, CE, Brazil, 2019

1.Content (CVI = 0.81)	RF	%	CVI
1.1 The purpose is evident, making it easy to readily understand the material	33	32.03	0.88
1.2 The content approaches information relevant to the patient’s assistance	24	23.31	0.89
1.3 The proposal of the material is limited to its purposes, so the professional can reasonably understand it in the time given to them	23	22.33	0.76
1.4 Summary or review included	23	22.33	0.76
Total	103	100%	-
2. Language (CVI = 0.85)	RF	%	CVI
2.1 The reading level is adequate to the professional’s understanding	30	22.39	0.93
2.2 The conversational style makes it easier to understand the text	31	23.13	0.91
2.3 The vocabulary uses common words	30	22.39	0.88
2.4 The context is given first	27	20.15	0.84
2.5 The material promotes learning	16	11.94	0.65
Total	134	100%	-
3. Graphics (CVI = 0.75)	RF	%	CVI
3.1 Purposeful graphics	25	21.00	0.78
3.2 Types of graphics	22	18.50	0.68
3.3 Relevance of the illustrations	25	21.00	0.78
3.4 Lists and tables are explained	24	20.18	0.75
3.5 Legends are used for graphic illustration	23	19.32	0.71
Total	119	100%	-
4. Layout and typography (CVI = 0.81)	RF	%	CVI
4.1 Layout factors	26	34.67	0.78
4.2 Typography	23	30.66	0.73
4.3 Subtitles used	26	34.67	0.81
Total	75	100%	-
5. Motivation (CVI = 0.71)	RF	%	CVI
5.1 The text and/or images interact with the reader, leading them to solve problems, make choices, and/or show skills	21	33.33	0.68
5.2 The desired behavior patterns are well modeled or well demonstrated	23	36.50	0.70
5.3 There is motivation to self-sufficiency - i.e., people are motivated to learn as they believe the tasks and behaviors are feasible	19	30.17	0.75
Total	63	100%	-
6. Cultural suitability (CVI = 0.80)	RF	%	CVI
6.1 The material is culturally suited to the logic, language, and experience of the target audience	28	50.00	0.82
8.2 It presents culturally adequate images and examples	28	50.00	0.78
Total	56	100%	-

• Source: Adapted from Doak, Doak and Root⁸.
• Legend: CVI (content validity index), RF (relative frequency).

Regarding the CVI, the reference values adopted for validation were CVI ≥ 0.78: excellent; CVI between 0.60 and 0.77: good; CVI < 0.59: bad - as proposed by Polit, Back, Owen¹⁵. To this end, the data were entered in a spreadsheet; then, the CVI was calculated by item and specialist in each dimension. Thus, the degree of agreement between the judges was verified.

Of the six topics in the instrument, four had a CVI higher than 0.78, considered excellent, namely: language (CVI = 0.85); Content (CVI = 0.81); cultural suitability (CVI = 0.80), layout and typography (CVI = 0.81). Graphics (CVI = 0.75) and motivation (CVI = 0.71) were evaluated as good, and the recommendations given by the specialists were taken into consideration.

Table 2:
Usability evaluation by NASF-AB professionals

Items	N°	%
1 I believe I would like to use this application frequently	21	9.90
2 I found the application unnecessary/complex	18	8.50
3 I found the application easy to use	26	12.26
4 I believe I would need technical support to use this application	17	8.02
5 I found the various functions in this application well integrated	23	10.84
6 I found this application very inconsistent	23	10.84
7 I believe most people would quickly learn how to use this application	22	10.38
8 I found the application too heavy to be used	20	9.44
9 I felt quite confident as I used it	20	9.44
10 There are many things I need to learn before I continue using this application	22	10.38
	212	100.00

• Source: Developed by the authors.
• Legend: N (Number of answers from the participants).

The usability evaluation was calculated as follows: one point was subtracted from odd answers (1, 3, 5), and from the even answers (2, 4), the answer of five was subtracted. The final mean was achieved by multiplying the result by 2.5. The SUS mean in this tool is 5.5%. The final score can range from 0 to 100 points.

Formula: 212 x 2.5 = 530 = 5.3%.

Hence, the instrument presented content validity index (83.5%) through the SAM, in which four dimensions were evaluated as excellent, and two, as good - considering the degree of agreement between the specialists, based on the CVI analysis. As for the usability indexes, they were evaluated as relevant.

Discussion

The application is intended to make it easier for users to take steps within the health care system, promoting greater integration between the actions and services. Hence, it provides information to the NASF professionals regarding the health services available in the network in their different levels of health care.

The data indicate that the professionals who work at the NASF-AB remain little time in the service, mainly due to the fragile employment relationships. Such a situation interferes with the health care management in the region; whenever someone new is hired, this professional will require some time invested in learning the proceedings and equipment available. In this context, the application is a tool that facilitates the professionals’ work process in the health system.

Onocko-Campos, Ferrer, Porto, Santos, Stefanello, Trapé, et al.¹⁶⁾ consider: “The fragile relationship between the services is due to the constant change of both professionals and health care models in every new municipal election, which interrupts the continuity” (p. 4645) of the services.

Souza and Medina³ highlight: “Including NASF professionals in the PHC either reinforces the fragmentations of the work in health or foments the integration of practices” (p. 146). This reflects the importance of health professionals’ work as they seek to integrate the many paths of health care. Therefore, constant updates of knowledge and advanced technologies are made possible, promoting comprehensive care.

To verify the validity of new instruments in general, some authors suggest a minimum agreement of 75% for it to be considered of good content validity¹¹. The items that were evaluated as partially adequate (1) were revised, and those considered inadequate (0) were discarded, as proposed by Polit and Beck⁵.

In a similar study, in which an instrument for health professionals was developed, nine out of the 28 items in the checklist needed changes - especially related to clarity, the instructional sequence of topics, and vocabulary. In the overall evaluation, the instrument obtained CVI of 0.94. The suggestions from the judges were accepted, aiming to make the items clearer, and the reading, understanding, and applicability of the instruments, easier¹⁷.

In general, the judges’ responses were concordant, as seen in the results presented. Based on the answers given by the content judges in the SAM (Table 2), the mean score was considered relevant for reliability and agreement of the high-score answers.

Doak, Doak and Root⁸, authors and developers of the SAM, consider that, for the educative material to be adequate, it must have a value equal to or higher than 40% in relation to the total scores of the instrument. This calculation is made by summing the total scores obtained and dividing the result by the total items in the questionnaire.

With the constant and intense changes taking place nowadays, in which technological innovation increases at a fast pace, various types of technology - such as educational, managerial, and assistive technologies - are made available to professionals and users¹⁸. Hence, it is important to have a critical and reflexive attitude regarding the use of technology, in the effort to make it suitable for the health needs in general.

Usability must be considered an important factor in the development of systems, from the blueprint to implementation. Therefore, the information systems must be continuously submitted to usability assessments to verify whether its interface is indeed efficient and satisfactory for users.

Regarding the management of public health care - in which the NASF and PHC teams need to work together towards the patients’ health care - this tool aims to contribute to overcoming some of the obstacles. Thus, usability is one of the main quality aspects of any interactive product. Usability in touchscreen mobile devices is essential and must be taken into consideration when a new product is launched. This could be a differential in a hasty market such as that of the present-day mobile devices¹⁹.

Obstacles, as described by Sousa, Albuquerque, Nascimento, Albuquerque and Lira²⁰, seem to cause or worsen these limitations - e.g., the lack of meetings between the teams of the PHC and other specialized services, the inexistence of electronic medical records, and improper use of referral sheets - which could help pass on information.

Hence, the instrument proposed can be useful to professional teams in tracking the integration between the NASF-AB and the primary health care. As a limitation to this study, there is the use of a non probabilistic sample; also, the application was restricted for use in the territorial limits of the municipality of Fortaleza, Ceará, Brazil.

Conclusion

The web application developed for the NASF-AB professionals was validated regarding its content and usability by judges and assistive professionals. The content validity index achieved a value considered adequate, making the product valid and apt for use in the health services.

The judges’ participation enabled adjustments and improvements to be made in the technology, as these professional’s suggestions were highly important to make the material better, adding knowledge and value to its final version.

The involvement of professionals from different areas was relevant in the process of validating the technology. It furnished a positive evaluation of the material’s suitability with a differentiated look brought by specialists in their field.

Acknowledgment

Gratitude is extended to the Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico (FUNCAP), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPQ), and to the Ministry of Health for the grant given for the project “Management in Shared Networks: Technology and innovation settings for the primary health care”.

Gratitude is also extended to those given a grant from the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), as they helped us develop the data collection.

REFERENCES

• ¹
  Brocardo D, Andrade CLT, Fausto MCR, Lima SML. Núcleo de Apoio à Saúde da Família (Nasf): panorama nacional a partir de dados do PMAQ. Saúde debate. 2018;42(1):130-44.
• ²
  Campos GWS, Pereira Júnior N. Primary care in Brazil, and the Mais Médicos (More Doctors) Program in the Unified Health System: achievements and limits. Ciênc. saúde coletiva. 2016;21(9):2655-63.
• ³
  Souza TS, Medina MG. NASF: fragmentação ou integração do trabalho em saúde na APS? Saúde debate. 2018;42(2):145-58.
• ⁴
  Santos AF, Fonseca Sobrinho D, Araujo LL, Procópio CSD, Lopes EAS, Lima AMLD et al. Incorporação de tecnologias de informação e comunicação e qualidade na atenção básica em saúde no Brasil. Cad. Saúde Pública [periódico na Internet]. 2017 [acesso em: 25 abr. 2019]; 33(5):1-14. Disponível em: http://www.scielo.br/pdf/csp/v33n5/1678-4464-csp-33-05-e00172815.pdf
  » http://www.scielo.br/pdf/csp/v33n5/1678-4464-csp-33-05-e00172815.pdf
• ⁵
  Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. 7 ed. Porto Alegre: Artmed; 2011.
• ⁶
  Bittencourt HR, Creutzberg M, Rodrigues ACM, Casartelli AO, Freitas ALS. Desenvolvimento e validação de um instrumento para avaliação de disciplinas na educação superior. Est Aval Educ. 2011;22(48):91-114.
• ⁷
  Fehring RJ. The Fehringmodel. In: Carrol-Jonhnson RM, Paquete M (eds). Classification of nursing diagnoses: proceedings of the Tenth Conference, 1994. Philadelphia: J.B. Limppincott; 1994. p. 55-62.
• ⁸
  Doak CC, Doak LG, Root JH. Learner verification and revision of materials. In: Doak CC, Doak LG, Root JH (eds). Teaching patients with low literacy skills. 2nd ed. Philadelphia: Lippincott Company; 1996. p. 167-88.
• ⁹
  Sousa CS, Turrini RNT, Proveda VB. Translation and adaptation of the instrument "suitability assessment of materials" (sam) into Portuguese. Rev Enf UFPE. 2015;9(5):7854-61.
• ¹⁰
  Tenório JM, Cohrs FM, Sdepanian VL, Pisa IT, Marin HF. Desenvolvimento e avaliação de um protocolo eletrônico para atendimento e monitoramento do paciente com doença celíaca. RITA. 2010;17(2):210-20.
• ¹¹
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc. saúde coletiva. 2011;16(7):3061-8.
• ¹²
  Sukumar SR, Natarajan R, Ferrel RK. Quality of Big Data in health care. Int J of Health Care Qual Assur. 2015;28(6):621-34.
• ¹³
  Pinto LF, Rocha CMF. Innovations in Primary Health care: the use of communications technology and information tools to support local management. Ciênc. saúde coletiva [periódico na Internet]. 2016 Maio [acesso em: 25 abr. 2019]; 21(5):1433-48. Disponível em: https://www.scielo.br/pdf/csc/v21n5/en_1413-8123-csc-21-05-1433.pdf
  » https://www.scielo.br/pdf/csc/v21n5/en_1413-8123-csc-21-05-1433.pdf
• ¹⁴
  Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and satisfaction with a New eConsult Service: a mixed-methods study of primary care providers. J Am Board Fam Med. 2015;28(3):394-403.
• ¹⁵
  Polit DF, Beck CT, Owen SV. O CVI é um indicador aceitável de validade de conteúdo? Avaliação e recomendações. Pesquisa enferm saúde. 2007;30(4):459-67.
• ¹⁶
  Onocko-Campos RT, Ferrer AL, Porto K, Santos DVD, Stefanello S, Trapé TL et al. Saúde mental na atenção primária à saúde: estudo avaliativo em uma grande cidade brasileira. Ciênc. saúde coletiva [periódico na internet]. 2011 [acesso em: 14 mar. 2019]; 16(12):4643-53. Disponível em: http://www.scielo.br/pdf/csc/v16n12/13.pdf
  » http://www.scielo.br/pdf/csc/v16n12/13.pdf
• ¹⁷
  Tiburcio MP, Melo GSM, Balduíno LSC, Costa IKF, Dias TYAF, Torres GV. Validação de instrumento para avaliação da habilidade de mensuração da pressão arterial. Rev Bras Enferm. 2014;67(4):581-7.
• ¹⁸
  Barra DCC, Paim SMS, Sasso GTMD, Colla GW. Methods for developing mobile apps in health: an integrative review of the literature. Texto contexto - enferm. 2017;26(4):1-12.
• ¹⁹
  Nilsson EG. Design patterns for user interface for mobile applications. Adv Eng Softw. 2009;40(12):1318-28.
• ²⁰
  Sousa FOS, Albuquerque PC, Nascimento CMB, Albuquerque LC, Lira AC. The role of Family Health Support Center in assistance coordination of Primary Health Care: limits and possibilities. Saúde debate [periódico na Internet]. 2017 [acesso em: 04 maio 2020]; 41(115):1075-89. Disponível em: https://www.scielo.br/pdf/sdeb/v41n115/en_0103-1104-sdeb-41-115-1075.pdf
  » https://www.scielo.br/pdf/sdeb/v41n115/en_0103-1104-sdeb-41-115-1075.pdf

• Research support source: Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico (FUNCAP).

Publication Dates

• Publication in this collection
  17 July 2020
• Date of issue
  2020

History

• Received
  22 May 2019
• Accepted
  09 June 2020