Intra-and inter-examiner reliability of Alberta Infant Motor Scale application in follow-up ambulatory of at-risk newborns

ABSTRACT

Prematurity is a risk factor for delayed motor development, and it is recommended to monitor these infants in the first two years of life. To verify the properties of intra and inter-examiner measurements of AIMS in an outpatient follow-up clinic for newborns at risk in a public maternity hospital. Prospective study conducted in an outpatient follow-up of high-risk newborns. The Intraclass Correlation Coefficient (ICC) was used to analyze reliability. To compare the intra-examiner evaluations, the paired T-test or Wilcoxon test was performed. The independent T-test was used to compare inter-examiner assessments. The correlation between variables was analyzed using the Pearson or Spearman test. The Bland Altman test was performed to assess the concordance between the scores. 31 preterm infants with 8,47 ± 4,49 of corrected age were evaluated. There was no significant difference between the evaluations intra and inter-examiner. The ICC values remained above 0.88 for both intra and inter-examiner evaluation. The scores showed high agreement. AIMS has intra- and inter-examiner reliability for assessing and monitoring preterm newborns for up to 18 months in a follow-up clinic.

Keywords | Motor Development; AIMS; Reliability

RESUMO

A prematuridade é fator de risco para atraso do desenvolvimento motor, e recomenda-se o acompanhamento desses lactentes nos primeiros dois anos de vida. Verificar a confiabilidade intra e interexaminadores da Escala Motora Infantil de Alberta (EMIA) em ambulatório de seguimento de recém-nascidos de risco de uma maternidade pública. Estudo prospectivo realizado em ambulatório de seguimento de recém-nascidos de risco. As avaliações do desenvolvimento motor foram realizadas por meio da EMIA, por dois avaliadores previamente treinados. O Coeficiente de Correlação Intraclasse (CCI) foi utilizado para análise das confiabilidades. Para a comparação entre as avaliações intraexaminadores foi realizado o Teste T pareado ou Teste de Wilcoxon. O Teste T independente foi utilizado para comparar as avaliações interexaminadores. A correlação entre as variáveis foi analisada a partir do Teste de Pearson ou Spearman. Para avaliar a concordância entre os escores foi realizada análise de Bland Altman. Foram avaliados 31 recém-nascidos pré-termo (RNPT) com idade corrigida média de 8,47 ± 4,49. Não houve diferença significativa entre as avaliações intraexaminadores e interexaminadores. Os valores de CCI se mantiveram acima de 0,88 para a confiabilidade intraexaminadores e interexaminadores. Os escores apresentaram alta concordância, analisada por meio do teste de Bland Altman. EMIA apresentou adequada confiabilidade intra e interexaminadores para avaliação e acompanhamento de RNPT até 18 meses em ambulatório de seguimento de lactentes de risco.

Descritores | Desenvolvimento Motor; EMIA; Confiabilidade

RESUMEN

La prematuridad es un factor de riesgo de retraso en el desarrollo motor de los lactantes y se recomienda la monitorización de ellos durante los dos primeros años de vida. Verificar la fiabilidad intraevaluadores e interevaluadores de la Escala Motora Infantil de Alberta (EMIA) en un seguimiento ambulatorio de recién nacidos de riesgo en una maternidad pública brasileña. Estudio prospectivo realizado en un seguimiento ambulatorio de recién nacidos de riesgo. Para evaluar el desarrollo motor, la EMIA fue empleada por dos evaluadores previamente capacitados. Se utilizó el coeficiente de correlación intraclase (CCI) para analizar la fiabilidad. Para comparar las evaluaciones intraevaluadores, se utilizó la prueba T pareada o la prueba de Wilcoxon. La prueba T independiente se utilizó para comparar las evaluaciones interevaluadores. La correlación entre las variables se analizó mediante la prueba de Pearson o Spearman. Para evaluar la concordancia entre los puntajes, se aplicó el análisis de Bland Altman. Se evaluaron a 31 recién nacidos pretérmino (RNPT) con un promedio de edad media corregida de 8,47 ± 4,49. No hubo diferencias significativas entre las evaluaciones intraevaluadores e interevaluadores. Los valores de CCI se mantuvieron por encima de 0,88 para la fiabilidad intraevaluadores e interevaluadores. Los puntajes mostraron un alto nivel de concordancia, que se analizó mediante el Bland Altman. La EMIA apuntó una adecuada fiabilidad intra e interevaluadores para evaluar y monitorear los RNPT hasta 18 meses en seguimiento ambulatorio de lactantes de riesgo.

Palabras clave | Desarrollo Motor; EMIA; Fiabilidad

INTRODUCTION

Prematurity is one of the major causes of infant mortality and morbidity, both in the neonatal and early childhood periods, due to general immaturity that can lead to systemic changes and dysfunctions, in addition to cognitive, motor, communicative, behavioral, learning and sensory delays^1),(2. Brazil ranks 10th in the world ranking of premature births, with an index of 11.5% of premature births, which generates an increase in the daily cost of hospitalization in a neonatal intensive care unit (ICU), according to studies^1),(3.

In the neonatal ICU, besides biological risk factors due to the low gestational age that imply several alterations in the essential systems⁴, the preterm newborn (PTNB) is exposed to environmental factors that will also contribute to changes in its motor development. Hospitalized in the neonatal ICU, the PTNB is subjected to a series of invasive and painful procedures, restricted physical space, absence of adequate stimuli, hospital routine that affect their mobility, contribute to a delayed and deficient motor development⁴.

Detecting these factors and minimizing them early, performing a careful evaluation in the first years of life and identifying disorders in motor development, allows determining an appropriate intervention that allows children with delays to follow the same pace of acquisitions of a child with normal motor development^4),(5. Evidence also shows that an early intervention program is more effective during the first two years of life due to the high neuroplasticity of PTNB, and, therefore, after the hospitalization period, multiprofessional follow-up in an outpatient clinic for at-risk newborns until the first two years of life is recommended^6),(7.

The scales for evaluating the motor development of infants such as Bayley Scales of Infant Development and Peabody Developmental Motor Scales, despite being reliable validated and tested in several countries, have high cost and require specific training⁸. The Alberta Infant Motor Scale (AIMS), however, has been widely used in the evaluation of motor skills because it is easy and fast to apply, and has been shown to be sensitive to the detection of motor deficits⁹. This scale was developed by Piper and Darrah in 1994 as an observational measurement scale of broad motor function used in term and preterm infants above forty gestational weeks, until the acquisition of independent gait¹⁰. AIMS was validated for the Brazilian population by Valentini and Saccani¹¹ and the results showed adequate reliability and validity for use in monitoring the development of Brazilian infants.

Previous studies have analyzed some AIMS measurement properties, but the measurement properties in preterm infants followed in follow-up outpatient clinic for at-risk newborns have not been studied. Thus, our study aims to verify the intra- and inter-examiner reliability of the Alberta Infant Motor Scale in a follow-up outpatient clinic for at-risk newborns.

METHODOLOGY

A prospective study conducted in the follow-up outpatient clinic for at-risk newborns of a reference maternity hospital in southern Brazil. The guardians were informed about the objective of the study and the procedures to be performed with the infants; after agreeing, all of them signed an informed consent form and the image and video consent form.

Infants of both sexes, between 0 and 18 months of corrected age, under follow-up at the outpatient clinic of the maternity, participated in the study; infants with a clinical history of congenital malformation, genetic syndromes, grade III and IV periventricular hemorrhage, clinically instable infants, and those who did not complete the study protocol were excluded. The sample was non-probabilistic and intentional.

The sample was characterized by means of an instrument specific to the study and answered by the guardians. The evaluations of motor development were made using the AIMS script, in a reserved environment, by two examiners previously trained. The training of the examiners consisted of 18 hours of theoretical-practical classes on the application of AIMS, in addition to practical experience in the application of assessments with AIMS in infants. The scale contains 58 items, divided into four subscales: prone (21 items), supine (nine items), sitting (12 items) and standing (16 items). The evaluation is observational, follows a script analyzing three points, namely: antigravitational movement, weight discharge and postural control¹⁰. The one-point score is given to each of the items that the infant performs and those before the opening of the motor window of each subscale, thus, the sum of the four subscales generates the total score¹⁰. Thus, by the total score and corrected age of the infant, the percentile is estimated, and infants with percentile below 25% were considered at risk^11),(12. In case of agitation and excessive crying, or any behavioral changes, the evaluation was interrupted until the infant returned to the initial condition, so as not to interfere with the results.

A Nikon Coolpix L120® camera was used, positioned on a tripod adjusted at a height of one meter, angle of 45° to 1.5 meters away from both the examiner and the infant.

For the inter-examiner reliability protocol, two examiners simultaneously analyzed the infants, marked their score, but an examiner made physical contact with the child during postures. The evaluations were timed and the order of the examiners that would apply the AIMS was randomized. To comply with the intra-examiner reliability protocol, the footage was evaluated again by the same examiner, between seven and fifteen days after the first evaluation, to reduce memory bias¹².

After the evaluations, the results were tabulated and the data analyzed in the Statistical Package for the Social Sciences version 20.0. The data were presented as measures of central tendency and dispersion, according to the normal distribution of the data. The Intraclass Correlation Coefficient (ICC) was used to analyze intra-examiner reliability. For the comparison between intra-examiner reliability assessments, the paired T Test or Wilcoxon test was used, according to the normal distribution of the data. The independent T-test was used to compare inter-examiner reliability assessments. The correlation between the variables was analyzed using the Pearson or Spearman correlation test, according to the data normal distribution. Bland-Altman analysis was applied to analyze if there is agreement between the scores of the evaluations of intra- and inter-examiner reliability from the mean and standard deviation of the differences and the lower and upper limits of agreement. The significance level determined was p<0.05 for all statistical analyses.

RESULTS

Thirty-one infants with a mean corrected age of 8.47±4.49 months were evaluated, and 10 evaluations were used for intra-examiner reliability analysis 1; 10 for intra-examiner reliability analysis 2; and 11 for inter-examiner reliability, whose data on the characterization of infants are shown in Table 1.

Table 1
Characterization of infants participating in intra- and inter-examiner reliability

Variables	Intra-examiner 1	Intra-examiner 2	Inter-examiner	Total	p-value
	(n=10)	(n=10)	(n=11)	(n=31)
Sex	6M/4F	8M/2F	6M/5F	20M/11F	-
GA (weeks)	33.16 (3.50)	33.21 (2.32)	32.80 (1.62)	33.06(2.69)	0.89
CA (months)	10.08 (4.87)	7.13 (4.48)	9.83 (4.17)	8.95 (3.89)	0.31
CA (months)	8.37 (4.71)	5.52 (4.33)	8.03 (4.23)	8.47 (4.29)	0.06
BW (g)	2146.25 (865.46)	1935 (831.75)	1790 (445.60)	1900[1013]	0.03

• M: male; F: female; GA: gestational age; CA: chronological age; CA: corrected age; BW: birth weight. Data presented by mean and standard deviation (SD).

Table 2 shows the results related to the scores and percentiles of the application of AIMS for the evaluation of intra-examiner and inter-examiner reliability.

Table 2
Intra-examiner (evaluations 1 and 2) and inter-examiner (examiner 1 and 2) reliability of AIMS scale applied in infants

		AIMS	Evaluation 1	Evaluation 2	95% CI		p
					LL	UL
Intra-examiner reliability	Examiner 1	Score	23.60±17.6	23.10±17			0.55+
		percentile	9.50±2.7	9.30±2.7	-0.46	0.86	0.51*
	Examiner 2	Score	32.9±20	32±20.7			0.12+
		percentile	2.40±1	2.4±1.1			1.00+
Inter-examiner reliability			Examiner 1	Examiner 2
Score		Score	34.64±17	33.73±16.9	-14.16	15.98	0.90#
percentile		percentile	6.36±2.6	6.09±2.9	-2.17	2.72	0.82#

• AIMS: Alberta Infant Motor Scale; LL: Lower limit; UL: Upper limit; CI: Difference confidence interval; data presented by mean±standard deviation; p=p value; ⁺Wilcoxon test performed;*performed paired t-test; ^#performed T-test.

Table 3 shows the results related to intra and inter-examiner reliability.

Table 3
AIMS Reliability Analysis

		ICC	95% ICC	P
			LL-UL
Score	1	1.00	1.00-1.00	<0.001
	2	0.99	0.97-1.00	<0.001
	1-2	0.99	0.97-1.00	<0.001
Percentile	1	0.89	0.63-0.97	<0.001
	2	0.88	0.59-0.97	<0.001
	1-2	0.96	0.86-0.99	<0.001

• 1: examiner 1; 2: examiner 2; 1-2: interexaminers; ICC: Interclass Correlation Coefficient; LL: Lower limit; UL: Upper limit; p: p-value

Figure 1 shows the data correlation, based on the correlation analysis of Pearson and Spearman.

Figure 1
Graphs of correlation analyses between scores 1 and 2; and percentiles 1 and 2 Data from the first and second evaluations represented on the x and y axis, respectively. p: p value; r: Pearson coefficient; rho: Spearman coefficient.

Figure 2 shows the data from qualitative analysis of agreement of the AIMS scores based on the Bland-Altman analysis.

Figure 2
Qualitative analysis of agreement of the data of the scores represented based on the Bland-Altman test

DISCUSSION

This study included as differential the analyses of agreement, homogeneity, correlation and reliability between the AIMS data obtained by two examiners. Thus, these results showed that AIMS is adequate to assess motor development and identify possible motor delays of infants in a follow-up outpatient clinic for at-risk newborns of a reference maternity hospital in southern Brazil. A simple training was appropriate to prepare professionals for the evaluation of these infants. Most premature infants, despite being considered at risk, after hospital discharge end up not being followed up with the recommended frequency of at least twice visits between 3-5 months of corrected age, and at 12 and 24 months of corrected age¹³. In this context, an existent delay in motor development is not identified, and may result in long-term problems of motor coordination, attention deficit, and delays in cognitive and school performance^14),(15. Therefore, in the first years of life, premature infants need periodic evaluations with standardized scales, so that any change in child’s overall development is detected early⁷.

The standardization of scales occurs by the evaluation of measurement properties, including reliability, since it addresses aspects about coherence, precision, stability, equivalence and homogeneity of an instrument, and especially in the case of a population at risk¹⁶. In our study, the percentile values showed no significant difference between the evaluations (LL_p=0.34% and UL_p=0.34%). Regarding inter-examiner reliability, results show no significant difference between the evaluations both in relation to the score at 95% CI (LLₑ=-0.42 points and ULₑ=2.23 points) and the percentile (LL_p=- 0.25% and UL_p=-0.80%). Moreover, in our study, AIMS percentile ICC of both intra- and inter-examiner remained above 0.88, whereas for the ICC scale score it was ≥0.99. The ICC, which is a measure of agreement assessment between the data, presented in our study values above 0.75, and approaching 1, representing high reliability for the application of tests according to the literature¹². Similarly, Almeida et al. ⁽¹⁷, to evaluate the concurrent validity and AIMS inter-examiner reliability in preterm infants in Rio de Janeiro, Brazil, verified AIMS inter-examiner reliability, with satisfactory ICC values in all ages evaluated, ranging from 0.76 to 0.99. These authors also verified that the concurrent validity between the AIMS raw scores and the Bayley Scale of Infant Development II had an excellent correlation (r=0.97; p<0.001). In 2013, Silva et al. ⁽¹⁸ analyzed AIMS reliability in 50 infants born preterm and full-term, aged around 4 months, whose results showed ICC above 0.8 (19). Besides, in the validation study of the scale for the population os the state of Rio Grande do Sul, intra-examiner reliability showed a strong agreement (ICC=0.92-0.99). Although several studies have evaluated AIMS reliability, the differential of our study is the evaluation of these measures in infants followed in follow-up outpatient clinic for at-risk newborns, using complementary and Bland-Altman analysis.

Studies recommend that, in addition to the reliability analyzed by the ICC, agreement should be evaluated as a complement, since the data may indicate a correlated but not concordant measure^18)-(20. For such purpose, besides the ICC, the agreement between the examiners was evaluated using Bland-Altman Analysis. Giavarina²⁰ describes the Bland-Altman method as a method to quantify the agreement between two quantitative measures by the construction of upper and lower limits, that is, by a specific analysis it is possible to quantify bias and a range of agreement between the measures²¹. In our study, this statistical test showed that the mean differences for intra-examiner 1 (p=0.108), intra-examiner 2 (p=0.475) and inter-examiner (p=0.157) analysis remained close to zero, with no statistically significant differences, indicating high intra- and inter-examiner agreement. The amplitude of the limits, both lower and upper, although low in our study, should be clinically evaluated by observing each infant evaluated, thus being observed whether this variation will be significant in the clinical area^19),(20. However, observing the limits of agreement, among the 31 evaluations, only 1 evaluation, of intra-examiner 2 reliability, was an outlier, remaining outside the limits of agreement (LL=-3.66 points; UL=4.66 points), not influencing the results, since about 95% of the points should remain within the recommended²².

Therefore, based on these data, the evaluation with AIMS in an outpatient clinic of at-risk infants presents high intra- and inter-examiner reliability for clinical practice, besides being highly concordant. This finding can be considered an important contribution for physical therapists working in outpatient clinics such as this clinic in which our study was developed. A differential of our study regarding AIMS is the clinical applicability by different physical therapists that work in outpatient clinics of at-risk newborns, provided that they are carefully trained to perform this activity.

However, one limitation of the study was the difficulty in selecting the sample. Most of the parents and/or guardians that were contacted did not want to attend the therapeutic evaluation, despite the pediatricians’ recommendation, and for this reason, the sample size of the study was not satisfactory. The homogeneous distribution of the age group of the sample is also a possible limitation, thus recommending that future studies perform the analysis of Bland-Altman agreement in a larger sample and in new age groups. We also suggest, for future studies, analyses similar to that of our study considering the subscores of the positions prone, supine, sitting and standing, since the total score can be masked of when analyzed together even if there is difference between the subscales.

CONCLUSION

This study showed that AIMS presents high intra- and inter-examiner reliability with preterm infants at risk of up to 18 months, and we suggest that this scale can be applied as a way of evaluating and monitoring the motor development of preterm newborns under follow-up in the follow-up outpatient clinics for at-risk newborns by trained professionals.

References

• ¹ Leal C, Esteves-Pereira AP, Nakamura-pereira M, Torres JA, Theme-filha M, Maria R, et al. Prevalence and risk factors related to preterm birth in Brazil. Reprod Health. 2016;13(Suppl 3):164-74. doi: 10.1186/s12978-016-0230-0
  » https://doi.org/10.1186/s12978-016-0230-0
• ² Voller SMB. Follow-Up care for high-risk preterm. Pediatr Ann. 2018;47(4):142-6. doi: 10.3928/19382359-20180325-03
  » https://doi.org/10.3928/19382359-20180325-03
• ³ Viana Filho JR, Pássari IA, Niveiros Filho SI. Gestão de custos hospitalares: um estudo de caso no Hospital Santa Casa de Misericórdia e Maternidade de Rondonópolis - MT. In: Machado MCK, editor. Controladoria, Gestão de Custos e Finanças. Ponta Grossa: Atena; 2017. p. 150-170. doi: 10.22533/at.ed.22318271210
  » https://doi.org/10.22533/at.ed.22318271210
• ⁴ Panceri C, Ruttnig K, Pereira G, Valentini NC, Helena R, Salazar A. A influência da hospitalização no desenvolvimento motor de bebês internados no Hospital de Clínicas de Porto Alegre. Clin Biomed Res [Internet]. 2012 [cited 2020 May 10];32(2):161-8. Available from: https://seer.ufrgs.br/hcpa/article/view/25819
  » https://seer.ufrgs.br/hcpa/article/view/25819
• ⁵ Guimarães CLN, Reinaux CM, Botelho ACG, Lima GMS, Cabral Filho JE. Desenvolvimento motor avaliado pelo Test of Infant Motor Performance: comparação entre lactentes pré-termo e a termo. Rev Bras Fisioter. 2011;15(5):357-62. doi: 10.1590/S1413-35552011005000021
  » https://doi.org/10.1590/S1413-35552011005000021
• ⁶ Khurana S, Kane AE, Brown SE, Tarver T, Dusing SC. Effect of neonatal therapy on the motor, cognitive, and behavioral development of infants born preterm: a systematic review. Dev Med Child Neurol. 2020;62(6):684-92. doi: 10.1111/dmcn.14485
  » https://doi.org/10.1111/dmcn.14485
• ⁷ Sociedade Brasileira de Pediatria. Seguimento Ambulatorial do Prematuro de Risco [Internet]. Porto Alegre; 2010 [cited 2020 May 10]. Available from: https://www.sbp.com.br/fileadmin/user_upload/pdfs/Seguimento_prematuro_ok.pdf
  » https://www.sbp.com.br/fileadmin/user_upload/pdfs/Seguimento_prematuro_ok.pdf
• ⁸ Albuquerque PL, Lemos A, Guerra MQDF, Eickmann SH. Accuracy of the Alberta Infant Motor Scale (AIMS) to detect developmental delay of gross motor skills in preterm infants: A systematic review. Dev Neurorehabil. 2015;18(1):15-21. doi: 10.3109/17518423.2014.955213
  » https://doi.org/10.3109/17518423.2014.955213
• ⁹ Lackovic M, Nikolic D, Filimonovic D, Petronic I, Mihajlovic S, Golubovic Z, et al. Reliability, Consistency and Temporal Stability of Alberta Infant Motor Scale in Serbian Infants. Children. 2020;7(3):16. doi: 10.3390/children7030016
  » https://doi.org/10.3390/children7030016
• ¹⁰ Piper MC, Darrah J. Motor Assessment of the Developing Infant. Philadelphia: Saunders; 1994.
• ¹¹ Valentini NC, Saccani R. Brazilian Validation of the Alberta Infant Motor Scale. Phys Ther. 2012;92(3):440-7. doi: 10.2522/ptj.20110036
  » https://doi.org/10.2522/ptj.20110036
• ¹² Valentini NC, Saccani R. Escala Motora Infantil de Alberta: Validação para uma população gaúcha. Rev Paul Pediatr. 2011;29(2):231-8. doi: 10.1590/S0103-05822011000200015
  » https://doi.org/10.1590/S0103-05822011000200015
• ¹³ Portney LG. Measurement Revisited: Reliability and Validity Statistics. In: Portney L. Foundations of Clinical Research: Applications to Evidence-Based Practice [Internet]. 4th ed. Philadeplhia: Saunders; 2020 [cited 2020 May 10]. Available from: Available from: http://fadavispt.mhmedical.com/content.aspx?aid=1172486561
  » http://fadavispt.mhmedical.com/content.aspx?aid=1172486561
• ¹⁴ McKinnon K, Huertas-Ceballos A. Developmental follow-up of children and young people born preterm, NICE guideline 2017. Arch Dis Child Educ Pract Ed. 2019;104(4):224. doi: 10.1136/archdischild-2017-314044
  » https://doi.org/10.1136/archdischild-2017-314044
• ¹⁵ Moreira RS, Magalhães LC, Alves CRL. Effect of preterm birth on motor development, behavior, and school performance of schoola-age children: A systematic review. J Pediatr. 2014;90(2):119-34. doi: 10.1016/j.jpedp.2013.05.009
  » https://doi.org/10.1016/j.jpedp.2013.05.009
• ¹⁶ Melo WA, Uchimura TT. Perfil e processo da assistência prestada ao recém-nascido de risco no Sul do Brasil. Rev Bras Epidemiol. 2011;14(2):323-37. doi: 10.1590/S1415-790X2011000200013
  » https://doi.org/10.1590/S1415-790X2011000200013
• ¹⁷ Almeida KM, Dutra MVP, Mello RR, Reis ABR, Martins PS. Validade concorrente e confiabilidade da Alberta Infant Motor Scale em lactentes nascidos prematuros. J Pediatr. 2008;84(5):442-8. doi: 10.2223/JPED.1829
  » https://doi.org/10.2223/JPED.1829
• ¹⁸ Silva LP, Maia PC, Lopes MMCO, Cardoso MVLML. Intraclass reliability of the Alberta Infant Motor Scale in the Brazilian version. Rev da Esc Enferm. 2013;47(5):1046-51. doi: 10.1590/S0080-623420130000500006
  » https://doi.org/10.1590/S0080-623420130000500006
• ¹⁹ Bunce C. Correlation, Agreement, and Bland-Altman Analysis: Statistical Analysis of Method Comparison Studies. Am J Ophthalmol 2009;148(1):4-6. doi: 10.1016/j.ajo.2008.09.032
  » https://doi.org/10.1016/j.ajo.2008.09.032
• ²⁰ Giavarina D. Understanding Bland Altman Analysis. Biochem Medica. 2015;25(2):141-51. doi: 10.11613/BM.2015.015
  » https://doi.org/10.11613/BM.2015.015

• Funding source: Notice UDESC-PAEX/2017; Natália Alves Menegol is a fellow of the Foundation for Research and Innovation Support of the State of Santa Catarina (FAPESC), commitment term 05/2019
• Approved by the Ethics Committee on Research with Human Beings under CAAE: 80831617.8.0000.0118.

Publication Dates

• Publication in this collection
  18 June 2021
• Date of issue
  Jan-Mar 2021

History

• Received
  02 Oct 2020
• Accepted
  12 Apr 2021