Validation of 19-items wearing-off (WOQ-19) questionnaire to Portuguese

Validação do questionário de 19 itens de wearing-off (WOQ-19) para a língua portuguesa

ABSTRACT

Background:  The treatment of Parkinson disease with dopaminergic therapy improves functionality and quality of life. However, as the disease progresses, the wearing-off phenomenon develops. To improve the recognition of this phenomenon, the 19-item wearing-off questionnaire (WOQ-19) was developed.

Objective:  To translate and validate the WOQ-19 into Portuguese.

Methods:  The questionnaire was translated into Portuguese and, subsequently, back-translated into English and analyzed. The final version was tested in Parkinson disease patients for reliability through the test-retest paradigm and internal consistency. Also, sensitivity and specificity were obtained in different cut-off positive items.

Results:  The WOQ-19 showed good test stability, with an intraclass correlation coefficient of 0.877 (95%CI 0.690-0.951; p<0.001), and good internal consistency, with Cronbach alpha of 0.815. Two items of positive cut-off showed the best accuracy: 0.873 (95%CI 0.791-0.954). Sensitivity was 0.975 (95%CI 0.892-1) and specificity was 0.714 (95%CI 0.565-0.863).

Conclusion:   The Portuguese version of the WOQ-19 showed excellent diagnostic properties and can be used to diagnose wearing-off phenomena.

Keywords: Parkinson Disease; Antiparkinson Agents

RESUMO

Introdução:  O tratamento da doença de Parkinson com terapia dopaminérgica melhora a funcionalidade e a qualidade de vida. Entretanto, com a progressão da doença, os fenômenos de flutuação motora e não motora se desenvolvem. Para melhorar o reconhecimento dessa situação, foi desenvolvido o questionário de 19 itens de wearing-off (WOQ-19)

Objetivo:  Traduzir e validar o questionário WOQ-19 para a língua portuguesa.

Métodos:  O questionário foi traduzido do inglês para o português. Em seguida, foi retrotraduzido para o inglês e analisado. A versão final foi testada em pacientes parkinsonianos com paradigma teste-reteste e consistência interna. A sensibilidade e especificidade foram medidas em relação a vários pontos de cortes de itens positivos.

Resultados:  O questionário apresenta boa estabilidade de teste, com coeficiente de correlação intraclasse de 0,877 (IC95% 0,690-0,951; p<0,001), e boa consistência interna, com alfa de Cronbach de 0,815. O ponto de corte com dois itens positivos teve a melhor acurácia: 0,873 (IC95% 0,791-0,954). A sensibilidade foi de 0,975 (IC95% 0,892-1) e a especificidade foi 0,714 (0,565-0,863).

Conclusão:  A versão em português do WOQ-19 mostrou excelentes propriedades diagnósticas e pode ser utilizada para diagnosticar as condições de flutuações motoras e não motoras na doença de Parkinson.

Palavras-chave: Doença de Parkinson; Antiparkinsonianos

INTRODUCTION

Parkinson disease (PD) is the second most prevalent neurodegenerative disorder, with no curative treatment. In the beginning, it can have a fair medical treatment with dopaminergic therapy, which improves functionality and quality of life¹. As the disease progresses, some complications of the treatment can be very disturbing, such as wearing-off (WO) phenomenon. It is the shortening of the drug effects that leads to complex medication posology or adjuvant therapies². It is associated with the worsening of quality of life and functionality. Sometimes, it is not easily recognized, if the symptoms are mild or non-motor.

To improve the recognition of this phenomenon, a 32-item questionnaire was developed³. It describes some symptoms and patients must check whether they have them and if they improve with medication. For practical reasons, using the same research, it was adapted to 19 items (19-item wearing-off questionnaire - WOQ-19), which should have the same properties⁴. It has been used in clinical studies and has been validated in several languages with some different clinimetric properties⁵. A recent review of the Movement Disorder Society set WOQ-19 as the recommended tool for wearing-off screening⁶. Our group performed a systematic review which showed a sensitivity of 0.8-0.88 and specificity of 0.68-0.8, when 2 cut-off items were used⁷, compared to the clinical evaluation, the gold-standard. This questionnaire was used in a sample of Brazilian patients, with similar properties. However, it was a plain translation, with no proper validation⁸.

Thus, we aimed to validate the WOQ-19 in Portuguese and to access its clinimetric properties.

METHODS

The work has two parts, language, cultural, and social translation, following Beaton’s guidelines⁹, and clinimetric properties testing.

Patient selection

One-hundred-fifty-six PD patients were selected from the movement disorder unit in Hospital de Clínicas de Porto Alegre, a public teaching hospital, in Porto Alegre, Brazil. Inclusion criteria were: Patients had to be diagnosed with PD by UK Brain Bank criteria¹⁰ and had to be on dopaminergic therapy. Also, the patient or caregivers had to be able to fill out the questionnaire. They were recruited from January 2015 to November 2017.

Sample size

A sample size of 20 patients was estimated for the test-retest step, with a confidence interval of 95% (CI95%), for an expected correlation of 0.858 (+/-0.119)¹¹. A sample of 156 patients was calculated for sensitivity 88% and specificity 80%, for an expected prevalence of 60 and 10% precision.

Language, cultural and social translation

The questionnaire (WOQ-19) was translated from English into Brazilian Portuguese by two physicians who were fluent in English. This version was back-translated into English by two translators with no medical background. The composition was compared and analyzed by both authors and translators for semantic, idiomatic, experimental, and conceptual equivalence, resulting in the final version. This version was administered to 20 PD patients as a semantic and conceptual validation test, as they were inquired about meaning, concept, and feelings. In case of incongruence, the items were reanalyzed.

Clinimetric properties

The questionnaire was explained by the physician before the clinical appointment and patients were asked to fill it out. Epidemiological and clinical data were collected, with a neurologic evaluation and the Mini-Mental State Examination¹² and the Movement Disorder Society Unified Parkinson’s Disease rating scale (MDS-UPDRS)¹³. Levodopa equivalent dose was calculated as described by Tomlinson¹⁴. All patients were examined by one of the two authors, CEM or MM, both neurologists with a special interest in Movement Disorders, in ON state, both blinded to the WOQ-19 results.

Twenty patients were evaluated by test-retest paradigm 2 weeks apart, through intraclass correlation coefficient (ICC) and kappa, to assess reliability. We used the Cronbach’s alpha for internal consistency in those patients, expecting a value greater than 0.75.

For sensibility and specificity the questionnaire results, were compared to the clinical evaluation as a gold-standard.

Statistical analysis

A descriptive analysis of all variables was conducted, expressed as mean and standard deviation (SD) or median and interquartile range (IQR). WOQ-19 results were compared to the WO clinical assessment of positive and negative patients. Also, the number of positive cut-off items for better sensibility and specificity was analyzed. Moreover, we compared the difference between patients with positive and negative questionnaires. We calculated the sensitivity, specificity, positive predictive value, negative predictive value and a receiver operating characteristic (ROC) curve with area under the curve (AUC). A post-hoc analysis was conducted, comparing these variables in a group of 20 patients with shorter disease duration. Internal consistency was tested with Cronbach’s alpha. The Shapiro-Wilk normality test was applied to determine the homogeneity of variance. Student’s t-test was used for parametric and Mann-Whitney U for non-parametric variables, chi-square for proportion. For statistical analysis, the SPSS 18 package was used. This study followed the STARD guideline¹⁵.

Ethical aspects

The project was approved by the Hospital de Clínicas Ethics Committee. The project was approved by Duke University and Dr. Stacy, developer of the questionnaire. Informed consent was obtained from all patients or their legal guardians.

RESULTS

Language, cultural, and social translation

After initial translation and back-translation, the item “Numbness” presented with discordance. The initial form Dormência, came back wrong. To clarify this issue, we consulted the Italian (Sensazione di addormentamento ad una parte del corpo¹¹), and Spanish (entumeciemento, hormigueos¹⁶) versions, languages more similar to Portuguese, and the item was settled as Dormência em parte do corpo. All 20 initial patients seem to understand the questionnaire.

Clinimetric properties

One hundred fifty-six PD patients were included. Demographics and clinical data are described in Table 1. Mean age was 64.7 (±10.2) years, 51% were male, and mean disease duration was 12.4 (±5.3) years. All patients were on levodopa therapy. Total MDS-UPDRS was 78.22 (±35.3). The median of the positive answers was 5 (3-8) for the entire questionnaire, 4 (2-6) for motor questions and 1 (0-2) for non-motor ones.

Table 1.
Epidemiological data.

	Total n=156 (100%)	WO by clinical assessment n=121 (77.6%)	No WO by clinical assessment n=35 (22.4%)
Age: mean (±SD), years	64.75 (10.21)	63.5 (10.34)	68.78 (8.73)	p=0.008
Male: n (%)	80 (51.3%)	66 (54.5%)	14 (40%)
Age at disease onset: mean (±SD), years	52.28 (10.36)	50.98 (10.42)	56.75 (8.91)	p=0.003
Disease duration: mean (±SD), years	12.47 (5.39)	12.60 (5.40)	12.02 (5,4)
LVD equivalent: mean (±SD), mg	1,188.02 (505.46)	1,236.31 (484.75)	1,021.09 (546.26)	p=0.026
MDS-UPDRS
Part 1: mean (±SD)	13.03 (7.88)	12.75 (7.71)	13.97 (8.45)	p<0.001
Part 2: mean (±SD)	19.49 (11.19)	19.33 (10.25)	20.06 (14.14)
Part 3: mean (±SD)	39.06 (20.82)	37.99 (20.07)	42.74 (23.17)
Part 4: mean (±SD)	6.64 (4.12)	7.86 (3.36)	2.43 (3.76)
Total: mean (±SD)	78.22 (35.30)	77.93 (33.18)	79.2 (42.37)
Questionnaire 2 items: n (%)	128 (82%)	118 (97.5%)	10 (28.5%)	p<0.001
Items: median (IQR)	5 (3-8)	6 (4-9)	1 (0-3)	p<0.001
Motor: median (IQR)	4 (2-6)	5 (3-7)	1 (0-2)	p<0.001
Non motor: median (IQR)	1 (0-2)	1 (0-3)	0 (0-1)	p<0.001

• LVD: levodopa; MDS-UPDRS: Movement Disorder Society Unified Parkinson's Disease rating scale; SD: standard deviation; IQR: interquartile range. Student’s t-test was used for parametric and Mann-Whitney U for non-parametric variables.

The test-retest paradigm was applied to 20 patients. The ICC, when considering the number of positive items, was 0.877 (95%CI 0.690-0.951; p<0.001). The kappa agreement for individual items, comparing the two moments, was equal to 0.604 (95%CI 0.044-0.69; p<0.001). The Cronbach’s alpha was obtained for internal consistency, 0.815, which was above the minimum previously set 0.75.

The questionnaire turned out in 128 WO-positive patients and 28 WO-negative ones. Clinical evaluation showed 121 WO-positive patients and 35 WO-negative ones. The median positive answer in clinical WO-positive patients was 6 (4-9) and clinical WO-negative patients, 1 (0-3); p<0.001.

Table 2 shows the sensitivity and specificity by number of positive items; 2 positive items have the best accuracy. Sensitivity was 0.975 (95%CI 0.947-1), specificity was 0.714 (0.565-0.863), positive predictive value (PPV) was 0.921 (95%CI 0.875-0.968), and negative predictive value was 0.892 (95%CI 0.778-1). A ROC curve was plotted with an AUC of 0.873 (95%CI 0.791-0.954) (Figure 1).

Table 2.
Items cut-off.

Number positive items	Sn	Sp
0	1	0
1	0.983	0.457
2	0.975	0.714
3	0.901	0.743
4	0.810	0.80
5	0.678	0.857
6	0.545	0.857

• Sn: sensitivity; Sp: specificity.

Figure 1.
Receiver operating characteristic curve.

Table 3 shows differences between patients with positive (2 or more items) and negative questionnaires. MDS-UPDRS part IV was higher in WO groups 7.51 (±3.65) × 0.68 (±3.88) and levodopa equivalent dose 1,227.66 (±475.76) mg × 1,006.83 (±600.45) mg, as expected.

A sample of 20 patients with shorter disease duration (4.93±1.36 years) analyzed with 2 positive cutoff points. Sensitivity was 1, specificity was 0.714.

Table 3.
Discriminatory ability.

	WOQ-19 (2 or more positive items)	WOQ-19 (1 or less positive items)
Equivalent dose: mean±), mg	1,227.66 (475.76)	1,006.83(600.45)	p=0.036
MDS-UPDRS 1: mean (±SD)	12.41 (7.61)	15.86(8.56)	p=0.035
MDS-UPDRS 2: mean (±SD)	18.84 (10.20)	22.46 (14.80)
MDS-UPDRS 3: mean (±SD)	36.84 (19.19)	49.18(25.07)	p=0.016
MDS-UPDRS 4: mean (±SD)	7.51 (3.65)	2.68 (3.88)	p<0.001
MDS-UPDRS total: mean (±SD)	75.6 (32.38)	90.18 (45.18)

• WOQ-19: 19-item wearing-off questionnaire; MDS-UPDRS: Movement Disorder Society Unified Parkinson's Disease rating scale; SD: standard deviation. Student’s t-test was used for parametric variables.

DISCUSSION

WOQ-19 is an important tool for clinical and research evaluation on PD. However, a proper validation for Portuguese was missing. Our linguistic validation followed Beaton’s guidelines¹⁰, and there have been no deviations. Since there was a divergence in an item during the procedure, the need for proper validation might be reinforced when a scale or questionnaire built in other cultural or language setting is used. In the era of multicentric trials, cultural and language barriers can be overcome by proper validation tools. The Movement Disorder Society has been doing this with its scales.

The questionnaire showed to be reliable, with excellent ICC, similar to the Italian version¹¹. Also, kappa was used for categorical correlation, resulting in a moderate correlation. To our knowledge, this is the first study to apply kappa. The Cronbach’s alpha showed good internal consistency.

As expected, the WOQ-19 showed a statistical difference in relation to MDS-UPDRS part IV, which measures levodopa’s complications and equivalent dose, which is associated with WO¹⁷. This reinforces the construct validation. When plotting the ROC curve, two positive items seemed to have the best accuracy. The same result was demonstrated by other authors^11,18. In addition, the 9-item questionnaire, which is a shortened version of this one, showed that the questionnaire loses specificity when a positive item is used as a cutoff point¹⁹.

This study has some limitations, as the specificity interval confidence was wide, partly because we have few patients without WO. This might happen because WO is associated with longer disease duration²⁰, and our patients had a long disease compared to others who validated this questionnaire in other languages^11,21. Also, our sample did not have many patients at the onset of the disease, we had only 2 patients with Hoehn and Yahr less than 2. It might be because our sample comes from a tertiary teaching hospital. This could make the symptoms milder and harder to identify. To mitigate this bias, we analyzed a sample of patients with 20 patients with 6 or fewer years of disease. It does not have the statistic power, however it showed similar results to those of the entire group. This approach must lessen this bias. However, WO was once thought to be a later complication, though it can be present since the onset of the disease. Stocchi, using the Italian version of the same questionnaire, reported as early as 2.5 years of disease duration for 41.8% of patients presented WO²². As having few patients early in the course of the disease, our validation loses power for this subset of patients. We do not report the time with levodopa use, because there was an important recall bias. Since WO is associated with levodopa use, the amount of time use could indicate a risk to develop WO. However, since all patients were on levodopa, we do not believe it could influence the diagnostic properties.

Another potential bias was that the questionnaire was explained to patients before they filled it out. There might be a difference among other groups, as it is not reported how the questionnaire was applied. Bares²³, when applying the shorter version of the questionnaire, stated it can have some misunderstanding about the questionnaire, so it was recommended it was explained to patients.

In conclusion, we validated the Portuguese version of the WOQ-19. The Brazilian version of the questionnaire demonstrated to be reliable and valid and can be considered a tool to diagnose wearing-off. Some caution must be taken when applying it to patients at very early stages of the disease.

ACKNOWLEDGMENTS

CRM Rieder was supported by the Brazilian government research grant agency - Conselho Nacional de Desenvolvimento Científico e Tecnológico [CNPq] (No. 311970/2014-1). We thank Prof. Stacy and Duke University for granting permission for translation and support.

References

• ¹ Aquino CC and Fox SH. Clinical spectrum of levodopa-induced complications. Mov Disord. 2015 Jan.30 (1):80-9. https://doi.org/10.1002/mds.26125
  » https://doi.org/https://doi.org/10.1002/mds.26125
• ² Pahwa R, Lyons KE. Levodopa-related wearing-off in Parkinson's disease: identification and management. Curr Med ResOpin. 2009 Apr; 25 (4):841-9. https://doi.org/10.1185/03007990902779319
  » https://doi.org/https://doi.org/10.1185/03007990902779319
• ³ Stacy M, Bowron A, Guttman M, Hauser R, Hughes K, Larsen JP, et al. Identification of motor and nonmotor wearing-off in Parkinson’s disease: comparison of a patient questionnaire versus a clinician assessment. Mov Disord. 2005 Jun; 20 (6):726-33. https://doi.org/10.1002/mds.20383
  » https://doi.org/https://doi.org/10.1002/mds.20383
• ⁴ Stacy M, Hauser R. Development of a patient questionnaire to facilitate recognition of motor and non-motor wearing-off in Parkinson’s disease. J Neural Transm (Vienna). 2007 Feb;114(2):211-7. https://doi.org/10.1007/s00702-006-0554-y
  » https://doi.org/https://doi.org/10.1007/s00702-006-0554-y
• ⁵ Stacy M. The wearing-off phenomenon and the use of questionnaires to facilitate its recognition in Parkinson’s disease. J Neural Transm (Vienna). 2010;117(7):837-46. https://doi.org/10.1007/s00702-010-0424-5
  » https://doi.org/https://doi.org/10.1007/s00702-010-0424-5
• ⁶ Antonini A, Martínez-Martín P, Chaudhuri RK, Merello M, Hauser R, Katzenschlager R, et al. Wearing-Off scales in Parkinson’s disease: critique and recommendations. Mov Disord. 2011 Oct;26(12):2169-75. https://doi.org/10.1002/mds.23875
  » https://doi.org/https://doi.org/10.1002/mds.23875
• ⁷ Mantese CE, Schumacher-Schuh A, Rieder CRM. Clinimetrics of the 9- and 19-item wearing-off questionnaire: a systematic review. Parkinsons Dis. 2018 Apr;2018:Article ID 5308491:1-5. https://doi.org/10.1155/2018/5308491
  » https://doi.org/https://doi.org/10.1155/2018/5308491
• ⁸ Melo LM, Chien HF, Barbosa ER. Identification of wearing-off manifestations (reduction of levodopa effect) in Parkinson's disease using specific questionnaire and comparison of the results with routine ambulatory evaluations. Arq Neuro-Psiquiatr. 2010;68(4):506-10. https://doi.org/10.1590/S0004-282X2010000400007
  » https://doi.org/https://doi.org/10.1590/S0004-282X2010000400007
• ⁹ Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186-91. https://doi.org/10.1097/00007632-200012150-00014
  » https://doi.org/https://doi.org/10.1097/00007632-200012150-00014
• ¹⁰ Hughes AJ, Daniel SE, Kilford L, Lees AJ. Accuracy of clinical diagnosis of idiopathic Parkinson’s disease: a clinico-pathological study of 100 cases. J Neurol Neurosurg Psychiatry. 1992;55(3):181-4. https://doi.org/10.1136/jnnp.55.3.181
  » https://doi.org/https://doi.org/10.1136/jnnp.55.3.181
• ¹¹ Abbruzzese G, Antonini A, Barone P, Stocchi F, Tamburini T, Bernardi L, et al. Linguistic, psychometric validation and diagnostic ability assessment of an Italian version of a 19-item wearing-off questionnaire for wearing-off detection in Parkinson's disease. Neurol Sci. 2012 Dec;33(6):1319-27. https://doi.org/10.1007/s10072-012-0943-y
  » https://doi.org/https://doi.org/10.1007/s10072-012-0943-y
• ¹² Brucki SMD, Nitrini R, Caramelli P, Bertolucci PHF, Okamoto IH. Suggestions for utilization of the mini-mental state examination in Brazil. Arq Neuro-Psiquiatr. 2003 Sep;61(3B):777-81. https://doi.org/10.1590/s0004-282x2003000500014
  » https://doi.org/https://doi.org/10.1590/s0004-282x2003000500014
• ¹³ Goetz CG, Tilley BC, Shaftman SR, Stebbins GT, Fahn S, Martínez-Martín P, et al. Movement Disorder Society-Sponsored Revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results. Mov Disord. 2008 Nov;23(15):2129-70. https://doi.org/10.1002/mds.22340
  » https://doi.org/https://doi.org/10.1002/mds.22340
• ¹⁴ Tomlinson CL, Stowe R, Patel S, Rick C, Gray R, Clarke CE. Systematic review of levodopa dose equivalency reporting in Parkinson’s disease. Mov Disord. 2010 Nov;25(15):2649-85. https://doi.org/10.1002/mds.23429
  » https://doi.org/https://doi.org/10.1002/mds.23429
• ¹⁵ Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015 Oct;351:1-9. https://doi.org/10.1136/bmj.h5527
  » https://doi.org/https://doi.org/10.1136/bmj.h5527
• ¹⁶ Martínez-Martín P, Tolosa E, Hernández B, Badia X. The Patient Card questionnaire to identify wearing-off in Parkinson disease. Clin Neuropharmacol. 2007 Sep-Oct;30(5):266-75. https://doi.org/10.1097/wnf.0b013e3180413da7
  » https://doi.org/https://doi.org/10.1097/wnf.0b013e3180413da7
• ¹⁷ Warren Olanow C, Kieburtz K, Rascol O, Poewe W, Schapira AH, Emre M, et al. Factors predictive of the development of Levodopa-induced dyskinesia and wearing-off in Parkinson's disease. Mov Disord. 2013 Jul;28(8):1064-71. https://doi.org/10.1002/mds.25364
  » https://doi.org/https://doi.org/10.1002/mds.25364
• ¹⁸ Martínez-Martín P, Tolosa E, Hernández B, Badia X. Validation of the “QUICK” Questionnaire - a tool for diagnosis of “wearing-off” in patients with Parkinson’s disease. Mov Disord. 2008 Apr;23(6):830-6. https://doi.org/10.1002/mds.21944
  » https://doi.org/https://doi.org/10.1002/mds.21944
• ¹⁹ Stacy MA, Murphy JM, Greeley DR, Stewart RM, Murck H, Meng X, et al. The sensitivity and specificity of the 9-item Wearing-off Questionnaire. Parkinsonism Relat Disord. 2008;14(3):205-12. https://doi.org/10.1016/j.parkreldis.2007.07.013
  » https://doi.org/https://doi.org/10.1016/j.parkreldis.2007.07.013
• ²⁰ Hauser R, McDermott MP, Messing S. Factors associated with the development of motor fluctuations and dyskinesias in Parkinson disease. Arch Neurol. 2006 Dec;63(12):1756-60. https://doi.org/10.1001/archneur.63.12.1756
  » https://doi.org/https://doi.org/10.1001/archneur.63.12.1756
• ²¹ Seki M, Takahashi K, Uematsu D, Mihara B, Morita Y, Isozumi K, et al. Clinical features and varieties of non-motor fluctuations in Parkinson’s disease: A Japanese multicenter study. Parkinsonism Relat Disord. 2013 Jan;19(1):104-8. https://doi.org/10.1016/j.parkreldis.2012.08.004
  » https://doi.org/https://doi.org/10.1016/j.parkreldis.2012.08.004
• ²² Stocchi F, Antonini A, Barone P, Tinazzi M, Zappia M, Onofrj M, et al. Early DEtection of wEaring off in Parkinson disease: the DEEP study. Parkinsonism Relat Disord. 2014 Feb;20(2):204-11. https://doi.org/10.1016/j.parkreldis.2013.10.027
  » https://doi.org/https://doi.org/10.1016/j.parkreldis.2013.10.027
• ²³ Bareš M, Rektorová I, Jech R, Farníková K, Roth J, Růžička E, et al. Does WOQ-9 help to recognize symptoms of non-motor wearing-off in Parkinson’s disease? J Neural Transm (Vienna). 2012 Mar;119(3):373-80. https://doi.org/10.1007/s00702-011-0683-9
  » https://doi.org/https://doi.org/10.1007/s00702-011-0683-9

• Support: Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq).

Publication Dates

• Publication in this collection
  14 Sept 2020
• Date of issue
  Oct 2020

History

• Reviewed
  29 Oct 2019
• Received
  11 Mar 2020
• Accepted
  06 Apr 2020