PAP SMEAR PERFORMANCE IN BACTERIAL VAGINOSIS DIAGNOSIS

REALIZACIÓN DE LA PRUEBA DE PAPANICOLAOU EN EL DIAGNÓSTICO DE LA VAGINOSIS BACTERIANA

ABSTRACT

Objective:   to assess Pap smear performance in bacterial vaginosis diagnosis in women of childbearing age.

Method:   a cross-sectional, retrospective study carried out in a health unit in a city in the interior of São Paulo. The study included information from 1,173 women who underwent collection of Pap smear and Gram-stained vaginal smears (reference standard test for bacterial vaginosis diagnosis) and analysis based on the Nugent score, from January 2013 to June 2020. Secondary data collected from the information and internal control systems were used.

Results:   the prevalence of bacterial vaginosis in the sample was 31.8%. The degree of agreement between the two diagnostic methods by the Kappa index was 0.54, considered moderate, with a value <0.001. Sensitivity was 55.2% and specificity was 94.1%, with accuracy of 81.7% and positive predictive value of 81.4% and negative of 81.8%.

Conclusion:  Pap smear showed agreement with the reference standard of 0.77, however, does not replace the Gram-stained vaginal smear, but can be used as an auxiliary method for bacterial vaginosis diagnosis.

DESCRIPTORS: Vaginosis; Bacterial. Papanicolaou Test. Clinical diagnosis. Sensitivity and specificity. Women’s health

RESUMEN

Objetivo:   evaluar el desempeño del examen citopatológico en el diagnóstico de vaginosis bacteriana en mujeres en edad fértil.

Método:  estudio transversal, retrospectivo, realizado en una unidad de salud de una ciudad del interior de São Paulo. El estudio incluyó información de 1173 mujeres a las que se les realizó recolección de pruebas citopatológicas y frotis vaginales teñidos con la técnica de Gram (prueba estándar de referencia para el diagnóstico de vaginosis bacteriana) y análisis con base en el puntaje de Nugent, desde enero de 2013 hasta junio de 2020. Datos secundarios recolectados de los sistemas de información y control interno.

Resultados:  la prevalencia de vaginosis bacteriana en la muestra fue de 31,8%. El grado de concordancia entre los dos métodos diagnósticos por el índice Kappa fue de 0,54, considerado moderado, con un valor <0,001. La Sensibilidad fue del 55,2% y la Especificidad del 94,1%, con una Precisión del 81,7% y un Valor Predictivo Positivo del 81,4% y Negativo del 81,8%.

Conclusión:  el examen citopatológico mostró concordancia con el patrón de referencia de 0,77, sin embargo, no sustituye al frotis vaginal teñido de Gram, pero puede utilizarse como método auxiliar para el diagnóstico de la vaginosis bacteriana.

DESCRIPTORES: Vaginosis bacteriana; Prueba de Papanicolaou; Diagnóstico clínico; Sensibilidad y especificidad; Salud de la mujer

RESUMO

Objetivo:   avaliar o desempenho do exame citopatológico no diagnóstico de vaginose bacteriana de mulheres em idade fértil.

Método:  estudo transversal, retrospectivo, realizado em uma unidade de saúde de um município do interior paulista. Foram inclusas no estudo as informações de 1173 mulheres submetidas à coleta dos exames citopatológico e esfregaço vaginal corado com técnica de Gram (exame padrão referência para diagnóstico de vaginose bacteriana) e análise a partir do score de Nugent, no período de janeiro de 2013 a junho de 2020. Utilizaram-se dados secundários, coletados a partir dos sistemas de informação e controle interno.

Resultados:  a prevalência de vaginose bacteriana na amostra foi de 31,8%. O grau de concordância entre os dois métodos diagnósticos pelo índice de Kappa foi de 0,54, considerado moderado, com valor<0,001. A Sensibilidade foi de 55,2% e a Especificidade de 94,1%, com Acurácia de 81,7% e Valor Preditivo Positivo de 81,4% e Negativo de 81,8%.

Conclusão:   o exame citopatológico apresentou concordância com o padrão referência de 0,77, no entanto, não substitui o esfregaço vaginal corado com técnica de Gram, mas pode ser utilizado como método auxiliar para o diagnóstico de vaginose bacteriana.

DESCRITORES: Vaginose bacteriana; Exame colpocitológico; Diagnóstico clínico; Sensibilidade e especificidade; Saúde da mulher

INTRODUCTION

Bacterial vaginosis (BV) is the main cause of vulvovaginitis in women of reproductive age,1 with a variable prevalence in different localities of around 32% in Brazil and 40 to 60% in Ethiopia. It is the result of an imbalance in the vaginal microbiota, with a reduction in the concentration of protective bacteria, such as Lactobacillus, and an increase in the pH and concentration of pathogenic bacteria, such as Gardnerella vaginallis and Mobiluncus spp2^-4.

There are several risk factors that are related to the occurrence of this imbalance, such as low socioeconomic conditions, smoking, practice of vaginal douching, use of intravaginal products, intrauterine device (IUD), multiple sexual partnerships and intercourse without condom use1^,5.

The presence of BV has impacts on women’s health, such as increased susceptibility to Sexually Transmitted Infections (STIs) such as HIV (Human Immunodeficiency Virus), genital herpes, gonorrhea and chlamydia as well as abortion, premature birth, postpartum complications and pelvic inflammatory disease1^,6.

For this reason, BV requires timely diagnosis, which in the Unified Health System (SUS - Sistema Único de Saúde) is carried out mainly in Primary Health Care (PHC)7 and can be clinical, mainly through the Amsel criteria, or laboratory, with Gram-stained vaginal smear analysis (reference standard exam) and assessment based on the Nugent Score, which quantifies the bacterial morphotypes and classifies the microbiota according to the score values (0-3 normal, 4-6 intermediate, and equal to or greater than 7, BV)1^,8.

Another diagnostic method capable of identifying the presence of the main bacteria that cause BV is Pap smear, although this is not its main objective, as it is recommended for screening cervical cancer and its precursor lesions in women aged 25 to 65 years9. It should be noted that Pap smear is offered in the PHC routine,10 which facilitates access to laboratory diagnosis, unlike the Gram staining technique, which depends on laboratory equipment and specialists for analysis and may be scarce in developing countries11.

In this context, the present study aims to assess Pap smear performance in BV diagnosis in women of childbearing age.

METHOD

This is a cross-sectional study with a quantitative approach, carried out at a School Health Center located in a city in the countryside of São Paulo.

Information from women who underwent collection of Pap smear and Gram-stained vaginal smears with analysis based on the Nugent score, from January 2013 to June 2020, were included in the study.

Regarding the selection criteria, pregnant women, women over 50 years of age and IUD users were excluded from the sample because they had conditions that could interfere with the normal vaginal microbiota (Figure 1)1^,5^,12.

Figure 1 -
Sample composition diagram.

For the sample, a minimum number of 127 participants was estimated, assuming sensitivity and specificity equal to 0.7, accuracy of 10 percentage points and prevalence of BV of 30.1%13 by the reference standard. However, the sample used in the present study was larger than estimated (1,173), considering the period of data collection, the selection criteria and the use of secondary data. In addition, the use of a larger sample improves the statistical analysis by making it more precise, especially in simple occurrence measures, such as accuracy, sensitivity and specificity.

The research used secondary data collected from information and internal control systems: e-SUS, Vivver (system adopted by the municipality), Cancer Information System (SISCAN - Sistema de Informação do Câncer) and internal document of the service that controls the performance and the result of Gram-stained vaginal smear analysis. The first two for obtaining sociodemographic data, the third for information regarding Pap smear and the fourth for information regarding Gram-stained vaginal smear.

In relation to obtaining data in the mentioned systems, age was considered from the date of birth reported by participants on the day the exams were collected. This provided date of birth has been converted to years. The education variable was obtained in levels of education (illiterate, literate, complete elementary school, incomplete elementary school, complete high school, incomplete high school, complete higher education, incomplete higher education, specialization/residency, master’s and doctoral degrees) and was converted into years of completed studies. The color variable was obtained as white, brown and black; and marital status was classified as married, stable, single, divorced and widowed. In relation to BV, data extraction was carried out according to exam results (Pap smear and Gram-stained vaginal smear analysis).

From the presentations in the systems, the present study considered for analysis and treatment the following composition of sociodemographic variables: age (in the age groups ≤20 years, 21-30, 31-40 and 41-50); color (white and non-white); education (≤8 years, 8-11 and ≥12); and marital status (with and without partnership). For the variables of the diagnostic methods, Pap smear (presence or not of signs suggestive of BV)14 and Gram-stained vaginal smear analysis (BV diagnosis yes or no) were used.

The accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of Pap smear were estimated in relation to Gram-stained vaginal smear analysis as a reference standard. The agreement between Pap smear and the reference standard was verified using the Kappa test, with the help of SPSS21 software, considering: k values <0, there is no agreement; 0-0.2, minimum; 0.21-0.40, reasonable; 0.41-0.60, moderate; 0.61-0.80, substantial; and 0.81 - 1.0 perfect.15.

The study complied with ethical precepts, according to Resolution N^o.466 of December 12, 2012.

RESULTS

The sample, consisting of 1,173 women, was predominantly aged between 21 and 30 years (34.3%), white (88.2%), with more than 12 years of education (39.8%) and without partnership (53.3%).

Based on the reference pattern, 373 women with BV were identified, a prevalence of 31.8%. The comparison of the results of Pap smear and the Gram staining technique is shown in Table 1, which also provides an understanding of the calculation of the performance measures of the diagnostic method illustrated in Table 2.

Table 1 -
Comparison between Pap smear and the reference standard† in bacterial vaginosis diagnosis. Botucatu, SP, Brazil, 2015-2020. (n=1173)

	Agreement among methods n(%)	Disagreement among methods n(%)	Total
Presence of BV*	206 (81.4%)	47 (18.6%)	253
Absence of BV*	753 (81.8%)	167 (18.2%)	920
Total	959	214	1173

• ^* BV: bacterial vaginosis; †Reference standard: Gram-stained vaginal smear and analyzed using the Nugent Score

Table 2
Performance measurements of Pap smear in relation to the reference standard in bacterial vaginosis diagnosis. Botucatu, SP, Brazil, 2015-2020. (n=1173)

Performance measures	Ratio	%
Accuracy	959/1173	81.7
Sensitivity	206/353	55.2
Specificity	753/800	94.1
PPV*	206/253	81.4
NPV†	753/900	81.8

• *PPV: positive predictive value; †NPV: negative predictive value

The degree of agreement between the two diagnostic methods by the Kappa index was 0.54. considered moderate, with p-value <0.001, which refutes the hypothesis of lack of agreement between the two methods.

Sensitivity, which assesses the ability of Pap smear to recognize the presence of BV, was 55.2%. Specificity, which consists of identifying the absence of BV, was 94.1%. Therefore, accuracy was 81.7%, with a PPV of 81.4% and a NPV of 81.8% (Table 2).

DISCUSSION

This study made it possible to assess Pap smear performance in BV diagnosis in the studied population. The prevalence of BV found in the present investigation was close to that found in international studies carried out in countries such as Ghana, Ethiopia and India, which observed a prevalence ranging from 30.9% to 44%,1,6,16 being the main cause of vulvovaginitis17.

In general, studies that analyzed Pap smear performance in BV diagnosis were carried out in emerging economies, such as Brazil and India2^,16. Faced with difficulties in accessing methods considered the reference standard, it is necessary to find other alternatives that help in this diagnosis, such as Pap smear.

In this regard, when assessing Pap smear performance, this study identified agreement between the methods, since the p-value was close to zero (<0.001). However, when tested by the Kappa method, moderate agreement was observed. A Brazilian study carried out in the southeast of the country found a good agreement of the test, of 0.77, when submitted to the Kappa method2.

When analyzed using different performance measures, BV diagnosis using Pap smear showed a moderate S and a high Ê, which was corroborated by a study that also compared the method to Gram-stained vaginal smear analysis as a standard reference. In this research, the S was 47.6% and the Ê was 95.418.

The moderate S of the test may be related to the fact that the cytopathological slide is made up of a cervical smear, unlike the reference standard that collects a vaginal smear16.

Countries such as Switzerland and Finland perform Pap smear from a triple collection, i.e., from the ectocervix, endocervix and vagina, which allows a better comparison with the Gram technique and even with the clinical diagnosis based on the criteria from Amsel19^-20.

A high Ê was found, of 95.2% and 96.9%, in other studies that made the same comparison, which made it possible to conclude that the method represents an adequate auxiliary tool in infection diagnosis, especially when the result is positive2^,21.

The method that assesses the predictive values indicates the probability of a result being confirmed by the reference standard after performing Pap smear. A PPV of 81.4% and a NPV of 81.8% were found in this research, values similar to those found in two other Indian and one Kenyan studies, in which the PPV ranged from 80 to 90.5% and NPV ranged from 80 to 88%18^,21^-22.

Other types of laboratory analysis have been studied and used in BV diagnosis, such as molecular analysis using the real-time polymerase chain reaction (rtPCR) technique and tests that identify specific bacterial nucleic acids. These tests have shown high sensitivity and specificity in BV diagnosis. However, these technologies are even more expensive and less accessible23^-24.

From the analyzes carried out, it is possible to state that Pap smear does not replace the Gram-stained vaginal smear. Although it is interesting to invest in implementing this technique in PHC, the context of the services and the reference standard method’s availability must be considered. In case of scarcity of the latter, Pap smear, as a test foreseen in SUS’ reality, can help in BV diagnosis.

As limitations of this study, it is noted that using secondary data from a locoregional sample may not represent the reality of other municipalities. Moreover, it is restricted to only two diagnostic methods and does not present a correlational analysis of diagnosis with data on sexual partnerships, for instance. However, it made it possible to assess Pap smear performance compared to the reference standard for BV diagnosis, which can directly contribute to the care of women, especially in the context of PHC.

That said, it is worth mentioning that the present study supports the actions of health professionals in PHC in the care of women.

CONCLUSION

Pap smear showed agreement with the reference standard of 0.77; accuracy of 81.7%, sensitivity of 55.2%, specificity of 94.1%, PPV of 81.4% and NPV of 81.8%. In this context, it is possible to conclude that Pap smear does not replace the Gram staining technique, but can be used as an auxiliary method for BV diagnosis.

REFERENCES

• 1. Coudray MS, Madhivanan P. Bacterial vaginosis - a brief synopsis of the literature. Eur J Obstet Gynecol Reprod Biol [Internet]. 2020 [cited 2023 Mar 27];245:143-8. Available from: https://doi.org/10.1016/j.ejogrb.2019.12.035
  » https://doi.org/10.1016/j.ejogrb.2019.12.035
• 2. Teixeira PM, Vital WC, Lima AA, Silva NNT, Carneiro CM, Teixeira LFM, et al. Bacterial vaginosis: prevalence, risk profile and association with sexually transmitted infections. Rev Epidemiol Controle Iinfecç [Internet]. 2020 [cited 2020 Oct 11];10(3):2238-3360. Available from: https://doi.org/10.17058/jeic.v10i3.14984
  » https://doi.org/10.17058/jeic.v10i3.14984
• 3. Bitew A, Abebaw Y, Bekele D, Mihret A. Prevalence of bacterial vaginosis and associated risk factors among women complaining of genital tract infection. Int J Microbiol [Internet]. 2017 [cited 2020 Jul 23];2017:491940. Available from: https://doi.org/10.1155/2017/4919404
  » https://doi.org/10.1155/2017/4919404
• 4. Nami Y, Haghshenas B, Khosroushahi AY. Molecular identification and probiotic potential characterization of lactic acid bacteria isolated from human vaginal microbiota. Adv Pharm Bull [Internet]. 2018 [cited 2020 Oct 11];8(4):683-95. Available from: https://doi.org/10.15171/apb.2018.077
  » https://doi.org/10.15171/apb.2018.077
• 5. Paladine HL, Desai UA. Vaginitis: diagnosis and treatment. Am Fam Physician [Internet]. 2018 [cited 2020 Oct 10];97(5):321-9. Available from: https://pubmed.ncbi.nlm.nih.gov/29671516/
  » https://pubmed.ncbi.nlm.nih.gov/29671516/
• 6. Konadu DG, Owusu-Ofori A, Yidana Z, Boadu F, Iddirisu LF, Adu-Gyasi D, et al. Prevalence of vulvovaginal candidiasis, bacterial vaginosis and trichomoniasis in pregnant women attending antenatal clinic in the middle belt of Ghana. BMC Pregnancy Childbirth [Internet]. 2019 [cited 2020 Oct 10];19(341):1-10. Available from: https://doi.org/10.1186/s12884-019-2488-z
  » https://doi.org/10.1186/s12884-019-2488-z
• 7. Gomes LS, Holanda VR, Barros MBSC. Identificação de infecções do trato reprodutivo em mulheres atendidas na Atenção Primária à Saúde. Rev APS [Internet]. 2019 [cited 2023 Mar 27];22(4):870-80. Available from: https://doi.org/10.34019/1809-8363.2019.v22.16228
  » https://doi.org/10.34019/1809-8363.2019.v22.16228
• 8. Nugent RP, Krohn MA, Hillier SL. Reability of diagnosing bacterial vaginosis is improved by a standardized method of Gram stain interpretation. J Clin Microbiol [Internet]. 1991 [cited 2020 Oct 10];29:297-301. Available from: https://doi.org/10.1128/jcm.29.2.297-301.1991
  » https://doi.org/10.1128/jcm.29.2.297-301.1991
• 9. Virtanen S, Rantsi T, Virtanen A, Kervinen K, Nieminen P, Kalliala I, et al. Vaginal microbiota composition correlates between pap smear microscopy and next generation sequencing and associates to socioeconomic status. Sci Rep [Internet]. 2019 [cited 2020 Nov 20];9:7750. Available from: https://doi.org/10.1038/s41598-019-44157-8
  » https://doi.org/10.1038/s41598-019-44157-8
• 10. Bezerra HS, Mendes TMC, Souza TA de, Nascimento AKF, Macedo HKS, Andrade FB. Coverage of screening for cervical cancer in a northeastern state of Brazil. J Hum Growth Dev [Internet]. 2021 [cited 2021 Nov 20];31(1):145-51. Available from I: Available from I: https://doi.org/10.36311/jhgd.v31.10319
  » https://doi.org/10.36311/jhgd.v31.10319
• 11. Bansal R, Garg P, Garg A. Comparison of Amsel’s criteria and Nugent’s criteria for diagnosis of bacterial vaginosis in tertiary care centre. Int J Reprod Contracept Obstet Gynecol [Internet]. 2019 [cited 2020 Nov 25];8(2):637-40. Available from: https://doi.org/10.18203/2320-1770.ijrcog20190297
  » https://doi.org/10.18203/2320-1770.ijrcog20190297
• 12. Lins LMR, Regis BC, Fernandes AST, Oliveira GMF, Araujo IM, Agra IKR, et al. Impactos da menopausa na saúde da mulher. Braz J Hea Rev [Internet]. 2020 [cited 2020 Oct 20];3(5):12018-31. Available from: https://doi.org/10.34119/bjhrv3n5-053
  » https://doi.org/10.34119/bjhrv3n5-053
• 13. Marconi C, Duarte MT, Silva DC, Silva MG. Prevalence of and risk factors for bacterial vaginosis among women of reproductive age attending cervical screening in southeastern Brazil. Int J Gynaecol Obstet [Internet]. 2015 [cited 2021 Jul 12];131(2):137-41. Available from: https://doi.org/10.1016/j.ijgo.2015.05.016
  » https://doi.org/10.1016/j.ijgo.2015.05.016
• 14. Pangarkar MA. The Bethesda System for reporting cytology. Cytojournal [Internet]. 2022 [cited 2023 Mar 26];19:28. Available from: https://doi.org/10.25259/CMAS_03_07_2021
  » https://doi.org/10.25259/CMAS_03_07_2021
• 15. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics [Internet]. 1977 [cited 2020 Nov 12];33:159-74. Available from: https://doi.org/10.2307/2529310
  » https://doi.org/10.2307/2529310
• 16. Anand KV, Pimple AS, Mishra GA, Sahare RV, Pathuthara S, Deodhar KK, et al. Reliability of conventional Papanicolaou smear in diagnosing bacterial vaginosis among women with clinical genital infection. South Asian J Cancer [Internet]. 2020 [cited 2021 Mar 15];9(1):13-6. Available from: https://doi.org/10.4103/sajc.sajc_421_18
  » https://doi.org/10.4103/sajc.sajc_421_18
• 17. Pathak R, Pradhan P, Pudasaini S, Maharjan S, Basnyat AS. Study of Thrichomonas vaginalis and bacterial vaginosis in Pap smear at a terciary health care centre of Nepal. Nepal Med Coll J [Internet]. 2020 [cited 2020 Nov 12];22(1-2):8-12. Available from: https://doi.org/10.3126/nmcj.v22i1-2.29926
  » https://doi.org/10.3126/nmcj.v22i1-2.29926
• 18. Manyu GC, Muchiri LW, Kahato MN. Comparison of PAP, modified Pap and Gram stained cervico-vaginal smears in the diagnosis of bacterial vaginosis in women attending thika district hospital. Int J Sci Res Publ [Internet]. 2018 [cited 2021 Jul 16];8(12):618-24. Available from: https://doi.org/10.29322/IJSRP.8.12.2018.p8479
  » https://doi.org/10.29322/IJSRP.8.12.2018.p8479
• 19. Eriksson K, Forsum U, Bjørnerem A, Platz-Christensen JJ, Larsson PG. Validation of the use of Pap-stained vaginal smears for diagnosis of bacterial vaginosis. APMIS [Internet]. 2007 [cited 2021 Jul 17];115:809-13. Available from: https://doi.org/10.1111/j.1600-0463.2007.apm_607.x
  » https://doi.org/10.1111/j.1600-0463.2007.apm_607.x
• 20. Vardar E, Maral I, Inal M, Ozguder O, Tasli F, Postaci H. Comparison of Gram stain and Papsmear procedures in the diagnosis of bacterial va-ginosis. Infect Dis Obstet Gynecol [Internet]. 2002 [cited 2021 Jul 17];10:203-7. Available from: https://doi.org/10.1155/S1064744902000236
  » https://doi.org/10.1155/S1064744902000236
• 21. Vandana G, Kumar KR, Khan S, Anil S. "Cytological findings of bacterial vaginosis in routine Pap Smears" A two yrs institutional study. J Med Dent [Internet]. 2018 [cited 2020 Nov 22];17(1):68-78. Available from: https://www.iosrjournals.org/iosr-jdms/papers/Vol17-issue1/Version-3/N1701036878.pdf
  » https://www.iosrjournals.org/iosr-jdms/papers/Vol17-issue1/Version-3/N1701036878.pdf
• 22. Ahmad I, Nirala RK, Poddar CK, Chaudhary PK. Comparative study of PAP Smear and microbiological pattern in bacterial vaginosis in a tertiary care hospital, South Bihar (India). Int J Contemp Med [Internet]. 2018 [cited 2020 Nov 22];5(12):5-10. Available from: http://doi.org/10.21276/ijcmr.2018.5.12.25
  » http://doi.org/10.21276/ijcmr.2018.5.12.25
• 23. Menard JP, Mazouni C, Fenollar F, Raoult D, Boubli L, Bretelle F. Diagnostic accuracy of quantitative real-time PCR assay versus clinical and Gram stain identification of bacterial vaginosis. Eur J Clin Microbiol Infect Dis [Internet]. 2010 [cited 2021 Jul 20];29:1547-52. Available from: https://doi.org/10.1007/s10096-010-1039-3
  » https://doi.org/10.1007/s10096-010-1039-3
• 24. Muzny CA, Balkus J, Mitchell C, Sobel JD, Workowski K, Marazzo J, et al. Diagnosis and management of bacterial vaginosis: summary of evidence reviewed for the 2021 Centers for Disease Control and Prevention Sexually Transmitted Infections Treatment Guidelines. Clin Infect Dis [Internet]. 2022 [cited 2023 Mar 26];74(Suppl 2):144-51. Available from: https://doi.org/10.1093/cid/ciac021
  » https://doi.org/10.1093/cid/ciac021

NOTES

• ORIGIN OF THE ARTICLE
  Extracted from the course completion work of Multidisciplinary Residency in Family Health at Universidade Estadual Paulista entitled “Acurácia do exame citopatológico como método diagnóstico de vaginose bacteriana”, in 2021.
• FUNDING INFORMATION
  The present work was carried out with the support of a project called: Technologies to support SNC and Management, contributions of the professional master’s degree for the central-south region of São Paulo: third stage of the CAPES/COFEN project, which belongs to Public Notice 8/2021 - Graduate Development Program (PDPG) - CAPES/COFEN. Aid: COFEN-20211953903P.
• APPROVAL OF ETHICS COMMITTEE IN RESEARCH
  Approved by the Ethics Committee in Research of the Faculty of Medicine of Botucatu, Opinion 1.011.708/2015, CAAE (Certificado de Apresentação para Apreciação Ética - Certificate of Presentation for Ethical Consideration) 42997315.5.0000.5411.

Edited by

• EDITORS
  Associated Editors: Renata Cristina de Campos Pereira Silveira, Ana Izabel Jatobá de Souza. Editor-in-chief: Elisiane Lorenzini.

Publication Dates

• Publication in this collection
  29 May 2023
• Date of issue
  2023

History

• Received
  11 Oct 2022
• Accepted
  30 Mar 2023