Validation of self-reported history of root canal treatment in a southern Brazilian subpopulationAbstract
The aim of this study was to assess self-reported history of root canal treatment (SRHRCT) as a method for detecting the presence of root canal treatment (RCT) and apical periodontitis (AP) in a southern Brazilian subpopulation. In this cross-sectional study, 136 military police officers from the city of Porto Alegre, Brazil, were included. The participants were interviewed and full-mouth periapical radiographs were taken. A calibrated examiner determined the presence of RCT and AP by applying standardized criteria. The diagnostic accuracy of SRHRCT was calculated separately for RCT and AP. Accuracy, sensitivity, specificity, positive and negative predictive values (+PV and -PV), efficiency, and positive and negative likelihood ratios (+LR and -LR) were estimated. The mean age of the participants was 34.1 ± 10.4 years and 88.2% were males. Overall, SRHRCT demonstrated high sensitivity and specificity for RCT, but not for AP: sensitivity (RCT = 0.960, AP = 0.757) and specificity (RCT = 0.835, AP = 0.631). The estimated values for PV and LR were: +PV (RCT=0.777, AP=0.396), -PV (RCT = 0.972, AP = 0.890), +LR (RCT = 5.853, AP = 2.057), and -LR (RCT = 0.046, AP = 0.383). SRHRCT proved to be a good predictor of the presence of RCT, but a weak predictor of AP in this subpopulation.
Endodontics; Periapical Periodontitis; Validation Studies
Introduction
Large-scale epidemiological studies are scarce in the field of endodontics, which challenges accurate estimates of pulp and periapical diseases and the need for endodontic care worldwide. Population-based surveys, especially those on pulp diseases, apical periodontitis (AP), and root canal treatment (RCT) have high costs because of the need of clinical and radiographic examinations, trained and qualified professionals, and time-consuming diagnosis. The use of accurate self-reported measures in these scenarios may significantly reduce costs and facilitate the reproducibility of such studies. 1,2
There is evidence that self-reported health status is a valid method for assessing a variety of diseases and health-related conditions, such as diabetes, 3,4 diet, 5 hypertension, 3,6 hypercholesterolemia, 7 and rheumatoid arthritis. 8 In dentistry, there are reports of accurate measures of self-reported number of teeth, 1,9 use of dental prosthesis 1,10 periodontal disease, 11 orofacial pain, 12 presence of fillings, 1 and the need for RCT. 13
To date, only two studies have been able to validate data on self-reported history of root canal treatment (SRHRCT), 1,2 and another two have used SRHRCT as a simplified method for estimating the patient’s history of RCT and pulp disease. 14,15 In addition, even though a radiographic image of RCT is assumed to represent a history of dental pulp or periapical pathosis, SRHRCT was found to be a poor predictor of AP in a previous study. 2
Note that both previous investigations validating SRHRCT were conducted in North-American populations, 1,2 and the extrapolation of those findings to populations with different social, cultural, and educational profiles may be prone to bias. Validation studies in Latin American countries are even scarcer. Regarding periodontal conditions, one single study was conducted to validate self-reported periodontal disease in a Brazilian sample. 16 To the best of the authors’ knowledge, there are no studies on the validity of SRHRCT in South American populations. Hence, the aim of this study was to quantify the validity of SRHRCT as a method for detecting the presence of RCT and AP in a southern Brazilian subpopulation.
Methodology
The study was reviewed and approved by the Research Ethics Committee of the Federal University of Rio Grande do Sul (process # 03735112.6.0000.5347) and by the Research Institute of the Military Police of Rio Grande do Sul, Brazil. All subjects read and signed an informed consent form before their participation. The data were identified prior to the analysis. This cross-sectional observational study follows the STROBE guidelines (supplemental material).
The target population of this study comprised military police officers from the city of Porto Alegre, Brazil, included in a major study aimed to assess the impacts of periodontal and apical diseases on their physical activity. 17,18 In total, 154 officers were eligible and invited to participate in the study. Of those, 138 agreed to participate, and 136 individuals with complete data on SRHRCT were included in the final analysis. The study was conducted from June 2012 to March 2013.
A structured questionnaire was applied and a clinical examination was conducted in all participants 17,18 to obtain sociodemographic and clinical data. The main exposure variable of this study (SRHRCT) was obtained by the following question applied in the questionnaire: “Have you ever had any root canal treatment?” The participants then answered “yes” or “no.”
The outcomes were obtained from full-mouth periapical radiographic examinations not exceeding one year from the time of the oral clinical examination, retrieved from the charts of the Health Department of the Military Police for each study participant. Radiographic analysis was conducted by one experienced and calibrated endodontist (MSG). The intra-examiner calibration resulted in kappa values of 0.85 for AP and 0.99 for RCT.
For the radiographic assessment of AP and RCT variables, pre-established criteria 19 were applied to all teeth, except to third molars. For analytical purposes, both AP and RCT variables were dichotomized as absent or ≥1 teeth with AP or RCT, respectively.
Descriptive statistics were used to describe the main characteristics of the study sample. The validity of SRHRCT was calculated based on RCT and AP data, assessed separately through the radiographic outcome. Accuracy, sensitivity, specificity, positive and negative predictive values (+PV and -PV), efficiency, and positive and negative likelihood ratios (+LR and -LR) were determined according to standard methods. In addition, Pearson’s correlation ( r ) was calculated between SRHRCT and RCT, SRHRCT and AP, and RCT and AP.
Results
The characteristics of the study sample are shown in Table 1 . Mean age was 34.1 ± 10.4, ranging from 20 to 53 years, with 88.2% males. Participants of a lower military rank predominated, with 44.1% soldiers and 25.0% sergeants.
Sociodemographic and dental characteristics of participants. Data shown as mean ± standard deviation or N (%). n = 136.
The mean number of teeth among participants was 30.7 ± 1.9, and most of the individuals (53.7%) reported no history of RCT. The prevalence of individuals with one tooth presenting RCT was 19.9% and with more than one tooth presenting RCT was 17.6%. In total, 51 patients (37.5%) presented RCT. The prevalence of individuals with one tooth with AP was 20.6% and with more than one tooth with AP was 3.7%, representing a total of 33 subjects (24.3%) with at least one episode of AP.
Table 2 is the contingency table for SRHRCT in relation to RCT and AP. Among individuals who reported no history of RCT, 71 (97.3%) had no RCT on their radiographs, and among those who reported history of RCT, 49 (77.8%) showed RCT radiographically. In relation to AP, 65 participants (89%) who answered negatively to SRHRCT did not present AP on their radiographs, whereas only 25 individuals (39.6%) who answered positively to SRHRCT presented radiographic signs of AP.
Contingency table (N/%) for self-reported history of root canal treatment (SRHRCT) in relation to the presence of root canal treatment (RCT) and apical periodontitis (AP).
Table 3 shows the values for all diagnostic tests estimated for SRHRCT in relation to RCT and AP. RCT presented high sensitivity (0.96), specificity (0.83), -PV (0.97), and +LR (5.85), while AP presented a low +PV (0.39). The r values between SRHRCT and RCT and between SRHRCT and AP were 0.773 (p < 0.001) and 0.334 (p < 0.001), respectively.
Quantification of accuracy, sensitivity, specificity, positive predictive value, negative predictive value, efficiency, positive likelihood ratio, and negative likelihood ratio for self-reported history of root canal treatment (SRHRCT) in relation to the presence of root canal treatment (RCT) and apical periodontitis (AP). n =136.
Table 4 shows the mean number of teeth in participants with true-positive, true-negative, false-positive, and false-negative results for SRHRCT in relation to RCT and AP. For RCT diagnosis, individuals with true-positive and true-negative results showed higher averages for the number of teeth (30.6 and 31.1, respectively) when compared to the group of individuals with false-positive and false-negative results (29.5 for both). In relation to the AP variable, the highest mean number of teeth was observed among individuals with true-negative results (31.1).
Average number of teeth in the samples with true-positive, true-negative, false-positive, and false-negative results for self-reported history of endodontic treatment (SRHRCT) in relation to root canal treatment (RCT) and apical periodontitis (AP) variables. N=136.
Table 5 shows the contingency table for RCT and AP. Among 33 participants with AP, 25 (75.8%) also presented RCT. Of the 103 participants with healthy periapical status, 77 (74.8%) did not exhibit RCT. Among the 51 individuals presenting RCT, 26 (25.2%) presented no AP. The correlation between RCT and AP was r = 0.447 (p < 0.001).
Contingency table (N/%) correlating the presence of root canal treatment (RCT) and the presence of apical periodontitis (AP).
Discussion
The present study is novel in evaluating the validity of SRHRCT as a method for predicting RCT and AP in a South American adult subpopulation. Results revealed that participants’ responses were highly accurate when they were asked to self-report their history of RCT, but the SRHRCT diagnostic tool was a poor predictor of the presence of AP in this sample. Interestingly, our present findings are in accordance with a similar study carried out by our research group in a North American population. 2 We confirm the need for routine clinical and radiographic examinations for the diagnosis AP, as patients are commonly unaware of the presence of this disease.
As previously discussed in detail, 2 the higher predictive ability of SRHRCT to detect RCT, compared to AP, may be attributable to many different factors, such as the patient’s perception of the treatment (RCT is a technically complex procedure, with extended appointments and high cost); however, a possible confounding factor to explain the reports of RCT in the absence of positive radiographic findings could be the failure to perform a complete RCT, i.e. , the final stage of this treatment, which encompasses the placement of the filling material that can be detected radiographically in the root canal space, was not carried out. Also, RCT may be indicated for vital or necrotic teeth, with no AP; when AP is present, it is expected to heal after RCT, which ends up skewing the false-positive results. On the other hand, AP is often an asymptomatic disease, which may keep individuals from seeking endodontic care, thereby increasing the frequency of false-negative results. Moreover, tooth loss due to AP may also increase the possibility of false results. Furthermore, AP is often associated with silent pulp necrosis. Taken together, these factors explain the relatively low correlation observed between RCT and AP both in the present sample (r=0.447) and in the previous study (r = 0.454). 2
Some methodological aspects of this study warrant discussion. Tooth loss may be an important confounder in diagnostic studies involving endodontic parameters, but we believe it was not a concern in the present sample, since the participants presented with a very high mean number of teeth (30.7 ± 1.9). Some authors suggest considering tooth loss a surrogate factor of AP history, 15 which would justify the exclusion of patients with fewer teeth to reduce the effect of this potential confounder, as done in a previous study. 2 However, the exclusion of these patients from the present sample was not necessary because of the high mean number of teeth present. Another feature is related to individuals’ low mean age (34.1 ± 10.4 years), which possibly contributed to the relatively low prevalence of both RCT (37.5%) and AP (24.3%) in this subpopulation.
Similarly to previous studies, 1,2 the population of this study has some peculiar and well defined characteristics, limiting the external validity of the present results. This sample consisted predominantly of young male adults, all of whom were military police officers with institutional access to full dental care and to regular dental visits. Therefore, this subpopulation is usually well informed and up-to-date regarding oral health conditions, which possibly contributed to the increased levels of true-positive and true-negative results. In addition, present findings on the accuracy of SRHRCT were very similar to the results of a previous study, 2 since the population of that study consisted largely of well-educated North American white adults, with similar patterns of access to dental care. Most importantly, it must be very clear that the validity indicators are context-dependent, that is, they may be influenced by cultural and socioeconomic factors and by the access to health services. Hence, the same instrument may present different levels of accuracy when performed in different population groups. 20,21,22 Therefore, it seems clear that future diagnostic studies using self-reported measures for detecting pulp diseases, AP, and RCT should be highly encouraged, especially in populations with different socioeconomic profiles.
The vast majority of epidemiological studies on AP use panoramic radiography as the method of evaluation. 23,24 Conversely, the present study used periapical radiographs, which provide more accurate information for the diagnosis of AP. 25 This is even more important when assessing anterior segments, since periapical diagnosis in this region is greatly impaired by anatomical interferences, an inherent characteristic of panoramic radiographic examination. Cone-beam computed tomography (CBCT) could provide even more robust results as it allows the visualization of related anatomical structures in three dimensions with higher resolution, resulting in excellent diagnostic accuracy. 26
An important limitation inherent to any imaging method for the detection of AP should be considered: a cross-sectional analysis does not allow the establishment of the diagnosis on the activity of the lesion, since the presence of a radiolucent area can represent either an active lesion or a healing process. In this case, the gold-standard examination would be histological, which is not feasible for obvious ethical reasons. Another research strategy to detect AP would be longitudinal screening, which was not feasible in the present study.
According to a previous study, 27 validity can vary according to the self-report variable analyzed. In the present study, the question “Have you ever had any root canal treatment?” was found to be a poor predictor of AP, as the results showed a 60.4% false-positive rate for this outcome. This result is very similar to the findings of a previous study, 2 and it is of great relevance, since it reveals the current lack of a reliable self-report instrument to estimate the presence of endodontic disease and consequent RCT needs. Thus, considering the importance of population screenings in detecting AP in different populations worldwide, there is a clear need to develop new diagnostic methods based on self-reported measures on oral health, capable of providing more accurate estimates of AP, with a lower number of false positives than that provided by the SRHRCT method.
Since tests with higher specificity values can detect more accurately those individuals who do not present the condition, they are best indicated when the treatment requires more invasive or costly measures. The higher the specificity of a test, the lower the chances of a false-positive result. In this study, considering AP, the specificity values (63%) were lower than those obtained for sensitivity (75%), which reinforces that this single self-reported measure is not suitable for a survey of the needs of endodontic interventions at a population-based level, as it would lead to a significant number of overtreatments and, consequently, to unnecessary public health costs.
Investigations focused on periodontal disease showed good accuracy between self-reported periodontal measures and clinical periodontal parameters in different populations. 22,28,29 In part, this may occur because signs and symptoms associated with both gingivitis and periodontitis are more easily perceived by patients than those associated with AP, which reflects in greater sensitivity to self-reported instruments. In addition, these studies demonstrate that a single self-reported measure is not sufficient to determine periodontal disease; however, a set of combined self-reported measures in a multivariate analysis may produce better predictive values. 16
Valid self-reported measures on pulp and periapical diseases may simplify and enable large-scale epidemiological surveys, contributing to the planning of health strategies and policies for the prevention and treatment of endodontic diseases. In addition, future accurate self-reported methods capable of detecting AP (e.g., involving a set of specific questions related to caries and pain history, restorative treatments or dental trauma, among others, besides the SRHRCT question) may contribute to the investigation of the relationship between endodontic infection and systemic diseases through population-based studies.
Conclusion
SRHRCT was found to be an accurate method for detecting RCT experience, but a weak predictor of the presence of AP in this southern Brazilian adult subpopulation.
Acknowledgements
The authors thank the Military Police of Rio Grande do Sul, Brazil, for allowing the use of their facilities for this study, and all the officers who voluntarily agreed to participate in this study. Also, the authors thank CAPES Foundation (Ministry of Education of Brazil, Brasília/DF, Brazil) and the Australian Academy of Science (Canberra, Australia), for granting a scholarship to Gisele Jung Franciscatto. The authors report no conflicts of interest related to this study.
References
-
1 Pitiphat W, Garcia RI, Douglass CW, Joshipura KJ. Validation of self-reported oral health measures. J Public Health Dent. 2002;62(2):122-8. https://doi.org/10.1111/j.1752-7325.2002.tb03432.x
» https://doi.org/10.1111/j.1752-7325.2002.tb03432.x -
2 Gomes MS, Hugo FN, Hilgert JB, Padilha DM, Simonsick EM, Ferrucci L, et al. Validity of self-reported history of endodontic treatment in the Baltimore Longitudinal Study of Aging. J Endod. 2012 May;38(5):589-93. https://doi.org/10.1016/j.joen.2012.02.006
» https://doi.org/10.1016/j.joen.2012.02.006 -
3 Huerta JM, Tormo MJ, Egea-Caparrós JM, Ortolá-Devesa JB, Navarro C. Accuracy of self-reported diabetes, hypertension and hyperlipidemia in the adult Spanish population. DINO study findings. Rev Esp Cardiol. 2009 Feb;62(2):143-52. https://doi.org/10.1016/S0300-8932(09)70156-3
» https://doi.org/10.1016/S0300-8932(09)70156-3 -
4 Jackson JM, DeFor TA, Crain AL, Kerby TJ, Strayer LS, Lewis CE, et al. Validity of diabetes self-reports in the Women’s Health Initiative. Menopause. 2014 Aug;21(8):861-8. https://doi.org/10.1097/GME.0000000000000189
» https://doi.org/10.1097/GME.0000000000000189 -
5 Rimm EB, Giovannucci EL, Stampfer MJ, Colditz GA, Litin LB, Willett WC. Reproducibility and validity of an expanded self-administered semiquantitative food frequency questionnaire among male health professionals. Am J Epidemiol. 1992 May;135(10):1114-26. https://doi.org/10.1093/oxfordjournals.aje.a116211
» https://doi.org/10.1093/oxfordjournals.aje.a116211 -
6 Alonso A, Beunza JJ, Delgado-Rodríguez M, Martínez-González MA. Validation of self reported diagnosis of hypertension in a cohort of university graduates in Spain. BMC Public Health. 2005 Sep;5(1):94. https://doi.org/10.1186/1471-2458-5-94
» https://doi.org/10.1186/1471-2458-5-94 -
7 Taylor A, Dal Grande E, Gill T, Pickering S, Grant J, Adams R, et al. Comparing self-reported and measured high blood pressure and high cholesterol status using data from a large representative cohort study. Aust N Z J Public Health. 2010 Aug;34(4):394-400. https://doi.org/10.1111/j.1753-6405.2010.00572.x
» https://doi.org/10.1111/j.1753-6405.2010.00572.x -
8 Wong AL, Harker JO, Mittman BS, Levy GD, Bulpitt KJ, Colburn KK, et al. Development and evaluation of a patient self-report case-finding method for rheumatoid arthritis. Semin Arthritis Rheum. 2004 Aug;34(1):484-99. https://doi.org/10.1016/j.semarthrit.2003.12.002
» https://doi.org/10.1016/j.semarthrit.2003.12.002 -
9 Douglass CW, Berlin J, Tennstedt S. The validity of self-reported oral health status in the elderly. J Public Health Dent. 1991;51(4):220-2. https://doi.org/10.1111/j.1752-7325.1991.tb02218.x
» https://doi.org/10.1111/j.1752-7325.1991.tb02218.x -
10 Palmqvist S, Söderfeldt B, Arnbjerg D. Self-assessment of dental conditions: validity of a questionnaire. Community Dent Oral Epidemiol. 1991 Oct;19(5):249-51. https://doi.org/10.1111/j.1600-0528.1991.tb00160.x
» https://doi.org/10.1111/j.1600-0528.1991.tb00160.x -
11 Abbood HM, Hinz J, Cherukara G, Macfarlane TV. Validity of self-reported periodontal disease: a systematic review and meta-analysis. J Periodontol. 2016 Dec;87(12):1474-83. https://doi.org/10.1902/jop.2016.160196
» https://doi.org/10.1902/jop.2016.160196 - 12 Nilsson IM, List T, Drangsholt M. The reliability and validity of self-reported temporomandibular disorder pain in adolescents. J Orofac Pain. 2006;20(2):138-44.
-
13 Farmer J, Ramraj C, Azarpazhooh A, Dempster L, Ravaghi V, Quiñonez C. Comparing self-reported and clinically diagnosed unmet dental treatment needs using a nationally representative survey. J Public Health Dent. 2017 Sep;77(4):295-301. https://doi.org/10.1111/jphd.12205
» https://doi.org/10.1111/jphd.12205 -
14 Joshipura KJ, Pitiphat W, Hung HC, Willett WC, Colditz GA, Douglass CW. Pulpal inflammation and incidence of coronary heart disease. J Endod. 2006 Feb;32(2):99-103. https://doi.org/10.1016/j.joen.2005.10.039
» https://doi.org/10.1016/j.joen.2005.10.039 -
15 Caplan DJ, Pankow JS, Cai J, Offenbacher S, Beck JD. The relationship between self-reported history of endodontic therapy and coronary heart disease in the Atherosclerosis Risk in Communities Study. J Am Dent Assoc. 2009 Aug;140(8):1004-12. https://doi.org/10.14219/jada.archive.2009.0311
» https://doi.org/10.14219/jada.archive.2009.0311 -
16 Cyrino RM, Cota LOM, Lages EJP, Lages EMB, Costa FO. Evaluation of self-reported measures for prediction of periodontitis in a sample of Brazilians. J Periodontol. 2011 Dec;82(12):1693-704. https://doi.org/10.1902/jop.2011.110015
» https://doi.org/10.1902/jop.2011.110015 -
17 Oliveira JA, Hoppe CB, Gomes MS, Grecca FS, Haas AN. Periodontal disease as a risk indicator for poor physical fitness: a cross-sectional observational study. J Periodontol. 2015 Jan;86(1):44-52. https://doi.org/10.1902/jop.2014.140270
» https://doi.org/10.1902/jop.2014.140270 -
18 Hoppe CB, Oliveira JA, Grecca FS, Haas AN, Gomes MS. Association between chronic oral inflammatory burden and physical fitness in males: a cross-sectional observational study. Int Endod J. 2017 Aug;50(8):740-9. https://doi.org/10.1111/iej.12686
» https://doi.org/10.1111/iej.12686 -
19 De Moor RJ, Hommez GM, De Boever JG, Delmé KI, Martens GE. Periapical health related to the quality of root canal treatment in a Belgian population. Int Endod J. 2000 Mar;33(2):113-20. https://doi.org/10.1046/j.1365-2591.2000.00295.x
» https://doi.org/10.1046/j.1365-2591.2000.00295.x -
20 Joshipura KJ, Pitiphat W, Douglass CW. Validation of self-reported periodontal measures among health professionals. J Public Health Dent. 2002;62(2):115-21. https://doi.org/10.1111/j.1752-7325.2002.tb03431.x
» https://doi.org/10.1111/j.1752-7325.2002.tb03431.x -
21 Gilbert GH, Litaker MS. Validity of self-reported periodontal status in the Florida dental care study. J Periodontol. 2007 Jul;78(7 Suppl):1429-38. https://doi.org/10.1902/jop.2007.060199
» https://doi.org/10.1902/jop.2007.060199 -
22 Heaton B, Gordon NB, Garcia RI, Rosenberg L, Rich S, Fox MP, et al. A clinical validation of self-reported periodontitis among participants in the black women’s health study. J Periodontol. 2017 Jun;88(6):582-92. https://doi.org/10.1902/jop.2017.160678
» https://doi.org/10.1902/jop.2017.160678 -
23 Timmerman A, Calache H, Parashos P. A cross sectional and longitudinal study of endodontic and periapical status in an Australian population. Aust Dent J. 2017 Sep;62(3):345-54. https://doi.org/10.1111/adj.12512
» https://doi.org/10.1111/adj.12512 -
24 Khalighinejad N, Aminoshariae A, Kulild JC, Sahly K, Mickel A. Association of end-stage renal disease with radiographically and clinically diagnosed apical periodontitis: a hospital-based study. J Endod. 2017 Sep;43(9):1438-41. https://doi.org/10.1016/j.joen.2017.04.014
» https://doi.org/10.1016/j.joen.2017.04.014 -
25 Estrela C, Bueno MR, Leles CR, Azevedo B, Azevedo JR. Accuracy of cone beam computed tomography and panoramic and periapical radiography for detection of apical periodontitis. J Endod. 2008 Mar;34(3):273-9. https://doi.org/10.1016/j.joen.2007.11.023
» https://doi.org/10.1016/j.joen.2007.11.023 -
26 Leonardi Dutra K, Haas L, Porporatti AL, Flores-Mir C, Nascimento Santos J, Mezzomo LA, et al. Diagnostic accuracy of cone-beam computed tomography and conventional radiography on apical periodontitis: a systematic review and meta-analysis. J Endod. 2016 Mar;42(3):356-64. https://doi.org/10.1016/j.joen.2015.12.015
» https://doi.org/10.1016/j.joen.2015.12.015 -
27 Blicher B, Joshipura K, Eke P. Validation of self-reported periodontal disease: a systematic review. J Dent Res. 2005 Oct;84(10):881-90. https://doi.org/10.1177/154405910508401003
» https://doi.org/10.1177/154405910508401003 -
28 Chatzopoulos GS, Tsalikis L, Konstantinidis A, Kotsakis GA. A two-domain self-report measure of periodontal disease has good accuracy for periodontitis screening in dental school outpatients. J Periodontol. 2016 Oct;87(10):1165-73. https://doi.org/10.1902/jop.2016.160043
» https://doi.org/10.1902/jop.2016.160043 -
29 Quiroz V, Reinero D, Hernández P, Contreras J, Vernal R, Carvajal P. Development of a self-report questionnaire designed for population-based surveillance of gingivitis in adolescents: assessment of content validity and reliability. J Appl Oral Sci. 2017 Jul-Aug;25(4):404-11. https://doi.org/10.1590/1678-7757-2016-0511
» https://doi.org/10.1590/1678-7757-2016-0511
Publication Dates
-
Publication in this collection
11 Feb 2019 -
Date of issue
2019
History
-
Received
03 July 2018 -
Reviewed
26 Nov 2018 -
Accepted
17 Dec 2018
