Abstract
The aim of the present study was to confirm the discriminant validity (obtained using traditional statistical methods) of the Early Childhood Oral Health Impact Scale (ECOHIS) between preschool children with and without caries (mean score) through an evaluation of the effect size. A systematic search of electronic databases and a manual search were performed for studies published up to December 2015 involving the use of the ECOHIS for the evaluation of the impact of dental caries on oral health–related quality of life (OHRQoL) among preschool children. Two independent raters performed the selection of the studies and data extraction. Only papers published in English and Spanish were selected. No restrictions were imposed regarding the year of publication. Twelve studies were included, and the magnitude of standardized differences between the means of the “without caries” and “with caries” groups was calculated using Cohen’s d. Most studies demonstrated a large magnitude in the difference between the groups evaluated. The estimate of the effect size confirmed the discriminant validity of the ECOHIS obtained through traditional statistics. Thus, the magnitude of the difference should be considered an important analytical tool for the confirmation of statistical findings regarding null hypotheses and demonstrates the clinical significance of these research results.
Child, Preschool; Dental Caries; Quality of Life; Surveys and Questionnaires
Introduction
Studies addressing oral health–related quality of life (OHRQoL) among preschool children using the Early Childhood Oral Health Impact Scale (ECOHIS) have demonstrated that dental caries exert an impact mainly on the “symptoms”, “function” and “psychological” domains of the Child Impact portion of the scale as well as the “distress” domain of the Family Impact portion.1,2,3,4,5,6,7 Assessing the discriminant validity of the ECOHIS in groups of children with and without caries, such studies have used traditional statistical methods involving the testing of the null hypothesis. However, some studies have demonstrated that this method is limited, as it may sometimes mask the presence of an effect.8,9 In contrast, results from the analysis of the effect size are more precise and detailed, demonstrating the clinical significance of findings rather than merely focusing on statistical significance.10 Thus, the analysis of the effect size is recommended to overcome the limitation of traditional hypothesis testing and can be used to compare the results obtained with the two methods.10
The analysis of the effect size results in information on the real significance of an effect of an adverse health condition or intervention in addition to the concept of statistical significance.11 Statistical significance depends considerably on the sample size, as large samples lead to a statistically significant result, even if the association between variables is of a small magnitude without clinical relevance. Real significance is given by the description of the observed effect size,12 which allows the reader to interpret the importance of the findings.13 Unlike hypothesis tests that either accept or reject differences between the groups studied, the analysis of the effect size furnishes information regarding the magnitude of the relationship found between the outcome and explanatory factors.14 This means that the effect size is useful in determining the practical or theoretical importance of an effect and the relative contribution of different factors or the same factor under different circumstances.13 Different effect size measures are available and several may be appropriate for a given dataset.11
The description of the effect size in scientific papers in the health field can be considered incipient. In papers published in the Journal of Experimental Psychology: General in 2009 and 2010, for example, the effect size of the findings was calculated in less than half of the studies. Moreover, nearly no interpretation of the effect size was offered in the studies that estimated this aspect.13
To confirm the discriminant validity of the ECOHIS obtained through traditional statistical methods, we evaluated the magnitude of the differences found in studies that have used this scale in groups of preschool children with and without dental caries.
Methodology
The inclusion criteria for the studies in the present investigation consisted of 1) the use of preschool children as the study subjects; 2) the use of the ECOHIS to determine OHRQoL; and 3) the use of dental caries (regardless of the index used) as the method of distinguishing groups within a study.
The ECOHIS is an OHRQoL assessment tool developed in the United States1 for the evaluation of the impact of oral problems on the quality of life of preschool children (0 to 5 years of age) and their families. This scale has 13 items distributed between the Child Impact Section (CIS) and Family Impact Section (FIS). The CIS has four domains: symptoms, function, psychological aspects and self-image/social interaction. The FIS has two domains: family distress and family function. Each item has five response options regarding the frequency of an event in the life of the child: 0 = never, 1 = hardly ever, 2 = occasionally, 3 = often, 4 = very often and 5 = “I don’t know” (which is not counted when the scores are summed). The score is calculated by the sum of the response codes of the items. The CIS and FIS scores range from 0 to 36 and 0 to 16, respectively. The total score ranges from 0 to 52, with higher scores indicating a greater impact on OHRQoL. The time of reference for the questions is the child’s entire life, given the infrequent nature of oral problems/treatment and the young age of the children for which the scale was developed.
Electronic searches of the Latin American and Caribbean Health Sciences, Web of Science and Medline databases were performed for articles published in English and Spanish using the following key words: Child, Preschool (MeSH term) OR Preschool Child OR Children, Preschool OR Preschool Children AND Dental caries (MeSH term) OR Dental Decay OR Caries, Dental OR Decay, Dental AND ECOHIS. Correlate key words in Spanish were also used. A systematic search of electronic databases was performed for studies published up to December 2015. Additionally, a manual search was performed of the list of references of all studies pre-selected from the electronic search.
Two independent raters (M.C.F. and M.L.R.J.) performed the analysis of the titles and abstracts, followed by the complete reading of pre-selected articles. Disagreements between raters were resolved by consensus. In cases for which no consensus was reached, a third rater (L.S.M.) was consulted to decide the eligibility of the study.
The following data were extracted from each selected article: study design, type of study (validation or application), type of sampling, sample size and mean + standard deviation of ECOHIS score (outcome) of groups with and without caries as well as respective p-values. The inclusion of validation and application studies is justified by the fact that both designs present data on children with and without dental caries. The difference between the two is that the psychometric properties of the measure are evaluated in a validation study, but not in an application study, as this type of evaluation has previously been performed. The validation studies included in the present review revealed discriminant validity, meaning that the total ECOHIS score is capable of distinguishing children with and without dental caries.
The groups (children with and without caries) in the selected studies were compared using the effect size, which was determined through the mean + standard deviation values of the ECOHIS scores (outcome variable). The effect size was calculated using the model proposed by Cohen8 for standardized differences between the means of two independent groups: Cohen’s d = X1 – X2/s’, with s’ = , for which s’ is the combined standard deviation of both groups, s1 is the standard deviation of the first group (without caries) and s2 is the standard deviation of the second group (with caries). In the present study, X1 was the mean of the first group and X2 was the mean of the second group. Thus, one of the means of the distribution is subtracted from the other and the result is divided by the standard deviation common to both groups. The standardized effect size is calculated considering variability (given by the standard deviation) as well as the differences between means.13 Based on the interpretation criteria proposed by Cohen,15 0.20 < d ≤ 0.40 indicates a small effect size, 0.40 < d < 0.80 indicates a moderate effect size and d ≥ 0.80 indicates a large effect size.
Results
Twenty-nine potentially relevant studies were identified from the electronic databases, 20 of which were pre-selected following the analysis of the title and abstract. Five of the seven studies identified through manual searches of the reference lists were also pre-selected. Thus, 25 pre-selected studies were submitted to full-text analysis, 12 of which (11 from the electronic search and one from the manual search) were included in the present investigation. Figure 1 displays the flowchart of the study selection process. Among the articles excluded in the pre-selection phase was an article that was published in Chinese.
Table 1 displays the effect size findings for validation studies with non-probabilistic sampling. The effect sizes for the studies by Pahel et al.1 and Peker et al.5 ranged from small to moderate when using a cutoff of one to three teeth with caries and large when a cutoff of four or more teeth with caries was used. The effect size was large for the three remaining studies.
Table 2 displays the effect size findings for validation studies with probabilistic sampling. The effect size was moderate for the study conducted by Scarpelli et al.4, who used a cutoff of one to three teeth with caries, and in the study conducted by Martins-Júnior et al.7, who used a cutoff of one or more teeth with caries. The effect size was large in the study by Scarpelli et al.4 when a cutoff of four or more teeth with caries was used.
Table 3 shows the effect size findings for application studies with non-probabilistic sampling. The effect size was moderate in the study that used a cutoff of one to five teeth with caries16 as well as in the study that used a cutoff of one or more teeth with caries.17 The effect sizes were large when a cutoff of one or more teeth with caries and six or more teeth with caries was used in studies conducted by Lee et al.18 and Abanto et al.16, respectively.
Table 4 displays the effect size findings for application studies with probabilistic sampling. The effect size was moderate or small in studies that used a cutoff of one to four teeth with caries, one or more teeth with caries and greater than four teeth with caries.6,19 The effect size was large for the study conducted by Ramos-Jorge et al.20, who dichotomized dental caries as zero and one or more teeth with caries.
Discussion
The importance of the effect size resides in highlighting the clinical significance of the results of studies. Large, clinically relevant, but non-significant effects may suggest the need for further studies with a greater test power12,21 to furnish sufficiently strong evidence to prove that the differences are indeed statistically significant. Small effects that are clinically irrelevant but statistically significant because of a large sample size may warn researchers against overvaluation of their statistical significance.12,21 Any estimate of the effect size should be evaluated within the context of the study. Thus, for any phenomenon under analysis, the effect size should be presented, along with the relevant conditions under which it was estimated.13 The type of sampling, sample size, floor effect rate for the sample, cutoff point of the independent variable and percentage of “I don’t know” responses can exert an influence on the findings, which should be analyzed with caution by the investigators so as to neither underestimate nor overestimate the differences encountered.
For all the studies analyzed here, comparisons were made between children with no caries and those with caries, although the cutoff points varied. The studies analyzed here established cutoff points equal to or greater than one, one to three, one to four, greater than four, equal to or greater than four, one to five and equal to or greater than six teeth with caries. When the cutoff point was closer to zero, the magnitude of the difference was smaller, which leads one to reflect upon the minimum number of teeth with caries capable of exerting an impact on the quality of life of preschool children. In contrast to the comparison between a group without caries and another group with a number of teeth with caries more distant from zero.1,5,16
In a probabilistic sample, a statistically significant result, but with a moderate effect size, was found in the comparison of children without caries and those with one to three teeth with caries, whereas a large effect size was found in the comparison of children without caries and those with four or more teeth with caries.4 A moderate effect size was also found for a probabilistic sample that demonstrated a high degree of statistical significance with a cutoff point of one or more teeth with caries.7 In this situation, the cutoff point of the explanatory factor (dental caries) for an impact on quality of life likely affected the magnitude of the differences.
The aim of reporting the effect size is to enable the reader a better understanding of the importance of the findings of a given study. A larger effect size denotes a greater impact of the central variable of the study (explanatory factor) on the issue that is being analyzed (outcome). In the present study, caries was found to exert a real (as well as significant) influence on the OHRQoL of preschool children. The discriminant validity of the ECOHIS was confirmed based on the large effect size found for the majority of studies analyzed,1,3,4,5,16,18,20,22 which means that this scale can be used to distinguish groups with different degrees of oral problems.
The size effect was moderate for two validation studies (one with non-probabilistic sampling and one with probabilistic sampling) and two application studies (one with non-probabilistic sampling and one with probabilistic sampling) that used cutoff points of one to three,1,4 one to five16 and more than four teeth with caries.19 Unlike the finding for the study conducted by Scarpelli et al.4 with probabilistic sampling (field study), a large magnitude of the difference was found between a group of children without caries and another group with one to three teeth with caries in a preliminary study with non-probabilistic sampling, which suggests that the magnitude of differences in population-based samples is more discrete. In contrast, a validation study with the same type of sampling had significant findings but a small to negligible effect size (0.27 and 0.08),5 despite the fact that the study used the same cutoff point for dental caries as that used in the previously cited investigation.4 The statistical significance combined with the small effect size in the study conducted by Peker et al.5 demonstrates that not every statistically significant result has clinical relevance.
Moderate effect sizes were found for the studies conducted by Wong et al.6, Martins-Júnior et al.7 and Guedes et al.17, in which the researchers compared ECOHIS scores between children without caries and those with one or more teeth with caries. Wong et al.6 and Martins-Júnior et al.7 used probabilistic sampling, whereas Guedes et al.17 used non-probabilistic sampling. Although sampling differences can result in different effect sizes, the common cutoff point for caries in these studies likely explains the similar findings.
Regarding effect size values, Cohen15 states that the use of reference values (given by the author himself) is recommended only when there is no basis of comparison, such as effects previously established within a field. This was the case in the present investigation, as the authors of most of the studies analyzed did not consider the effect size of their findings, with the exception of Lee et al.3, Martins-Júnior et al.7 and López Ramos et al.22. When typically found values are on the order of 0.2, an effect size of 0.3 may indicate an important effect. In contrast, an effect size of 0.6 may not be considered important if the values typically found in the field of interest are on the order of 0.8.14
The value obtained through the calculation of the effect size can be understood as a probability. If the presupposition of normality of the data distribution is accepted, an interpretation of the effect size value can be made based on a Z distribution table. Such a table provides the proportions of a localized area beneath sections of the standardized normal curve, allowing the determination of what percentage of individuals in the experimental group will exhibit an effect greater than the mean of the control group in future experiments.12,14 Presuming normality in the data distribution in the validation study conducted by Scarpelli et al.4 with a probabilistic sample for which the effect size regarding child impact was 1.11 in the comparison of children without caries and those with four or more teeth with caries. Can be expected that 87% of children with four or more teeth with caries will exceed the mean impact value of the group without caries in future studies, as based on the Z distribution.
The high floor effect rate observed in some of the studies selected exerted an influence on the moderate effect size found for certain comparisons. If most of the respondents answer “never” for questions on the ECOHIS, the groups studied will likely be similar with regard to the impact given by the mean score of the OHRQoL scale, which will obviously lead to a difference with a small magnitude. In the study by Pahel et al.,1 approximately half of the parents/caregivers reported no impact from caries on the quality of life of their children. The characteristics of dental caries in the sample of this study may explain the floor effect for the ECOHIS, as only 69 children had caries and 112 were caries free. In the study by Lee et al.,3 the floor effect for the child and family impact sections (22.6 and 33.0%, respectively) did not exert an influence on the effect size, which was large. The lower floor effect for this study can be related to the fact that the data came from a clinical convenience sample and therefore those individuals already exhibited some type of oral problem, although this is not a general rule, as in a study involving a convenience sample from a dental clinic wherein the floor effect rate was 40.1 and 59.9% for the child and family impact sections, respectively,16 which is nearly twice the rate found for the study by Lee et al.3
The considerable floor effect rate (56.7 and 64.7% for the child and family sections) for the study by Scarpelli et al.4 with a non-probabilistic sample did not exert an influence on the effect size, which suggests the occurrence of a significant difference regarding the floor effect between the groups with and without dental caries (data not shown in the study). However, with a floor effect rate of 51.5 and 64.6% for the child and family impact sections, respectively, in the probabilistic sampling, the effect size was moderate for both sections between the group with one to three teeth with caries and the group without caries. The same occurred in the application study conducted by Wong et al.6 with probabilistic sampling. Most of the parents/caregivers (60.6 to 89.8%) reported that neither their children nor families ever experienced the problems addressed by the ECOHIS. Two population-based studies found a high percentage of “never” responses.7,19 According to Martins-Júnior et al.,7 this may be explained by the fact that the children were recruited during a national child vaccination campaign and were therefore not seeking dental treatment, which could affect the impact measured by the OHRQoL scale. In the study by Kramer et al.,19 the floor effect rate was higher than 80%. According to the authors, this distribution of scores is a characteristic of population-based studies and is likely indicative of individuals with a genuinely low degree of problems. In these population-based studies, the floor effect was probably less discrepant between groups, which may have exerted an influence on the moderate effect size encountered.
As a subjective evaluation, the ECOHIS allows parents/caregivers to have greater awareness of the impact that oral problems have on their children and themselves. Indeed, a lack of awareness in the general public is the reason why the oral health of preschool children is often relegated to a position of secondary importance; thus, many children with untreated dental caries experience physical pain as well as difficulties chewing, sleeping and socializing, with possible effects on self-esteem, growth, weight gain16,18,23,24 and quality of life. Besides the possibility of raising awareness on the part of parents/caregivers, the outcomes derived from the assessment of OHRQoL are considered better measures for communication with policy administrators than merely the use of quantitative measures, such as caries indices.25 Thus, considering the clinical significance of OHRQoL findings derived from the ECOHIS, this scale can be used to furnish more concrete data regarding the impact of oral problems on the daily lives of preschool children.
The use of the effect size in the quantitative description of study outcomes is an important complement to the description of results derived from hypothesis tests. It is not enough to identify that some findings were significant and others were not, as the value of a study goes beyond statistical significance. In the present investigation, the determination of the effect size demonstrated the clinical significance of dental caries, proving that this oral problem exerts an impact on the OHRQoL of preschool children.
Conclusion
The determination of the effect size confirmed the discriminant validity of the ECOHIS. For most of the studies analyzed, the effect size was moderate to large. This means that the ECOHIS furnishes valid findings regarding the OHRQoL of preschool children with caries through the reports of parents/caregivers.
Acknowledgments
This study was supported by the following Brazilian fostering agencies: Coordination of Higher Education (CAPES) of the Ministry of Education, the National Council for Scientific and Technological Development (CNPq) of the Ministry of Science, State of Minas Gerais Research Assistance Foundation (FAPEMIG) and the State of Maranhão Research and Scientific/Technological Assistance Foundation (FAPEMA).
References
-
1 Pahel BT, Rozier RG, Slade GD. Parental perceptions of children’s oral health: the Early Childhood Oral Health Impact Scale (ECOHIS). Health Qual Life Outcomes. 2007;5(1):6. https://doi.org/10.1186/1477-7525-5-6
» https://doi.org/10.1186/1477-7525-5-6 -
2 Li S, Veronneau J, Allison PJ. Validation of a French language version of the Early Childhood Oral Health Impact Scale (ECOHIS). Health Qual Life Outcomes. 2008;6(1):9. https://doi.org/10.1186/1477-7525-6-9
» https://doi.org/10.1186/1477-7525-6-9 -
3 Lee GH, McGrath C, Yiu CK, King NM. Translation and validation of a Chinese language version of the Early Childhood Oral Health Impact Scale (ECOHIS). Int J Paediatr Dent. 2009;19(6):399-405. https://doi.org/10.1111/j.1365-263X.2009.01000.x
» https://doi.org/10.1111/j.1365-263X.2009.01000.x -
4 Scarpelli AC, Oliveira BH, Tesch FC, Leão AT, Pordeus IA, Paiva SM. Psychometric properties of the Brazilian version of the Early Childhood Oral Health Impact Scale (B-ECOHIS). BMC Oral Health. 2011;11(1):19. https://doi.org/10.1186/1472-6831-11-19
» https://doi.org/10.1186/1472-6831-11-19 -
5 Peker K, Uysal O, Bermek G. Cross - cultural adaptation and preliminary validation of the Turkish version of the Early Childhood Oral Health Impact Scale among 5-6-year-old children. Health Qual Life Outcomes. 2011;9(1):18. https://doi.org/10.1186/1477-7525-9-118
» https://doi.org/10.1186/1477-7525-9-118 -
6 Wong HM, McGrath CP, King NM, Lo EC. Oral health-related quality of life in Hong Kong preschool children. Caries Res. 2011;45(4):370-6. https://doi.org/10.1159/000330231
» https://doi.org/10.1159/000330231 -
7 Martins-Júnior PA, Ramos-Jorge J, Paiva SM, Marques LS, Ramos-Jorge ML. Validations of the Brazilian version of the Early Childhood Oral Health Impact Scale (ECOHIS). Cad Saúde Publica. 2012;28(2):367-74. https://doi.org/10.1590/S0102-311X2012000200015
» https://doi.org/10.1590/S0102-311X2012000200015 -
8 Cohen J. A power primer. Psychol Bull. 1992;112(1):155-9. https://doi.org/10.1037/0033-2909.112.1.155
» https://doi.org/10.1037/0033-2909.112.1.155 -
9 Zakzanis KK. Brain is related to behavior (p<.05). J Clin Exp Neuropsychol. 1998;20(3):419-27. https://doi.org/10.1076/jcen.20.3.419.821
» https://doi.org/10.1076/jcen.20.3.419.821 - 10 Ferreira FO, Lima EP, Lana-Peixoto MA, Haase VG. [Selecting neuropsychological tests for use in multiple sclerosis and temporal lobe epilepsy: relevance of effect size estimations]. Interam J Psychol. 2008;42(2):203-17. Portuguese.
-
11 Kirk RE. Practical significance: a concept whose time has come. Educ Psychol Meas. 1996;56(5):746-59. https://doi.org/10.1177/0013164496056005002
» https://doi.org/10.1177/0013164496056005002 - 12 Lindenau JD, Guimarães LSP. [Calculating the effect size in SPSS]. Rev HCPA. 2012;32(3):363-81. Portuguese.
-
13 Fritz CO, Morris PE, Richler JJ. Effect size estimates: current use, calculations, and interpretation. J Exp Psychol Gen. 2012;141(1):2-18. https://doi.org/10.1037/a0024338
» https://doi.org/10.1037/a0024338 - 14 Conboy J. A magnitude de efeito como forma de análise em ciências sociais. Lusiada Psicologia. 2002;1(1):187-210.
- 15 Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale: Erlbaum; 1988.
-
16 Abanto J, Carvalho TS, Mendes FM, Wanderley MT, Bönecker M, Raggio DP. Impact of oral diseases and disorders on oral health-related quality of life of preschool children. Community Dent Oral Epidemiol. 2011;39(2):105-14. https://doi.org/10.1111/j.1600-0528.2010.00580.x
» https://doi.org/10.1111/j.1600-0528.2010.00580.x -
17 Guedes RS, Piovesan C, Antunes JLF, Mendes FM, Ardenghi TM. Assessing individual and neighborhood social factors in child oral health-related quality of life: a multilevel analysis. Qual Life Res. 2014;23(9):2521-30. https://doi.org/10.1007/s11136-014-0690-z
» https://doi.org/10.1007/s11136-014-0690-z -
18 Lee GH, McGrath C, Yiu CK, King NM. A comparison of a generic and oral health-specific measure in assessing the impact of early childhood caries on quality of life. Community Dent Oral Epidemiol. 2010;38(4):333-9. https://doi.org/10.1111/j.1600-0528.2010.00543.x
» https://doi.org/10.1111/j.1600-0528.2010.00543.x -
19 Kramer PF, Feldens CA, Ferreira SH, Bervian J, Rodrigues PH, Peres MA. Exploring the impact of oral diseases and disorders on quality of life of preschool children. Community Dent Oral Epidemiol. 2013;41(4):327-35. https://doi.org/10.1111/cdoe.12035
» https://doi.org/10.1111/cdoe.12035 -
20 Ramos-Jorge J, Pordeus IA, Ramos-Jorge ML, Marques LS, Paiva SM. Impact of untreated dental caries on quality of life of preschool children: different stages and activity. Community Dent Oral Epidemiol. 2014;42(4):311-22. https://doi.org/10.1111/cdoe.12086
» https://doi.org/10.1111/cdoe.12086 - 21 Abelson RP. Statistics as principled argument. Hillsdale: Erlbaum; 1995.
- 22 López Ramos RP, García Rupaya CR, Villena-Sarmiento R, Bordoni NE. Cross cultural adaptation and validation of the Early Childhood Health Impact Scale (ECOHIS) in Peruvian preschoolers. Acta Odontol Latinoam. 2013;26(2):60-7.
-
23 Feitosa S, Colares V, Pinkham J. The psychosocial effects of severe caries in 4-year-old children in Recife, Pernambuco, Brazil. Cad Saúde Pública. 2005;21(5):1550-6. https://doi.org/10.1590/S0102-311X2005000500028
» https://doi.org/10.1590/S0102-311X2005000500028 -
24 Oliveira LB, Sheiham A, Bönecker M. Exploring the association of dental caries with social factors and nutritional status in Brazilian preschool children. Eur J Oral Sci. 2008;116(1):37-43. https://doi.org/10.1111/j.1600-0722.2007.00507.x
» https://doi.org/10.1111/j.1600-0722.2007.00507.x -
25 Allen PF. Assessment of oral health related quality of life. Health Qual Life Outcomes. 2003;1(1):40. https://doi.org/10.1186/1477-7525-1-40
» https://doi.org/10.1186/1477-7525-1-40
Publication Dates
-
Publication in this collection
2017
History
-
Received
21 July 2016 -
Reviewed
13 Feb 2017 -
Accepted
03 Mar 2017