Impact of Non-Weighting in the Analysis of Data Obtained from Complex Samples

Abstract

Objective:

To compare the estimates obtained, considering or not the weighting data.

Material and Methods:

Secondary data from the Oral Health Survey of the State of São Paulo (SBSP2015) was used for calculation of mean estimates, standard errors of the mean and confidence intervals (CI) for the DMFT index and components (decayed, lost and filled), in the age group of 35-44 years. Multiple logistic regression models were estimated, considering or not the weighting from the sampling plan (p<0.05).

Results:

It was observed that the estimates of the DMFT index and the carious component did not vary much when the design was considered or not (1.1% and 2.0%, respectively). However, the data referring to the lost and filled component showed greater differences between the values of the means. The averages fluctuated up and down by up to 6.7% for weighted versus unweighted analyses. The standard error was underestimated in the unweighted analysis and the confidence interval showed variations. Differences between the regression models obtained by the weighted and unweighted analysis of the data were detected.

Conclusion:

Although weighted and unweighted models presented differences of less than 10% in estimates of the mean, confidence intervals, as well as statistical inferences, were different. Thus, weighting should be applied in the population base data analysis collected by sampling with complex designs.

Keywords:
DMF Index; Statistics; Health Surveys

Introduction

Population-based epidemiological studies require a design that allows the characteristics of the studied population to be analyzed, using a representative sample plan of the population, where each element of the population has a known and non-zero probability of being selected. Thus, one of the ways to guarantee the representativeness of the data in the population is the randomization by probabilistic sampling [^1,2].

Studies by probabilistic sampling allow the researcher to observe the representation of the phenomenon studied in the population and calculate confidence intervals and statistical significance. For this, the researcher must define in the study planning which sampling method will be used, among them, Simple Random Sampling (SRS), Systematic Sampling (SS), Stratified Random Sampling (AE) and Cluster Sampling (AC). The last one, which guides our research, is used in large-scale population surveys, where each sampling unit is a group of elements [³].

A major challenge for population-based surveys is the analysis of data collected by complex sampling (a combination of several probabilistic sampling methods for selecting a sample). Using a combined method of sampling allows the researcher to visit compact areas instead of dispersed areas individually, reducing time and expenses [⁴].

In cluster sampling, the population is divided into groups that function as Primary Sampling Units (PSU), which are linked to the clusters. Each cluster is composed of several units and these are called Secondary Sampling Units (SSU). Within the PSU, a sample can be selected that can have one or more stages. Thus, this type of sampling is recommended for populations of states, cities, neighborhoods, and census sectors [⁵].

It is worth noting that, when investigating only the selected clusters, there is an increase in imprecision due to possible differences in the sample units within and between clusters, making it necessary to increase the sample size. Therefore, the data for analysis should be treated, for they cannot be considered as being independent, as occurs in simple random sampling [^6,7]. The main challenges encountered in analyzing data from complex samples are the measurement of correct point estimates and the estimation of standard deviation and variance and this is very important because to search to estimate an overall mean rather than to estimate means for each group or area. In the studies, there is the analytic choice, but little attention has been paid to the theoretical justification for this way of proceeding [^8,9].

When the sample units are presented with variant probabilities, it is advisable to assign weight to the analysis. As an example, there is the State of São Paulo, where there is a city with more than 12 million inhabitants and another one with less than 3 thousand. One way to make the representativeness of a grievance be compared among the locations would be by assigning weights to these samples [¹⁰,¹¹]. However, the literature has many studies that use regressions to assess the association of variables that did not weigh their data for complex samples.

In the literature, we find clear evidence of the importance of weighting for data from complex samples. However, many researchers do not understand or are not concerned with the impact on estimates (mean, standard deviation, and confidence interval), in addition to the significance of logistic regressions.

For that matter, the objective of the present study was to present a trial on the influence of weighting on population estimates and on the statistical significance of the variables analyzed in logistic regressions, using the database from the epidemiological survey on oral health of the State of São Paulo, in the adult population aged 35 to 44 years.

Material and Methods

Study Design

This study is a statistical trial, which compares estimates obtained from weighted and unweighted samples. To this end, secondary data were used referring to the State Oral Health Research (SB São Paulo Project - SBSP) in the adult population between 35 and 44 years old [¹¹].

The SBSP2015 survey used the WHO methodology to estimate the prevalence of oral diseases (caries experience, use and need of prosthesis, periodontal condition, toothache, socioeconomic characteristics, and impact on oral health) for each of the six macroregions (domains) and for the State of São Paulo, considering the age groups under analysis. The total number of municipalities in the state of São Paulo is 645. A total of 178 cities plus the state capital were drawn for the study (1st draw stage - PSU Primary Sampling Units) with representation for six macroregions (São Paulo Capital, Metropolitan Region of São Paulo and the Regional Health Departments – RHD - 2 to 17- as shown in Figure 1).

Figure 1
Regional Health Departments of the State of São Paulo and the macro-regions.

In the second stage, 390 census sectors, or Secondary Sampling Units - SSU (2 sectors for 178 municipalities and 36 sectors for the city of São Paulo) were drawn. Given the characteristics of this research, only urban sectors were used. The drawing of sectors was carried out with probability proportional to the number of inhabitants in each of the municipalities. The depletion technique with minimum sample size for each SSU was used, where all households in the SSU were visited, with individuals from the studied age group being examined.

The census sector is a spatial division of the territory defined by the IBGE with approximately 300 households. Therefore, having in hand the sector map and the IBGE projection for the size of the population stratified for the age group in each sector drawn, a team of “Enrollers/Beaters” visited all households in the census sectors, identifying all residents who were eligible (age indexes of the survey) and informing them of the survey. As each census sector presented distinct characteristics of population density by age group, all occupied households were visited, and eligible residents in the age groups were registered on the enrollment form.

Subsequently, the teams of examiners and markers traveled throughout the sector, inviting them to participate in the research (signing the informed consent form, examining and interviewing people in the index age) until the sector was depleted and the minimum number expected was reached. If this did not happen, the sector was covered again. All information from the residents examined/interviewed, as well as those who were absent and those who refuse to participate in the survey, served as data to weigh the results, calculating the density rate by census sector and the non-response rate, being fundamental information for correction analysis, for they generate weights for weighting within the sector of the municipality and the macroregion.

For the calculation of the sample of adults (age range 35-44 years) and adjustment of the sample to the size of the population, the parameters of the SB Brazil 2010 survey for the southeast region were used [¹¹].

Each domain had a minimum sample size for each of the six macroregions (n = 687 for calculating the sample for dental caries). However, as other conditions were also investigated (periodontal disease, use and need of prosthesis), the minimum sample size per macroregion was 951 people, which was divided by 30 (considering that this is the minimum PSU per macroregion) and reached a minimum n for each examiner per PSU of 32 people. At the end of the survey, 6,051 adults were examined.

In the respective primary and secondary sampling units, information about the conditions and other socioeconomic information was included in the original database. Each macroregion of the State of São Paulo (n = 6) was called a domain, where the cities were drawn. Therefore, in each municipality (PSU), it was drawn within its domain (macro) and had a probability proportional to its population size (PPS). In the second stage, the census sectors were drawn in each of the municipalities. Therefore, the census sector is considered the secondary sampling unit (SSU).

It was not possible to carry out examinations in all selected municipalities (PSU) in their respective domains, as well as in some census sectors (SSU), resulting in a number of examinees below the total calculated in the sample, which was already foreseen. This fact causes an imbalance in the sampling process using PPS as a reference. Thus, the non-response rate of the municipalities participating in the research was calculated according to the municipalities drawn.

With the acquisition of the weights for each of the PSUs, the next step is to carry out the integration of the two databases, that is, the weight database with the database of the examined individuals. Both databases are on the same Excel base, where the information was merged so that they had the same reference variable.

To this end, the chosen variable was the name of the municipality (PSU), where, from the identification of this variable, the spelling of the municipalities was corrected using the SEERRO function of the Excel program, which nests the information of two databases, having in common the same reference variable.

Data Analysis

Estimates of means, standard errors of the mean and confidence intervals were calculated for the DMFT index (decayed, missing, and filled teeth), considering the complex design (with weighting) and also considering as simple random sampling (without weighting).

Then, the DMFT index was dichotomized by the median and the associations with the independent variables sex, ethnicity, income, time of study, and visits to the dentist were analyzed. For that, simple and multiple logistic regression models were estimated, considering or not the complex design. For the analysis considering the complex design, the weights were calculated from the sampling plan and used to adjust the estimates according to the distribution within the regions.

Variables with p<0.20 in the individual analyses were tested in the multiple regression models, and those with p=0.05 remained in the models. The adjustment quality was assessed by the Akaike information criterion (AIC) and -2 Log L. All the analyses were performed using the SAS program.

Results

Table 1 shows the means, standard errors of the means and the confidence intervals (CI) for the DMFT index and its components, estimated with and without weighting. It was observed that the estimates of the DMFT index and the carious component did not vary much when the design was considered or not (1.1% and 2.0%, respectively). However, the data referring to the lost and filled component showed greater differences between the values of the means. For these parameters, an increase of 6.1% was observed for the lost component and a reduction of 6.7% for the filled component. It is also noted that the standard error was underestimated when weighting was not considered. Consequently, the confidence interval of the weighted estimates is larger, which can influence the statistical significance.

Table 1
Descriptive analyzes of the DMFT index variables and their components (decayed, missed and filled teeth) in adults aged 35-44 years, considering the complex weighted and unweighted design.

Variables	Unweighted Design			Weighted Design
	Average	S.E.	95% C.I.	Average	S.E.	95% C.I.
DMFT Index	16.01	0.09	15.84-16.19	15.84	0.18	15.48-16.19
Decayed Teeth	1.50	0.04	1.43-1.57	1.53	0.05	1.43-1.64
Missed Teeth	5.94	0.09	5.77-6.11	6.30	0.13	6.04-6.55
Filled Teeth	8.57	0.07	8.43-8.72	8.00	0.11	7.80-8.21

• DMFT: Decayed, Missed and Filled Index; S.E: Standard Error; 95% C.I.: 95% Confidence Interval.

Table 2 shows the results for the simple and multiple logistic regression analyses, considering or not the weights. It is observed that the weighted and unweighted analyses present variables with different significance. The choice of these independent variables was made to reproduce other regression models that seek to estimate the effect of socio-demographic variables and the use of dental services on the caries experience.

Table 2
Results of bivariate and multivariate logistic regression analyzes, considering the unweighted and weighted complex design.

Variables	N(%)	Unweighted Design				Weighted Design
		DMFT<16	DMFT≥16	OR (95% C.I.) (Bivariate)	OR (95% C.I.) (Multivariate)	DMFT<16	DMFT≥16	OR (95% C.I.) (Bivariate)	OR (95% C.I.) (Multivariate)
Sex
Male	1943 (32.1)	1033 (53.2)	910 (46.8)	Ref	Ref	5008363 (54.8)	4137886 (45.2)	Ref	Ref
Female	4108 (67.9)	1815 (44.2)	2293 (55.8)	1.43 (1.29-1.60)	1.40 (1.25-1.57)	9546249 (45.6)	11369567 (54.4)	1.44 (1.31-1.58)	1.45 (1.33-1.59)
Ethnicity
White	3763 (62.2)	1758 (46.7)	2005 (53.3)	Ref		8563626 (47.1)	9623382 (52.9)	Ref	Ref
Not White	2288 (37.8)	1090 (47.6)	1198 (52.4)	0.96 (0.87-1.07)		5983968 (50.4)	5884071 (49.6)	0.88 (0.78-0.98)	0.80 (0.72-0.90)
Family Income
Up to $1500	2224 (41.9)	1026 (46.1)	1198 (53.9)	1.12 (1.00-1.25)		5769096 (49.6)	5865893 (50.4)	1.02 (0.88-1.17)
Above $1500	3085 (58.1)	1509 (48.9)	1576 (51.1)	Ref		6709568 (50.0)	6705469 (50.0)	Ref
Education
Up to 8 years	2470 (43.7)	1043 (42.2)	1427 (57.8)	1.43 (1.28-1.59)	1.49 (1.34-1.66)	5422951 (43.5)	7054236 (56.5)	1.47 (1.30-1.65)	1.50 (1.34-1.68)
9 or more years	3183 (53.3)	1626 (51.1)	1557 (48.9)	Ref	Ref	8089334 (53.0)	7177186 (47.0)	Ref	Ref
Visit to dentist
Never	9 (0.2)	9 (100.0)	0 (0.0)	-		28679 (100.0)	0 (0.0)	-
Once, at least	5817 (99.8)	2743 (47.2)	3074 (52.8)			14122783 (49.0)	14703627 (51.0)
Time since last visit to dentist
Less than 1 year	3185 (55.9)	1448 (45.5)	1737 (54.5)	1.14 (1.02-1.26)	1.18 (1.06-1.31)	7163594 (48.3)	7663029 (51.7)	1.04 (0.93-1.16)
More than 1 year	2509 (44.1)	1221 (48.7)	1288 (51.3)	Ref	Ref	6686898 (49.3)	6872067 (50.7)	Ref
Where visited the dentist
Public Service	2288 (39.7)	1042 (45.5)	1246 (54.5)	1.10 (0.99-1.22)		5339725 (47.0)	6014414 (53.0)	1.12 (1.05-1.20)
Others	3469 (60.3)	1664 (48.0)	1805 (52.0)	Ref		8601896 (50.0)	8618146 (50.0)	Ref

• DMFT: Decayed, Missed and Filled Index; OR: Odds Ratio; 95% C.I.: 95% Confidence Interval.

Differences were observed between the regression models, with and without statistical weighting of the analysis. The ethnicity variable was statistically significant in the weighted analysis (gross and adjusted), but it was not significant in the unweighted analysis. However, the variable “time since the last consultation” was statistically significant in the unweighted analysis (gross and adjusted), but it was not for the weighted analysis.

Discussion

This study used the secondary database of SBSP2015 to exemplify the estimation of parameters with and without the attribution of weights (weighting). The estimates obtained from the mean, standard error of the mean and confidence interval differed from the measurements obtained when weights were not assigned, which can change the significance of the variables. In this sense, the failure to assign weighting to the analysis could lead to an underestimation of the observed standard error [¹²]. In addition, the efficiency of complex sampling can be misused if unequal weight is not attributed due to its unequal probability of representing the sample [¹³].

A cluster sample has the advantage, compared to the simple random sample, that it is cheaper in the cost per element sampled due to a lower expense in the elaboration of registrations and in the location of individuals. However, on the other hand, it implies greater complexity and difficulty in statistical analysis, in addition to the increase in the magnitude of the variance, which, when using a cluster design, is difficult to predict, with a consequent decrease in the precision of the study [^14,15].

As noted, the weighting of the analysis resulted in averages different from those obtained without the weighting. Although the weighting resulted in the expansion of the standard error measure, possibly associated with the imputation of the non-response rate in the calculation of the estimates, this procedure tends to reproduce more accurately the calculation of the estimates in complex samples. If a complex sample is analyzed as if it were a Simple Random Sample (AAS), the magnitude of the variability will be underestimated, which results in a loss of efficiency in the analysis [¹⁶].

Although the importance of weighting data in complex samples is already clear, we observe the non-observance of these procedures in many researches. Obviously, the estimates of the weighted data are different; however, the question is whether these data would be implicated in the estimates of the association and impact measures using logistic regressions. As observed in this study, the weighting resulted in different multiple statistical models from that obtained when the weighting was not performed.

The present study found that the unweighted analyses showed a standard error of the mean lower than the weighted samples, which makes the confidence interval estimate underestimated, which is already expected due to the characteristics of the sample itself. However, the unweighted analysis results in estimates that are incompatible with the sample design of the study. Therefore, the incorporation of weights is strongly recommended. Some authors cite the increase in the standard error measurement as a disadvantage; however, they argue that the estimates obtained by unweighted samples are not valid for the population [^17,18].

When in a population-based survey, data analysis is carried out without taking into account the weighting, it assumes equiprobability among the individuals examined/interviewed, the use of weights corrects the probability of a draw, as the organization in clusters results from the unequal fraction probability of inter and intracluster draw and weighting makes the final result less likely to be irregular [^19,20]. However, there is an increase in variance and confidence intervals, as individuals within clusters tend to be more similar than the population, compared to unweighted analyses [²¹].

Greenland suggests that changes of 10% or more could be considered a criterion for bias selection. Although we cannot find mean with differences of more than 10%, we clearly see them in the SE, in some cases with 100% difference (DMFT), consequently increasing the CI range [²²].

Although it seems obvious the need to weigh complex samples and this is a peaceful point in statistical designs, this quality parameter is not always used in scientific publications, even in impact journals. Our aim was to show how much the means, SE, CI and statistical inference can be modified by weighting. It seems clear that this is the case, but we found a very limited literature on the subject, and many researchers still have doubts about this need, which requires some degree of knowledge of statistics and consideration in complex sample designs. In addition, we could cite as a limitation the fact that this study uses a single sample. However, this sample contains more than 6000 individuals and, therefore, the possibility of using smaller samples would decrease the possibility of verifying the magnitude of the difference in relation to the average, SE and CI.

This finding brings us to the discussion of weighting, as it controls possible measurement biases, while unweighted samples have standard errors and underestimated confidence intervals, which impacts the estimates of significance values, as observed in this trial. The fact that weighting the data and using programs that allow the use of weights in the analyses makes it possible to correct the values, since there is a difference between the sample drawn and the expected exams/surveys, in addition to what is observed and the real one. Thus, it is recommended that non-response rates are corrected in weighting with the use of weights [²³].

There are criticisms regarding the results obtained in the SBBrasil 2003 nationally based survey, precisely because they do not carry out the weighting and use of weights in the data analysis. The draw considered the organization by clusters, but weights were not assigned for the analysis of the results, thus disrupting the estimates. It is emphasized that the greater the variability of the studied variable, the greater the bias of the sample estimate, so the lack of weight makes it difficult to estimate the population results [²⁴].

Conclusion

Although means vary less than 10% between weighted and unweighted analyses, standard errors, confidence intervals and statistical inferences were different, which can be considered a potential bias. Thus, it is recommended that the weighting be applied in the analysis of population-based data collected by sampling with complex designs.

• Financial Support
  None.
• Data Availability
  The data used to support the findings of this study can be made available upon request to the corresponding author.

References

• ^[1]
  Bloch KV, Luiz RR, Werneck GL. Amostragem. In: Medronho RA, Bloch KV, Luiz RR, Werneck GL. Epidemiologia. 2. ed. Rio de Janeiro: Atheneu; 2009. [In Portuguese].
• ^[2]
  Pereira GR. Uma revisão sobre o uso analítico de dados provenientes de amostras complexas. Dissertação (Mestrado) – Esalq/USP, Piracicaba; 2016. [In Portuguese]. https://doi.org/10.11606/D.11.2016.tde-28112016-144856
  » https://doi.org/10.11606/D.11.2016.tde-28112016-144856
• ^[3]
  Hulley SB, Newman TB, Cummings SR. Escolhendo Sujeitos do Estudo: Especificação, Amostragem e Recrutamento. In: Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Delineando a Pesquisa Clínica. 4. ed. Porto Alegre: Artmed; 2015. [In Portuguese].
• ^[4]
  Cunningham SD, Huguet N. Weighting and Complex Samples Design Adjustments in Longitudinal Studies. New York, NY: Routledge Taylor & Francis Group; 2012.
• ^[5]
  Bolfarine H, Bussab WO. Elementos de Amostragem. São Paulo: Edgar Blücher; 2005. [In Portuguese].
• ^[6]
  Luiz RR, Magnanini MMF. The logic of sample size determination in epidemiological research. Cad Saúde Colet 2000; 8(2):9-28.
• ^[7]
  Szwarcwald CL, Damacena GN. Complex sampling design in population surveys: planning and effects on statistical data analysis. Rev Bras Epidemiol 2008; 11(Suppl 1):38-45. https://doi.org/10.1590/S1415-790X2008000500004
  » https://doi.org/10.1590/S1415-790X2008000500004
• ^[8]
  Kreuter F, Valliant R. A survey on survey statistics: what in done and can be done in Stata. Stata J 2007; 7(1):1-21. https://doi.org/10.1177/1536867X0700700101
  » https://doi.org/10.1177/1536867X0700700101
• ^[9]
  Frohlich N, Carriere KC, Potvin L, Black C. Assessing socioeconomic effects on different sized populations: to weight or not to weight? J Epidemiol Community Health 2001; 55(12):913-20. https://doi.org/10.1136/jech.55.12.913
  » https://doi.org/10.1136/jech.55.12.913
• ^[10]
  Lumley T. Analysis of complex survey samples. J Stat Softw 2004, 9(1):1-19. https://doi.org/10.18637/jss.v009.i08
  » https://doi.org/10.18637/jss.v009.i08
• ^[11]
  Pereira AC, Vieira V, Frias AC. Pesquisa Estadual de Saúde Bucal - Relatório Final. Águas de São Pedro: Livronovo; 2016. 122p. [In Portuguese].
• ^[12]
  Korn EL, Graubard BI. Epidemiologic studies utilizing surveys: accounting for the sampling design. Am J Public Health 1991; 81(9):1166-73. https://doi.org/10.2105/AJPH.81.9.1166
  » https://doi.org/10.2105/AJPH.81.9.1166
• ^[13]
  Sakshaug JW, West BT. Important considerations when analyzing health survey data collected using a complex sample design. Am J Public Health 2014; 104(1):15-6. https://doi.org/10.2105/AJPH.2013.301515
  » https://doi.org/10.2105/AJPH.2013.301515
• ^[14]
  Cordeiro R. Effect of design in cluster sampling to estimate the distribution of occupations among workers. Rev Saúde Pública 2001; 35(1):10-5. https://doi.org/10.1590/S0034-89102001000100002
  » https://doi.org/10.1590/S0034-89102001000100002
• ^[15]
  Cochran WG. Técnicas de Amostragem. Rio de Janeiro: Agência Norte-Americana para o Desenvolvimento Internacional e Editora Fundo de Cultura; 1965. [In Portuguese].
• ^[16]
  Perez MC, Utra IB, León AA, Roche RG, Sagué KA, de la Rosa MC, et al. Estimate methods used with complex sampling designs: their application in the Cuban 2001 health survey. Rev Panam Salud Publica 2004; 15(3):176-84. https://doi.org/10.1590/S1020-49892004000300006
  » https://doi.org/10.1590/S1020-49892004000300006
• ^[17]
  Du Mouchel WH, Duncan GJ. Using sample survey weights in multiple regression analysis of stratified samples. J Am Stat Assoc 1983; 78(383):535-43. https://doi.org/10.2307/2288115
  » https://doi.org/10.2307/2288115
• ^[18]
  Ciol MA, Hoffman JM, Dudgeon BJ, Shumway-Cook A, Yorkston KM, Chan L. Understanding the use of weights in the analysis of data from multistage surveys. Arch Phys Med Rehabil 2006; 87(2):299-303. https://doi.org/10.1016/j.apmr.2005.09.021
  » https://doi.org/10.1016/j.apmr.2005.09.021
• ^[19]
  Kalton G. Introduction to Survey Sampling. Beverly Hills: Sage Publications; 1983. https://doi.org/10.4135/9781412984683
  » https://doi.org/10.4135/9781412984683
• ^[20]
  Cox BG, Cohen SB. Methodological Issues for Health Care Surveys. New York: Marcel Dekker; 1985. https://doi.org/10.1002/bimj.4710290103
  » https://doi.org/10.1002/bimj.4710290103
• ^[21]
  Sousa MH, Silva NN. Estimates from a complex survey. Rev Saúde Pública 2003; 37(5):662-70. https://doi.org/10.1590/s0034-89102003000500018
  » https://doi.org/10.1590/s0034-89102003000500018
• ^[22]
  Greenland S. Invited Commentary: Variable selection versus shrinkage in the control of multiple confounders. Am J Epidemiol 2008; 167(5):523-9; discussion 530-1. https://doi.org/10.1093/aje/kwm355
  » https://doi.org/10.1093/aje/kwm355
• ^[23]
  Cochran WG. Sampling Techiniques. 3rd ed. New York: John Wiley & Sons; 1977.
• ^[24]
  Queiroz RCS. Portela MC. Vasconcelos MTLV. Brazilian Oral Health Survey (SB Brazil 2003): data do not allow for population estimates, but correction is possible. Cad Saúde Pública 2009; 25(1):47-58. https://doi.org/10.1590/S0102-311X2009000100005
  » https://doi.org/10.1590/S0102-311X2009000100005

Edited by

Academic Editor: Ana Maria Gondim Valença

Publication Dates

• Publication in this collection
  30 July 2021
• Date of issue
  2021

History

• Received
  21 Aug 2020
• Reviewed
  11 Jan 2021
• Accepted
  19 Jan 2021