Agreement and reliabilty of an eletronic questionnarie for children (WEBDAFA)

Abstracts

O objetivo deste estudo foi analisar a concordância entre a versão impressa (VI) e a versão eletrônica (VE) de um instrumento para avaliar atividade física (AF) em crianças e verificar a fidedignidade entre o teste-reteste da VE. Participaram do estudo, 230 crianças, de sete a 10 anos, de duas escolas, divididas em dois grupos: a) análise de concordância (n=130), duas aplicações do questionário (VE versus VI); b) análise da fidedignidade (n=127), duas aplicações do questionário eletrônico (teste-reteste). Não houve diferença significativa entre os escores de AF obtida pela VI versus VE (48,6 vs 46,8). O coeficiente de correlação intraclasse (análise de concordância) foi significativo para todos os tipos de AF (p<0,05). Na análise de fidedignidade, a diferença entre os escores de AF do teste-reteste foi de 2,70, a concordância foi superior a 70% e correlação intraclasse foi de 0,84 (IC95%=0,78-0,89). O questionário eletrônico (Webdafa) apresentou índices psicométricos de concordância e fidedignidade adequados quando aplicado em crianças de sete a 10 anos.

Atividade física; Questionário; Web questionário

ORIGINAL ARTICLE

^IUniversidade Federal do Paraná, Pós-graduação em Educação Física, Curitiba, PR, Brasil

^IICentro de Pesquisa em Exercício e Esporte da UFPR, Curitiba, PR, Brasil

^IIIUniversidade Tecnológica Federal do Paraná, Campus Curitiba, PR, Brasil

^IVUniversidade Estadual de Ponta Grossa, Ponta Grossa, PR, Brasil

^VUniversidade de Pernambuco, Escola Superior de Educação Física, Pernambuco, Brasil

^VIUniversidade Federal de Santa Catarina, Departamento de Nutrição, Florianópolis, SC, Brasil

Corresponding author

ABSTRACT

The objective was to analyse the agreement between the printed version (PV) and electronic version (EV) of an instrument to assess PA in children and to verify the reliability between two applications of the EV. The sample included 230 seven to tenyear-old children, from two schools (public and private). The sample was divided into two groups: a) analysis of agreement (n = 130), two applications of the questionnaire (EV vs. PV); b) reproducibility analysis (n = 127), two applications of the EV (test-retest). There was no significant difference between the PV and PE in the PA domain (48,6 vs 46,8). The intraclass correlation coefficient (analysis of concordance) was significant for all types of PA (p <0.05). In the reliability analysis, the difference between the scores of PA test-retest was of 2.70, the concordance was over 70% and the intraclass correlation was 0.84 (95% CI = 0.78 to 0.89). The electronic questionnaire (WEBDAFA) presented appropriate psychometric indices of agreement and reliability when applied to children aged seven to ten years.

Key words: Physical activity; Questionnaire; Web questionnaire.

INTRODUCTION

Physical activity (PA) is associated with health benefits for children and adolescents^1,2. Children who are physically active have better bone and muscle growth³, besides presenting lower chance of developing non-contagious chronic diseases such as obesity, diabetes, high blood pressure, and metabolic syndrome^1-3. Evidence also suggests that behaviors acquired and consolidated during childhood and adolescence are more likely to remain in adulthood^4-6. Thus, the quantification of PA in children becomes important in order to identify its prevalence and also to diagnose AP patterns of population subgroups.

Objective measurements of PA such as direct observation, accelerometry, pedometers, and energy expenditure are accurate and valid in children, however, they are costly and require specific equipment, which limits their application in epidemiological studies^7-9. Subjective measures such as questionnaires, interviews, and diaries based on the self-report of a child or parent are widely used to measure PA^8,9. However, the use of questionnaires to assess the habitual practice of PA in children has limitations because children are involved in spontaneous and intermittent activities, which can result in errors of recall bias in the self-report of PA^10,11, not to mention that there are few valid tools for this population.

In an attempt to minimize these errors and possible misunderstandings in the interpretation of complex questions, questionnaires to assess PA in children have used specific tools with appropriate language for this public¹². In Brazil, the questionnaire typical day of PA and food intake (DAFA, in Portuguese) is the only tool developed and validated specifically for Brazilian children of ages between seven and ten years old. This tool identifies the PA done during a typical day based on illustrative drawings, making the evaluation more attractive and less complex for children¹². The PA section of this tool was validated based on the reports from parents and teachers¹².

Recently studies have used electronic questionnaires via internet to measure different outcomes related to health^13-16. It is believed that this type of application can be more attractive among children and result in more accurate values to measure PA¹⁷. Further advantages relate to financial savings because it eliminates the printing of the questionnaires, it saves time, and it is also possible to apply the tool at different locations simultaneously. This technology gives direct access and greater accuracy in the collection and storage of the data^13,18. Despite all these benefits listed above, studies conducted in Brazil were not found in the literature that have evaluated the agreement and reliability of electronic tools for measuring PA in children.

Faced with the need to provide an tool via the internet that is a consistent and appropriate method to measure the practice of habitual PA in children, minimize costs, and also improve the quality of information obtained, the present study presents the following objectives: analyze the agreement of the measurements of PA in children obtained by applying the printed version (PV) of the DAFA questionnaire versus the electronic version of the WEBDAFA (EV) questionnaire, as well as check the reliability (test-retest) of the PA measurements obtained through the WEBDAFA electronic questionnaire in children.

METHODOLOGICAL PROCEDURES

Study Profile

We selected 230 children between the age of seven and ten who were enrolled in two schools (one public and one private). In each school two groups were selected: a) concordance analysis (n=130) and b) the reproducibility analysis (n=127). The selection of the schools was done intentionally. Eight groups were randomly selected in each school (1st through 4th grade), two from each grade, and all students from each group selected were invited to participate in the study (n=400). After sending consent forms, 250 (return rate of 57.5%) of these returned and 20 of these students were excluded because they were over 10 years old.

In the analysis of the agreement, the students first answered the PV and then latter the EV. The reliability was checked by applying the electronic tool on two occasions. The interval between applications was of one day for the procedures of agreement and test-retest, respectively. The data collection procedures were conducted between the months of October and November 2010 in a computer lab room under the guidance of the researcher and supervision of the teacher responsible for the class.

In the application of both tools, the researcher presented the questionnaire to the students with a concise explanation of all the sections, highlighting the concept of a typical day, meaning the PA done on most days of the week¹² using a vocabulary appropriate for the age group of children and gestures to emphasize the actions and with the help of a projector of images.

Application of printed tool

The printed tool used as the reference method was the DAFA (Typical Day of Physical Activities and Food Intake) questionnaire, which is a structured questionnaire developed with the purpose of obtaining information about the PA habits of a typical day and the Food Intake of children between the ages of seven and ten years old. It was tested with Brazilian schoolchildren and presented adequate psychometric measurements of validity and reproducibility for measuring PA¹². This study's project was approved by the Ethics Committee on Human Research at the State University of Western Paraná (decision 494/2010).

This tool consists of three sections: a) identification of personal data (gender, grade, shift, body mass, and height); b) types of PA done in a usual day (typical day); c) food consumed in five meals during a typical day. This study will present the results regarding the PA section. The representation of each type of PA is done by means of 11 drawings that indicate each category (dancing, walking, playing with a pet, doing household chores, riding a bike, jumping rope, climbing stairs, playing ball, swimming, skateboarding, and doing gymnastics or stretching) and intensity (slow, fast, and very fast) of the PA. In the analysis of the physical activity of each category, a value was assigned for each intensity, slow (=3), fast (=5), and very fast (=9), which can generate scores by category as follows: (0) zero for children who said they had not done any physical activity and 3.5 or 9 for children who marked only one of the three intensities. In the other cases the possibilities of the PA scores would be as follows: 8 (slow + fast), 12 (slow + very fast), 14 (fast + very fast), and 17 (slow + fast + very fast). The overall score of PA would be the result of the sum of all categories of PA, which may result in a PA overall score of 187 presented in the tool. To apply the printed tool we followed the guidelines proposed in the original study¹².

Application of Electronic tool (internet)

The EV of the Typical Day of Physical Activities and Food Intake) questionnaire, named by the authors as Webdafa, was developed through an internet (web) application that used the programming language Hypertext Preprocessor 5 (PHP 5)¹⁹, Java Script²⁰. It has a graphical interface where users interact using Adobe Flash technology²¹. The project is hosted on the site http://www.criancaativaesaudavel.com.br and the form can be accessed through web browsers (Internet Explorer, Mozilla, Google Chrome).

The Webdafa consists of the same structure of the printed tool, differing in only the interface mode for filling out the answers. A management system was added of a database adopted for storing information via the MySQL 5 method^19,22, which makes it possible to generate reports that can be exported to spreadsheets in Excel for Windows (xls). In order to submit answers in the electronic version, the children were instructed to move the mouse's cursor over the picture and double click on it (Figure 1).

Analysis of data

The information on the printed questionnaire (DAFA) was entered into the program Epidata 3.1 and inconsistencies checked by the double entry method. The data from the Webdafa questionnaire were obtained from a spreadsheet from the Excel for Windows program generated by the electronic MySQL5 system¹⁹.

The data was analyzed using the SPSS version 17.0 program for Windows. A descriptive analysis (mean, standard deviation, and frequency percentage) was done to describe the sample. The normality of the data was analyzed using the Kolmogorov-Smirnov test. All the variables of the habitual practice of PA differed from the normality curve and because of this non-parametric tests were used to compare the groups. There was no significant difference (p<0.05) between the scores of PA according to gender and type of school so the analyses were performed grouped. The comparison between the percentages was obtained by the chi-square test.

The agreement was analyzed by comparing the mean of the scores (mean scores of posts) obtained by the printed and electronic versions. Individual analyses were done per question and by the overall PA score through the Mann-Whitney U test. Furthermore, the intraclass correlation coefficients and the percentage of agreement between the answers of the first and second application were used. The overall score of PA was obtained by adding the scores of each type of PA. The dispersion was tested between the methods by the Bland and Altmann test²³. The Kappa test was used to analyze the agreement between the classifications obtained from both methods and because of this the overall score was dichotomized in relation to the median.

For the reliability analysis, the mean difference was checked between the scores of the electronic questionnaire Webdafa obtained on two occasions (test-retest) through the Mann-Whitney U test. The percentage of agreement and intraclass correlation coefficient were also employed in this analysis.

Ethical aspects

All participants in the study were volunteers and the parents or guardians signed the informed consent form authorizing participation in the study. The methodological procedures used in this study were analyzed and approved by the Ethics Committee on Human Research at the State University of Western Paraná (decision 494/2010).

RESULTS

The study sample consisted of two groups from public (42.2%) and private schools with the following characteristics: age of 8.4 years (sd=1.1 years), height of 1.30 m (sd=0.7), body mass of 32.1 kg (sd=7.1), and body mass index of 17.4 kg/m ² (sd=2.6). The sample of agreements was composed of 103 children (50.5% girls) with a predominant age of eight years old. The reproducibility sample was composed of 127 children (45.7% girls). In this sample a higher proportion of girls from public schools and overweight was observed in relation to the number of boys (p<0.05). Other characteristics of the sample are presented in table 1.

There was no significant difference between the PA scores obtained from the PV versus the EV (Table 2). The intraclass correlation coefficient was significant for all types of PA (p<0.05), ranging from 0.79 (household chores) to 0.93 (swimming). Among the 11 types of PA analyzed, only the items playing ball (25.2%), jumping rope (36.8%), and swimming (41%) showed relative agreement of less than 70%. However there was high agreement (88.3%) of the overall score of PA and a kappa index of 0.76 (p<0.001) when comparing the questionnaires printed versus electronic (data not shown). The dispersion analysis (figure 2) indicated a mean error of 1.7 (CI95% = -25.6; 29.1). PA scores, low and medium, showed good agreement, but the higher scores presented higher dispersion.

Reliability was checked by test-retest (table 3). All items presented agreement (%C) above 70% and an intraclass correlation coefficient between 0.84 (riding bicycle) and 0.95 (playing with a skateboard). There was a significant difference (p<0.05) in the PA scores between the two applications of the EV of the WEBDAFA questionnaire only for the types of PA "household chores and riding bicycle" (table 3).

DISCUSSION

The analysis of agreement between the printed and electronic methods showed significant intraclass correlation values for most items analyzed (ICC> 0.79). The overall PA score obtained by the two methods did not differ statistically, indicating agreement between the methods. The reliability analysis indicated an agreement percentage higher than 70% and a high intraclass correlation coefficient (ICC> 0.84) for all items of the electronic tool. Among the 11 categories of PA, only three had mean scores that differed (p<0.05). This indicates appropriate values of reliability for the tool applied in the electronic version on children.

The Webdafa tool was designed based on the use of new electronic technologies to collect health-related information. The mode of application aroused the interest of the child to answer the tool and to fill it out was fun and fast. The electronic application of the tool has advantages such as how to eliminate the cases missing from the analysis. There is no need to enter the results from the questionnaires into the database, thus eliminating the chance of typing errors, saving time in applying the questionnaire, and the data can be available in real time via internet for different locations. These advantages listed have encouraged some researches and surveys via internet with various health-related outcomes^24,25 .

In general, tools to assess PA in an electronic format arouse the interest in children the most^25-29. In general, when analyzed individually, the measures of the habitual PA obtained through electronic questionnaires are undervalued and have low or moderate correlation with measurements obtained from accelerometers^27-29. However, according to some authors, the electronic questionnaires are more reliable estimates for the PA of groups than the individual estimates^28,29.

Although this study did not conduct objective measures of PA, no differences were observed between the sum of the PA scores obtained by the printed questionnaire versus the electronic questionnaire. Even when the analyses were made between gender, these results corroborate with studies done previously with this type of tool^25,27,29.

In the reliability analysis, the Webdafa electronic questionnaire showed intraclass correlation coefficients between 0.73 to 0.93. This data is superior to those found in the original version, which ranged from 0.50 to 0.83³⁰. In general, electronic questionnaires have shown high intraclass correlation coefficients^17,28,29. This demonstrates that electronic questionnaires have a good internal consistency and stability, which suggests that they are promising to use in epidemiologic surveys because it is an inexpensive method, viable, and useful for monitoring levels (behavior) of habitual PA of children by group level.

Currently a significant proportion of schools in southern and southeastern Brazil have computers connected to the worldwide network, which can provide a large number of children with immediate access to the Webdafa electronic questionnaire with relative ease. In theory, researchers from various regions of Brazil could investigate the PA in children simultaneously. Thus, we suggest that further studies be conducted using the Webdafa questionnaire with different population groups.

The intentional selection of schools and classes, the short period between the first application (test) and second application (retest), as well as the lack of a reference measurement for validating the PA scores obtained by the electronic questionnaire may be considered the main limitations of this study. Therefore, we recommend that further studies of agreement and validation of the WEBDAFA questionnaire be done in other regions of Brazil with children from different social classes, especially where they do not have much familiarity with computers. Because of this, the results of this study should be treated with caution and cannot be extrapolated to other regions of Brazil.

CONCLUSION

Finally, it is concluded that the Webdafa questionnaire applied in its electronic form presented values of agreement and reliability suitable for measuring PA in children between seven and ten years old.

REFERENCES

• 1. Hallal PC, Victora CG, Azevedo MR, Wells JCK. Adolescent physical activity and health: a systematic review. Sports Med 2006;36(12):1019-30.
• 2. Twisk JWR. Physical activity guidelines for children and adolescents: a critical review. Sports Med 2001;31(8):617-27.
• 3. Janssen I. Physical activity guidelines for children and youth. Appl Physiol Nutr Metab 2007;32(2):S109-S121.
• 4. Azevedo MR, Araújo CL, Silva MC, Hallal PC. Tracking of physical activity from adolescence to adulthood: a population-based study. Rev Saude Publica 2007;41(1):69-75.
• 5. Boreham C, Robson PJ, Gallagher AM, Cran GW, Savage JM, Murray LJ. Tracking of physical activity, fitness, body composition and diet from adolescence to young adulthood: The Young Hearts Project, Northern Ireland. Int J Behav Nutr Phys Act 2004;1(1):1-14.
• 6. Gordon-Larsen P, Nelson MC, Popkin BM. Longitudinal physical activity and sedentary behavior trends: Adolescence to adulthood. Am J Prev Med 2004;27(4):277-83.
• 7. Kohl HW, Fulton JE, Caspersen CJ. Assessment of physical activity among children and adolescents: a review and synthesis. Prev Med 2000;31(2):S54-S76.
• 8. Sallis JF, Saelens BE. Assessment of physical activity by self-report: status, limitations, and future directions. Res Q Exerc Sport 2000;71(2 Suppl):S1-14.
• 9. Farias Júnior JC, Lopes AS, Florindo AA, Hallal PC. Validity and reliability of self-report instruments for measuring physical activity in adolescents: a systematic review. Cad Saúde Pública 2010;26(9):1669-91.
• 10. Cale l, Harris J. Self-report measures of children's physical activity: recommendations for future development and a new alternative measure. Health Educ J 1994;53:439-53.
• 11. Welk GJ, Corbin CB, Dale D. Measurement issues in the assessment of physical activity in children. Res Q Exerc Sport 2000;71(2 Suppl):S59-73.
• 12. Barros MVG, Assis MAA, Pires MC, Grossemann S, Vasconcelos FAG, Luna MEP, et al. Validity of physical activity and food consumption questionnaire for children aged seven to ten years old. Rev Bras Saude Mater Infant 2007;7(4):437-48.
• 13. Davis RN. Web-based administration of a personality questionnaire: Comparison with traditional methods. Behav Res Methods 1999;31(4):572-7.
• 14. Resiti TMP, Annelette WM, Johan EVDBR, Harry VDL, Hein J de K, Hein R. Internet-administered adolescent Health questionnaires compared with a paper version in a randomized study. J Sch Health 2005;36:70.e1-70.e6.15.
• 15. Ekman A, Dickman PW, Klint A, Weiderpass E, Litton JE. Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol 2006;21(2):103-11.
• 16. Mangunkusumo RT, Duisterhout JS, de Graaff N, Maarsingh EJ, de Koning HJ, Raat H. Internet Versus Paper Mode of Health and Health Behavior Questionnaires in Elementary Schools: Asthma and Fruit as Examples. J Sch Health 2006;76:80-6.
• 17. Ridley K, Dollman J, Olds T. Development and validation of a computer delivered physical activity questionnaire (CDPAQ) for children. Pediatr Exerc Sci 2001;13(1):35-46.
• 18. Bälter KA, Bälter O, Fondell E, Lagerros YT. Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiology 2005;16(4)577-9.
• 19. Welling L. PHP e MySQL desenvolvimento Web. Elsevier; 2005.
• 20. Powers S. Learning JavaScript. O'Reilly Media, Inc; 2006.
• ²¹
  Adobe Flash. Available from: <http://pt.wikipedia.org/wiki/Adobe_Flash> [2011 ago 31]
  » link
• 22. Williams HE, Lane D. Web database applications with PHP & MySQL. O'Reilly & Associates, Inc; 2004.
• 23. Martin Bland J, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;327(8476):307-10.
• 24. Woodruff SJ, Hanning RM, McGoldrick K, Brown KS. Healthy eating index-C is positively associated with family dinner frequency among students in grades 6-8 from Southern Ontario, Canada. Eur J Clin Nutr 64(5):454-60.
• 25. Moore HJ, Ells LJ, McLure SA, Crooks S, Cumbor D, Summerbell CD, Batterham AM. The development and evaluation of a novel computer program to assess previous-day dietary and physical activity behaviours in school children: the Synchronised Nutrition and Activity Program (SNAP). Br J Nutr 2008;99(6):1266-74.
• 26. Lévesque L, Cargo M, Salsberg J. Development of the Physical Activity Interactive Recall(PAIR) for Aboriginal children. Int J Behav Nutr Phys Act 2004;1(1):1-8.
• 27. McMurray RG, Harrell JS, Bradley CB, Webb JP, Goodman EM. Comparison of a computerized physical activity recall with a triaxial motion sensor in middle-school youth. Med Sci Sports Exerc 1998;30(8):130-9.
• 28. Ridley K, Olds TS, Hill A. The Multimedia activity recall for children and adolescents (MARCA): development and evaluation. Int J Behav Nutr Phys Act 2006;3(1):1-10.
• 29. Mclure SA, Reylly JJ, Crooks S, Summerbell CD. Development and evaluation of a novel computer-based tool for assessing physical activity levels in schoolchildren. Pediatr Exerc Sci 2009;21(4):506-19.
• 30. Da Costa FF, Assis MAA. Nível de atividade física e comportamentos sedentários de escolares de sete a dez anos de Florianópolis-SC. Rev Bras Ativ Fís Saúde 2011;16(1):48-54.

Agreement and reliabilty of an eletronic questionnarie for children (WEBDAFA)

Elto Legnani^I,II,III; Rosimeide Francisco Santos Legnani^II,IV; Cassiano Ricardo Rech^IV; Mauro Virgilio Gomes de Barros^V; Wagner de Campos^I,II; Maria Alice Altenburg de Assis^VI

Publication Dates

• Publication in this collection
  28 Jan 2013
• Date of issue
  Feb 2013

History

• Received
  13 May 2012
• Accepted
  20 Aug 2012
• Reviewed
  12 July 2012