Escala de Engajamento dos Alunos na Escola Inferido por Professores para o Ensino Fundamental

Elementary School Student Engagement Scale Inferred by Teachers

Escala de Compromiso de Estudiantes de primaria Inferido por Profesores

Resumo

Avaliar o engajamento escolar é crucial para identificar alunos em risco de evasão e monitorar o progresso acadêmico. O objetivo deste estudo foi construir e obter evidências de validade para uma Escala de Engajamento dos Alunos na Escola Inferido por Professores (EEAE-IP). Participaram do estudo 488 alunos do 2ª ao 5ª ano do Ensino Fundamental de escolas de Minas Gerais. A EEAE-IP apresenta boas propriedades psicométricas, com Análise Fatorial Confirmatória atestando o modelo de quatro fatores (comportamental, cognitivo, afetivo e agente). A evidência da consistência interna da escala foi boa com um Alfa de Crombach de 0,92 e a fidedignidade interobservador sugere que o construto é avaliado da mesma forma por diferentes observadores. A EEAE-IP é de rápida aplicação e utiliza o professor como informante. Sugerimos novos estudos que forneçam dados para a validade em outras populações. Este estudo reforça o papel da EEAE-IP em futuras pesquisas psicoeducacionais.

Palavras-chave:
psicologia educacional; comportamento na sala de aula; avaliação psicológica

Abstract

Evaluating school engagement is crucial to identifying students at risk of dropping out and monitoring academic progress. The aim of this study was to develop and obtain evidence of validity for a Student School Engagement Scale Inferred by Teachers (EEAE-IP). The study included 488 students from the 2nd to the 5th year of elementary school in Minas Gerais. The EEAE-IP presents good psychometric properties, with Confirmatory Factor Analysis attesting to the four-factor model (behavioral, cognitive, affective and agent). Evidence of the scale’s internal consistency was good with a Cronbach’s Alpha of 0.92 and the inter-rater reliability suggests that the construct is assessed in the same way by different observers. The EEAE-IP is quick to apply and uses the teacher as an informant. Further studies that provide data for validity in other populations were suggested. This study reinforces the role of the EEAE-IP in future psychoeducational research.

Keywords:
educational psychology; classroom behavior; psychological assessment

Resumen

Evaluar la participación escolar es crucial para identificar a los estudiantes en riesgo de abandonar la escuela y monitorear el progreso académico. El objetivo de este estudio fue construir y obtener evidencias de validez para una Escala de Compromiso Estudiantil Inferida por los Profesores (EEAE-IP). El estudio incluyó a 488 alumnos del 2º al 5º año de la enseñanza fundamental de Minas Gerais. La EEAE-IP presenta buenas propiedades psicométricas, con el Análisis Factorial Confirmatorio dando fe del modelo de cuatro factores (conductual, cognitivo, afectivo y agente). La evidencia de la consistencia interna de la escala fue buena con un Alfa de Cronbach de 0,92 y la confiabilidad entre evaluadores sugiere que el constructo es evaluado de la misma manera por diferentes observadores. La EEAE-IP es rápida de aplicar y utiliza al profesor como informante. Sugerimos más estudios que proporcionen datos para la validez en otras poblaciones. Este estudio refuerza el papel de la EEAE-IP en futuras investigaciones psicoeducativas.

Palabras clave:
psicologia educacional; conducta en la sala de clase; evaluación psicológica

School engagement is the degree to which a student is involved in school tasks and has four distinct (cognitive, behavioral, affective, and agent) but highly related dimensions (Reeve et al., 2019). Interest in school engagement has increased in recent decades thanks to its contribution to dropout prevention, its relationship to good academic performance (Archambault et al., 2019), and the association of lower levels of engagement in students who experience bullying (Valle & Williams, 2021). However, comparing the results of studies is challenging because each instrument measuring engagement uses a different theoretical model and definition of the construct (Fredricks et al., 2005, 2019).

In the literature, there are instruments that measure engagement globally, considering it with two continuums engagement/disengagement and others that adopt a two-dimensional approach, and the dimensions of engagement can be defined as cognitive and psychological (Appleton et al., 2006) or behavioral and affective (Skinner et al., 2009). In this study, engagement is considered as a four-dimensional construct, with its dimensions being cognitive, behavioral, affective, and agent (Mameli & Passini, 2019). The cognitive dimension is relative to the effort to understand complex ideas and mastery of difficult skills. The behavioral dimension is defined as participation in school activities and positive conduct in school. The affective dimension is students’ relating to their school and their positive and negative feelings toward the school context (Fredricks et al., 2005). The agent dimension is students’ proactive contribution to enrich and personalize learning (Reeve & Tseng, 2011). In this sense, it is important to note that the four-dimensional approach does not negate the three dimensions of engagement most investigated in the literature, but adds the agent dimension, making it a more comprehensive approach.

Another aspect to consider about research on school engagement is that the literature, as well as the assessment instruments found, are mostly aimed at children and adolescents aged 12 to 18 years (Appleton et al., 2006; Fredricks et al., 2005; Furrer and Skinner, 2003; Justi et al., 2021; Mameli & Passini, 2019; Veiga, 2016). Furthermore, none of these scales include the agent dimension of engagement, thus highlighting the need for instruments that encompass the agent dimension to assess school engagement in this age group. The importance of including the agent dimension in the assessment of engagement is that it brings to light students’ proactive behavior and their ability to recruit teacher support for learning (Reeve et al., 2022; Veiga, 2016). These aspects are generally not addressed in the cognitive, affective, and behavioral dimensions.

Bearing the above context in mind, the purpose of this study was to develop and obtain validity of evidence for a Student School Engagement Scale Inferred by Teachers (EEAE-IP). The justification for developing the scale is the need for an instrument that addresses all four dimensions of engagement (cognitive, behavioral, affective, and agent), given that recent research in the school setting on the construct highlights the importance of agent engagement (Reeve et al., 2022) and other scales aimed at other stages of education already use all four dimensions (Justi et al., 2021; Mameli & Passini 2019; Veiga, 2016). Another justification is that the proposal of a teacher-inferred scale seeks to minimize inconsistencies in the assessment of the construct, given that the scale is intended for children in the literacy process and that self-report measures depend on the student’s reading comprehension and the validity of responses depends on the level of that comprehension (Fredricks et al., 2019). Furthermore, from an applied point of view, assessing student engagement in school can contribute to the prevention of truancy (Archambault et al., 2019) and bullying (Valle & Williams, 2021).

The development of the EEAE-IP will be presented in four studies, as follows: (a) Study 1: construction of the scale items and evidence of content validity; (b) Study 2: evidence of construct validity through Confirmatory Factor Analysis; (c) Study 3: evidence of internal consistency and interobserver reliability and; (d) Study 4: evidence of external validity-comparison with the adaptation to Brazil of the self-report scale Student Involvement in School: A Quadridimensional Scale (EAE-E4D by Veiga (2016) adapted to Brazil by Justi et al. (2021) and between the EEAE-IP with academic performance in mathematics and Portuguese.

Study 1: Scale Item Construction and Evidence of Content Validity

This study aimed to build the items of the EEAE-IP, considering the four dimensions of engagement, and, to check whether the item wording was suitable for teachers.

Method

Participants

Seven judges participated in this study, six of whom had doctoral degrees and a master’s degree in the subject, all with conceptual experience in the area and methodological knowledge regarding the construction of psychological assessment instruments and psychometrics. Ten 2nd to 5th-grade teachers from public and private schools in a medium-sized city in the inner state of Minas Gerais also participated. Both the judges and the teachers were selected by convenience, and the teachers did not participate in the other subsequent studies.

Instruments

Twenty-four items of the Student School Engagement Scale Inferred by Teachers (EEAE-IP) were drafted based on the literature on the construct (Fredricks et al., 2005; Justi et al., 2021; Skinner et al., 2009; Veiga, 2016). These addressed aspects related to the dimensions of engagement with language suitable for teachers.

Procedures

Data collection. The wording and classification of the items within the four dimensions of engagement were evaluated by seven judges. Each of the judges was sent an e-mail invitation containing the objectives of the study, the purpose of the instrument, the need to evaluate the items, and the link to the questionnaire. The guidelines were about the description, the layout, and the possibility of indicating which dimension the item was most representative of. Thus, each item should be classified by the judges in only one dimension and, for cases in which the analysis of the item generated a dubious classification; the judge should express their opinion in the space for comments.

The version of the EEAE-IP, already evaluated by the judges, was submitted to 10 teachers from the 2nd to the 5th grades of the Elementary School to evaluate the understanding of this population. The invitation procedure for the teachers was identical to the one used for the judges. On the link, the teachers found the scale that had already been analyzed by the judges and had to classify the items as understandable or not. If they felt the need, there was a space for writing suggestions or observations.

Data analysis. The index of raw agreement was calculated to infer the agreement evidence among the judges. For the purposes of this study, the index of raw agreement is more relevant since agreement measures how often two or more judges attribute exactly the same rating to the item, or in other words, it is the degree to which two or more raters, using the same rating scale, provide equal ratings for the same item. The formula used to calculate this index was $\text{Agreement Index}=\frac{\text{Agreement}}{\text{Agreement}+\text{disagreement}}\times 100$. The concordance for each of the engagement dimensions was: 0.87 for behavioral; 0.84 for agent; 0.93 for affective; and, 0.79 for cognitive. The items received a strong agreement index except for the cognitive engagement dimension which had a moderate agreement (Landis & Koch, 1977).

Ethical Considerations

This research is in accordance with ethical principles and was submitted to and approved by the Research Ethics Committee of the Universidade Federal de Juiz de Fora (Opinion No. 2.876.895 - CAAE No. 89272418.5.0000.5147). The 2nd to 5th-grade teachers signed the Free and Informed Consent Term and the Confidentiality Agreement; the 5th-grade students signed the Free and Informed Consent Term; and their respective guardians signed the Free and Informed Consent Term.

Results

After the judges’ evaluation, four items were removed from the EEAE-IP, one item from each dimension, that had high scores on a dimension that did not match the one designated by the authors. They are: the agent engagement item “Hardly expresses his/her opinion spontaneously on a subject discussed in class” rated by three judges as behavioral and one as affective; the cognitive item “Talks with the teacher about the steps of the assignment, before presenting the final assignment” with two ratings within the expected dimension; The behavioral item “Is rude to teachers” which obtained three ratings within the affective dimension; and the affective item “Is careful with school supplies and the school’s facilities and equipment” which was rated by five judges as behavioral, one as agent, and one as affective.

When teachers evaluated the item wording, two items raised doubts: “Does not think before doing the task in class” and “Is often alone at school”. The phrasing of these items was modified for better understanding, and they were worded as follows: “When he/she has a class assignment to do, he/she does it right away, without thinking about what was asked of him/her” and “He/she is isolated at school.

Discussion

To obtain the final version of the EEAE-IP, adjustments were made due to the judges’ and teachers’ evaluations. The item “Seeks help when unable to perform an assignment” had a higher score in the agent dimension, however, it is an item of the cognitive dimension. The characterization of an agent item predicts proactivity that modifies, enriches, and personalizes learning. An item that describes the agentive dimension demonstrates a break from the linear model in which the teacher presents an activity and students perform the task (Reeve & Tseng, 2011). To give greater emphasis to the item’s cognitive dimension, it was rewritten as follows “When you cannot perform an assignment, you ask for help”. In this way, the item sought to portray effort and positive coping with failures, characteristics of cognitive engagement.

The item “Is often alone at school” when evaluated by teachers, 70% of them considered the item dubious. Based on the teachers’ comments, one reason for the misunderstanding may be the difficulty in defining what an alone student is. Due to the students’ age, the word “alone” may have given room for interpretation of being able to go to school without parents or parents never attending school. It is expected by Early Years teachers that the family is present and follows the child’s school development (Poli et al., 2020; Soares & Farias, 2019). Based on this, the final version of the item was modified to: “Stays isolated at school”.

It can be said that, from a qualitative point of view, the judges’ and teachers’ analyses indicated that, in general, the scale items are clear and consistent with the four dimensions of school engagement. The few suggestions made by both groups were taken on board and contributed to improving the understanding of some items. Finally, it should be noted that the scale showed robust rates of agreement among the judges.

Study 2: Evidence of Factor Validity Through Confirmatory Factor Analysis.

Considering engagement a multidimensional construct that contains four dimensions, a Confirmatory Factor Analysis was performed to test the four-factor model.

Method

Participants

Data from 523 students aged 7 to 12 years old from three public schools and 30 2nd to 5th-grade teachers from three public schools and five private schools in a medium-sized city in the inner state of Minas Gerais were used for the sample. The data were generated by 30 2nd to 5th-grade teachers who taught these students’ classes and evaluated their students’ school engagement using the EEAE-IP. The sample size was determined based on Field’s (2021) recommendations, which propose between 5 and 10 participants per instrument item. Sample selection was by convenience and due to incomplete responses, data from 11 students were discarded. Of the teachers who completed the scales 93.3% were women and 6.6% were men, and among the students evaluated 47.5% were girls and 52.5% were boys. Students from several years of elementary school were evaluated, 11.5% from 2nd grade, 22% from 3rd grade, 32.2% from 4th grade, and 34.3% from 5th grade. Regarding the type of school, 39.8% of the student data came from private schools and 60.2% from public schools.

Instruments

The Student School Engagement Scale Inferred by Teachers (EEAE-IP) was used. It presents 20 items on a six-point Likert scale, five of each dimension (behavioral, cognitive, affective, and agent) and five inverted (“When he/she has a class assignment to do, he/she does it right away, without thinking about what was asked of him/her”; “At school, he/she seems anxious”; “He/she disturbs classes on purpose”; “In tests and assignments, he/she answers the minimum necessary, without developing his/her answers. At school, he seems anxious”; “He disturbs classes on purpose”; “During tests and assignments, he answers the minimum required, without developing his answers”).

Procedures

Data collection. Prior contact was made with the school management to authorize the research and to agree on the distribution of the EEAE-IP to the teachers. Each teacher who agreed to participate in the research was individually instructed on how to fill out the scale. Teachers participated in the study during April and May 2019.

Data analysis. Data from the three-factor and four-factor models were analyzed in SPSS (Statistical Package for the Social Science, version 22). Confirmatory Factor Analysis (CFA) was performed using Amos (Analysis of Moment Structures, version 23) software, employing maximum likelihood estimation (MLE) and a covariance matrix between items. The fit quality of the model was assessed using the following indicators: Chi-square (χ²), non-significant Chi-square values (p ≥ 0.05) indicate that the model fits the data (Kyriazos, 2018); Root Mean Square Error of Approximation (RMSEA), values below 0.08 indicate acceptable model and values below 0.05 indicate adequate model; the Comparative Fit Index (CFI), with a minimum value of 0.90 for satisfactory fit; the Cross Validation Index (ECVI), with the model with the lowest ECVI value being considered more stable within the population studied; and, χ²/gl., being expected values below 5 for adequate adjustment (Bentler & Bonnet, 1980).

Results

Screening was performed on the data for the multivariate and univariate normality indices. Mahalanobis distance calculation was used and multivariate outliers with p < 0.001 were excluded as suggested by Kline (2015). Twenty-four cases were excluded, leaving data from 488 students that were used in the subsequent statistical analyses. Asymmetry and kurtosis for each item were calculated, with items with asymmetry and kurtosis values between -3 and 3 being considered normal (Kline, 2015). One item extrapolated the kurtosis value (4.88), but it was decided to keep that one item since its asymmetry value did not extrapolate the criterion and the sample size is quite reasonable to tolerate some deviations from normality (Kline, 2015).

Confirmatory factor analysis was conducted based on the theoretical assumptions of school engagement, with the four-factor model consisting of ‘behavioral engagement’, ‘cognitive engagement’, ‘affective engagement’, and ‘agent engagement’ (Reeve & Tseng, 2011) and a three-factor model (Fredricks et al., 2005) consisting of ‘behavioral engagement’, ‘cognitive engagement’, and ‘affective engagement’ being tested. Items that, if removed, would significantly change the fit indicators of the scale were used to choose the item with the highest weight.

Modifications were made whenever covariation was high and measurement errors referred to items of the same engagement type. The modification rates that contributed the most to model fit and met the above criteria were: items 4 and 5, and, 5 and 17 for agent engagement; items 7 and 13 for behavioral engagement; items 2 and 14, and, 14 and 20; and 2 and 20 for affective engagement. The chi-square and ECVI indexes showed more significant changes after the modification, thus the four-dimensional model showed a good fit to the data as can be seen in the indicators in Table 1.

Table 1
EEAE-IP Model Fit Index

Model	χ 2	Gl	RMSEA	CFI	ECVI	x²/gl
Four-factor model	717,268	158	0.08	0.91	1,769	4,540
Three-factor model	1038,981	16	0.10	0.86	2,417	6,453

• Note. χ2 the smaller the better; RMSEA < 0.08 acceptable; CFI >0.90 ; ECVI the smaller the better; x²/gl <5.

We highlight that all regression weights are above 0.36, indicating good construct validity, the parameters have the expected signs and the correlations between the four factors are positive. Between the factor ‘agent’ and ‘behavioral’ is found the lowest correlational value, 0.60, and between the factor ‘cognitive’ and ‘affective’ the highest with 0.96 as shown in Figure 1.

Figure 1
Illustrative diagram of the four-factor model for the EEAE-IP

A three-factor model was tested which was built by modifying the four-dimensional model. The latent factor ‘agent’ was excluded and the items grouped to the latent factor ‘cognitive’. Such a modification is justified because items reflecting proactivity and personalization of learning, characteristics of the agent dimension, are found in the cognitive dimension when the scale is three-dimensional (Fredricks et al., 2005). The three-factor model showed lower fit rates than the four-factor model, with x²/g.l showing a value above 5. The regression weights were similar to the four-factor model, ranging from 0.36 to 0.95, however, the ECVI of the three-factor model was higher, indicating that the four-factor model fits the data better.

Discussion

The four-factor model was the one that showed the best fit to the data, with the ECVI lower than the three-factor model and the CFI above 0.90. The three-factor solution did not show a good fit. The analyses confirmed four highly correlated first-order factors, as engagement theory already promulgated about the relationship of the dimensions (Reeve & Tseng, 2011; Veiga, 2016) and as per the theoretical expectations that supported the construction of the EEAE-IP.

Agent engagement was the latent factor that registered factor loadings of the highest magnitude (β > 0.80), showing that the items represent this factor well. The importance of the agent dimension of school engagement is highlighted due to the possibility of recognizing how students engage in learning activities and show initiative to personalize learning (Reeve & Tseng, 2011; Veiga, 2016). Thus, the four-factor model proved adequate to assess engagement in the population of children aged 7 to 12 years.

Study 3: Evidence for Internal Consistency and Interobserver Reliability

Once the factor structure of the EEAE-IP was known, the evidence for the internal consistency of this instrument was sought, and interobserver reliability was evaluated.

Method

Participants

Three pairs of teachers participated in the study, totaling 6 teachers from 2nd to 5th grade in public schools in a medium-sized city in the inner state of Minas Gerais. Each duo divided a class, with one teacher to teach Portuguese and another to teach mathematics.

Instruments

The Student School Engagement Scale Inferred by Teachers (EEAE-IP) described in study 2 of this paper was used.

Procedure

Data collection. The data collection was carried out together with Study 2. With the help of the school’s coordinators, a survey was made of how many classes had two teachers, and both were invited to participate in the study. All teachers who agreed to participate in the study were included. The procedures for approaching the teachers and the dates were the same as in Study 2.

Data analysis. As evidências de consistência interna da EEAE-IP foram calculadas pelo Alpha de Cronbach e com o objetivo de avaliar a fidedignidade interobservadores foi realizada a correlação de Pearson entre os escores assinalados pelos professores de matemática e português que preencheram a EEAE-IP.

Results

The evidence of internal consistency was calculated using Cronbach’s alpha coefficient for the total scale and for each dimension of the EEAE-IP. The EEAE-IP presented a total Cronbach’s alpha of 0.92, being considered a good index, as well as each dimension of the scale also presented satisfactory indices (Streiner, 2003): behavioral = 0.74; cognitive = 0.79; affective = 0.70; agent 0.93.

The correlation between the total scores of the scales completed by the math and Portuguese teachers was r = 0.66 and the correlations between the scores of the same dimensions were moderate to strong (Dancey & Reidy, 2019): r = 0.40 for cognitive, r = 0.70 for affective, r = 0.71 for agent, and, r = 0.75 for behavioral. All correlations were statistically significant.

Discussion

Regarding reliability, the EEAE-IP obtained an excellent Cronbach’s alpha for the total items (α = 0.92) and good indices (α ≥ 0.70) on all four dimensions. Interobserver reliability obtained moderate to strong indices (0.40 to 0.75), suggesting that the observations of both teachers are consistent. The strongest correlation between the scales completed by the two teachers was in the behavioral dimension. This can be explained by the ease of observing items related to this dimension, such as the items “rarely misses class” or “follows class rules”. On the other hand, the cognitive dimension of engagement showed a moderate correlation. This can be explained by the intrinsic characteristics of cognitive processes that make it difficult for teachers to observe them. The results of the evidence of reliability show that there is good internal consistency and homogeneity of the items of the EEAE-IP.

Study 4: External Validity Evidence - Relationship Between the EEAE-IP and the Self-Report Scale EAE-E4D ( Veiga, 2016 ) Adapted for Brazil ( Justi et al., 2021 ) and Between the EEAE-IP and Academic Performance in Mathematics and Portuguese

Correlation analyses were performed between Veiga’s (2016) EAE-E4D, adapted for Brazil by Justi et al. (2021), and the EEAE-IP, and, from the latter, with the 2nd bimester grades of the school year in Portuguese and mathematics of 5th-grade students.

Method

Participants

Seven 5th-grade teachers who were part of study 2 of Confirmatory Factor Analysis and 87 5th-grade students from two public schools and one private school in the city of a medium-sized city in the inner state of Minas Gerais participated in the study. The average experience of the teachers was 10.7 years, ranging from 6 to 30 years of classroom experience. Among the teachers six were female and one was male. The average age of the students was 10.9 years, ranging from 10 to 11 years, 51.7% were boys and 48.3 girls.

Instruments

Veiga’s EAE-E4D Self-Report Scale (2016) was adapted by Justi et al. (2021) for the Brazilian context in a study that assessed engagement in 5th and 6th-grade students in elementary schools in Brazil. The scale language of the original version by Veiga (2016) was adapted and an exploratory factor analysis study was conducted, in which four factors were extracted that corresponded to the four dimensions of school engagement contained in the original scale (behavioral, cognitive, agent, and affective). The reliability (Cronbach’s alpha) of the scale was 0.81 for affective engagement; 0.78 for behavioral; 0.74 for agent; and 0.62 for cognitive. Recently, the version adapted to the Brazilian context of the EAE-E4D was published in full in the paper by Justi et al. (2021).

EEAE-IP has already been described in Study 2 of this paper.

Procedures

Data collection. The adaptation for the Brazilian context of the EAE-E4D (Justi et al., 2021) and the EEAE-IP were applied to the 5th-grade students who participated in this research. The assessment was carried out collectively at the children’s school, according to the day and time scheduled with the principal, in June and July 2019. The scale was distributed to the students, who filled out the header and the items following the instructions read by the researcher. Questions could be spoken aloud at any time during the assessment. No time limit was given for completion.

Data analysis. Spearman’s Correlation Analysis was performed between the EEAE-IP scores and the Portuguese and mathematics scores for the two-month period in which the research took place and Pearson’s Correlation between the EAE-E4D and the EEAE-IP scores.

Results

The overall correlation between the EEAE-IP and the EAE-E4D was significant but weak (r = 0.32, p < 0.01). The correlations between the scores of the two scales for the behavioral dimension (r = 0.28, p < 0.01) and the affective dimension (r = 0.26, p < 0.01) were also significant. The correlations between the scores of the cognitive dimension (r = 0.17) and the agentic dimension (r = 0.13) between the two scales were not statistically significant.

The overall correlation of the EEAE-IP with Portuguese and mathematics scores was moderate and statistically significant in most dimensions, as can be seen in Table 2. The strongest correlation found was between the agency dimension and Portuguese grades. A lower, but significant correlation occurred between the affective dimension and math grades. The correlation of EAE-E4D with grades was not significant.

Table 2
Spearman’s Correlation between EEAE-IP with Portuguese and Mathematics Grades

Grades	EEA-IP
	Behavioral	Cognitive	Affective	Agent	Total
	r	r	r	r	r
Portuguese	0.31*	0.54**	0.35**	0.53**	0.55**
Mathematics	0.38**	0.38**	0.29*	0.20	0.41**

• Note. * = p < 0.05; ** = p < 0.01.

Discussion

The overall correlation between the Brazilian adaptation of the EAE-E4D (Justi et al., 2021) and the EEAE-IP was significant but weak. One explanation for the weak correlation is that the student fills out the scale looking at his or her current situation, while the teacher evaluates the student considering his or her academic trajectory. For example, a student who spent half the year skipping classes and not doing activities may, in the middle of the year, after seeing that his grades are not desirable, want to change his behavior in order to improve his grades and not repeat the year. In this case, the teacher would make an analysis based on what has already occurred and would probably see this student with low engagement. On the other hand, the student would tend to evaluate himself in a more prospective way, believing that he will now do the activities. This argument is possible in this study because both scales were filled out in June and July, in other words, in the middle of the school year.

Another possibility is the metacognition requirement that completing a self-report scale demands and the fact that, at the age surveyed in this study, students are still in the period of recognizing their own thoughts (Smortchkova & Shea, 2020). In addition, Veiga’s (2016) own Brazilian adaptation study of the EAE-E4D by Justi et al. (2021) found difficulties in measuring cognitive engagement through self-report. These results may demonstrate that teacher assessment of engagement in children aged 7-11 years may be more consistent than a self-report assessment.

Finally, the students’ concern in answering the scale in a socially desirable way, making their answers not reflect the real situation is another possibility that we cannot rule out. Social desirability, in the study by Reis and Sampaio (2019), corroborates this justification by finding that children aged 8 years to 12 years tend to perform actions in a game-based way on what would be socially desired/expected of them. With all these possibilities in mind, perhaps the best way to assess the external validity evidence of the EEAE-IP, is to assess its correlation with academic performance.

Overall, the correlation between the scores and the EEAE-IP is moderate and statistically significant. This is further indication that to assess engagement in 7-12-year-old children, an instrument that uses third-party observation may be preferable to a self-report instrument. Furthermore, the correlation between EAE-E4D and grades was not significant. Such a result was already expected, as in Price’s (2015) research, teacher reports are positively correlated with student grades and the correlation between scores on self-report engagement scales and teacher reports showed a weak correlation (Fredricks et al., 2005; Furrer & Skinner, 2003).

General Discussion

It is possible to state that the objective of developing and obtaining validity evidence for a Scale of Student School Engagement Inferred by Teachers (EEAE-IP) was achieved quite satisfactorily. After all, a strong inter-rater agreement was observed for the scale items, except for the cognitive dimension which showed a moderate agreement rate. Thus, the items reflected the aspects of the four dimensions already enacted by the existing literature of the construct (Fredricks et al., 2005; Reeve & Tseng, 2011; Skinner et al., 2009; Veiga, 2016). With regard to statistical properties, the result of the confirmatory factor analysis demonstrated that the four-factor model showed a better fit to the data than the three-factor model and that the four-factor model showed four highly correlated first-order factors. Such result is in line with Veiga’s study (2016) which tested a one-latent factor model that showed a poor fit for the data and a model of four first-order latent factors that were correlated that showed an acceptable fit. The internal consistency of the scale for all dimensions was satisfactory and the interobserver reliability obtained moderate to strong rates (0.40 to 0.75), indicating that the completion of the scale is consistent. In addition, the correlation of the scale with Portuguese and math scores used for external validity was significant and moderate, as expected based on the engagement literature (Archambault et al., 2019; Reeve & Tseng, 2011).

In summary, one of this study’s contributions is the creation of an engagement scale for the Early Years of Elementary School. Another contribution is the possibility of assessing engagement through teacher inference since children have difficulty completing self-report scales (; Smortchkova & Shea, 2020). In addition, the EEAE-IP is quickly applied and can contribute to the mapping of engagement throughout the school years, since the scale allows the assessment of children and youth from 7 to 12 years.

A limitation of this study is the use of grades from only one semester to analyze the correlation between the scale and academic performance, and the use of grades from math and Portuguese subjects only, which provides data from a portion of the year and subject. For future studies, it is suggested that grades from all semesters or the final grade be used, as well as the inclusion of more subjects. This would allow the identification of variations in engagement that occur during the school year, as well as more comprehensive information, both for the external validity of the scale and to demonstrate whether the subject the teacher teaches is a variable that influences engagement. The second limitation refers to the fact that the study performed only convergent validity, and it is suggested that future studies should also analyze discriminant validity.

Given that teachers’ knowledge of students is reduced at the beginning of the year, it is recommended that the EEAE-IP not be used in the first three months of the school year. It is recommended that in future research with the EEAE-IP, the scale be applied during different periods of the year in a longitudinal study to verify whether there is a change in teachers’ perceptions of student engagement. Finally, it is understood that a scale of teacher-inferred engagement can assist in research evaluating school engagement, and because this research is a study that provides initial psychometric data, it is emphasized that further research is needed to provide new evidence of validity in other populations.

References

• Archambault, I., Janosz, M., Goulet, M., Dupéré, V., & Gilbert-Blanchard, O. (2019). Promoting student engagement from childhood to adolescence as a way to improve positive youth development and school completion. In J. A. Fredricks, A. L. Reschly, & S. L. Christenson (Eds.), Handbook of student engagement interventions: Working with disengaged students (pp. 13-29). Academic Press.
• Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44(5), 427-445. https://doi.org/10.1016/j.jsp.2006.04.002
  » https://doi.org/10.1016/j.jsp.2006.04.002
• Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), p. 588-606. https://doi.org/10.1037/0033-2909.88.3.588
  » https://doi.org/10.1037/0033-2909.88.3.588
• Dancey, C. P., & Reidy, J. (2019). Estatística sem matemática para psicologia [Statistics without math for psychology]. (7a ed.). Artmed.
• Field, A. (2021). Descobrindo a estatística usando o SPSS [Discovering stats using SPSS]. (5a ed.). Artmed.
• Fredricks, J. A., Blumenfeld, P., Friedel, J., & Paris, A. (2005). School engagement. In K. A. Moore & L. H. Lippman (Eds.), What do children need to flourish: Conceptualizing and measuring indicators of positive development (pp. 305-321). Springer Science + Business Media.
• Fredricks, J. A., Hofkens, T. L., & Wang, M. (2019). Addressing the challenge of measuring student engagement. In K. A. Renninger & S. E. Hidi (Eds.), The Cambridge handbook of motivation and learning (pp. 689-712). Cambridge University Press.
• Furrer, C., & Skinner, E. (2003). Sense of relatedness as a factor in children’s academic engagement and performance. Journal of Educational Psychology, 95(1), 148-162. https://doi.org/10.1037/0022-0663.95.1.148
  » https://doi.org/10.1037/0022-0663.95.1.148
• Kline, R. B. (2015). Principles and practices of structural equation modelling (4th ed.). Guilford.
• Kyriazos, T. A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 9(8), 2207-2230. https://doi.org/10.4236/psych.2018.98126
  » https://doi.org/10.4236/psych.2018.98126
• Landis, R. J., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
  » https://doi.org/10.2307/2529310
• Mameli, C., & Passini, S. (2019). Development and validation of an enlarged version of the student agentic engagement scale. Journal of Psychoeducational Assessment, 37(4), 450-463. https://doi.org/10.1177/0734282918757849
  » https://doi.org/10.1177/0734282918757849
• Poli, O. L., Zago, N., & Bortoleto, E. J. (2020). Transformações sociais e interações escola-família no ensino fundamental [Social transformations and school-family interactions in fundamental education]. Reflexão e Ação, 28(1), 220-234. https://doi.org/10.17058/rea.v28i1.13161
  » https://doi.org/10.17058/rea.v28i1.13161
• Price, H. E. (2015). Principals’ social interactions with teachers: How principal-teacher social relations correlate with teachers’ perceptions of student engagement. Journal of Educational Administration, 53(1), 116-139. https://doi.org/10.1108/JEA-02-2014-0023
  » https://doi.org/10.1108/JEA-02-2014-0023
• Reeve, J., Cheon, S. H., & Jang, H. R. (2019). A teacher-focused intervention to enhance students’ classroom engagement. In J. A. Fredricks, A. L. Reschly, & S. L. Christenson (Eds.), Handbook of student engagement interventions (pp. 87-102). Academic Press.
• Reeve, J., Jang, H. R., Shin, S. H., Ahn, J. S., Matos, L., & Gargurevich, R. (2022). When students show some initiative: Two experiments on the benefits of greater agentic engagement. Learning and Instruction, 80, Article 101564. https://doi.org/10.1016/j.learninstruc.2021.101564
  » https://doi.org/10.1016/j.learninstruc.2021.101564
• Reeve, J., & Tseng, C. M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36(4), 257-267. https://doi.org/10.1016/j.cedpsych.2011.05.002
  » https://doi.org/10.1016/j.cedpsych.2011.05.002
• Reis, R. M. A., & Sampaio, L. R. (2019). Child distributive behavior and inhibitory control in a private context. Paidéia (Ribeirão Preto), 29, 1-9. https://doi.org/10.1590/1982-4327e2933
  » https://doi.org/10.1590/1982-4327e2933
• Soares, G. F., & Farias, J. S. (2019). Com quem a escola pode contar? A coprodução do Ensino fundamental público por familiares de estudantes [Who can the school count on? The co-production of public elementary education by students’ families]. Revista de Administração Pública, 53(2), 310-330. https://doi.org/10.1590/0034-761220170301
  » https://doi.org/10.1590/0034-761220170301
• Justi, F. R. R., Veiga, F. H., & Silveira, M. E. (2021). Adaptação para o Brasil da Escala Envolvimento dos alunos na escola: Uma escala quadridimensional (EAE-E4D): Versão ensino fundamental [Adaptation to Brazil of the Scale Student engagement in school: A four-dimensional scale (EAE-E4D): Elementary school version]. In J. F. Salles & A. L. Navas (Orgs.), Avaliação da linguagem oral, escrita e de habilidades relacionadas (pp. 223-230). Vetor.
• Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), p. 99-103. https://doi.org/10.1207/S15327752JPA8001_18
  » https://doi.org/10.1207/S15327752JPA8001_18
• Skinner, E. A., Kindermann, T. A., & Furrer, C. J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493-525. https://doi.org/10.1177/0013164408323233
  » https://doi.org/10.1177/0013164408323233
• Smortchkova, J., & Shea, N. (2020). Metacognitive development and conceptual change in children. Review of Philosophy and Psychology, 11(4), 745-763. https://doi.org/10.1007/s13164-020-00477-7
  » https://doi.org/10.1007/s13164-020-00477-7
• Valle, J. E., & Williams, L. C. A. (2021). Engajamento escolar: Revisão de literatura abrangendo relação professor-aluno e bullying [School engagement: A literature review covering teacher-student relationship and bullying]. Psicologia: Teoria e Pesquisa, 37,1-13. https://doi.org/10.1590/0102.3772e37310
  » https://doi.org/10.1590/0102.3772e37310
• Veiga, F. H. (2016). Assessing student engagement in school: Development and validation of a four-dimensional scale. Procedia - Social and Behavioral Sciences, 217, 813-819. https://doi.org/10.1016/j.sbspro.2016.02.153
  » https://doi.org/10.1016/j.sbspro.2016.02.153

• 1
  Article derived from the master’s thesis of the first author under the supervision of the second and third, defended in 2019, in the Graduate Program in Psychology at the Federal University of Juiz de Fora

Editado por

Associate editor: Luciana Carla dos Santos Elias

Datas de Publicação

• Publicação nesta coleção
  23 Out 2023
• Data do Fascículo
  2023

Histórico

• Recebido
  22 Mar 2022
• Revisado
  02 Set 2022
• Revisado
  14 Mar 2023
• Aceito
  18 Abr 2023