Acessibilidade / Reportar erro

Development and validation of a questionnaire on teaching and learning in remote education* * Traslated by Verónica Yañez, all responsability by authors and the tradutor.

Abstract

This article presents a questionnaire proposal to inquire about teaching and learning in remote education. The questionnaire consists of 6 dimensions, 38 indicators, and 2 open-ended questions. The dimensions pertain to instructional planning, teaching resources, assessment for learning, social interaction, and pedagogical support. The instrument was constructed through an iterative process according to criteria of coherence and relevance considering the literature on the subject. The questionnaire was validated by five expert trials with the same criteria for its construction. The instrument was applied to a non-probabilistic sample of 202 teachers at different levels of school education. The overall reliability of the questionnaire was 0.943, so it has a high consistency between the indicators and the construct. The factorial analysis performed on the questionnaire indicates that there are seven factors that explain 61.6% of the total variance. These results indicate that it is a valid and reliable tool to inquire about the teaching and learning process in remote education. It is concluded that this instrument is a proposal that collaborates with the elaboration of relevant research criteria to study this emerging issue.

Questionnaire; Teaching and learning; Pandemic; Remote education

Resumen

Este artículo presenta una propuesta de cuestionario para indagar sobre la enseñanza y aprendizaje en educación remota. El cuestionario consta de 6 dimensiones, 38 indicadores y 2 preguntas abiertas. Las dimensiones corresponden a planificación de la enseñanza, recursos didácticos, evaluación para el aprendizaje, interacción social y apoyo pedagógico. El instrumento fue construido mediante un proceso iterativo según criterios de coherencia y pertinencia considerando la bibliografía sobre la temática. El cuestionario fue validado por cinco juicios de expertos con los mismos criterios para su construcción. El instrumento se aplicó a una muestra no probabilística de 202 profesores de diferentes niveles de educación escolar. La confiabilidad general del cuestionario fue de 0,943, por lo que posee una alta consistencia entre los indicadores con el constructo. El análisis factorial realizado al cuestionario indica que existen siete factores que explican el 61,6% de la varianza total. Estos resultados evidencian que es un instrumento válido y confiable para indagar acerca del proceso de enseñanza y aprendizaje en educación remota. Se concluye que este instrumento es una propuesta que colabora con la elaboración de criterios investigativos relevantes para estudiar esta temática emergente.

Cuestionario; Enseñanza y aprendizaje; Pandemia; Educación remota

Introduction2 2 - The data set supporting the results of this study is not publicly available due to the ethics protocol and the characteristics of the information collected. Requests for access to the data can be made directly to the lead author by e-mail: david.herrera@usach.cl

As the health emergency had a negative impact on the development of face-to-face education, education systems had to adopt multiple measures to mitigate its effects on students’ educational trajectory through the implementation of remote education processes (REIMERS; SCHLEICHER, 2020REIMERS, Fernando M.; SCHLEICHER, Andreas. A framework to guide an education response to the COVID-19 pandemic of 2020. Paris: OECD, 2020.). This involved challenges and problems for educational communities on how to deal with the formative processes in the school (JELIŃSKA; PARADOWSKI, 2021). Thus, school education institutions had to carry out actions to construct pedagogical alternatives with the aim of promoting methodological, didactic, and evaluative transformations in virtual support to give continuity to the teaching and learning process (PORTILLO PENUELAS et al., 2020PORTILLO PEÑUELAS, Samuel et al. Enseñanza remota de emergencia ante la pandemia covid-19 en educación media superior y educación superior. Propósitos y Representaciones, Lima, v. 8, p. 1-17, 2020.).

This situation caused one of the worst educational crises worldwide (PORTILLO PENUELAS et al., 2020) and made it impossible to conduct face-to-face classroom classes, affecting 94% of students (GARCIA-ARETIO, 2021). Thus, the absence of face-to-face classes deepened the formative gaps (MORENO-CORREA, 2020MORENO-CORREA, Sandra Milena. La innovación educativa en los tiempos del coronavirus. Salutem Scientia Spiritus, Cali, v. 6, n. 1, p. 14-26, 2020.) and implied a change in pedagogical practice (PORTILLO PENUELAS et al., 2020) in a “fast and improvised way to a modality where there is, in general, a lack of experience and skills on the part of teachers, students, and other actors of the system” (RUZ-FUENZALIDA, 2021RUZ-FUENZALIDA, Carlos. Educación virtual y enseñanza remota de emergencia en el contexto de la educación superior técnico-profesional: posibilidades y barreras. Revista Saberes Educativos, Santiago de Chile, v. 6, p. 128-143, 2021., p. 139). However, this new educational context is also an opportunity to advance the design of innovations that take on more open, dynamic, diverse, combined and flexible approaches and pedagogical perspectives to guarantee the training of students (GARCIA-DE-PAZ; SANTANA BONILLA, 2021GARCÍA-DE-PAZ, Sergio; SANTANA BONILLA, Pablo Joel. La transición a entornos de educación virtual en un contexto de emergencia sanitaria: estudio de caso de un equipo docente en formación profesional básica. Revista de Educación a Distancia, Murcia, v. 21, n. 65, p. 1-24, 2021.).

Considering the impact of emergency remote education on education systems, it is essential for educational research to develop proposals to account for the transformations, opportunities, and challenges experienced by teaching and learning in this new context. Therefore, the construction of research tools for the production of relevant and reliable knowledge is a compelling demand for proposing reflections, discussions, and responses aimed at strengthening remote education (QIU et al., 2020; REIMERS; SCHLEICHER, 2020REIMERS, Fernando M.; SCHLEICHER, Andreas. A framework to guide an education response to the COVID-19 pandemic of 2020. Paris: OECD, 2020.).

The construction of a questionnaire to inquire about teaching and learning experiences in remote education is a proposal for an instrument that collaborates with educational research to collect valid and reliable data with the purpose of building knowledge that contributes to the educational challenges faced by different educational institutions (HIRAOKA; TOMODA, 2020HIRAOKA, Daiki; TOMODA, Akemi. Relationship between parenting stress and school closures due to the COVID-19 pandemic. Psychiatry and Clinical Neurosciences, Tokyo, v. 74, n. 9, p. 497-498, 2020.). Indeed, given the scarce evidence of how teaching and learning processes have developed (STEWART, 2021STEWART, William H. A global crash-course in teaching and learning online: a thematic review of empirical Emergency Remote Teaching (ERT) studies in higher education during Year 1 of COVID-19. Open Praxis, Oslo, v. 13, n. 1, p. 89-102, 2021.; TRUNG et al., 2020) and in view of the possibilities of continuing with this educational modality, this article aims to present the development and validation of a questionnaire to explore teaching and learning in remote education as a robust alternative for the production of empirical evidence in school contexts.

Teaching and learning in remote education: background and conceptual approaches

Remote education is a response to the contingency brought about by the current health crisis. This is conceived as a temporary and provisional migration of the teaching and learning process to virtual spaces (ABREU, 2020ABREU, José Luis. Tiempos de coronavirus: la educación en línea como respuesta a la crisis. Daena, Monterrey, v. 15, n. 1, p. 1-15, 2020.) as a way to support and give continuity to educational training. This type of teaching modality, with limited technical resources and limited design times, requires an active teacher to adapt quickly to changes (HODGES; FOWLER, 2020HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.). Teachers become emergency educational managers to adapt their classroom pedagogical practice to the virtual one without necessarily having the professional skills to implement it optimally (ABREU, 2020ABREU, José Luis. Tiempos de coronavirus: la educación en línea como respuesta a la crisis. Daena, Monterrey, v. 15, n. 1, p. 1-15, 2020.).

Remote teaching has two modalities: Synchronous and asynchronous. In the synchronous mode, students participate in a virtual class in real time. This allows teachers and students to share a temporary space in social interaction and organized according to a formal structure – with defined times – for the connection. The asynchronous modality is intended for students to interact in virtual learning platforms in an autonomous way, as they lead their own learning process according to the educational requirements that contemplate the work and/or activities of the digital modules (PICON, 2020PICÓN, María Laura. ¿Es posible la enseñanza virtual? Foro Educacional, Santiago de Chile, v. 34, p. 11-34, 2020.). Therefore, this type of virtual education is not intended to create a system with robust technical and methodological support, as it seeks to provide a rapid response and contribute to the continuity of the teaching and learning process (HODGES; FOWLER, 2020HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.).

Research in remote education has focused on three thematic areas (GAJARDO ESPINOZA; DÍEZ-GUTIÉRREZ, 2021): evaluation practices, perceptions or knowledge about Covid-19, and reflections and proposals for the management of health emergencies. Thus, in the evaluation practices and perceptions or knowledge of COVID-19, the studies are focused on experiences and proposals in school and higher education. Specifically, the research uses questionnaires and surveys without indicating the validation processes and, in turn, its results are partial according to recommendations for improving pedagogical practices (GILLES; CHARLIER, 2020GILLES, Jean-Luc; CHARLIER, Bernadette. Dispositifs d’évaluation à distance à correction automatisée versus non automatisée: analyse comparative de deux formes emblématiques. Évaluer, Paris, n. 1, p. 143-154, 2020.), offering methods and experiences for training in telematic education (GONZALES-ZAMORA et al., 2020) or how to develop the learning capacities of the different technological tools (GEWIN, 2020GEWIN, Virginia. Five tips for moving teaching online as COVID-19 takes hold. Nature, London, v. 580, n. 7802, p. 295-296, 2020.).

There are also studies that have more extensively investigated the teaching and learning process in remote education. These investigations present the design and validation of questionnaires that have aimed to characterize the processes of this modality of education through instruments of online perception, centered on technological conditions, digital competencies, teacher-student relationship, emotions, and teaching-evaluation in higher education (VILLARROEL et al., 2021). In addition to a questionnaire to evaluate the promotion of subject-focused metacognition, task, context, planning, monitoring ,and evaluation at different educational levels—school and university—(ROMO-SABUGAL; JUAREZ-HERNANDEZ; TOBON, 2021). There is also research on the assessment of satisfaction in remote education on its difficulties and challenges in higher and school education regarding the usefulness of technology and social networks in training (FANDOS-IGADO et al., 2021).

In this way, the aforementioned investigations apply reliability processes by means of internal consistency analysis of Cronbach, yielding general indices of 0.89 to 0.83 (VILLARROEL et al., 2021), 0.88 (ROMO-SABUGAL; JUAREZ-HERNANDEZ; TOBON, 2021) and 0.87 (FANDOS-IGADO et al., 2021). Most of these studies incorporate complementary statistics to reinforce the internal consistency and correlations of the items with factorial tests. Although they present important samples for the execution of statistical tests (between 346 and 2600) and high reliability indices, none is focused on the production of data in the process of teaching and learning in remote context in school education.

Teaching planning in remote education

The organization and incorporation of elements that fit remote education into planning is essential to ensure that no student is left behind (AZNAR SALA, 2020AZNAR SALA, Francisco Javier. La educación secundaria en España en medio de la crisis del covid-19. Revista Internacional de Sociología de la Educación, Barcelona, v. 9, n. 1, p. 53-78, 2020.). This has meant curricular adjustments according to the criteria of virtual times (synchronous and asynchronous), resources, methodologies, strategies, and evaluations that allow the achievement of curricular objectives prioritized to mitigate the effects of the learning gap resulting from the health emergency.

The adaptations to the teaching planning “have demanded greater dedication in the preparation of the subjects than the one destined in the face-to-face format” (PEREZ-LOPEZ; VAZQUEZ ATOCHERO; CAMBERO RIVERO, 2021, p. 343). Thus, curriculum design in remote education is constituted by the preparation and adaptation of teaching and learning according to curricular decision-making; the specification of this design as a class session or development that requires synchronous or asynchronous didactic strategies.

Methodologies in remote education

The virtual context has required a pedagogical change oriented toward the implementation of methodologies that facilitate self-learning, interdisciplinary work, collaborative learning, based on projects, problems, or other active methodologies. This new reality requires greater dynamism, flexibility, and autonomy for the development of student-centered learning (BUSTAMANTE, 2020BUSTAMANTE, Roberto. Educación en cuarentena: cuando la emergencia se vuelve permanente (segunda parte). Aportes para el Diálogo y la Acción, Lima, n. 5, p. 1-12, 2020. Disponible en: http://www.grade.org.pe/creer/archivos/articulo-5.pdf. Acceso en: 6 mayo 2022.
http://www.grade.org.pe/creer/archivos/a...
). In fact, methodologies or strategies that are specific to classroom teaching cannot be replicated (STEWART, 2021STEWART, William H. A global crash-course in teaching and learning online: a thematic review of empirical Emergency Remote Teaching (ERT) studies in higher education during Year 1 of COVID-19. Open Praxis, Oslo, v. 13, n. 1, p. 89-102, 2021.).

The challenge is to build a teaching and learning process that collaborates with student autonomy and active participation (MORGAN, 2020MORGAN, Hani. Best practices for implementing remote learning during a pandemic. The Clearing House, London, v. 93, n. 3, p. 135-141, 2020.). Consequently, it is hoped that these methodologies will allow the development of “activities that motivate analysis, critical thinking, reflection, and the collective construction of knowledge” (GALINDO et al., 2020, p. 10), so that they respond to contextual, cognitive, and emotional needs differentiated according to the virtual spaces of link between teachers, students, and among students. Likewise, this implies how synchronous and asynchronous modes allow interaction, metacognition, and academic motivation. These are fundamental and critical aspects (ZACCOLETTI et al., 2020) that have had a significant impact on students due to the absence of presence and the increased sense of social isolation.

Didactic resources in remote education

According to Arriagada Toledo (2020)ARRIAGADA TOLEDO, Patricio. Pandemia covid-19: educación a distancia o las distancias en la educación. Revista Internacional de Educación para la Justicia Social, Madrid, v. 9, n. 3, p. 1-3, 2020., teachers have had to reinvent themselves and incorporate new educational tools and materials to confront teaching and learning in the different virtual environments imposed by remote education. This leads to the need to use a range of digital teaching resources that are essential to ensure the teaching and learning process (PICÓN, 2020PICÓN, María Laura. ¿Es posible la enseñanza virtual? Foro Educacional, Santiago de Chile, v. 34, p. 11-34, 2020.).

Due to the absence of classroom classes, didactic resources or educational materials must be adapted to the new digital methods that are used in the formative processes because they have to consider the technological gaps among the students, which represents a great challenge for the teachers (MACINTYRE; GREGERSEN; MERCER, 2020). Indeed, in the face of this unknown educational reality, the demand for the preparation of teaching resources requires more time for their production and preparation (PÉREZ-LOPEZ; VAZQUEZ ATOCHERO; CAMBERO RIVERO, 2021).

Evaluation for learning in remote education

An e-assessment in virtual learning environments (BARBERA, 2016BARBERÁ, Elena. Aportaciones de la tecnología a la e-evaluación. Revista de Educación a Distancia, Murcia, v. 50, n. 4, p. 1-13, 2016.) becomes a possibility for the consolidation of an authentic assessment or for learning in remote education. This demands a formative assessment with a strong component of effective feedback (RUÍZ-PRIMO; BROOKHART, 2018), which allows the construction of learning in synchronous and asynchronous spaces. The transformation of evaluative practice for online learning requires the development of relevant assessment to enhance formative evaluative coherence (COVID-19 SOCIAL EDUCATION TABLE PROPOSALS, 2021) and, thus, avoid replicating a face-to-face evaluative practice (HODGES; FOWLER, 2020HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.).

The challenge lies in the development of a diverse, continuous, feedback-oriented formative evaluation model with active student participation (PHALARADRA; ABEYWARDENA, 2016; RAHIM, 2020RAHIM, Ahmad Fuad Abdul. Guidelines for online assessment in emergency remote teaching during the COVID-19 pandemic. Education in Medical Journal, Penang, v. 12, n. 2, p. 59-68, 2020.; YAO et al., 2020). Online evaluative practice implies a flexible attitude on the part of teachers for the implementation of evaluative strategies and instruments relevant to the remote modality.

Finally, remote education requires a redesign of the evaluation system aimed at developing authentic, contextualized, decentralized evaluation experiences that integrate learning in synchronous and asynchronous spaces, accompanied by systematic feedback practices as a way to strengthen teaching and learning processes (GARCIA-PENALVO et al., 2020; MUNOZ RIVERS; HERRERA ARAYA, 2020).

Social interaction in remote education

Social interaction as socio-affective relationships in the classroom (ESCAMILLA et al., 2020) has had a negative effect on the achievement of learning in the remote modality. In this way, the social construction of teaching-learning is fundamental for the exchange of ideas, as it enables multiple communications based on argument and reflection. These socio-educational processes require socio-affective support linked to motivation and academic achievement (ZACCOLETTI et al., 2020).

It is important to mention that this link is not only related to academics, since the closure of schools has generated a heightened sense of loss and isolation among students (MILLER, 2021MILLER, Karyn E. A light in students’ lives: K-12 teachers’ experiences (re)building caring relationships during remote learning. Online Learning, Massachusetts, v. 25, n. 1, p. 115-134, 2021.). Therefore, teachers have had to emotionally assist their students (BUSTAMANTE, 2020BUSTAMANTE, Roberto. Educación en cuarentena: cuando la emergencia se vuelve permanente (segunda parte). Aportes para el Diálogo y la Acción, Lima, n. 5, p. 1-12, 2020. Disponible en: http://www.grade.org.pe/creer/archivos/articulo-5.pdf. Acceso en: 6 mayo 2022.
http://www.grade.org.pe/creer/archivos/a...
) in order to accompany and reinforce socio-affective support and academic motivation as facilitators of learning in remote education (MILLER, 2021MILLER, Karyn E. A light in students’ lives: K-12 teachers’ experiences (re)building caring relationships during remote learning. Online Learning, Massachusetts, v. 25, n. 1, p. 115-134, 2021.). Thus, the construction of spaces in remote education based on trust is of vital importance (GALINDO et al., 2020) and as an opportunity to reassess and reinvent relational ties through a sociopolitical practice of active-authentic care (MILLER, 2021MILLER, Karyn E. A light in students’ lives: K-12 teachers’ experiences (re)building caring relationships during remote learning. Online Learning, Massachusetts, v. 25, n. 1, p. 115-134, 2021.).

Pedagogical support in remote education

Institutional pedagogical support is essential to consolidate the teaching and learning processes in remote education, both at the level of technical support and in the professional development of teachers (GARCIA ARETIO, 2021). This support not only consists of guidelines and training for the effective use of digital resources, but also involves continuous digital training processes that enable the creation of virtual learning environments (PORTILLO PENUELAS et al., 2020). Pedagogical accompaniment refers to the assistance they received for planning and coordinating teaching actions for the effective construction of virtual learning environments, together with the strengthening of their methodological and technological skills for remote education (GARCIA ARETIO, 2021).

Thus, teachers in this context of health emergency with migration to remote training are exposed to different challenges related to methodological and didactic changes, evaluation strategies, the use of technologies, and the link they establish with their students. Both for pedagogical challenges and for the development of practice in remote education, collaborative work, together with agile pedagogical-institutional support, is fundamental to face these new forms of teaching (AZNAR, 2020).

Instrument design

The purpose of the questionnaire is to collect evidence to analyze the teaching and learning processes promoted by school teachers in remote education. This questionnaire is composed of dimensions that account for the central aspects of the teaching and learning process as indicated by the specialized literature (SIMÓN; MUÑOZ-MARTÍNEZ; PORTER, 2021). These dimensions are instructional planning, methodologies, teaching resources, assessment for learning, social or pedagogical interaction, and pedagogical support.

Construction of the questionnaire

The questionnaire was constructed through an iterative process (LÓPEZ-ROLDÁN; FACHELLI, 2015LÓPEZ-ROLDÁN, Pedro; FACHELLI, Sandra. Metodología de la investigación social cuantitativa. Barcelona: Universidad Autónoma de Barcelona, 2015.) by three evaluators. This process was structured by technical processing sequences presented in detail in Table 1.

Table 1
Technical sequences for the elaboration of the questionnaire

Based on the technical sequences of the questionnaire, the instrument is composed of the dimensions that were defined according to the revised literature (GALINDO et al., 2020; HODGES; FOWLER, 2020HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.; PEREZ-LOPEZ; VAZQUEZ ATOCHERO; CAMBERO RIVERO, 2021; SIMON; MUNOZ-MARTINEZ; PORTER, 2021; YAO et al., 2020). These dimensions considered the research focuses linked to the design and validation of questionnaires in remote education (FANDOS-IGADO et al., 2021; ROMO-SABUGAL; JUAREZ-HERNANDEZ; TOBON, 2021; VILLARROEL et al., 2021), with the aim of establishing greater precision and delimitation with respect to the operationalization of the teaching-learning construction in remote education. The dimension are presented in Table 2.

Table 2
Questionnaire dimensions

According to these definitions, the six dimensions that make up the questionnaire were operationalized in different indicators (items) in the form of positive phrases. For each of the items, a Likert scale with four levels of agreement was used. These indicate are described in Table 3.

Table 3
Indicators of questionnaire dimensions

Validation

After completing the analysis of the overall and specific relevance and coherence of the questionnaire, which made it possible to develop the internal adjustment through the application of an iterative intra- and inter-item review for each dimension of the instrument, we proceeded to the process of validation by expert judgment to ensure the content validity of the questionnaire. Content validity is defined as the degree of positive relationship to measure the effectiveness of the theoretical construct according to the items that compose it (RUIZ BUENO, 2014RUIZ BUENO, Antoni. La operacionalización: de elementos teóricos al proceso de medida. Barcelona: Universidad de Barcelona, 2014. Disponible en: http://hdl.handle.net/2445/53152. Acceso en: 6 mayo 2022.
http://hdl.handle.net/2445/53152...
). This process is known as an item analysis (LOPEZ-ROLDAN; FACHELLI, 2015LÓPEZ-ROLDÁN, Pedro; FACHELLI, Sandra. Metodología de la investigación social cuantitativa. Barcelona: Universidad Autónoma de Barcelona, 2015.), which makes it possible to guarantee descriptions or inferences from the results obtained.

The questionnaire was validated by five expert trials by adapting the Escobar-Pérez and Cuervo-Martinez validation criteria (2008). The validators have graduate degrees in education with a specialization in curriculum and evaluation, work in higher education and school contexts, and have more than ten years of teaching and research experience. A coefficient of expert competence was applied to each of the experts (CABERO ALMENARA; BARROSO OSUNA, 2013CABERO ALMENARA, Julio; BARROSO OSUNA, Julio Manuel. La utilización del juicio de experto para la evaluación de TIC: el coeficiente de competencia experta. Bordón, Madrid, v. 65, n. 2, p. 25-38, 2013.), according to the assessment variables on the level of knowledge of the subject (content) and its level of argumentation. The results showed a high value of expert competence for each one (0.8-0.9).

Thus, the criteria used for the validation process were:

  1. Relevance. Analysis according to the degree of logical integration of the aspects that constitute the indicator.

  2. Consistency. Analysis of the indicator in relation to its degree of logical articulation in relation to its evaluative dimension.

  3. Evaluation range. According to four levels of validation based on the application of the relevance and consistency criteria: Does not meet criterion (0); low level of compliance with criterion (1); moderate level of compliance with criterion (2); and high level of compliance with criterion (3).

Based on these criteria for content validation or item analysis, the questionnaire experienced a minor adjustment in the wording of some indicators related to verbal and temporal formulations. These adjustments were: (a) verb tenses from present to past, (b) lexical revision to specify the element(s) that constitute the indicators, and (c) precision of elements with the use of parentheses to specify some indicators in relation to categories such as “learning,” “educational materials,” “technological tools,” and “virtual platforms.” In addition, it obtained the maximum score for each of the dimensions based on the evaluation made by the five expert judgments according to the high level of compliance with relevance and coherence, which allows guaranteeing its intra- and inter-item validity.

Reliability

The questionnaire was applied to a sample of 202 teachers with a dual purpose: to test its overall reliability and by dimensions (RUIZ BUENO, 2014RUIZ BUENO, Antoni. La operacionalización: de elementos teóricos al proceso de medida. Barcelona: Universidad de Barcelona, 2014. Disponible en: http://hdl.handle.net/2445/53152. Acceso en: 6 mayo 2022.
http://hdl.handle.net/2445/53152...
) and to collect information on the construct of teaching and learning in remote education to evidence its validity. Thus, the purpose of evaluating the reliability of the questionnaire was to:

  1. To check the reliability of the instrument (general and specific) to investigate teaching and learning in emergency remote education.

  2. Determine that dimensions and indicators (items) are consistent and relevant to research on the subject.

  3. To analyze whether the indicators (items) allow to obtain adequate information for research in the subject.

In this way, the questionnaire was analyzed internally by calculating Cronbach’s Alpha coefficients globally and for each of its component dimensions. This methodological exercise ensures the consistency of the questionnaire for the collection of empirical data (TABER, 2018TABER, Keith S. The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, Amsterdam, v. 48, n. 6, p. 1273-1296, 2018.).

The non-probabilistic sample distribution used for the application of the questionnaire is presented in Table 4.

Table 4
Sample distribution for questionnaire reliability analysis

From the application of the questionnaire, a reliability analysis was carried out with the application of SPSS software, version 22, to determine the properties of the instrument by calculating the reliability measures of scales and the intra-item correlation of the instrument in order to establish the internal consistency according to the average of the correlations between the items. From the analysis carried out, it can be indicated that the result of the intra- and inter-item consistency analysis is reliable, given that for all the indicators that constitute the dimensions of the questionnaire it is positive (MUÑIZ; FONSECA-PEDRERO, 2019MUÑIZ, José; FONSECA-PEDRERO, Eduardo. Diez pasos para la construcción de un test. Psicothema, Oviedo, v. 31, n. 1, p. 7-16, 2019.). The results of the confidence of the questionnaire are detailed in Table 5.

Table 5
Reliability according to Cronbach’s Alpha

It should be noted that the value of Alpha equal to or greater than 0.7 is considered an adequate internal consistency of the instrument GONZÁLEZ ALONSO; PAZMIÑO SANTA CRUZ, 2015). In the analysis of all the items, a 0.943 of reliability was achieved according to Cronbach’s Alpha. However, the planning dimension of teaching is the only one with a reliability range slightly below 0.7, which does not affect its consistency.

As a complement, a Kendall’s T correlational analysis (PÉREZ JUSTE et al., 2009) was applied to verify and reinforce the reliability measurements using SPSS software, version 22. The correlations by dimension are presented in Table 6.

Table 6
Kendall test correlational analysis by dimensions

The results of the Kendall correlational test are summarized below:

  1. Dimension A. Most items show statistically significant levels of association. However, the levels of association are mostly low (two moderate), which corresponds to the 0.644 value of Cronbach’s Alpha.

  2. Dimension B. Items show statistically significant levels of association, but are mostly low (two moderate). It should be noted that the values are higher than those found in dimension A, which corresponds to the value of 0.789 of Cronbach’s Alpha.

  3. Dimension C. Only one item shows no statistically significant association levels. The number of moderate associations is greater than those found in dimensions A and B, which corresponds to the 0.767 value of Cronbach’s Alpha.

  4. Dimension D. All items show statistically significant association levels, with the number of low associations being balanced with moderate associations, which corresponds to the Cronbach Alpha value 0.851.

  5. Dimension E. The items present statistically significant levels of association, being mostly moderate and in two cases high, which corresponds to the value of 0.884 of Cronbach’s Alpha.

  6. Dimension F. The levels of association of the items are statistically significant, being mostly moderate with enough values in the upper limit of the range that defines them, which corresponds to the value of 0.879 of Cronbach Alpha.

In summary, the study of Kendall’s intra-dimensional correlations reinforces the levels of reliability found with Cronbach’s Alpha.

In addition, an exploratory factorial analysis was performed using SPSS software version 22. This type of analysis allows to establish a greater precision on the latent and observed variables or constructs (LLORET-SEGURA et al., 2014). Therefore, it seeks to establish whether there are matches between the factorial analysis and the dimensions defined a priori in the study.

As the first statistical sufficiency test, it was observed that the Kaiser-Meyer-Olkin (KMO) measurement analysis and the Bartlett sphericity test showed very positive values, between 0.920 and 0.000 3 3 - The KMO value requires a minimum of 0.6 and Bartlett’s sphericity test should be less than 0.05. , so that exploratory factor analysis is feasible (Table 7).

Table 7
KMO and Bartlett tests

Considering the results of this first test, a second analysis is carried out on the basis of the following criteria:

  • Removal of main components: This is based on the fact that this is an exploratory analysis

  • Orthogonal rotation or Varimax, since the correlations between the items are of very low significance

  • Suppression of variance coefficients less than 0.3.

These results indicate that these seven factors explain 61.6% of the total variance, which allows to describe the teaching experience in the teaching process and learning in schools in remote education (Table 8). It should be noted that an appropriate model is considered when aggregations of the extracted factors exceed 50% of the variance explained (FLORA; LABRISH; CHALMERS, 2012).

Table 8
Total variance explained

The identification of factors and their influence on the total variance explained is presented in Table 9.

Table 9
Identification of factors according to explanation of variance

With respect to the dimensions defined in the original study design, there is a medium agreement with the factors found. The most relevant areas according to the factorial analysis are: The items linked to didactic resources (Dimension C), pedagogical support (Dimension F), interpersonal relations (Dimension D) and evaluative processes (Dimension E) indicated in Table 5.

Regarding the two open questions on facilitators and hinderers, these have a very high degree of validation by expert judgment (ESCOBAR-PÉREZ; CUERVO-MARTÍNEZ, 2008ESCOBAR-PÉREZ, Jazmine; CUERVO-MARTÍNEZ, Ángela. Validez de contenido y juicio de expertos: una aproximación a su utilización. Avances en Medición, Bogotá, v. 6, n. 1, p. 27-36, 2008.), since they delve into particular components of these areas associated with technological access, training in digital competencies and curricular-evaluative innovation processes, which allows us to maintain that they have consistent content validity.

Discussion of results

The results of the validation of the questionnaire show a high and significant level of internal consistency, both inter- and intra-dimensions. The reliability tests and factorial analysis carried out show that the questionnaire on teaching and learning processes in remote education is a reliable instrument for producing knowledge in educational research.

These findings are consistent with studies linked to questionnaire and test design and validation processes (RESTREPO-PALACIOS; SEGOVIA CIFUENTES, 2020). Thus, the six dimensions that constitute the questionnaire integrate two fundamental aspects: Operationalization of teaching-learning and the context of remote education. These dimensions are supported by theory (GALINDO et al., 2020GALINDO, Diana et al. Recomendaciones didácticas para adaptarse a la enseñanza remota de emergencia. Revista Digital Universitaria, México, DC, v. 21, n. 5, p. 1-13, 2020.; HODGES; FOWLER, 2020HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.; PEREZ-LOPEZ; VAZQUEZ ATOCHERO; CAMBERO RIVERO, 2021; SIMON; MUNOZ-MARTINEZ; PORTER, 2021; YAO et al., 2020YAO, Jijun et al. What role should teachers play in online teaching during the COVID-19 pandemic? Evidence from China. Science Insights Education Frontiers, Jamestown, v. 5, n. 2, p. 517-524, 2020.), thus allowing an adequate integration that collaborates with the analysis of teaching and learning in remote education in school contexts. .In this way, the questionnaire offers a structural articulation (SÁNCHEZ-MARTÍ; MORENO; ION, 2019) that makes possible the inquiry of perceptions, declared practices and technological, pedagogical and institutional conditions for the discussion and problematization of the data (GAJARDO ESPINOZA; DÍEZ-GUTIÉRREZ, 2021).

In addition, the questionnaire presents a systematic process of design and validity, which allows strengthening research in remote education with a multidimensional perspective and enriches the production of data in a broader way with the purpose of deepening partial statistical analyses (GILLES; CHARLIER, 2020GILLES, Jean-Luc; CHARLIER, Bernadette. Dispositifs d’évaluation à distance à correction automatisée versus non automatisée: analyse comparative de deux formes emblématiques. Évaluer, Paris, n. 1, p. 143-154, 2020.) in order to subject this type of instrument to a more complex discussion on the type of knowledge it produces (CAMIZÃO; CONDE; VICTOR, 2021; ROMO-SABUGAL; JUÁREZ-HERNÁNDEZ; TOBÓN, 2021).

In summary, it is possible to state the following: (a) that the overall high internal consistency of the instrument is checked according to the statistical tests performed; (b) a high consistency is determined for each of the dimensions of the instrument; (c) the indicators (items) are relevant and consistent for the collection of data for each of the dimensions that constitute the instrument; and (d) the dimensions of the questionnaire allow the description of teaching experience in remote education.

Conclusions

The questionnaire to inquire about the teaching and learning process in remote education has 6 dimensions, 38 items and 2 open questions that have a very high intra- and inter-item consistency, which allows to argue that it is a valid and reliable instrument for educational research. The theoretical construct and the factor analysis performed according to the Cronbach’s Alpha values per dimension allow testing its overall reliability, which is 0.943 and for each of the items between a range of 0.644 and 0.879. Thus, this is an instrument for the collection of data according to the assessments and perceptions that teachers have regarding the development of teaching and learning in remote education.

The limits of the questionnaire are expressed in three aspects: Sample, dimensions, and methodological adjustments. The sample used (n=202 teachers) for the application of the questionnaire needs to be expanded to consolidate the validation process. In addition, it would be appropriate to incorporate students with an adjusted questionnaire to perform comparative statistical analyzes for the purpose of developing more sophisticated explanatory designs. The dimensions have adequate internal consistency, however, they do not integrate items related to learning diversity, inclusion, and differentiated training. These are fundamental aspects that the current school system strongly incorporates in the processes of educational quality and equity. It would also be interesting to apply inferential statistics in order to establish, with greater clarity and precision, the impact of each of the dimensions of teaching and learning in this form of education.

This work offers opportunities and lines of research that are under construction. The opportunities include the relevance of this type of instruments for the production of robust and pertinent statistical information that contributes to the knowledge of remote education in schools. The data collected by means of this questionnaire become a substantial basis for knowing, discussing and reflecting on the impact of this type of education on teaching and student learning. Indeed, the results of this article open questions and lines of research related to the study of perceptions and conceptions of teachers and students about the experiences and impact of remote education; tensions, opportunities and challenges of remote education in schools in relation to diversity and inclusion; the processes of teacher-student and student-student interaction; initial and continuing teacher training in remote and/or hybrid school education; management, leadership and pedagogical support by the school for the development of remote education.

Due to the requirements and demands of research in the context of the Covid-19 pandemic, the construction of this type of instrument is a contribution not only to exploring how educational processes in remote education develop. it can also contribute to the production of empirical evidence, the discussion of data and the design of proposals aimed at improving teaching and learning in the current remote context. This is why the systematic design of the questionnaire, describing in detail the phases of elaboration, validation and reliability analysis, becomes a significant support for the production of scientific knowledge and thus respond to the new research challenges that arise from the current context of remote education.

Referencias

  • ABREU, José Luis. Tiempos de coronavirus: la educación en línea como respuesta a la crisis. Daena, Monterrey, v. 15, n. 1, p. 1-15, 2020.
  • ARRIAGADA TOLEDO, Patricio. Pandemia covid-19: educación a distancia o las distancias en la educación. Revista Internacional de Educación para la Justicia Social, Madrid, v. 9, n. 3, p. 1-3, 2020.
  • AZNAR SALA, Francisco Javier. La educación secundaria en España en medio de la crisis del covid-19. Revista Internacional de Sociología de la Educación, Barcelona, v. 9, n. 1, p. 53-78, 2020.
  • BARBERÁ, Elena. Aportaciones de la tecnología a la e-evaluación. Revista de Educación a Distancia, Murcia, v. 50, n. 4, p. 1-13, 2016.
  • BUSTAMANTE, Roberto. Educación en cuarentena: cuando la emergencia se vuelve permanente (segunda parte). Aportes para el Diálogo y la Acción, Lima, n. 5, p. 1-12, 2020. Disponible en: http://www.grade.org.pe/creer/archivos/articulo-5.pdf Acceso en: 6 mayo 2022.
    » http://www.grade.org.pe/creer/archivos/articulo-5.pdf
  • CABERO ALMENARA, Julio; BARROSO OSUNA, Julio Manuel. La utilización del juicio de experto para la evaluación de TIC: el coeficiente de competencia experta. Bordón, Madrid, v. 65, n. 2, p. 25-38, 2013.
  • CAMIZÃO, Amanda Costa; CONDE, Patricia Santos; VICTOR, Sonia Lopes. A implementação do ensino remoto na pandemia: qual o lugar da educação especial? Educação e Pesquisa, São Paulo, v. 47, p. 1-17, 2021.
  • ESCAMILLA, José et al. ¿Cómo abordar la dimensión socioafectiva en la enseñanza remota de emergencia? Revista Digital Universitaria, México, DC, v. 21, n. 5, p. 1-10, 2020.
  • ESCOBAR-PÉREZ, Jazmine; CUERVO-MARTÍNEZ, Ángela. Validez de contenido y juicio de expertos: una aproximación a su utilización. Avances en Medición, Bogotá, v. 6, n. 1, p. 27-36, 2008.
  • FANDOS-IGADO, Manuel et al. Desarrollo y validación de un instrumento para determinar la utilidad del smartphone y las redes sociales en los niveles educativos no universitarios. Revista Meta, Rio de Janeiro, v. 13, n. 41, p. 860-883, 2021.
  • FLORA, David; LABRISH, Cathy; CHALMERS, R. Philip. Old and new ideas for data screening and assumption testing for exploratory and confirmatory factor analysis. Frontiers in Psychology, Lausanne, v. 3, p. 1-21, 2012.
  • GAJARDO ESPINOZA, Katherine; DÍEZ GUTIÉRREZ, Enrique-Javier. Evaluación educativa durante la crisis por covid-19: una revisión sistemática urgente. Estudios Pedagógicos, Valdivia, v. 47, n. 2, p. 319-338, 2021.
  • GALINDO, Diana et al. Recomendaciones didácticas para adaptarse a la enseñanza remota de emergencia. Revista Digital Universitaria, México, DC, v. 21, n. 5, p. 1-13, 2020.
  • GARCÍA ARETIO, Lorenzo. Covid-19 y educación a distancia digital: preconfinamiento, confinamiento y posconfinamiento. Revista Iberoamericana de Educación a Distancia, Madrid, v. 24, n. 1, p. 9-32, 2021.
  • GARCÍA-DE-PAZ, Sergio; SANTANA BONILLA, Pablo Joel. La transición a entornos de educación virtual en un contexto de emergencia sanitaria: estudio de caso de un equipo docente en formación profesional básica. Revista de Educación a Distancia, Murcia, v. 21, n. 65, p. 1-24, 2021.
  • GARCÍA-PEÑALVO, Francisco José et al. La evaluación online en la educación superior en tiempos de la covid-19. Education in the Knowledge Society, Salamanca, v. 21, p. 1-26, 2020.
  • GEWIN, Virginia. Five tips for moving teaching online as COVID-19 takes hold. Nature, London, v. 580, n. 7802, p. 295-296, 2020.
  • GILLES, Jean-Luc; CHARLIER, Bernadette. Dispositifs d’évaluation à distance à correction automatisée versus non automatisée: analyse comparative de deux formes emblématiques. Évaluer, Paris, n. 1, p. 143-154, 2020.
  • GONZALES-ZAMORA, José A. et al. Videoconferences of infectious diseases: an educational tool that transcends borders: a useful tool also for the current COVID-19 pandemic. Le Infezioni in Medicina, Salerno, v. 28, n. 2, p. 135-138, 2020.
  • GONZÁLEZ ALONSO, Jorge; PAZMIÑO SANTACRUZ, Mauro. Cálculo e interpretación del Alfa de Cronbach para el caso de validación de la consistencia interna de un cuestionario, con dos posibles escalas tipo Likert. Revista Publicando, Quito, v. 2, n. 1, p. 62-77, 2015.
  • HIRAOKA, Daiki; TOMODA, Akemi. Relationship between parenting stress and school closures due to the COVID-19 pandemic. Psychiatry and Clinical Neurosciences, Tokyo, v. 74, n. 9, p. 497-498, 2020.
  • HODGES, Charles; FOWLER, Denver. COVID-19 crisis and faculty members in higher education: from emergency remote teaching to better teaching through reflection. International Journal of Multidisciplinary Perspectives in Higher Education, New York, v. 5, n. 1, p. 118-122, 2020.
  • JELIŃSKA, Magdalena; PARADOWSKI, Michal. Teachers’ engagement in and coping with emergency remote instruction during COVID-19-induced school closures: a multinational contextual perspective. Online Learning Journal, Massachusetts, v. 25, n. 1, p. 303-328, 2021.
  • LLORET-SEGURA, Susana et al. El análisis factorial exploratorio de los ítems: una guía práctica, revisada y actualizada. Anales de Psicología, Murcia, v. 30, n. 3, p. 1151-1169, 2014.
  • LÓPEZ-ROLDÁN, Pedro; FACHELLI, Sandra. Metodología de la investigación social cuantitativa. Barcelona: Universidad Autónoma de Barcelona, 2015.
  • MACINTYRE, Peter; GREGERSEN, Tammy; MERCER, Sarah. Language teachers’ coping strategies during the COVID-19 conversion to online teaching: correlations with stress, wellbeing and negative emotions. System, Amsterdam, v. 94, p. 1-13, 2020.
  • MILLER, Karyn E. A light in students’ lives: K-12 teachers’ experiences (re)building caring relationships during remote learning. Online Learning, Massachusetts, v. 25, n. 1, p. 115-134, 2021.
  • MORENO-CORREA, Sandra Milena. La innovación educativa en los tiempos del coronavirus. Salutem Scientia Spiritus, Cali, v. 6, n. 1, p. 14-26, 2020.
  • MORGAN, Hani. Best practices for implementing remote learning during a pandemic. The Clearing House, London, v. 93, n. 3, p. 135-141, 2020.
  • MUÑIZ, José; FONSECA-PEDRERO, Eduardo. Diez pasos para la construcción de un test. Psicothema, Oviedo, v. 31, n. 1, p. 7-16, 2019.
  • PÉREZ JUSTE, Ramón et al. Estadística aplicada a la educación. Madrid: Uned-Pearson, 2009.
  • PÉREZ-LÓPEZ, Eva; VÁZQUEZ ATOCHERO, Alfonso; CAMBERO RIVERO, Santiago. Educación a distancia en tiempos de covid-19: análisis desde la perspectiva de los estudiantes universitarios. Revista Iberoamericana de Educación a Distancia, Madrid, v. 24, n. 1, p. 331-350, 2021.
  • PHALACHANDRA, Bhandigadi; ABEYWARDENA, Isahn S. Open educational resources in the Commonwealth. Burnaby: Commonwealth of Learning, 2016.
  • PICÓN, María Laura. ¿Es posible la enseñanza virtual? Foro Educacional, Santiago de Chile, v. 34, p. 11-34, 2020.
  • PORTILLO PEÑUELAS, Samuel et al. Enseñanza remota de emergencia ante la pandemia covid-19 en educación media superior y educación superior. Propósitos y Representaciones, Lima, v. 8, p. 1-17, 2020.
  • PROPUESTAS EDUCACIÓN MESA SOCIAL COVID-19: recomendación para una evaluación pertinente en tiempos de crisis. Santiago de Chile: Universidad de Chile: Pontificia Universidad Católica de Chile, 2021.
  • QIU, Jianyin et al. A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: implications and policy recommendations. General Psychiatry, Shanghai, v. 33, n. 2, p. 1-2, 2020.
  • RAHIM, Ahmad Fuad Abdul. Guidelines for online assessment in emergency remote teaching during the COVID-19 pandemic. Education in Medical Journal, Penang, v. 12, n. 2, p. 59-68, 2020.
  • REIMERS, Fernando M.; SCHLEICHER, Andreas. A framework to guide an education response to the COVID-19 pandemic of 2020. Paris: OECD, 2020.
  • RESTREPO-PALACIO, Sonia; SEGOVIA CIFUENTES, Yasbley de María. Diseño y validación de un instrumento de evaluación de la competencia digital en educación superior. Ensaio, Rio de Janeiro, v. 28, n. 109, p. 932-961, 2020.
  • RÍOS MUÑOZ, Daniel; HERRERA ARAYA, David. La descentralización de la práctica evaluativa orientada al autoaprendizaje del estudiante. Educação e Pesquisa, São Paulo, v. 46, p. 1-16, 2020.
  • ROMO-SABUGAL, Claudia; JUÁREZ-HERNÁNDEZ, Luis Gibran; TOBÓN, Sergio. Validez de constructo de un instrumento para evaluar la promoción de la metacognición en el aula. Estudios Pedagógicos, Valdivia, v. 47, n. 3, p. 191-205, 2021.
  • RUIZ BUENO, Antoni. La operacionalización: de elementos teóricos al proceso de medida. Barcelona: Universidad de Barcelona, 2014. Disponible en: http://hdl.handle.net/2445/53152 Acceso en: 6 mayo 2022.
    » http://hdl.handle.net/2445/53152
  • RUZ-FUENZALIDA, Carlos. Educación virtual y enseñanza remota de emergencia en el contexto de la educación superior técnico-profesional: posibilidades y barreras. Revista Saberes Educativos, Santiago de Chile, v. 6, p. 128-143, 2021.
  • RUIZ-PRIMO, María Araceli; BROOKHART, Susan M. Using feedback to improve learning. New York: Routledge, 2018.
  • SÁNCHEZ-MARTÍ, Angelina; MORENO, José Luís Muñoz; ION, Georgeta. Diseño y validación de un cuestionario de percepción del aprendizaje a través del feedback entre iguales en educación superior. Revista Iberoamericana de Diagnóstico y Evaluación Psicológica, Lisboa, v. 4, n. 53, p. 113-128, 2019.
  • SIMÓN, Cecilia; MUÑOZ-MARTÍNEZ; Yolanda; PORTER, Gordon L. Classroom instruction and practices that reach all learners. Cambridge Journal of Education, Cambridge, v. 51, n. 6, p. 607-625, 2021.
  • STEWART, William H. A global crash-course in teaching and learning online: a thematic review of empirical Emergency Remote Teaching (ERT) studies in higher education during Year 1 of COVID-19. Open Praxis, Oslo, v. 13, n. 1, p. 89-102, 2021.
  • TABER, Keith S. The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, Amsterdam, v. 48, n. 6, p. 1273-1296, 2018.
  • TRUNG, Tran et al. Dataset of Vietnamese student’s learning habits during COVID-19. Data in Brief, Amsterdam, v. 30, p. 1-7, 2020.
  • VILLARROEL, Verónica et al. Educación remota en contexto de pandemia: caracterización del proceso educativo en las universidades chilenas. Formación Universitaria, La Serena, v. 14, n. 6, p. 65-76, 2021.
  • YAO, Jijun et al. What role should teachers play in online teaching during the COVID-19 pandemic? Evidence from China. Science Insights Education Frontiers, Jamestown, v. 5, n. 2, p. 517-524, 2020.
  • ZACCOLETTI, Sonia et al. Parents’ perceptions of student academic motivation during the COVID-19 lockdown: a cross-country comparison. Frontiers in Psychology, Lausanne, v. 11, p. 1-13, 2020.
  • *
    Traslated by Verónica Yañez, all responsability by authors and the tradutor.
  • 2
    - The data set supporting the results of this study is not publicly available due to the ethics protocol and the characteristics of the information collected. Requests for access to the data can be made directly to the lead author by e-mail: david.herrera@usach.cl
  • 3
    - The KMO value requires a minimum of 0.6 and Bartlett’s sphericity test should be less than 0.05.

Publication Dates

  • Publication in this collection
    09 Sept 2022
  • Date of issue
    2022

History

  • Received
    09 Sept 2021
  • Reviewed
    08 Dec 2021
  • Accepted
    08 Feb 2022
Faculdade de Educação da Universidade de São Paulo Av. da Universidade, 308 - Biblioteca, 1º andar 05508-040 - São Paulo SP Brasil, Tel./Fax.: (55 11) 30913520 - São Paulo - SP - Brazil
E-mail: revedu@usp.br