Brazilian Version of Online Self-Regulated Learning Questionnaire (OSLQ): Evidence of Validity

Abstract

Self-regulated learning refers to intentional, planned, temporal, dynamic, and complex action. Self-regulatory assessment instruments valid for traditional teaching are not suitable for online environments. This study examinates evidence for validation of the Online Self-Regulated Learning Questionnaire (OSLQ) applied to a sample of 1,434 students from an on-line Pedagogy course, 95.12% women and 4.88% men, ages from 21 to 40 years, from different regions of Brazil. AFE and AFC considering Parallel Analyzes. The results indicated a moderate adjustment for the hypothetical six-factor model. The extraction of two factors by AFE indicated that the model has not been confirmed. This result points to the need of further studies using the instrument, aiming at its availability to research.

KEYWORDS:
self-regulated learning; distance learning; evaluation tool; evidence of measure validity

Resumo

A autorregulação da aprendizagem refere-se à ação intencional, planejada, temporal, dinâmica e complexa. Instrumentos de avaliação da autorregulação válidos para o ensino tradicional não são adequados para ambientes on-line. Neste estudo, buscaram-se evidências de validade do Online Self-Regulated Learning Questionnaire (OSLQ), aplicado em amostra de 1.434 estudantes de um curso de Pedagogia on-line, sendo 95,12% mulheres e 4,88% homens, com idades entre 21 e 40 anos, de diversas regiões do Brasil. Foram realizadas AFC e AFE mediante Análises Paralelas. Os resultados indicaram ajuste moderado para o modelo hipotético de seis fatores. Foram extraídos dois fatores pela AFE, cujo modelo não foi confirmado. Aponta-se para a necessidade de realização de novos estudos utilizando o instrumento, visando a sua disponibilidade para uso em pesquisas.

Palavras-chave:
autorregulação da aprendizagem; ensino a distância; instrumento de avaliação; evidências de validade de medida

The advent of technologies aimed to information and communication brought changes to teaching, being highlighted the offer of Distance Learning (DL), an on-line process of teaching and learning that happens by means of the use of several media of social communication, with students spread and distant between themselves and also physically separated from the teacher. For Belloni (2009) and Valente (2005), DL allows great flexibility of access to courses, curriculum and innovative methodologies and, in the scene of professional qualification, represents an opportunity of updating.

For Barnard-Brak et al. (2008), one of the main characteristics of on-line learning is the experience of autonomy of the student, considering that, in this environment, teaching is not limited to a given place, time and physical material. However, in order for the autonomy propitiated to the student in the on-line environment to be enjoyed in a proper way, self-regulation becomes a critical factor for the success of learning.

Self-regulation of learning is a topic that is raising the interest of researchers from different approaches to psychology, always aiming to understand and explain this important construct of human learning, such as, for example, Bandura (2008), Pintrich (2004), Schunk (2001), and Zimmerman and Schunk (2011). Despite some points of disagreement between the various authors, there is a relative consensus that self-regulation of learning refers to the degree in which students activate their own learning process metacognitively, motivationally and behaviorally (Simão & Frison, 2013; Zimmerman & Schunk, 2011). The analysis of the proposed models, in order to explain the self-regulation process, allows inferring that the self-regulation of learning is an intentional, planned, temporal, dynamic and complex action.

Zimmerman and Schunk (2011), when considering that the human being has the innate ability to self-regulate, defined self-regulation as the control exercised by the individual over its thoughts, feelings and actions that are planned and cyclically adapted in order to obtain goals and objectives. Expressed in a different way, self-regulation is considered a cyclical and multidimensional process, inherent to all human beings, in which the student plays an active role, in a differentiated process according to the requirements of each situation (Simão & Frison, 2013).

In the current context, there has been an increase in the number of students enrolling in online courses (Broadbent & Poon, 2015), as well as an increasing amount of research focused on self-regulated learning (Araka et al, 2020).Based on a survey of studies carried out from 2004 until 2014, regarding the use of learning strategies in the online environment, Broadbent and Poon (2015) highlighted the strategies of time management, metacognition, effort regulation and critical thinking, which were positively correlated with academic results, although these effects were minor when compared to those found in traditional education. In contrast, the essay, elaboration and organization had the lower empirical support, indicating less use of those strategies by students in the online environment.

Araka et al. (2020), when seeking to identify recent advancements and trends in the area, carried out a systematic review (within the period from 2008 to 2018) regarding the techniques and tools used to assess the self-regulation of learning in online platforms. The authors showed that the same traditional methods of assessment used for the classroom environment have been used to measure the self-regulated learning in online environments, pointing out gaps existing regarding the tools currently used to measure and support the management of self-regulated learning. Also, Barnard-Brak et al. (2008) stated that most self-regulation researchers were convinced that there is a differentiated and specific process for each learning context, being necessary to create studies that investigate the self-regulation construct, both in the face-to-face context as well as in courses offered online. Under this perspective, instruments of research and assessment valid for traditional learning environments such as, for instance, the Motivated Strategies for Learning Questionnaire (MSLQ), would not be suitable to be used in online environments, due to the great differences between the two learning environments: face-to-face and distance learning. Barnard-Brak et al. (2009) argued that the main differences between the online learning environment, when compared with the face-to-face one, lie in the flexibility of time and space, in the indirect social interaction, in the range of World Wide Web (www) information sources available at moments of study and in the acquisition of dynamic learning interfaces. In this sense, it is expected that students effectively make use of different learning strategies.

Measurement of Self-Regulated Learning in On-Line Teaching Environments

The On-line Self-Regulated Learning Questionnaire (OLSQ) - was developed by Barnard-Brak et al. (2009) for the evaluation of self-regulated learning by American students in online learning environments and has been the main instrument used in research. OLSQ consists of 24 items, categorized into six dimensions of self-regulated learning: goals setting (GS), environment structuring (ES), tasks strategies (TS), time management (TM), help-seeking (HS) and self-evaluation (SE).

Since the proposition of the Questionnaire, analyzes of validity, reliability and consistency of the instrument have been performed. Barnard-Brak et al. (2009) raised evidence of construct measure validity by means of Confirmatory Factor Analysis (CFA) in two studies, with 434 students of courses in hybrid format, part of them face-to-face classes and one part online (E1), and 204 students of 100% online courses (E2). For E1, CFA results revealed a significant chi-square, indicating adequate adjustment of data to the proposed model, with a value of χ2 / df ratio of 3.08; RMSEA of .04 TLI and CFI .96. The results found for E2 were significant chi-square, with a value of χ2 /df ratio of 2.77; RMSEA as a compensation for model's complexity was .06; TLI of .93 and CFI of .95, indicating a good fit of the model. According to authors, the results indicated evidence of the validity of the instrument for assessing the construct in students enrolled, both in conventional teaching models, as well as in models based on Virtual Learning Environments (VLE). The internal consistency indexes, obtained using Cronbach’s alpha, were for ES .79; GS .53; TS .85; TM .72; HS .64; SE .59.

Barnard-Brak et al. (2010) conducted a study with 276 American university students enrolled in online courses. They were based on the analysis of the instrument previously carried out by Barnard-Brak et al. (2009), indicating only the internal consistency indexes obtained with the application of Cronbach’s alpha, for ES .92; GS .88; TS .85; TM .91; HS .92; SE .89.

Korkmaz and Kaya (2012) also aimed to determine the self-regulated learning levels of 222 students in the online environment, adapting OSLQ to the Turkish language. Confirmatory factor analysis revealed a significant chi-square with ratio value χ2 /327.28; RMSEA= .045; S-RMR= .047; GFI= .89; AGFI= .85; CFI= .99; NNFI= .99; IFI= .99. Indicators showed a good fit for the six-factor model, except the results for GFI and AGFI of acceptable fit. Additionally, the internal consistency indexes of the scale were obtained, by means of Cronbach’s alpha: ES .92; GS .95; TS .87; TM .96; HS .93; SE .94.

Tabuenca et al. (2015) used OSLQ as one of the measures used in a longitudinal study, involving 89 students from two universities in the Netherlands, enrolled in online courses. As an indicator of OSLQ validity, the reliability coefficient measured by Cronbach’s alpha used was: ES .83; GS .78; TS .41; TM .76; HS .69; SE .5.

In Brazil, Rodrigues et al. (2016) investigated evidence of OSLQ validity, for measuring self-regulation characteristics of learning, according to the guidelines of distance education in Brazil. In order to do this, they conducted a survey of 408 participants in online courses with an average age of 30 years (SD = 18.23). The authors sought to verify the adjustment of the model by means of confirmatory factorial analysis: RMSEA (.062), CFI (.91) and NFI (.89). Results indicated adjustment rates below those obtained in previous studies using the same scale, which demonstrated the relevance of new studies about scale validation including other samples, such as, research in other contexts.

Lin et al. (2017) conducted a survey at a virtual language school in the Midwestern U.S., with 466 high school students. They used a version adapted to the level of participants including 4 subscales, with a reduction of two dimensions (GS and TS) and with number of OSLQ items: ES four items, TS four items, HS three items and SE three items. Exploratory factor analysis was used, with Kaiser's eigenvalue rule greater than 1; eigenvalue map and scale consistency indexes measured by Cronbach’s alpha. The 2-factor solution, with an explanation of the 76% variance was considered, however, in the second factor. Only two items were loaded.

Martinez-Lopez et al. (2017) described the process of translating and adapting OSLQ to Russian. They also looked for evidence regarding instrument validity from data of 45 engineering students enrolled in an online course, whose results were limited by the small number of participants. The Exploratory Factor Analysis was performed from results of Kaiser Meyer Olkin test (KMO) of .644 and from Bartlett’s test of sphericity (p ¼ .000). Eigenvalue criterion above 1 was used, with Promax rotation, revealing a 4-factor structure, with 54.12% explanation of variance. Factorial loads of items for each factor were greater than .40, with the exception of items 1 (.122) and 22 (0322). Internal consistency indexes (Cronbach’s alpha) were: ES .65; GS .83; TS .65; TM .77; HS .78; SE .71.

Kiliscec and Yıldırım (2018) sought the validity and reliability of a translation into the Turkish language of OSLQ. Participants were 321 students enrolled in an online course at a public university in Adana, Turkey. The model adjustment values resulting from the Confirmatory Factor Analysis for the 6-factor model, according to the structure proposed by Barnard-Brak et al. (2009), were: χ2/df (Chi-Square/Degree of Freedom) =2.45, RMSEA=.06, RMR=.08, SRMR=.06, TLI (Tucker-Lewis Index) =.89, CFI=.90, GFI= .86, AGFI=..84 and NFI=.8. It is observed that the results for adjustment indicators GFI, AGFI, CFI, TLI and NFI were significantly below the acceptable, but for indicators RMSEA, SRMR, and RMR values were close to acceptable. The total scale consistency value, measured by alpha value, was .95.

Fung et al. (2018) sought evidence regarding the validity of a Chinese version of OSLQ applied to 412 students from the fourth to the ninth year of elementary school with an average age of 12 who participated in several online courses offered by schools in Hong Kong. The 6-factor structure was assessed by CFA, the results of which indicated: χ2 (224) = 346.642, p <.000; CFI = .977; TLI = .971; RMSEA = .036 indicating a proper adjustment of data to the model. The internal consistency indexes, measured by Cronbach’s alpha, were ES .83; GS .83; TS .82; TM .85; HS .76; SE .85.

Handoko et al. (2019) sought to find out differences in self-regulation, measured by OSLQ, among 643 students who had completed their online courses or not, offered by a public university in the United States. Only indexes of internal consistence of items (alpha of Cronbach) were searched, being ES.75; GS .84; TS .65; TM .67; HS .78; SE .75.

Taghizade et al. (2020) investigated the validity and reliability of a version of OSLQ for the Persian language in the Iranian context. Students (418) enrolled in online university courses in Tehran participated in the survey. The results of factorial analysis, by extracting the main components, confirmed the six-factor structure, which explained 56.78% of the total variance of data. Confirmatory factor analysis showed adjustment of data to model, according to results in tests χ2 / df = 1.930 <3, RMSEA = .064, GFI = .94, NFI = .92, CFI = .94. Consistency coefficients by Cronbach’s alpha ranged from .84 to .94 for each subscale.

Vilkova and Shcheglova (2020) raised the validity of OSLQ for a sample of 913 Russian university students enrolled in online courses. Eight experts in the field evaluated the items of the instrument and agreed to exclude four items, with the justification of them being vague or confusing for the Russian context, in addition to logical discrepancies. Items excluded were of GS dimension: I read in a high voice the instruction materials posted online in order to fight against distractions; as well as two items of SE: I summarize my learning in order to examine my understanding about what I learned in the distance learning material.

Confirmatory Factor Analysis was performed in two test models, the first one considering the six dimensions included in the original model (Barnard-Brak et al., 2009) and the second one with five, excluding the HS dimension because, according to the arguments of authors, it would not be compatible with the online learning situation. Results of data adjustment for the first model were: χ2 / df (164) = 120.68, p = .00; RMSEA = .08, CFI = .88 and TLI = .86. For the second five-dimensional model, the following were adjustment indexes: χ2 / df (99) = 513.09, p = .00; RMSEA = .07, CFI = .94, and TLI = .93.

Vilkova and Shcheglova (2020) reiterated the importance of investigations about self-regulated learning in completely online environments and claim that OSLQ has been the instrument used. However, dimensions of self-regulated learning are not present in this context, with a mismatch between the reality of the virtual environment and the learning skills evaluated by the questionnaire. For example, Baker et al. (2018) found that only 7% of students participating in their research received feedback from their instructors in the online environment. The level and quality of student participation in the platforms were low, with 90% of review activities about the same information presented (Breslow et al., 2013); 94% of students did not participate in online discussions (Qiu et al., 2016) and only students’ interaction with content was emphasized, despite the importance of relationships with instructors. Regarding the validity of OSLQ, Vilkova and Shcheglova (2020) emphasize the need of studies with samples of at least 540 participants, given the dimensions evaluated by the instrument, and because there is no support in literature for the construction of the instruments and criterion validity.

The majority of OSLQ validity studies used CFA and the internal consistency coefficient measured by Cronbach’s alpha. Only two studies raised the factorial structure of the instrument items by EFA: Lin et al. (2017) with a version of four dimensions and exclusion of items prepared before the analysis with the justification of adapting to the education level of participants; and Martinez-Lopez et al. (2017) with a sample of only 45 students. Table 1 shows the summary of results of EFAs executed. It is worth mentioning that the small sample size and the modification of the instrument for EFA made it difficult to compare the results and are points to be considered during the analysis of the few studies available in literature.

Table 1
OSLQ Validity Studies

Article	Participants	Construct Validity
Barnard-Brak et al. (2009)	Universitários Americanos	CFA
	434 cursos híbridos
	204 totalmente on-line
Rodriques et al. (2016)	Universitários brasileiros	CFA
	408 totalmente on-line
Martinez-Lopez et al. (2017)	Universitários russos	CFA
	45 totalmente on-line
Lin et al.(2017)	Ensino médio	EFA
	466 (cursos de curta duração on-line)
Fung et al. (2018)	Ensino elementar	CFA
	(cursos de curta duração on-line)
Taghizade et al. (2020)	Universitários Iran	Principal Component Factoring e CFA
	418 (cursos de curta duração on-line)
Vilkova e Shcheglova (2020)	Universitários russos	CFA
	913 (cursos de curta duração on-line)

• Note. CFA, Confirmatory Factor Analysis; EFA, Exploratory Factor Analysis.

Taghizade et al. (2020) made a Principal Components Analysis (PCA), discovering 6 factors. EFA and PCA are techniques for reducing variables or components and used for factorial structure analysis. However, there is an important difference between the two techniques (Costello & Osborne, 2005; Damásio, 2012). The two analyzes assume that the variance of each variable is composed by the specific variance (of the variable itself), common variance (shared by all items of the factor or component) and error variance (a part of the item not explained by the component or factor). PCA considers specific and common variance resulting in factorial loads and inflated communalities. Regarding EFA, whose objective is to discover the latent construct structure explaining the variance of items, only the portion of shared variance (common) of the items is considered, being considered, for the extraction of factors, the methods of Maximum Likelihood for samples with normal distribution, and Main Factorial Axis (Principal Axis Factoring) for non-normal distributions (Damásio, 2012).

In the present study, we searched for the evidence of validating a Portuguese version for OSLQ, with a sample of 1,434 participants, enrolled in a 100% online course, using the statistical techniques recommended by scholars of Psychometry today, for instance Damásio (2012), Lara (2014), Lorenzo-Seva and Ferrando (2019).

Method

Participants

One thousand four hundred thirty-four students from a Pedagogy course in distance learning answered to the OSLQ, from a universe of 89,000 students enrolled. The type of sampling was non-probabilistic by adherence. They were 1,364 (95.12%) females and 70 (4.88%) males. The average age of participants was 31.2 years (SD = 8.75), with a range from 17 to 63 years. The greatest concentration was in the 21 to 30 age group, accounting for 4.9% of the sample, followed by the 31 to 40 age group, with 33.75%. As for the distribution of participants by Brazilian regions, 474 participants (33.05%) were from the Southeast, 291 (2.29%) from the South, 267 (18.62%) from the Northeast, 208 (14.5%) from the Central West and 194 (13.52%) from the Northern region. Most respondents, 27.96% (401 students), were in the fourth semester, while the lowest concentration was in the seventh semester, with 3.28% of the sample (47 students).

Instrument

With the authors’ consent, the Self-Regulated Learning Questionnaire (Barnard-Brak et al., 2009) was translated to Portuguese by a professional translator fluent in both Portuguese and English languages. Following guidelines from Borsa et al. (2012), linguistic, cultural, contextual and scientific considerations about the evaluated construct were considered. The Portuguese version of the instrument was applied to a sample of 1,434 Brazilian students.

The instrument was assessed regarding the relevance of language, content and structure by six experts in construction, standardization, validation and review of psychological assessment instruments. Four returned with suggestions that were analyzed and adopted. Subsequently, the preliminary version was sent to five Distance Learning students, to evaluate the adequacy of the language. Participants did not indicate the need for any adjustments.

As in the original version of Barnard-Brak et al. (2009), the questionnaire was composed by 24 statements, elaborated about six constructs linked to the self-regulation of learning: setting goals, structuring the environment, strategies for performing tasks, time management, seeking help and self-assessment. According to the instrument, participants are instructed to indicate the option that best represents how they perceive themselves regarding their course in Distance Learning, on a 5-point Likert scale (5 totally agree with to 1 strongly disagree). All items on the scale are positive and in accordance with Barnard-Brak et al. (2009), values between 1 and 2 indicate low frequency of self-regulation, between 2.1 and 3.9, moderate self-regulation and between 4 and 5 high frequency of self-regulation.

Procedures

The research followed the principles present in Resolution 466/12 of the National Health Council and was submitted for appreciation and approval of the Ethics Committee in Research involving Human Beings, under the number 1,597,451.

The final version of the questionnaire was made available online, using the resource Google Forms (application for creation and management of Google forms), and presented on the online platform of the course to which the student was enrolled. A link to access the instrument was sent to the students’ e-mail, being available for a period of 30 days. After accessing it, the student was invited to participate in the research and, only after agreeing with the Free and Informed Consent Form (ICF), the questionnaire became accessible. Data collected were coded and input in software SPSS statistics, version 21 and transported to software Factor Analysis, free and available in (http://psico.fcep.urv.es/utilitats/factor/Download.html).

In Study 1, data from the total sample of 1,434 participants were submitted to a confirmatory study of factorial structure indicated by Barnard-Brak et al. (2009) and to the survey of internal consistency of the items of each subscale, measured by Cronbach’s alpha, using the IBM SPSS Statistic 21 program. In Study 2, data were transferred to the software Factor Analysis in order to perform the Exploratory Factor Analysis (EFA), using the method of Parallel Analysis, and using Robust Unweighted Least Squares (RULS), with Robust Oblimin rotation (Lorenzo-Seva & Ferrando, 2019).

Results and Discussion

Study 1

We sought to verify the adjustment of the hypothetical 6-factor model to data of the sample with 1,434 participants. For the construction of the model to be confirmed, measurement errors of the 24 items and specifications between the five constructs or latent variables were considered. In order to calculate the fit of data for the hypothetical model, it was assumed that latent variables were correlated with each other. Chi-square and Root Mean Square (RMR) statistics were used as absolute models for adjusting indexes. The value of Chi-square test for adjusting the model was significant (χ² = 1379.368, p <.000), indicating the existence of statistically significant differences between the matrix predicted by the hypothetical model and the analyzed data matrix.

This may be justified due to the size of the sample, and due to the sensitivity of Chi-square to this variable. The ratio between chi-squared and degrees of freedom χ²=1379.368/df 237= 5.820 shows adjustment under the reasonable, according to Carmines and McIver (1981). RMR was .063, indicating a proper adjustment. Simultaneously, other measures of adjusting the model were used: CFI (.906), a value below .95, described by literature as the minimum indicator for a good adjustment (Hu & Bentler, 1998), GFI (.915) and AGFI (.892), which compares the residues of data arrays observed and estimated, revealed a bad adjustment of the model, since values were near and smaller than .9. RMSEA was .058 indicating adequate adjustment of the model, since values below .05 reflect a good fit and results between .05 to .08 reflect reasonable errors, considered to be adequate ones. Generally, the model revealed a proper adjustment, similar to those obtained in previously carried researches using the same instrument. The observation of high residues possibly reveals the presence of redundant items. This fact, that would increase the probability of being correlated with error-variances of some items, led to the attempts of new specifications and analysis of items with the insertion of covariances between measurement errors for each item (Figure 1).

Figure 1.
OSLQ Diagram with insertion of co-variances between item measurement errors

As it can be seen in Figure 1, items comprising the subscales for the evaluation of self-assessment strategies and searching for help had correlated error variances in most of their items. The new specification, with the insertion of covariance, improved the adjustment indexes of the Model. The Ratio between chi-square and degrees of freedom was χ² = 741.294 / df 225 = 3.294; the RMR .045 and RMSEA .040 indexes were also reduced as well as the GFI .958; the AGFI .943 and the CFI .957 improved. The present CFA differs from those carried out by surveys indicated in Table 1 due to sample size, as those present 100% students from online courses and use all OSLQ items translated and adapted to Portuguese. Results obtained, however, were aligned with the others, indicating a reasonable adjustment of the six-dimensional model.

Study 2 - EFA by Maximum Likelihood and Oblimin Rotation

The execution of EFA, with the Portuguese version of OSLQ, was made due to lack of analysis in literature, in other words, evaluating the complete instrument, as in the original version, and with a large size of the sample. According to Damásio (2012), although there is controversy regarding the number of observations per item of the instrument for evaluation, if the measure has a reduced number of items per factor and presents low loads factors and commonalities, there is a high probability that unstable factor solutions will be found with a small sample.

The same data were then submitted to analysis in program Factor 1.3.01 (Lorenzo-Seva & Ferrando, 2015), which provides techniques considered more accurate for studies seeking evidence of instrument validity. AFE was executed using the Unweighted Least Squares (ULS) method and Direct Oblimin Rotation. Parallel Analysis based on Minimum Rank Factor Analysis was used in order to determine the number of factors to be extracted. The distribution of OSLQ scores revealed asymmetries from -1.588 to -.586 and kurtosis from -.547 to 4.704, with extreme values being identified. In this case, when the univariate distributions of ordinal items are asymmetrical or have an excess of kurtosis, the polychoric correlation matrix (Lara, 2014) is advised.

The polychoric correlation matrix of items is shown in Table 2. KMO test provided a value of .92679 (considered very good) and Bartlett Sphericity Test had values, χ² 16384.8 (df = 276; p = .000010), both indicating the adequacy of the data for factorial analysis. Five factors were extracted with eigenvalues > 1.0, with an indication for retention of two factors (Table 3). Those results did not align with those obtained by Taghizade et al. (2020), who found the 6-factor solution to be coherent between OSLQ items and predicted dimensions; however, as already pointed out, the authors used ACP that, although being widely used, is not considered genuine AFE (Damásio, 2012, Hernandes et al., 2017).

Table 2
Matrix of Polychoric Correlations of OSLQ Items

Item	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
1	1.0
2	.66	1.0
3	.56	.51	1.0
4	.61	.60	.60	1.0
5	.31	.28	.30	.34	1.0
6	.42	.37	.33	.44	.29	1.0
7	.44	.38	.39	.49	.27	.77	1.0
8	.50	.48	.49	.53	.32	.68	.76	1.0
9	.45	.43	.45	.50	.24	.70	.69	.72	1.0
10	.28	.27	.26	.28	.13	.24	.25	.31	.30	1.0
11	.11	.14	.15	.13	.04	.15	.14	.16	.22	.36	1.0
12	.34	.31	.40	.39	.15	.27	.28	.34	.35	.27	.31	1.0
13	.35	.34	.38	.38	.23	.26	.30	.37	.33	.28	.26	.50	1.0
14	.42	.38	.35	.41	.23	.36	.36	.40	.40	.35	.27	.39	.44	1.0
15	.32	.29	.33	.38	.10	.26	.34	.29	.31	.21	.21	.39	.39	.45	1.0
16	.40	.35	.42	.45	.07	.25	.35	.35	.36	.25	.16	.45	.38	.45	.61	1.0
17	.27	.22	.29	.27	.16	.21	.23	.28	.25	.36	.22	.29	.27	.30	.22	.26	1.0
18	.23	.18	.26	.24	.12	.12	.20	.21	.18	.31	.20	.30	.31	.27	.30	.28	.49	1.0
19	.23	.20	.23	.26	.13	.21	.24	.27	.24	.26	.20	.29	.28	.23	.26	.27	.49	.43	1.0
20	.31	.25	.34	.29	.13	.16	.25	.31	.22	.21	.18	.33	.36	.28	.32	.34	.33	.38	.35	1.0
21	.41	.39	.44	.42	.17	.29	.33	.44	.38	.34	.18	.38	.48	.39	.39	.43	.36	.34	.36	.40	1.0
22	.36	.37	.38	.41	.17	.30	.35	.41	.38	.30	.25	.40	.48	.38	.38	.38	.34	.27	.36	.38	.64	1.0
23	.22	.22	.23	.27	.14	.22	.26	.31	.27	.26	.23	.32	.39	.32	.31	.31	.37	.42	.50	.38	.40	.46	1.0
24	0,26	.23	.27	.27	.13	.22	.26	.28	.30	.27	.24	.31	.41	.31	.34	.31	.42	.47	.51	.37	.42	.48	.79	1.0

Table 3
Parallel Analysis based on Minimum Rank Factor Analysis

Variables	% of variance
	Eigenvalue	Proportion of Variance	Proportion of accumulated Variance
1	4.2697*	8.2829	9.1883
2	1.3962*	7.8378	8.6997
3	1.38458	.05769	.52532
4	1.17510	.04896	.57429
5	1.07429	.04476	.61905

• Note. * Retention of 2 factors with a proportion of variance greater than the accumulated variance

According to Table 4, in Factor 1, 10 items were loaded with a factorial load ranging from .42 to .93. The five items designed to assess the strategies for Goals Setting (GS): Item 1: I set standards for my tasks in the distance learning course, Item 2: I set short-term goals (daily or weekly) as well as long-term goals (monthly or half-yearly), Item 3: I keep a high level for my distance learning course, Item 4: I set goals to help manage my time of study in the distance learning course, and Item 5: I do not compromise the quality of my work because it was made as a distance one. The four items regarding Environment Structuring (ES): Item 6: I choose the place where I study in order to avoid too much distraction, Item 7: I find a comfortable place to study, where I can study more efficiently for the distance course, Item 8: I know where I can study most efficiently for online courses, Item 9: I choose a time with few distractions in order to study for the distance course. A single item about time management strategy (TS): Item 14: I allocate extra study time for my online courses because I know that they take time.

Table 4
EFA factorial loads with Direct Oblimim Rotation of items OSLQ and Communalities

Item	Fator 1	Fator 2	h 2 *
1. I set standards for my assignments in online courses	.73		.73
2. I set short-term (daily or weekly) goals as well as long-term goals (monthly or for the semester)	.70		.70
3. I keep a high standard for my learning in my online courses	.62		.66
4. I set goals to help manage study time for my online courses	.74		.69
5. I don't compromise the quality of my work because it is online	.42		.34
6. I choose the location where I study to avoid too much distraction.	.93		.89
7. I find a comfortable place to study	.92		.85
8. I know where I can study most efficiently for online courses	.93		.88
9. I choose a time with few distractions for studying for my online courses	.87		.78
14. I allocate extra studying time for my online courses because I know it is time-demanding	.44		.56
1. I try to take more thorough notes for my online courses because notes are even more important for learning online than in a regular classroom		.31	.48
11. I read aloud instructional materials posted online to fight against distractions		.35	.50
13. I work extra problems in my online courses in addition to the assigned ones to master the course content		.42	.60
17. I find someone who is knowledgeable in course content so that I can consult with him or her when I need help		.55	.61
18. I share my problems with my classmates online, so we know what we are struggling with and how to solve our problems		.66	.63
19. If needed, I try to meet my classmates face-to-face		.61	.56
2. I am persistent in getting help from the instructor through e-mail		.47	.48
23. I communicate with my classmates to find out how I am doing in my online classes		.77	.79
24. I communicate with my classmates to find out what I am learning that is different from what they are learning		.82	.88
12. I prepare my questions before joining in discussion forum	.31	.31	.67
15. I try to schedule the same time every day or every week to study for my online courses, and I observe the schedule	.30	.35	.65
16. Although we don't have to attend daily classes, I still try to distribute my studying time evenly across days	.38	.31	.69
21. I summarize my learning in online courses to examine my understanding of what I have learned	.34	.44	.73
22. I ask myself a lot of questions about the course material when studying for an online course	.30	.47	.72
Variance	6.49	4.18
Proportion of variance (%)	.40	.25
Reliability	.97	.94
Correlation between factors	.57

• Note.^*Communalities

Nine items were loaded regarding Factor 2, with factorial loads from .31 to .82, three items regarding the assessment of strategies for performing tasks (TS): Item 10: I try to take more complete notes in the distance course because notes are even more important for online learning than for normal classroom learning, Item 11: I read the instructional materials posted online to fight against distractions, Item 13: I work on extra problems in the distance course, in addition to those indicated, to master the course content. Three items about looking for help (HS): Item 18: I share my problems with colleagues online so that we know what brings difficulties to us and how to solve our problems, Item 19: If necessary, do I try to meet my colleagues in person, and Item 20: I am able to get help from the tutor using email. And two items regarding self-evaluation (SE): Item 23: I communicate with my colleagues to find out how I am doing in my distance classes, and Item 24: I communicate with my colleagues to see if what I am learning is different from what they are learning. The remaining five items of the instrument carried, on both factors, a factor load between .30 and 044. Bentler's Simplicity Index (1977) was .78 (100 percentile) and the Load Simplicity Index was .50 (100th percentile). These values indicated that each item represents, predominantly, a single dimension and the global solution had a high degree of simplicity. The Root Mean Square of Residuals (RMSR) was .0254, while the expected average value for an acceptable model was until .0264.

The extraction of two factors by the analysis performed did not corroborate the six dimensions proposed underlying the items of the instrument. The grouping of items around the two factors did not include the logic of their elaboration either, since items of different dimensions were allocated around the same factor. We then sought to confirm the two-factor model with the 19 items, excluding those that loaded both factors (Figure 2). The ratio between chi-square and degrees of freedom χ2 (164) = 120.68 / 164 = 7.321, p = .00 revealed an adjustment below the reasonable, according to Carmines and McIver (1981). The CFA adequacy indexes for the two-factor model were poor; RMR was .192, CFI .78, GFI .84, AGFI .806 and RMSEA .09.

Figure 2.
2-factor model OSLQ diagram

Final Considerations

Distance education is a reality in Brazil and studies seeking to understand the requirements for the successful learning of students in the online environment are incipient. Self-regulation of learning is revealed an important factor, in view of the implicit autonomy for this learning environment. Literature points out that instruments used to assess the self-regulation of learning in face-to-face contexts are not suitable for online learning, considering the unfolding of autonomy offered and required by students, such as structuring the study environment, flexibility of time, seeking help, self-assessment, and others. The present study sought evidence regarding the validity of a Brazilian version of OSLQ, an instrument designed to assess six dimensions of the self-regulation of learning in online environments. The original model, proposed by the authors, revealed an adjustment in the limit of the criteria considered for its adequacy, comparable to results found in other studies available in literature. EFA indicated a new hypothetical two-factor model, grouping items designed for the three-dimensional assessment (F1: Environment Structuring, Tasks Strategies and Time Management and, for Factor 2, items designed to four-dimensional evaluation, Task Strategies, Help-Seeking and Self-Evaluation.

Since its proposal by Barnard-Brak et al. (2009) the validity studies of the instrument were made with reduced and modified versions and/or with small samples, not much indicated for the case of an instrument with factors and items not yet confirmed by psychometric analyzes. The method of Parallel Analyzes, according to Damásio (2012), is consolidated in international research and, although little used in Brazil, allows more accurate results for data extraction and analysis, essential for studies using questionnaires with statements in Likert-type scales. In the present study, in addition to the CFA of the original 6-item model, using the classic analyzes available in the most well-known statistical packages such as, for example, SPSS, STATA, STATISTICA, an EFA was performed using the most current and recommended techniques, available in the Factor software.

According to Cronbach (1996), the validity of an assessment measure and the underlying construct are inseparable. After preparing the measure, the subsequent process is the search for evidence that demonstrates its suitability, with a possible revision when faulty or inadequate aspects are recognized. In these cases, the self-criticism and criticism of those who argue for other interpretations play an important role in the complex task of validation. Also according to the author, when a measure is developed to evaluate a well-accepted construct, it is at greater risks for its adequacy than the construct, however, the evidence that emerges from the set of analyzes made may also guide the review of the construct to which it was planned.

The availability of instruments for assessing the self-regulation of learning in online environments is necessary for the knowledge and understanding of strategies used by students in this type of teaching. Although studies with the OSLQ made with samples from foreign students have obtained satisfactory results about its adequacy, the characteristics of the studies made did not support the proposed structure. The need of new studies is indicated regarding the properties of the instrument and, probably, the elaboration of new evaluation items if the same subjacent constructs remain as they are.

References

• Araka, E., Maina, E., Gitonga, R., & Oboko, R. (2020). Research Trends in Measurement and Intervention Tools for Self-Regulated Learning for E-Learning Environments-Systematic Review (2008-2018). Research and Practice in Technology Enhanced Learning, 15(6), 1-21. https://doi.org/10.1186/s41039-020-00129-5
  » https://doi.org/10.1186/s41039-020-00129-5
• Baker, R., Dee, T., Evans, B., & John, J. (2018). Bias in Online Classes: Evidence from a Field Experiment. CEPA Working Paper No. 18-03. Stanford Center for Education Policy Analysis
• Bandura, A. (2008). A Teoria Social Cognitiva na Perspectiva da Agência [The Social Cognitive Perspective in the Agentic Perspective] . In A. Bandura, R. G. Azzi & S. Polydoro (Orgs.), Teoria Social Cognitiva: Conceitos Básicos (pp.69-95). Artmed.
• Broadbent, J., & Poon, W. L. (2015). Self-Regulated Learning Strategies & Academic Achievement in Online Higher Education Learning Environments: A systematic review. Internet and Higher Education, 27, 1-13. https://doi.org/10.1016/j.iheduc.2015.04.007
  » https://doi.org/10.1016/j.iheduc.2015.04.007
• Barnard-Brak, L., Paton, V., & Lan, W. (2008). Online Self-Regulatory Learning Behaviors as a Mediator in the Relationship between Online Course Perceptions with Achievement. The International Review of Research in Open and Distributed Learning, 9(2), 1-7. https://doi.org/10.19173/irrodl.v9i2.516
  » https://doi.org/10.19173/irrodl.v9i2.516
• Barnard-Brak, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. L. (2009). Measuring Self-Regulation in Online and Blended Learning Environments. The Internet and Higher Education, 12(1), 1-6. https://doi.org/10.1016/j.iheduc.2008.10.005
  » https://doi.org/10.1016/j.iheduc.2008.10.005
• Barnard-Brak, L., Lan, W. Y., & Paton, V. O. (2010). Profiles in Self-Regulated Learning in the Online Learning Environment. International Review of Research in Open and Distance Learning, 11(1), 149-56. https://doi.org/10.19173/irrodl.v11i1.769
  » https://doi.org/10.19173/irrodl.v11i1.769
• Belloni, M. L. (2009). Educação à Distância 5. ed. Autores Associados.
• Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying Learning in the Worldwide Classroom Research into edX's First MOOC. Research & Practice in Assessment, 8, 13-25.
• Borsa, J. C., Damásio, B. F. & Bandeira, D. R. (2012). Adaptação e Validação de Instrumentos Psicológicos entre Culturas: Algumas Considerações. Paidéia, 22(53), 423-432. https://doi.org/10.1590/S0103-863X2012000300014
  » https://doi.org/10.1590/S0103-863X2012000300014
• Carmines, E. G., & McIver, J. P. (1981). Analyzing Models with Unobserved Variables: Analysis of Covariance Structures. In G. W. Bohrnstedt, & E. F. Borgatta (Eds.), Social Measurement: Current Issues (pp. 65-115). Sage Publications, Inc.
• Costello, A. B., & Osborne, J. (2005). Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis. Practical Assessment, Research, and Evaluation, 10(7). https://doi.org/10.7275/jyj1-4868
  » https://doi.org/10.7275/jyj1-4868
• Cronbach, L. J. (1996). Fundamentos da Testagem Psicológica [Fundamentals of Psychological Testing]. 5. ed. Artmed.
• Damasio, B. F. (2012). Uso da Análise Fatorial Exploratória em Psicologia [Uses of Exploratory Factorial Analysis in Psychology]. Avaliação Psicológica, 11(2), 213-228.
• Lara, A. S. D. (2014). ¿ Matrices Policóricas/Tetracóricas o Matrices Pearson? Un Estúdio Metodológico [Polychoric/Tetrachoric Matrix or Pearson Matrix? A Methodological Study]. Revista Argentina de Ciencias del Comportamiento, 6(1), 39-48.
• Fung, J. J., Yuen, M., & Yuen, A. H. (2018). Validity Evidence for a Chinese Version of the Online Self-Regulated Learning Questionnaire with Average Students and Mathematically Talented Students. Measurement and Evaluation in Counseling and Development, 51(2), 111-124. https://doi.org/10.1080/07481756.2017.1358056
  » https://doi.org/10.1080/07481756.2017.1358056
• Gameel, B. G. (2017). Learner Satisfaction with Massive Open Online Courses. American Journal of Distance Education, 31(2), 98-111. https://doi.org/1.1080/08923647.2017.1300462
  » https://doi.org/1.1080/08923647.2017.1300462
• Handoko, E., Gronseth, S. L., McNeil, S. G., Bonk, C. J., & Robin, B. R. (2019). Goal setting and MOOC completion: A study on the role of self-regulated learning in student performance in massive open online courses. The International Review of Research in Open and Distance Learning, 20(3), 39-58. https://doi.org/10.19173/irrodl.v20i4.4270
  » https://doi.org/10.19173/irrodl.v20i4.4270
• Hu, L., & Bentler, P. M. (1998). Fit Indices in Covariance Structure Modeling: Sensitivity to Under Parameterized Model Misspecification. Psychological Methods, 3, 424-453. https://doi.org/10.1037/1082-989X.3.4.424
  » https://doi.org/10.1037/1082-989X.3.4.424
• Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-Regulated Learning Strategies Predict Learner Behavior and Goal Attainment in Massive Open Online Courses. Computers & Education, 104, 18-33. https://doi.org/10.1016/j.compedu.2016.10.001
  » https://doi.org/10.1016/j.compedu.2016.10.001
• Korkmaz, O., & Kaya, S. (2012). Adapting Online Self-Regulated Learning Scale into Turkish. Turkish Online Journal of Distance Education-TOJDE, 13(1), 1302-8.
• Lin, C-H., Zhang, Y., & Zheng, B. (2017). The Roles of Learning Strategies and Motivation in Online Language Learning: A Structural Equation Modeling Analysis. Computers & Education, 113, 75-85. https://doi.org/10.1016/j.compedu.2017.05.014
  » https://doi.org/10.1016/j.compedu.2017.05.014
• Lorenzo-Seva, U., & Ferrando, P. J. (2019). A General Approach for Fitting Pure Exploratory Bifactor Models. Multivariate Behavioral Research, 54(1), 15-3. https://doi.org/10.1080/00273171.2018.1484339
  » https://doi.org/10.1080/00273171.2018.1484339
• Martinez-Lopez, R., Yot, C., Tuovila, I., & Perera-Rodríguez, V. H. (2017). Online Self-Regulated Learning Questionnaire in a Russian MOOC. Computers in Human Behavior, 75, 966-74. https://doi.org/10.1016/j.chb.2017.06.015
  » https://doi.org/10.1016/j.chb.2017.06.015
• Pintrich, P. R. (2004). A Conceptual Framework for Assessing Motivation and Self-Regulated Learning in College Students. Educational Psychology Review, 16(4), 385-407. https://doi.org/10.1007/s10648-004-0006-x
  » https://doi.org/10.1007/s10648-004-0006-x
• Qiu, J., Tang, J., Liu, T. X., Gong, J., Zhang, C., Zhang, Q., & Xue, Y. (2016). Modeling and Predicting Learning Behavior in MOOCs. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining (pp. 93-102). ACM.
• Rodrigues, R. L., Ramos, J. L. C., Silva, J. C., Gomes, A. S., & Fonseca, J. A. V. (2016). Validação de um Instrumento de Mensuração de Autorregulação da Aprendizagem em Contexto Brasileiro Usando Análise Fatorial Confirmatória [Validation of a self-Regulation Learning Measurement Instrument in the Brazilian Context Using Confirmatory Factor Analysis]. Novas Tecnologias na Educação, 14(1). https://doi.org/10.22456/1679-1916.67337
  » https://doi.org/10.22456/1679-1916.67337
• Simão, A. M. V., & Frison, L. M. B. (2013). Autorregulação da Aprendizagem: Abordagens Teóricas e Desafios para as Práticas em Contextos Educativos [Self-Regulation Learning: Theoretical Approaches and Challenges for Practice in Educational Contexts]. Cadernos de Educação, 45, pp. 2-20. https://doi.org/10.15210/caduc.v0i45.3814
  » https://doi.org/10.15210/caduc.v0i45.3814
• Schunk, D. H. (2001). Self-Regulation Through Goal Setting. ERIC Digests, CG-01-08.
• Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time Will Tell: The Role of Mobile Learning Analytics in Self-Regulated Learning. Computers & Education, 89, 53-74. https://doi.org/1.1016/j.compedu.2015.08.004
  » https://doi.org/1.1016/j.compedu.2015.08.004
• Taghizade, A., Azimi, E., & Mirzaee, R. (2020).Validity Evidence for a Persian Version of the Online Self-Regulated Learning Questionnaire. Interdisciplinary Journal of Virtual Learning in Medical Sciences, 11(1), 13-24. https://dx.doi.org/10.30476/ijvlms.2020.84802.1017
  » https://dx.doi.org/10.30476/ijvlms.2020.84802.1017
• Valente, J. A. (2005). Pesquisa, comunicação e aprendizagem com o computador: o papel do computador no processo ensino-aprendizagem. Em BRASIL. Ministério da Educação. Integração das tecnologias na educação Brasília: Ministério da Educação, SEED.
• Vilkova, K., & Shcheglova, I. (2020). Deconstructing self-regulated learning in MOOCs: In search of help-seeking mechanisms. Education and Information Technologies https://doi.org/10.1007/s10639-020-10244-x
  » https://doi.org/10.1007/s10639-020-10244-x
• Zimmerman, B. J., & Schunk, D. H. (2011). Self-Regulated and Performance: An Introduction and an Overview. In Zimmerman, B.; Schunk, D. H., Handbook of Self-Regulation of Learning Band Performance, pp. 1-12. Routledge: Taylor; Francis Group.

Publication Dates

• Publication in this collection
  08 Oct 2021
• Date of issue
  2021

History

• Received
  08 Nov 2018
• Reviewed
  13 Sept 2019
• Accepted
  28 Oct 2020