Oldenburg Burnout Inventory - student version: cultural adaptation and validation into Portuguese

Inventário de Oldenburg para estudantes: adaptação cultural e validação para o português

PSYCHOLOGICAL ASSSESSMENT

Oldenburg Burnout Inventory – student version: cultural adaptation and validation into Portuguese

Inventário de Oldenburg para estudantes: adaptação cultural e validação para o português

Juliana Alvares Duarte Bonini Campos^I; Mary Sandra Carlotto^II; João Marôco^III

^IUniversidade Estadual Paulista, Araraquara, Brasil

^IIUniversidade Luterana do Brasil, Canoas, Brasil

^IIIInstituto Superior de Psicologia Aplicada (ISPA-IU), Lisboa, Portugal

^{Endereço para correspondência}

ABSTRACT

The Oldenburg Burnout Inventory for college students (OLBI-S) was adapted to Brazilian Portuguese and its reliability and validity were measured in a sample of both Brazilian (n=958) and Portuguese (n=602) college students. The confirmatory factor analysis of the OLBI-S showed good fit but two items were removed since they lacked individual reliability (λ<.50). The new structure showed good fit to 2/3 of the test sample and was invariant to the other 1/3 of the total sample. Convergent validity and internal consistency were low, but discriminant, concurrent and divergent validity were good. The OLBI-S was not invariant in the Brazilian and Portuguese samples. The adapted OLBI-S did not show cross-cultural validity.

Keywords: Burnout, college students, evaluation, Psychometrics.

RESUMO

Realizou-se adaptação cultural do Inventário de Oldenburg para estudantes (OLBI-S) em português e estimou-se sua confiabilidade e validade. O OLBI-S foi preenchido por 958 estudantes universitários brasileiros e 602 portugueses. O modelo fatorial original apresentou ajustamento adequado mas foram removidos dois itens com confiabilidade individual baixa (λ<0,5). A nova estrutura apresentou bom ajustamento a 2/3 da amostra total sendo invariante no 1/3 restante da amostra. Verificou-se baixa consistência interna e validade convergente, confiabilidade compósita aceitável, boa validade discriminante, concorrente e divergente. O OLBI-S não foi invariante nas amostras de Brasil e Portugal. O OLBI-S apresentou limitações e ausência de validade transcultural nas amostras estudadas.

Palavras-chave: Burnout, estudantes universitários, avaliação, Psicometria.

The burnout syndrome was initially defined as a psychological condition related to work in help professions and that require interaction with others, and is characterized by high levels of emotional exhaustion, high disbelief in the function and low professional accomplishment. Initially, it was believed that the burnout syndrome affected only professionals with extensive human interaction as social workers, nurses, doctors and psychologists. However, currently it is known that the burnout syndrome exceeds care services, since the stressors that can initiate it may be present in any workplace (Demerouti, Bakker, Nachreiner, & Schaufeli, 2001). More recently, higher education students have been identified in the literature as a vulnerable group for the development of Burnout (Balogun, Helgemoe, Pellegrini, & Hoeberlein, 1995; Marôco & Tecedeiro, 2009; Marôco, Tecedeiro, Martins, & Meireles, 2008; Martinez, Pinto, & Silva, 2000; Schaufeli, Martinez, Pinto, Salanova, & Bakker, 2002) due to socio-economic pressures, concerns about their professional future, relationships with peers and teachers, tests and papers to which they are constantly exposed, and therefore constitute a risk population to be studied.

Carlotto and Câmara (2006), Dyrbye et al. (2010), Martinez et al. (2000) and Salanova, Schaufeli, Martinez, and Breso (2010) and emphasize that early detection of symptomatic significant levels of burnout can be an indicator of potential problems, in both school and professional contexts, enabling preventive interventions. Marôco and Tecedeiro (2009) also indicate that the occurrence of Burnout can seriously limit students psychosocial well-being and the academic performance. From this perspective, the authors argue that the detection of Burnout and its determinants, in higher education, is an asset for understanding and developing psychological interventions in this area.

Despite the obvious need for studies on the burnout syndrome in students, these are incipient which, according Marôco and Tecedeiro (2009) and Marôco et al. (2008), may be occurring due to a lack of appropriate instruments to assess this syndrome in students. The only instrument proposed in the literature to assess the syndrome in students is the Maslach Burnout Inventory – Student Survey (MBI-SS; Schaufeli et al., 2002) whose psychometric properties have been evaluated and their stability attested in several studies, including university students in Portugal (Marôco & Tecedeiro, 2009; Schaufeli et al., 2002) and Brazil (Carlotto & Câmara, 2006). However, this scale has been the target of criticism leading several authors to propose alternative assessment tools (Demerouti, Bakker, Vardakou, & Kantas, 2003; Kristensen, Borritz, Villadsen, & Christensen, 2005). Demerouti et al. (2001), Demerouti et al. (2003) and Halbesleben and Demerouti (2005) criticize the fact that the MBI dimensions Exhaustion and Cynicism have a positive formulation, while the Professional Efficacy dimension has negative formulation, which can compromise the inventory's sensitivity and its discriminatory power. Another aspect highlighted by the authors is that the MBI only considers the emotional aspects of the exhaustion dimension excluding the physical and cognitive elements.

Given the criticisms made regarding the MBI, Demerouti and Nachreiner (1998) proposed the Oldenburg Inventory (OLBI) which consists of 16 items describing different states of emotional exhaustion and detachment. The degree of agreement with each item is expressed in 4-point ordinal scale (from 1 – totally disagree to 4 – totally agree). Half of the items were formulated positively and half were formulated negatively. This instrument was proposed originally in German, and was based on a theoretical model that assumes that Burnout is a two-dimensional syndrome that can occur regardless of the individual's occupation. The validity of OLBI was tested in different population groups (Demerouti et al., 2001; Demerouti et al., 2003; Demerouti, Mostert, & Bakker, 2010; Halbesleben & Demerouti, 2005; Peterson, Demerouti, Bergstrom, Asberg, & Nygren, 2008).

Taking into consideration that OLBI was developed to meet different occupational groups, its use in students can be an interesting alternative to MBI-SS. Thus, we carried out this study with the objective of proposing and assessing the reliability and validity of the Portuguese version of the Oldenburg Burnout Inventory adapted for students (OLBI-S).

Method

Participants

Participants were higher education students, volunteers, enrolled in institutions in Brazil and Portugal. The invitation to participate in the study was done for each institution by researchers, in person or via an e-mail sent to the schools Direction. The instruments were completed by a total of 1052 Brazilian students and 612 Portuguese. However, only 958 Brazilians (Response Rate – RR = 91.1%) and 602 Portuguese (RR = 98.4%) completed all OLBI's items and were included in the study. The mean age of the Brazilian students was 23.3 (SD = 5.4) years and of the Portuguese 22.9 (SD = 5.0) years. The characteristics of the participants are presented in Table 1.

To characterize the sample, information on gender, age, area of undergraduate course attended, type of institution, classes shift, dwelling, studies' financing, use of medication due to studies and thought about quitting the course, were assessed.

To evaluate the Burnout Syndrome we used the Oldenburg Inventory (OLBI) proposed by Demerouti et al. (2001) for screening the syndrome in the general population. Previous studies of the authors (Demerouti et al., 2003; Halbesleben & Demerouti, 2005) presented initial evidence of OLBI's factorial validity [χ²/df (ratio chi-square and degrees of freedom) =1.90; GFI (goodness of fit index) =.94; NFI (normed fit index) =.90; CFI (confirmatory fit index) =.95; RMSEA (root mean square error of approximation) =.062].

Instruments

Given that we did not find in the literature a Portuguese version of OLBI, in this study translation, and back translation, from the original English version by Halbesleben and Demerouti (2005) was performed, for development of the instrument based on the Portuguese language orthographic agreement (Ministry Science, 2008), being adapted for its application in university students. The adapted instrument is called by us the Oldenburg Burnout Inventory – Student Version (OLBI-S).

To allow an estimate of the concurrent validity of the OLBI-S, the Portuguese version of the Maslach Burnout Inventory (MBI-SS) validated by Campos, Zucoloto, Bonafé, Jordani and Marôco (2011) χ²/df=2.360; CFI=.913; GFI=.856; RMSEA=.068; AVE=.551-.786; CR=.830-.916) was used. We also used the Portuguese version of the Beck Depression Scale (BDI) to assess the divergent validity of OLBI-S.

Procedures

A website was created for the sociodemographic questionnaire and OLBI-S' Portuguese version. The questionnaires were available for completion online for 7 months (May to November). Each web page hosted one instrument, so that the respondent could view all items simultaneously. Non-responses to items were allowed, and the participant could return to verify and/or correct the answer to each inventory before its submission. Before any analysis, all items with negative statements were reversed so that the scores of all items were in the same conceptual direction (lower scores represent disagreement, higher scores correspond to agreement with the statements).

Psychometric Analyzes

Face Validation. The process of face validation involved the participation of a multidisciplinary team (portuguese-speaking psychologists) with eight members. The idiomatic, semantic, cultural and conceptual equivalence of the instrument were analyzed in order to obtain agreement and consensus. Thus, an intermediate version of the instrument was obtained. This version was pre-tested in a group of 20 students to assert the misunderstanding index of each question.

Content Validation. A total of 13 Psychology professionals (judges) Participated in this step, to analyze each OLBI-S' items regarding essentiality and classified them into "essential", "useful but non-essential" and "not necessary". For each item, the number of judges categorizing the item as "essential" was computed for calculation of the Content Validity Ratio (CVR). To decide the significance of each item Laewshe's proposal (1975) was used, and a 5% significance level was adopted.

Psychometric Qualities

Sensitivity Analysis of Items. The psychometric sensitivity was investigated with the dual purpose of evaluating both the shape of the responses' distribution to each item and the approximate normality of the items to be used in the confirmatory factor analysis. The psychometric sensitivity was assessed for each item by calculating descriptive statistics, including measures of central tendency and measures of shape. We considered that items with skewness (Sk) and kurtosis (Ku) close to 0 had maximum sensitivity, and that the greater the difference from 0 the lower the items' sensitivity. Note that, in the population, the distribution of the responses to each item must follow an approximately normal distribution to maximize the ability of the item to discriminate structurally different individuals. Moreover, it is expected that the items' scores present a normal distribution if they are good projections of the construct, which is supposed to be normally distributed in the population. We considered that items with a Sk greater than 3, in absolute value, and Ku greater than 7, in absolute value, had sensitivity and normality problems that recommend against their use in factor analysis (Marôco, 2010).

Construct Validity

Factorial Validity. We carried out confirmatory factor analysis (CFA) of the Portuguese version for students (OLBI-S) to verify the two-dimensional structure proposed by Halbesleben and Demerouti (2005). In this sense the following goodness of fit indices were used χ²/df (ratio chi-square and degrees of freedom), CFI (confirmatory fit index), GFI (goodness of fit index) and RMSEA (root mean square error of Approximation). The model's adjustment was considered good for CFI and GFI values above .9 and RMSEA values below .06 (Boomsma, 2000; Byrne, 2001; Marôco, 2010; McDonald & Ho, 2002). To assess the AFC we used AMOS ® 18.0.

Factorial Invariance. Initially, to verify the stability of the factor solution obtained, a cross-validation of the model was made in order to compare the indexes observed in the validation sample with another independent test sample coming from the same population (Hair, Black, Babin, Anderson, & Tatham, 2005; Marôco, 2010). For this the total sample was divided into three equal parts with two parts constituting the "validation sample" and one part the "Test sample" to perform a cross testing of the hypothetical model. The invariance test was conducted imposing equality restrictions to the factorial weights of the two groups, being the statistical test the difference between the χ² of the model fixed factor weights and the model with equal factor weights. When the hypothesis of factorial invariance was accepted, the analysis of the invariance of the correlations between factors was performed, and finally the invariance of specific factors (residues; Kaplan, 2000). Then, the same procedure was made to assess the invariance of the factor solution obtained in the Portuguese sample and in the sample Brazilian.

Convergent Validity. To examine whether each dimension's observed variables were strongly related to each other, the convergent validity was evaluated, to this end the Average Variance Extracted (AVE) and the Composite Reliability (CC) were estimated (Fornell & Larcker, 1981; Marôco, 2010). According to Hair et al. (2005) values VEM_j >.5 and CC_j >.7 are indicative of convergent validity and adequate construct reliability being that, for exploratory research only, values above these cutoff points may be acceptable.

Discriminant Validity. Discriminant validity assesses whether the items that reflect a dimension are not correlated with another dimension, i.e., if the average variance extracted for each factor is greater than the average variation shared between each factor and other factors in the model (Marôco, 2010). The discriminant validity was estimated according to Fornell and Larcker proposal (1981) as described in Marôco (2010) who claim that for two factors i and j, if VME_i and VME_j > ρ_ij² (ρ_ij²: squared correlation between factors i and j) evidence of discriminant validity exists.

Criterion Validity. The criterion-related validity was assessed through the concurrent validity and discriminant validity. For estimating both the concurrent validity of the divergent validity, Pearson's correlation analysis was used. For the former, the mean score of OLBI-S each factor was correlated with those obtained in each dimension of the Maslach Burnout Inventory (MBI-SS). For divergent validity OLBI-S' factors scores were correlated with the average of the scores obtained with the Beck Depression Scale (BDI).

Reliability. The internal consistency was assessed using standard Cronbach's alpha coefficient (α) for each dimension proposed in the inventories, and to calculate the consistency of the total scale stratified alpha Coefficient (α_estr) was estimated.

Ethical Aspects. The present study was approved by the Ethics Committee on Human Research of the Lutheran University of Brazil, Canoas / RS (protocol: 2010-188H).

Results

Table 2 presents the Portuguese version of the Oldenburg Inventory adapted for students (OLBI-S) after face validation.

It should be clarified that the statements in Table 2 were grouped according to their dimension to facilitate the understanding of OLBI's structure (Demerouti et al., 2001).

In the pre-test we observed that no item showed index misunderstanding <.20, thus, the calculation of the content validity ratio was made (CVR; Table 3).

In the opinion of the judges only 4 items of the inventory are essential to investigate the Burnout Syndrome in students.

Summary measures of OLBI-S' items to characterize the psychometric sensitivity of the Portuguese and Brazilian sample can be found in Table 4.

All items presented skewness and kurtosis values close to those of a normal distribution (Sk = 0, Ku = 0) in both samples.

Items that had factorial weights (λ) negative and lower than .50 were removed. Items whose errors are correlated with the factors or other errors, according to the modification indices provided by AMOS®, were also removed. The decision on reducing the items was defined with the goal of producing a scale with greater reliability and validity, preserving the original factor structure.

The Confirmatory Factor Analysis pointed to an appropriate adjustment (χ²/df = 4.543, CFI = .913, GFI = .944, RMSEA = .058), however, the factorial weights of items 5 and 12 of the Exhaustion factor, and item 13 of the Disengagement factor were below .5. Thus, there was a refinement of the original model using the modification indices for both obtained with AMOS ®. Item 5 was deleted, since the modification indices indicated a correlation of the measuring error with the Disengagement dimension and item 13 of the Exhaustion dimension. None of the removed items, except one, were considered "essential" by the content validity panel, therefore usuring that these items remotion did not affect the content validity of the factors under study.

Thus, we propose a reduced scale, with the dimensions Exhaustion and Disengagement consisting of 7 items each (Figure 1).

The values shown in the figure are standardized estimates of covariance between factors, factorials weights and explained variance of each item respectively. We can observe that the factor model has appropriate quality of adjustment indices. All items of the scale (OLBI-S), with the exception of item 14, showed factorial weights (λ≥.5) and adequate individual reliability (R²≥.25). The option of maintaining item 14 on the scale was due to the fact that the modification indices did not suggest significant changes in the quality of the adjustment associated with the removal of that item.

In the external validation of the modified structure, we analyzed the invariance of the model in the test sample and the validation sample. The evaluation in both samples, simultaneously, showed very good adjustment indices (χ²/df = 3.777, CFI: .911; GFI: .938, RMSEA: .042).

The differences in the goodness of fit of the factor model to the two samples revealed no significant differences. Similarly, neither the covariances between factors nor the residues associated with the items revealed significant differences between the two samples. These observations point to the invariance of the model in the two independent samples confirming the stability of the factor structure proposed in this study of adaptation and validation [λ: χ²(12)_dif=19.666, p=.074; Cov: χ²(15)_dif=24.181, p=.062; Residues:, χ²(29)_dif=37.252, p=.140].

The convergent validity for OLBI-S was low (VEM_Ex=.358; VEM_Dist=.363; CC_Ex=.793; CC_Dist=.797) while discriminant validity was adequate (r²= .067).

The internal consistency of the Exhaustion dimension was below the desired levels (α = .565) and was adequate for the Disengagement dimension (α = .700). The total scale OLBI-S showed adequate consistency (α_estr = .952).

The correlational analysis between the reduced Oldenburg Inventory for students (OLBI-S) and the Maslach Burnout Inventory – Student Survey (MBI-SS) and Beck's Depression Inventory (BDI) is presented in Table 5.

The correlation coefficients found between the OLBI-S and the MBI-SS dimensions point to a moderate concurrent validity. Moreover, the low correlation between the dimensions of OLBI-S and BDI denote good divergent validity from a measurement scale for depression distinguishing the two constructs.

In the evaluation of the adjustment simultaneously for the Portuguese and the Brazilian sample, adequate adjustment indices were found (χ²/df = 5.914; CFI: = .839, GFI = .906, RMSEA = .056). However, we did not observe invariance of the models [λ: χ²(12)_dif=129.656, p<.001; Cov: χ²(15)_dif=153.470, p<.001; Residues: χ²(29)_dif=331.203, p<.001]. Figure 2 presents the factorial models adjusted separately in the sample in the Portuguese and the Brazilian sample.

A cross-cultural equivalence between countries was not found. In the Brazilian sample items 14, 1, 7 and 11 showed factor weights lower than .50, while in the Portuguese sample this only occurred for item 6. The correlation between the two scales of OLBI-S is also considerably higher in the Portuguese sample when compared to the Brazilian sample (r = .69 vs. r = .59).

The mean Exhaustion scores (Brazil: 2.70 ± .48; Portugal: 2.47 ± .37) and Disengagement (Brazil: 2.25 ± .46; Portugal: 2.03 ± .53) were significantly different (p <.001).

Discussion

This study presents a version in Portuguese of the Oldenburg Burnout Inventory, adapted for college students, and examines its reliability and validity when applied to a sample of Brazilian and Portuguese students. Despite any discussion, it is worth noting that so far the psychometric properties of OLBI had been tested only on workers from various fields (Demerouti et al., 2001; Demerouti et al., 2003; Demerouti et al., 2010; Demerouti & Nachreiner, 1998; Halbesleben & Demerouti, 2005, Peterson et al., 2008), to the best of our knowledge, this is the first study to use this inventory and test its properties when applied to students.

The results of this study confirm the two-dimensional structure of OLBI, previously described by Demerouti et al. (2001), Demerouti et al. (2003), Demerouti et al. (2010), Demerouti and Nachreiner (1998) and Halbesleben and Demerouti (2005), however, it should be emphasized that to achieve adequate factorial validity, an item of each dimension of the instrument was excluded (Figure 1). This occurred because the confirmatory factor analysis indicated factorial weights of the items of the original version of the instrument were below desired values. This structure was then confirmed in a second similar sample, but independent from the first sample, where the factor structure was refined. After the refinement of the model, the only quality adjustment index that was inadequate was χ²/df, however, this fact is not relevant since this index is extremely sensitive to the number of variables in the model and sample size, as already emphasized by Bentler (1990) and Marôco (2010).

Another aspect to be highlighted is that in the content analysis (Table 2) only four items from OLBI were considered, by the experts, as essential for evaluation of the Burnout syndrome in students. One of these four items, the item O5 belonging to the Exhaustion dimension, was subsequently removed from the inventory, as it presented factorial weight lower than .50. Given that the sample is big and includes students from different areas of study, from different years and two Lusophone countries, it is sustainable in the conclusion that this item is a weak expression of Exhaustion in the student population. It should also be noted that this item also presented weak correlations (r = .3 for Exhaustion, r = .2 for Cynicism and Effectiveness) with scores on MBI-SS' factors. These observations contradict the assessment of the relevance of the item by the experts.

OLBI's concurrent validity when applied to workers from different professional categories has been ascertained with the Maslach Burnout Inventory – General Survey (MBI-GS). Demerouti et al. (2003) and Halbes-leben and Demerouti (2005) found a high correlation between the dimensions Exhaustion and Disengagement/ Cynicism from OLBI and MBI-GS, pointing to OLBI's high concurrent validity. Because it is a research with students, the concurrent validity of OLBI-S was measured before with the student version of the MBI (MBI-SS; previously validated for the population of Portugal and Brazil by our research team Campos et al., 2011; Carlotto & Câmara, 2006; Marôco et al., 2008) and moderate correlations can be noted between the dimensions of the instruments (Table 5) indicating that they are part of the same conceptual model. In addition, the low convergent validity observed may indicate that the scale is not measuring the intended concept (Hair et al., 2005), in other words, items that should be reflective of a factor are not heavily saturating in this factor (Marôco, 2010)

When the refined model is implemented separately for the Brazilian and Portuguese sample there is an absence of cross-cultural equivalence, which reflects mainly the differences found between the factor weights of each item of the instrument in the different samples (Figure 2). Before attributing this disparity to cultural differences between the countries, it should be noted that in this study the characteristics of the samples were not paired (Table 3). This fact is a limitation that must be considered, since the psychometric properties of an instrument are specifically related to an individual population and is not, therefore, an absolute characteristic of each test, as already pointed out by Honaker (1988) and by Suris, Borman, Lind and Kashner (2007). However, the possible influence of the sample characteristics in the factor validity of the model can point to low factor stability of OLBI-S when used in different populations. It is, however, important to highlight that despite significant differences between the factorial weights of the items on the two factors considered (exhaustion and cynicism) in samples from Portugal and Brazil, the same items were retained in the analysis of factorial validity in both the Portuguese sample and the Brazilian sample. This indicates the stability of the factor structure proposed for OLBI-S adapted into Portuguese, not dismissing however other studies in different samples.

Despite confirming the two-dimensional structure of the Portuguese version of OLBI-S, it had limitations regarding the convergent validity and internal consistency. Factor structure was not invariant between the Brazilian and Portuguese sample pointing to the absence of cross-cultural validity

Recebido: 03/02/2011

1ª revisão: 19/12/2011

Aceite final: 26/12/2011

Acknowledgements: We would like to thank Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) for the financial support (processes: 2010/15062-3 e 2010/09295-5).

• Balogun, J. A., Helgemoe, S., Pellegrini, E., & Hoeberlein, T. (1995). Test-retest reliability of a psychometric instrument designed to measure physical therapy students' burnout. Perceptual & Motor Skills, 81(2), 667-672.
• Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238-246.
• Boomsma, A. (2000). Reporting analyses of covariance structures. Structural Equation Modeling- A Multidisciplinary Journal, 7(3), 22.
• Byrne, B. M. (2001). Structural equation modeling with Amos: Basic concepts, applications and programming Mahwah, NJ: Lawrence Erlbaum.
• Campos, J. A. D. B., Zucoloto, M. L., Bonafé, F. S. S., Jordani, P. C., & Marôco, J. (2011). Reliability and validity of self reported burnout in college students: A cross randomized comparison of pencil-and-paper vs online administration. Computers in Human Behavior, 27(5), 1875-1883.
• Carlotto, M. S., & Câmara, S. G. (2006). Características psicométricas do Malasch Burnout Inventory Student Survey (MBI-SS) em estudantes universitários brasileiros. Psico-USF 11(2), 167-173.
• Demerouti, E., Bakker, A. B., Nachreiner, F., & Schaufeli, W. B. (2001). The job demands-resources model of burnout. Journal of Applied Psychology, 86(3), 499-512.
• Demerouti, E., Bakker, A. B., Vardakou, I., & Kantas, A. (2003). The convergent validity of two burnout instruments A multitrait-multimethod analysis. European Journal of Psychological Assessment, 19(1), 12-23. doi: 10.1027//1015-5759. 19.1.12
• Demerouti, E., Mostert, K., & Bakker, A. B. (2010). Burnout and work engagement: A thorough investigation of the independency of both constructs. Journal of Occupational Health Psychology, 15(3), 209-222.
• Demerouti, E., & Nachreiner, F. (1998). Zur Spezifität von Burnout für Dienstleistungsberufe: Fakt oder Artefakt [The specificity of burnout for human services: Fact or artefact]. Zeitschrift für Arbeitswissenschaft, 52, 82-89.
• Dyrbye, L. N., Thomas, M. R., Power, D. V., Durning, S., Moutier, C., Massie, F. S., Jr., et al. (2010). Burnout and serious thoughts of dropping out of medical school: A multi-institutional study. Academic Medicine, 85(1), 94-102.
• Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.
• Hair, J. F., Black, W. C., Babin, B., Anderson, R. E., & Tatham, R. L. (2005). Multivariate data analysis (6^th ed.). Upper Saddle River, NJ: Prentice Hall.
• Halbesleben, J. R. B., & Demerouti, E. (2005). The construct validity of an alternative measure of burnout: Investigating the English translation of the Oldenburg Burnout Inventory. Work and Stress, 19(3), 208-220. doi: 10.1080/026783705 00340728
• Honaker, L. M. (1988). The Equivalency of Computerized and Conventional Mmpi Administration - A critical-review. Clinical Psychology Review, 8(6), 561-577.
• Kaplan, D. (2000). Structural equation modeling: Foundations and extensions Thousand Oaks, CA: Sage.
• Kline, R. B. (1998). Principles and practice of structural equation modeling New York: The Guilford Press.
• Kristensen, T. S., Borritz, M., Villadsen, E., & Christensen, K. B. (2005). The Copenhagen Burnout Inventory: A new tool for the assessment of burnout. Work and Stress, 19(3), 192-207. doi: 10.1080/02678370500297720
• Laewshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 13.
• Marôco, J. (2010). Análise de equações estruturais Lisboa, Portugal: ReportNumber.
• Marôco, J., & Tecedeiro, M. (2009). Inventário de Burnout de Maslach para estudantes portugueses. Psicologia, Saúde & Doenças 10(2), 227-235.
• Marôco, J., Tecedeiro, M., Martins, P., & Meireles, A. (2008). Estrutura fatorial de segunda ordem da Escala de Burnout de Malasch para estudantes numa amostra portuguesa. Análise Psicológica, 4(26), 639-649.
• Martinez, I. M. M., Pinto, A. M., & Silva, A. L. (2000). Burnout em estudantes do ensino superior. Revista Portuguesa de Psicologia, 35, 17.
• McDonald, R. P., & Ho, M. H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64-82. doi: 10.1037//1082-989x.7.1.64
• Ministry Science. (2008). Acordo ortográfico de 1990 Retrieved October 27, 2008, from http://www.portaldalingua-portuguesa.org
• Peterson, U., Demerouti, E., Bergstrom, G., Asberg, M., & Nygren, A. (2008). Work characteristics and sickness absence in burnout and nonburnout groups: A study of Swedish health care workers. International Journal of Stress Management, 15(2), 20. doi: 10.1037/1072-5245.15.2.153
• Salanova, M., Schaufeli, W., Martinez, I., & Breso, E. (2010). How obstacles and facilitators predict academic performance: The mediating role of study burnout and engagement. Anxiety Stress and Coping, 23(1), 53-70. doi: 10.1080/106158 00802609965
• Schaufeli, W. B., Martinez, I. M., Pinto, A. M., Salanova, M., & Bakker, A. B. (2002). Burnout and engagement in university students A cross-national study. Journal of Cross-Cultural Psychology, 33(5), 464-481.
• Suris, A., Borman, P. D., Lind, L., & Kashner, T. M. (2007). Aggression, impulsivity, and health functioning in a veteran population: Equivalency and test-retest reliability of computerized and paper-and-pencil administrations. Computers in Human Behavior, 23(1), 97-110. doi: 10.1016/j.chb. 2004.03.038

Publication Dates

• Publication in this collection
  24 Jan 2013
• Date of issue
  2012

History

• Received
  03 Feb 2011
• Accepted
  26 Dec 2011
• Reviewed
  19 Dec 2011