Acessibilidade / Reportar erro

BULLYING AND MECHANISMS OF MORAL DISENGAGEMENT: SYSTEMATIC LITERATURE REVIEW WITH META-ANALYSIS

ABSTRACT

Bullying is one of the main problems faced by children and adolescents in schools. This study aimed to examine evidence available in the literature on the effect of moral disengagement on the involvement of students in situations of bullying. To this end, a systematic review was carried out with meta-analysis and consultation of eight databases. Eligible publications were analyzed using conceptual analysis and the Metafor statistical package. The study was registered on the PROSPERO database. In total, 15 articles were included in the review. The data show that moral disengagement is a strong predictor for bullying behavior (OR 0.27; 95% CI: 0.18, 0.35) and students prone to morally disengage can practice more bullying when compared to students with greater moral engagement. It is concluded that cognitive factors related to moral disengagement must be included in the discussion of the phenomenon, as well as be considered in the anti-bullying intervention proposals.

Keywords:
moral development; school violence; bullying

RESUMO

O bullying é um dos principais problemas enfrentados por crianças e adolescentes nas escolas. Este estudo teve por objetivo examinar evidências disponíveis na literatura sobre o efeito do desengajamento moral no envolvimento de estudantes em situações de bullying. Para tanto, foi realizada revisão sistemática com metanálise e consulta a oito bases de dados. As publicações elegíveis foram analisadas por meio da análise conceitual e do pacote estatístico Metafor. O estudo foi registrado na base PROSPERO. No total, 15 artigos foram incluídos na revisão. Os dados evidenciam que o desengajamento moral é um forte preditor para o comportamento de bullying (OR 0,27; IC95%: 0,18, 0,35) e estudantes propensos a se desengajarem moralmente podem praticar mais bullying quando comparados com estudantes com maior engajamento moral. Conclui-se que fatores cognitivos relacionados ao desengajamento moral devem ser incluídos na discussão do fenômeno, bem como ser considerados nas propostas de intervenção antibullying.

Palavras-chave:
desenvolvimento moral; violência escolar; bullying

RESUMEN

El bullying es uno de los principales problemas enfrentados por niños y adolescentes en las escuelas. En este estudio se tuvo por objetivo examinar evidencias disponibles en la literatura sobre el efecto del no compromiso moral en el envolvimiento de estudiantes en situaciones de bullying. Para tanto, se realizó revisión sistemática con metaanálisis y consulta a ocho bases de datos. Las publicaciones elegibles se analizaron por intermedio del análisis conceptual y del paquete estadístico Metafor. El estudio fue registrado en la base PROSPERO. En total, 15 artículos se incluyeron en la revisión. Los datos evidencian que el no compromiso moral es un fuerte predictor para el comportamiento de bullying (OR 0.27; IC95%: 0.18, 0.35) y estudiantes propensos a no comprometerse moralmente pueden practicar más bullying cuando comparados con estudiantes con más compromiso moral. Se concluye que factores cognitivos relacionados a la falta de compromiso moral deben ser incluidos en la discusión del fenómeno, así como ser considerados en las propuestas de intervención antibullying.

Palabras clave:
desarrollo moral; violencia escolar; bullying

INTRODUCTION

The occurrence of bullying has a negative impact on the development, health and teaching-learning process of children and adolescents (Crochik, 2016Crochík, J. L. (2016). Hierarchy, violence and bullying among students of public middle schools. Paidéia (Ribeirão Preto), 26(65), 307-315. doi: 10.1590/1982-43272665201608
https://doi.org/10.1590/1982-43272665201...
; Fernandes, Yunes, & Finkler, 2020Fernandes, G., Yunes, M. A. M., & Finkler, L. (2020). The social networks of adolescent victims of domestic violence and bullying. Paidéia (Ribeirão Preto), 30, e3007. doi: 10.1590/1982-4327e3007
https://doi.org/10.1590/1982-4327e3007...
; Zequinão, Medeiros, Silva, Pereira, & Cardoso, 2020Zequinão, M. A., Medeiros, P., Silva, J. L., Pereira, B. O., & Cardoso, F. L. (2020). Sociometric status of participants involved in school bullying. Paidéia(Ribeirão Preto), 30, e3011. doi:10.1590/1982-4327e3011
https://doi.org/10.1590/1982-4327e3011...
). The phenomenon corresponds to a type of violence characterized by the intentionality of the aggressors to inflict pain or suffering on the victims, by the reiteration of aggressive behaviors and by the imbalance of power among the students involved (Olweus, 2013Olweus, D. (2013). School bullying: Development and some important challenges. Annual Review of Clinical Psychology, 9(1), 751-780. doi: 10.1146/annurev-clinpsy-050212-185516
https://doi.org/10.1146/annurev-clinpsy-...
). It is a group phenomenon, as each participant has a tendency to reverberate or maintain the rules and conditions established by the members with higher status or popularity (Sijtsema et al., 2014Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
https://doi.org/10.1037/a0037145...
).

Although a group phenomenon, individual issues must be considered in the analysis of their dynamics. This aspect has been valued and highlighted since the first studies about the theme conducted by Dan Olweus since the 1970s (Olweus, 2013Olweus, D. (2013). School bullying: Development and some important challenges. Annual Review of Clinical Psychology, 9(1), 751-780. doi: 10.1146/annurev-clinpsy-050212-185516
https://doi.org/10.1146/annurev-clinpsy-...
). It is essential to understand what are the individual mechanisms underlying bullying behavior, through research, in order to provide subsidies for prevention and intervention programs in schools (Sijtsema et al., 2014Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
https://doi.org/10.1037/a0037145...
). Cognitive factors and other dimensions of development are included in this discussion of individual factors and they can contribute to the occurrence of bullying.

Thus, with regard to individual factors, moral disengagement refers to a series of cognitive processes considered valid to elucidate the aggressive or deviant behavior of children and adolescents (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.). According to the author, the processes of moral disengagement correspond to mechanisms of cognitive self-justification for transgressive actions that were hypothetically learned and socialized in specific social contexts. Such theoretical construct was elaborated in line with assumptions of Social Learning Theory and Cognitive Social Psychology.

Moral disengagement consists of eight mechanisms: (1) moral justification, (2) advantageous comparison, (3) euphemistic language, (4) displacement of responsibility, (5) diffusion of responsibility, (6) consequences distortion, (7) dehumanization and (8) blaming the victim (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.). The first three mechanisms refer to cognitive restructuring, on the basis of which people perceive negative or immoral behavior in a more positive way (moral justification) or interpret that such behavior is less immoral / harmful when compared to another (advantageous comparison). In other situations, people can use language as a resource to minimize or obscure the negative impact of behavior (euphemistic language) (Bandura et al., 1996; Hymel & Bonanno, 2014Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
https://doi.org/10.1080/00405841.2014.94...
).

The following two mechanisms, displacement and diffusion of responsibility, allow to minimize the role of agent of immoral or negative behavior, shifting personal responsibility to other people or to the group of peers. The distortion of consequences involves distorting or disregarding the consequences of the behavior perceived as more positive. The mechanisms of dehumanization and blaming allow the victim to be considered deserving of immoral or negative behaviors, as well as unworthy of receiving treatments given to the human being itself (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.; Hymel & Bonanno, 2014Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
https://doi.org/10.1080/00405841.2014.94...
).

These eight mechanisms are considered to be self-regulating and create a link among cognition, moral emotion and morally engaged or disengaged behavior (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.; Hymel & Bonanno, 2014Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
https://doi.org/10.1080/00405841.2014.94...
). According to scientific literature, aggressive behaviors and the practice of bullying among students have been associated with moral disengagement, as aggressors, as a rule, rationally justify their actions, do not have a feeling of guilt and do not recognize the damage caused to the victims (Bandura et al., 1996; Sijtsema et al., 2014Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
https://doi.org/10.1037/a0037145...
). Thus, this study aimed to examine the evidence available in the literature on the effect of moral disengagement on the involvement of students in situations of bullying.

METHOD

Type of study

The method of systematic literature review with meta-analysis was used to identify, evaluate and interpret data from studies carried out on the interface between bullying and moral disengagement. The following execution process was adopted: (1) formulation of the guiding question, (2) selection of databases, (3) definition of the search strategy, (4) data collection and selection procedure, (5) analysis and presentation of results and (6) presentation of the systematic review (Carvalho, Pianowski, & Santos, 2019Carvalho, L. F., Pianowski, G., & Santos, M. A. (2019). Guidelines for conducting and publishing systematic reviews in Psychology. Estudos de Psicologia(Campinas ), 36, e180144. doi:10.1590/1982-0275201936e180144
https://doi.org/10.1590/1982-0275201936e...
). The protocol for conducting this systematic review was registered on the PROSPERO (International Prospective Register of Systematic Reviews) database under the number: CRD42018099782.

Formulation of the Guiding Question

In order to define the research question, the PVO model was applied, an acronym formed by P - participants / context / phenomenon studied, V - variables under study and O - outcome / results. Thus, the guiding question of this study was: What is the relationship between the mechanisms of moral disengagement and the involvement of students in school bullying situations? The PVO model was structured as follows: P - students involved in bullying situations, V - bullying and moral disengagement, O - identifying, describing and analyzing results on the relationship between the chosen variables.

Databases Selection

The search included seven sources of information, which index the literature in the following areas: Psychology, Health, Education and Multidisciplinary. The Web of Science, PsycINFO, Scopus, PubMed, SciELO, PePSIC and LILACS databases were consulted.

Search Strategy Definition

The descriptors bullying and moral disengagement were used in all databases consulted. The inclusion criteria were: (1) studies with cross-sectional or longitudinal designs, (2) studies evaluating the prevalence of bullying and moral disengagement, (3) studies with school-age children and adolescents, and (4) texts published in English, Spanish or Portuguese in the period from January 2013 to August 2018. Studies that did not address or that only secondarily referred to the themes in question (moral disengagement and bullying) were excluded, as well as books, book chapter, thesis, dissertation, editorial, letters to the editor and annals of scientific event. This review did not include studies about cyberbullying, as the specialized literature considers this phenomenon to be related, but distinct in terms of characteristics and dynamics when compared to traditional bullying.

Data Collection and Selection Procedures

The search for articles in the databases took place in September 2018 and it was carried out by two independent reviewers. Considering the different ways of presenting the authorship of articles in the databases, duplicates were eliminated via EndNote reference management software and manually in Excel. The first screening was carried out by the two researchers by reading the titles and abstracts identified at the intersection of the descriptors used. In the second moment, the researchers retrieved and read the selected articles in full. Divergences between the two were discussed until consensus was reached.

Analysis and Presentation of Results

The data storage and management of the articles were carried out in an Excel spreadsheet with the following indicators: title, authors, year and journal, country where the study was developed, type of study, number of participants, instrument used to measure bullying episodes, instrument used to measure moral disengagement, study objectives, main results and limitations presented. This systematization facilitated the descriptive analyzes and the critical appraisal of the selected studies. Conceptual analysis was also developed (Walker & Avant, 2018Walker, L. O.; Avant, K. C. (2018). Strategies for theory construction in nursing. New York: Pearson.) to identify how the construct of moral disengagement is being outlined in studies about bullying.

In the meta-analysis process, the measurement effect used for the analysis was Pearson’s r coefficient between the scores obtained for bullying and moral disengagement in the reviewed studies. The analyzes were performed using the Metafor statistical package (Viechtbauer, 2010Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor Package. Journal of Statistical Software, 1(3), 1-48. doi: 10.18637/jss.v036.i03
https://doi.org/10.18637/jss.v036.i03...
) of software R, version 3.5.1. The evaluation of the heterogeneity between the studies and their magnitude was estimated using Cochran’s Q tests and the I² index. The reference percentages for heterogeneity, considering the I² index, were: close to 25% (low), close to 50% (moderate) and close to 75% (high) (Higgins & Thompson, 2002Higgins, J. P. T.; Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539-1558. doi: 10.1002/sim.1186
https://doi.org/10.1002/sim.1186...
). The level of significance was set at 5%.

The articles that make up this review were also evaluated by two independent researchers for methodological quality, using two instruments: (1) Prevalence Critical Appraisal Instrument and (2) Appraisal tool for CrossSectional Studies (AXIS tool). The Prevalence Critical Appraisal Instrument contemplates 10 questions to assess the methodological quality of studies that present prevalence data (Munn et al., 2014Munn, Z.; Moola, S.; Riitano, D.; Lisy, K. (2014). The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. International Journal of Health Policy and Management, 3(3), 123-128. doi: 10.15171/ijhpm.2014.71
https://doi.org/10.15171/ijhpm.2014.71...
). The AXIS tool is an instrument built to support intervention decisions by assessing the quality and risk of cross-sectional research bias in the scientific literature (Downes et al., 2016Downes, M. J.; Brennan, M. L.; Williams, H. C.; Dean, R. S. (2016). Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open, 6(12), e011458. doi: 10.1136/bmjopen-2016-011458
https://doi.org/10.1136/bmjopen-2016-011...
). Considering the guiding question of the research, the AXIS tool was adapted to also evaluate the five articles with longitudinal design.

RESULTS

Descriptive Analysis

In the first stage of the study, all results were considered eligible and totaled 27 articles, eliminating repetitions among intersections or among databases. The flow of the study corpus construction process is shown in Figure 1.

Figure 1
PRISMA flowchart of the Search and Selection Process for Articles.

The descriptive analysis of the articles included in the review made it possible to identify the distribution of publications by year over the period studied (2013-2018). There were no trends in concentration, as the studies were distributed evenly over the years. The articles were published in 11 journals, 47% were published in journals with a high level of expertise (Journal of School Violence, Journal of Interpersonal Violence and Aggressive Behavior). Regarding the country of origin of the studies reviewed, Italy, followed by Australia, Greece and Sweden were the contexts for the greatest development of research on bullying and moral disengagement. No studies developed in Brazil were found.

Of the total of studies included, 11 had a cross-sectional design and used robust samples of participants, that is, with a total number that allows generalizations. All articles reported quantitative results. The number of surveys that adopted the Olweus Bully / Victim Questionnaire or adaptations of this questionnaire was significant. With regard to the assessment of moral disengagement, most studies used the Moral Disengagement Scale (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.).

In general, the forms of manifestation of bullying were significantly positively correlated with moral disengagement, just as boys reported more moral disengagement and involvement in bullying situations when compared to girls (Kokkinos & Kipritsi, 2018Kokkinos, C. M.; Kipritsi, E. (2018). Bullying, moral disengagement and empathy: Exploring the links among early adolescents. Educational Psychology, 38(4), 535-552. doi: 10.1080/01443410.2017.1363376
https://doi.org/10.1080/01443410.2017.13...
; Sijtsema et al., 2014Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
https://doi.org/10.1037/a0037145...
; Thornberg et al., 2015Thornberg, R.; Pozzoli, T.; Gini, G.; Jungert, T. (2015). Unique and interactive effects of moral emotions and moral disengagement on bullying and defending among school children. Elementary School Journal, 116(2), 322-337. doi: 10.1086/683985
https://doi.org/10.1086/683985...
; Van Noorden et al., 2014Van Noorden, T. H. J.; Haselager, G. J. T.; Cillessen, A. H. N.; Bukowski, W. M. (2014). Dehumanization in children: The link with moral disengagement in bullying and victimization. Aggressive Behavior, 40(4), 320-328. doi: 10.1002/ab.21522
https://doi.org/10.1002/ab.21522...
; Wang, Ryoo et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). Students who reported practicing bullying also tended to justify their behaviors through the mechanisms that make up the moral disengagement construct previously presented (Orue & Calvete, 2016Orue, I.; Calvete, E. (2016). Psychopathic traits and moral disengagement interact to predict bullying and cyberbullying among adolescents. Journal of Interpersonal Violence, 34(11), 2313-2332. doi: 10.1177/0886260516660302
https://doi.org/10.1177/0886260516660302...
; Thornberg & Jungert, 2014Thornberg, R.; Pozzoli, T.; Gini, G.; Jungert, T. (2015). Unique and interactive effects of moral emotions and moral disengagement on bullying and defending among school children. Elementary School Journal, 116(2), 322-337. doi: 10.1086/683985
https://doi.org/10.1086/683985...
)

In this scenario, students with higher scores on scales of moral disengagement presented as a related result the increase in the practice of bullying against colleagues (Obermann, 2013Obermann, M.-L. (2013). Temporal aspects of moral disengagement in school bullying: Crystallization or escalation? Journal of School Violence, 12(2), 193-210. doi: 10.1080/15388220.2013.766133
https://doi.org/10.1080/15388220.2013.76...
; Wang, Ryoo et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
; Wang, Yang et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). In addition, in the dynamics of bullying, boys showed lower levels of guilt when attacking their colleagues (Fitzpatrick & Bussey, 2017Fitzpatrick, S.; Bussey, K. (2017). The role of moral disengagement on social bullying in dyadic very best friendships. Journal of School Violence, 17(4), 417-429. doi: 10.1080/15388220.2017.1355810
https://doi.org/10.1080/15388220.2017.13...
; Mazzone et al., 2016Mazzone, A.; Camodeca, M.; Salmivalli, C. (2016). Interactive effects of guilt and moral disengagement on bullying, defending and outsider behavior. Journal of Moral Education, 45(4), 419-432. doi: 10.1080/03057240.2016.1216399
https://doi.org/10.1080/03057240.2016.12...
; Menesini et al., 2015Menesini, E.; Palladino, B. E.; Nocentini, A. (2015). Emotions of moral disengagement, class norms, and bullying in adolescence: A multilevel approach. Merrill-Palmer Quarterly-Journal of Developmental Psychology, 61(1), 124-143. doi: 10.13110/merrpalmquar1982.61.1.0124
https://doi.org/10.13110/merrpalmquar198...
).

Conceptual Analysis

From the conceptual analysis it was possible, first, to identify the uses or definitions of moral disengagement used in the selected studies. There was consensus and the construct in question was defined as a series of cognitive processes or mechanisms used to alleviate guilt and allow the adoption of antisocial or aggressive behaviors. The data related to the analysis level are shown in Figure 2.

Figure 2
Conceptual Framework for Moral Disengagement in Studies on School Bullying.

The model presented highlights the theoretical integration between cognitive and emotional aspects as two moral dimensions that contribute to explain individual differences that are relevant in bullying behaviors (Thornberg et al., 2015Thornberg, R.; Pozzoli, T.; Gini, G.; Jungert, T. (2015). Unique and interactive effects of moral emotions and moral disengagement on bullying and defending among school children. Elementary School Journal, 116(2), 322-337. doi: 10.1086/683985
https://doi.org/10.1086/683985...
). The propensity of students to moral disengagement was associated with an increase in bullying, considered as a group phenomenon (Fitzpatrick & Bussey, 2017Fitzpatrick, S.; Bussey, K. (2017). The role of moral disengagement on social bullying in dyadic very best friendships. Journal of School Violence, 17(4), 417-429. doi: 10.1080/15388220.2017.1355810
https://doi.org/10.1080/15388220.2017.13...
). In addition, it was possible to verify that moral disengagement is a significant predictor of bullying behavior (Travlos et al., 2018Travlos, A. K.; Tsorbatzoudis, H.; Barkoukis, V.; Douma, I. (2018). The effect of moral disengagement on bullying: Testing the moderating role of personal and social factors. Journal of Interpersonal Violence, 36(5-6), 2262-2281. doi: 10.1177/0886260518760012
https://doi.org/10.1177/0886260518760012...
). On the other hand, the concepts related to moral disengagement can be interpreted as responses that would help in the fight against bullying, being useful in interventions that focus on the clarification and analysis of values, as well as the development of empathy, solidarity and altruism (Kokkinos & Kipritsi, 2018Kokkinos, C. M.; Kipritsi, E. (2018). Bullying, moral disengagement and empathy: Exploring the links among early adolescents. Educational Psychology, 38(4), 535-552. doi: 10.1080/01443410.2017.1363376
https://doi.org/10.1080/01443410.2017.13...
; Wang, Ryoo et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
).

Meta-analysis and Evaluation of Methodological Quality

Eight studies met the required criteria (scores on bullying and moral disengagement) for the development of the meta-analysis. In one of the studies (Travlos et al., 2018Travlos, A. K.; Tsorbatzoudis, H.; Barkoukis, V.; Douma, I. (2018). The effect of moral disengagement on bullying: Testing the moderating role of personal and social factors. Journal of Interpersonal Violence, 36(5-6), 2262-2281. doi: 10.1177/0886260518760012
https://doi.org/10.1177/0886260518760012...
) this measure was not available, being obtained by transforming the value of F to value Z and then to the value of r according to the procedure described in the literature (Hughett, 2007Hughett, P. (2007). Accurate computation of the f-to-z and t-to-z transforms for large arguments. Journal of Statistical Software, 1(1), 1-5.). The results of the meta-analysis are shown in Figure 3

Figure 3
Results of the Meta-analysis on Moral Disengagement in Studies on School Bullying.

The results presented demonstrate statistically significant effects about the relationship between bullying and moral disengagement with a combined mean difference: 0.27 (95% CI: 0.18, 0.35). There was heterogeneity between the studies, with high variability (I2 = 92.7%). Thus, in the revised data set, moral disengagement was significantly associated with the adoption of bullying behaviors.

Regarding the methodological quality of the studies reviewed, assessed based on the application of the Prevalence Critical Appraisal Instrument, seven studies were classified as level A. The same occurred in the assessment of the risk of bias and quality according to the assessment items of the AXIS tool instrument. The studies evaluated as level B or of moderate quality showed weakness in the sample definition procedures in relation to the population, in the care for non-responding participants and in the determination of the criteria to establish statistical significance. The article assessed as having a high risk of bias and low methodological quality (Kollerová et al., 2014Kollerová, L.; Janošová, P.; Říčan, P. (2014). Moral disengagement from bullying: The effects of gender and classroom. New Educational Review, 37(3), 280-291.) also it had limitations in the detailed description of the data collection procedures. The results of the assessment of the methodological quality of the articles are summarized in Box 1.

Box 1
Methodological Quality Level of the Revised Studies According to the Criteria: Representativeness and Sample Size, Type of Recruitment, Presentation of the Theme and Design, Data Analysis and Internal Consistency.

The four main limitations mentioned in the reviewed studies were also summarized, as presupposed the guidelines for the construction of systematic reviews (Carvalho et al., 2019Carvalho, L. F., Pianowski, G., & Santos, M. A. (2019). Guidelines for conducting and publishing systematic reviews in Psychology. Estudos de Psicologia(Campinas ), 36, e180144. doi:10.1590/1982-0275201936e180144
https://doi.org/10.1590/1982-0275201936e...
). First, the main limitation presented refers to the cross-sectional nature of most studies (n = 11), as this design prevents causal interpretations from the results (Kokkinos & Kipritsi, 2018Kokkinos, C. M.; Kipritsi, E. (2018). Bullying, moral disengagement and empathy: Exploring the links among early adolescents. Educational Psychology, 38(4), 535-552. doi: 10.1080/01443410.2017.1363376
https://doi.org/10.1080/01443410.2017.13...
; Mazzone et al., 2016Mazzone, A.; Camodeca, M.; Salmivalli, C. (2016). Interactive effects of guilt and moral disengagement on bullying, defending and outsider behavior. Journal of Moral Education, 45(4), 419-432. doi: 10.1080/03057240.2016.1216399
https://doi.org/10.1080/03057240.2016.12...
; Menesini et al., 2015Menesini, E.; Palladino, B. E.; Nocentini, A. (2015). Emotions of moral disengagement, class norms, and bullying in adolescence: A multilevel approach. Merrill-Palmer Quarterly-Journal of Developmental Psychology, 61(1), 124-143. doi: 10.13110/merrpalmquar1982.61.1.0124
https://doi.org/10.13110/merrpalmquar198...
). Second, the size and characteristics of the studied samples were pointed out as a strong limitation in two specific cases (Orue & Calvete, 2016Orue, I.; Calvete, E. (2016). Psychopathic traits and moral disengagement interact to predict bullying and cyberbullying among adolescents. Journal of Interpersonal Violence, 34(11), 2313-2332. doi: 10.1177/0886260516660302
https://doi.org/10.1177/0886260516660302...
; Robson & Witenberg, 2013Robson, C.; Witenberg, R. T. (2013). The influence of moral disengagement, morally based self-esteem, age, and gender on traditional bullying and cyberbullying. Journal of School Violence, 12(2), 211-231. doi: 10.1080/15388220.2012.762921
https://doi.org/10.1080/15388220.2012.76...
). The third limitation refers to the use of self-report instruments, which tend to increase the occurrence of the phenomenon called social desirability (Thornberg & Jungert, 2014Thornberg, R.; Jungert, T. (2014). School bullying and the mechanisms of moral disengagement. Aggressive Behavior, 40(2), 99-108. doi: 10.1002/ab.21509
https://doi.org/10.1002/ab.21509...
; Wang, Ryoo et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
).

DISCUSSION

The revised results confirmed that moral disengagement is a strong predictor of bullying behavior, not only as a direct influence, but also as an indirect influence through the absence of feelings of empathy and guilt, inverse to moral disengagement. In summary, the data suggest that students prone to moral disengagement may practice more bullying when compared to students with greater moral engagement. Boys were more morally disengaged than girls, as well as more bullying.

Most of the studies are transversal in nature and confirmed the existence of a strong relationship between moral disengagement and bullying. These data were consistent with the longitudinal studies included in the review and are also supported by the classic literature in the area (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.). Meanwhile, with regard to the prevalence of moral disengagement and bullying behaviors among boys, these data are not considered conclusive, especially those from cross-sectional studies (Wang, Ryoo, et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). This finding indicates the need for specific future investigations on the relationship between three variables: gender, bullying and moral disengagement.

With regard to the location of the reviewed surveys, it was observed that they were limited to the countries considered to be developed. This aspect indicates that developing countries, such as Brazil or others in Latin America, should invest in investigating other dimensions, such as cognitive processes or family interactions that also influence the dynamics of bullying.

The perspective adopted regarding the concept of moral disengagement in studies on bullying has broadened the understanding of this problem. According to a holistic and functional approach, the antecedents, attributes, consequences and concepts related to this cognitive process stimulated attention to other dimensions of aggression among peers and their associated interactive or individual environmental factors, such as moral development (Bandura et al., 1996Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency. Washington: American Psychological Association.). Students who adopt bullying behaviors, in general, have difficulties in this regard - intolerance and little ability to resolve conflicts in an assertive way (Menesini et al., 2015Menesini, E.; Palladino, B. E.; Nocentini, A. (2015). Emotions of moral disengagement, class norms, and bullying in adolescence: A multilevel approach. Merrill-Palmer Quarterly-Journal of Developmental Psychology, 61(1), 124-143. doi: 10.13110/merrpalmquar1982.61.1.0124
https://doi.org/10.13110/merrpalmquar198...
).

The variable moral disengagement addressed in the studies about bullying refers to the individual vulnerability of double meaning. First, it is about flexible cognitive and emotional functioning in the face of the rupture of rules and norms of social interaction (Menesini et al., 2015Menesini, E.; Palladino, B. E.; Nocentini, A. (2015). Emotions of moral disengagement, class norms, and bullying in adolescence: A multilevel approach. Merrill-Palmer Quarterly-Journal of Developmental Psychology, 61(1), 124-143. doi: 10.13110/merrpalmquar1982.61.1.0124
https://doi.org/10.13110/merrpalmquar198...
; Thornberg & Jungert, 2014Thornberg, R.; Jungert, T. (2014). School bullying and the mechanisms of moral disengagement. Aggressive Behavior, 40(2), 99-108. doi: 10.1002/ab.21509
https://doi.org/10.1002/ab.21509...
; Thornberg et al., 2015). Second, the absence of guilt perpetuates aggressive behaviors (Mazzone et al., 2016Mazzone, A.; Camodeca, M.; Salmivalli, C. (2016). Interactive effects of guilt and moral disengagement on bullying, defending and outsider behavior. Journal of Moral Education, 45(4), 419-432. doi: 10.1080/03057240.2016.1216399
https://doi.org/10.1080/03057240.2016.12...
; Obermann, 2013Obermann, M.-L. (2013). Temporal aspects of moral disengagement in school bullying: Crystallization or escalation? Journal of School Violence, 12(2), 193-210. doi: 10.1080/15388220.2013.766133
https://doi.org/10.1080/15388220.2013.76...
; Wang, Ryoo et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). On the other hand, empathy, as a capacity to respond to the suffering of others, it is an important concept and related to moral disengagement (Hymel & Bonanno, 2014Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
https://doi.org/10.1080/00405841.2014.94...
).

A theoretical essay considered that the theory of moral disengagement provides a conceptual system for understanding the behavior of bullying among children and adolescents, mainly due to the characteristic of the aggressors of not having feelings of self-condemnation (Hymel & Bonanno, 2014Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
https://doi.org/10.1080/00405841.2014.94...
). In this sense, it is inferred that cognition and moral emotions can explain negative or positive behaviors based on individual differences (Thornberg et al., 2015Thornberg, R.; Pozzoli, T.; Gini, G.; Jungert, T. (2015). Unique and interactive effects of moral emotions and moral disengagement on bullying and defending among school children. Elementary School Journal, 116(2), 322-337. doi: 10.1086/683985
https://doi.org/10.1086/683985...
). Thus, moral disengagement would explain, to some extent, bullying behaviors.

This understanding is broadened, as the lack of moral content in school-age children and adolescents indicates that these individuals do not see their peers as worthy or of human existence in themselves (Obermann, 2013Obermann, M.-L. (2013). Temporal aspects of moral disengagement in school bullying: Crystallization or escalation? Journal of School Violence, 12(2), 193-210. doi: 10.1080/15388220.2013.766133
https://doi.org/10.1080/15388220.2013.76...
; Wang, Ryoo, et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). Theoretically, this process can be explained by different social / psychosocial mechanisms and issues important for understanding moral engagement as systemic and not just as belonging to the individual field. Even in the face of the components related to the individual’s uniqueness, one cannot ignore the interpersonal relationships that promote processes of internalization of social rules and the ways in which social interactions will happen throughout the life cycle (Lambe et al., 2017Lambe, L. J.; Hudson, C. C.; Craig, W. M.; Pepler, D. J. (2017). Does defending come with a cost? Examining the psychosocial correlates of defending behaviour among bystanders of bullying in a Canadian sample. Child Abuse & Neglect, 65, 112-123. doi: 10.1016/j.chiabu.2017.01.012
https://doi.org/10.1016/j.chiabu.2017.01...
).

However, even in the face of these results and their implications, the reviewed studies have gaps related to the diversity of information or the way they were collected. By listing the main limitations of the studies reviewed, it is intended to encourage new research that is dedicated to clarifying the analyzed variables. These recommendations aim to prevent other researchers and professionals from taking inadequate directions in the development of agendas for practices or research. In parallel to this dimension, the instruments used in the data collection processes were not able to measure perceptions or sensitively subjective aspects, a condition that could be minimized through the development of qualitative research, for example.

Regarding the analysis of the methodological quality of the studies reviewed, a certain heterogeneity was identified, as the quality of the evidence was assessed as moderate in most studies, with no serious inaccuracies being detected at this level of analysis. This result allows us to infer that the revised evidence is valid, but that additional studies are still required, with different designs and different sample sizes, so that the limitations mentioned can be mitigated or overcome. These observations are suggested by the authors of the research, which as a rule are not considered conclusive.

From the results reviewed, it appears that intervention programs aimed at reducing bullying in schools must also focus on individual issues and encourage empathic relationships based on positive behaviors such as help and positive appreciation of the other, solidarity and moral engagement (Sijtsema et al., 2014Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
https://doi.org/10.1037/a0037145...
; Wang, Ryoo, et al., 2017Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
). As boys were identified as the ones who most exhibited behaviors of moral disengagement and bullying (Kokkinos & Kipritsi, 2018Kokkinos, C. M.; Kipritsi, E. (2018). Bullying, moral disengagement and empathy: Exploring the links among early adolescents. Educational Psychology, 38(4), 535-552. doi: 10.1080/01443410.2017.1363376
https://doi.org/10.1080/01443410.2017.13...
; Wang, Ryoo, et al., 2017aWang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
https://doi.org/10.1016/j.chiabu.2017.04...
), it is suggested that these programs focus on this group. These initiatives may favor the development of pro-social behaviors, characterized by altruism and citizenship, and have a positive impact on students’ academic performance.

FINAL CONSIDERATIONS

The strength of this study lies in the unprecedented systematization of evidence on the relationship between the variables of moral disengagement and school bullying. All the results analyzed indicated that moral disengagement increases the risk of students becoming involved in bullying situations. However, these findings must be interpreted in light of its three main limitations. First, the analysis is restricted to published concepts about moral disengagement and bullying and it ignores variables that could eventually be prevalent in some of the reviewed studies (abuse and empathy, for example). Second, the search strategy with an emphasis on the terms moral disengagement and bullying may have omitted scientific contributions that did not explicitly state these variables in their titles or abstracts.

Finally, it is highlighted that the present study offers practical and research implications. In practical terms, aspects related to empathy and tolerance to diversity should be included in the agenda of anti-bullying intervention programs. At the same time, students who practice bullying should participate in initiatives that point out the negative effects of their behavior and how conflicts can be resolved in a non-violent way. For future studies it is imperative to use different research models (exploratory, longitudinal and qualitative, for example) in order to address the relationship between the disengagement construct and the occurrence of bullying.

REFERENCES

  • Bandura, A.; Barbaranelli, C.; Caprara, G. V.; Pastorelli, C. (1996). Mechanisms of moral disengagement in the exercise of moral agency Washington: American Psychological Association.
  • Carvalho, L. F., Pianowski, G., & Santos, M. A. (2019). Guidelines for conducting and publishing systematic reviews in Psychology. Estudos de Psicologia(Campinas ), 36, e180144. doi:10.1590/1982-0275201936e180144
    » https://doi.org/10.1590/1982-0275201936e180144
  • Crochík, J. L. (2016). Hierarchy, violence and bullying among students of public middle schools. Paidéia (Ribeirão Preto), 26(65), 307-315. doi: 10.1590/1982-43272665201608
    » https://doi.org/10.1590/1982-43272665201608
  • Downes, M. J.; Brennan, M. L.; Williams, H. C.; Dean, R. S. (2016). Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open, 6(12), e011458. doi: 10.1136/bmjopen-2016-011458
    » https://doi.org/10.1136/bmjopen-2016-011458
  • Fitzpatrick, S.; Bussey, K. (2017). The role of moral disengagement on social bullying in dyadic very best friendships. Journal of School Violence, 17(4), 417-429. doi: 10.1080/15388220.2017.1355810
    » https://doi.org/10.1080/15388220.2017.1355810
  • Fernandes, G., Yunes, M. A. M., & Finkler, L. (2020). The social networks of adolescent victims of domestic violence and bullying. Paidéia (Ribeirão Preto), 30, e3007. doi: 10.1590/1982-4327e3007
    » https://doi.org/10.1590/1982-4327e3007
  • Higgins, J. P. T.; Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539-1558. doi: 10.1002/sim.1186
    » https://doi.org/10.1002/sim.1186
  • Hughett, P. (2007). Accurate computation of the f-to-z and t-to-z transforms for large arguments. Journal of Statistical Software, 1(1), 1-5.
  • Hymel, S.; Bonanno, R. A. (2014). Moral disengagement processes in bullying. Theory into Practice, 53(4), 278-285. doi: 10.1080/00405841.2014.947219
    » https://doi.org/10.1080/00405841.2014.947219
  • Kokkinos, C. M.; Kipritsi, E. (2018). Bullying, moral disengagement and empathy: Exploring the links among early adolescents. Educational Psychology, 38(4), 535-552. doi: 10.1080/01443410.2017.1363376
    » https://doi.org/10.1080/01443410.2017.1363376
  • Kollerová, L.; Janošová, P.; Říčan, P. (2014). Moral disengagement from bullying: The effects of gender and classroom. New Educational Review, 37(3), 280-291.
  • Lambe, L. J.; Hudson, C. C.; Craig, W. M.; Pepler, D. J. (2017). Does defending come with a cost? Examining the psychosocial correlates of defending behaviour among bystanders of bullying in a Canadian sample. Child Abuse & Neglect, 65, 112-123. doi: 10.1016/j.chiabu.2017.01.012
    » https://doi.org/10.1016/j.chiabu.2017.01.012
  • Mazzone, A.; Camodeca, M.; Salmivalli, C. (2016). Interactive effects of guilt and moral disengagement on bullying, defending and outsider behavior. Journal of Moral Education, 45(4), 419-432. doi: 10.1080/03057240.2016.1216399
    » https://doi.org/10.1080/03057240.2016.1216399
  • Menesini, E.; Palladino, B. E.; Nocentini, A. (2015). Emotions of moral disengagement, class norms, and bullying in adolescence: A multilevel approach. Merrill-Palmer Quarterly-Journal of Developmental Psychology, 61(1), 124-143. doi: 10.13110/merrpalmquar1982.61.1.0124
    » https://doi.org/10.13110/merrpalmquar1982.61.1.0124
  • Munn, Z.; Moola, S.; Riitano, D.; Lisy, K. (2014). The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. International Journal of Health Policy and Management, 3(3), 123-128. doi: 10.15171/ijhpm.2014.71
    » https://doi.org/10.15171/ijhpm.2014.71
  • Obermann, M.-L. (2013). Temporal aspects of moral disengagement in school bullying: Crystallization or escalation? Journal of School Violence, 12(2), 193-210. doi: 10.1080/15388220.2013.766133
    » https://doi.org/10.1080/15388220.2013.766133
  • Olweus, D. (2013). School bullying: Development and some important challenges. Annual Review of Clinical Psychology, 9(1), 751-780. doi: 10.1146/annurev-clinpsy-050212-185516
    » https://doi.org/10.1146/annurev-clinpsy-050212-185516
  • Orue, I.; Calvete, E. (2016). Psychopathic traits and moral disengagement interact to predict bullying and cyberbullying among adolescents. Journal of Interpersonal Violence, 34(11), 2313-2332. doi: 10.1177/0886260516660302
    » https://doi.org/10.1177/0886260516660302
  • Robson, C.; Witenberg, R. T. (2013). The influence of moral disengagement, morally based self-esteem, age, and gender on traditional bullying and cyberbullying. Journal of School Violence, 12(2), 211-231. doi: 10.1080/15388220.2012.762921
    » https://doi.org/10.1080/15388220.2012.762921
  • Sijtsema, J. J.; Rambaran, J. A.; Caravita, S. C. S.; Gini, G. (2014). Friendship selection and influence in bullying and defending: Effects of moral disengagement. Developmental Psychology, 50(8), 2093-2104. doi: 10.1037/a0037145
    » https://doi.org/10.1037/a0037145
  • Thornberg, R.; Jungert, T. (2014). School bullying and the mechanisms of moral disengagement. Aggressive Behavior, 40(2), 99-108. doi: 10.1002/ab.21509
    » https://doi.org/10.1002/ab.21509
  • Thornberg, R.; Pozzoli, T.; Gini, G.; Jungert, T. (2015). Unique and interactive effects of moral emotions and moral disengagement on bullying and defending among school children. Elementary School Journal, 116(2), 322-337. doi: 10.1086/683985
    » https://doi.org/10.1086/683985
  • Travlos, A. K.; Tsorbatzoudis, H.; Barkoukis, V.; Douma, I. (2018). The effect of moral disengagement on bullying: Testing the moderating role of personal and social factors. Journal of Interpersonal Violence, 36(5-6), 2262-2281. doi: 10.1177/0886260518760012
    » https://doi.org/10.1177/0886260518760012
  • Van Noorden, T. H. J.; Haselager, G. J. T.; Cillessen, A. H. N.; Bukowski, W. M. (2014). Dehumanization in children: The link with moral disengagement in bullying and victimization. Aggressive Behavior, 40(4), 320-328. doi: 10.1002/ab.21522
    » https://doi.org/10.1002/ab.21522
  • Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor Package. Journal of Statistical Software, 1(3), 1-48. doi: 10.18637/jss.v036.i03
    » https://doi.org/10.18637/jss.v036.i03
  • Walker, L. O.; Avant, K. C. (2018). Strategies for theory construction in nursing New York: Pearson.
  • Wang, C. X.; Ryoo, J. H.; Swearer, S. M.; Turner, R.; Goldberg, T. S. (2017). Longitudinal relationships between bullying and moral disengagement among adolescents. Journal of Youth and Adolescence, 46(6), 1304-1317. doi: 10.1007/s10964-016-0577-0
    » https://doi.org/10.1007/s10964-016-0577-0
  • Wang, X. C.; Yang, L.; Gao, L.; Yang, J. P.; Lei, L.; Wang, C. Y. (2017). Childhood maltreatment and Chinese adolescents’ bullying and defending: The mediating role of moral disengagement. Child Abuse & Neglect, 69, 134-144. doi: 10.1016/j.chiabu.2017.04.016
    » https://doi.org/10.1016/j.chiabu.2017.04.016
  • Zequinão, M. A., Medeiros, P., Silva, J. L., Pereira, B. O., & Cardoso, F. L. (2020). Sociometric status of participants involved in school bullying. Paidéia(Ribeirão Preto), 30, e3011. doi:10.1590/1982-4327e3011
    » https://doi.org/10.1590/1982-4327e3011
  • This study was carried out with support by Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - financing code 001. The first, penultimate and last authors arescholarship researchers of productivity in research of Conselho Nacional de Desenvolvimento Científico e Tecnológico - CNPq.
  • This paper was translated from Portuguese by Ana Maria Pereira Dionísio.

Publication Dates

  • Publication in this collection
    11 Aug 2021
  • Date of issue
    2021

History

  • Received
    01 May 2019
  • Accepted
    08 Sept 2020
Associação Brasileira de Psicologia Escolar e Educacional (ABRAPEE) Associação Brasileira de Psicologia Escolar e Educacional (ABRAPEE), Rua Mirassol, 46 - Vila Mariana , CEP 04044-010 São Paulo - SP - Brasil , Fone/Fax (11) 96900-6678 - São Paulo - SP - Brazil
E-mail: revista@abrapee.psc.br