THE REVIEWER: ANONYMOUS PROTAGONIST OF QUALITY

In my tenure in the editorship of the Direito GV Law Review, I have had the very interesting opportunity of getting to know the inner workings of the system of double-blind peer review. Among the several challenges of an editor, one of the most crucial is to ensure the quality and speed of the functioning of this system in relation to the texts submitted to the journal. On the one side, we are interested that the scientific community have access as quickly as possible to the articles that record progresses in the field to which the periodical is dedicated; on the other side, the decision to publish must reflect an impartial review that concludes for the quality of the text in several senses. I have noticed that promoting these two noble objectives simultaneously is a great challenge for the Brazilian legal community.

The Coordination for the Improvement of Higher Educational Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES) - the agency that regulates graduate programs in Brazil - established the double-blind peer review as the mechanism for the review of articles published in scientific periodicals in the area of law in this country.¹ This does not mean that CAPES compels periodicals to adopt this mechanism. But it means that, if periodicals do not adopt it, they will be negatively evaluated by CAPES according to the Qualis criteria.

The double-blind peer review system is not the sole model available or practiced in academia. There are other models.² Each one of them has its merits and difficulties. For the good functioning of each one of them, one must understand their inner logic and the points where they might present weaknesses. Given that the system of double-blind peer review is ubiquitous among the scientific periodicals in the area of law in Brazil, it is important to analyze how it works.

The idea behind this review system is that the scientific community with which the article dialogues is entrusted with reading the submitted article and freely engaging with it. It is the scientific community that will decide if the article has potential for publication, if it should be perfected or not as a condition for publication, and, in the end of the review process, if the article should be published. This is why it is said that the review is done by peers: the participants in the scientific community exercise themselves the quality control over the texts submitted for publication. In other words, sometimes the members of the scientific community are authors and, in other times, they are reviewers.

The review is doubly blind because it guarantees, in this way, that the reviewers do not know the identity of the author of the article while they perform the review of the text, and the author of the article does not know the identity of the reviewers that have evaluated her or his text.

Of course, in order to have anonymity between author and reviewers, there must be someone to manage the process. This is the work of the editor of the periodical and her or his team. They are the ones who will guarantee the anonymization of the articles submitted and of the reviews. Once it is guaranteed that the author cannot be identified in the text, the editor forwards the article for reviewers so that they read it and send back a review to the editor. The reviewer may suggest the rejection of the article, the approval of it, or may also indicate that the article has to be modified as a condition for publication. Based on the content of the received reviews, the editor decides among proceeding with the review process - by giving feedback to the author with suggestions to modify the article -, publishing the article without modification, or rejecting the article. Even if the editor’s decision is discretionary, i.e., it is not bound to what the reviews say, the idea is that the editor may base her or his decision - and justify this decision to the author of the text - mainly on the content of the reviews. Whatever the editor decides, she or he usually sends her or his decision to the author together with the content of the reviews that helped to make the decision.

At this stage of reflection, I think it is important to notice that, even if the editor follows all the steps described in the previous paragraph - even if she or he fulfills all the procedures imposed by the rules of double-blind peer review - she or he cannot guarantee the quality of the interlocution between reviewers and articles/authors. There is nothing in the rules of double-blind peer review that might constrain the reviewers to well justify their engagement with the reviewed articles. The rules guarantee anonymity, which in principle allows for a freer and less biased engagement of the reviewers with the texts. However, the reviewer must choose to exercise this freedom of engagement. The system assumes the collective interest of the scientific community in establishing and keeping a robust and productive interlocution in the realm of the review process of articles for scientific periodicals. The system assumes that the community values this interlocution and understands its importance for the development of the field.

In my tenure in the editorship of the Direito GV Law Review, this assumption, however, did not always verify as a fact. The lack of quality engagement by the reviewers with the texts submitted to their review negatively impacts the process of perfecting and selecting the articles for publication and, in the end, the substantive quality of the published texts. This happens because the justification of the reviewer’s judgment has at least two important purposes. Firstly, it explains to the editor the reasons of the merits and demerits that the reviewer identifies in the article. As mentioned before, even though the editor is not bound by the reviews, she or he counts on the contribution of the academic community to take positions as well founded as possible. Journal editors usually have academic experience. However, it is not reasonable to expect that the editor and her or his team are experienced in all fields of research contemplated by the periodical under their management, even more so in the case of generalist periodicals. Therefore, editors make better decisions - and give better founded feedback to the authors - when they can have access to the reasons why experienced readers found problems and virtues in the texts submitted for publication. Secondly, a well-founded review indicates to the author how an experienced reader in the field identifies strengths and weaknesses in the text and in the underlying research. The review process of articles consists in a very important opportunity of free interlocution between researchers. Authors can benefit from the attentive reading of their article by qualified readers and, therefore, can perfect their work in an important way. The review process is a privileged chance of learning.

I present in the sequence some personal reflections that may contribute to facing the difficulties surrounding the blind peer review in the field of law in Brazil.

Scarce Incentives

In several occasions, the editorial team and I faced the difficulty of identifying reviewers that were experienced in the field of problems and structuring methodologies of the submitted article, and that were available to do the review work. Oftentimes, we had to send more than three invitations until two reviewers accepted to review the article. This difficulty slows down the review process of the articles. In order to not compromise the quality of the review process, we often had to sacrifice speed in making the articles available to readers.

In many other situations, the reviewer accepts to review the article, but the review comes back “empty”. An empty review is a barely founded - or not founded at all - review. Usually review forms are divided in two parts: objective questions and one or more fields open to qualitative explanation. In empty reviews, the reviewers usually answer the objective requirements of the review. For example, they choose between the options whether the article has or has not brought original contributions (there may be intermediate options or not, depending on the review form). They also indicate, in a further example, whether the abstract corresponds or not to the content of the article. Lastly, they opt between rejection, approval or suggestion to modify the article. However, the review is almost totally deprived of justification for the selected options in the objective part. In some cases, the review is even sent without the reviewer writing a single line to explain her or his position. In those cases, the delay in the review process of the article is even greater because, if the editor wants to uphold the quality of the peer review process, she or he will have to invite a new reviewer for an effective review to take place.

Those difficulties are reflected in a set of interesting data relative to the review flux of the Direito GV Law Review. Between 1st September 2019 and 14th February 2020, which roughly corresponds to the period I spent involved with the activities of the journal, 68.89% of the articles in relation to which invitations were sent for reviews gave rise to extra invitations because of some (or more than one) of the following reasons: (i) lack of answer from the reviewer in relation to the invitation; (ii) rejection of the invitation by the reviewer; or (iii) lack of delivery of review after the invitation was accepted. It is important to stress that articles that received extra invitations because of the insufficiency of the provided reviews were not taken into account.

The figures related to the period when I was involved with the activities of the editorship do not constitute an isolated event, but rather reflect a pattern of what has been happening for some time. The following Graph 1 shows the percentages of the articles that gave rise to extra invitations for some of the described reasons in the years of 2016,³ 2017, 2018, and 2019 and in the first months of 2020.⁴ Comparing the last four years, 2017 presented the smallest percentage of articles that received extra invitations, reaching 64.9%, while 2018 presented the highest percentage, reaching 71.1%. The first three months of 2020 showed a significant increase in this percentage - that went above 86% -, but it is still not possible to define if this increase will translate into a tendency for the year 2020.

GRAPH 1
Percentage of manuscripts with extra invitations by year (March 2016 to March 2020)

These data are not conducive to a straightforward denunciation of our attitude as academics wearing the hat of periodical reviewers. Academic life in Brazil can be very demanding. Brazilian academics are usually torn between preparing and giving classes (and everything that the conduction of a course involves), research activities and often also the management of the institutions where they work. To this must be added the organization of and participation in events, the orientation of undergraduate, master’s and PhD students, the participation in undergraduate and graduate examining committees, and the participation in faculty admission committees.

In sum, the academic professional works a lot. And, while this may not be a professional sector that operates with proper quantifiable goals, such as the ones set in some commercial sectors, it is possible to say that the academic professional does have specific objectives to pursue. Graduate programs are assessed periodically, and an important part of this assessment is based on the quantification of faculty production. This assessment has important consequences for the academic institutions and the work conditions it provides for its professionals.

One of the factors that academics must take into consideration when reflecting on how to prioritize their numerous activities is to guarantee that those that will impact positively the assessment of the institution where they work will get done. In this reflection, the elaboration of reviews for scientific periodicals has inevitably a lesser weight, because, according to the rules of the CAPES, the drafting of reviews is not count as relevant academic production. It is true that the Lattes platform has a field to insert the review activity for periodicals, but as of now this work is not graded in the assessment of the area of law by the CAPES.

One of the possibilities to increase the incentive for the elaboration of good reviews would be precisely for the CAPES to value this activity in the assessment of graduate programs. Grading reviews in itself would create an incentive for more academics to accept invitations to draft reviews and send them to the editorial teams of periodicals. It is true that, in the short term, this scoring rule of reviews would not solve by itself the problem of the elaboration of good reviews. This is because the rule would take into account the quantity of drafted reviews, but would not assess their quality. In the medium or long term, however, the rule might also foster the quality of reviews by means of a reputational mechanism: an editor committed to the quality of reviews who once receives a badly elaborated review will shun asking another review from the same reviewer. That, in a scenario where reviews are valued, would be prejudicial to the reviewer, for it would reduce the amount of invitations from editors.

Revision of the CAPES metrics in order to grade reviews is not the single logic of incentive guiding academic work. Peer recognition is also important. Nevertheless, in the case of review drafting in the system of double-blind peer review, anonymity impairs the creation of recognition and prestige mechanisms. The authorship of specific reviews cannot be published, under penalty of compromising the anonymous nature of the process. A possible solution would be the creation of periodic lists of the best reviewers by journal editors. The fact that a researcher drafted a review for a particular periodical is of now publishable in a proper field of the Lattes curriculum. A list with the best reviewers of the year would not increase the level of publicity already existing in relation to the participation of academics in the drafting of reviews for particular periodicals. And it could be an interesting mechanism to value the effort of quality interlocution in the review process of articles for legal periodicals in Brazil.

Review Culture

The observed problems are neither restricted to the low availability of reviewers nor to the short time and attention dedicated by some reviewers to the reviews. In some cases, even reviews consisting of long commentaries fail to reach the objective of a good review.

Sometimes reviewers delve into very specific criticism of particular words used by the authors. Other times, they meticulously point out grammatical or spelling mistakes. There are cases of long criticisms to the author for failing to cite a particular work. Those might be useful contributions, but they do not address what is central in the article. The editor and the author expect the reviewer to give feedback regarding the structural aspects of the academic work.

Researchers in general are familiar with the criteria that define a good academic work. As covered in another editorial (BARBIERI and PASQUA, 2018), the work must be built around one or more clearly formulated questions. The questions must be relevant for the field of study where they belong. The research methodology and/or argumentation must be pertinent to tackle the work's guiding question or questions. The study must bring some original contribution to the field, i.e., it must show relevance taking into account the state of the debate and the investigations in the area. If these are central characteristics of a good academic work, a good review must present an evaluation precisely about these issues. Accordingly, a good review must address the following topics:

Make a judgment about the quality of the question or questions tackled in the work. Is the question clear? Is the question original and relevant for the field of study to which the work belongs?

Make a judgment about the appropriateness of the research methodology and argumentation to tackle the guiding question of the work. Are the investigation steps adequate to solve the issues put by the research problem?

Make a judgment about the structure of the work. Do the parts of the work establish an adequate relation among them? Do they develop in a logic manner? Does the argumentation developed in the work justify the conclusions?

These are not the only important topics. One must also check, for example, if the employed references are sufficient and relevant. The text must be well written. However, the substantive contributions of the work and the quality of its structure are what, more fundamentally, define if the work is pertinent to its field. The review must take a justified stance about what is more relevant.

An empty review that negatively evaluates the article is little useful both for the editor and the author. When the editor receives an empty review, she or he has little support to decide if it is a case of straightforward rejection of the article, or if the reviewer identifies potential in the text. With an empty review in hand, the editor has few possibilities to promote a productive interlocution between the reviewer and the author. As for the author, she or he loses the opportunity to receive substantive feedback about her or his work. This feedback is important both in the cases where the editor decides on the direct rejection of the article and in the cases where she or he decides to proceed with the review.

A substantive review is crucial even if the reviewer makes a positive review of the article. Oftentimes, reviewers understand that, when they estimate that the article has a good quality and does not need many modifications, the review must only say that the article is good and should be approved. For the editor, however, this attitude is insufficient, among other reasons, because the other reviewer might have a different position. The justification of the reviewer’s judgment allows the editor to relate the strengths that the reviewer sees in the article to the periodical’s academic scope, and it also offers the editor elements to compare the opinion of one reviewer with that of another one. For the author, a positive and justified review of the text is also useful: it helps her or him to understand - from the point of view of a qualified reader - what she or he is getting right and why. In sum, justified reviews, pointing either to a favorable or unfavorable judgment of the article, are useful for editors and authors. More than that, they are crucial to perfect the academic community’s conception of quality work and to promote this quality within the research in the area.

Conclusion

The work of the reviewer in the system of double-blind peer review is practically invisible. However, it constitutes one of the central opportunities of free interlocution among the members of the academic community. The anonimity of the reviewer is important to preserve the logic of double-blind peer review. However, strategies such as the modification of the CAPES rules to start valuing reviews in the assessment of academic production and reputational benefits for those who drafted with good reviews to periodicals might create incentives for the elaboration of good reviews.

Well drafted reviews aim for genuine exchanges about the central elements of the reviewed articles. These reviews greatly enhance the reflection on the conception of quality itself and the promotion of this quality in each area of study. The exchange between authors and reviewers strengthens the consistency of articles and generally raises the quality of the academic production. Almost every academic is, at some point, called upon to be also a reviewer. Beside the duly recognized contributions that they offer as authors, academics must be aware of how much they also contribute to the quality work in their area when they serve anonymously on the other side of the counter.

• 1
  The Quadriennial Review Report 2017 (CAPES, 2017, p. 12-13), that evaluated graduate law programs in the quadrennium 2013-2016, indicates the adoption of the double-blind peer review as the minimum requirement to classify a periodical in the Qualis system.
• 2
  Ross-Hellauer (2017) conducted a study to investigate the several meanings of the notion of open peer review. The study asserts that open peer review is an umbrella notion that designates different innovations in relation to one or more elements characteristic of the blind peer review, innovations that are in line with the wider movement of open science. Ross-Hellauer identified 22 models designated by the notion of open peer review, consisting of distinct combinations of one or more of the following characteristics: (i) openness of the identity of the reviewers for the authors combined with openness of the identity of the authors for the reviewers; (ii) publication of the review reports with the articles, so that the reports can also be quoted and referenced; (iii) possibility of the community in general contributing to the review process; (iv) permission and stimulus of direct discussion between reviewers and/or between reviewers and authors; (v) public availability of the manuscripts immediately after their submission for review; (vi) openness of the final version of the articles for public commentary; and (vii) review offered by an entity distinct from that of the publication.
• 3
  Data for 2016 correspond to review invitations sent between March and December of that year. This is because in March 2016 Direito GV Law Review changed its peer-review management system. Data for January and February 2016, therefore, were not taken into account.
• 4
  Data for 2020 correspond to review invitations sent between January and March of this year.

REFERÊNCIAS

• BARBIERI, Catarina Helena Cortada; PASQUA, Juliana Silva. O que buscamos em um artigo científico? Revista Direito GV, v. 14, n. 3, São Paulo, p. 810-813, set./dez. 2018. Disponível em: https://www.scielo.br/scielo.php?pid=S1808-24322018000300810&script=sci_arttext Acesso em: 11 maio 2020.
  » https://www.scielo.br/scielo.php?pid=S1808-24322018000300810&script=sci_arttext
• COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR (CAPES). Relatório da Avaliação Quadrienal 2017 Direito, 2017. Disponível em: https://capes.gov.br/images/documentos/Relatorios_quadrienal_2017/20122017-Direito_relatorio-de-avaliacao-quadrienal-2017_final.pdf Acesso em: 11 maio 2020.
  » https://capes.gov.br/images/documentos/Relatorios_quadrienal_2017/20122017-Direito_relatorio-de-avaliacao-quadrienal-2017_final.pdf
• ROSS-HELLAUER, Tony. What is open peer review? A systematic review.F1000Research, v. 6, p. 588, 2017. Disponível em: https://doi.org/10.12688/f1000research.11369.1 Acesso em: 11 maio 2020.
  » https://doi.org/10.12688/f1000research.11369.1

Publication Dates

• Publication in this collection
  11 Sept 2020
• Date of issue
  2020