EVALUATIONS OF SCIENTIFIC PRODUCTIONS: CHALLENGES AND MOTIVATIONS OF EDITORS AND REVIEWERS

Farias, Maria Giovanna Guedes; Santos, Gildenir Carolino

doi:10.5007/1518-2924.2023.e92858/54023

Maria Giovanna Guedes Farias

Conception and preparation of the manuscript

Data collection

Data analysis

Review and approval

Universidade Federal do Ceará, Department of Information Sciences, Fortaleza, CE, Brazil. mgiovannaguedes@gmail.com

https://orcid.org/0000-0002-2690-3350

Gildenir Carolino Santos

Conception and preparation of the manuscript

Data collection

Discussion of results

Review and approval

Universidade Estadual de Campinas, Library System, Campinas, SP, Brazil. gilldenir@unicamp.br

https://orcid.org/0000-0002-4375-6815

CONFLICT OF INTEREST

Not applicable

EDITORS

Edgar Bisset Alvarez, Ana Clara Cândido, Patrícia Neubert, Genilson Geraldo, Mayara Madeira Trevisol, Jônatas Edison da Silva, Camila Letícia Melo Furtado and Beatriz Tarré Alonso.

Evaluators	Quantity	Percentage
Range 41 to 55	17	41%
Range 58 to 68	11	27%
Range 24 to 40	9	22%
Range 71 to 75	4	10%
Total respondents	41

Time of performance	Quantity	Percentage
l to 2 years	2	5%
2 to 3 years	1	3%
3 to 4 years	3	7%
4 to 5 years	5	12%
Over de 5 years	30	73%
Total respondents	41

Nº Journals	Quantity	Percentage
l	3	7%
2	4	10%
3	6	15%
4	3	7%
Over de 5	25	61%
Total respondents	41

Editors	Quantity	Percentage
Range 33 to 40	4	10%
Range 4l to 49	13	32%
Range 50 to 59	12	30%
Range 60 to 66	6	15%
Range 71 to 8l	5	13%
Total respondents	40

Time of performance	Quantity	Percentage
1 to 2 years	2	5%
2 to 3 years	5	12%
3 to 4 years	5	12%
4 to 5 years	3	8%
Over de 5 years	25	63%
Total respondents	40

It's important to give this feedback to the authors. It's a commitment of those in academic careers to collaborate with research. If everyone did a bit, there wouldn't be so many problems and wear and tear. In addition, the number of evaluations could be reduced. (A15)

Yes, because it's something I enjoy doing, it's a pleasure to contribute to scientific communication. But if there were some form of reward, it would certainly increase my motivation even more. (A13)

The motivation is intellectual, in the learning that an evaluation offers the evaluator and the possibility of internalizing this learning in their research practice, in addition to the possible possibility of projection at curriculum level. (A19)

From the point of view of scientific academic altruism, promoting science and encouraging researchers, yes. But I feel that other motivators are lacking (more traffic to publish in journals, slow response from some of them, bureaucracy that prevents progress, as well as the accumulation of requests over several periods). (A22)

Yes. I understand that it's a partnership, because as an author myself, I want my articles to be carefully evaluated. The basis of science is peer review. (A26)

The invitation to an evaluation of this nature denotes acceptance of your expertise in the subjects and of the titles you hold as recognition of your academic and scientific importance to the areas for which you collaborate. (A29).

Yes and no. Yes, because it's a way of entering the educational field. No, because this is just another unpaid job that teachers do. (A30)

I always feel motivated to evaluate and collaborate with scientific communication and the journal. In addition, I value the work of the reviewer, as I am a journal editor, and I know how much we need reviewers to boost the editorial process. (A33)

First of all, it's a way of finding out about the various research projects underway in my area of research, although at no time have, I used this knowledge for my own benefit. But it is a way of keeping up to date. I like evaluating because it's something I can do to improve publications and teach authors how to improve their writing and organization of results. (A38)

In some situations where the articles are based on statistical and even empirical data, I believe that incorporating the scientific data that underpins the research has increased the demands on the evaluators, since they need to check that the results are correctly represented. In this case, it requires more time and competence in the subject being assessed. (A1)

The constant updating of the themes made available. (A3)

Not using Publons and/or ReviewerCredits accreditors (A6)

Decide whether or not the article should be rejected or if it can be redone and resubmitted. Another precaution is to know how to choose your words so as not to discourage or attack the author. Knowing how to make suggestions within the theoretical and methodological framework chosen by the author. This requires empathy and study on the part of the referee. (A8)

Finding the courage to evaluate, to give a quick and quality response, knowing that our own submissions take months to be evaluated. (A11)

It takes a long time to send the evaluation voucher. It's not uncommon for me to have to ask, and that's very bad." (A12)

I'm speaking as an editor - it's a very disparate quality of evaluation. Sometimes they are opposite, which means more commitment, time and delay. Another problem is finding evaluators who agree to take part. (A16)

Aiming to evaluate without belittling the evaluated, paying attention to the characteristics of the submitted manuscript and, at the same time, looking for ways to contribute to its improvement. (A19)

Being fair and understanding that sometimes flaws can be pointed out, fixed and the writing still has merit for dealing with the subject in a way that contributes to the area of knowledge to which it was submitted. (A20)

There are still many evaluators who don't have the maturity to give complete opinions that really state the reason for acceptance or rejection. (A27)

At the moment, the biggest challenge is self-plagiarism. There are submissions that have aspects of "rehashed" work, which forces me - as a reviewer - to read work by that author in order to copyedit that work with what the author has submitted. (A29).

Evaluating an article presents the same difficulties for the evaluator as those involved in evaluating any academic-scientific work. Demanding linguistic adequacy, scientific rigor, originality and obedience to the technical standards of reference records and formatting is becoming almost impossible in the editorial space of Brazilian education. Evaluating work that fails to meet these requirements is always very difficult. (A30)

The illegibility of texts with problems of the author's mental organization; problems of plagiarism; unfinished research that presents poorly-written results in order to say that it has been completed; texts from areas of knowledge that are not in the evaluator's domain. (A38)

Lack of normative standardization in line with ABNT. There are different standards for each journal. (A40)

I feel there is a lack of natural incorporation as a reviewer into the Scientific Committee of journals after a considerable number of reviews. It is notorious that some journals add evaluators due to the representation of the curriculum, seeking to bring greater credibility to the journal. This model is recurrent in journals seeking internationalization and is repeated even among the names of researchers at a national level. This practice seems understandable and even necessary to me, but I think it's important for journals to adopt a balance between the quantity and quality of reviews, in order to incorporate new members of the scientific committees, i.e. regular reviewers who make a significant contribution. I don't know if any journals already do this, but I think it would be a satisfactory way of rewarding reviewers. (A1)

No. Most Brazilian journals are not professionalized, so volunteering not only increases the risk of thematic or theoretical conflicts of interest on the part of editors, it can offer nothing to technical work that could not be volunteered. What's more, Sucupira doesn't consider an opinion to be noteworthy. Technical work is worth almost nothing in terms of teaching output. (A8)

I'm in favor of training in evaluation, whether through Anped, Coneped, courses (A25)

Publication of a newsletter at the end of each editorial year with a body of evaluators; publication of the average evaluation time at the end of the cycle. Obviously, always preserving the impersonality of the journal in terms of the evaluation process. That's the basics (A31)

In addition to listing the evaluators who collaborate the most, evaluate with quality and meet deadlines, perhaps highlighting the 3 most prominent evaluators in this regard in the Official Report. (A33)

Sending messages to the evaluators to follow up the process of accepting or not accepting articles, so that they know what has been used and what has not; tutorials on evaluation style; good evaluation practices for the journal. (A38)

Problems	Quantity	Percentage
Conflict of interest	2	3%
Incoherent opinions	22	28%
Crudeness	7	9%
Delay in evaluations	36	46'%
Others	11	14 %
Total	78	100%

Tempo	Quantity	Percentage
Other	17	39%
One month	19	43%
Four weeks	3	7%
Three weeks	4	4%
Two weeks	3	7%
Total	46	100%

Tempo	Quantity	Percentage
Others	17	17%
Allow more time to evaluate the manuscript	26	40%
Contact to reviewer via WhatsApp	7	11%
Sending a direct message to the reviewer convincing to stay on the list of reviewers	21	32%
Total	65

Ranking	Indicadores	Quantity
1°	Adoption of institutional policy	8
2°	Academic Recognition	8
3°	Editorial Policy	5
4°	Creation of a system of accreditation	3

Suggested actions	Indicators
Motivation policies for evaluators to be adopted by publishers. (E3)	Adoption of institutional policy
It would be important if these participations were taken into account in the institutions where the researchers work. (E4)
In the case of private HEIs, organize institutional publishing labels and charge the price of books to be published administered by the HEI treasury so that evaluations are remunerated. (E14)
It's a very complex issue because we generally rely on the kindness of professionals for this task. It is known that article evaluation is more of a demand and there is no remuneration. Perhaps it could be scored more highly in assessments such as RAAD and in the lattes themselves, being considered a relevant task that contributes to the program's score, if it is a professor linked to a postgraduate program. (E17)
Strengthen symbolic remuneration for evaluation (E18)
It was discussed at the last CAPES Mid-Term Seminar (2019) that CAPES area coordinators should sensitize colleagues to contribute to the area's sister journals. I think this should be reinforced." (E26)
Policies for valuing journals, editors and evaluators (E28)
There should be an exchange of information between editors in the sense of sharing information. For example: a list of reviewers with their contacts, etc. I think IBICT could promote this integration. (E39)
National base of evaluators (E8)	Creation of an accreditation system for assessors
Searching databases, searching for themes-researchers, requesting nominations for evaluators from the organizers of thematic sections, from well-known researchers. (E8)
Every year, we turn to the list of the Association for Research and Postgraduate Studies in Information Science - ANCIB - for new contacts and to colleagues in the area of Communication and Information, through e-mails, with a view to inviting colleagues to be referees for the journal. (E26)
I've thought about structuring a formal report for them to fill in so that they have more evaluation criteria that aren't based on their practices alone. We're still evaluating it with the team." (E31)	Editorial policy
I do almost all of the above, but I don't think it's innovative, I think it's part of the editor's role to have more direct contact with the reviewer. Thanking them for their opinions, giving them clear guidelines on how to prepare the opinion, passing on a statement. Unofficially, we give a slight precedence in the evaluation to articles by reviewers who are always collaborating. Precedence in evaluation, but a small one, due to the demands of the journal, and in publication in aheadof print AOP. When it comes to publication in AOP, i.e. after it has been evaluated and approved for publication, I always ask when the author, who is also an evaluator, needs the text to be published in SciELO. (E9)
The Journal's Editorial Board always invites new evaluators and adopts the Ad-Hoc Evaluator. (E27)
For new reviewers, we make personalized invitations to authors who have published with us and are not yet part of the reviewer team. There are also contacts made at events. (E34)
The Editorial Board is periodically re-evaluated and only people who actively contribute are retained. Unfortunately, this does little to motivate reviewers. (E35)
Evaluation training for new doctors; mini-courses at scientific events for evaluation training. (E23)	Academic recognition
We commend the work of the reviewer (who has already contributed his work to the Journal) and say that the opinion of such a reviewer is very important for improving the manuscript and for science itself. (E5)
Make the relationship more personalized; value the evaluators with personalized messages. (E12)
Invitation to organize dossiers and/or participate in thematic dossiers (E20)
Thank-you events; publicly visible lists of the most active evaluators by subject, for example; list of the best-rated evaluators (badge/star/patent system or similar). (E33)
I think it would be interesting for the journal to make its evaluators aware of the importance of their work for scientific communication and for the journal, through messages at certain times; for new evaluators, to open up spaces for new masters and doctors to work as evaluators. (E34)
Existence of CAPES and CNPq public notices that can pay for evaluations (E34)
We are always looking for new solutions to guarantee the permanence and activity of the evaluators, as well as constantly renewing the available list. However, we don't have any other options at the moment to improve the performance of the evaluators. We don't use the telephone to communicate with evaluators, as we consider it too invasive. It is an inherent activity of research, but the excess of competing publications overloads the evaluators. We are considering holding regular events to thank everyone who works on producing our publications, especially the authors and reviewers. (E33)

[1] CONFLICT OF INTEREST

Not applicable

Brasil

Brasil

EVALUATIONS OF SCIENTIFIC PRODUCTIONS: CHALLENGES AND MOTIVATIONS OF EDITORS AND REVIEWERS

ABSTRACT

Objective:

Methodology:

Results:

Conclusion: