Simulation of the New Proposed Method by CAPES for the Qualis 2017-2020 Classification of the Brazilian Archives of Biology and Technology

Abstract

This article aimed to simulate the new method proposed by the Brazilian Coordination for the Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, CAPES) for the classification of Qualis 2017-2020 by tracing the path that led to the B3 stratification attributed to the journal Brazilian Archives of Biology and Technology (BABT). Data collection was based on the evaluation of articles published between 2017 and 2018 to obtain the Lattes curricula of the author(s). Other goals were to verify their participation in stricto sensu postgraduate programs recognized by CAPES; to identify the program’s evaluation area according to CAPES; and to establish the association of authors/co-authors with their respective countries. Overall 199 articles were identified, with 897 authors/co-authors, 359 of which are associated with Brazilian institutions. A participation of at least one Brazilian author/co-author could be verified in 69 articles, out of which 60 had at least one professor affiliated to a stricto sensu postgraduate program recognized by CAPES. There was professors linked to 26 of the 49 CAPES evaluation areas, whereby the highest number of publications were Food Science and Agrarian Sciences I, both with 16 occurrences each. It was verified that the criteria established by the CAPES’ guiding document were not adequately followed. It could be concluded that the journal BABT should have been classified in the stratum A2, with a possible oscillation between A1 and A4, so that its classification as a B3 publication is out of the range foreseen in CAPES’ established criteria.

Keywords:
postgraduate; Qualis; Brazilian Archives of Biology and Technology

INTRODUCTION

Since the last decades there has been a great evolution and consolidation of stricto sensu postgraduation courses in Brazil, chiefly due to the efforts made by the Brazilian Coordination for the Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, henceforth CAPES) [¹]. In light of the growth of the offers of educational systems, the existence of criteria for their evaluation is of crucial significance for the creation of strategies to achieve the goals of the programs [²].

CAPES is an agency associated with the Brazilian Ministry of Education and works closely with stricto sensu postgraduate studies in Brazil. Among other functions, CAPES is responsible for the evaluation of institutions based on Brazilian policies within the respective scope, as well as for the management of the Qualis system for the classification of journals [³].

The systems for the evaluation of postgraduate programs were established in the country in 1977 [³,⁴] and have undergone major changes since 2000 due to the evolution of the number and level of publications. This led to changes in the way of conducting and applying scientific research in Brazil [⁵].

According to Barata [⁶], the management of postgraduate programs is supposed to provide tools or to develop methods that can have an impact on the evaluation of the courses, with a focus on scientific production. Postgraduate courses are responsible for the majority of the scientific production and its qualitative and quantitative growth [⁷,⁸]. In order to achieve it, the evaluation criteria play a fundamental role: they give greater weight to the accreditation and re-accreditation of the programs, reaching between 35% and 40% of the course’s concept [⁸].

As stated in Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [⁹], the evaluation systems must be continuously improved and serve as guide for researchers, programs and institutions as a standard of excellence in search for better academic quality for master’s and doctoral courses.

The Brazilian evaluation system employs Qualis Journals (Qualis Periódicos) as a classification factor for scientific journals and is understood as a list of vehicles used for the dissemination of intellectual production of stricto sensu postgraduate programs [⁶,¹⁰,¹¹].

For CAPES, Qualis is an instrument that aims to assess the quality of the scientific journals for postgraduate research in Brazil [⁴]. According to Amoras [¹²], the Qualis system is based on the entry of publications of an article into the Lattes Curriculum of the authors associated with stricto sensu postgraduate programs.

Since the emergence of Qualis in 1998, the assessment of intellectual production has become more unified and objective. The classification varies between the areas of knowledge, so that each area receives an independent assessment. After the evaluation, the publications are included in a stratified list. Finally, this list is incorporated on a basis, upon which the programs will be evaluated [¹³,¹⁴].

In accordance with Soma, Alves and Yanasse [¹⁵], these lists comprise the set of journals with publications of postgraduate students and professors. CAPES currently divides stricto sensu postgraduate programs into 49 evaluation areas, which are grouped into nine major areas of knowledge: Agrarian Sciences; Biological Sciences; Health Sciences; Exact and Earth Sciences; Engineerings; Multidisciplinary; Human Sciences; Applied Social Studies; and Linguistics, Letters and Arts. These areas are, in turn, grouped into three fields: Life Sciences; Exact, Technological and Multidisciplinary Sciences; and Humanities [¹⁶].

From 1998 to 2006, Qualis underwent several adaptations due to (according to CAPES) its inadequate use. In 2007, Qualis underwent a reformulation, whereby a new classification form was conceived and comprised eight strata: A1, A2, B1, B2, B3, B4, B5 and C. This way, the journals with the best and worst evaluation are classified respectively as A1 and B5. Journals that do not meet the minimum required criteria in their areas receive the classification C, a non-scoring stratum in the postgraduate evaluation [⁶].

In 2019, CAPES proposed a new method for the evaluation of Qualis Journals for the 2017-2020 period, as published in the Official Letter no. 6/2019-CGAP/DAV/CAPES, under the justification that this new formula allows for more balanced comparisons between the evaluation areas [¹⁷]. Furthermore, the need to review the assessment is one of the points foreseen in the Brazilian National Plan for Post-Graduation 2011-2020 [¹⁸].

Taking into account the publications of the years 2017 and 2018, the proposal was based on four principles: (i) A unified classification of the journals, valid for all areas; (ii) A classification attributed by a mother area, this being the area in which the journal obtained the largest number of publications; (iii) Qualis Reference (Qualis Referência), where the classification is attributed by means of the combination of bibliometric indicators and a mathematical model employed on all areas, whereas each area may propose changes in up to 10% of the journals in two strata, and up to 20% of the journals in one stratum, up or down; and (iv) Bibliometric indicators that take into consideration, respectively, the number of citations in three databases: Scopus (CiteScore), Web of Science (Impact Factor), and Google Scholar (h5-index): CiteScore Percentile and/or Impact Factor shall be regarded as the first stratification criterion. When the journal has percentile values in more than one base and in more than one category, the highest value among all will be taken into account [¹⁷].

This method resulted in a sectioned classification into eight strata calculated at equal intervals of 12.5 points of the final percentile. Journals with percentiles above the median comprised the strata “A” (A1, A2, A3 and A4), and journals with percentiles below the median comprised the strata “B” (B1, B2, B3 and B4) [¹⁷].

Shortly after the disclosure of such information, although not officially published by CAPES, a preliminary version of Qualis 2017-2020 was widely disseminated in the Brazilian academic environment. This version considered the application of the aforementioned method, as well as the results of the period 2017-2018 of the four-year evaluation period of the then in effect postgraduate program.

The journal Brazilian Archives of Biology and Technology (BABT), object of study of the present work, received the B3 classification. The goal of the present article was to simulate CAPES’ proposed method for the classification of Qualis in order to investigate the path that resulted in the stratification that was attributed to the referred journal.

MATERIAL AND METHODS

The website of the BABT journal on the SciELO platform was consulted in order to locate all articles published in 2017 and 2018.

Considering that the articles are presented in an Open Access format and available free of charge on the platform, the extraction of the data necessary to carry out the analysis took place. The publication year, the articles’ titles, as well as the names of the authors and their institutional associations were collected.

Due to the fact that CAPES’ proposed method for the Qualis classification contemplates only publications produced by professors within the scope of the national postgraduate program, and that the data for the assessment of the stricto sensu postgraduate program in Brazil consist of the filling out of the Coleta application, which integrates CAPES’ Sucupira Platform, which is annually fed through the data extraction from the researchers’ Lattes Curricula, the access links to the Lattes Curricula of the Brazilian National Council for Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq) of all authors who had publications in the researched period were raised.

By means of the located Lattes Curricula, these were accessed in order to establish the authors’ association with stricto sensu postgraduate programs in Brazil during the year of the publication. After the verification of the association, the area of evaluation of the postgraduate programs was searched in the section of Evaluated and Recognized Courses of CAPES’ Sucupira platform.

For the cases when the Lattes Curriculum could not be identified, or the present information were insufficient to identify the country of the institution to which the authors are associated, a search was conducted on Google’s website. When more than one affiliated institution was exhibited, only the first presented institution was considered, based on the presumption that it was the author’s main institution.

In summary, the data collection script was performed according to the following scheme Figure 1.

Figure 1
Synthetic scheme of data collection.

RESULTS AND DISCUSSION

During the researched period, the journal BABT presented three published editions: volumes 60 and 61 and a special issue, which were all considered in the present study. The number of published articles sums up to 199.

The articles had a participation of 897 authors/co-authors, out of which 870 were distinct: five with three publications, 17 with two publications and 848 with one publication.

Concerning the country of affiliation, 359 authors/co-authors were associated with Brazilian institutions, while 538 were affiliated with foreign institutions, as can be seen in the table below Table 1.

Table 1
Distribution of countries of institutions with which authors/co-authors are associated

Country	Number of authors/co-authors
Brazil	359
China	112
Iran	103
India	99
Turkey	47
Pakistan	44
Saudi Arabia	24
Egypt	19
Korea	10
Mexico	10
Other (23 countries)	70

At least one Brazilian author/co-author participated in 69 articles, while only foreign authors/co-authors appeared in 130 articles. Out of the articles with the participation of Brazilians, 60 were authored/co-authored by at least one professor associated with a stricto sensu postgraduate program recognized by CAPES. 139 articles had no authorship/co-autorship of any professor associated a stricto sensu postgraduate program recognized by CAPES.

128 professors associated with stricto sensu postgraduate programs could be identified, who participated in 158 postgraduate programs: two professors participated in three postgraduate programs, 26 in two, and 100 in one.

Excluding the redundancies comprised by the situation in which more than one person from the same postgraduate program was the author/co-author of the same article, it was verified that the publications were associated to 122 affiliations with a stricto sensu postgraduate program recognized by CAPES.

The participation of professors associated with 26 of the 49 CAPES evaluation areas was identified. These results comprise eight of the nine major areas (with the exception of Linguistics, Letters and Arts) and all three fields. With the exclusion of redundancies, the following configuration could be obtained, whereby the number of publications, organized by field, major area and evaluation area, is marked in parentheses Table 2.

Table 2
Distribution of publications of postgraduate programs according to CAPES’ classification

Field	Major Area	Evaluation Area
Life Sciences (78)	Agrarian Sciences (47)	Food Science (16)
		Agrarian Sciences I (16)
		Veterinary Medicine (13)
		Zootechny / Fishery Resources (2)
	Biological Sciences (16)	Biodiversity (10)
		Biological Sciences I (1)
		Biological Sciences II (4)
		Biological Sciences III (1)
	Health Sciences (15)	Physical Education (1)
		Nursing (1)
		Pharmacy (6)
		Medicine I (1)
		Medicine II (3)
		Medicine III (1)
		Nutrition (1)
		Dentristry (1)
Exact, Technological and Multidisciplinary Sciences (41)	Exact and Earth Sciences (4)	Chemistry (4)
	Engineerings (17)	Engineerings I (7)
		Engineerings II (9)
		Engineerings IV (1)
	Multidisciplinary (20)	Biotechnology (9)
		Environmental Sciences (2)
		Interdisciplinary (8)
		Materials (1)
Humanities (3)	Human Sciences (1)	Education (1)
	Applied Social Studies (2)	Public and Business Administration, Accounting and Tourism (2)

• In light of the obtained numbers, it was verified that the BABT journal’s main area was attributed to Agrarian Sciences. However, a tie could be detected between the areas Food Science and Agrarian Sciences I, both with 16 occurrences.

It was not possible to identify precisely which of the aforementioned evaluation areas constituted the mother area in the journal’s classification, given that, in event of a tie, as described in the Official Letter no. 6/2019-CGAP/DAV/CAPES, the mother area is the one in which the number of publications in the journal was more representative concerning the total production of the area [¹⁷]. However, the total number of productions of the area was not disclosed by CAPES.

The standardization of objective indicators for the Qualis classification in all areas of knowledge can be regarded as a positive factor because it enables the process of academic comparison [¹⁷]. However, it should be highlighted that, in terms of the postgraduate evaluation at national level, it is impossible to disregard indicators that are related to the SciELO database, which has proven to be an important platform for the dissemination of scientific knowledge specially in Brazil, as well in Latin America, the Caribbean [¹⁹,²⁰].

Although the standardization process appears to be an innovative initiative, considering the extensive number of existing areas, each with their respective peculiarities, the adoption of a unique criterion for the assessment is questionable, since it can lead to disastrous results for journals that are already consolidated [²⁰].

In a similar analogy made by the journal [²⁰], it is pointed out that the definition of the mother area, regarding the highest number of articles published in a journal, tends to emphasize the areas with the highest numbers of postgraduate programs, thus depreciating smaller areas. According to the Brazilian Association of Public Health [²¹], this can lead to the “escape of scientific articles to other (similar) areas in order to achieve better qualifications; migration of journals from the collective health area to others, whose parameters are more favorable; underfunding of national journals of the area”.

In turn, the universalization of the evaluation and classification criteria of journals associated with the decentralization of the evaluation is incoherent, since it was individualized by area. Such a proposal may be valid for monodisciplinary journals, but not for journals that comprise a significant number of areas, as is the case of BABT. Furthermore, as highlighted by Amoras [¹²], the classification is carried out without regard to the area of the journals and articles, considering solely the area of the postgraduate program of the researchers.

Finally, with the evaluation of the journals being carried out exclusively through publications made by professors associated with stricto sensu postgraduate programs recognized by CAPES, the publications of foreign authors, who represent the majority in several journals, in particular those considered of excellence, were not taken into consideration. In this sense, it is worth noting that the internationalization represents a factor that has been valued by CAPES during its process of reformulation of postgraduate evaluation, as can be seen in PNPG 2011-2020. This suggests that the classification of journals in the new Qualis goes against the desirable indicators that CAPES itself emphasizes, a fact corroborated by Cueto [²²]. In addition to it, publications carried out by high-level researchers associated exclusively with industry were also disregarded, as well as egresses who are still not part of the faculty of stricto sensu postgraduate programs in Brazil [¹²] since the articulation between university and industry constitutes one of the central focuses of the new postgraduate evaluation system and is currently in implementation process.

CONCLUSION

The percentage of authors/co-authors affiliated with foreign institutions was 59.97%. In 65.32% of the articles published during the period, no affiliation with Brazilian institutions was identified. Out of the articles published in the period, only 30.15% had an author/co-author associated with a stricto sensu postgraduate program recognized by CAPES, while the other articles were disregarded for the journal’s classification according to the new Qualis method. Only 14.26% of the total number of authors/co-authors had such an affiliation.

The publications were distributed in 53.06% of CAPES’ assessment areas, in 88.88% of the major areas, and in all fields. Although it was not possible to specify the mother area assigned to BABT, it was determined by allocating a percentage of 26.67% of the publications whose author/co-author was associated to a stricto sensu postgraduate program recognized by CAPES. If the percentage of all published articles is considered, this represents only 8.04%.

In the metrics published by the Scopus database for the period of 2018 and calculated on April 30, 2019 [²³], the journal BABT presented the CiteScore value of 0.97, being the percentile of this indicator 75. Therefore, the journal should have been classified in the stratum A2, as described in the Office Letter no. 6/2019-CGAP/DAV/CAPES [¹⁷]. Considering the possibility of alteration by the mother area in up to two strata, it could range from A1 to A4, thus in disagreement with the attributed classification in the stratum B3, which lies outside the predicted range established by CAPES’ criteria.

It is known that the stricto sensu postgraduate evaluation system in Brazil needs changes and that the course of the transformations of the improvement processes transitions between mistakes and successes. However, it is expected that the errors found during the course of this process will be corrected and not repeated in the future, especially due to the fact that the analysed classification consisted only of a preview and is, therefore, subject to revision. In this sense, it is expected that the shortcomings highlighted in this partial process will be suppressed in its final version.

Acknowledgments

This study was financed in part by the Fundação Araucária.

REFERENCES

• ¹
  Miranda IS. Descredenciamento dos cursos de pós-graduação stricto sensu do Brasil: impactos e motivos. Espacios. 2018;39(23):9.
• ²
  Fagá IT, Quoniam LM. A relação entre produção científica e avaliação da Capes: um estudo cientométrico de um programa das Engenharias II e de um programa interdisciplinar. Rev Bras Pós-Grad. 2016 Jun 6;12(29):651-79.
• ³
  Martínez-Ávila D. Qualis Periódicos: el sistema brasileño de evaluación de revistas. Anu Think EPI. 2019 Jan;13(1):1-12.
• ⁴
  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Internet]. Ofício circular no 23/2015-DAV/CAPES; c2015 [cited 2020 Apr 19]. Available from: http://capes.gov.br/sala-de-imprensa/noticias/7693-esclarecimento-qualis-periodicos
  » http://capes.gov.br/sala-de-imprensa/noticias/7693-esclarecimento-qualis-periodicos
• ⁵
  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Internet]. História e missão; c2008 [cited 2020 Apr 19]; Available from: http://www.capes.gov.br/historia-e-missao
  » http://www.capes.gov.br/historia-e-missao
• ⁶
  Barata RCB. Dez coisas que você deveria saber sobre o Qualis. Rev Bras Pós-Grad. 2016 Dec 22;13(30):13-40.
• ⁷
  Dantas F. Responsabilidade social e pós-graduação no Brasil: ideias para (avaliação). Rev Bras Pós-Grad. 2004 Nov;1(2):160-72.
• ⁸
  Azevedo MLN, Oliveira JF, Catani AM. O Sistema Nacional de Pós-Graduação (SNPG) e o Plano Nacional de Educação (PNE 2014-2024): regulação, avaliação e financiamento. Rev Bras Pol Adm Educ. 2016 Dec;32(3):783-803.
• ⁹
  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Internet]. Sobre a avaliação; c2014 [cited 2020 Apr 19]; Available from: http://www.capes.gov.br/avaliacao/sobre-a-avaliacao
  » http://www.capes.gov.br/avaliacao/sobre-a-avaliacao
• ¹⁰
  Rocha-e-Silva M. O novo Qualis, ou a tragédia anunciada. Clinics. 2009 Jan;64(1):1-4.
• ¹¹
  Castro ALS. Sistema Qualis/CAPES periódicos: reflexões sobre as distorções dos critérios de avaliação e de seus múltiplos usos. Multi-Science Journal. 2015;1(3):1-2
• ¹²
  Amoras FC. Editorial: o qualis único e área mãe. Estação Científica (UNIFAP). 2019 Jun;9(2):7-8.
• ¹³
  Ribeiro DA, Macedo MSAN, Couto EF. Critérios Qualis A1 da Educação: a avaliação sob a perspectiva da disputa acadêmica. Linhas Críticas. 2016 Apr;22(47):229-48.
• ¹⁴
  Pinto AL, Matias M, González JA. Produção brasileira da Ciência da Informação na Web of Science entre 1994 e 2013 e a lista Qualis/Capes da Área. Ibersid. 2016 Jun;10(1):51-61.
• ¹⁵
  Soma NY, Alves AD, Yanasse HH. O Qualis Periódicos e sua utilização nas avaliações. Rev Bras Pós-Grad. 2016 Dec 22;13(30):45-61.
• ¹⁶
  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Internet]. Sobre as áreas de avaliação; c2017 [cited 2020 Apr 20]; Available from: http://www.capes.gov.br/avaliacao/sobre-as-areas-de-avaliacao
  » http://www.capes.gov.br/avaliacao/sobre-as-areas-de-avaliacao
• ¹⁷
  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Internet]. Ofício nº 6/2019-CGAP/DAV/CAPES; c2019 [cited 2020 Apr 19]; Available from: http://www.prppg.ufpr.br/site/wp-content/uploads/2019/07/oficio_1015556.pdf
  » http://www.prppg.ufpr.br/site/wp-content/uploads/2019/07/oficio_1015556.pdf
• ¹⁸
  Brasil. Ministério da Educação. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Plano Nacional de Pós-Graduação - PNPG 2011-2020. Brasília: Capes; 2010.
• ¹⁹
  Campos R. O efeito Qualis: estudo de caso comparativo do sistema brasileiro de classificação de revistas avaliadas por pares. Ci Inf. 2015 Dec;44(3):447-52.
• ²⁰
  Cadernos de Saúde Pública, et al. Contributions to the debate on the assessment of scientific research production. Cad Saúde Pub. 2019 Oct;35(10):e00173219.
• ²¹
  Associação Brasileira de Saúde Coletiva [Internet]. Critérios para classificação das revistas: apreciação da proposta de Qualis Periódico Referência (online); c2019 [cited 2020 Apr 19]; Available from: https://www.abrasco.org.br/site/outras-noticias/notas-oficiais-abrasco/apreciacao-da-proposta-de-qualis-periodico-referencia/42328
  » https://www.abrasco.org.br/site/outras-noticias/notas-oficiais-abrasco/apreciacao-da-proposta-de-qualis-periodico-referencia/42328
• ²²
  Cueto M. A história das ciências e o Qualis Periódicos. Hist Cienc Saúde-Manguinhos. 2019 Dec;26(4):1083-4.
• ²³
  Scopus [Internet]. CiteScore metrics for journals and serials; c2019 [cited 2020 Apr 20]; Available from: https://www.scopus.com/sources
  » https://www.scopus.com/sources

HIGHLIGHTS

• 1
  CAPES’ proposed method for the Qualis classification of BABT was simulated.
• 2
  359 authors/co-authors affiliated with Brazilian institutions and 538 with foreign ones.
• 3
  Agrarian Sciences I and Food Science were the areas with largest number of publications.
• 4
  The criteria established by CAPES were not adequately followed.

Publication Dates

• Publication in this collection
  31 Aug 2020
• Date of issue
  2020

History

• Received
  11 Apr 2020
• Accepted
  25 Apr 2020