Ten-year growth in the scientific production of Brazilian Psychiatry: the impact of the new evaluation policies

Dez anos de crescimento da produção científica da Psiquiatria brasileira: o impacto das novas políticas de avaliação

ORIGINAL ARTICLE

Ten-year growth in the scientific production of Brazilian Psychiatry: the impact of the new evaluation policies

Dez anos de crescimento da produção científica da Psiquiatria brasileira: o impacto das novas políticas de avaliação

Jerônimo Gerolin^I; Rodrigo A. Bressan^I,II; Ricardo Pietrobon^III; Jair de Jesus Mari^I,IV

^IUniversidade Federal de São Paulo (UNIFESP), São Paulo (SP), Brasil

^IIInstituto Israelita de Ensino e Pesquisa Albert Einstein, São Paulo (SP), Brasil

^IIIDuke University Health System, North Carolina, USA

^IVInstitute of Psychiatry, King's College London, London, UK

^{Correspondence}

ABSTRACT

OBJECTIVE: Developed by the Ministry of Education, the Qualis evaluation criteria have strongly impacted the scientific production of Post-Graduation Programs. A new set of more stringent criteria has been proposed for Qualis. Our aim was to evaluate the impact of the new Qualis criteria on the scientific production of Post-Graduation Programs in psychiatry over the last 10 years.

METHOD:We extracted data from annual reports published between 1998 and 2008, and compared performance measured in terms of the old Qualis rating criteria and the new set of criteria.

RESULTS: There was a 25% increase in the number of Information Science Institute-indexed articles in the second five-year period, which rose from 1,213 to 1,518. While, according to the old Qualis criteria, 84% of the Information Science Institute production would have been classified as highly-rated (IF > 1), only 17% of the papers were classified as highly-rated (A1) according to the new Qualis rating criteria. Most papers (65%) were assigned to intermediate categories (B1 and B2) with an IF < 2.29.

DISCUSSION:All psychiatric Post-Graduation Programs have increased their production, but by favoring quality over quantity, the new rules have proved to be more useful for discriminating among the scientific production.

Descriptors: Bibliometrics; Developing countries; Psychiatry/statistics & numerical data; Publishing/statistics & numerical data; Mental health

RESUMO

OBJETIVO: Nosso objetivo foi analisar o impacto dos novos critérios de avaliação de publicações científicas (Qualis) sobre a produção científica de Psiquiatria ao longo dos últimos 10 anos. Os critérios de avaliação do Ministério da Educação têm refletido em forte impacto na produção científica dos Programas de Pós-Graduação.

MÉTODO: Extraímos dados dos relatórios anuais publicados entre 1998 e 2008 e comparamos os desempenhos medidos em termos dos antigos critérios de avaliação Qualis e dos novos conjuntos de critérios.

RESULTADOS: Houve um aumento de 25% no número de artigos indexados no Information Science Institute no segundo período de cinco anos, que passou de 1.213 para 1.518. Embora, de acordo com os critérios de anteriores do Qualis, 84% da produção Information Science Institute tenha sido classificada como de alta qualificação (IF > 1), apenas 17% dos trabalhos foram classificados como de alta qualificação (A1), de acordo com os novos critérios de avaliação Qualis. A maioria dos trabalhos (65%) foi atribuída a categorias intermediárias (B1 e B2), com IF < 2,29.

DISCUSSÃO:Todos os Programas de Pós-Graduação de psiquiatria aumentaram sua produção, porém, favorecendo a qualidade ao invés da quantidade, as novas regras provaram ser de mais utilidade para discriminar a produção científica.

Descritores:Bibliometria; Países em desenvolvimento; Psiquiatria/estatísticas & dados numéricos; Editoração/estatísticas & dados numéricos; Saúde mental

Introduction

The recently released National Science Indicators (NSI), published by the Thomson Institute for Scientific Information (ISI) have revealed that, globally, Brazil currently ranks 13^th; it ranks 1st in Latin America.¹ Several articles on the growth of scientific production in Brazil have been published in the last few years.^2-7 Leta et al.² demonstrated that the number of publications about psychiatry increased 168% between 1981 and 1995. Razzouk et al.⁴ showed that ISI articles authored by Brazilian researchers in the fields of psychiatry/psychology doubled in the 1999-2003 period. In Brazil, the vast majority of psychiatric research publications are launched by Post-Graduation Programs (PGPs) which are, in turn, associated with the psychiatry departments of public universities.⁸ Although previous studies have demonstrated that considerable growth has been achieved, little is known about the quality of this growth.

Our group relied on PGP reports from the Coordenadoria de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), which is the Ministry of Education's Coordination Agency for the Educational Development of Post-Graduates, to show how Brazil's scientific production grew steadily between 1998 and 2002, especially with respect to the number of ISI-indexed articles.⁵ Moreover, Mari et al. identified the conditions that made this growth possible, and suggested that one of the main reasons behind this sustainable growth was CAPES' evaluation of PGPs.⁶ By implementing a hierarchical evaluation system to oversee the quality of scientific publications, CAPES has played a fundamental role in promoting the growth of Brazil's scientific production.⁹ The Qualis rating system, which rates scientific production according to the journals in which articles are published, is based on two related parameters, i.e., the scope of distribution (international, national or local) and the quality (A-high, B-medium, C-low) of the publication. The Qualis system has been using value 1 as the median Impact Factor (IF) for psychiatry and mental health publications. Thus, papers published in journals with IF > 1, were classified as International A (IA); papers published in ISI journals without an IF or with an IF < 1, as International B (IB); and papers published in journals currently indexed in Medline but not in ISI were classified as International C (IC).

Recent data from Gonçalves et al.⁷ have shown that Brazilian investigators are increasingly publishing their articles in high-impact journals, and that between 2004 and 2006, the mean IF of the scientific production on mental health issues was found to be 2.21 (the median applied is 2, and the range is 0 to 31.44). These data have shown that Qualis ratings have become obsolete, thus encouraging CAPES to formulate a new set of evaluation criteria in order to be able to better discriminate between the quality of different top-ranked articles, since articles published in very high-impact journals such as Science and Nature would be measured as being the same as articles published in any journal with IF larger than 1.

In April 2009, CAPES developed new rules based on more stringent criteria for Qualis. These criteria establish four different categories for articles published in ISI- indexed journals: (A1 - FI > 3.8); (A2 - 3.8 > FI > 2.3); (B1 - 2.3 > FI < 1.0); and (B2 - FI < 1.0). In this article, we have aimed at evaluating the impact of the new Qualis criteria by examining 10 years' worth of scientific production (1998-2007) published by Post-Graduation Programs (PGPs) in Psychiatry and comparing it to the old Qualis criteria.

Method

Annual reports published by CAPES on PGPs dedicated to research in psychiatry were studied to estimate the number of their publications between 1998 and 2007. To determine the quality and relevance of this scientific production, the impact factor (IF) of the journals where these articles were published was obtained from the Internet site of the Journal Citation Reports.¹⁰ Only the ISI publications were selected as a parameter index of scientific production because they can be reliably traced and represent one of the best markers of scientific impact. We accessed the Journal of Citation Report - JCR (2008), available at http://esi3.isiknowledge.com/rankdatapage.cgi to obtain a general estimate of the impact factor of the journals in which papers written by Brazilian investigators had been published (the site was visited in May 2009).

The number of articles published in journals indexed in the Institute of Scientific Information (ISI) between 1998 and 2007 was extracted from CAPES's annual reports on the five PGPs dedicated to research in psychiatry as defined by the Brazilian Ministry of Education: Universidade de São Paulo (USP); Universidade Federal de São Paulo (UNIFESP); Universidade Federal do Rio Grande do Sul (UFRGS); Universidade Federal do Rio de Janeiro (UFRJ); and Universidade de São Paulo campus Ribeirão Preto (USP/RP). The old Qualis criteria considered as International A (IA) those articles published in journals with IF > 1, and as International B (IB) those articles published in journals with IF < 1. The new Qualis evaluation system will be based on the following classification categories: (A1 - FI > 3.8); (A2 - 3.8 < FI < 2.3); (B1 - 2.3 < FI < 1.0); and (B2 - FI < 1.0). The comparison was carried out by dividing the production into two 5-year periods (1998-2002 and 2003-2007). The program of the Universidade Federal de Pernambuco was excluded because it could not provide data covering the 10-year period established for comparison purposes. The available data on UFRGS covered only nine years instead of 10.

Results

A total of 1,213 articles were published in the first 5-year period, and 1,518 in the second 5-year period (Table 1), thus representing an increase of 25%. The median Impact Factor (IF) was 1.04 for the first period, increasing to 2.09 for the second 5-year period. The number of publications found in journals with an IF above 3.8 was 90 (7.4%) in the first period, and 174 (11.4%) in the second period, nearly double. The largest share of the scientific production fell into the B1 category (486/1518, 32%), which presented the highest increase (244%) and whose IF ranges between 1.0 and 2.3. It is noteworthy that, when comparing the two periods, all programs showed positive improvements with respect to quantity and quality. The largest number of program advisers was found at USP while UFRGS had the lowest number.

The highest number of A1 articles was found to have been produced by USP. UNIFESP and UFRGS showed similar parameters, followed by UFRJ and USP/RP. The percentage of A1 papers varied among the PGPs: USP had 20.1%, UFRGS 19.6%, UNIFESP 17.4%, UFRJ 11.8% and USP/RP 7.95%. The increase in the number of articles if the previous criteria were to be applied is shown in Figure 1. While in 1998 the number of articles with IF > 1 was below 50, in 2007 it had increased to approximately 250 articles per year. Meanwhile, the number of articles published in journals with IF < 1.0 remained fairly stable, with fewer than 50 articles per year. Figure 2 shows the distribution of articles broken down into the four categories (A1, A2, B1, and B2) following the newly adopted criteria applied to the 10-year period. The total number of international articles was below 50 per year in 1998, rising to more than 300 in 2007. However, Figure 1 shows that the highest growth, at 140 articles per year, was seen in the B1 stratum, i.e. journals whose impact factors ranged between 1 and 2.3. In 1998, the number of A1 articles written amounted to fewer than 5, while, in 2007, an average of 40 articles were written, thus representing a substantial quality improvement. If the new criteria were to be applied to the entire period, USP would have accounted for the publication of 250 A1 articles, followed by UNIFESP with over 150, and UFRGS with approximately 120 (Figure 3).

Discussion

We provided an estimate of the number of ISI articles published over the last 10 years (1998-2007) by PGPs dedicated to psychiatric research in Brazil, and showed that the number of articles has increased significantly in this period. Based on this estimate, we evaluated the impact of the new CAPES-Qualis evaluation criteria compared to the old criteria, and showed that the increase in the number of published papers does not necessarily reflect an improvement in the quality of such scientific production. The total number of international articles was less than 50 per year in 1998, rising to more than 300 in 2007; however, the highest growth was seen in the B1 stratum. Nonetheless, there was also an increase in the number of A1 papers, which, in 1998, amounted to fewer than 5, while in 2007 there were around 40, thus showing a substantial quality improvement. If the new criteria were to be applied to the whole period, USP would have accounted for the publication of 250 A1 articles, followed by UNIFESP and UFRGS. According to the old Qualis rating system, the largest share of the ISI production (84%) would have been classified as highly-rated (International A), while according to the new Qualis, only 17% of these papers would have been highly-rated (A1). Most papers (65%) belonged to intermediate categories, that is, B1 and B2, which include ISI papers published in journals whose impact is below 2.29.

According to our survey, in 2007, around 40 ISI articles were published in journals with an IF higher than 3.8, thus proving the competitiveness of Brazil's scientific production in psychiatry. Indeed, considering that the percentage of articles varied from the 8% produced by USP-RP to the 20% produced by USP, these data suggest that part of the scientific production of psychiatric PGPs can be compared to that of leading international research groups. This paper did not aim at drawing comparisons between PGPs since, as pointed out in a previous publication,⁵ the productivity of PGP supervisors varies significantly. However, considering that the percentage of A1 papers produced has varied among PGPs, one would expect this to greatly impact the way in which CAPES grades these programs in the future.

The data reported in this article do not reflect the entire scientific production in psychiatry produced by Brazilian authors. A proportion of the research on public health, neurophysiology, neuropathology, psychopharmacology, genetics and the molecular biology of psychiatric disorders was developed in basic research PGPs and has not been included here. Furthermore, the total number of articles evaluated here was based on the number produced by each PGP individually. We believe that there is collaboration between PGPs and cross-authorship is to be expected; thus it is possible that some articles have been counted twice. This would introduce an over-estimate of the total number of publications, which may well affect the total by as much as 10%. On the other hand, inadequate listing of the publications in the CAPES reports has led to the loss of some articles. To provide a benchmark for any improvement in the rating of the journals in this period we have used the current impact factor of the journals to classify the papers, to make a uniform comparison between methods. Therefore the current data do not perfectly match our previous classification of ISI papers according to Qualis.⁵

The IF is a bibliometric tool to be used as an indirect affiliated indicator of research quality.¹¹ There are many reasons to believe that the IF can be only a limited proxy of the quality of articles: 1) editors require authors to cite work in their journals;¹² 2) editors give preference for longer and peer-reviewed articles to increase IF; 3) the impact factor of a journal is driven by a few papers that are frequently cited;¹² and 4) almost a quarter of the citations of any paper come from the author him- or herself.¹³ Under these circumstances the journal can become more important than the content of the article. Performance indicators do not measure the impact on patients' lives, nor the use of the information by policy-makers or non-governmental organizations. Impact Factor (IF) is therefore a poor measure of overall impact.¹⁴ The impact factor used here is only a proxy of the quality of the articles produced by Brazilian investigators. The ideal would be to have the real citations derived by the papers in order to draw conclusions about improvements in quality. Thus, the IF criterion applied in these comparisons should be seen as only a probable indicator of quality. In addition, the number of publications was not divided by the number of program advisers, leading to a bias towards the largest programs.

From the data presented here it is certain that Brazilian scientific production in psychiatry has shown a quantitatively and qualitatively sustained rise over the last 10 years.¹⁵ These data will provide important information for CAPES on how to evaluate the scientific production by giving different weights for the new Qualis classification, which will have a major impact on PGP appraisal.

Acknowledgments

JJ Mari is a level I-A researcher and RA Bressan is a level II researcher at the Brazilian Research Board (CNPq). We acknowledge the important contributions made by the anonymous referees to the previous version of this paper.

Click to extend

References

1. Ministério da Ciência eTecnologia. Indicadores nacionais de ciência e tecnologia (C&T): comparações internacionais - produção científica. Brasília, DF MCT. Atualização: 2009 Nov 27. [ citado 11 Maio 2009]. Disponível em: http://acessibilidade.mct.gov.br/index.php/content/view/2078.html.

2. Leta J, Jacques R, Figueira I, Meis L. Central international visibility of Brazilian psychiatric publications from 1981 to 1995. Scientometrics. 2001;50:241-54.

3. Mari JJ, Bressan RA, Miguel EC. Mental health and psychiatric research in Brazil. Br J Psychiatry. 2004;184:273.

4. Razzouk D, Zorzetto R, Dubugras MT, Gerolin J, Mari JJ. Mental health and psychiatry research in Brazil: the scientific production from 1999 to 2003. Rev Saude Publica. 2006;40(Spe):93-100.

5. Bressan RA, Gerolin J, Mari JJ. The modest but growing Brazilian presence in psychiatric, psychobiological and mental health research: assessment of the 1998-2002 period. Braz J Med Biol Res. 2005;38(5):649-59.

6. Mari JJ, Bressan RA, Almeida-Filho N, Gerolin J, Sharan P, Saxena S. Mental health research in Brazil: policies, infrastructure, financing and human resources. Rev Saude Publica. 2006;40(1):161-9.

7. Gonçalves R, Kieling C, Bressan RA, Mari JJ, Rohde LA. The evaluation of scientific productivity in Brazil: an assessment of the mental health field. Scientometrics. 2009;80:529-37.

8. Velloso D, Lannes L, Meis L. Concentration of science in Brazilian governmental universities. Scientometrics. 2004;61:207-20.

9. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. Qualis: classificação de periódicos, Anais, Revista e Jornais [Internet]. Brasilia: CAPES. [citado 17 Maio 2009]. Disponível em: http://qualis.capes.gov.br/webqualis/.

10. Journal of Citation Report - JCR 2008. [Internet]. New York: Thompson Reuters; c2009; [cited 2009 Jun 13]. Available from: http://esi3.isiknowledge.com/rankdatapage.cgi.

11. Dong P, Loh M, Mondry A. The «impact factor» revisited. Biomed Digit Libr. 2005;5;2:7.

12. Smith R. Commentary: the power of the unrelenting impact factor--is it a force for good or harm? Int J Epidemiol. 2006;35(5):1129-30. Comment on: Int J Epidemiol. 2006;35(5):1123-7; discussion 1127-8.

13. Porta M, Fernandez E, Bolúmar F. Commentary: the 'bibliographic impact factor' and the still uncharted sociology of epidemiology. Int J Epidemiol.2006;35(5):1130-5. Comment on: Int J Epidemiol. 2006;35(5):1123-7; discussion 1127-8.

14. The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Med. 2006;3(6):e291. Comment in: PLoS Med. 2006;3(9):e410.

15. Razzouk D, Zorzetto R, Dubugras MT, Gerolin J, Mari Jde J. Leading countries in mental health research in Latin America and the Caribbean. Rev Bras Psiquiatr. 2007;29(2):118-22.

Submetido: October 16, 20

Aceito: January 7, 2010

Publication Dates

• Publication in this collection
  22 Mar 2010
• Date of issue
  Mar 2010

History

• Accepted
  07 Jan 2010
• Received
  16 Oct 2009