Pursuing efficiency: international visibility of the scientific production of Brazilian graduate programs in child and adolescent health from 1998 through 2003

Abstracts

OBJETIVO:Avaliar a tendência do número de publicações oriundos dos programas de pós-graduação brasileiros em saúde da criança e do adolescente e a proporção de citações desses artigos no MEDLINE e no Journal Citation Reports (JCR), utilizando a primeira base de dados como medida de eficiência e a última como indicador de visibilidade. MÉTODOS: Avaliamos 14 programas de pós-graduação quanto ao número de teses, dissertações e artigos citados no MEDLINE e JCR, através de dados secundários das duas últimas avaliações trienais realizadas pela Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), de 1998 até 2000 e de 2001 até 2003. RESULTADOS:O número de artigos publicados aumentou (de 1.520 para 1.917), bem como o número mediano de artigos citados tanto no MEDLINE (de 32,5 para 45) como no JCR (de 24,5 para 27). O número mediano de dissertações aumentou de 19,5 para 26,5; o número mediano de teses cresceu de 12 para 13,5. O número mediano de orientadores diminuiu (de 21,5 para 18,4). CONCLUSÃO: Os programas de pós-graduação em saúde infantil e do adolescente tornaram-se mais eficientes quanto à produção de conhecimento através da publicação de mais artigos com maior visibilidade internacional. Tal tendência foi acompanhada contraditoriamente pela redução no número de orientadores.

Bases de conhecimento; bibliometria; indicadores de publicação científica; políticas de apoio e financiamento para publicações científicas; publicações técnicas e científicas; comunicação e difusão científica

ORIGINAL ARTICLE

Pursuing efficiency: international visibility of the scientific production of Brazilian graduate programs in child and adolescent health from 1998 through 2003

Marcelo Z. Goldani^I; Ricardo Q. Gurgel^II; Danilo Blank^I; Jerônimo Gerolin^III; Jair J. Mari ^IV

^IProfessor adjunto, Departamento de Pediatria, Faculdade de Medicina, Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre, RS, Brazil

^IIProfessor adjunto, Departamento de Medicina e Núcleo de Pós-Graduação em Medicina, Universidade Federal de Sergipe (UFSE), São Cristóvão, SE, Brazil

^IIIAnalista de sistemas, Centro de Avaliação e Integração de Dados Institucionais, Universidade Federal de São Paulo (UNIFESP), São Paulo, SP, Brazil

^IVProfessor titular, Departamento de Psiquiatria, UNIFESP, São Paulo, SP, Brazil

^{Correspondence}

ABSTRACT

OBJECTIVE: To assess the trend in the number of published articles by Brazilian graduate programs in child and adolescent health and the proportion of such publications cited in MEDLINE and Thomson Scientific's Journal Citation Reports (JCR), using the former database as a proxy for efficiency and the latter as an indicator of visibility.

METHODS: We assessed the trends of 14 graduate programs concerning the number of theses, dissertations, and articles cited in MEDLINE and JCR, through secondary data from the latest two triennial evaluations carried out by the Brazilian Federal Agency for the Improvement of Higher Education (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, CAPES) between 1998 and 2000 and between 2001 and 2003.

RESULTS: The number of published articles increased (1,520 to 1,917), as did the median number of articles cited both in MEDLINE (32.5 to 45) and in JCR (24.5 to 27). The median number of dissertations rose from 19.5 to 26.5; the median number of theses went up from 12 to 13.5. The median number of faculty advisors decreased (21.5 to 18.4).

CONCLUSION: Graduate programs in child and adolescent health became more efficient in producing knowledge through the publication of more articles with broader international visibility. Such trend was contradictorily accompanied by a diminishing number of advisors.

Keywords: Knowledge bases, bibliometrics, scientific publication indicators, support and finance policies on scientific publishing, scientific and technical publications, scientific communication.

Introduction

Efficiency of academic research is a matter of pressing interest to practicing clinicians - pediatricians included - considering the growing tendency towards following evidence-based medicine, whose trait of orderly incorporating research evidence into medical practice is directly derived from the research and development concept. Under a broader public health focus, quality academic research is essential for devising national health policies, designing health actions, and delivering health services efficiently.^1-4

On the other hand, most theoretical academics currently agree that a nation's research and development drive must place society and the environment at the core of a research system based upon quality, transparency and value for money.^2,5 Such a concept is often translated by the equation that combines efficiency with the sum of proper utilization of financial supplies and knowledge output.^6,7

Given that the so-called knowledge society has been increasingly viewed as knowledge economy in today's globalizing context, and as it has called for a production of knowledge with the widest possible reach, the notion of visibility has been definitely infused into the academic research lexicon.^8-14 According to many authors, as far as the mainstream peer-reviewed periodicals scene is dependable, publication acceptance equals quality, while the proportion of publications cited in prestigious databases such as MEDLINE and Thomson Scientific's Journal Citation Reports (JCR) can be viewed as a proxy for visibility.^14-16 Such combination of visibility and quality is embraced by most research funding agencies, such as Brazil's Federal Agency for the Improvement of Higher Education (CAPES), which bases its evaluation of graduate education programs on publication output, chiefly international visibility, and advisors' performance. ¹⁷

Brazil is one of the few of the so-called middle-income countries having a consistent state program directed to financing projects of knowledge production in the health field, in spite of the small budget assigned to scientific research, considering the country's gross national income.^15,18,19

Governmental policies of health research funding in Brazil are mainly operationalized through three agencies with countrywide reach (the already cited CAPES, the National Council for Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq), and the Studies and Projects Funding Agency (Financiadora de Estudos e Projetos, FINEP), and through state foundations for research support. CAPES stands out among them, having a strong aim at organizing and planning the development of human resources at the graduate level. Its process of graduate program evaluation began in the late 1970s, but significant changes in methods and evaluation criteria were made from 1996 on, placing great emphasis on knowledge production, particularly on the number of published articles, international visibility, and researchers' individual performance.^17,20

During that period, the overall participation of Brazilian scientific publications in the international indexing databases underwent a significant rise from 0.56% to 1.49%, so that Brazil now ranks 17th on the list of countries with greatest scientific output, and 9th on the list of those countries with largest increase in scientific production over a 10-year period. ²¹

Recent studies show significant advances concerning the publication of Brazilian human health articles in international periodicals. ²² Amongst all knowledge fields that have been following such trend, that of child and adolescent health stands out, with a 213% increase in MEDLINE citations along the past 12 years. ²³ Multiple health specialties contribute to this figure, particularly public health, internal medicine and surgery research centers. Although the contribution of pediatric academic institutes and departments is also significant, recent research suggests that they are slower in keeping up with the upward trend of other specialties. ²³

The objective of this study was to assess the trend in the number of published articles by Brazilian graduate programs in child and adolescent health and the proportion of such publications cited in MEDLINE and JCR, using the former database as a proxy for efficiency and the latter as an indicator of visibility.

Methods

This is an ecological study on a time trend of secondary data derived from reports of the continuing evaluation of graduate programs in child and adolescent health. We included all programs belonging to the referred field, credentialed by CAPES, and evaluated during the latest two triennia (1998 through 2000, 2001 through 2003).

We examined the following variables: number of term papers (both dissertations and theses), number of advisors, number of advisees, and number of published articles. These data were assessed in the annual reports issued by each graduate program according to the criteria prescribed by CAPES. Since the top rating of the CAPES Qualis system - a process that ranks the journals and other types of publications used by graduate programs to publicize the intellectual output of their faculty and students - adopts JCR as the reference for classifying periodicals of international circulation (Qualis International A and B), and MEDLINE indexing as the prime database for the second rank (Qualis International C), we chose these two criteria as proxy measures for visibility. ²⁴ The overall visibility of each program was inferred from the ratio between the number of articles cited in MEDLINE and/or JCR and the total number of articles published in the same period. It is important to point out that those two databases show some degree of overlap, since the same article can be indexed in both.

The data analysis was descriptive through measures of frequency and central tendency. We used the Wilcox test to assess significant differences between medians of dependent samples and linear regression to find the trend of medians of advisors and articles per year during the study period. The level of significance was p < 0.05 for all calculations.

Results

Fourteen graduate programs in mother-infant health, pediatrics and child and adolescent health were included in the analysis, 10 of which were credentialed by CAPES to issue doctorate degrees and four to issue master's degrees only.

All evaluated indices of knowledge production, i.e. dissertations, theses and published papers, showed an overall upward trend, as seen by the analysis of medians of papers issued by all 14 programs. The total number of dissertations underwent a 35.9% increase from the first to the second triennium, while the number of theses had a 12.5% increase (Table 1). The total number of published articles was 1,520 during the first triennium and 1,917 during the second one, corresponding to a 26% increase. When separately considering the medians of articles per program cited in MEDLINE and JCR, there was an increase of 38.5% and 10.2%, respectively. In contrast, there was an overall 14.7% decrease in the number of advisors from 1998 through 2003 (p < 0.01) (Figure 1).

The increase in the number of MEDLINE indexed articles was higher compared with JCR rates, although there was a noticeable increment in the visibility of the articles published along the study period, as inferred by citation both in MEDLINE and JCR (Table 2).

Discussion

The results of this study indicate a moderate positive increment in the scientific production of graduate programs in child and adolescent health, in accordance with studies that assessed other areas of knowledge. ²⁵ However, one of our main findings is the contrasting decrease in the number of advisors, while the number of both term papers and published articles went up.

Some issues can be raised when trying to explain such results. First, has the continuing evaluation process carried out by CAPES been having a positive impact on the performance of graduate programs? The new method of evaluation that was implemented by CAPES in 1998 considers both the production and the higher quality of scientific articles as performance indicators. Thus, the impact factor of the periodicals in which papers get published is one of the chief quality criteria, whose top excellence goal is the publication in journals with impact factor equal to or above the median for periodicals in a given specialty. In a study that assessed performance indicators for cancer, cardiovascular and malaria research in Brazil in the late 1980s and early 1990s, the findings already showed continuous growth of the scientific community and foresaw a positive impact of scientific activity on biomedical education.^26,27 A recent study that looked into knowledge production in pediatrics showed a rise in the number of published articles since 1993, even though this started before the implementation of the new process of evaluation by CAPES and the study did not focus specifically on graduate program output. ²³ On the other hand, the diminishing number of advisors along the period of the present study may indicate a reaction to the new evaluation criteria, which tend to drive away faculty members with low academic output. This current scenario may hinder the aggregation of new researchers and advisors to the graduate programs, with the ensuing limitation of their production capacity.

Second, does a greater financial investment in research projects in the area of human health correlate with a significant increase in knowledge production in mother-infant health? In general, there has not been any remarkable rise in the human health research funding in Brazil along the past 20 years. The competition dynamics within the knowledge production arena is not yet fully understood; there are no theoretical references that may adequately explain the behavior and relations among the various actors in such a highly rivaled field. Science is certainly involved with the production of well-being, but it does not follow such basic market principles as consumer pressure, nor is it sensitive to investments in areas that are adjacent to science and technology. ²⁸ However, as King points out, scientific production in Brazil is quite small when compared to the gross national income, indicating that investments in science and technology are yet to be viewed as a strategic priority of the country. ¹⁵ Moreover, according to CAPES, in spite of a 205% increase in graduate degrees issued along the past 10 years, there has only been a 30% raise in the financing budget for graduate programs, with a marked resulting reduction of scholarships granted to new students during this period. ²⁹ Therefore, these findings allow us to suggest that the growth of scientific production in Brazil has occurred as a reaction of the institutions to a policy of excellence pursuit, in which the evaluation process played a fundamental role.

Some limitations of the present study may hamper a deeper analysis of the quality of graduate program publications. First, the study lacks a longer time trend series that would allow a more proper definition of the correlation between the new evaluation method and the scientific production trend. Second, we could not assess the impact factor of the periodicals in which the involved papers could get published. Third, we could not assess the number of citations that those papers elicited. In this regard, the improvement of evaluation criteria should permit access to that information in the future.

In conclusion, scientific production of graduate programs in child and adolescent health underwent a striking increase in international visibility advanced by a growing number of published articles of higher quality. The attainment of such efficiency level was accompanied by a reduction in the number of advisors and an increase in the number of students, in a scenario of financial constraint. The implementation of the new continuing evaluation process by CAPES in 1998 brought about new efficiency parameters that seem to have fostered, at least partially, the referred noticeable increase in the scientific output of graduate programs.

However, by comparing the results, there was an overall increase of 95.6% in the number of articles related to child health published by general Brazilian scientific institutions cited in MEDLINE from 1998 to 2003, while pediatric institutions presented a lesser performance, having increased their output by only 38.5%.^23,30 These results suggest, even in face of a significant increase, that pediatric institutions did not respond equally to the general inputs for increased performance in Brazilian science production. The reasons for this scenario must be evaluated promptly, among other reasons to investigate the total budget allocated to research in pediatric institutions and the administrative performance of programs through new criteria, such as scales of difficulties, opportunities and administrative pressure.

Finally, if we view such scenario in the light of the report by de Meis et al., ²⁹ who described the great level of stress of graduate students and faculty members caused by the conflict between growing competition and dwindling resources, and taking into account all limitations imposed by sparse investment, we must still question the further growth of graduate programs.

Acknowledgements

We are grateful to the reviewer for pertinent comments, which significantly improved the quality of this manuscript.

References

Manuscript received Jan 25 2007, accepted for publication June 20 2007.

References

• 1. Vianna CM, Caetano R, Ortega JA, Façanha LO, Mosegui GB, Siqueira M, et al. Flows of financial resources for health research and development in Brazil, 2000-2002. Bull World Health Organ. 2007;85:124-30.
• 2. Guimarães R. Pesquisa em saúde no Brasil: contexto e desafios. Rev Saude Publica. 2006;40(N Esp):3-10.
• 3. Schwartzman S. Equity, quality and relevance in higher education in Brazil. An Acad Bras Cienc. 2004;76:173-88.
• 4. Pope C. Resisting evidence: the study of evidence-based medicine as a contemporary social movement. Health (London).2003;7:267-82.
• ⁵ London, Department of Health, Research and Development Directorate. Best research for best health: A new national health research strategy. London: Department of Health; 2006. http://www.dh.gov.uk/prod_consum_dh/idcplg?IdcService=GET_FILE&dID=23849&Rendition=Web Access: 10/06/2007.
• 6. Levy D. Latin America's tertiary education: accelerating pluralism. In: Levy D. Higher Education and Science and Technology in Latin America and the Caribbean: Responding to Expansion and Diversification. 2002 March 8; Fortaleza: Brazil; 2002. http://www.iadb.org/sds/doc/FortalezaLevy.pdf Access: 10/06/2007.
• 7. Salerno C. What we know about the efficiency of higher education institutions: the best evidence. 2003. http://www.minocw.nl/documenten/bhw-99-bgo99.pdf Access: 10/06/2007.
• 8. Coimbra CE Jr. Dezoito anos de Cadernos de Saúde Pública. Cad Saude Publica.2002;18:4-5.
• 9. Li L, Zhang F. Developing English-language academic journals of China. Scientometrics.2003;57:119-25.
• 10. Ugolini D,Casilli C. The visibility of Italian journals. Scientometrics.2003;56:345-55.
• 11. Brown L. New impact factors reflect NPG's investments. 2004 [site na Internet]; http://www.nature.com/press_releases/if_pr.doc Access: 10/06/2007.
• 12. Zhao D. Challenges of scholarly publications on the web to the evaluation of science: a comparison of author visibility on the web and in print journals. Inf Process Manage. 2005;41:1403-18.
• 13. Campanario JM, González L, Rodríguez C. Structure of the impact factor of academic journals in the field of Education and Educational Psychology: citations from editorial board members. Scientometrics. 2006;69:37-56.
• 14. Buela-CasalG, PerakakisP, TaylorM, ChecaP. Measuring internationality: reflections and perspectives on academic journals. Scientometrics.2006;67:45-65.
• 15. King DA. The scientific impact of nations. Nature.2004;430:311-6.
• 16. Patel V. Closing the 10/90 divide in global mental health research. Acta Psychiatr Scand.2007;115:257-9.
• ¹⁷ Brasil, Ministério da Educação (MEC), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), Diretoria de Avaliação (DAV). Quesitos, indicadores e critérios para a avaliação trienal da grande área de ciências da saúde - 2001-2003. 2003.
• 18. Pinheiro-Machado R, de Oliveira PL. The Brazilian investment in science and technology. Braz J Med Biol Res.2001;34:1521-30.
• 19. YeFY. A quantitative relationship between per capita GDP and scientometric criteria. Scientometrics.2007;71:407-13.
• 20. Brasil, Ministério da Educação (MEC), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), Diretoria de Avaliação (DAV). Principais características do Qualis. 2007.
• 21. Glänzel W, Leta J, Thijs B. Science in Brazil. Part 1: A macro-level comparative study. Scientometrics. 2006;67:67-86.
• 22. BressanRA, GerolinJ, MariJJ. The modest but growing Brazilian presence in psychiatric, psychobiological and mental health research: assessment of the 1998-2002 period. Braz J Med Biol Res.2005;38:649-59.
• 23. Blank D, Rosa LO, Gurgel RQ, Goldani MZ. Brazilian knowledge production in the field of child and adolescent health. J Pediatr (Rio J).2006;82:97-102.
• ²⁴ Brasil, Ministério da Educação (MEC), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Classificação de periódicos, anais, revistas e jornais. 2007. http://qualis.capes.gov.br/webqualis/ Access: 10/06/2007.
• 25. Pellegrini Filho A, Goldbaum M, Silvi J. [Production of scientific articles about health in six Latin American countries, 1973-1992]. Rev Panam Salud Publica.1997;1:23-34.
• 26. Rodrigues PS, Fonseca L, Chaimovich H. Mapping cancer, cardiovascular and malaria research in Brazil. Braz J Med Biol Res. 2000;33:853-67.
• 27. Zorzetto R, Razzouk D, Dubugras MT, Gerolin J, Schor N, Guimares JA, et al. The scientific production in health and biological sciences of the top 20 Brazilian universities. Braz J Med Biol Res. 2006;39:1513-20.
• ²⁸ Brasil, Ministério da Educação (MEC), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Relatório de Atividades - Exercício 2004.
• 29. de Meis L, Velloso A, Lannes D, Carmo MS, de Meis C. The growing competition in Brazilian science: rites of passage, stress and burnout. Braz J Med Biol Res. 2003;36:1135-41.
• 30. Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, TrowM. The new production of knowledge. Competitiveness, collaboration and globalization. London: Sage; 1994.

Publication Dates

• Publication in this collection
  09 Nov 2007
• Date of issue
  Oct 2007

History

• Accepted
  20 June 2007
• Received
  25 Jan 2007