Brazilian impact factor of physics journals - the third side of the coin

Abstracts

The lack of correlation between the Journal Impact Factors and the most cited Brazilian papers in physics is statistically demonstrated. The existence of an effective "Brazilian Impact Factor" is observed, being its values, in general, lower than the international Impact Factors. In some cases, discrepancies from the international values are huge, leading to doubts on whether it is appropriate to use this indicator to judge Brazilian scientists.

scientometrics; Impact Factor; Physics journals; Brazil

A ausência de correlação entre o Fator de Impacto das revistas e os artigos brasileiros mais citados da área de Física é demonstrada estatísticamente. A existência de um "Fator de Impacto Brasileiro" efetivo é observada, sendo seus valores, em geral, menores que os Fatores de Impacto internacionais. Em alguns casos, as diferenças dos valores internacionais são enormes, questionando a adequação do uso deste indicador para o julgamento de cientistas brasileiros.

cientometria; Fator de Impacto; revistas de Física; Brasil

INTRODUCTION

The principle behind the definition of the journal Impact Factor (IF) can be traced back to a classical paper by Garfield (Garfield 1972). Its use worldwide has been boosted by the publication of the Journal Impact Factor index by the Institute of Scientific Information (ISI) since 1972. The original intention in creating the IF was to help librarians in choosing what journals they should order. A collection of papers concerning different aspects of the use of IF was recently published (Scientometrics 2012).

The subsequent history developed in a much unexpected way. As a simple and quantitative criterion for judging scientists, otherwise a quite hard job, IF has seduced many people, particularly those from the science funding agencies around the world. Presently, however, we witness a kind of insurrection against the IF, more intense in the field of biomedicine as well as in developed countries (Marder et al. 2010, Alberts 2013), which has culminated in the recent DORA declaration (DORA 2012). Criticisms have focused mainly on the bad influence the IF fever has on the formation of young scientists, who may feel constrained to avoid risks of innovative researches and fields. This debate promises to extend itself and to become more intense. In the last section the discussion is particularized to Brazilian issues.

The coin has a third side, however. Those who advocate the use of the IF may argue that, due to its transparency, competition among IF-ranked scientists occurs on clear and fair conditions. But while the IF is an international index, based on the total of publications in international indexed journals, there are reasons to believe that it does not apply homogeneously around the world. This suspicion becomes more relevant as one realizes that funding agencies are local! In Brazil, as possibly in other scientifically peripheral countries, the main governmental agencies for scientific development use the IF as a determinant factor to decide who is going to get money for research and who is not. The principle is simple and apparently correct: the most impacting scientists should have the money. A hasty corollary is the source of possible distortions: the most impacting scientists are those who publish in the journals with the largest IF.

In fact, scientists in these countries, Brazil in particular, usually have more difficulties in getting published in high-IF journals than their colleagues in the international scientific hard core. The reasons are multiple and it is not our objective to discuss them here. Instead, we want to quantitatively show that, as a result, in these peripheral countries the IF may not only be the presently criticized metrics but in fact, it may actually be a fake. In other words, not even the fairness and transparency of the competition based on the IF are assumptions that can be taken for granted. This proposition is developed on quantitative grounds in the next sections.

DO THE MOST CITED BRAZILIAN PAPERS IN PHYSICS CORRELATE WITH IF?

Take the N_p papers published in a journal in two subsequent years, say 2010-2011, and the N_c citations of these papers in the following year, 2012, for example. The two-year 2012-IF of that journal is defined as

IF= N_c / N_p [1]

Hence, in view of the proportionality between IF and N_c, the answer to the subtitle question should be an obvious yes. But, once we consider a representative sub-system of the whole scientific community it seems that this implication cannot be assured.

To check this hypothesis we collected data of the most cited papers on the Brazilian physics community published in 2012. The field of physics was chosen in view of its major international insertion in Brazil, being therefore reasonable to consider that for other fields our conclusions could hold or even step up.

The systematics of the search in ISI Web of Science (WoS), made in April 26, 2014, is as follows: Address: Brazil or Brazil; Time span: 2012; Databases: Science Citation Index and Conference Proceedings Citation Index - Science; Research Area: Physics; Document types: articles and proceeding papers; group-authors: exclude all.

The exclusion of group-authors papers aimed at restricting our search to the physics papers having substantial contribution from Brazilian physicists. Since the "exclude group-authors" facility of WoS does not eliminate all group papers, a one-by-one check had to be performed and papers with more than 15 non-Brazilian authors were arbitrarily eliminated. Note that review papers were also ruled out at this stage in view of them not being specific to our present goals. For the sake of comparison we performed the same search for papers originated in the UK (just changing, in the WoS search, the address to England or UK). UK was chosen randomly among the forefront countries of scientific production. The data are plotted against the IF in Figure 1.

A simple linear analysis exposes relevant comparative aspects (note that if all papers had been included, the centroids of N_cit for each journal would lie on a straight line). The linear analysis of the UK results, in green (fit line in black), are as expected, with a correlation factor r²=0.095 and slope=0.13, which are quite reasonable values in view of the IF definition. On the other hand, in the Brazilian case, in red (fit line in blue), the lack of correlation meets the eye and is confirmed by the quite vanishing correlation factor r²=0.000013 and slope=0.001. Furthermore, these results also mean that in 2012, for the same number of citations, on average Brazilian physicists published in journals of smaller impact factor than the UK physicists. Though not shown, if we go back in time, this feature becomes clearer.

Figure 1.
Journal impact factor (IF) versus number of citations of papers published in 2012 from Brazil (red triangles, blue linear fit) and UK (green circles, black linear fit). For clarity data are limited to IF>20 (UK) and IF>10 (Brazil). Data collected in April 26, 2014.

Even though this linear correlation analysis is not enough to lead to definitive conclusions, it clearly points out that something is wrong and justifies the next procedure.

THE EFFECTIVE BRAZILIAN IF

The previous results, if extended to all years, can only be explained once we devise a particular Brazilian IF for the physics journals. To check this proposition in quantitative grounds we first calibrate our procedure, trying to replicate the published 2012-IF (symbolized simply as IF in what follows) of the most important physics journals, our IF_calc. The Create Citation Report tool on WoS allows us to do it by the IF definition, eq. (1). Following the guidance at the Journal of Citation Reports (JCR) website we restrict N_c to the total of citations to articles, proceeding articles and reviews, the so-called scholar papers. Even though, the fact that N_c and N_p come from different sources, JCR and WoS respectively, prevents us from obtaining the exact published IFs. Our values are systematically smaller, but most of them are close enough to our purposes (see Tables below).

Following the same systematics we then define two Brazilian IFs: one including reviews and group-authors papers, IF_Braz1, and another excluding them, IF_Braz2. Dozens of physics journals have been searched, advancing a general rule that effective Brazilian IFs are almost systematically lower than both IF_calc and IF. Again it is pointed out that Brazilian papers are less cited than the international average. On the other hand, it is not our intention here to rank all physics journals from our particular systematics so that, in the following tables, we sample just a few singular and illustrative cases. For some important journals the number of Brazilian papers published in 2010-2011 is not enough to grant reliable statistics. For them we thus focus on the similarly defined 5-year (2007-2011) IF, fixing an arbitrary minimum of 15 papers to enrol a journal in the search.

In Table I we show some journals, distri buted in sub-areas of physics, presenting large variations of the IFs. For the sake of consistency, the percentual variation of IF_Braz2 in the last column is calculated with reference not to IF but to IF_calc, except for JACS. In this single case, the value of N_p exceeds 10,000, thus being unavailable, therefore we refer to IF for the percentual variation. Unexpected variations above 40%, are detected, reaching values as large as -59%. The exclusion of one review paper for Nano Letters could be seen as contributing to overthrow its IF_Braz2 to an artificial value. To check this hypothesis one would need to know the weight of this type of article in the construction of the international IF. Anyway, the fall of both IF_Braz1 and IF_Braz2in the other cases is visible (the elimination of group-authors papers is better analyzed in the following table). Against the main trend, a remarkable surprise is the performance of Icarus, with a variation of +73%.

In Table II, which shows the 2-year IFs for some other journals, we see that group-authors raises the IF_Braz1 of Physical Review Letters, Physics Letters B, New Journal of Physics and Journal of High Energy Physics to values even much higher than the corresponding IFs. Of course this means an unbalanced participation of group author papers in Brazil, since group-authors also participate in the overall building of the IFs. Yet IF_Braz2, without group-authors, collapses by large amounts, more than 50% in some cases. Again, representing a few cases, Physical Review A and The Astronomical Journal have their IF_Braz2 raised. In contrast to the broader distribution in sub-areas in Table I, Table II is dominated by journals grouped in a sub-area of physics, roughly Nuclear and Particle Physics. The journals in this sub-area present the highest decreases of IF_Braz2.

Table 1.
5-year 2012 impact factor of some selected physics journals. Ni is the total number of papers, Nexc the number of papers excluded (G for group-authors and R for reviews) and Nf=Ni-Nexc is the number of papers considered for the statistics. The acronyms, in decreasing order, mean Nano Letters, ACS Nano, Journal of the American Chemistry Society, Biophysical Journal, Laser Physics Letters. Nuclear Physics B and Icarus. *Unavailable from WoS.

Journal	IF	IFcalc	IFBraz1	IFBraz2	Ni	Nexc	Nf	%
NanoLett	14.13	14.01	13.60	9.25	29	1R	28	-34
ACSNano	12.50	12.03	6.53	6.53	15	-	15	-46
JACS	10.24	*	6.06	6.06	34	-	34	-41
Biophys J	3.98	3.93	2.78	2.78	32	-	32	-29
Las Phys Lett	4.97	4.94	4.00	3.77	36	1R	35	-24
Nucl Phys B	3.67	3.54	2.10	1.44	42	1G	41	-59
Icarus	3.19	3.14	5.44	5.44	18	-	18	+73

Table 2.
2-year 2012 impact factor of some selected physics journals. Ni is the total number of papers, Nexc the number of papers excluded (G for group-authors and R for reviews) and Nf=Ni-Nexc is the number of papers considered for the statistics. The acronyms, in decreasing order, mean Physical Review Letters, Physics Letters B, Journal of Physics G, New Journal of Physics, Journal of High Energy Physics, Journal of Cosmology and Astroparticle Physics, Physical Review A and Astronomical Journal.

Journal	IF	IFcalc	IFBraz1	IFBraz2	Ni	Nexc	Nf	Qualis	%
PRL	7.94	7.73	11.65	6.83	188	88G	100	A1	-12
Phys Lett B	4.57	4.44	8.55	2.05	154	76G	78	A2-->B1	-54
J Phys G	5.33	5.11	1.91	2.28	23	5G	18	B1-->B2	-55
New J Phys	4.06	3.77	4.26	2.72	27	2G	25	A2-->B2	-28
JHEP	5.62	4.29	5.34	1.87	92	1R,38G	53	A1-->B2	-56
JCAP	6.04	4.85	2.77	2.72	31	2G	29	A1-->B1	-44
PRA	3.04	3.01	3.38	3.38	152	-	152	A2	+11
Astron J	4.97	4.92	5.91	5.91	32	-	32	A2	+20

Table II also shows the mostly more important IF-based Brazilian government ranking of physics journals, namely the Qualis factor (Qualis), and how this ranking would change (follow the arrow) in case IF_Braz2 were to be considered. The Qualis ranges for this ranking are: A1 (FI ≥ 6.0), A2 (6.0 > FI ≥ 3.5), B1 (3.5 > FI ≥ 2.0), B2 (2.0 > FI ≥ 1.5).

DISCUSSION AND CONCLUSIONS

In recent years some works appeared under the scope of the evaluation of scientific work in Brazil, which deserves to be emphasized here. Meneghini et al. (2008) showed that papers originated from Latin America with no collaboration from authors in the first world are systematically less cited. Rocha e Silva (2010, 2011) have harshly criticized the construction of the Qualis ranking, proposing further evaluation paradigms, which are more focused on the quality of the articles and not on the journal FI. He also considered possible new metrics excluding review articles and a reassessment of local journals. Finally, Meneghini (2011) used data consisting of Brazilian publications in physics to approach the evaluation of institutions instead of individual researchers.

This literature differs essentially from the present work in the sense that here we did not necessarily put into question the appropriateness of the use of the IF but, instead, we question its calculation method. Nevertheless, some common points meet the eye. There is a clear connection between the observation of Meneghini et al. (2008) that the IF of a journal is affected detrimentally by publications from Latin America and the existence of an effective Brazilian IF. Furthermore, the exclusion of group authors exemplified here, (not assessed by previous criticism) in the construction of IF_Braz2 does increase the incidence of purely local papers. Thus, in some aspects IF_Braz2 also appears as a possible new metric designed to more positively evaluate local scientific production, as recommended by Rocha e Silva (2010, 2011). On the other hand, IF_Braz2 is unable to meet the prospect of a particular evaluation of local journals or to assess the particularities of different areas of knowledge, advocated in Rocha e Silva (2010, 2011). Finally, a particular point, the exclusion of review papers (from any evaluation index), appears as an overall agreement.

In conclusion, ranking scientists by IF implies a reversal of values by which the scientific journals turn to lend prestige to their authors, not the opposite as would be normally expected. We showed here that this process is even more harmful for science in Brazil, perhaps in other peripheral countries as well.

In fact, Figure 1 clearly shows that either Brazilian papers are i) less cited or ii) are published in journals of lower IF; presumably both. The tables show that different areas of physics have different international insertion of Brazilian scientists. Furthermore, the participation of group-authors papers and, in a lesser extent, review papers, is unbalanced.

These factors, together, lead to effective Brazilian IFs of physics journals, which in general are smaller than the published ones, but with quite different percentual shifts. We thus conclude that the international published IF does not properly depict the quality of the physics research in Brazil, a conclusion that could be extended to other fields and to other developing countries. These features show that the IF-based ranking of physicists and its consequence, the distribution of scientific resources in Brazil can be being strongly distorted.

ACKNOWLEDGMENTS

We acknowledge professors Maria Celeste R.L. de Vasconcelos, Ivan D. Aronni and José E. Lara for useful discussions on the subject.

REFERENCES

• Alberts B. 2013. Impact factor distortions (Editorial), Science 340: 787.
• DORA. 2012. San Francisco Declaration on Research Assessment. http://am.ascb.org/dora/
  » http://am.ascb.org/dora/
• Garfield E. 1972. Citation analysis as a tool in journal evaluation. Science 178: 471-479.
• JCR. http://thomsonreuters.com/journal-citation-reports/
  » http://thomsonreuters.com/journal-citation-reports/
• Marder E, Kettenmann H and Grillner S. 2010. Impacting our young. Proc Nat Acad Sci 107: 21233.
• Meneghini R. 2011. Citations to papers from Brazilian institutions: a more effective indicator to assess productivity and the impact of research in graduate programs. Braz J Med Biol Res 44: 738.
• Meneghini R, Packer AL and Nassi-Calò L. 2008. Articles by Latin American Authors in Prestigious Journals Have Fewer Citations. PLoS ONE 3: e3804. doi:10.1371/journal.pone.0003804
  » https://doi.org/10.1371/journal.pone.0003804
• Qualis. http://qualis.capes.gov.br/webqualis/principal.seam
  » http://qualis.capes.gov.br/webqualis/principal.seam
• Rocha e Silva M. 2010. Qualis 2011-2013: os três erres. Clinics 65: 935.
• Rocha e Silva M. 2011. Reflexões críticas sobre os três erres, ou os periódicos brasileiros excluídos. Clinics 66: 3.
• Scientometrics. 2012. 92(2): 207-503.
• WoS. http://thomsonreuters.com/thomson-reuters-web-of-science/
  » http://thomsonreuters.com/thomson-reuters-web-of-science/

Publication Dates

• Publication in this collection
  15 May 2015
• Date of issue
  Apr-Jun 2015

History

• Received
  09 June 2014
• Accepted
  08 Sept 2014