Relationships of field-normalized and non-normalized scientometrics with the Physical Education Web-Qualis grades

Relações de métricas normalizadas e não normalizadas por área com a classificação do Web-Qualis na Educação Física

abstract

This study aimed (1) to verify the relationship between the JIF, Eigenfactor, Citescore and SJR metrics of Physical Education journals throughout Qualis 2013-2016; and (2) to analyze their associations with the field-concerned Web-Qualis grades (based on the 2013-2016 quadrennium criteria). WebQualis grades, which refers to 2015-2018 data, were acquired by accessing officially available data. The scientometrics Journal Impact Factor (JIF), Citescore, Eigenfactor, and Scimago Journal Rank (SJR) for the years 2013 to 2016 and 2018 were extracted from official Clarivate Analytics and Elsevier sources. Approximately half of Physical Education journals included in the 2018 WebQualis database were indexed in Web of Science Collection and Scopus databases. The main results demonstrated high and stable appeared bivariate correlations between all scientometrics from 2013 to 2016. Both JIF, Eigenfactor, Citescore and SJR were associated with WebQualis grades. However, better adjustment of parameters was observed in the model that included SJR as the WebQualis predictor. The field normalized SJR appeared to better predict the WebQualis defined by the 2013-2016 Qualis criteria, which includes the adherence indicator. If researchers consider that distinct research areas are not equally weighted and require specific evaluation, as the authors suggest, use the normalized metrics to classify journals in Brazilian scenarium.

Key words:  Bibliometrics; Communication; Physical Education; Scientific journals

resumo

O objetivo deste estudo foi verificar a relação entre as métricas JIF, Eigenfactor, Citescore e SJR de periódicos de Educação Física ao longo do Qualis 2013-2016; e (2) analisar suas associações com as classificações do Web-Qualis (critério de classificação do quadriênio 2013-2016). As classificações do WebQualis, referentes aos dados de 2015-2018, foram adquiridas acessando fontes de dados oficiais. As métricas Journal Impact Factor (JIF), Citescore, Eigenfactor e Scimago Journal Rank (SJR) para os anos de 2013 a 2016 e 2018 foram extraídas de fontes oficiais da Clarivate Analytics e Elsevier. Aproximadamente metade dos periódicos de Educação Física incluídos no banco de dados WebQualis de 2018 foram indexados nos bancos de dados Web of Science Collection e Scopus. Os principais resultados demonstram altas e estáveis correlações bivariadas entre todas as métricas ​​de 2013 a 2016. Ambos o JIF, o Eigenfactor, o Citigenore e o SJR foram associados à classificação do WebQualis. No entanto, melhores parâmetros de ajuste foram observados no modelo que incluiu a SJR como preditor do WebQualis. O SJR normalizado por área pareceu melhor predizer o WebQualis definido pelo critério Qualis de 2013 de 2016, que inclui o indicador de aderência. Se os pesquisadores considerarem que áreas distintas de pesquisa não são igualmente ponderadas e requerem avaliação específica, como sugestão dos autores, a utilização das métricas normalizadas para classificar os periódicos no cenário brasileiro.

Palavras-chave:  Bibliometria; Comunicação; Educação Física; Revistas cientificas

INTRODUCTION

Quality metrics in science are being increasingly used by the need to measure and analyze journal-level knowledge production and research numbers¹. The criteria used to meet these metrics need to be accepted by the scientific community in each area of ​​knowledge, being called bibliometrics^1,2. Bibliometrics is a multidisciplinary field whose main focus is to analyze scientific production in a given context, such as in a country, an institution, a research area or a researcher³. In a special case, bibliometric indicators promoted by bibliometrics are used by universities to evaluate graduate programs, as in the Brazilian scenario⁴. Nowadays, the most commonly applied bibliometric indicators include the Journal Impact Factor (JIF), the Eigenfactor, Citescore, and the Scimago Journal Rank (SJR)⁵.

In the Brazilian scenario, the quality metric criteria for the use of bibliometric indicators are organized by the Coordination for the Improvement of Higher Education Personnel (CAPES). According to CAPES, Brazilian scientific production is evaluated by a system called Qualis⁶. The Qualis system is a set of procedures used by CAPES to stratify the quality of intellectual production (i.e. scientific journals articles) of graduate programs at grade levels (e.g., A1, A2, B1, B2, B3, B4, B5, and C)^7-9. It is important to highlight that the previous Physical Education Qualis system (2013-2016 quadrennium criteria) used of a single citation-based metric, the JIF. In this way, a previous concern emerges with the following question: What would be the best indicator to minimize the differences between areas? Field normalized metrics were proposed by both Scopus (SJR) and Clarivate Analytics (Eigenfactor) to advance on evaluating scientific journals.

Therefore, this study aimed (1) to verify the relationship between the JIF, Eigenfactor, Citescore and SJR metrics of Physical Education journals throughout Qualis 2013-2016; and (2) to analyze their associations with the field-concerned 2013-2016 Physical Education WebQualis.

METHODS

Data sample

The previous Physical Education WebQualis grades, defined for the 2013-2016 quadrennium, were acquired by accessing officiall CAPES repository¹⁰. Scientometrics data (i.e., JIF, Citescore, Eigenfactor, and SJR) for the years 2013 to 2016 and 2018 were extracted from official Clarivate Analytics¹¹ and Elsevier sources¹². These specific years were chosen to evaluate the data corresponding to the 2013-2016 Qualis quadrennium criteria and the most up-to-date WebQualis grades so far.

WebQualis grades data were treated by cleaning journal title and International Standard Serial Number (ISSN) entries (e.g., excessive spaces and grammatical variations). Thus, data from WebQualis grades and scientometrics were merged by matching journal title and ISSN. Duplicate journal entries (e.g., online and printed versions) were counted and merged to avoid bias. The final dataset included all journals reported in the Physical Education WebQualis and their respective WebQualis grade and scientometrics.

Statistical analysis

The absolute and relative frequencies were used to describe the data. The scientific metrics (i.e., JIF, Citescore, Eigenfactor, and SJR) were standardized to present an intuitive interpretation of findings (mean equal to zero and standard deviation [SD] equal to one).

The Spearman correlation coefficient was used to analyze each bivariate correlation between scientometrics according to evaluation year (i.e.; 2013 to 2016). The Spearman coefficient was chosen due to the known skewed distribution of outcomes. The magnitude of correlation coefficients was interpreted as moderate (0.50 to 0.70), high (0.70 to 0.90) and very high (0.90 to 1.00). This procedure included journals with complete data among all evaluation years. Thus, journals without information on at least one metric from 2013 to 2016 were excluded.

Multinominal logistic regression models were applied to evaluate the effect of each metric (i.e., JIF, Citescore, Eigenfactor, and SJR) on WebQualis grades. McFadden's pseudo-R-squared and Akaike Information Criterion (AIC) were computed for model comparisons. Higher values of pseudo-R-squared and lower values of AIC indicates better model fit/prediction when comparing models. The model referring to the metric which better predicted the WebQualis grades was described by presenting the predicted probabilities of a journal being allocated to a grade (B4-A1). This procedure included journals with complete data for the reference year of 2018. Analyses were performed using the Stata SE, version 15 (StataCorp. College Station, TX: StataCorp LLC).

RESULTS

From a total of 2,219 journals listed in the 2013-2016 Physical Education WebQualis, 250 duplicates were identified, and 1,969 journals were included for analysis. The proportion of journals indexed on the Web of Science Collection and Scopus databases in 2018 was 44.2% and 54.5%, respectively. Only 42.6% of journals were indexed on both databases.

The magnitude of correlations for the relationships between JIF, Citescore, and SJR ranged from high to very high (r>0.85, p-value<0.001). High correlations were also observed for all metrics with Eigenfactor score (r≈0.75, p-value<0.001). The bivariate correlations between all scientometrics appeared to be stable from 2013 to 2016 (Figure 1).

Figure 1
Spearman correlation coefficients of the relationships between science metrics throughout the 2013-2016 Qualis quadrennium (n=828).

JIF, Citescore, Eigenfactor, and SJR rankings were correlated with WebQualis grades in 2018 (Figure 2). A higher correlation coefficient was observed for the Scopus SJR metric (r=0.50) when compared to JIF (r=0.39), Citescore (r=0.40) and Eigenfactor (r=0.30). It is interesting to note that from 12 journals allocated to B3 WebQualis grade, three ranked over the 90^th sample percentile on all scientometrics (e.g., JIF>5.0).

Figure 2
Distribution of journal rankings allocated on WebQualis Grades in 2018 according to scientometrics (n=847). Note: Journal rankings are presented in reverse order, with higher values suggesting higher metrics’ scores. r: Spearman correlation coefficients.

Four multinomial logistic regression models, one for each scientometric, were applied to further explore the associations between standardized metrics and the probability of being allocated on distinct WebQualis grades (Table 1). Better goodness of fit parameters was observed in the model that included Scopus SJR as the WebQualis predictor (higher McFadden’s R² and lower AIC). However, the pattern of classification appeared to be similar among all predictors (see Supplementary Material).

Table 1
Goodness of fit parameters of polynomial logistic regression models to predict 2018 WebQualis Grade (n=847).

Models	Goodness of fit parameters
	McFadden’s R2	AIC
Model 1 - JIF	0.0724	2439.02
Model 2 - Citescore	0.0787	2422.84
Model 3 - SJR	0.0999	2367.41
Model 4 - Eigenfactor	0.0296	2550.72

The probabilities of WebQualis classification according to the increase of Scopus SJR (z-score) are presented in Figure 3. The increase of SJR was associated with higher probabilities of a journal being allocated to higher WebQualis grades. The probability of B1 categorization increased until reaching 0.4 SD below the mean SJR, and then decreased. A similar pattern was observed for the allocation to A2 grade, with a probability increase until reaching 0.2 SD above the mean SJR. Higher probabilities of being a journal being allocated to the A1 grade were observed with the increase of the SJR score.

Figure 3
Multinomial logistic regression model applied to evaluate the association between SJR z-scores and WebQualis Grades (n=847).

DISCUSSION

This study aimed to evaluate the magnitude of the correlations between scientometrics during the period of evaluation of the WebQuails system (2013 to 2016) and highlight the importance of considering normalized metrics for classifications in the Brazilian scenario. It is important to state that the 2013-2016 Qualis system applied an indiscriminate use of a single citation-based metric, the JIF, which scarcely represents scientific reality^6,13. Impact factors are influenced by many aspects that are far from assessing the quality of production, such as the size of the scientific community of knowledge fields, the prestige of their subareas in the same field, the average number of authors per article, the language of publication and the country of origin of the authors^4,14. For this perspective, field-normalized scientometrics, such as SJR and Eigenfactor, should be considered as they may improve evaluation criteria for the Brazilian system.

High to very high correlations were observed for between JIF, Citescore, SJR, and Eigenfactor over the years of 2013 to 2016. Recently, Waris et al.⁵ considering journals of sports science verified high correlations between JIF and SJR (r= 0.90) and between JIF and Eigenfactor (r= 0.84). The authors advocate that SJR may be considered as an alternative to the JIF for sports science journals since the system is free and easily accessible for submission of manuscripts. Similar results were found considering other scientific areas, for example specific respiratory medicine¹⁵ and orthopedic journals¹⁶.

We observed better goodness of fit parameters in the predictive model which included SJR as the WebQualis predictor, which was in line with our hypothesis. It is worth mentioning that than some metrics (i.e. SJR and Eigenfactor) are field-normalized, and thus, allow the comparison between scientific areas. The SJR algorithm takes into account field differences, which are included in the 2013-2016 Qualis criteria by applying the field adherence criteria. However, small differences were observed between the effect of all the predictors and the findings should be interpreted with caution.

It is worth highlighting the main limitation of this study was that our analysis considered only the quadrienium from 2013 to 2016, as the evaluation of journals in the national Brazil scenario included the adherence criteria during this period.

CONCLUSION

We concluded that SJR, JIF, Citescore, and Eigenfactor are correlated between them over the years of 2013 to 2016, with strong correlations among SJR, JIF, and Citescore. The SJR was the metric that best discriminates the allocation of Physical Education journals to WebQualis Grades. It is suggested the use of field normalized scientometrics over metrics that account for only citations count to evaluate Brazilian scientific journals.

SUPPLEMENTARY MATERIAL

Supplementary material accompanies this paper.

Supplementary File 1. Free access in https://osf.io/9wpxg/

Supplementary File 2. Free access in https://osf.io/9wpxg/

Supplementary File 3. Free access in https://osf.io/9wpxg/

This material is available as part of the online article from http://www.scielo.br/j/rbcdh

• How to cite this article Kons RL, Lopes MVV, Milan FJ, Milistetd M, Silva KS, Detanico D, Nascimento JV. Relationships of field-normalized and non-normalized scientometrics with the Physical Education Web-Qualis grades. Rev Bras Cineantropom Desempenho Hum 2023, 25:e89743. DOI: https://doi.org/10.1590/1980-0037.2023v25e89743
• COMPLIANCE WITH ETHICAL STANDARDS
• Funding
  This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. This study was funded by the authors.
• Ethical approval
  This research is in accordance with the standards set by the Declaration of Helsinki.

REFERENCES

• ¹ Oliveira AB, Rodrigues RS, Blattmann U, Pinto AL. Comparação entre o Qualis/CAPES e os índices H e G: o caso do portal de periódicos UFSC. Inf Inf. 2015;20(1):70-91. http://dx.doi.org/10.5433/1981-8920.2015v20n1p70
  » http://dx.doi.org/10.5433/1981-8920.2015v20n1p70
• ² Souza Pinto MD, Santos RNM, Santos Bahia EM. Análise de citação da revista eletrônica Arquivística.net: uma aplicação das técnicas bibliométricas. Em Questão. 2009;15(1):27-42.
• ³ Araújo RF, Alvarenga L. A bibliometria na pesquisa científica da pós-graduação brasileira de 1987 a 2007. Encontros Bibli. Rev Biblio Ciênc Inf. 2011;16(31):51-70.
• ⁴ Barata RCB. Dez coisas que você deveria saber sobre o Qualis. Bol Tec PPEC. 2016;2(1):1-17.
• ⁵ Waris A, Ahmad S, Isam C, Abdel-Magid M, Hussain A. Comparison among Journal Quality Indicators of Sports Science Journals. Library Herald. 2017;55(3):339-51. http://dx.doi.org/10.5958/0976-2469.2017.00032.X
  » http://dx.doi.org/10.5958/0976-2469.2017.00032.X
• ⁶ Andriolo A, Souza AFM, Farias AQ, Barbosa AJA, França AS No, Hernandez AJ, et al. Classificação dos Periódicos no Sistema QUALIS da CAPES-a Mudança dos Critérios é URGENTE! Arq Bras Cardiol. 2010;94(3):290-1. http://dx.doi.org/10.1590/S0066-782X2010000300001 PMid:20730252.
  » http://dx.doi.org/10.1590/S0066-782X2010000300001
• ⁷ Pinto AL, Fausto S. Revistas Internacionais para a área de Ciência da Informação: outra visão além do sistema Qualis/Capes. Informação & Informação. 2012;17(3):23-48.
• ⁸ Campos JNB. Qualis periódicos: conceitos e práticas nas Engenharias I. Rev Bras Pós-Graduação. 2010;7(14):477-503. http://dx.doi.org/10.21713/2358-2332.2010.v7.14
  » http://dx.doi.org/10.21713/2358-2332.2010.v7.14
• ⁹ Brasil. Ministério da Educação. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. Qualis Periódicos [Internet]. Brasília; 2013 [cited 2020 Jan 12]. Available from: http://www.capes.gov.br/avaliacao/qualis
  » http://www.capes.gov.br/avaliacao/qualis
• ¹⁰ Brasil. Ministério da Educação. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. Sucupira [Internet]. Brasília; 2020 [cited 2020 June 12]. Available from: https://sucupira.capes.gov.br/sucupira
  » https://sucupira.capes.gov.br/sucupira
• ¹¹ Clarivate [Internet]. 2020 [cited 2020 June 12]. Available from: https://jcr.clarivate.com
  » https://jcr.clarivate.com
• ¹² Scopus [Internet]. 2020 [cited 2020 June 12]. Available from: https://scopus.com
  » https://scopus.com
• ¹³ Rocha-e-Silva M. O Novo Qualis, que não tem nada a ver com a ciência do Brasil: carta aberta ao presidente da CAPES. Clinics. 2009;64(8):721-4. http://dx.doi.org/10.1590/S1807-59322009000800002 PMid:19722071.
  » http://dx.doi.org/10.1590/S1807-59322009000800002
• ¹⁴ Ruano-Ravina A, Álvarez-Dardet C. Evidence-based editing: factors influencing the number of citations in a national journal. Ann Epidemiol. 2012;22(9):649-53. http://dx.doi.org/10.1016/j.annepidem.2012.06.104 PMid:22819434.
  » http://dx.doi.org/10.1016/j.annepidem.2012.06.104
• ¹⁵ García-Pachón E, Arencibia-Jorge R. Comparación del factor de impacto y el índice Scimago Journal Rank en las revistas del sistema respiratorio. Arch Bronconeumol. 2014;50(7):308-9. http://dx.doi.org/10.1016/j.arbres.2013.10.006 PMid:24315384.
  » http://dx.doi.org/10.1016/j.arbres.2013.10.006
• ¹⁶ Moverley R, Rankin KS, McNamara I, Davidson DJ, Reed M, Sprowson AP. Impact factors of orthopaedic journals between 2000 and 2010: trends and comparisons with other surgical specialties. Int Orthop. 2013;37(4):561-7. http://dx.doi.org/10.1007/s00264-012-1769-1 PMid:23329405.
  » http://dx.doi.org/10.1007/s00264-012-1769-1

Publication Dates

• Publication in this collection
  13 Mar 2023
• Date of issue
  2023

History

• Received
  22 June 2022
• Accepted
  07 Feb 2023