Brazilian scientific journals that use the Open Journal Systems (OJS): a quality analysis

Brazilian scientific journals that use the Open Journal Systems (OJS): a quality analysis

Luisa Veras de Sandes-Guimarães^I; Sely Maria de Souza Costa^II

^IFundação Getúlio Vargas, São Paulo, Brazil

^IIUniversidade de Brasília, Brasília, Brazil

^{Address for correspondence}

ABSTRACT

This study aims to assess the quality of Brazilian journals that use the Open Journal Systems (OJS) in order to be published. For this purpose, the main criteria used to assess quality were sought in the available literature, and a selection was made of those considered to be of great relevance. The universe of the research was comprised of journals that were correctly registered under their respective subject heading in the system website by June 2010, a total of 236. The sample's nature is intentional and, according to the established criteria, 78 journals were selected for the study. The quality aspects evaluated were: editorial board, authors, rules for article submission, peer review, age, format, language of publication and the journal impact. The criteria were assessed by indicating the presence/absence of the quality indicator or by attributing a poor/reasonable/good, quality scale, according to the criterion assessed. The results indicate that the criteria established for authors, rules for article submission, and format were positively evaluated. The editorial board was also positively assessed, although 38% of the journals analysed did not present the composition of this group on their websites. The peer review criterion was negatively assessed, since only 17% of the journals correctly inform how the article evaluation process occurs and what criteria are used in the assessment of the submitted articles. Furthermore, it was identified the existence of a moderate positive relation between the H-index of the journals and their ages, and that there is no correlation between the language of article publication and the H-index of the journals.

Keywords: Scientific Electronic Journal, Open Access, Scientific Literature, Open Journal Systems (OJS), Quality of Journals.

1. INTRODUCTION

The scientific journal is one of the most frequently used channels for communication among academics and researchers in scientific communities. According to Meadows (1999), the foremost reason for the emergence of the scientific journal was the need to establish a more efficient means of communication between scientists. Tenopir and King (2001) have shown that articles published in journals constitute the most important informational resource used by scientists in their work, both in their teaching and research activities.

It is important to note that, in order to consider the results of research work as scientific knowledge, these must first be evaluated by peers and published in a scientific journal, book or other established means of scientific communication.

However, the subscription costs of scientific journals have increased substantially over the last few decades, unlike the budgets of most libraries, which have not. This makes it difficult to maintain the same number of journal subscriptions and has created an access barrier to published knowledge (King & Tenopir, 1998). Other authors such as McCartan (2010), McGuigan and Russell (2008), DigitalKoans (2010) and Luethi (2008) support this argument.

One example of the impact of the crisis that the price of journals has created is that of the University of Princeton which recently began to prohibit its researchers from ceding to publishers the copyright of their articles (except when a waiver is granted), thereby maintaining free access to almost all of their institutional publications. They justify this measure by stating that:

Universities pay millions of dollars a year for subscriptions to academic journals. People without subscriptions, which can cost up to $25.000 a year for some journals or hundreds of dollars for a single issue, are often prevented from reading research work that was funded by the taxpayer. Individual articles are also commonly locked behind pay walls (The Conversation Media Group, 2011, September 28).

This situation coincides with the development of information and communications technologies, especially from the second half of the 20th century. There was an increase in the creation of electronic journals, which made the publication of research results a far more agile and dynamic process. One of the most recent effects of information technologies in scientific communication is related to the open access to this type of literature.

Alberts (2002) highlights that, since scientific research is basically publicly funded, these results, scientific knowledge, should be recognized as a global public asset and be freely accessible to everyone. It should also be taken into account that what the great majority of researchers really want is not financial returns, but rather to have their work widely disseminated, so that these may gain greater visibility within the scientific community (Harnard, 1995, 2011).

This situation favored the emergence of a movement in the scientific community in support of open access to scientific information. This initiative established two strategies to enable open access which are defined by Harnard et al (2004) as the Green Road (self-archiving in open access repositories) and the Golden Road (open access electronic journals)

An important Brazilian initiative to provide open-access was the adoption by the Brazilian Institute of Information in Science & Technology (IBICT) of the Electronic System for Journal Editing (SEER). This is a customization of the Open Journal Systems (OJS) software, aimed at constructing and managing all the editing stages of an electronic journal (Márdero Arellano, Santos & Fonseca, 2005).

There has been a rapid growth in the use of this system to create and edit journals. In Brazil the use of SEER made it possible to create 300 new journal titles by March 2008. By June 2010 this number had increased by 160%, giving a total of 780 journals, rising to 924 by November 2011, an increase of 18.5%. The fact that the system is freely available and easy to use may lead to the non-observance of questions relating to the quality of scientific journals.

The question, therefore, that this study aimed to answer was: does the quality of Brazilian scientific journals that use the SEER correspond to the criteria that are generally considered when creating and maintaining these journals, such as the profile of the editorial board, the authors, rules for the submission and assessment of manuscripts, regularity, language and impact?

The relevance of this study lies in its potential to create knowledge in an area that is little explored by the present literature and to offer suggestions that may serve as a basis to improve editorial policies of scientific journals in Brazil.

2. THE QUALITY OF SCIENTIFIC JOURNALS

Assessing the quality of scientific journals is not a new subject in the academic environment. In 1964 the United Nations Educational, Scientific & Cultural Organization (UNESCO) prepared a model to assess Latin-American journals, which served as the basis for most of the assessments and models that subsequently emerged. Ferreira and Krzyzanowski (2003) list some of the first research works on the subject, which are summarized as follows:

* 1968 – Arends bases her work on the model created by UNESCO in 1964 and proposes that assessment criteria should be related to: the presentation of the material; the regularity of the publication; the period that the journal has been edited; periodicity, receptivity of the journal with regards to collaboration from other institutions; indexation, amongst others.

* 1982 – Braga and Oberhofer propose a model, based on the one prepared by UNESCO, where the criteria for assessment would cover normalization, periodicity, indexation, dissemination, authority, and so on. These criteria present different scorable variables and the level of performance of a journal depended on the number of points scored (very good, fair, poor).

* 1985 – Yahn, when assessing Agricultural journals, changed the Braga and Oberhofer model adding the assessment of the journal's content, as well as a format assessment.

• 1986 – Martins carried out an assessment on 224 journals in the fields of Science and Technology, with the intention of checking if these conformed to items related to normalization, as described in the precepts established by the Brazilian Technical Standards Association (ABNT).

These studies on quality assessment were essential to alert the scientific community to questions regarding the quality of journals. Later, other authors changed or created new criteria that complement the existing literature on the subject and adapt them to the new standards and changes of the 21^st century.

Stumpf (2003) carried out a study on 26 journals in the area of Communications where, with the collaboration of professors and researchers of this field, she assessed the journals in accordance with the following criteria: quality of the journal; the prestige that the journal has within the community; quality of the articles published; contribution that the journal makes to the subject; rigor in assessing articles; regularity of the publication; graphic presentation and distribution. Each researcher or professor awarded a score from 1 to 5 of for each item that was assessed.

Ferreira, Neubhaher, Reis & Gomes (2009) assessed electronic scientific journals in the field of Law, more specifically journals that use the SEER platform and are classified as A or B in the Qualis evaluation. The assessment was made based on criteria of normalization, periodicity, indexation and navigation. The journals were rated and classified in accordance with their performance (varying from "Poor" to "Very Good").

Trzesniak (2006) suggests that the creation and development of the Qualis evaluation system, sustained by the Brazilian Federal Agency for the Support and Evaluation of Graduate Education (CAPES), made the process of assessing journals better known and respected by researchers in different fields. The Qualis evaluation system assesses the quality of journals and assigns them one of following classifications: A1 (highest), A2, B1, B2, B3, B4, B5 and C (lowest). In the words of Trzesniak (2006, p. 347): "At present, it is unusual for a researcher, when submitting an article, not to take into consideration the journal's Qualis classification. It is increasingly common for an author to take an interest in how the Qualis assessment is made and on what criteria this system is based".

As a result, editors have begun to take a much greater interest in the formal aspects of a scientific journal, which are taken into consideration when quality is assessed, such as: the International Standard Serial Number (ISSN), guidelines for publication, instructions to authors, the regularity and periodicity of the publication. In addition to these formal aspects, Yamamoto and Costa (2009, p. 196) believe that:

[...] the assessment made editors seek to qualify the journals under their responsibility from the point of view of content. Indirect indicators are the valorization of a more transparent and qualified system of arbitration (describing the process, providing the names of counselors and consultants), providing opening for institutions other than those of origin, with emphasis on those who are in other Units of the Federation and, most importantly, seeking to index their journals in consecrated databases.

An analysis of the present literature about the quality of scientific journals enables us to conclude that this feature may be evaluated by focusing on the process or on the result. In the first instance, factors related to editorial process should be considered, such as the training and expertise of the editorial team (editors, the editorial board and editorial council), rules for the submission and evaluation of manuscripts, assessment deadlines and how approved articles will be made available, among others.

In an assessment that is focused on result, it is important to know what level of impact that articles published in a particular journal exerts on the scientific community and, thereby, in what measure these contribute to the growth of a particular field of knowledge. Following this line of thought, the more often articles in a journal are quoted by other authors, the greater the impact level of that journal.

These methodologies are not excluding, since an adequate editorial process will certainly contribute to the quality of the articles that are published. Therefore, the higher the level of quality of the articles, the more probable it is that the journal will attain a high level of impact.

2.1 Assessing a journal by focusing on the process and by focusing on the result.

An assessment focused on the process includes two quality dimensions regarding scientific journals, which Trzesniak (2006) described as the quality of the product's goal and the quality of the productive process. When considering the first dimension, a quality assessment usually analyzes the following items:

a) a highly qualified scientific editorial board, that successfully covers the journal's whole thematic reach, that is both institutionally and geographically diverse (aspects that are relatively easy to assess) and that is involved in revising "computer scripts" (an effective involvement is not easy to evaluate ....);

b) well qualified ad hoc consultants with institutional and geographical diversity (Trzesniak, 2006, p. 350).

To establish the quality of the productive process, the items normally included in an assessment are as follows: compliance with the proposed periodicity (bi-monthly, tri-monthly, etc.), and inclusion of the dates of receipt and acceptance of articles (Trzesniak, 2006). A journal's productive process is an indication of the efficiency or sluggishness of the assessment process of articles.

In addition to the characteristics mentioned previously, an assessment focused on process also ascertains the quality of the instructions given to authors. It is hoped that journals provide information regarding the conditions required to consider an article for evaluation, for example, by explaining: the models of formatting and normalization that are used (references, citations, etc.); the language in which articles should be submitted; number of pages required, and so on. It is also helpful if the journal explains the process by which a manuscript should be submitted. In addition, it is important that the journal lets the authors know how the process of evaluation occurs for submitted articles and what criteria are used in such assessments.

In the case of an assessment focused on results, it is necessary to establish the journal's impact. Such a procedure requires organizing and maintaining specific databases, an activity that hitherto has usually only been carried out in developed countries. In the case of Brazil, SciELO¹ tends to transform itself into an indexer capable of generating information related to the journal's impact. However, this is still the only experience of its kind in this country.

Calvert and Zengzhi (2001) believe that some quality criteria assessed in journals are basically extrinsic, and that only some actors involved in the publication of a journal consider them important to assess its quality. Such criteria would be: composition, the journal's reputation, the Editor's affiliation, the Editor's reputation, the Editorial Board, assessment policy for articles, and the journal's outlook. Instead, focus should really be on journal content and the quality of the articles it publishes (Calvert & Zengzhi, 2001).

In order to assess the quality and impact of journals, it is usual to employ bibliometric techniques. One recent example of a bibliometric study involving citation analysis was carried out by Machado-da-Silva, Guarido Filho, Rossoni & Graeff (2008). These authors assessed Brazilian scientific journals in the Administration field, using the impact factor and relating this index with the classification of these journals in the Qualis system of CAPES. They analyzed 21 journals and proceedings from the Brazilian Academy of Management Meeting (EnANPAD) edited from 2005 to 2007, dividing the journals into three groups, according to the Qualis ranking: 'A' Old National (that had already received the assessment score of 'A' for some time); "A' Recent National (the assessment score from 2007); 'B' National.

The authors confirmed that the citations and the impact factor are higher in the 'A' Old National group. With regards to the other two groups, no significant difference was found in the impact factors of either. In addition, they were able to ascertain that self-citation had no particular influence on the impact factor. The authors also stated that impact factor is a viable and relevant criterion when assessing journals. This does not mean, however, that only this criterion should be used, but rather that it should be recognized as being important and used in conjunction with others assessment criteria (Machado-da-Silva et al., 2008).

Mugnaini and Strehl (2008) mentioned that recently the only data that gave an idea of the impact of scientific journals were those generated from the data bases contained in the Web of Science, compiled by the Institute for Scientific Information (ISI). However, as time went by, other databases that are also able to index citations emerged, such as SciELO, Scopus and Google Scholar. Thus, the ISI continues to share space with other index services, since it is no longer the only mechanism used to calculate the impact of scientific production.

3. METHODOLOGY

The aim of this study was to analyze the quality of Brazilian scientific journals that use SEER. For this purpose, the main criteria used to assess quality were sought in the available literature, and a selection was made of those considered to be of great relevance. We worked with quality indicators of processes and results, used by agencies that promote education and research in Brazil. The research work is descriptive by nature and used the survey method, adopting a quantitative approach.

The universe of the research consisted of Brazilian scientific journals that use the SEER system and which, at the time when data were collected, were registered under their respective subject heading at the afore-mentioned system website (http://seer.ibict.br). In June 2010, it was possible to access 236 journals, registered under their particular subject headings, although 780 journals were registered in the system. That is to say, 544 journals did not specify their area of knowledge. For this reason, the universe considered was composed of 236 journals.

Selecting which journals would be included in our sampling was done based on previously established criteria, characterizing an intentional sampling. The selection criteria used were the following:

a. Journals from the most productive areas, according to the bibliographic production indicators of the post-graduate programs made available by CAPES, and related to the triennium of 2004-2006;

b. Journals that began to circulate either in print or in electronic form in 2006, or before².

It should be underlined, however, that these criteria could result in only better-quality journals being selected. However, it is necessary to confirm if at least these journals, because they belong to the most productive areas of knowledge and have been in circulation for long enough to become consolidated, present the minimum standards of quality required for scientific journals.

The field productivity was calculated based on the number of researchers and the number of articles produced during the triennium period. Based on the productivity indexes, a calculation was made to establish the measure that adequately represented the central position of the distribution of the areas within their greater areas of knowledge. The areas that were above this central measure would be considered as productive.

It was ascertained that all the great areas of knowledge were shown to have a very strong positive or negative asymmetry. This means that the distribution is shown to be concentrated in values of high or low magnitude. That is, within each great area, there are many areas of low productivity or many areas of high productivity.

In these cases, the average is not the best central distribution measurement according to Barbeta (1999, p. 108):

In general, given a set of values, the average is the most adequate measurement of the central position, when it is assumed that these values have a reasonably asymmetric distribution, while the median emerges as an alternative to represent the central position in very asymmetric distributions. The average is strongly influenced by discrepant values. [Author's emphasis]

Therefore, since the distribution asymmetry is strong, the median was chosen to represent the central position of the distribution, since the average is influenced by very high or very low values. The most productive areas should therefore present greater productivity than the median of their greater area.

A list was then made of all the journals that belonged to the most productive areas of knowledge, which came to a total of 184 journals. Of these, 84 were excluded because they had been published for less than four years, which reduced the total number of journals to just 100.

After the journals were filtered according to their starting year of publication, it was seen several presented problems, such as: publication being delayed for up to one year or more, problems of access, and a non-scientific profile. Thus, a further 22 journals were excluded from the survey, leaving a final sampling of 78 journals (33% of the universe) (see complete list in ^{Appendix A}).

3.1 Variables analyzed

The variables studied in this survey consist of a series of selected criteria considered to be relevant to ascertain the quality of a scientific journal. These criteria were identified through literature, mainly by the works of Ferreira (2005) and Trzesniak (2006, 2009), as well as other sources cited during the presentation of the research results.

3.1.1 Editorial Board

Affiliation: This refers to the number of members with no ties to the institution responsible for editing the journal.

Qualification: Level of academic degree held by members of the editorial board. The following scale was used: graduation; master's degree; doctorate.

Productivity: Number of articles published by members of the editorial board during the triennium of 2007-2009. Due to the high number of members, 1,340 in all, this part of the analysis was carried out with 10% of the total, that is, 134 members, divided equally among the journals and randomly selected. This information was obtained by consulting their Lattes curriculums. The editorial board was considered to be productive if the average number of articles written by the members in the triennium was higher than the average number produced in the field to which the journal is connected. It should be said that 31 (24%) of the journals did not provide information about the members of their editorial body and 41 (32%) used another nomenclature to describe that board, such as: editorial council, scientific committee, editorial commission, technical scientific committee and scientific council.

Due to the variety of nomenclatures used, it was necessary to analyze if the members mentioned represented the Editorial Board or the Editorial Council. For this, the following criteria were used to identify the Editorial Board, according to Trzesniak (2009): Inclusion of more than nine members (according to the author, an editorial council generally has up to nine members); Inclusion of members with different institutional and geographical backgrounds. In addition to being characteristic of the board, their members do not need to act as a group and, in theory, do not have to meet regularly, unlike the editorial council (Trzesniak, 2009).

3.1.2 Authors

The affiliation of the first author of each article published in 2009 was ascertained. The number of Brazilian and foreign authors was verified for the same year.

3.1.3 Rules for the Submission of Articles

Percentage of inedited works: It was calculated for each journal the proportion of inedited works published during 2009. A minimum rate of 40% inedited articles was used (Ferreira, 2005). Ideally, each journal should provide information about the percentage of inedited articles they publish, but this rarely occurs.

Criteria for formatting and normalization: Completeness and clarity when specifying guidelines related to the formatting and normalization of the articles.

3.1.4 Peer Review

Assessment procedures: Provide full and clear information regarding the assessment of manuscripts. Dates of the assessment process: Provide clear information relating to the dates of the assessment process, and at least include the dates of submission and approval of the articles.

3.1.5 Other formal aspects

The age of the journal; Format (if printed and electronic or only electronic); language in which the articles are published.

3.1.6 Impact of the journals

The Publish or Perish software was used to calculate the impact of each journal. The program retrieves and analyzes the academic citations of a particular author or journal. It uses Google Scholar to obtain the raw citations, then analyzes the citations and presents a great variety of metrics in a user-friendly format (Harzing & Van de Val, 2008).

One of the metrics presented is the H-index. This index was proposed by Hirsch (2005) to calculate the impact of scientists. The calculation is made as follows: an author has an H-index if H of his/her N articles has at least H citations each, and the other articles have less than H citations each. For example, a researcher with a "7" H-index value has seven articles with at least seven citations each. Braun, Glanzel and Schubert (2006) suggested that the H-index could be useful to calculate the impact of the journals. Firstly, because it is insensitive to an accidental excess of un-cited articles, as well as to one or several highly-cited articles. Secondly, because it combines the effect of quantity (number of articles) and quality (citations) in a fairly specific and balanced way, that should reduce the apparent overvaluation of some of the smaller revision journals. The H-index of a journal should not be calculated for the whole life of the journal, but only for a specific period (Braun, Glanzel & Schubert, 2006).

It was decided to use this software, since it uses Google Scholar to carry out the citation analysis, mechanism by which, in addition to being free, have been shown to be efficient in calculating the impact of journals and researchers (Harzing & Vander Wal, 2009). In addition, most of the journals analyzed are not indexed on the Web of Science or Scopus databases, which would therefore make if far more difficult to calculate an index to represent the impact of the selected journals.

4 RESULTS AND DISCUSSION

4.1 Editorial Board

The information relating to the Editorial Board, as shown on Graph 1, was gathered from 48 journals, since the remaining 30 (38% of the total) did not provide any information about their members or did not present any form of Editorial Board. It should be said, however, that providing the names of members of the Editorial Board is an essential criterion to indicate the quality of a journal. For example, the Scientific Electronic Library Online [SciELO] (2004) uses this criterion to select journals to be included into its collection.

In addition, Trzesniak (2009) reminds that an Editorial Board is one of the guarantees of a journal's scientific credibility. This group is responsible for assisting the editor in the decision making process regarding the publication of original articles. Furthermore, as suggested by Trzesniak (2009, p. 97):

Preparing a scientific journal requires a good deal of dedication and implies giving your time to improving the work of other researchers, in detriment to your own projects. It is only with the involvement of a team that it is possible to do this without the editor being seriously undermined in his own career as a researcher.

Thus, it may be perceived how important an Editorial Board and a well-structured team are to ensure the good management of a scientific journal and, consequently, to guarantee its quality as an instrument for information flow.

When analysing the information of Graph 1 about the Editorial Board, it may be seen that most of the members are affiliated to institutions other than to the one that edits the journal. This factor shows the quality of the journal, as stated by Trzesniak (2009, p. 90):

[...] this is a necessarily multi-institutional collegiate, formed by specialist researchers evenly distributed, both scientifically (that is to say, by the areas and sub-areas of the sciences and technology to which the journal is dedicated), as well as geographically (that is to say, by the regions where the journal intends to circulate representatively). [Author's emphasis]

The diversity of the members of the Editorial Board is one of the criteria for journal assessment used by at least three institutions: Thomson Scientific (2009), to accept journals on the ISI Web of Science data base; SciELO (2004), to accept journals into their collection; in the Brazilian Association of Scientific Editors (Associação Brasileira de Editores Científicos [ABEC], 2009), to select scientific electronic journals for financing.

Similar criteria were also found in the studies of: i) Krzyzanowski and Ferreira (1998) when assessing Brazilian scientific and technical journals; ii) Ali, Young and Ali (1996), which outlined a checklist of quality aspects used for financing decisions; iii) Trzesniak (2006), as one of the criteria used to assess journals in the Education area.

As shown on Graph 2, the academic qualification of almost all the members (97%) of the Editorial Board of the journals surveyed is appropriate, since they hold doctorates. In addition, the productivity of the great majority (96%) of the members that took part in this sampling exceeds or equals the average productivity of the field of knowledge to which the journal belongs. This could mean that the members of the editorial board that were surveyed have a sufficient level of involvement in scientific and research activities, which is one of the factors that is considered necessary for members of a professional group who will be responsible for maintaining a journal's quality standard.

Even though these results are positive, it is necessary to remember that 38% of the journals did not provide information about the members of their Editorial Board, which reduces the overall quality of these journals. If the analysis had covered all of the journals, the results would show that, taken together, these have less quality than the minimum required.

4.2 Authors of published articles

Graph 3 shows the percentage distribution of authors who have had articles published in the journals included in this sampling, distributed by affiliation and nationality.

The results reveal that the percentage of outside authors (those not affiliated to the editorial institution) exceeds the absolute predomination (70%), which is the minimum recommended by Ferreira (2005) to avoid endogeneity. In accordance with the author, Trzesniak (2006) recommends that more than 60% of the articles published should be by authors from institutions other than the one editing the journal, not including foreign authors. In the case of the latter, Trzesniak (2006) recommends the publication of at least 10% of articles by foreign authors over the last three years. The Thomson Scientific (2009) establishes as a more general criterion, the international diversity of authors whose articles are published in the journal.

With respect to the nationality of the authors, it should be underlined that this assessment was carried out with the first authors only and in the year of 2009. This explains, therefore, why there are not the 10% of foreign authors, as recommended by Trzesniak (2006). However, the journals showed that 5% of their authors were foreign nationals, which can be seen as a good proportion of the total surveyed.

4.3 General data of the journals surveyed

As explained in the methodology, journals included in the sampling should necessarily be classified in their respective areas of knowledge. However, as can be seen in Graph 4, some areas were represented with more journals than others.

This difference may also be explained by other aspects, such as the publication standards of areas of knowledge. Mueller (2005) carried out a survey to ascertain the preferred publication channels for each area of knowledge. She collected data from the tables produced by CAPES from 1995 to 2002, checking publications in (national and foreign) journals, proceedings (national and foreign), books and book chapters and concluded that:

Researchers in the areas of Exact, Natural and Biological Sciences prefer to publish their work in foreign journals and very little through other channels; Researchers in Health Sciences prefer to publish in national journals, but also publish in foreign journals, rarely using other channels; Researchers in Engineering prefer to publish their work in conference papers, in a far greater number that they publish in journals; Researchers in Applied Social Sciences prefer to publish their work in national journals and books. However, they also publish, though on a lesser scale, in foreign journals, national events and in book chapters; Researchers in Human Sciences, Linguistics, Language Studies & Arts prefer to publish their work in national journals and in book chapters.

Therefore, it is interesting to note that the areas of Mathematical Sciences and Natural and Biological Sciences probably have the least number of journals registered at SEER, since preference is given to publishing in foreign journals. As a result, fewer journals on these subjects are published and edited in Brazil. A similar fact occurs in the areas of Engineering and Computational Sciences, which prefer to publish their work in conference papers.

These findings corroborate studies undertaken by Leite, Mugnaini and Leta (2011), which propose a new approach to investigate scientific productivity. The International Publication Ratio (IPR) was developed to allow for a distinction to be made between groups with different publication tendencies. Using the curriculum Lattes data bank, the authors gathered information about those within the Brazilian scientific community holding doctorate degrees, including their areas of knowledge, affiliations and publications. A total of 34.390 researchers had their curriculums analyzed and their publications were classified into five groups according to the International Publication Ratio (IPR): (1) highly international (with between 80.1-100% of international publications), (2) mainly international (with 60.1-80%), (3) intermediate (with 40.1-60%), (4) mainly national (with 20.1-40%) and (5) highly national (with 0-20%). The IPR data was linked to the researchers' areas of knowledge.

Leite, Mugnaini and Leta (2011) found evidence that international performance is a variable that is dependent on the field of knowledge. Areas dedicated to questions of international interest, such as Biology, Engineering, Exact and Earth Sciences, present a greater fraction of researchers with a high IPR. However, this is not the same for areas that are essentially devoted to questions of local and national interest. Using the IPR offers a good example of the importance of the idiosyncrasies that exist in each field as critical factors to be considered when comparing performance in different areas, "within a scenario where general evaluation determines resources destination" (Leite, Mugnaini & Leta, 2011).

To observe and to analyze the age distribution of journals involve a sustainability analysis, which, in spite of being an important area of investigation, was not part of the focus of the present study. As can be seen on Graph 4, a greater number of journals have been edited for between 4 and 20 years and fewer journals between 21 and 50 years. However, it is not possible to state that fewer journals survive longer than twenty years. This is an interesting hypothesis to be studied later on.

The age variable was used to evaluate whether older journals, or rather, more consolidated journals, had a higher H-index. That is, if the impact of a journal increased with age. In order to ascertain this factor, it was necessary to calculate the Pearson's Correlation coefficient, which produced a result of 0.39. The correlation of the H-index of the journals with their age can be represented by the following dispersion graph:

It may be said that a moderate positive correlation exists, which shows that there is a tendency for the H-index to increase as the journals get older. However, in the case of this sampling, there are exceptions – journals that do not follow the same tendencies and have a lower H-index with increased age. So, it is not possible to say with certainty that older journals have greater impact. This also depends on other factors that were not studied in this research.

Graph 7 shows the language in which the journals are published. This information was gathered from the journals websites, generally from the section stating the instructions to authors, where information is provided about the languages the journal accepts to publish an article.

The survey sought to assess the correlation between the publication languages with the H-index of the journals themselves. It was assumed that the journals that published in the languages most widely spoken in the world would have greater reach and visibility, and potentially greater impact, since their articles could be read by a greater number of people.

In order to calculate this correlation, a classification was made of the principle languages used by the journals being surveyed, from the language that was the least spoken to the one that was spoken the most, in accordance with the number of native speakers, and based on information taken from the book Ethnologue: languages of the world (2009). The final classification was as follows: 1) Portuguese, 2) English, 3) Portuguese and English, 4) Portuguese and Spanish, 5) Portuguese, English and Spanish, 6) Portuguese, English, Spanish and French.

The Pearson Correlation obtained was -0,038. This means that there is practically no correlation between the variables; that is to say, the values are almost independent of one another. However, conclusive statements cannot be drawn from this analysis. The information related to the languages of the articles, as supplied by the journals, does not mean that the specific languages have actually been used in publications on a regular basis.

When analyzing the publication format of the different journals, our research material shows that 85% of the journals surveyed are edited both in printed and electronic format and that 15% are edited only by electronic means. It should be emphasized that five of the 12 electronic journals have migrated; that is to say, these were previously published in print and then opted to retain only their electronic format. Furthermore, it should be said that two of the journals that are only published electronically (and which have not migrated from a printed format) have an H-index of 5 and are among the 27% of journals with the highest H-index. These journals belong to the areas of Biological and Health Sciences.

With regards to the criteria for formatting and normalization, Graph 8 shows our data.

The journals were classified on a scale ranging from "poor" to "good", in terms of clarity and completeness of the instructions given to authors regarding the journal's rules about formatting and normalization. It was observed that 68% of the journals obtained the best scores in terms of formatting and normalization. This means that these journals presented their instructions in a clear and comprehensive manner, which made it easier for the authors to understand and to avoid any uncertainties and, thereby, any errors when submitting their manuscripts. Thus, the editorial process can be quicker, since it is probable that few articles would have to be returned to their authors because of errors in formatting and normalization (identified in the first stage of the process, during the preliminary assessment by the editors).

Examples of good presentation of rules for formatting and normalization are as follows: indication of the norms to be followed for formatting and normalization (ABNT, APA, Vancouver, etc.); indication of how articles should be structured (for example, standard names to be used for section titles); minimum and maximum number of pages; format of the archive (.pdf, .doc, .rtf etc.) and means by which it should be sent; indication of how figures and tables should be positioned; languages accepted for publication; examples of references and citations; other information specifically related to the journal. Several of the journals that were surveyed provided a model for each type of document (article, literature review, book review etc.) together with instructions on how to fill it out.

The journals classified as "Reasonable" did not provide all of the information necessary for an author to fully understand the precepts for formatting and normalization. That is to say, they present the rules, but fail to explain these clearly (failure to provide all information necessary to ensure a clear understanding) and lack clarity (do not provide information in a didactic and understandable way), making it difficult for the authors to understand. The journals that were classified as "Poor" provided no rules for formatting and normalization.

In that which refers to the assessment of the articles, Graph 9 shows that only a small minority of the journals (17%) fulfill this requisite adequately.

The journals classified as "Good" explain how the whole assessment process for written works is carried out and, in addition, indicate the criteria used in these assessments. Those classified as "Reasonable" only provide information about the assessment process, that is, which actors in the editorial assessment process handle the article, how many reviewers assess the work, what happens if the evaluations are contradictory and what procedures an author should follow once he/she receives feedback on an article they have submitted. The journals classified as "Poor" specify neither the process nor the assessment criteria, or provide any clear or satisfactory explanation about how the articles are evaluated.

Stumpf (2008) studied journals in the area of Communications and found that reviewers do not normally receive a list of items that need to be observed in the articles they evaluate. In general, the editors ask for a descriptive evaluation, in the format of an essay, highlighting both the strong and the weak points in the work, as well as suggestions for improvements.

A descriptive assessment is obviously important, since it makes it possible to explain in greater detail the points evaluated in the article. However, an assessment should also be made by using a checklist, with specific points that need to be observed in the article being reviewed. Such a procedure would make it possible to attain a greater level of standardization in the assessment process, as well as making it easier for the reviewers to carry out their appraisal, since this will enable them to learn which points are essential for articles to be accepted by the journal in question.

In addition, specifying the criteria to be observed in the assessment and approval of the articles, also enables the authors to become more aware of the most important points they need to bear in mind to ensure that their articles are accepted for publication. This can help avoid articles having to be passed back and forth between the journal and the author, thereby making the process of evaluation much quicker.

With regards to the date of the arbitration process, the results of the survey indicate that 73% of the journals provide specific dates for the receipt and acceptance of each article, while 27% of the journals did not do so. As well as being considered an important criteria in the assessment process from the point of view of the promotion agencies and organizations that maintain data bases and indexers, to publicize these dates shows how concerned editors are about the agility of the process to assess and publish the articles, as well as with the isonomy between authors in relation to deadlines.

Graph 10 classifies journals in accordance with the percentage of inedited works published in 2009.

It can be seen that only 4% of the journals published less than 40% of inedited works, which Ferreira (2005) considers to be the minimum requirement for scientific journals of quality. However, in 2009, the great majority of the journals (96%) published more than the minimum number of inedited manuscripts. This means that these journals excel in publishing innovative articles in their respective areas of knowledge.

Graph 11 shows that 73% of the journals have a 0 to 4 H-index and 27% have a 5 to 15 H-index during the period from 2007 to 2009. These results may mean that the majority of the journals do not have a significant impact on the scientific community within their own areas of interest. However, this low H-index may be explained by other factors, such as the journal's visibility, age, indexation, and so on.

In addition to the raw data concerning the H-index, a comparison was also made between the H-index averages for two distinct groups. The first group is composed of the following areas: Agronomic and Veterinarian Sciences, Biological Sciences, Mathematical and Natural Sciences, Medical and Health Sciences, and Engineering and Computational Sciences, involving a total of 34 journals. The second group is composed of the areas of: Humanities, Applied Social Sciences, Language Sciences, and Arts, involving a total of 44 journals.

The t-test was used to compare the H-index average of both groups, since both samplings achieved sufficient observations. The null hypothesis and the alternative one are, respectively, as follows:

H₀→ Average of Group 1 = Average of Group 2. The difference between the averages observed in both groups can be justified by casual factors.

H₁→ Average of Group 1 ≠ Average of Group 2. There is a real difference between the averages of both groups.

Statistics and data needed to accept or reject the Null Hypothesis (H₀) were calculated using the Excel software. The following table shows the results:

The test carried out returned a significance probability "P" of 0,000873568 which is less than the 0.05 significance level adopted. Therefore, real differences exist between Group 1 and Group 2 in terms of their H-index averages. This test showed that journals in the areas of natural sciences, mathematics and health have a greater impact factor, when compared to journals in the areas of humanities, social sciences and language studies. This fact may be explained by different factors.

One of these factors is that, in Group 1, there are usually a far greater number of co-authored articles, where it is quite common to have articles written by more than six authors. However, in Group 2 this is not a common factor, since it is more usual to find articles produced by only one author. It was also observed that articles in Group 1 consist of only a few pages, whilst those in Group 2 are generally much longer. The fact that articles in Group 1 have fewer pages and are written by more authors means that there is a higher production of articles. Consequently, by adopting this practice, there is a greater chance of several articles being quoted more often and, therefore, having a greater impact on the scientific community.

5 CONCLUSIONS

The main objective of this study was to assess the quality of Brazilian scientific journals that use the SEER system. The results of this research permitted to obtain relevant data that should be taken into account to ensure that these journals can become quality journals and be recognized as having real scientific value. This is because this is the only way they will gain greater visibility and acceptance within the academic milieu which, in turn, implies that they will have a far greater impact.

The results obtained make it possible to identify both the positive and the negatives aspects in relation to the journals that were surveyed.

Positive aspects:

The journals included in the sample adequately fulfilled the criteria considered for authors evaluation;

The Editorial Board of 62% of the journals is considered to be productive, with good qualifications and with few members affiliated to the editing institution;

The majority (68%) of the journals present clear and full instructions related to the formatting and normalization of articles;

The majority (73%) provide information in their articles about the dates for the process of arbitrage and, principally the dates of the submission and acceptance of articles;

Most of the journals (63%) publish more than 40% of inedited articles;

Negative aspects:

Only 17% of the journals provide information about how the assessment process is carried out and what criteria is used to evaluate the articles;

38% of the journals provide no information about the members of their Editorial Board;

Most of the journals have a low H-index.

Other aspects identified:

There is a moderate positive relation (0,39) between the increase in a journal's H-index and its age;

There is no correlation between the H-index and the language in which the articles are published;

There is a significant difference between the average H-index in the areas of natural sciences, mathematics and health when compared to areas of social and human sciences and linguistics.

The results of this survey make it possible to state that the journals studied only partially met the quality criteria as defined for this study. The journals attained good results in questions relating to: authors, formatting and normalization, dates for the arbitration process and percentage of first-time articles. However, the survey obtained a worse result than expected in the criteria established for the Editorial Board and the Assessment Process for articles.

The low H-index factor of the majority of the journals cannot be seen as a negative factor. This is because over 70% of the journals have been edited for less than 21 years, and the H-index is related, albeit moderately, to the age of a journal.

However, it is important to emphasize the possibility that the criteria that were established for the purpose of this sampling have influenced the journals selection. It is also worth highlighting that this survey only evaluated a relatively small sample, considering the total number of journals that exist within the system. It would be interesting, therefore, to carry out a survey with a sample more representative of the system as a whole. However, this survey has produced results that will serve as a point of reflection for scholars in the area, as well as highlight aspects that may be studied in more depth in the future.

Sely Maria de Souza Costa

PhD in Information Science at Loughborough University

Senior Lecturer at Universidade de Brasília (UnB)

Faculdade de Ciência da Informação (FCI)

Endereço: Campus Darcy Ribeiro, Asa Norte

70910-900 – Brasília, DF, Brasil

Telefone: (61) 33071205 Fax: [phone]

Email: [e-mail]

Manuscript first received 28/10/2010

Manuscript accepted 24/02/2012

References

• Alberts, B. (2002). Engaging in a worldwide transformation: our responsibility as scientists for the provision of global public goods. Annual Meeting of the National Academy of Sciences, Washington, D.C., USA, 139.
• Barbetta, P. (1999). Estatística aplicada às Ciências Sociais (3a ed.). Florianópolis: UFSC.
• Braun, T., Glanzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169-173.
• Calvert, P. J., & Zengzhi, S. (2001). Quality versus quantity: contradictions in LIS journal publishing in China. Library Management, 22(4/5), 205-211.
• DigitalKoans (2010). Archive for the Serials Crisis Category. Recuperado em 22 outubro, 2011 de http://digital-scholarship.com/digitalkoans/category/serials-crisis/
• Ferreira, A. A., Neubhaher, B., Reis, E., & Gomes, M. S. (2009, setembro). Avaliação de periódicos científicos on-line na área do direito. CRB-8 Digital, 2(2), 12-26.
• Ferreira, M. C. G., & Krzyzanowski, R. F. (2003). Periódicos científicos: critérios de qualidade. Pesquisa Odontológica Brasileira, 17
• Ferreira, S. (2005). Critérios de qualidade para as revistas científicas em comunicação. In S. Ferreira & M. Targino (Orgs.). Preparação de revistas científicas: teoria e prática. São Paulo: Reichmann & Autores.
• Harnard, S. (1995). Electronic scholarly publication: quo vadis? Serials Review, 21(1), 70-72.
• Harnad, S. (2011). Open access is a research community matter, not a publishing community matter. Lifelong Learning in Europe, 16(2). Recuperado em 22 outubro, 2011 de http://eprints.ecs.soton.ac.uk/22403/
• Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Gingras, Y, Oppenheim, C., Stamerjohanns, H., & Hilf, E. (2004). The access/impact problem and the green and gold roads to open access. Retrieved September 24, 2007 from http://eprints.ecs.soton.ac.uk/9939/1/impact.html
• Harzing, A. W., & Van der Wal, R. (2009). A Google Scholar H-Index for journals: an alternative metric to measure journal impact in economics and business. Jounal of the American Society for Information Science and Technology, 60(1). Retrieved March 15, 2011, from http://onlinelibrary.wiley.com/doi/10.1002/asi.20953/abstract
• Harzing, A., & Van der Val, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8, 61-73. Retrieved June 21, 2010 from http://www.int-res.com/articles/esep2008/8/e008p061.pdf
• Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, USA, 102(46), 16569-16562. Retrieved August 15, 2011, from http://www.pnas.org/content/102/46/16569.abstract
• King, D., & Tenopir, C. (1998). A publicação de revistas eletrônicas: economia da produção, distribuição e uso. Ciência da Informação, 27(2), 176-182. Recuperado em 30 outubro, 2007 de http://www.scielo.br/pdf/ci/v27n2/king.pdf
• Krzyzanowski, R., & Ferreira, M. (1998). Avaliação de periódicos científicos e técnicos brasileiros. Ciência da Informação, 27(2), 165-175. Recuperado em 5 dezembro, 2008 de http://revista.ibict.br/index.php/ciinf/article/viewArticle/357
• Leite, P., Mugnaini, R., & Leta, J. (2011). A new indicator for international visibility: exploring Brazilian scientific community. Scientometrics, 88(1), 311-319. Retrieved August 26, 2011, from http://www.springerlink.com/content/l855606h71611865/
• Lewis, P. (Ed.). (2009). Ethnologue: languages of the World (16a ed.). Dallas: SIL International. Retrieved June 12, 2010 from http://www.ethnologue.com
• Luethi, M. (2008). Self-governance in science: what can we learn from FOSS? DIME working papers, number 34. Recuperado em 22 outubro, 2010 de http://www.dime-eu.org/files/active/0/WP34-osterlohLuethi_IPROSS-up.pdf
• Machado-da-Silva, C., Guarido Filho, E., Rossoni, L., & Graeff, J. (2008). Periódicos brasileiros de Administração: análise bibliométrica de impacto no triênio 2005-2007. RAC-Eletrônica, 2(3), 351-373. Recuperado em 10 fevereiro, 2009 de http://www.anpad.org.br/periodicos/arq_pdf/a_821.pdf
• Márdero Arellano, M. A., Santos, R., & Fonseca, R. (2005). SEER: disseminação de um sistema eletrônico para editoração de revistas científicas no Brasil. Arquivística.net, 1(2), 75-82. Recuperado em 16 outubro, 2007 de http://dici.ibict.br/archive/00000897/02/AN-2005-33[1].pdf
• McCartan, P. (2010). Journals and the production of knowledge: a publishing perspective. British Journal of Political Science, 40(2). Recuperado em 22 outubro, 2011 de http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=7604012
• McGuigan, G., & Russell, R. (2008). The business of academic publishing: a strategic analysis of the academic journal publishing industry and its impact on the future of scholarly publishing. Electronic Journal of Academic and Special Librarianship, 9(3). Recuperado em 22 outubro, 2011 de http://southernlibrarianship.icaap.org/content/v09n03/mcguigan_g01.html
• Meadows, A. (1999). A comunicação científica. Brasília: Briquet de Lemos.
• Meneghini, R. (1998). Avaliação da produção científica e o Projeto SciELO. Ciência da Informação, 27(2), 219-220.
• Mueller, S. (2005). A publicação da ciência: áreas científicas e seus canais preferências. DataGramaZero, 6(1). Recuperado em 15 maio, 2010 de http://dgz.org.br/fev05/Art_02.htm
• Mugnaini, R., & Strehl, L. (2008). Recuperação e impacto da produção científica na era Google: uma análise comparativa entre o Google Acadêmico e a Web of Science. Encontros Bibli, 13(n. esp). Recuperado em 25 abril, 2010 de http://www.periodicos.ufsc.br/index.php/eb/article/view/1127
• Scientific Electronic Library Online (2004). Critérios SciELO Brasil: critérios, política e procedimentos para a admissão e a permanência de periódicos científicos na coleção SciELO Brasil. Recuperado em Outubro de 2004 de http://www.scielo.br/criteria/scielo_brasil_pt.html
• Stumpf, I. (2003). Avaliação das revistas de comunicação pela comunidade acadêmica da área. Em Questão, 9(1), 25-38. Recuperado em 10 maio, 2010 de http://www.seer.ufrgs.br/index.php/EmQuestao/article/view/57/17
• Stumpf, I. (2008). Avaliação pelos pares nas revistas de comunicação: visão dos editores, autores e avaliadores. Perspectivas em Ciência da Informação, 13(1), 18-32. Recuperado em 12 junho, 2010 de http://www.scielo.br/pdf/pci/v13n1/v13n1a03.pdf
• Tenopir, C., & King, D. (2001). A importância dos periódicos para o trabalho científico. Revista de Biblioteconomia de Brasília, 25(1), 15-26.
• The Conversation Media Group. (2011, September 28). Princeton goes open access to stop staff handing all copyright to journals – unless waiver granted. Retrieved 03 November, 2011, from http://theconversation.edu.au/princeton-goes-open-access-to-stop-staff-handing-all-copyright-to-journals-unless-waiver-granted-3596
• Thomson Scientific. (2009). The Thomson Scientific journal selection process Retrieved 05 february, 2009 from http://www.thomsonreuters.com/business_units/scientific/free/essays/journalselection/
• Trzesniak, P. (2009). A estrutura editorial de um periódico científico. In A. Sabadini, M. Sampaio & S. Koller (Orgs.). Publicar em psicologia: um enfoque para a revista científica São Paulo: Associação Brasileira de Editores Científicos de Psicologia.
• Trzesniak, P. (2006). As dimensões da qualidade dos periódicos científicos e sua presença como instrumento da área da educação. Revista Brasileira de Educação, 11(32), 346-361. Recuperado em 20 novembro, 2008 de: http://www.periodicos.ufrgs.br/admin/sobrelinks/arquivos/As_dimensoes_da_qualidade.pdf
• Yamamoto, O. & Costa, A. (2009). A avaliação de periódicos científicos brasileiros da área da psicologia. In A. Sabadini, M. Sampaio & S. Koller (Orgs.). Publicar em psicologia: um enfoque para a revista científica São Paulo: Associação Brasileira de Editores Científicos de Psicologia.

Appendix A

1

Scientific Electronic Library Online. According to Meneghini (1998, p. 219), "an instrument to enable national production to become more visible and accessible via electronic means and, at the same time, to create a data base by which it will be possible to assess the country's scientific output and increase its international visibility."

2

An estimate of the minimum age of the journals was based on criteria consolidated in the reference documents for each area of knowledge published on the CAPES website.

Publication Dates

• Publication in this collection
  15 May 2012
• Date of issue
  Apr 2012

History

• Received
  28 Jan 2010
• Accepted
  24 Feb 2012