Assessment of scientific programs: a necessary procedure for Brazilian scientific policy - the Young Investigator Program of the State of São Paulo Research Foundation

HUMAN SCIENCES

Assessment of scientific programs: a necessary procedure for Brazilian scientific policy – the Young Investigator Program of the State of São Paulo Research Foundation

Carlos A. de Pian^{I, III}; Rogerio Meneghini^{I, II, III, *}

^IFAPESP, Rua Pio XI, 1500, 05468-901 São Paulo, SP, Brasil

^IIBIREME - PAHO - WHO, Latin American and Caribbean Center on Health Sciences Information

Rua Botucatu, 862, 04023-901 São Paulo, SP, Brasil

^IIIUNIFESP, Department of Information Technology in Health, Graduate Studies Rua Botucatu, 862, 04023-901 São Paulo, SP, Brasil

^{Correspondence to}

ABSTRACT

Programs of Science and Technology research have grown significantly in Brazil over the last decades. Until the 1980s the so-called undirected programs, without specific goals and requiring only scientific merit, prevailed. The few programs with defined goals in this period were never objectively assessed. The same situation occurred in developed countries. In the early 1990s, the assessment of programs supported by public funding became mandatory in US and some European countries. In Brazil, program assessment has so far not been implemented yet. The Fundação de Amparo à Pesquisa no Estado de São Paulo (FAPESP) (Brazilian funding agency) Young Investigator (YI) Program is in its eleventh year, with approximately eight hundred projects awarded. Although it is free-demand based as concerns areas of knowledge, it has specific goals : (1) conceding grants to YI in view of the balance between funding, merit and real needs so as to enable satisfactory working conditions in the short term, (2) providing priority for institutions with a less extensive background in research, (3) granting a special fellowship to YI with no employment connection and (4) introduction of new research fronts in centers with a sound research background. This assessment provided evidence for the achievement of first three goals. The fourth one is still pending on additional data requiring survey assessment. Actions in this direction are recommended.

Key words: evaluation, assessment of scientific programs, scientific production, statistical sampling.

RESUMO

Programas de Ciência e Tecnologia cresceram significativamente no Brasil nas últimas décadas. Até a década de 80 os assim chamados programas não dirigidos, sem metas específicas e requerendo apenas mérito científico prevaleciam. Os poucos programas com metas definidas neste período não foram jamais objetivamente avaliados. A mesma situação ocorria nos países desenvolvidos. No início da década de 90, a avaliação de programas dependentes de recursos públicos tornou-se mandatória nos Estados Unidos e em alguns países europeus. No Brasil a avaliação de programas não foi até agora implementada. O programa de Jovem Pesquisador (JP) da FAPESP está em seu décimo primeiro ano, com a concessão de aproximadamente 800 projetos. Embora não seja dirigido em termos de áreas específicas de conhecimento, ele tem metas específicas: (1) conceder auxílios a JP tendo em vista um balanço entre fundos, mérito e reais necessidades para permitir condições satisfatórias de trabalho dentro de um curto período, (2) conceder prioridade à instituições menos consolidadas em pesquisa, (3) conceder bolsa especial a JP sem vínculo empregatício à instituição e (4) introduzir novas frentes de pesquisa em centros cientificamente bem consolidados. A presente avaliação forneceu evidências de que as três primeiras metas foram alcançadas.A quarta está ainda pendente de dados adicionais que requerem avaliação por enquete. São sugeridas ações nesta direção.

Palavras-chave: avaliação, avaliação de programas científicos, produção científica, amostragem estatística.

INTRODUCTION

In 1993 the American Congress passed the "Government Performance and Results Act" which required that all federal agencies, including those that support scientific and technological research, defined goals for assessment of performance and reported the results every year. This measure arose from societal pressure for a payback of its taxes. This act gained new momentum over the following years but agencies with technology goals (for example, the National Science Foundation, National Institute of Health) were able to add a supplementary format which included a descriptive approach for performance goals and required peer assessment (Cozzens 1997) while using numeric indicators, bibliographic metrics, patents, and even "user satisfaction" metrics (Cozzens 1995).

Similar initiatives had been implemented in Europe since the beginning of the 1990s. The English government was also under pressure to account for its investment in Research & Development (R&D, and established a landmark to this end through an academic department evaluation process defined in the early 1990s on the "Research Assessment Exercise", program which included guidelines for the distribution of funds to public universities at the national level (McNay 2003). At the end of the 90s, the English Department of Health established a multidimensional procedure to define categories of benefits. Its health economics research group assessed the impact of the funds invested in R&D on public health services, which entailed large-scale assessment of research programs (Buxton et al. 2000).

The process of assessing R&D programs grew increasingly important and new experiences have emerged. Multidimensional analysis has become paramount in reviewing the performance and achievement of goals of large R&D programs. This frequently requires people with competencies in different areas, ranging from basic research to economics, who are able to interact and fully understand complex interfaces and achieve more reliable results.

In Brazil there is little experience in prospecting to establish R&D programs and assess its results. We have learned how to deal with scientific assessment in general and with the evaluation of scientific projects in particular, especially as compared with other Latin American countries. As regards the direction of programs, the most important experience might have been carried out by FAPESP, in the Biochemistry-FAPESP (Bioq-FAPESP) Project in the beginning of the 1970s. This project proposed the fast advancement of research in biochemistry and had international advice to assess individual projects (Perez 2002). It had a defined goal and its assessment included prominent investigators and even a Nobel Prize winner, an unprecedented fact at that time. This program is mentioned as a paradigm and it was so to a certain extent. However, its main intended goal was to enable the formation of a critical mass of investigators in biochemistry in the State of São Paulo. The project was never actually evaluated, although the community's perception is that this goal has been achieved.

In the 1980s Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) introduced aresearch program on Chagas Disease which was notevaluated as to its cost/benefit ratio. It is therefore impossible to say how much this program impacted the training of the competent Brazilian scientific investigators in this area nowadays.

The same can be said of the CNPq - National Academy of Science of the United States Program which ran in the 1970s and 1980s in the area of chemistry (Toma 2005). This program was extremely original since it involved the permanence of prominent North American professors in Brazil for relatively long periods thus allowing them to provide effective guidance to Brazilian students.

In all of these programs, the assessment of results was carried out through peer perception, which was also the way it was carried out in other countries. This delegation of power was accepted and arose from the great prestige enjoyed by scientists after World War II, especially due to the development of nuclear weapons. In debates between scientists and the United States government, the conclusion emerged that science would best serve the public interest if scientists, as private citizens, had the power to determine how funds should be spent to support scientific activity (Irvine and Martin 1980). This era came to an end in the 1980s especially because of the high public investment in R&D in the United States and the need to provide taxpayers with a clear picture of the benefits arising from such funding.

In Brazil, it is known that taxpayer pressure is not as strong. Directed programs have been implemented by support agencies and ministries but the discussion is conducted by technicians and scientists. Although key decisions have to be validated by the Brazilian Congress and approved by the President of Brazil, the programs are not discussed in depth in these instances. As a natural consequence, the assessment of program results is not a requirement and is not carried out routinely.

FAPESP has always been acknowledged as anagency that supports high quality research. Until recently it focused mainly on the so-called "counter projects" that is, undirected and demand-based projects: support grants, thematic grants and fellowships. The way these projects were reviewed and assessed as to their development and conclusions made FAPESP's model of scientific quality management recognized. More recently directed programs have focused on genomics, technology innovation, establishment of excellence centers and public policies among others (Perez 2002). The Young Investigator (YI) Program itself, object of this study, although open to free demand, had clear goals that went beyond the assessment of good scientific standards. It was created to offer concrete possibilities of progress in technology and/or science to young investigators with a high potential and who might be unemployed (FAPESP 2006). The main goals are stated below:

1 – To provide conditions for young PhD's, whose potential has been proven by relevant scientific publication and quality projects, to dedicate themselves fully to their project.

2 – To encourage YIs to go preferably to institutions and research centers which are not yet fully consolidated as such so that their research may contribute to raise the scientific standard of these institutions.

3 – The YIs should not be prevented from going to institutions with a stronger research background if their presence in such institutions should enable these institutions to have an edge in terms of new lines and fronts of research.

4 – The program is to benefit not only investigators who are employed, but also those who are not, through a special fellowship, even without a prior definition as to the institution they are going to when their application is accepted.

This program was designed in 1995 and implemented in 1996. It was a pioneer program in a way. A similar program was only offered by major research agencies in the United States in July 1999, under the coordination of the National Science and Technology Council (Nat Sci Tech Council 1999) and under the Presidential Early Career Awards for Scientists and Engineers, established by the United States President.

This pioneer program had peculiar characteristics and one might ask why. One important point is that the post-doctoral programs in the United States are quite unique in that after finishing their PhD studies, graduates are encouraged to pursue their own projects, normally with a grant from the group they have chosen which is different from the group where they did their PhD studies. This post-doctoral project requires great effort on the part of investigators to make progress on their own. It is the time for young researcher to affirm themselves. Later, as junior faculty (assistant professor) in another institution, pursuing a career though not yet in a stable position under the tenure tracking system, researcher have to look for funding and compete with researcher with a strong track record. The funding supposedly provides them with the conditions to set up an infrastructure, usually for a new research line in the department. The "Presidential Early Career Awards for Scientists and Engineers" project, established by the United States President, was created to make this path easier so long as the candidate is recognized as a highpotential investigator. There is great prestige attached to the award, and institutions strive hard to assist candidates in the submission of their proposal.

The situation is different in Brazil. Prior to the university reform in 1969, young professors would associate with a senior professor early in their career, usually a full professor, and collaborate in the line of research the senior professor was engaged in. Although the possibility of breaking this link was allowed by the reform, few investigators have been able to work in a more independent manner in the beginning of their career. In the eighties, investigators were mostly "endogenous" (Meneghini 1996) while in the nineties changes have been observed which will be discussed in this study. One of the obstacles for young investigators to achieve independence was that it was difficult for them to set up an infrastructure that could allow them to start working according to their competencies in a shorter term. This was one of the goals of the YI program. There was an understanding that there should be extra incentive for them to become independent in the short term. The situation here was different from the United States where the independence of young investigators was recognized after a long post-doctoral period (four years on average), and where the greatest problem was the competition for large grants with well established groups, with a high level scientific production.

In addition to being a researcher able to present high scientific outputs, a YI Program recipient was to be regarded as an important asset by a less consolidated research center, since young investigators would be able to provide an infrastructure for project development to match what the institution offered to him. YI recipients would also have a reputation of scientific independence assessed by FAPESP.

The assessment of merit and the funding to meet YI projects have been widely discussed by the Scientific Management Committee of FAPESP (DC-FAPESP) and gave rise to concepts that were cascaded down to the coordinators and advisors who acted in an independent though consistent manner. This was a key point as the YI program, despite having some defined goals, was not theme-based, but rather allowed the free choice of topics.

The implementation of a program where the recipient of a YI grant did not have to be an employee of an institution was a real innovation in Brazil, and as far as we know, it was an innovative approach worldwide as well. Several aspects warrant the granting of special fellowships to young investigators. As we will show, a large percentage of the candidates approved in the YI category with no employment relationship had at some time a post-doctoral grant in Brazil or abroad. Despite their excellent resume - an item which is also assessed in this study - they had not yet found a position.

The possibility of entering these programs was very good for these young people as it allowed them to negotiate with the institution they wanted to work at, with a fellowship and grant in hand, on how to collaborate with the institution's graduation studies and undergraduation teaching activities. Both activities are allowed by FAPESP, although some restrictions apply to the latter. This was also advantageous for the institutions as they were able to have a young professor, with the prestige attached to the program, without allocating a position. As we will show, these two facts were observed and others as well which had not been predicted, and which were marked by conflicts and litigation. These other facts should also be taken into account in the continuation of the program.

METHODS

SAMPLE SIZE

The YI program started in 1995 and this survey covered from 1995 through March 2004, when the data were collected and tabulated. During this period 492 YI projects were approved. The collection of data of all the projects would be far too comprehensive, and therefore we projected a sample with a confidence interval of 8% and a level of confidence of 95%. As a consequence, the size of the sample was calculated to include 114 projects (Creative Research System). This means that, assuming a random choice, the result found for a certain indicator should be within a ± 8% error interval and that there is a 95% degree of security that the result is within this interval. For example, if we hypothetically found that 30% of the publications were in English, this means that in the entire population of 492 projects there should be, with a 95% degree of security, 30% ± 2.4% (8% out of 30%) of articles written in English.

STRATIFIED SAMPLE

Stratified sampling is advised when the representation of certain variables within a population is known (Stak Trek). In this sample of 114 three stratifications were performed: 1 - For the representation of sub-categories of YI projects with coordinators with or without employment relationship; 2 - For area of knowledge, according to the FAPESP classification, and 3 - For institutional origin. The proportion of these three variables was established for the population of 492 projects and the sample of 114 projects was developed according with these proportions.

SAMPLING OF RESEARCH SUPPORT (RS) PROJECTS

Research Support is a traditional FAPESP program open to all investigators employed by a São Paulo State Institution. The sample of 113 RS projects, a number equivalent to the YI sample, corresponds to the first RS projects of the recipients, granted in the same period of the YI projects (1995-March 2004). The sampling aimed at reflecting the same proportions of institutions and areas of knowledge as the YI sampling. Therefore, the RS sample does not necessarily reflect the RS project population, but rather mirrors the YI population. It was designed for comparison purposes. This is certainly the statistically valid path for this purpose.

Since both projects, YI and RS, corresponded almost entirely to the first project granted to investigators, within the same time interval, it is no surprise that the recipients had the same academic age, i.e., PhD obtained in 1996 ±2.4 for YI and 1995.9 ±1.7 for RS.

RESULTS

CHARACTERISTICS OF THE PROJECTS SELECTED FOR THE SAMPLE OF THIS STUDY

The projects that make up the sample of the study of the YI and RS programs were selected considering the items described under Methodology. Table I shows the representation per area of knowledge of the projects selected. We were also concerned about including projects starting in 1996 through 2000 to cover projects ended March 2004.

Among the 114 YI projects, 66 were coordinated by investigators with employed by an institution, and 44 had a YI fellowship. The possibility of a project coordinator not having an employment relationship was unique to the YI program, and was first implemented at FAPESP. It allowed investigators to have a special fellowship with a maximum duration of four years.

The 113 project sample selected for RS corresponds to the first RS project approved for each investigator. In this case, all recipients had an employment connection, as established by FAPESP's rules. The selection was performed so as to ensure the same representation per area and per institution as in the universe of YI projects.Table I shows that, while in Biology the number of projects was very high, there were no YI projects in Astronomy and Mathematics, since very few or even no projects were submitted. The percentage in the area of Biology is higher in this program than the average in other FAPESP programs. This will be discussed further below.

The distributions of the YI and RS projects as concerns the institutions involved are shown in Table II. This distribution will also be discussed further below. For now it should be noted that some private universities stand out as to the acceptance of YI projects, especially the Universities of Mogi das Cruzes, São Francisco, Vale do Paraíba and Universidade Paulista. This required a higher or lesser degree of engagement from their management, and the implementation of a new philosophy of work in these private universities, the results of which will be discussed in the final part of this paper.

Among other goals, the YI program aimed at offering conditions to allow young investigators with high potential to move faster towards pursuing a scientific career. This required higher investment especially in projects that required funding for equipment and input. Table III shows the profile of investments in YI projects as compared with RS projects in the samples selected. The investment in YI projects was clearly higher than in RS projects. Figure 1 presents the percentage values of the funding granted in different ranges. For YI projects, most of the grants were in the range of R$ 100-200,000 (Brazilian reais) whereas most RS grants were below R$ 50,000. However, it is necessary to consider that YI projects last four years whereas RS projects last two years. Although most of the funding is spent in the first year, the difference in project terms has probably impacted the amounts granted, that is, longer projects require more funding.

REQUEST ASSESSMENT PROCESS OF THE YI PROGRAM

One question arises: what were the parameters used to assess the potential of candidates in each of the two categories? The need for a stricter assessment by peers for the YI program (see Methodology) was apparent in several indicators. One of them was scientific production at the time of project submission (Table IV). YI candidates had a considerably higher performance in paper publication, especially in international periodicals. This advantage regarding international publications was more observed in the following areas: biology, engineering and physics.

Another indicator was the degree of rigor applied by FAPESP's Scientific Management Committee in the assessment of the merit of the projects submitted. As a rule, three advisors were heard and sometimes more were called in, as happens with the FAPESP Thematic Program. As a result, more submissions were turned down. Table V shows data on the approval of programs per area of knowledge for the year 2004, and it shows that while 57% of RS requests were approved, only 27% of YI requests were approved. Surprisingly the percentage of approvals in the YI program was even lower than the percentage of the thematic program (45%).

This table also shows that there are different trends in different areas. Traditionally submissions in the health area tend to prevail in the RS and Thematic programs (759 and 50 requests respectively, Table V), which is not the case in the YI program where most projects are in the area of biology and engineering (33 and 37 requests respectively, Table V). The reason for this is not obvious. It could be partly due to the fact that projects in biology and engineering require more expensive and diversified equipment for a smaller number of users which justifies more active investment in the program. But other factors may also play a major role and deserve a specific study. Other considerations will be made under Discussion.

The distribution of project grants per programpresents yet another noteworthy aspect as concerns the institutions that resort to them which is shown in Table VI (there is a apparent discrepancy between the total grants in Tables V and VI, which arises from the fact that Table V includes a few grants to institutions which are not included in Table VI). The thematic program, the most prestigious offered by FAPESP, shows a prevalence of Universidade de São Paulo (USP) and Universidade Estadual de Campinas (UNICAMP), with 55% and 23% of the grants (data of 2004 which are, however, close to the averages of all the years). The values found for these institutions drop when the RS program is considered and drop even further when we consider young YI, where the percentages are 28% and 10% respectively. These data reflect the fact that these two universities are the ones with the largest number of consolidated research groups which are not the focus of the YI program. Among other goals, this program was designed to open new research fronts in more needy institutions able to work within the scope of the program. Typically, UNESP and private universities were the most benefited by the program, and in these universities we observed the opposite of what can be observed in USP and UNICAMP. There was a percentual increase when we observe YIs granted and thematic projects granted: from 5% to 20% for Universidade Estadual Paulista (UNESP) and from 3% to 18% for private universities.

This should not lead to the conclusion that projects from different institutions were analyzed with different degrees of rigor. Table VII shows similar rates of approval for institutions of the YI program. What apparently happened was that institutions with a smaller track record in terms of research made more requests for the YI program. For instance, let us compare the number of requests for the YI and thematic programs made by UNESP and private universities. If the requests of these institutions are added up, 21% were made for thematic projects in 2004 and 47% were made for the total YI. Apparently the philosophy of the program was understood by the scientific community and generated different demands.

Another noteworthy indicator was the participation of investigators in other programs prior to receiving the YI and RS grants in post-doctoral programs in Brazil and abroad (Table VIII). The percentage of investigators who participated in post-doctoral programs presents no significant differences, with a slightly larger number of YI's taking part in such programs (78) as compared to RS recipients (69). Interestingly enough however, YIs with fellowship had, in percentual terms, greater presence in post-doctoral programs in Brazil prior to the YI grant. This will be discussed further in the Discussion section.

It is worth noting that the percentages of "endogenous" projects (i.e. the fact that the institution where the YI or RS project was started is the same where the Investigators pursued their PhD) were very low, and were the same for both samples analyzed. Only 12.3% of YI project coordinators started their program in the same institution where they pursued their PhD, whereas this percentage is 12.4% for RS projects (results not shown).

INSTITUTIONAL MOBILITY OF YOUNG INVESTIGATORS

The YI program was the first in Brazil to allow submissions by investigators without an employment relationship with an institution. The idea is that young investigators with a clear potential could, with a considerable grant, negotiate with an institution to try to obtain space and infrastructure to establish their group. This was one of the drivers of the YI program. However, young investigators who were formally employed by institutions were also granted grants. In the sample of 114 recipients, 48 received a fellowship and 66 were formally employed. Table IX shows the institutional destination of the recipients of YI and RS grants in the two samples of projects. It is worth noting that the sample of YI projects reflected the areas and institutions of recipients in the universe of 492 projects. The RS project sample does not represent the universe of the first RS projects, but rather seeks to reflect stratifications concerning institutions, areas of knowledge and academic age, as close as possible to the YI sample.

The first data that stands out in Table IX is a relatively high number of YI projects started at UNESP (21) and private universities (23) in the sample. This makes sense since, as stated above this can be explained by the goals of the program. The fact that Investigators moved from an institution to another in the time elapsed between the granting and the start of the program is also noteworthy. As expected, the mobility of YI recipients was higher than the mobility of RS recipients since part of YI recipients had no employment relationship and pursued possibilities that were frequently available in other institutions (see below). As to YI recipients, 29.2% of them moved to another institution as compared with 19.5% of RS recipients. The high degree of mobility of YI recipients from USP (55%) stands out as compared with the others. However, it is also a curious thing that most YIs from USP who moved to other institutions went to other USP units (59%, results not shown). It is important to note that among those YIs with fellowships that remained in their original institution all but one were eventually hired (data of a survey carried out in April 2006 based on the Lattes resume database).

The degree of mobility of recipients in private universities is quite surprising. Unlike some reports suggest, the percentage of changes was relatively low for YI recipients (17.4%), and was in fact the lowest among the different categories of institutions and lower than the percentage of change of RS recipients linked to private universities.

If more in-depth analysis is to be carried out, the YI projects would have to be subdivided into two sub-categories: YIs with fellowships and YIs with employment relationship (Table X). The participants of these two sub-categories show opposite trends in terms of institutional mobility which is very high (66.7%) for those YIs with fellowship and very low (15.2%) for YIs with employment relationship, and these percentages are even lower than the percentages for RS (Table IX - 19.5%). This is critical to the project and will be further discussed in the Discussion section.

PERFORMANCE OF YI AND RS RECIPIENTS DURING THE GRANT PERIOD

Projects were monitored through the publication of complete papers in periodicals. Table XI shows the numbers of publications for both programs in different institutions. The YI program was divided into YI with fellowship and YI with employee relationship. Especially as concerns international publications, the performance of YI recipients was significantly higher. To make comparison easier, it is more appropriate to take productivity values, presented in parenthesis, normalized for the number of projects. There were 6.3 and 5.1 publications per project for the YI program with fellowship and with employment relationship respectively, whereas for the RS program there were 3.4 papers per project. But it is difficult to compare the number of papers published for the YI and RS programs due to two factors: firstly because YI projects last longer than RS projects (on average 4 and 2 years respectively). In the comparison, this favors YI projects. Secondly, part of the papers is not related with the projects underway but with previous projects whose results, in terms of publications, were published during the term of the projects at hand.

Table IX had shown that most project coordinators had received a post-doctoral grant. Therefore, part of the papers shown in Table XI results from these post-doctoral and doctoral studies. Assessing this effect is not only laborious but difficult to do if we analyze the papers listed on their resumes. This point will be discussed below following the discussion on the scientific production for both programs after the completion of the projects.

PERFORMANCE OF INVESTIGATORS ENGAGED IN YI AND RS PROJECTS AFTER THE END OF THE PROJECTS

The scientific production of Investigators after the end of their projects is shown in Table XII. YI projects are shown separately in two sub-categories: those with fellowship and with employment relationship. Productivity in Brazilian and international periodicals alike is higher for the RS program than for each of the YI sub-categories. This might come as a surprise at first.

Here again we should consider that YI and RS have different terms. Due to the design of the samples, these projects started at the same time and, since the survey was carried out in March 2004, the post-grant period was longer for the RS than for the YI program. This factor increases the number of papers for the RS program. It becomes clear then that the fact that the programs have different terms makes it difficult to make a direct comparison of the productivity of both programs.

One way to better understand what happened was to tabulate the sum of scientific production in both periods, during and after the projects (Table XIII). The period covered by this analysis was the same for both the YI and the RS projects, therefore eliminating the impact of the different duration of projects. We have to admit that the productivity of YI project coordinators was not superior to that of RS project coordinators when the two periods - during and after the project - is considered. The meaning of this important conclusion will be discussed in detail in the Discussion section.

It is important to highlight specific aspects found in Table XIII that will be relevant for discussing productivity differences later. If we consider the international publication of papers, UNICAMP has a significantly higher productivity for RS projects. Even though it contributes less than 10% of the projects of the samples, it impacted the global production significantly. International productivity was higher in YI projects for UNESP and private universities, coincidently those that have a smaller number of consolidated research centers, and for which a higher contribution was expected from YI coordinators.

NUMBER OF MASTER'S AND DOCTORATE DEGREES ADVISED, UNDERWAY OR COMPLETED, PER INSTITUTION, IN MARCH 2004

Table XIV shows the number of graduate students that the YI program allowed to integrate into the researchsystem of the State of São Paulo, a total of 756 in the period between 1995 and 2004. It is worth noting that the sample we are dealing with for this period includes 23% of the universe of YI projects to March 2004. The RS program reached a total of 976 in the sample. Here again it is impossible to observe at first an advantage of the YI program over the RS program. When the YI projects with employment relationship are considered, the total figures are equivalent to RS project figures. However, the number of students advised by each advisor was smaller for YIs without employment relationship. The Discussion section will show how this can be explained.

NUMBER OF RESEARCH SUPPORT GRANTS OR GRANTS UNDER FAPESP'S SPECIAL PROGRAMS GRANTED AFTER THE YI PROJECT OR THE FIRST RS GRANT

Table XV shows different types of grants awarded after the completion of YI and RS projects. Note that the grants to coordinators (Grant/Proj in the Table) are higher for RS recipients. But because of the data collection design (YI and RS beginning in the same year and the survey in 2004) the difference regarding the terms - four and two years for YI and RS, respectively, allows longer time for RS coordinators to obtain more grants. Therefore, another procedure should be adopted to compare YI and RS with respect to this indicator. It is worth noting that YIs with fellowship received lower amounts than YIs with employment relationship.

DISCUSSION

One of the reasons for creating the YI program was the long period that elapsed before a young PhD could pursue an independent academic career. This perception became stronger in the last three decades after the university reform of USP in 1969. Before then, it was not uncommon for young investigators to be hired even before they had completed their PhD, and their path was smoother and continuous, with no disruption. They generally started working with their professor, usually a full professor, and pursued the same theme, in the same place, with the same infrastructure. Since then, the situation has been changing and tends towards earlier intellectual and institutional emancipation as a result of a new university infrastructure and a new culture of academic affirmation much in line with the United States trend, also followed by Europe. The largest number of students who pursued their doctoral or post-doctoral studies abroad has contributed to this change. Breaking away from the system that favored "endogeny" was neither sudden nor painless, but endogeny is increasingly less common. In this study, for example, "endogenous PhDs" reached approximately 12% in both samples.

FAPESP has played an important role in State of São Paulo to consolidate the new model. Young people began to resort to FAPESP to obtain research grants not only as a pragmatic means but to seek academic affirmation from a symbolic point of view (and by the way, it is worth noting that at first there was a strong reaction on the part of older scholars against the "premature" concession of grants). What became clear in time is that although grants were important, they were not enough to allow, in most areas, the creation of conditions to develop more robust projects in the short term, despite the fact that intellectual conditions existed. It was not uncommon for one to have to wait from five to ten years to get to this point. In view of the above, the paradigmatic role of the BIOQ-FAPESP program should not be forgotten (Perez 2002). In the beginning of the 1970s, this program enabled young investigators with proven potential in biochemistry to reach conditions that allowed them to develop projects assessed by international advisors. This program may be considered a precursor of the YI program, although the latter had some other important features arising from recent challenges. As said in the Introduction, this and three other points were the basis of the YI program and our discussion will focus on these aspects.

MERIT

Item 1 covers two important aspects of the YI program: scientific merit for the grant and the budget allocation. The way these two parameters are dealt with can be appreciated through a set of indicators obtained in this study. Scientific merit is sometimes measured by the degree of approval given during an approval process with many competitors. This is typically an important indicator for a scientific journal ("acceptance rate"), which has an inverse relation with the publication's impact factor (Kirby et al. 2002). The YI project assessment process involved the review by at least three advisors. They were informed on the specific characteristics of the program. Their opinions were forwarded to the coordination offices of DC-FAPESP which consolidated them for a final decision. In a 2004 survey (Table V) we showed a more recent scenario that is reflected in the longer period (not shown). We observed that the degree of acceptance of YI projects was 27%, therefore lower than the degree of acceptance of RS projects (57%) and thematic projects (45%). Given the prestige of thematic projects, the last piece of data is of especial interest. Another relevant indicator concerning the merit can be seen in the scientific production of YI project coordinators in terms of the number of complete publications in periodicals at the time of submission, which was more than twice the number of publications of RS projects in internationaljournals and 46% higher than in Brazilian journals(Table IV).

As regards merit in the assessment of the requests we could use the indicator of participation in post-doctoral programs prior to the YI and RS programs. This was relatively high and had similar percentages for YI and RS (61%-77%, Table VIII). It is however surprising that the participation of YIs with post-doctoral fellowships in Brazil was this higher. It might have to do with a policy that prevailed in FAPESP's Scientific Committee during this study. This policy favored post-doctoral fellowships in Brazil rather than abroad. Investigators with employment relationship were supposedly not so affected by this measure whereas Investigators from the State of São Paulo with no employment relationship might have opted for a local post-doctoral fellowship and might have been driven, during the grant period, to the YI program.

YI recipients had a clear edge in terms of meritindicators. This allows to conclude that the procedures adopted by DC-FAPESP, and supported by the advisors' assessment, were appropriately understood and employed.

BUDGET FUNDING

There was no allocation of specific funding for the YI program and no difference was made as to area of knowledge or other item other than the four key points of the program. There was a significant degree of freedom that allowed the flow of a spontaneous demand which is not new to FAPESP's most traditional programs. Since the YI program would allow more significant funding without establishing defined levels, there was the implicit need for DC-FAPESP to work diligently to consider merit and realistic needs, balancing them within the context of FAPESP's global budget. FAPESP's previous experience with thematic projects may have helped it resist to pressures towards the establishment of a directed program.

Budget allocations were not stratified in the sampling of YI projects, that is, they do not necessarily reflect the population of YI projects, but the likelihood of their reflecting it is 95% with a percentual of error of 8% (see Methods). The total funding of YI projects in the sample reached R$ 23.4 million whereas the funding for RS projects was R$ 7.4 million. Considering that 60% of the budget is spent in permanent material and this amount is almost fully spent in the first year of the project, R$ 13.2 million were spent in this item line in the YI program and R$ 4.4 million in the RS program. The rest was primarily spent in consumption material and third-party services which, divided by the number of years (four for YI and two for RS), means R$ 2.2 million for YI per year and R$ 1.5 million for RS. The major difference is in the investment in permanent material which can be three times as high for YI. This meets the goal of providing YIs with the means to have appropriate working conditions in the short term. As we have said, these results do not arise from the application of established rules, but rather from the review of the justification of the budget estimates submitted by the applicants during the assessment process. This means that merit and needs have to be balanced and hard work is required. It is not just the case of making a decision based on an RFP with a defined budget for a defined number of projects.

This procedure does not necessarily lead to a percentage of approvals of proposals in different areas.The areas of agronomy/veterinary and mathematics, for instance, had a lower percentage of approvals for YI projects than biology (Table V). It is impossible to infer from this raw data that the agronomy/veterinary and mathematics committees were stricter in their assessment or that the proposals had less merit from the start.

DESTINATION OF YOUNG INVESTIGATORS

The goal of prioritizing YIs to institutions and research centers that were not fully consolidated was not defined based on quotas either. It arose from the assessment of merit and the analysis of the institutions where the projects were to be carried out. Once again, a good balance was reached, especially when the YI program grants are compared with the thematic program which supposedly serves more organized and established research groups (year 2004, Table VI). UNESP, the newest and most heterogeneous university of the three state universities of the State of São Paulo, and that for this very reason has numerous research centers striving to build their reputation, participated with only 5% of the thematic projects but 20% of YI projects. State research institutes had 4% of the thematic project grants and 8% of YI grants. Something interesting was observed in private universities which received 3% of the thematic grants but 18% of YI grants!

Table VII shows the results of Table VI in a different manner to allow the observation from the point of view of the degree of approval at the institutional level. A curious fact is that there has not been a different degree of approval in the percentages of grants for different categories of institutions, unlike thematic projects. Two pieces of evidence arise from these results. Firstly, FAPESP as a whole (DC-FAPESP and advisors) has acted in a relatively homogenous manner concerning YI grants. This is no easy task and requires efficient management. Secondly, the Investigator community of the State of São Paulo has assimilated the spirit and the purposes of the program which reflects the degree of maturity of this community. So much so that groups resorted to the program in a different manner, with considerable demand on the part of institutions that had certain priority.

There was also different demand according to the area. Interestingly enough, the area of health, the main focus of thematic and RS projects, with 33% and 40% of the grants respectively, had only 22% of YI projects in 2004 (Table V) and 13% in the period 1995-March 2004 (Table I) as a result of decreased demand. In the area of biology, the demand for YI projects (29% and 25%, Tables I and V) was above the demand for RS and thematic projects (11% and 17%, Table V). These inverse trends are very clear and we do not have a final explanation. One possibility is that the YI program, because it is new, reflects a new trend of attraction per area. The area of biology at FAPESP includes the sub-areas of biochemistry and genetics, closely linked to biotechnology and genomics. It also includes the sub-areas of botany and zoology which are the basis for ecology and biodiversity studies. These are all contemporary and attractive themes that fascinate young investigators. The areas of biology and health traditionally account for 50% of FAPESP projects, with health accounting for two thirds and biology for approximately one third. Both areas still account for 50% of the projects, but biology now accounts for two thirds and health for one third. Because it is still a "counter program" just like the RS and thematic programs, the YI program is like a laboratory for new research trends.

Similar data for institutional destination were used for the samples (Table II), but these data covered a period prior to the period of the data presented in Tables VI and VII. As we have said, the YI sample was stratified in terms of institutional destination regarding the YI population for the period 1995-March 2004 (which showed great consistency in the trends of institutional destination in two different periods - a positive fact). Table II shows in greater detail than Table VI the institutional destination of the participants of the sample and of the "mirror" RS sample. The effort to stratify samples is always arduous especially when yet one more stratification factor has to be included (and this was the case here with the inclusion of stratification for institutions, areas of knowledge and types of project, with or without employment relationship). For instance, in the case of private universities, the representation of each one of them in the YI sample was effected, but it was impossible to "mirror" it in the RS sample except for the total number. This was so because there was a concentration of YI projects in a few private universities, especially the Universities of Mogi das Cruzes, São Francisco, Vale do Paraíba and Universidade Paulista. The management committees of several universities have been increasingly interested in encouraging the participation of their professors in YI projects, as this not only translates into prestige but also into resources for research infrastructure. There are serious obstacles for these institutions to fulfill the commitments they assume with FAPESP in terms of the maximum number of teaching hour load (12 hours per week) and the availability of institutional infrastructure. This will be discussed further below.

PERMANENCE OF YOUNG INVESTIGATORS

Two first comments have to be made about the permanence of investigators. Mobility can be a good thing from an academic point of view if it is not exaggerated, especially in the beginning of one's career, and specifically in the post-doctoral period. It was uncommon in the past but has become increasingly common over the last two decades. Just to give an idea, in the mid nineties 67% of USP professors worked in the same units where they had pursued their PhD (Meneghini 1996). The situation is very different today. As has been said before, the percentage of recipients who carry out YI and RS projects in institutions other than those where they have pursued their PhD is 86% in both cases. "Endogeny" has become insignificant!

The mobility of investigators with employment relationship since the beginning of the YI or RS project up to March 2004 was very low: 20% for RS and 15% for YI with employment relationship (Tables IX and X). Mobility was a lot higher in the case of YI without employment relationship. This is understandable. Because they had fellowships, they sought more stable positions. A survey made in April 2006 in the Lattes database for the sample of 48 YIs that had held a fellowship showed that all but one held a position in a university or research institution. Of these, 67% left the original institutions and went to other institutions in the State of São Paulo without giving up their YI projects in the new institution. FAPESP has created an interesting procedure to deal with this. The YI fellowship recipients did not have to sign a term of granting of permanent material to the institution in the beginning of the project term, as is the rule for all other FAPESP projects, but only at the end. This procedure was implemented to accommodate changes that might happen, and for the benefit of the recipient that would be able to take the material acquired with him.

One question that arises is what this institutional change has meant to YI coordinators with fellowship. It has surely brought problems for many reasons ranging from the readaptation to a new place, the pursuit of institutional space and infrastructure, and even prior problems, in the institution of origin, of which there are many reports. It is understandable that YIs with a fellowship in search of undergraduate and graduate students, of the recognition of his/her status as an advisor, investigators that resist teaching for more hours than the agreement with FAPESP allows and with other privileges have caused conflict within the institution. These investigators had virtually no link with the institutions but they had to compete, within the institution, with those who had such links.

When the YI project was conceived, it was meant to be attractive to institutions for different reasons. It would entail the [institution's approval] by FAPESP ina merit-based program. The investigator would contribute as a professor and as a scientist, without taking a position or salary. These positive aspects were acknowledged by the institutions when they accepted the YI program. But the problems described above prevailed, from an institutional point of view, over the advantages. It must have been difficult for young investigators to deal with conflict. This type of experience may be considered premature and untimely in an academic career. And in fact, this aspect of the program had only partly been anticipated by those who designed it. These difficulties had impacts as we will discuss below.

Thirty three per cent of the YIs with fellowship remained in the institution (Table X). This percentage varied between institutions. Surprisingly, at USP only 13% of the YI'S with fellowship remained in the original institution, but they mainly moved to other institutions within USP. Private universities presented the lower rates of change, 43%, probably due to a strong trend towards investigators being absorbed by the institution, a good aspect of the program.

ROLE OF THE YI PROGRAM IN THE INTEGRATION OF STUDENTS INTO THE GRADUATE SYSTEM OF THE STATE OF SÃO PAULO

The sample of 114 YI projects mobilized 756 graduate students in the period covered by this study (Table XIV). Since the sample corresponds to 23% of the population, the total of graduate students mobilized was approximately 3,300 students. This figure is quite significant in view of the total number of graduate students in State of São Paulo registered at the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) in 2004 - 42,342 students (CAPES 2004). However, the sample of RS projects evidenced an even greater number of graduate students mobilized, with a similar figure for student /project for YIs with employment relationship. YI projects of investigators with fellowship accounted for the trend of lower figures for graduate student mobilization for the total YI. Giving the difficulties described above, it is no surprise that this has been so.

GENERAL PERFORMANCE OF YI RECIPIENTS

In order to compare the scientific production of YI and RS projects we had to cover the project term and the post-project period up to March 2004 due to the methodological reasons presented above in the Result section (Table XIII). This is basically due to the different terms of both programs, four and two years for YI and RS, respectively. As we have shown in the Result section, the total of international publications was not significantly different, whereas the total of Brazilian publications favored the RS program. YI projects with fellowship had a slightly lower productivity (papers/project) as compared to the total for YI projects. The scientific productivity of the YI program as compared with the RS program can be summarized as follows:

1 – The productivity levels for coordinators of YI projects with employment relationship and coordinators of RS projects were similar. There was no clear advantage for YI projects in this regard.

2 – The productivity of YI project coordinators with fellowships was lower, which negatively impacted the total productivity of YI projects.

These two key points of our study have to be discussed separately. As for the first point, the question is: how many YI project recipients can be distinguished from RS project recipients? It would be reasonable to expect that YIs with employment relationship would bring a relevant and original contribution to the institution. In an institution with a less solid background, this is automatically achieved. But in institutions with a strong research background, the YI program was expected to enable candidates to add a new element to the research line. The proposals were to add a new front of research, unrelated with the lines pursued in the institution, preferably focusing on the more exciting frontiers of each area of knowledge.

There are no indicators to measure to what extent this pre-requisite has been met. This aspect must surely have been one of the most difficult to assess by advisors and by FAPESP scientific committee. At this point, it is useful to quote the impression of the coordinators of DC-FAPESP who took part in the decision making process on YI projects: "in the case of investigators who sought to develop a project in an institution/department with a strong research track record, it was not easy to see how much their proposal was innovative in that environment".

In many cases the YI project was clearly innovative. Proposals were frequently based on post-doctoral experiences in other institutions or countries. Whether this innovative path prevailed among YIs cannot be assessed now, with the indicators presently available.

Innovating, in this sense, requires a program, organization and ad hoc infrastructure. This certainly requires time and effort to allow a new stage of consistent progress in research. It is no surprise that there is a period without much progress in terms of publication. If this has been the case for most YI projects in consolidated institutions, it can be explained by the not so exciting productivity figures when these projects are compared with RS projects. It is worth noting that smaller productivity figures for YI projects as compared with RS projects are noted after the end of the projects (Table XII). This post-project period reflects undoubtedly what happened during the project term, probably due to the difficulties discussed above.

As for RS projects, we could not observe obstacles that would prevent the emergence of new leaders in innovative lines. But it is also true that the path for RS researcher was more open for them to work in lines of research that were underway, in association with groups that operated in an institution/department, which probably happened in most cases.

It is worth mentioning that in institutions with a weaker research background (UNESP and private universities), where the same degree of difficulties for YI and RS project recipients could be expected (after all, they would all be "innovating") YI recipients' productivity was higher than RS recipients' productivity. Apparently, it was the difference in potential identified inthe assessment of YI projects as compared with RSprojects that enabled this result.

As for the second point, i.e. the lower productivity of YI projects with fellowship, it is easier to see the difficulties, as was discussed above. The problems are twofold: the first cannot be measured by this methodology, and has to do with the hostility that YIs have to deal with in their work environment. The second has to do with the move to other institutions during the term of the projects, and this is measurable as it occurred in most cases (67%) and has certainly translated into problems for recipients of this type YI project grants.

FINAL REMARKS AND SUGGESTIONS FOR CONTINUING THE ASSESSMENT OF THE YOUNG INVESTIGATOR PROGRAM

The first that comes to mind when we think of assessing a program is the question of whether it has achieved the desired goals. The indicators presented here are figures of a database of samples of YI and RS projects, built on sound statistical principles that allow statistically reliable results about the total projects developed in almost ten years of program. These goals were stated in the Introduction and in the Discussion sections.

The figures clearly show that:

1 – There was a process in place to assess the projects and this process was strict and consistent within a universe of areas of knowledge and institutions. These two attributes were achieved through continuous debates within DC-FAPESP and not due to an established, programmed direction. It was the result of the continuous exercise of balancing merits and resources.

2 – The estimates approved [for YI projects] presented on average a total for permanent material that was three times higher than estimates for RS projects, with total yearly expenses 50% higher than RS projects. Therefore greater attention was given to implementing appropriate working conditions as fast as possible through the acquisition of equipment and permanent material. Here again there was no program guideline but rather a discussion of each individual proposal which in turn gave rise to these results.

3 – The priority of having YIs go to institutions and research centers which were not yet fully consolidated was clearly achieved as a result of two factors. The first was increased demand by centers with a profile that fit the purposes of the program and the second was the assessment of advisors and DC-FAPESP.

4 – Approximately 42% of the recipients of YI projects were investigators without employment relationship who received a special fellowship that enabled them to start working. This experience, unique worldwide as far as we know, allowed to mobilize 200 investigators in the period 1995-2003 (considering the total population). According to the sample, approximately 30 went to private universities and more than half remained in these universities, and were hired as professors. This was the case of four universities (Mogi das Cruzes, São Francisco, Vale do Paraíba and Paulista). Based on these results, this process is more that potentially viable to encourage the hiring of professors-investigators in private universities. Although the figures are not outstanding, they are significant.

Recipients of YI projects in general clearly had superior merit than recipients of RS projects. Additionally, they were granted funding well above the average of RS grants. They were therefore expected to have higher productivity than RS recipients. In general, they presented similar results, with a clear decline for YI recipients when the post-project period is considered, that is, when the repercussions of what happened during the project period in terms of publications became more evident. This fact was discussed in depth at the end of the Discussion section. Unanticipated difficulties were found while the projects were underway and some cannot be measured using numeric indicators (institutional mobility, for instance). Other specific situations known to those that monitored the development of the projects are difficult to assess (litigation in the institutions where projects with fellowships were carried out). Additionally, another question arises: to what extent did YIs who went to institutions with a stronger research background followed an innovative research line and thus faced the difficulties inherent to these circumstances?

All of these points are important, but the assessment process would benefit from data which are not available in the FAPESP documents, in the Lattes databases and in international databases. Other methods should be adopted and should be considered by DC-FAPESP. One of them is to use the help of ad hoc advisory committees for research programs. This is something we have little experience in and international assistance could be of use. Agencies such as FAPESP, CNPq and Financiadora de Estudos e Projetos (FINEP) (Research and Project Funding Agency) have gained competence to set up and monitor research projects of different sizes, but still lack the competence to deal with larger programs.

It is strongly advisable that FAPESP conduct a survey with YI project recipients. This would contribute to conclude whether some assumptions put forward in this paper bear out, and, in case they do, whether they point at new ways to go. The assumptions made allow to structure a questionnaire that would complement this assessment process.

On the other hand, in view of the list of difficulties that may eventually be revealed by such a survey, we strongly suggest that a new assessment be carried out three years after this one, possibly in 2007, with the same samples used in this study. At that time in the future, the effects of the supposed difficulties encountered by program recipients should have been mitigated and the results should allow more comprehensive conclusions. The framework of this process and a large part of the assessment work have been completed. Only a supplementary, non intensive effort is required.

Manuscript received on January 17, 2007; accepted for publication on June 5, 2007; contributed by ROGERIO MENEGHINI

• BUXTON M, HANNEY S, PACKWOOD T, ROBERTS S AND YOULL P. 2000. Assessment benefits from Department of Health and National Health Service Research and Development Public Money and Management, October-December.
• CAPES. 2004. Available at http://ged.CAPES.gov.br/AgDw/silverstream/pages/frPesquisaColeta.html
• COZZENS SE. 1997. The knowledge pool: measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning 20: 7789.
• COZZENS SE. 1995. U.S. research assessment: recent developments. Scientometrics 34: 351362.
• CREATIVE RESEARCH SYSTEM. Available at http://www.surveysystem.com/sscalc.htm
• FAPESP. 2006. Available at http://www.FAPESP.br/materia.php?data[id_materia]=48
• IRVINE J AND MARTIN BR. 1980. The economic effects of big science: the case of radio astronomy. In: SCIENCE AND TECHNOLOGY INDICATORS CONFERENCE, (Proceedings) Paris: OECD.
• KIRBY PL, SCHOTLAND M, BACCHETTI PE AND BERO LA. 2002. Association of journal quality indicators with methodological quality of clinical research articles. JAMA 287: 28052808.
• MCNAY I. 2003. Assessing the assessment: an analysis of the UK research assessment exercise, and its outcomes, with special reference to research in education. Science and Public Policy 30: 18.
• MENEGHINI R. 1999. USP, de onde vêm seus doutores. Jornal da USP, São Paulo, SP, Brasil. Nat Sci Tech Council, 1999. Available at http://www.nsf.gov/pubs/1999/pecase99/pecase99.htm#four
• PEREZ JF. 2002. A construção de novos paradigmas. São Paulo, SP, Brasil (The construction of new paradigms). Perspec 6: 3035.
• STAK TREK. Available at http://www.stattrek.com/Default.aspx
• TOMA H. 2005. Henry Taube e sua herança para a química no Brasil. J Braz Chem Soc 16: 12.

*

Member Academia Brasileira de Ciências

Publication Dates

• Publication in this collection
  27 Aug 2007
• Date of issue
  Sept 2007

History

• Received
  17 Jan 2007
• Accepted
  05 June 2007