SCIENTIFIC LITERACY: A COMPARATIVE STUDY BETWEEN BRAZIL AND JAPAN

Abstract

This study compares the Scientific Literacy of Brazilian and Japanese students based on PISA results, aiming to contribute with evidence to the discussion of educational inequalities. It analyzes empirical data obtained through observation of Science classes in schools in Brazil and Japan, a questionnaire answered by teachers who were observed and interviews with specialists in the science area and with those responsible for PISA in the two countries considered in this study. The analysis shows that the differences in performance of Brazilian and Japanese students are associated with the way in which the education system of each country approaches the curriculum and continuing teacher education, and also with the different use that they make of PISA results.

SCIENTIFIC LITERACY; SCIENCE; PISA

Resumo

Este estudo compara o letramento científico de estudantes brasileiros e japoneses com base em resultados do Pisa, visando a contribuir com evidências para a discussão das desigualdades educacionais. Analisa dados empíricos obtidos por meio de observação de aulas de Ciências em escolas do Brasil e do Japão, de questionário aplicado aos professores observados e de entrevistas com especialistas da área de ciências e com responsáveis pelo Pisa nos dois países considerados neste estudo. A análise mostra que as diferenças de desempenho dos estudantes brasileiros e japoneses estão associadas à forma como o sistema educacional de cada país aborda o currículo e a formação continuada de professores, e, ainda, ao uso diferenciado que fazem dos resultados do Pisa.

LETRAMENTO CIENTÍFICO; CIÊNCIAS; PISA

Resumen

Este estudio compara la alfabetización científica de estudiantes brasileños y japoneses con base en los resultados de Pisa, con el objetivo de aportar evidencia a la discusión de las desigualdades educativas. Analiza datos empíricos obtenidos a través de la observación de clases de Ciencias en las escuelas de Brasil y Japón, un cuestionario aplicado a profesores y entrevistas con especialistas en el campo de la ciencia y con los responsables de Pisa en los dos países considerados en este estudio. El análisis muestra que las diferencias en el desempeño de los estudiantes brasileños y japoneses están asociadas con la forma en que el sistema educativo de cada país aborda el currículo y la formación continua de los profesores, y, aún, al uso diferenciado que hacen de los resultados de Pisa.

LETRAMIENTO CIENTÍFICO; CIENCIAS; PISA

Résumé

Cette étude compare les connaissances scientifiques des élèves brésiliens et japonais sur la base des résultats du Pisa, dans le but d’apporter des evidences à la discussion sur les inégalités éducatives. Il analyse les données obtenues grâce à l’observation des classes de Sciences dans les écoles du Brésil et du Japon, à un questionnaire appliqué aux enseignants observés ainsi qu’à des entretiens avec des spécialistes en sciences et des responsables du Pisa dans les deux pays. L’analyse montre que les différences de performance entre les élèves brésiliens et japonais sont associées à la manière dont le système éducatif de chaque pays aborde le curriculum et la formation continue des enseignants, aussi bien qu’à l’utilisation différente qu’ils font des résultats du Pisa.

CULTURE SCIENTIFIQUE; SCIENCES; PISA

Despite the progress made in Brazil by public education policies in terms of the universalization of elementary education and the expansion of access to high school education and early childhood education, serious problems still persist in relation to the quality of teaching and student performance in basic education, as shown repeatedly by the large-scale national assessments conducted by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixiera [National Institute of Educational Studies and Research] (Inep), notably the Sistema de Avaliação da Educação Básica [System for the Assessment of Basic Education] (SAEB) and Prova Brasil [National Assessment of School Performance].

Although the growing importance of scientific knowledge is taken into consideration, national assessments have been restricted, as a rule, to the Portuguese Language and Mathematics. Even though there is agreement among science teachers, scientists and public policy makers about the importance of developing scientific literacy, in the sense of emphasizing students’ abilities to use scientific knowledge in real world situations, the specialized literature also registers an imbalance between the development of Science and Technology and the science education that citizens receive (Bybee, 1993; Maienschein, 1998; Millar et al., 1995; Deboer, 2000; Roberts, 2007; Roberts & Bybee, 2014).

The Programme for International Student Assessment (PISA), developed and coordinated internationally by the Organization for Economic Cooperation and Development (OECD), is a three-year assessment that focuses on three cognitive domains - science, reading and mathematics - in addition to contextualization of results through questionnaires answered by students, school principals, teachers and parents. In each round, one of these cognitive domains is the main focus of the test, with most of the items centered on this domain (approximately two-thirds of the total test time) and the rest focused on the other two domains - although they can still provide sufficient elements for comparisons between years. In 2015, for example, the focus of PISA was on science. With this alternating schedule of major domains, a thorough analysis of student achievement in each of the three core areas is presented every nine years; an analysis of trends is offered every three years (OECD, 2016).

PISA tests students between the ages of 15 years and three months (completed) and 16 years and two months (completed) at the beginning of the period in which the test is administered. This age range assumes completion of compulsory basic schooling in most countries and a student population that is at least in the 7^th grade. In the 2006 and 2015 editions of PISA, the focus was on the domain of science, which allows for an in-depth analysis of student performance in the focus area and an analysis of trends in the other areas. In this study, we used data from these two editions and verified the performance of students in the skills and knowledge assessed in the area of science in two specific contexts, Brazil and Japan.

The vision of scientific literacy, which forms the basis of PISA, can be summarized by the following question: What is it important for young people to know, value and be able to accomplish in situations involving science and technology? Scientific literacy in PISA refers to both the understanding of scientific concepts and the ability to apply these concepts and to think from a scientific perspective. Therefore, it is associated with the ability to go beyond the simple acquisition of knowledge to demonstrate competence in applying this knowledge in everyday situations. PISA seeks to examine students’ ability to analyze, reason and reflect actively on their knowledge and experiences, focusing on competencies that will be relevant for their future lives.

In a larger study, namely the doctoral thesis of the first author under the supervision of the second author (Muri, 2017), aiming to understand Brazil’s performance in Science in PISA, we compared Brazil’s results with those of other countries participating in the program, especially with Japanese students, due to the opportunity arising from the experience gained in a teacher training program offered by Japan, between 2007 and 2009, as well as the prominent position held by Japan in international comparative tests. In large-scale assessments such as PISA, this comparability is very complex, since not all test items have the same functioning. According to Soares (2005), in order to enable comparability of results it is essential that the model used in the assessment ensures the assumption that the item has the same functioning for the various population groups being evaluated. For a good comparison between results of groups of students that are so different, as is the case of Brazilian and Japanese students, it is essential to pay special attention to how the items are built, so that they do not have what is referred to as Differential Item Functioning (DIF).

An item has DIF when students with the same cognitive ability do not have the same probability of getting the item right. Thus, in estimating proficiency, it is ideal to avoid using items with high DIF, that is, items that favor a certain group of students. However, when DIF is moderate and restricted to a few items, it interferes minimally in estimation of proficiency, and its analysis can be a very useful diagnostic tool of the education system in terms of curricular and sociocultural differences, and, in the case of international studies such as PISA, the diversity of educational realities and the disparity of results between countries. For this reason, we chose this methodological approach to conduct a comparative study between two countries with such different socioeconomic and cultural realities, as is the case of Brazil and Japan.

Also considering that the distinctive characteristics of the education systems of these two countries have consequences in the different ways in which the curriculum is prepared and developed, and also that the contents are selected by teachers and addressed with different emphases in Science classes, we initially sought in the doctoral thesis (Muri, 2017): (a) to identify whether the differential functioning of the items of the 2006 PISA Science test can explain the performance gap between students assessed in Brazil and Japan; (b) to verify whether the differential behavior of the test items hinders comparability of the results; and (c) to identify different curricular emphases and/or teaching practices in Science teaching in Brazil and Japan that can explain the different PISA performance of Brazilian and Japanese students.

The first two objectives unfolded into research questions that have been answered in articles already published by the authors (Muri et al., 2017, Muri Leite & Bonamino, 2020) and their results, in general terms, are brought to the discussion in this article, because they are directly related to the third objective, which is an unfolding of the general objective of this work. We will specifically discuss in this article the results of the empirical study conducted in a total of six schools, three in Japan and three in Brazil, with observation of Science classes, completion of questionnaires by the teachers observed, and interviews with experts in the area of Science and with those responsible for PISA in the two countries considered in this study.

As our specific interest was to explore the concept of scientific literacy adopted by PISA in the results of this international assessment and in real classroom situations, this concept was the basic reference for the development of the set of instruments used in our research. We adopted the constructs used by the Program to access the students’ point of view about Science teaching and learning and also the perspective of the teachers participating in this research, as well as to guide the classroom observations carried out in both countries. We thus sought to contextualize the quantitative results obtained with the DIF approach by adding the perceptions of students, teachers and experts to the research design.

In short, the specific objectives of this work are: a) to observe the use of time in Science classes in the final grades of elementary school in Brazil and in Japan; b) to observe whether in the Science classes of these countries activities related to interaction, investigation, experimentation and application occur, and with what frequency; c) to seek to understand which activities teachers state doing in their classes and how often they are done; and d) to seek to understand, with the help of specialists and PISA managers, the difference in performance found between Brazil and Japan by PISA.

This article presents the analysis of the qualitative evidence from the observation-related research in the schools of the two countries in the following dimensions: use of time in Science classes in the final grades of elementary school; carrying out and frequency of activities related to interaction, investigation, experimentation and application; teachers’ perception of the activities carried out in class and their frequency; and the performance gap between Brazil and Japan based on the interviews conducted with PISA experts and PISA managers in each country.

Besides the introduction and the conclusions, this paper is organized into three further sections. In the next section, we briefly discuss scientific literacy in PISA. Following that, the methodology adopted and the instruments used in the study are described. The following section presents and discusses the results of our qualitative research.

Scientific literacy in PISA

International experts from participating countries have developed a framework and conceptual support for each PISA assessment area (OECD, 1999, 2003, 2006). The framework begins with the general concept of Literacy, which is related to students’ ability to go beyond what they have learned, to apply their knowledge in new contexts; and to analyze, argue and communicate effectively as they present, solve and interpret problems in a variety of situations (OECD, 2007).

Specifically in the case of the area of science, Scientific Literacy has been defined as a set of competencies that would be expected of a scientifically literate individual. The competencies assessed by PISA are broad and include aspects related to personal usefulness, social responsibility and the intrinsic and extrinsic value of scientific knowledge (OECD, 2006). As such, the program’s perspective differs from perspectives grounded exclusively in the Science curriculum and discipline, since it includes problems situated in educational as well as professional contexts and recognizes the essential place of the knowledge, methods, attitudes and values that define scientific disciplines (OECD, 2006). From this perspective, the term that best describes the general purposes of PISA science assessment is Scientific Literacy, according to the conceptions of Bybee (1997a), Fensham (2000), Gräber & Bolte (1997¹ apud Guimarães, 2003), Mayer (2002), Roberts (1983), and United Nations Educational, Scientific and Cultural Organization (Unesco) (1993).

According to OECD (2006), the term Scientific Literacy was chosen to represent the goals of science education for all students, in a broad perspective and of an applied nature represented as a continuum of scientific knowledge and cognitive skills associated with scientific inquiry, which incorporates multiple dimensions and includes the relationships between science and technology. Together, the scientific competencies are at the heart of the definition and characterize the basis of Scientific Literacy and the purpose of science assessment, which is to evaluate the degree to which basic education students have developed these competencies (Bybee, 1997b; Fensham, 2000; Law, 2002; Mayer & Kumano, 2002).

Furthermore, the perspective adopted by PISA recognizes that there is an affective element in a student’s demonstration of these competencies (OECD, 2006, 2013, 2016), which is represented by their attitudes or disposition towards science and will determine their level of interest, sustain their engagement and motivate them to act (Schibeci, 1984). Thus, the scientifically literate person generally demonstrates interest in scientific issues, engagement with problems related to science, concern about technology, resource and environment issues, and reflection on the importance of science from a personal and social perspective. These requirements do not mean that these people are necessarily in favor of science itself, but that they recognize science, technology and research in this domain as an essential element of contemporary culture that structures much of our thinking (OECD, 2013, 2016).

These are the considerations that led to the definitions of Scientific Literacy in the 2006 and 2015 PISA. In short, PISA, as well as more recent contributions in the search for a consensus definition of Scientific Literacy, brings together the two broad domains focused on understanding the scientific content and social function of science and technology, including attitudes, beliefs and interests that influence decisions and actions based on a personal, social and cultural perspective.

Methods

This article presents a qualitative study conducted in Brazil and Japan with the aim of refining and expanding the potential for interpretation of the quantitative results of student performance in these two countries and the study of DIF in the science items of the 2006 PISA, by including the perception of students, teachers and experts. This triangulation meant looking at the same phenomenon, in this case the concept of Scientific Literacy in PISA, from more than one perspective. In the specialized literature, triangulation is not just one of the ways of combining qualitative methods with each other (Flick, 2005a, 2005b) and articulating quantitative and qualitative methods (Fielding & Schreier, 2001; Flick, 2005b), but also a break away from the hegemony of methodological monism or the single method (Tashakkori & Teddlie, 1998).

The concept of Scientific Literacy as adopted by PISA served as the reference for the development of all our research instruments, which generated data that were used in our analyses. In order to access the students’ point of view about Science teaching and learning and the perspective of the teachers participating in the research, and also to guide our observations of the classes, we adopted constructs used by the Program in 2006. In addition to obtaining the view of teachers and observing Science classes, in order to understand Scientific Literacy conceptually and operationally in the countries studied, we conducted open interviews with teachers specialized in the teaching of this subject, in Japan, and with people responsible for PISA, in Japan and Brazil. These interviews were extremely important, since they contributed to raising hypotheses that, in the view of experts, may contribute to explaining the difference in performance between the countries involved, as well as the peculiarities of their different education systems.

The schools

Between November 2015 and October 2016 we observed a total of 147 Science lessons in the three final years of elementary school in Japan and Brazil (64 hours of lessons in Japan and 83 in Brazil) in three schools in each country. We also followed a total of 16 teachers, eight in each country.

The three Japanese schools selected to participate in the field research partner with Gakugei University in Tokyo, where we did our Ph.D. sandwich internship, which ensured the possibility of videotaping the classes we observed. The Brazilian schools, on the other hand, were defined based on the characteristics of the Japanese schools that authorized us to carry out the research. All schools were located in Tokyo, two of them were associated with Gakugei University and were similar to university-maintained schools in Brazil, while the third was a municipal school, also a partner of that university. In order to conserve some degree of correspondence, in Brazil, the three schools selected were located in the municipality of Rio de Janeiro, two of them were federal - one was a university-maintained school, while the other was a school with a tradition of excellence - and the third, like the Japanese school, was also a municipal school that was selected for having the highest Índice de Desenvolvimento da Educação Básica [Basic Education Development Index] (IDEB) in Rio de Janeiro at the time. We observed Science lessons in the three final years of basic education, which correspond to the 7^th, 8^th and 9^th grades of basic education in Brazil, and to the three years of Junior High School/中学校 in Japan.

The observation schedule

The script for observation of more general aspects of classrooms and teachers is an adaptation of the observation form used by the Geres project.² The items relating to teaching practices regarding Science classroom management and student learning took PISA as their reference. The result was an observation schedule that covers three classroom dimensions: use of time, curriculum emphases and teaching practices.

Use of time

During our observations, we recorded the official and effective start and end times of Science classes. The main references for analyzing time were based on the study by Marcel Crahay (2002), who, in defense of equal knowledge as an expression of a fair school, places a primordial role on use of time for teaching and learning activities. We used Crahay’s concept of Time Allocated to Educational Action (TAEA) as a reference. This concept expresses the real amount of time allocated to activities directly related to the teaching-learning process, excluding time spent on other issues during the lesson. We used this concept to support the analysis of the data obtained during the observation of the classes, expressed as per the following formula:

TAEA = TO - TM - TE - TS, where:

TAEA - Time Allocated to Educational Action

TO - Official time

TM - Unused time (Pauses, Inactivity and Interruptions)

TE - Time “lost” entering the classroom

TS - Time “lost” leaving the classroom

Curriculum emphases

The results of the DIF analyses suggested that the emphases of the Science curricula in Brazil and Japan were different (Muri et al., 2017). We then considered what was emphasized from a curriculum perspective in the lessons we observed, seeking to record whether the subjects addressed were related to the area of natural and biological sciences, physics or chemistry.

Teaching practices

Teaching practices regarding classroom management and promotion of student learning were observed using a script based on PISA. These practices refer to the frequency with which certain activities take place in Science classes and relate specifically to the frequency with which:

1. Students are invited to put forward their ideas.

2. Students do laboratory experiments.

3. The teacher asks the students to imagine how a given scientific issue could be investigated in the laboratory.

4. The teacher asks the students to apply a scientific concept to everyday problems.

5. Students are invited to give their opinion about the topics that are being dealt with.

6. The teacher asks the students to draw conclusions about an experiment they have done.

7. The teacher explains how a scientific notion can be applied to several phenomena (e.g., to the movement of bodies or to substances with identical properties).

8. Students are allowed to devise their own experiments.

9. There is a debate or exchange of ideas during the lesson.

10. Experiments are performed by the teacher, in the form of a demonstration.

11. Students can choose their research assignments.

12. The teacher uses science to help students to understand the outside world.

13. Students debate on the topics that are being dealt with.

14. Students perform experiments following the teacher’s instructions.

15. The teacher clearly explains the importance of scientific concepts in the lives of all people.

16. The teacher asks the students to undertake a study in order to test their own ideas.

17. The teacher gives examples of technological applications to show how science is important for society.

The questionnaire for teachers

Initially, this instrument was built based on models of questionnaires used by national and international external assessments, such as SAEB/Prova Brasil,³ SARESP,⁴ Geres and PISA. The questionnaire for teachers was divided into three blocks. The first block, intended to obtain the profile of the teachers taking part, collected general information such as sex, age, education, experience and mode of employment. The second block contained questions regarding their participation in various types of continuing education and their perception of the impact of these activities on their teaching practice. The third block was dedicated to collecting information about Scientific Literacy, by incorporating the PISA items already listed and that were emphasized in our analyses.

Analysis of teaching practice

The initial step in analyzing the data obtained from the observations and the questionnaire answered by the teachers was a careful reading of the PISA conceptual framework. PISA investigates the process of Science teaching and learning by focusing on the strategies used at school and their variation across different types of teaching and schools participating in the program. To this end, the Program created four scales based on the 17 activities described above, which allow the process of student Scientific Literacy to be characterized. The categories used by the Program and also by us in the analysis of the data were the following: Interaction, Experimentation, Investigation and Application.

The Interaction category involves, above all, activities related to active participation of students in class. Experimentation refers to doing experiments, using of the Science laboratory, etc. The Investigation dimension examines the degree of student autonomy to seek and build knowledge. Finally, Application is more related to the use of science and, consequently, its importance for society. The distribution of the activities between the categories listed by the Program can be visualized better in the following box (Chart 1).

CHART 1
PISA 2006 SCIENTIFIC LITERACY ANALYSIS CATEGORIES

Index	Activities
Interaction	1. Students are invited to put forward their ideas. 2. Students are invited to give their opinion about the topics that are being dealt with. 3. There is debate or exchange of ideas during the lesson. 4. Students debate on the topics that are being dealt with.
Experimentation	1. Students do laboratory experiments. 2. The teacher asks the students to image how a given scientific issue could be investigated in the laboratory. 3. The teacher asks the students to draw conclusions about an experiment they have done. 4. Experiments are performed by the teacher, in the form of a demonstration. 5. Students perform experiments following the teacher’s instructions.
Investigation	1. The teacher asks the students to apply a scientific concept to everyday problems. 2. Students are allowed to devise their own experiments. 3. Students can choose their research assignments. 4. The teacher asks the students to undertake a study in order to test their own ideas.
Application	1. The teacher explains how a scientific notion can be applied to several phenomena (e.g., to the movement of bodies or to substances with identical properties). 2. The teacher uses science to help students to understand the outside world. 3. The teacher clearly explains the importance of scientific concepts in the lives of all people. 4. The teacher gives examples of technological applications to show how science is important for society.

• Source: Prepared by the authors based on OECD (2007).

The same constructs and categories were used in the analyses of the data obtained based on the observations and the questionnaire for teachers. In short, our field work sought to answer the following questions:

1. Based on the observations, what is the Time Allocated to Educational Action in Science in Brazil and Japan?

2. What is the perception of students taking part in PISA as to the frequency with which Science teaching-learning activities take place? Can this perceived frequency be related to the PISA Science performance of Brazilian and Japanese students?

3. How frequently do these activities occur in the Science classes observed in Brazil and Japan? Do the Science classes of these two countries concentrate more activities characteristic of any of the PISA scales (interaction, experimentation, investigation and application)?

4. Do the teachers observed adopt these activities in their classes? How frequently?

5. Why is performance in Brazil and in Japan so different? What lessons can be drawn from the Japanese experience?

In order to answer these questions, we sought to conduct our analysis in two main ways: descriptive analysis and explanatory analysis. The former is guided mainly by the discussion of the PISA results and the students’ answers to the questions proposed in the student context questionnaire in the international assessment. In this case, we investigated the correlations between the frequencies of use of teaching methods/practices, student performance and the time devoted to teaching Science. In the second dimension, we sought to explain the results found by our study through observation of the classes and the questionnaires answered by the teachers, and also to verify whether the practices that impact student PISA performance are especially frequent in the Science classes observed in these two countries and whether they are reported by the teachers in their answers to the questionnaires.

Results and discussion

Brazil is at a disadvantage compared to almost all the countries taking part in PISA, not just Japan. Brazil came in 52^nd place out of 57 PISA participants in 2006 and in 63^rd place out of some 70 countries participating in 2015. Japan came in 6^th place in 2006 and 2^nd place in 2015. On the performance scale, in 2006 and 2015, Brazil remained at level 1, while Japan went from level 3 to level 4. The comparison of PISA results obtained by Brazilian and Japanese students made it clear that the age/grade mismatch continues to be one of the most important dimensions of Brazilian educational inequalities, despite being one of the problems most targeted by federal, state and municipal public policies. This aspect has been widely discussed, especially in the case of Brazil, since in Japan there is no grade repetition, by Muri Leite and Bonamino (2020).

Only Brazilian students enrolled in the final grades of high school and, therefore, at a higher schooling level than Japanese students were able to achieve the initial thresholds considered by OECD as necessary for active participation in society (Muri, 2017). Brazilian students with age/grade mismatch had poorer performance in Science than Brazilian students in the overall sample. However, when compared to Japanese students, the difference for Brazilian students with age/grade mismatch was as much as 200 points in some skills. These results are in line with the findings of specialized literature showing that grade repetition is not the best solution for teaching-learning problems in Brazil (Freitas, 1947; Brandão et al., 1983; Ribeiro, 1991; Alves et al., 2007; Correa et al., 2014). Grade repetition has been shown to have a negative impact on PISA Science performance, because it deprives these students of the opportunity to learn what is typically assessed by the Program among 15-year-olds who should be in the final year of elementary school or the first grade of high school (Muri Leite & Bonamino, 2020).

The comparative analyses also showed that, despite the care that surrounds the preparation and selection of PISA items, there was significant presence of DIF in the Science items of the 2006 edition when comparing Brazil and Japan (Muri et al., 2017). It is worth noting that the items diagnosed as having DIF did not compromise the assessment process by favoring one group over the other. This was verified in the analyses of Brazil and Japan by Muri et al. (2017), which show that the differences favor one group of students from one country over the other, although they suggested the existence of different curriculum emphases in Science teaching, which were investigated in loco by our study.

Indeed, the Science performance of Japanese students according to PISA 2006 and 2015 was much better than that of Brazilian students, but despite the occurrence of DIF in some test items, it was not possible to state that significant differences in performance were explained solely and exclusively by the differential behavior of the items, that is, by the fact that the items were favoring Japan to the detriment of Brazil. Among countless other factors, such as students’ socioeconomic status, their parents’ level of education, etc., the literature points out that effective teachers positively impact student learning (Hanushek & Woessmann, 2010; Hattie, 2003, 2009, 2012; Hanushek, 2002; Taylor et al., 2010). The identification of practices that teachers incorporate in their teaching activities can, therefore, contribute to the understanding of what can make a difference in student learning and performance, and also result in elements to inform the adoption of measures to promote school effectiveness and improve the quality of education.

In Japan the majority of the teachers observed were male (five out of eight) while in Brazil the majority were female (six out of eight). The Brazilian teachers participating in the survey were younger (most of the Brazilian teachers were under 39 years of age, while most of the Japanese were over 40) and in general had a higher level of education than the Japanese (half of the Brazilian teachers observed had a Master’s degree and the other half had a Ph.D. In Japan, half the teachers also had Master’s degrees, but the other half had Licentiate/Bachelor’s degrees). All Brazilian teachers had degrees in Biology, while in Japan there were teachers with degrees in Chemistry, Physics and Geosciences. The continuing education of Brazilian teachers emphasizes initiatives directly related to research and extension. Usually, these activities are also related to attending postgraduate courses. The continuing education of Japanese teachers consists of initiatives related to peer interaction, that is, initiatives that can only be implemented when training is on-the-job. In Japan, collaborative work and research groups formed by teachers are a constituent part of professional development and have been considered successful initiatives.

Time

The empirical study showed that class time is a key indicator of the Science learning opportunities offered to students. In both Brazil and Japan, in the schools observed, lesson time is 50 minutes. Given that we observed 64 class hours in Japan and 83 class hours in Brazil, our total Official Time (OT) was exactly 7,350 minutes. Approximately one and a half lesson hours out of the 64 observed in Japan were devoted to issues other than Science teaching and learning. This is less than 2% of the total hours observed. In Brazil, of the observed class hours (83), at least 15 class hours were lost, almost 20% of the total time. Therefore, the Brazilian schools we observed waste ten times more lesson time than the Japanese schools. As in Bruns and Javier (2014), the main causes of this wasted time are arriving late, excessive bureaucratic tasks (registering attendance, cleaning the blackboard and distributing assignments) and interruptions.

Loss of teaching time as an indicator of educational opportunity is grounded in a robust body of research on the importance of learning time. More than half a century ago, John Carroll (1963) placed time at the center of his model of school learning. Carroll’s overall idea was that what students learn is closely related to the time they spend learning. Some of the most extensive research on the availability of learning time has been done in developing countries such as Brazil and other Latin American (e.g. Colombia and Peru) and Caribbean countries (see Bruns & Javier, 2014). International studies have focused attention on institutional factors that shape learning time and educational opportunities. Most of these studies point out that in some countries with relatively low average incomes, students are often only taught in a fraction of the total learning time allocated - the case of Brazil, Ghana, Tunisia, and others (see Abadzi, 2009).

Curriculum emphases

Another dimension that may explain the differences in PISA Science performance between Brazilian and Japanese students is curriculum emphases. The curriculum is distributed more homogeneously throughout the school years in Japan, while it is very fragmented in Brazil, which still has a strong emphasis on the natural sciences. As a rule, in Brazil, only in the 9^th grade do students have contact with school contents related to physical, chemical and technological systems. Our observations and the content of the vast majority of Brazilian textbooks support this conclusion. In Japan, although content related to the natural and biological sciences is dealt with more in each school year, it was possible to observe a more even distribution of content. In Japan, each year students study content related to chemistry, physics and biology. From an early age, Japanese students have contact with topics related to the advancement of science and technology and their importance for life.

We observed that the different contents of the PISA reference matrix are given to Japanese students in practically all school years, in such a way that continuous contact of these students with the different themes of Science education is established throughout elementary school. We perceived that such continuity offers not only greater and longer contact with the various fields of science, but above all greater opportunity to consolidate scientific concepts that are returned to year after year.

We analyzed the proposed or formal curriculum, as Forquin (1992) defines it, in both countries. In Brazil, we focused on the Parâmetros Curriculares Nacionais [National Curricular Parameters] (PCN), while in Japan we focused on the Course of Study or 学習指導要領 (Gakushū shidō yōryō) in Japanese. In the case of Brazil, although as in Japan we have total coverage of the contents proposed in the Program’s reference matrix, we noted that these contents are presented in a more sectioned way throughout the years of elementary school. This more watertight distribution of contents can be added to another curriculum difficulty in Brazil, namely the lack of implementation of a common national curriculum.⁵ If there is a relevant aspect that can be learned from the Japanese experience, it is the attention that is paid in Japan to the details of the national curriculum and the insistence that this curriculum is actually taught and learned.

According to OECD (2010), the Japanese curriculum is coherent, carefully focused on essential topics and deep conceptual exploration, logically sequenced, and set at a very high level of cognitive challenge. The result of this is that high school graduates in that country have a level of mastery of topics that matches that of university graduates in many Western countries. Japan has a history of sustained excellence and has been at the top of international education research rankings since international assessments began. Ryo Watanabe, Director of International Research at the National Institute for Educational Research, in an interview with OECD, stated that Japanese students have been very successful in terms of PISA results because of their curriculum. Japan has national curriculum standards, or courses of study, that define the content to be taught by grade and subject, and which must be updated every ten years. According to Watanabe, throughout the country teachers teach based on national curriculum standards (OECD, 2010). Also according to OECD (2010), the Japanese curriculum is very demanding, yet it is also highly coherent, in that it progresses step by step, logically from year to year, focusing each year on the topics that must be mastered in order to understand what will be presented the following year.

Didactic approach and teaching practice

Didactic approach and pedagogical practice are also important curricular factors. The quality of teaching and its effectiveness have been the subject of several national and international studies that seek to highlight the characteristics of teachers’ work that have a positive effect on student performance (Hanushek & Woessmann, 2010; Hattie, 2003, 2009, 2012; Hanushek, 2002; Taylor et al., 2010). Brazilian students, despite not performing well in Science as assessed by PISA, state that they are more frequently exposed to constructivist approaches involving ‘investigation”, “interaction”, “application” and “experimentation” than Japanese students. The literature on the subject indicates that these approaches to Science teaching are extremely important for the development of Scientific Literacy skills. However, despite the fact that Japan has one of the best PISA Science results compared to other OECD countries, Japanese students rarely state that these teaching strategies are frequently used by their teachers. They claim that, in general, such activities occur in just over 15% of lessons. Brazilian students stated that experimentation, although it is the least frequent activity, occurs in at least 30% of lessons.

The perceptions of Brazilian and Japanese teachers are similar in recognizing “interaction” and “application” as the activities most frequently undertaken in their classes, accounting for more than 50% of classes. “Experimentation” activities, recognized for their power to attract students’ interest in science, are the least used by Brazilian teachers and the most used by Japanese teachers.

Finally, Japanese and Brazilian teachers tend to be similar with regard to less frequent use of investigative activities in their classes. Investigative activities are basically related to the level of autonomy granted to students to choose their research topics, test their own ideas, do their own experiments, etc.

Guimarães (2003) reinforces the idea of autonomy, arguing that individuals are naturally inclined to perform an activity because they believe they are doing it of their own free will and not because they are forced to do so by external demands. As a result of this perception, their behavior can be intrinsically motivated, setting personal goals, demonstrating their successes and difficulties, planning the necessary actions to achieve their goals and adequately evaluating their progress.

The objective of investigations with regard to science teaching and learning is to engage students in the broad process of scientific inquiry, including, for example, opportunities to test their own ideas using scientific methods. While these teaching activities give students the opportunity to develop the three core competencies (“identify scientific issues”, “explain phenomena scientifically” and “use scientific evidence”) of the PISA Science chart (Kobarg et al., 2011), they were less frequent in the perception of students and teachers, as well as in the observations carried out in both Brazil and Japan. They occurred in only 1% of the classes observed in Brazil and in 6% of the classes observed in Japan. A recent literature review provides positive evidence about the impact of investigative activities on students’ scientific knowledge. The quality of the way in which science is represented by scientific inquiry and investigations is more important than students doing experiments (Harlen, 1999), that is, experiments can support the development of Scientific Literacy when integrated into teaching units aimed at improving students’ ability to think and reason scientifically. Thus, it should be noted that hands-on activities alone do not promote students’ Scientific Literacy; they need to know the broader process of scientific investigation in order to develop a deep understanding of scientific content (Kobarg et al., 2011).

Activities related to “experimentation” were most frequently observed in the Japanese school classes, where use of the laboratory and conducting experiments occurred in about 40% of the classes. Japanese teachers reported using this practice in about 70% of their classes. In Brazil, we observed these activities in only 16% of classes, and Brazilian teachers stated that this was the practice they used least in their classes (just over 10% of classes). Hands-on activities are an important aspect of Science teaching (Harlen, 1999) because they provide students with the opportunity to perform tasks, such as planning and conducting an experiment and drawing conclusions from their results (Kobarg et al., 2011). Hands-on activities provide students with the opportunity to collect their own scientific evidence, which can help them develop the competence to use this type of evidence. However, one must consider that hands-on activities are often limited to data collection and do not go very far with regard to interpreting data.

Students’ active interaction with the science teaching and learning process was observed by us and recognized by the teachers in their answers to the questionnaire for more than 50% of the total number of classes, both in Brazil and in Japan. In their answers to the context questionnaire, Brazilian students also identified these activities in more than 50% of classes, while Japanese students recognized them as being part of just over 15% of classes. During our research in the classroom, this was the most frequently observed activity in Brazil and also in Japan. “Interaction” activities can be used to engage students in the classroom. For example, students can be invited to state their own opinion, explain their ideas, or participate in discussions about scientific issues. Today, science educators agree that teaching science should place emphasis on interactive learning activities (Hofstein & Lunetta, 2004). Transmissive Science teaching has a strong focus on scientific knowledge and on theoretical lessons, textbooks and demonstration of experiments, while interactive Science teaching is, in contrast, oriented towards cooperative learning through collaborative discussions and interactions between students and teachers (Kobarg et al., 2011).

Activities related to the applicability of science, especially the applicability of scientific knowledge to everyday problems and the application of a scientific concept to various phenomena, as well as the importance of science in the lives of all people and in society, were observed much more frequently in Brazil (almost 60% of lessons) than in Japan (less than 30% of lessons). The different approaches to constructivist teaching emphasize real-life applications as a means of creating authentic and meaningful contexts for students (Collins et al., 1989), but real-life applications are generally not a major focus in traditional Science teaching (Seidel, 2003; Tesch, 2004; Widodo, 2004). As a rule, the results of studies on including real-life applications in science teaching show a positive correlation between students’ motivation to learn and their attitudes towards science (Kobarg et al., 2011).

Use of the PISA results

Another point we addressed to attempt to explain the significant difference in performance between Brazil and Japan was the use these two countries make of PISA results. For both program managers, in Brazil and Japan, the objective of the assessment is to inform public policies intended to improve the quality of education. However, unlike what happens in Brazil, where there are still many questions about what is effectively done with the diagnosis provided by large- -scale assessments, in Japan the results of the assessment are directly used in decision making to improve the quality of education. In view of the results of the first round of PISA (2000, 2003 and 2006), for example, three educational and curricular reforms have been carried out in Japan. In Brazil, the assessment manager points out that even with the experience of participating in the six editions of PISA, the country has not taken many steps towards, for example, improving teaching materials, textbooks, teacher training, curriculum, etc.

Japan’s high performance and Brazil’s low performance

In the view of the PISA manager in Japan, that country’s good performance can be explained by the reforms the country’s education has undergone based on the results of this international assessment. Briefly, the curriculum was revised, teachers received a publication with instructions about PISA, and the national assessment system was revived with items that, like PISA, assess skills. What OECD and the PISA manager in Japan pointed out as the reasons for the high performance of Japanese students appear in the words of the PISA manager in Brazil as the main difficulties and problems encountered by Brazilian students with regard to the Program and as factors explaining the low performance of our students and schools, and this involves lack of students’ familiarity with the test and lack of preparation for it, lack of infrastructure and lack of use of evidence provided by external assessments.

Japan has been at the top of the international rankings since the first education assessments, and there are characteristics of the Japanese educational system from which some lessons can be learned. During the development of our research, especially based on the contributions of experts in Science teaching, we observed some of these characteristics and verified how they relate to Japan’s high performance in large-scale external assessments, such as PISA. These are: a) the student promotion policy; considering that 100% of students are in the right grade for their age and that Japan is one of the few countries in the world where students do not fail to move on to the next grade from one year to the next; b) the common national curriculum which is revised every ten years; c) continuing teacher training done on-the-job; d) use of the time allocated to teaching action; and e) use of the assessment results to monitor, diagnose and promote the performance of the education system.

Conclusions

Based on PISA as well as the use of a qualitative approach, our objective was to identify different curriculum emphases and/or pedagogical practices in Science teaching in Brazil and Japan that could contribute to the understanding of the differences in student performance between these two countries. To this end, we sought to refine and broaden the potential for interpreting the quantitative results of our research by observing the use of time in Science classes; recording curriculum emphases and the occurrence of activities related to “interaction”, “investigation”, “experimentation” and “application” through observation of classes and perceptions of students and teachers; as well as interviewing experts and managers.

In Japan and Brazil, between November 2015 and October 2016, we observed a total of 147 Science lessons in the three final years of elementary school (64 hours of lessons in Japan and 83 in Brazil), given by 16 teachers (eight in Brazil and eight in Japan) in three schools in each country. The conclusions we reached can be summarized as follows:

1. More than 20% of the official class time observed in Brazil is wasted on issues other than the effective teaching of Science; ten times more than in Japan.

2. In Brazil the curriculum places stronger emphasis on natural and biological sciences. The curriculum is more evenly distributed in Japan, whereas it is sectioned in Brazil.

3. According to the Japanese students, interaction, investigation, experimentation and application activities are not frequent. The most common activities observed and perceived by Japanese teachers are experimentation and interaction; while in Brazil they are interaction and application.

4. Interviews with Science teaching specialists and PISA managers in Brazil and Japan showed that Japan’s success in this assessment is associated with the existence of a common national curriculum and with continuing on-the-job training of teachers, as well as with reforms of the Japanese education system prompted by the PISA results. The low PISA performance of Brazilian students appears to be related to their lack of preparation, their unfamiliarity with the test, poor teacher training and limited use of the evidence produced by large-scale assessments.

The results we analyzed suggest that Science teaching policies need to be accompanied by curriculum and teacher training policies that incorporate evidence and diagnoses produced by large-scale assessments. In Japan, this type of initiative shows more promise for generating changes in the curriculum and in teaching practices that positively influence student performance, rather than initiatives that hold students and teachers accountable and reward them and which, despite having been used for decades in Brazil, do not seem to have provided clear positive effects on student learning or on teachers’ curricular and professional development.

Acknowledgements

This work was conducted with support from the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil [Coordination for the Improvement of Higher Education Personnel - Brazil] (CAPES) - Funding Code 001.

References

• Abadzi, H. (2009). Instructional time loss in developing countries: Concepts, measurement, and implications. The World Bank Research Observer, 24(2), 267-290.
• Alves, F., Ortigão, M. I. R., & Franco, C. (2007). Origem social e risco de repetência: Interação entre raça--capital econômico. Cadernos de Pesquisa, 37(130), 161-180.
• Brandão, Z., Baeta, A. M. B., & Rocha, A. D. (1983). Evasão e repetência no Brasil: A escola em questão. Achiamé.
• Bruns, B., & Javier, L. (2014). Great teachers: How to raise teacher quality and student learning in Latin America and the Caribbean (Overview Booklet). World Bank Group.
• Bybee, R. W. (1993). Reforming science education: Social perspectives and personal reflections. Teachers College Press.
• Bybee, R. W. (1997a). Toward an understanding of scientific literacy. In W. Graber, & C. Bolte, Scientific literacy (pp. 37-68). Institute for Science Education.
• Bybee, R. W. (1997b). Achieving scientific literacy: From purposes to practices. Heinemann.
• Carroll, J. A. (1963). Model of school learning. Teachers College Record, 64, 723-733.
• Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning and instruction. Essays in honor of Robert Glaser (pp. 453-494). Erlbaum.
• Correa, E. V., Bonamino, A., & Soares, T. M. (2014). Evidências do efeito da repetência nos primeiros anos escolares. Estudos em Avaliação Educacional, 25(59), 242-269.
• Crahay, M. (2002). Poderá a escola ser justa e eficaz? Da igualdade das oportunidades à igualdade dos conhecimentos. Instituto Piaget.
• Deboer, G. E. (2000). Scientific literacy: Another look at its historical and contemporary meanings and its relationship to science education reform. Journal of Research in Science Teaching, 37(6), 582-601.
• Fensham, P. J. (2000). Time to change drivers for scientific literacy. Canadian Journal of Science, Mathematics, and Technology Education, 2, 9-24.
• Fielding, N., & Schreier, M. (2001). Introduction: On the Compatibility between qualitative and quantitative research methods. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 2(1), 1-21.
• Flick, U. (2005a). Triangulation in qualitative research. In U. Flick, E. Kardorff, & A. Steinke (Eds.), Companion to qualitative research (pp. 178-183). Sage.
• Flick, U. (2005b). Qualitative research in sociology in Germany and the US: State of the art, differences and developments. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 6(3), 1-21.
• Forquin, J. C. (1992). Saberes escolares, imperativos didáticos e dinâmicas sociais. Teoria & Educação, 5(1), 28-49.
• Freitas, M. A. T. (1947). A escolaridade média no ensino primário brasileiro. Revista Brasileira de Estatística, 8(30-31), 295-474.
• Guimarães, S. E. R. (2003). Avaliação do estilo motivacional do professor: Adaptação e validação de um instrumento [Tese de doutorado]. Universidade Estadual de Campinas.
• Hanushek, E. (2002). Teacher quality. In I. Lance, & E. Williams (Eds.), Teacher quality. Hoover Press.
• Hanushek, E., & Woessmann, L. (2010). Education and economic growth. In D. Brewer, & P. McEwan (Eds.), Economics of education (pp. 60-67). Elsevier.
• Harlen, W. (1999). Effective teaching of science. The Scottish Council for Research in Education (SCRE).
• Hattie, J. (2003). Teachers make a difference: What is the research evidence? https://research.acer.edu.au/research_conference_2003/4/
  » https://research.acer.edu.au/research_conference_2003/4/
• Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.
• Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning. Routledge.
• Hofstein, A., & Lunetta, V. N. (2004). The laboratory in science education: Foundations for the twenty-first century. Science Education, 88(1), 28-54.
• Kobarg, M., Prenzel, M., Seidel, T., Walker, M., McCrae, B., Cresswell, J., & Wittwer, J. (2011). An international comparison of science teaching and learning: Further results from PISA 2006. Waxmann.
• Law, N. (2002). Scientific literacy: Charting the terrains of a multifaceted enterprise. Canadian Journal of Science, Mathematics, and Technology Education, 2, 151-176.
• Maienschein, J. (1998). Scientific literacy. Science, 281, 917.
• Mayer, V. J. (Ed.). (2002). Global science literacy. Kluwer Academic Publishers.
• Mayer, V. J., & Kumano, Y. (2002). The Philosophy of Science and global Science literacy. In V. J. Mayer (Ed.), Global science literacy. Kluwer Academic Publishers.
• Millar, R., Lubben, F., Gott, R., & Duggan, S. (1995). Investigating in the school science laboratory: Conceptual and procedural knowledge and their influence on performance. Research Papers in Education, 9(2), 207-248.
• Muri, A. F. (2017). Letramento científico no Brasil e no Japão a partir dos resultados do PISA [Tese de doutorado]. Pontifícia Universidade Católica do Rio de Janeiro.
• Muri, A. F., Bonamino, A., & Soares, T. M. (2017). Funcionamento diferencial dos itens de ciências do PISA: Brasil e Japão. Estudos em Avaliação Educacional, 28(68), 538-570.
• Muri Leite, A. F. M., & Bonamino, A. (2020). Defasagem idade-série e letramento científico no Pisa. Estudos em Avaliação Educacional, 31(77), 393-420.
• Organisation for Economic Co-operation and Development - OECD. (1999). Measuring student knowledge and skills: A new framework for assessment. OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills. OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2006). The PISA 2006 assessment framework for science, reading and mathematics. OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2007). Competências em ciências para o mundo de amanhã: Análise (Volume 1). OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2010). Strong performers and successful reformers in education: Lessons from PISA for the United States. OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2013). PISA 2015 draft science framework. OECD Publishing.
• Organisation for Economic Co-operation and Development - OECD. (2016). PISA 2015 assessment and analytical framework. OECD Publishing.
• Ribeiro, S. A. C. (1991). Pedagogia da repetência. Estudos Avançados, 5(12), 7-21.
• Roberts, D. A. (1983). Scientific literacy: Towards a balance for setting goals for school science programs. Minister of Supply and Services.
• Roberts, D. A. (2007). Scientific literacy/science literacy. In S. Abell, & N. Lederman, Handbook of research on science education. Lawrence Erlbaum Associates.
• Roberts, D. A., & Bybee, R. W. (2014). Scientific literacy, science literacy and science education. In N. Lederman, & S. Abell (Eds.), Handbook of research on science education (Volume II). Routledge.
• Schibeci, R. A. (1984). Attitudes to science: An update. Studies in Science Education, 11(1), 26-59.
• Seidel, T. (2003). Lehr-Lernskripts im Unterricht. Waxmann.
• Soares, T. M. (2005). Utilização da Teoria de Resposta ao Item na produção de indicadores socioeconômicos. Pesquisa Operacional, 25(1), 83-112.
• Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Social Research Methods Series, 46.
• Taylor, J., Roehrig, A., Hensler, B., Connor, C., & Schatschneider, C. (2010). Teacher quality moderated the genetic effects on early reading. Science, 328(5977), 512-514.
• Tesch, M. (2004). Experimentieren im Physikunterricht - Ergebnisse einer Vide ostudie. Zeitschrift für Didaktik der Naturwissenschaften, 10, 51-69.
• United Nations Educational, Scientific and Cultural Organization - Unesco. (1993). International Forum on Scientific and Technological Literacy for All (Final Report). Unesco.
• Widodo, A. (2004). Constructivist oriented lessons. Peter Lang.

• Data availability
  The data underlying the text of the study are informed in the article.

• 1
  Gräber, W., & Bolte, C. (Eds.). (1997). Scientific Literacy: An International Symposium. Institute for Science Education at the University of Kiel.
• 2
  This was a longitudinal panel study that began in 2005. It observed the same sample of schools and students over a four- -year period. The Geres project involved association between six university centers with a tradition of education evaluation, namely: the Laboratório de Avaliação da Educação [Education Evaluation Laboratory] (LAEd) of the Pontifical Catholic University of Rio de Janeiro (PUC-Rio), the Grupo de Avaliação e Medidas Educacionais [Education Measures Evaluation Group] (Game) of the Federal University of Minas Gerais (UFMG), the Laboratório de Avaliação [Evaluation Laboratory] (Loed) of the State University of Campinas (Unicamp), the Linha de Pesquisa de Avaliação da Educação do Programa de Pós-Graduação em Educação [Education Evaluation Research Line of the Postgraduate Program in Education] of the Federal University of Bahia (UFBA), the Centro de Avaliação da Educação [Education Evaluation Center] (CAEd) of the Federal University of Juiz de Fora (UFJF) and the state University of Mato Grosso do Sul (UEMS).
• 3
  Prova Brasil and the SAEB are assessments for large-scale diagnosis developed by the INEP/Ministry of Education. Their objective is to evaluate the quality of teaching offered by the Brazilian education system through standardized tests and socioeconomic questionnaires.
• 4
  The Sistema de Avaliação de Rendimento Escolar do Estado de São Paulo [São Paulo State School Performance Evaluation System] (SARESP) is applied by the São Paulo State Education Department in order to produce a diagnosis of the basic schooling situation in São Paulo State, aiming to guide education system managers in the monitoring of policies directed towards improving the quality of education.
• 5
  The Base Nacional Curricular Comum [Common National Curriculum Base] (BNCC) was still under discussion at the time data was collected for this study. The BNCC was published in 2018, but still faces challenges with regard to national implementation.

Publication Dates

• Publication in this collection
  10 Dec 2021
• Date of issue
  2021

History

• Received
  04 Sept 2020
• Accepted
  27 July 2021