Assessment instruments and self-evaluation of reflective portfolios: a theoretical-conceptual construction

Cotta, Rosângela Minardi Mitre; Costa, Glauce Dias da

doi:10.1590/1807-57622014.1303

Abstracts

Reflective portfolios form part of the context of assessment of learning that is guided by feedback and interaction. There is a consensus in the literature and among specialists and students that the portfolio assessment process is weak. This study aimed to present and analyze an instrument for assessment and self-assessment of the skills to be used in reflective portfolios. The theoreticalconceptual construction was based on the authors’ nine years of experience of using reflective portfolios and on discussion of the literature. This led to a design that was tested and constructed collectively between teachers and students. The skills delineated in the instrument were communication and management of information, systems, people and groups, and were constructed focusing on university education that integrates academic and professional activities. Since the assessment process is consensual, it ensures commitment, accountability and transparency. These are produced through identifying and recognizing the instrument as legitimate.

Reflective portfolio; Active methods; Evaluation

O portfólio reflexivo inscreve-se no contexto de uma avaliação formativa pautada no feedback e na interação. A fragilidade no processo de avaliação dos portfólios é consenso na literatura e entre especialistas e estudantes. Este estudo objetivou apresentar e analisar o Instrumento de avaliação e autoavaliação das competências a serem trabalhadas no portfólio reflexivo (IAVCP). A construção teórico-conceitual baseou-se na experiência de nove anos de uso dos autores e na discussão da literatura, resultando em um desenho testado e construído coletivamente entre docentes e estudantes. As competências delineadas no IAVCP – de comunicação, gestão da informação, sistêmicas, pessoais e de grupo – foram construídas com enfoque na educação universitária que integra atividades acadêmicas e profissionais. O processo de avaliação, por ser consensuado, propiciou o compromisso, a corresponsabilização e a transparência, produzidos pela identificação e pelo reconhecimento do instrumento como legítimo.

Portfólio reflexivo; Métodos ativos; Avaliação

La cartera reflexiva se inscribe en el contexto de una evaluación formativa regida por el feedback y la interacción. La fragilidad en el proceso de evaluación de las carteras es consensos en la literatura y entre especialistas y estudiantes. El objetivo de este estudio fue presentar y analizar el instrumento de evaluación y auto-evaluación de las competencias por trabajar en la cartera reflexiva (IAVCP). La construcción teórico-conceptual se basó en la experiencia de nueve años de uso de los autores y en la discusión de la literatura, resultando en un diseño probado y construido colectivamente entre docentes y estudiantes. Las competencias delineadas en el IAVCP (de comunicación, gestión de la información, sistémicas, personales y de grupo) se construyeron con enfoque en la educación universitaria que integra actividades académicas y profesionales. El proceso de evaluación, por ser consensuado, propició el compromiso, la co-responsabilidad y la transparencia producidos por la identificación y el reconocimiento del instrumento como legítimo.

Cartera reflexiva; Métodos activos; Evaluación

Introduction

The current context of university education challenges the traditional paradigms of teaching, learning and assessment, which are linear and verticalized, and bets on a new holistic approach that enhances a dialogic education between teachers-students and students-students¹-³.

Among the different active methodologies of university education, the reflective portfolio stands out as an innovative method that “enables students to collect their opinions, doubts, difficulties, reactions to contents, to the studied texts and teaching techniques, feelings and situations experienced in interpersonal relationships. This methodology offers subsidies to the assessment of the student, the educator, the contents and the teaching and learning methodologies. In education, the portfolio offers many possibilities, and its main learning factor is that it is constructed by the student him/herself or by a group of students. Little by little, during the school semester, the student organizes his/her productions, which reveal his/her process of knowledge construction”¹ (p. 416).

In this perspective, the reflective portfolio is included in the context of competency-based education and formative assessment, in which learning is guided by constant feedback between teacher-student and student-student. The results must focus on the knowledge construction process in a dialogic and creative way, transcending the crystallized, restricted and classificatory format used in traditional assessments. According to Lizarraga⁴ and Cano⁵, competency-based education – which focuses on the necessary skills (competencies) that enable students to work in an active, responsible and creative way in the construction of their project of personal, social and professional life – must be conceived as a holistic and integrating view. The aim must be to overcome discipline fragmentation and the focus must be on the development of knowledge beyond a merely instrumental learning, incorporating understanding, reflection and criticism as dimensions inherent in it. Therefore, competency-based assessment is essentially formative, and this requires a radical change in the assessment paradigm: a shift away from an assessment of learning towards an assessment for learning.

Therefore, the university has the function of developing, in youths, cognitive, metacognitive, social, emotional, affective, technological and instrumental competencies, in view of the fact that these competencies need to become the main object of the educational system. According to Cotta et al.²,³, the exercise of competencies enabled by the reflective portfolio stimulates autonomy and the critical, reflective, creative and supportive spirit of students. However, at the same time that the scientific evidence highlights the portfolio’s potential as a didactic method that restructures the teaching and learning process, the same literature points to the fragility of the assessment process of students and portfolios, showing that it is necessary to create reliable, transparent and consensual assessment and self-assessment instruments¹-¹².

Aiming to overcome the fragility of the assessment of the portfolio as a method in the daily routine of universities, the objective of this essay was to present and analyze the instrument for teachers’ assessment and students’ self-assessment of the competencies used in the construction process of the reflective portfolio. To create this instrument, the authors relied on their broad experience of nine years working with reflective portfolios in different contexts and on a discussion of the scientific literature. We present and analyze the instrument here after it was tested, assessed and transformed into praxis, in a consensual, dialogic and critical way, by postgraduate and undergraduate students and teachers in different periods.

Methods

The Theoretical-Conceptual Framework and the Praxis that subsidized the construction of the Portfolio Assessment Instrument

The theoretical-conceptual construction of the Instrument for the assessment and self-assessment of the competencies used in reflective portfolios (in Portuguese, the acronym is IAVCP) was based on the authors’ experience and on discussions about the scientific literature of the area. Their experience includes both the construction (as students who constructed academic and professional portfolios, an experience that was fundamental to the acquisition of teachers’ skills related to the use of this method) and the orientation for the construction of portfolios (as teachers and facilitators in undergraduate and postgraduate programs).

Studies carried out by Driessen¹¹, Tartwijk and Driessen⁷, Lizarraga⁴, Cano⁵ and Cotta et al.¹-³ have systematized the cognitive and metacognitive competencies to be exercised in the portfolio, the criteria for a good practice of competency-based assessment and the applicability of these dimensions in the daily practice of portfolio construction in the university environment.

The Competencies Exercised in and Assessed by theIAVCP:

Nowadays, as never before in history, we need reflective individuals who understand the information, assess it and act over it; but we also need individuals who generate varied and original ideas, who avoid mistakes made in the past, who mitigate the large educational and economic inequalities, and who definitely create social surroundings characterized by wellbeing, justice and equity: an attractive and ambitious goal that students must achieve.⁴ (p. 9).

In the present study, the reflective portfolios, constructed according to a holistic approach in the context of significant learning and dialog between teacher-student and student-student, were based on the propositions of the report released by the United Nations Educational, Scientific and Cultural Organization (UNESCO), delineated in five fundamental pillars, as described below¹³,⁸,¹⁰,¹,³:

Learning to LEARN– Accepting new challenges and committing to the learning process, to the university and curriculum reform and to the implementation and (re)construction of social policies.
Learning to BE – Acting with autonomy, judgment and personal responsibility, that is, equipping oneself, as a human being, with freedom (thought, feelings, imagination, creativity…); crafting one’s own fate and history (empowerment); stimulating the development of a critical spirit and allowing the decoding of reality; acquiring an important autonomy of judgment with accountability in knowledge construction, academic “gains” and/or “losses”.
Learning to KNOW – Looking for and assimilating general and specific scientific and cultural knowledge that will complete one another and will be updated throughout the subject’s life.
Learning to DO – Acquiring procedures that help to cope with the difficulties that emerge in the subject’s life and profession.
Learning to LIVE and WORK TOGETHER – Understanding the others, the world and their inter-relations. The possibility of exercising, in teamwork, alterity, compassion, resilience and other “humanities”, dimensions that are very important nowadays, at the workplace and as a student and citizen; furthermore, the sensitization of looking and the qualification of listening, as well as the valuation of self-esteem.

In this context, both the teacher and the student play a central role in the development of the competencies that are necessary to the exercise of citizenship and professional practice. Students must commit, with responsibility and autonomy, to their learning process, aiming to acquire various technical, scientific, artistic and human capacities. Teachers must plan, guide, orientate, facilitate and stimulate the learning process, using teaching and technological information and communication methods (ICT) that help the future professionals to reach an integral and holistic education. Thus, in view of the current society’s demand for professionals who know how to understand the information and how to assess, create, make decisions and solve problems, what is intended, at the end, is to educate people who think for themselves and prove to be entrepreneurs, modifiers and regulators of their own actions, in a conscious, continuous and rapid way⁴.

The competencies delineated in the IAVCP were constructed based on a perspective of university education that overcomes the sum of discipline-based and fragmented knowledge, invests in a broad view that integrates the scholar and the professional, as well as knowledge, skills, attitudes and different intelligences, and aims at the education of an integral and integrated Human Being who reflects, analyzes and decides⁵. From this perspective, students are considered agents. According to Sen¹⁴ (p. 33), agent is “someone who acts and promotes changes, whose achievements can be judged according to their own values and objectives, no matter if we evaluate them according to some external criterion or not”.

The process of construction of the portfolio as an instrument that potentializes the development of competencies by students was intentionally instigated by its didactic divisions: Individual and collective construction of the concept of portfolio and identification of the main elements that characterize it as reflective (before and after the construction of the portfolio); My trajectory (record of memories through the answer to the questions: “Who am I? Where did I come from? Where am I going?”); Learning with the group (activities constructed collectively in face-to-face and distance moments); and Creativity space (a place that stimulates the exercise of inventiveness). The student is stimulated to search for information, select, analyze, make connections, make discoveries and innovate. The essential objective is to encourage students to interpret information (comprehensive thought), assess information (critical thought) and generate information (creative thought)¹,³,⁴

The Instrument for the Assessment and Self-assessment of the Competencies used in the Reflective Portfolio (IAVCP)

The Design of the Instrument

The construction of the IAVCP presented on Table 1 followed the criteria for a good practice of competency-based assessment proposed by Cano⁵, as described below:

Thumbnail

Table 1
Instrument for the Assessment and Self-assessment of the Competencies used in the Construction of the Reflective Portfolio

Consensual design: it requires collective work performed by the important agents of the process (teachers and students), taking into account the competencies described in legal norms and guidelines, after they have been debated, reformulated and amplified.
Collegiate articulation and connection between the broader objectives and the discipline and/or academic module:the articulation must occur in the academic space and with the external agents (health services, communities, professional entities, etc.).
Integration and application of learning: the competency is demonstrated in practice, when students solve and/or effectively understand and argue about a certain problem in a specific context. This exercise aims to enable students to analyze a certain situation and estimate which knowledge they have to apply. This requires reflection on practice, as students will constantly exercise their adaptation to the social and labor worlds, which are always changing.
Coherence and adequacy to the education design:intertwining, in a significant way, the contents of the discipline or academic module; resorting to evidences of the form in which students progress in the acquisition of the outlined objectives and goals in a way that is coherent with the methodology that was chosen.
Diagnostic or initial assessment: the point-of-departure is the profile of the graduate that we want to form. The competency has the evolutional character of what one gradually achieves; it is something that is constantly evolving. From the initial diagnosis, paths for progression can be established.
Formative assessment: the competencies must be in constant progression and growth. What is important is the capacity to learn how to learn and self-regulation processes for continuous improvement throughout life. In this context, we highlight the importance of feedback mechanisms – collective, individual, written, oral etc. – andself-regulation mechanisms of learning processes.
Assessment that registers the progression process in a continuous and longitudinal way: It enables both the teacher and the student to understand the evolution of learning and of the competencies, showing paths and strategies to overcome challenges and obstacles.
Multi-agent assessment: in the context of formative assessment, there must be the articulation of assessment processes whose assessment agents are the teachers and students themselves (self-assessment), focusing on the work of their peers (co-assessment or peer assessment), based on criteria that have been agreed upon and which are coherent with the competencies that will be developed.
Assessment that generates learning: competency-based assessment must advance beyond tests characterized by reproductive learning and memorization, which are little comprehensive and superficial, and must invest in assessments of execution that generate learning which is deeper, more functional, more interesting and significant; that is, a type of assessment for awareness-raising, decision-making and autonomy. This type of assessment must have regulation and feedback elements.
Assessment that generates satisfaction: when students are involved in practical activities connected with the social reality and/or the labor reality, they become more satisfied with the working process and with the assessment, provided it is performed in a coherent and transparent way. Designing significant and motivating assessment proposals generates more satisfaction.

The context in which the IAVCP was used, tested and assessed by students and teachers

The portfolio has been used as a method of teaching, learning and assessment by the team that coordinates the research line in Public Health and Teaching at Universidade Federal de Viçosa (UFV) since 2008. Based on this experience and on the students’ qualified demands, the IAVCP was created in 2010 and started to be used in 2011. Thus, the IAVCP was tested, assessed and modified in a consensual way during the construction of portfolios by undergraduate students (collective portfolios) and postgraduate students (individual portfolios) from programs in the area of Health, during the years of 2011, 2012, 2013 and 2014, in the disciplines of Health Policies and Health Promotion Policies, respectively. Overall, 587 undergraduate students used and tested this instrument and 97 collective reflective portfolios were constructed; in addition, 30 postgraduate students used and tested this instrument and 30 individual collective portfolios were constructed.

The assessment process of the reflective portfolios occurred on a monthly basis, in real time and in a continuous and longitudinal way, within a university teaching environment, totaling four assessments during each semester. The in situ assessment, in the presence of the students and in a dialogic way, facilitates the process of continuous conceptual study by means of feedback, and allows the timely and real-time regulation of distinct conflicts and problems, enabling conditions of dynamic stability and progressive development of autonomy and identity. This facilitates assessment and self-assessment procedures by means of the understanding of processes, orientations, feedback, resumption of directions and adjustments in their development³,¹².

During the school semesters, in each of the four moments of portfolio assessment (one assessment per month), simultaneously with their analysis with students or groups of students, one co-coordinating teacher noted down all the orientations, fragilities, strengths and potentialities presented in the portfolio and discussed with the students. These notes were read aloud by all and, after everybody had understood them and agreed that they were adequate, the page was signed and placed at the end of the portfolio, serving as a guide to the continuity of the portfolio construction process by the students. It was reread in the following assessment (in the following month) by the teachers, as it was considered an instrument that guided the assessment. Then, immediately after the presentation, discussion and orientation, firstly the students made the self-assessment and, at the end, the teachers in charge assessed the portfolios using the IAVCP. Finally, the students compared the teachers’ assessment with their self-assessment and, whenever necessary, they discussed conflicting points, if any.

Thus, after each moment of assessment and self-assessment, the IAVCP plays the role of guide and driver of the teaching and learning process, enabling the recognition, the revival, the recapture of the experience and the development of new understandings and appraisals.

Ethical aspects

This study was approved by the Research Ethics Committee – Protocol no. 091/2010, in accordance with Resolution no. 466/2012 of Brazil’s National Health Council, which regulates research involving human beings.

Results

The findings of this study have shown that the design of the assessment process, which was agreed by teachers and students, enabled the commitment and co-accountability of both parties and a transparent assessment process, produced by the identification and recognition of the instrument as legitimate. Taking the students’ needs into account and having their involvement, as well as the teachers’, in testing and transforming the IAVCP based on the portfolio construction and orientation experience, resulted in significant and motivating assessment processes.

The IAVCP proved to be a different type of assessment. It is coherent with the presuppositions of competency-based assessment, as it focused on the execution rather than on results, on processes rather than on products, in a diversified way and with different approaches and assessment agents (incorporating self-assessment by students), coherent with modes of assessing that include dimensions to analyze students’ competencies (knowledge, skills and attitudes). More than fragmenting or accumulating knowledge, students experience criteria in order to know how to select and integrate them, in a pertinent way, into a given context.

In addition to the assessments and self-assessments performed on a monthly basis in the classroom, individually (individual portfolios) and in groups (collective portfolios), totaling four in each semester, the students answered the question: “What factors motivate you (Up) and demotivate you (Down) to construct the reflective portfolio?” (Figure 1). The answers were written, individually and anonymously, on small pieces of paper that came in two different colors and were glued on the Poster “Up and Down Panorama”.

The Up and Down Panorama is a technique that aims to reveal, to teachers and students, in a pedagogical and playful way, what encourages and discourages students to construct the portfolio. Therefore, this technique aimed to understand the context that interferes in the process, not to quantify the answers. The strength of this technique is not in the frequency of the answers per se (positive or negative), but in the graphic representation of these answers (illustrated by means of “wordles”), in order to impact the agents involved in the process. This reveals strengths and fragilities and stimulates these actors to find ways of enhancing the work and overcoming difficulties. Therefore, the emphasis should be placed on potentialities and on the search for solutions.

Figura 1
Factors that motivate (Up) and demotivate (Down) students to construct the reflective portfolio (2011, 2012, 2013, 2014).

In the analysis of the Up and Down Panorama, we observed that the size of the words and/or expressions is proportional to the frequency with which they are cited by the students. Thus, the following elements emerged as the main factors that motivate (Up) the use of the reflective portfolio: teamwork, the type of learning, content, relationship to current themes and creativity, followed by connection between theory and practice, being critical-reflective, enabling discussions, type of innovative assessment and use of active methodologies. The following elements were the main demotivating factors (Down): time (lack), overload, difficulty in teamwork and the fact that it is a laborious method.

The students reported that the portfolio enables them to connect the contents studied in the classroom with the real world, the world of life. As students are stimulated to pay attention to what happens around them and to think about these events, actively searching for news transmitted by the media (lay and technical-scientific), a critical and reflective practice is instituted and, at the same time, these students become committed to the exercise of citizenship and to Sistema Único de Saúde (Brazil’s National Healthcare System). Furthermore, they reported that the criteria used in the assessment were transparent and clear, and this facilitated the visualization of their education process in a continuous and process-oriented way. Finally, they highlighted the development of competencies to understand and interpret the legislation and sanitary norms, which facilitated the decision-making process, not to mention the exercise of oral and written communication.

In view of the fact that knowledge and experience of learning are the bases of competencies, the results showed the pertinence of using the criteria of a good practice of competency-based assessment of portfolios employed in the construction of the IAVCP. Thus, we highlight the consensual design, the integration and application of learning and the diagnostic and formative assessment, which register the progression process in a continuous, longitudinal and multiagent way, generating learning and satisfaction. The link between assessment and acquisition of competencies in the construction process of the reflective portfolio contributes to the understanding of health policies (knowledge) and also to the development of skills and attitudes that are important for a professional action and for citizenship. We highlight the capacity for reflection, criticism and creativity to deal with the different opportunities and adversities that the daily routine imposes on us.

Discussion

Through the analysis of the portfolios, we identified that, after the implementation of the assessment and self-assessment instrument in 2011, the organization, the comprehensive thought, reflective thought, critical thought and creative thought have been gradually amplified and used in all the sub-divisions, as indicated in the studies carried out by Lizarraga⁴ and Cotta et al.¹,³

The findings of this study corroborate the results of studies conducted by Cano⁵, which show that the instrument is not merely an intuitive innovation; rather, it is grounded on a theoretical-conceptual framework associated with a previous reflection on the literature and with the teachers’ vast experience in the use of the portfolio as a method. Thus, it proved to be an excellent didactic and assessment instrument and revealed the different possibilities of the portfolio. While the students constructed the portfolios and paid attention to the competencies included in the IAVCP, they reflected on their own learning and progress based on the evidences that the method provided, with an emphasis on those that they accumulated during the studies and the semester that showed their progress, advances and potentialities, as well as their fragilities.

In this perspective, the IAVCP proved to be an assessment instrument that is adequate to the learning process. The teachers’ feedback, both oral and written, helped in the self-assessment process, and this made the assessment moments become excellent opportunities for growth and learning.⁵,¹⁰Therefore, the students’ responsibility for their own learning and assessment was developed. Their reflective activity was visible in the portfolios, documents and in the assessments and self-assessments that were performed in the four moments of use of the IAVCP, which compose their development in each school semester.

Thus, the IAVCP was “a different assessment that focused more on executions than on results, on processes rather than on products, diversified, with diverse instruments and assessment agents (incorporated into the co- and self-assessment), coherent with modes of working situated in scenarios which were as real as possible”⁵ (p. 48).Therefore, it constituted an assessment process beyond the analysis of yields. It enabled reflections on one’s own thought and on the work that was performed, the discovery of the mistakes that were made, the valuation of learning and evolution, the valuation of the level of personal satisfaction, and the specification of the relations among the developed activities, other academic contents and one’s personal or professional life⁴.

Conclusion

The IAVCP proved to be an important instrument for the assessment and self-assessment of the reflective portfolio. It enabled the teachers’ feedback to the students, the students’ feedback to the teachers, and the students’ feedback to the students. Furthermore, it guided the resumption of directions and the improvement in the competencies that were used. It was potentially important to stimulate the agents involved in the construction and/or orientation process to think about their own actions, know what they were doing and set goals and objectives for the continuity of the work. Moreover, it was important to outline changes in paths and directions, stimulating critical, reflective and creative thoughts, as well as teamwork.

The continuous and longitudinal assessments, conducted at four moments in each semester and based on the reflection and analysis of the reality portrayed in the different portfolios, enabled to capture information on the theme of Health Policies. Thus, the students could understand them and learn the causes of problems that emerge in the contemporary world, and could question these policies in order to enable the change of this reality. With this, the students develop the skill of observing the world around them, paying attention to the written, oral and television media, searching for new information in a curious, reflective and creative way, evaluating and drawing conclusions individually and in teams. In each assessment and self-assessment conducted by means of the IAVCP, students and teachers reflect on the advances that have been reached and set new goals and strategies, aiming to develop potentialities, to consolidate strengths and to overcome challenges and fragilities.

One limitation of the instrument is the lack of a column related to peer assessment, which would complement the assessment process.

Acknowledgment

This study is part of an innovation project in university teaching funded by Capes (Processo n. 23038.009788/2010-78, AUX-PE-Pró-Ensino Saúde 2034/2010).

References

¹ Cotta RMM, Mendonça ET, Costa GD. Portfólios reflexivos: construindo competências para o trabalho no Sistema Único de Saúde. Rev Panam Salud Publica. 2011; 30(5):415-21.
² Cotta RMM, Silva LS, Lopes LL, Gomes KO, Cotta FM, Lugarinho R, et al. Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem. Cienc Saude Colet. 2012; 17(7):787-96.
³ Cotta RMM, Costa GD, Mendonça ET. Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências. Cienc Saude Colet. 2013; 18(6):1847-56.
⁴ Lizarraga MLSA. Competências cognitivas em educación superior. Madrid: Narcea AS Ediciones; 2010.
⁵ Cano E. Buenas prácticas en la evaluación de competências: cinco casos de educación superior. Barcelona: Laertes Educación; 2011.
⁶ Friedrich DBC, Gonçalves AMC, Sá TS, Sanglard LR, Duque DR, Oliveira GMA. O portfólio como avaliação: análise de sua utilização na graduação de enfermagem. Rev Latino-am Enferm. 2010; 18(6):1-8.
⁷ Tartwijk JV, Driessen, EW. Portfolios for assessment and learning: AMEE Guide n. 45. Med Teach. 2009; 31:790-801.
⁸ Blanco A, coordenador. Desarrollo y evaluación de competencias en educación superior. Madrid: Narcea SA Ediciones; 2009.
⁹ Klenowski V. Desarrollo de portafolios: para el aprendizaje y la evaluación. Madri: Narcea SA de Ediciones; 2007.
¹⁰ Noguero FL. Metodologías participativas en la enseñanza universitaria. 2a ed. Madrid: Narcea SA Ediciones; 2007.
¹¹ Driessen EW, Overeem K, Tartwijk J, Vleuten CPM, Muijtjens AMM. Validity of portfolio assessment: which qualities determine ratings? Med Educ. 2006; 40:862-6.
¹² Sá-Chaves I. Portfólios reflexivos: estratégia de formação e de supervisão. Aveiro: Universidade de Aveiro; 2000.
¹³ Delors J. La educación encierra un tesoro. Madrid: Santillana; 1996.
¹⁴ Sen AK. Desenvolvimento como liberdade. São Paulo: Companhia das Letras; 2000.

Publication Dates

Publication in this collection
Jan-Mar 2016

History

Received
28 Jan 2015
Accepted
27 June 2015

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

[1] ¹ Cotta RMM, Mendonça ET, Costa GD. Portfólios reflexivos: construindo competências para o trabalho no Sistema Único de Saúde. Rev Panam Salud Publica. 2011; 30(5):415-21.

[2] ² Cotta RMM, Silva LS, Lopes LL, Gomes KO, Cotta FM, Lugarinho R, et al. Construção de portfólios coletivos em currículos tradicionais: uma proposta inovadora de ensino-aprendizagem. Cienc Saude Colet. 2012; 17(7):787-96.

[3] ³ Cotta RMM, Costa GD, Mendonça ET. Portfólio reflexivo: uma proposta de ensino e aprendizagem orientada por competências. Cienc Saude Colet. 2013; 18(6):1847-56.

[4] ⁴ Lizarraga MLSA. Competências cognitivas em educación superior. Madrid: Narcea AS Ediciones; 2010.

[5] ⁵ Cano E. Buenas prácticas en la evaluación de competências: cinco casos de educación superior. Barcelona: Laertes Educación; 2011.

[6] ⁶ Friedrich DBC, Gonçalves AMC, Sá TS, Sanglard LR, Duque DR, Oliveira GMA. O portfólio como avaliação: análise de sua utilização na graduação de enfermagem. Rev Latino-am Enferm. 2010; 18(6):1-8.

[7] ⁷ Tartwijk JV, Driessen, EW. Portfolios for assessment and learning: AMEE Guide n. 45. Med Teach. 2009; 31:790-801.

[8] ⁸ Blanco A, coordenador. Desarrollo y evaluación de competencias en educación superior. Madrid: Narcea SA Ediciones; 2009.

[9] ⁹ Klenowski V. Desarrollo de portafolios: para el aprendizaje y la evaluación. Madri: Narcea SA de Ediciones; 2007.

[10] ¹⁰ Noguero FL. Metodologías participativas en la enseñanza universitaria. 2a ed. Madrid: Narcea SA Ediciones; 2007.

[11] ¹¹ Driessen EW, Overeem K, Tartwijk J, Vleuten CPM, Muijtjens AMM. Validity of portfolio assessment: which qualities determine ratings? Med Educ. 2006; 40:862-6.

[12] ¹² Sá-Chaves I. Portfólios reflexivos: estratégia de formação e de supervisão. Aveiro: Universidade de Aveiro; 2000.

[13] ¹³ Delors J. La educación encierra un tesoro. Madrid: Santillana; 1996.

[14] ¹⁴ Sen AK. Desenvolvimento como liberdade. São Paulo: Companhia das Letras; 2000.