Improvement of Educational Products Validation form in professional postgraduate programsABSTRACT
Educational Product and Technical-Technological Product constitute a specificity of postgraduate programs in the professional modality, a product that must be carefully evaluated by the programs and recorded on platforms of evaluating bodies. In 2019, a working group from the teaching area was assembled to create a Validation Form for these products. Thus, this article aims to present a proposal for improving this Validation Form for Educational/Technical-Technological Products, based on reflections treated in scientific articles and scientific events from the area, focusing on the relevance and need for this type of evaluation in a standardized way. It is intended to collaborate with the improvement of the evaluation processes and procedures of Educational/Technical-Technological Products for the necessary accuracy, representativeness, and homogeneity, which will allow the real dimensioning of the contributions of these products. This involves not only training qualified professionals, but also transforming practices in the Sistema Único de Saúde (SUS) [Brazil's Unified Health System], promoting advances in science and in its fields of application.
Descriptors: Health Postgraduate Programs; Evaluation Of Health Research; Educational Technology; Scientific Research and Technological Development; Evaluation Studies as a Topic
RESUMEN
Producto Educacional y Producto Técnico-Tecnológico constituidos como especificidad de programas de posgrado en modalidad profesional, producción que debe ser cuidadosamente evaluada por programas y registrada en plataformas de instancias evaluadoras. En 2019, fue estructurado equipo laboral del área de enseñanza para criar Ficha de Validez de esos productos. Así, este artículo objetiva presentar una propuesta de perfeccionamiento de esta Ficha de Validez de Productos Educacionales/Técnico-Tecnológicos, basado en reflexiones tratadas en artículos científicos y eventos científicos de la área, enfocado en la pertinencia y necesidad de eso tipo de evaluación de manera estandarizada. Pretende colaborar con el perfeccionamiento de procesos y procedimientos evaluativos de Productos Educacionales/Técnico-Tecnológicos para necesaria precisión, representatividad y homogeneidad, que permitirá el real dimensionamiento de las contribuciones de esos productos. Eso envuelve no apenas formación de profesionales calificados, sino transformación de prácticas del Sistema Único de Salud, promoviendo avanzos en la ciencia y sus campos de aplicación.
Descriptores: Programas de Posgrado en Salud; Evaluación de la Investigación en Salud; Investigación Científica y Desarrollo Tecnológico; Tecnología Educacional; Estudios de Evaluación como Asunto
RESUMO
Produto Educacional e Produto Técnico-Tecnológico constituem-se como especificidade dos programas de pós-graduação na modalidade profissional, produção que deve ser cuidadosamente avaliada pelos programas e registrada em plataformas de instâncias avaliadoras. Em 2019, foi estruturado grupo de trabalho da área de ensino para criar Ficha de Validação desses produtos. Assim, este artigo objetiva apresentar uma proposta de aprimoramento desta Ficha de Validação dos Produtos Educacionais/Técnico-Tecnológicos, com base em reflexões tratadas em artigos científicos e eventos científicos da área, com foco na pertinência e necessidade desse tipo de avaliação de forma padronizada. Pretende-se colaborar com o aprimoramento dos processos e procedimentos avaliativos dos Produtos Educacionais/Técnico-Tecnológicos para necessária acurácia, representatividade e homogeneidade, que permitirá o real dimensionamento das contribuições desses produtos. Isso envolve não apenas formação de profissionais qualificados, mas também transformação de práticas do Sistema Único de Saúde, promovendo avanços na ciência e em seus campos de aplicação.
Descritores: Programas de Pós-Graduação em Saúde; Avaliação da Pesquisa em Saúde; Pesquisa Científica e Desenvolvimento Tecnológico; Tecnologia Educacional; Estudos de Avaliação como Assunto
INTRODUCTION
In 2019, during the Teaching Area Mid Term Seminar - Area 46, one of the evaluation areas of the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) [Coordination for the Improvement of Higher Education Personnel], which involves several professional master's and doctorate courses as well as some academic master's degrees, a working group (WG) was formed to discuss the Educational Product (EP)(1), with the task, among others, of proposing an Educational Product Validation Form. This proposal was organized using as reference the Report of the CAPES Technological Production WG(2) and detailed some possibilities of Educational Products and Processes that contemplate the typologies already defined within documents of the area(3).
In March 2020, an online format of the evaluation form proposed by this CAPES WG was disseminated at events linked to graduate programs and scientific journals - for example, through the article "The educational products and processes of postgraduate professional programs: proposals from a group of collaborators"(4).
In the aforementioned article, Rizzatti et al.(4) presented the fundamental elements for the discussion and construction of this evaluation instrument to be applied in postgraduate programs (PPG) that involve the production of an EP or Technical-Technological Product (TTP), most often produced in professional master's and doctoral degrees.
The discussion regarding the meaning and importance of the TTP or EP has been a target of interest and concern, both by the PPGs and by researchers in the CAPES teaching area. In addition to the mandatory nature of creating an EP/TTP in the professional modalities, there are unique aspects, which are not always evident or shared in the academic community:
In the professional modality, unlike the academic modality, students need to develop an Educational Product/Process (EP) that has to be applied in a real world scenario, and can have different formats. The academic community's lack of understanding of the characteristics of the professional modality has become a fertile field for criticism and disputes(4)
Zaidan, Reis, and Kawasaki(5) emphasize that there is a need to highlight the specificities of professional postgraduate programs, and this means the need to strengthen the identities of this type of program, highlighting, in the scientific community, their contributions to the training of professionals and for the production of knowledge and resources.
Another problem was pointed out in the study by Moreira et al.(6), in which a survey of educational products made in a professional master's degree in Science Teaching was carried out, and they identified that there is difficulty in providing visibility in certain EPs/TTPs, such as projects within extension programs and scientific dissemination activities. According to these authors, this is also due to the lack of financial resources from funding agencies, which determine the structuring of EP and TTP depending on the availability of this financing. As a result, they emphasize that there is a risk of "choosing" to develop a less "expensive" product in detriment to another that requires more financial resources.
Even though such difficulties are present in the daily lives of PPGs, in 2016, the Technical Production Classification was developed(7), which made it possible to include various educational materials in CAPES' Sucupira Platform and consolidated the relevance of this type of technical production as of the 2013-2016 quadrennial.
In addition to the classification of EPs/TTPs, their evaluation processes are of interest within this discussion. Araújo-Jorge, Sovierzoski, and Borba(8) present pertinent reflections on the importance of the EP evaluation processes, in which the authors highlight the need for changes. There are several proposals, among them, the emphasis on networking, the search for cooperative actions between programs and greater appreciation of EPs/TTPs. Another point considered crucial by these researchers is the issue of social inclusion, which was the subject of in-depth discussions within the scope of the Area Seminar held in November 2017, in which relevant aspects were pointed out to give visibility to the evolution of the programs in terms of reach or full implementation of EPs.
In view of these experiences and reflections, the CAPES WG responsible for discussing the EPs/TTPs developed a work aimed at responding to the demands and needs of the area and engaged in a valuable work to retake the area documents, seeking the memory and trajectory of construction of concepts and regulations. The WG participants critically returned to the concepts and typologies about EP and TTP, highlighting what kind of relationship these products must (or should) have with regard to the dissertation and thesis traditionally required in PPG. Therefore, the types of EP adopted were based on products declared by PPG in previous evaluations(4).
Another assignment of this WG was to formalize the need for the dissertations and theses of the professional modality to present a methodological section for the research developed and a section destined to explain the methodology of the EP development, described as follows:
[...]the dissertations and theses must contain a section or chapter in the body of the text addressing the methodology for the development of the EP: 1) containing the description of the steps to delimit the problem to be addressed; 2) definition of the stages, idealization, and elaboration of the EP; 3) prototyping (when applicable); 4) application, evaluation, validation (1st instance, minimum recommended for the MP); and 5) analysis in light of the theoretical and methodological framework(4)
For Rizzatti et al.(4), the expectation is that the EPs resulting from the MPs fully reach the first instance of assessment, which is recommended for the MP course and is mandatory for PD. For this working group, there are several forms of validation of the EP, such as focus groups, narratives, opinion polls, expert reviewers, among others.
The WG also clarified the need for the second instance of validation to take place formally, which can be done by the committee of a dissertation or thesis. To this end, they proposed a validation form that was developed based on proposals that are already in use by some PPGs and that was approved by the 2019 Area Seminar coordinators plenary(1). At this meeting, the proposal to apply this form to qualify EPs in Qualis Educacional (Quadrennium of 2021-2024) was also approved, which can be attached to the Sucupira Platform as a complementary information document on Technical-Technological Production.
Rizzatti et al.(4) propose, based on an active listening to the demands of various program coordinators present at the Seminar held by the teaching area in 2019, an EP evaluation and validation form, so that "The committee can evaluate the dissertation or thesis by issuing minutes and will analyze the EP with the issuance of this document, which should be considered as the validation document by specialized peers".
The WG, based on CAPES' technical report on technical production(1) and its expertise in the field, proposed the following concepts that should be considered in the analysis of EP/TTP: complexity, registration, impact, applicability, adherence, and innovation(4).
Thus, Rizzatti et al.(4), when proposing a Validation Form, they are also proposing an educational product and, as such, a product with important articulations and innovations that deserve recognition. However, like any product or technical technological process, it is evident that there is a need for visibility and debate to encourage its effective application, as well as continuous improvement. As these authors note(4), "Perhaps, it is necessary to evaluate how to give greater visibility to the EPs produced in professional PPGs, in the same way that we publicize articles in journals in the area".
It was in this "spirit" of contribution and improvement that the authors of the present article proposed an evaluation instrument based on the proposal of Rizzatti et al.(4), with some modifications that bring reflections of a theoretical and structural nature of the form proposed by the WG. In agreement with Moreira et al.(6), the evaluation of EPs/TTPs end up having a broader function than imagined:
Much more than innovative, an EP needs to be subversive [...] one that, even though it was designed for a specific context, can and must be adapted and reinterpreted. It must be proposed with an interdisciplinary character, establishing a bridge between the human, social, and nature sciences, licensed in the perspective of creative commons, free of charge, open to new inclusions.
In this line of reasoning and considering that the Validation Form proposed by Rizzatti et al.(4) can be considered a technical product, we agree with the reflections of Rôças, Moreira, and Pereira(9) when stating that the EP should not be taken "per se", but as a process of elaboration and transformation of those involved. Thus, the author of the EP is fundamentally involved in the problem identification process, considering theoretical and practical aspects, which leads him to reflect and propose solutions to address the identified problem.
This article, therefore, aims to present a proposal to improve this EP/TTP Validation Form, based on reflections on scientific articles and scientific events in the area, focusing on the relevance and need for this type of evaluation in a standardized way. Based on the proposal of Rizzatti et al.(4), the authors of the present article understand that there is room for improvements, which will be pointed out, point by point, indicating the reasons and suggestions of the new proposals for the Evaluation Form.
REFLECTIONS
Header related questions and proposals
The originally proposed form indicates the following aspects to be filled in: HEI; Student; Title of dissertation/thesis; Supervisor; Co-supervisor (if any).
In the present proposal for a new form, we understand that the HEI must be previously present in it, as it is an evaluation of a specific product of that institution and may carry the logo of the institution and/or the PPG. As for the item "Title of the dissertation/thesis", it is proposed that the new form contains the final title of the EP/TTP and the final title of the dissertation/thesis, since, considering that they are different scientific and/or technical productions, they may have different titles.
As for the names of the supervisor and co-supervisor, it is proposed that the new form contains the item "Authorship", in which all the authors and co-authors involved in the construction of the EP/TTP can be listed, in order to indicate the participation of the master's student, doctor, supervisor, co-supervisor and others, in the order that the authors deem to be the most appropriate, according to each one's contribution. This aspect is important as a way of responding to a discussion that was dealt with in the context of a CAPES online events on the topic, in which it was observed that many EPs/TTPs registered on the CAPES Sucupira Platform do not contain the names of the master's/doctoral students and advisors, which undermines the proper registration of such products on that platform.
With this proposition, the authors of the present article wish to provide an important reflection on the relationship between the dissertation/thesis and the EP/TTP in the context of professional PPGs. To this end, the work of Zaidan, Reis, and Kawasaki(5) is made explicit, who understand that there is an intrinsic relationship between the master's dissertation and the technical product produced, however, they recognize that, many times, the technical product ends up being prepared in a hurry and at the end of the process, which may incur in the development of a production unrelated to the dissertation carried out. Thus, these authors emphasize that "Reflections on experience show that the preparation of an educational product requires intentionality in guidance, already in methodological procedures".
So, it is noteworthy that, in addition to the need to treat EP/TTP with due attention, it is necessary that those involved in the process assume an explicit posture of shared authorship and insertion of the trajectory of the construction of EP/TTP in an articulated way to the dissertation/thesis, without losing sight of the fact that they are distinct and, at the same time, intimately articulated productions.
Questions and proposals on the title of the Validation Form proposed by the working group
The instrument proposed by the CAPES WG uses the term "validation" in its title, which has different meanings and interpretations depending on the field in which it is employed.
According to Villela(10), we can highlight at least two possibilities to understand this term:
Validation: confirmation through examination and objective evidence that certain requirements for the purpose of specific use can be adequately met [...] Validation protocol: written plan that establishes how the validation will be carried out, comprising the test parameters, the product characteristics, production equipment and decisive points about what constitutes acceptable test results.
Considering these different conceptualization possibilities, it is understood that the most common is the tendency to treat the term "validate" as an action of verification and comparison with an established benchmark or parameter. It is an operation with specific procedures that assign scientific qualities in comparison to a previous standard (gold standard). As the CAPES WG Form is a first proposition, there is no way to carry out a stricto sensu validation. Therefore, this article proposes to use the title "Educational Product (EP)/Technical-Technological Product (TTP) Evaluation Sheet", as the use of the term "evaluation" proves to be heuristically useful and meets the interests of the proposal.
Questions and proposals regarding the use of terms and concepts for evaluation
Although Rizzatti et al.(4) use the term "dimensions" in their article, when presenting the "EP/TTP Validation Form", in the body of the Form itself, they use the term "concepts". The authors of the present study propose the use of the term "dimensions" in the Form itself, as it allows us to foresee that the evaluated concepts are broad and complex. This gives visibility to the importance of the assessment task to be performed by members of the board, since the term "dimensions" is related to the extent, proportion, importance, value, magnitude, and influence of phenomena.
Proposal for the construction of levels between the different dimensions evaluated
The form originally proposed by the CAPES WG has two columns with no specific title for each one. In the first, it presents the concepts to be evaluated, with a brief description of them. In the second column, it presents alternatives with items that can be marked by the evaluators.
In a new proposal for an Evaluation Form, it is understood as an instrumental part that the evaluation carried out by the board can bring a reflection that standardizes assessment levels, proposed as follows: basic level, median level, and advanced level - as presented in Figure 1. This type of leveling can not only establish a relationship between the conceptual categories (name suggested by the CAPES WG) or dimensions (name suggested by the authors of the present work), but it can also favor the PPG to carry out a more detailed analysis (and, at the same time, an overall analysis) of the characteristics of the EPs/TTPs produced.
Proposal for an Educational Product/Technical-Technological Product Evaluation Form
1. Mark with an X the alternatives that correspond to the EP/TTP evaluation level; 2. In this part of the form, more than one item can be marked. The evaluation level will be formed by counting the considered items; 3. Add the total number of items marked on each level; 4. In the final opinion, make a qualitative assessment of the scores of the levels, strengths, weaknesses and suggestions.
In addition, this leveling proposal at three main levels seeks to bring an equivalent or articulated relationship between the different dimensions. Thus, based on the sub-items of the CAPES WG proposal, there was an effort to establish parallels or equivalences between the dimensions and the different levels of EP/TTP evaluation, as it was observed, in the form proposed by the WG, concepts with two, three, or four items to be analyzed.
The proposal of the authors of this article sought to treat items in a heuristic manner within a logic that allows a more accurate and standardized assessment, leading the interlocutor to recognize, in the three levels of assessment, an alignment that allows the different dimensions evaluated to have inter-relationship, consistency, and logic.
Questions and proposals on the evaluation of the scope and replicability concepts
The original form asks for a brief report on the aspects of the comprehensiveness and replicability of the EP/TTP. In this proposal, the intention was to align these concepts as well as the other dimensions. Then, concepts and items were proposed to be marked at the appropriate EP/TTP assessment levels. The conceptualization suggested for these terms (scope and replicability) was based on the article by Rizzatti et al.(4) and on the knowledge and experience of the authors of this article.
Questions and proposals on final registration in the information sheet of the Educational Product / Technical-Technological Product and the evaluation board participants
In the original form, a brief report on the scope and/or replicability of the EP is requested; signature of the president and members of the committee; and the date of the defense.
In the proposal of the authors of this article, it is understood that there are other elements that need to be explained in the form. First, when placing the scope and replicability assessment in the body of the form, there is no need for a general appraisal at the end of it.
Another issue is that, in this new proposal, it is necessary to highlight that this is an evaluation of the EP/TTP, which should not be confused with the minutes of the defense of the dissertation/thesis. On the other hand, following the reflections of the CAPES WG itself, it is important that the detailed information on the path of the construction of the EP/TTP is articulated with the dissertation/thesis and, thus, included as a specific part or item of this work. This need reflects both a specificity of the professional postgraduate area and the fact that EP/TTP is a production with its own characteristics and must be included in the CAPES system platforms as an autonomous product, but which is based on the research carried out during postgraduate school. Therefore, it is worth highlighting, in the proposal for the new Evaluation Form, whether or not there is a detailed description of the EP/TTP in a different item in the dissertation/thesis, explaining that they are different productions, but that they must have an articulation, although the proposed form designates a specific EP/TTP assessment.
Another point to consider is that the EP/TTP, according to the CAPES WG, can be evaluated in several ways, and, for them, the "EP/TTP Validation Form" reflects that:
[...]second instance of validation is mandatory for everyone and will be done by the defense board of the dissertation or thesis, based on the validation instrument proposed throughout this essay, having been developed from proposals that are already in use by some PPGs[...](4)
The need for an evaluation board for the EP/TTP is recognized, but this could be done on the date of the defense or not, being procedures with the possibility of occurring jointly or separately. Therefore, it is suggested that the date to be filled in the Form is that of the defense of the EP/TTP, even if it is the same date as the defense of the dissertation/thesis.
Finally, it is recommended that, at the end of the form, the EP/TTP evaluation panel build a brief final opinion, with a qualitative assessment of the marked items and their levels (basic, median, and advanced), identifying strengths and weaknesses, as well as, if desired, point out suggestions for improving the EP/TTP or the processes that involve the dimensions of the evaluation carried out.
In view of the considerations made, below, in Figure 1, the proposal to improve the original EP/TTP evaluation form is presented, in which the suggestions considered above are highlighted.
FINAL CONSIDERATIONS
Despite recognizing the efforts of the CAPES WG in organizing a document that provides a standardized assessment, as it is essential to obtain a more accurate overview of the EPs/TTPs, the authors of this article seek to advance proposals for improvement.
In fact, EPs/TTPs should be widely disseminated within the scope of HEIs, as well as in the areas of scope and interest related to the proposed material. However, it is necessary to recognize that PPGs do not always understand and value this type of technical production, and this means that professional postgraduate programs are not fully recognized (or known) in their specificities and unique contributions.
In addition to these considerations, taking as reference the area documents, it is understood that it is essential that the programs strive to standardize the evaluation processes (and subsequent validation, when applicable) of the EPs/TTPs in order to obtain an overview with the necessary accuracy, representativeness, and homogeneity. This will allow the real dimensioning of this fundamental work that not only involves the training of qualified professionals, but also contributes effectively to the transformation of ideas and practices that can promote effective advances in science and its fields of application.
REFERENCES
-
1 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Produção técnica. Grupo de trabalho[Internet]. Brasília: CAPES; 2019 [cited 2020 Dec 12]. Available from: https://www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-producao-tecnica-pdf
» https://www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-producao-tecnica-pdf -
2 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Relatório do Seminário de Meio Termo - Área de Ensino[Internet]. Brasília: CAPES; 2019 [cited 2020 Dec 12]. Available from: https://www.gov.br/capes/pt-br/centrais-de-conteudo/Ensino2.pdf
» https://www.gov.br/capes/pt-br/centrais-de-conteudo/Ensino2.pdf -
3 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Documento de Área - Ensino [Internet]. Brasília: CAPES; 2019 [cited 2020 Dec 12]. Available from: https://www.gov.br/capes/pt-br/centrais-de-conteudo/ENSINO.pdf
» https://www.gov.br/capes/pt-br/centrais-de-conteudo/ENSINO.pdf -
4 Rizzatti I, Mendonça AP, Mattos F, Rôças G, Silva MABV, Cavalcanti RJS, et al. Os produtos e processos educacionais dos programas de pós-graduação profissionais: proposições de um grupo de colaboradores. ACTIO: Doc Ciência. 2020;5(2): 1-17. https://doi.org/10.3895/actio.v5n2.12657
» https://doi.org/10.3895/actio.v5n2.12657 -
5 Zaidan S, Reis DAFR, Kawasaki TF. Produto educacional: desafio do mestrado profissional em educação. Rev Bras Pós-Grad. 2020;16(35):1-12. https://doi.org/10.21713/rbpg.v16i35
» https://doi.org/10.21713/rbpg.v16i35 -
6 Moreira MCA, Rôças G, Pereira MV, Anjos MB. Produtos educacionais de um curso de mestrado profissional em ensino de ciências. Rev Bras Ens Ciência Tecnol. 2018;11(3):344-363. https://doi.org/10.3895/rbect.v11n3.5697
» https://doi.org/10.3895/rbect.v11n3.5697 -
7 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Considerações sobre Classificação de Produção Técnica - Área de Ensino. [Internet]. Brasília: Capes; 2016 [cited 2020 Dec 3]. Available from: http://www.uern.br/controledepaginas/ppge-documentos-capes/arquivos/3404consideraa%C2%A7a%C2%B5es_sobre_classificaa%C2%A7a%C2%A3o_de_produa%C2%A7a%C2%A3o_ta%C2%A9cnica.pdf
» http://www.uern.br/controledepaginas/ppge-documentos-capes/arquivos/3404consideraa%C2%A7a%C2%B5es_sobre_classificaa%C2%A7a%C2%A3o_de_produa%C2%A7a%C2%A3o_ta%C2%A9cnica.pdf -
8 Araújo-Jorge TC, Sovierzoski HH, Borba MC. A Área de Ensino após a avaliação quadrienal da CAPES: reflexões fora da caixa, inovações e desafios em 2017. Rev Bras Ens Ciência Tecnol. 2017;10(3):1-15. https://doi.org/10.3895/rbect.v10n3.7744
» https://doi.org/10.3895/rbect.v10n3.7744 -
9 Rôças G, Moreira MCA, Pereira MV. "Esquece tudo o que te disse": os mestrados profissionais da área de ensino e o que esperar de um doutorado profissional. Rev ENCITEC. 2018;8(1):59-74. https://doi.org/10.31512/encitec.v8i1.262
» https://doi.org/10.31512/encitec.v8i1.262 -
10 Villela JRA. Validação de processos: um modelo utilizando ferramentas de qualidade e estatísticas [Dataset] [Internet]. 2004 [cited 2020 Dec 15]. Repositório da Produção Científica e Intelectual da Unicamp. Available from: http://repositorio.unicamp.br/jspui/bitstream/REPOSIP/265055/1/Villela_JoseRoqueAlves_M.pdf
» http://repositorio.unicamp.br/jspui/bitstream/REPOSIP/265055/1/Villela_JoseRoqueAlves_M.pdf
Edited by
-
EDITOR IN CHIEF: Antonio José de Almeida Filho
-
ASSOCIATE EDITOR: Maria Itayra Padilha
Publication Dates
-
Publication in this collection
18 Oct 2021 -
Date of issue
2022
History
-
Received
13 Feb 2021 -
Accepted
02 May 2021
