Brazil
RDBCI: Revista Digital de Biblioteconomia e Ciência da Informação
Table of contents previous current next
Table of contents
Article • Rev. Digit. Bibl. and Cienc. Inf. 24 • 2026 • https://doi.org/10.20396/rdbci.v24i00.8681266/en linkcopy

Open-access A tool for validating core and supplementary bibliographies for the INEP/MEC evaluation: a collaboration among libraries, faculty, and administrators

AuthorshipSCIMAGO INSTITUTIONS RANKINGS

ABSTRACT

Introduction:  Brazilian undergraduate programs are evaluated by the Ministry of Education, and university libraries play a key role in this process, particularly by providing information on collections and infrastructure. The INEP/MEC Evaluation Instruments, which guide institutional assessments, include two indicators related to core and supplementary bibliographies. Validating the adequacy of these bibliographies, however, is a complex process that involves librarians, faculty, and administrators and requires considerable time, effort, and financial resources.

Objective:  To propose an instrument that streamlines the bibliography adequacy validation process in view of MEC evaluations and supports the Núcleo Docente Estruturante (Structural Faculty Committee). Methodology: This is an experience report on the development of the instrument, grounded in: (1) documentary and bibliographic research; (2) brainstorming sessions with the library team and a group discussion with departmental representatives to identify challenges and propose solutions; and (3) direct observation of the workflow.

Results:  An Excel spreadsheet was developed that, based on data entered by faculty members, automates the identification of gaps in the bibliography and estimates the MEC score. Additionally, a website with tutorials was created to guide spreadsheet use and support the preparation of the Bibliographic Adequacy Report. These tools improved the efficiency of the verification process, enhanced communication among stakeholders, and provided greater predictability regarding evaluation indicators.

Conclusion:  Despite challenges related to data updates, the solution represented a significant advancement in managing undergraduate course collections and strengthened the library's strategic role in institutional evaluations.

KEYWORDS
Collection management; Academic libraries; Library administration; Evaluation.

RESUMO

Introdução:  Os cursos de graduação brasileiros são avaliados pelo Ministério da Educação e as bibliotecas universitárias desempenham papel importante nesse processo, especialmente no fornecimento de informações sobre acervo e estrutura. Os Instrumentos de Avaliação do INEP/MEC, que orientam a avaliação, incluem dois indicadores relacionados às bibliografias básica e complementar. A validação da adequação dessas bibliografias, contudo, é um processo complexo que envolve bibliotecários, docentes e gestores, demandando tempo, esforço e recursos financeiros significativos.

Objetivo:  Propor um instrumento que agilize o processo de validação da adequação das bibliografias com vistas à avaliação do MEC e sirva de apoio ao Núcleo Docente Estruturante. Metodologia: Trata-se de relato de experiência de desenvolvimento do instrumento embasado por: 1) pesquisa documental e bibliográfica; 2) brainstorming com a equipe da biblioteca e sessão de discussão em grupo com os representantes departamentais para identificação de dificuldades e proposta de soluções e 3) observação direta do fluxo de trabalho.

Resultados:  Foi desenvolvida uma planilha de Excel que, a partir dos dados informados pelos docentes, automatiza a identificação de pendências nas bibliografias e estima o conceito MEC, além de um website com tutoriais para uso da planilha e elaboração do Relatório de Adequação Bibliográfica. Tais ferramentas ampliaram a eficiência do processo de verificação, melhoraram a comunicação entre os atores e deram previsibilidade aos indicadores avaliativos.

Conclusão:  Apesar dos desafios na atualização dos dados, a solução representou um avanço na gestão do acervo dos cursos de graduação e fortaleceu o papel estratégico da biblioteca nas avaliações.

PALAVRAS-CHAVE:
Desenvolvimento de coleções; Bibliotecas universitárias; Processos de gestão; Avaliação.

1 INTRODUCTION

University libraries regularly participate in the Ministry of Education's (MEC) evaluation processes of undergraduate programs. During these evaluations, university libraries host committees of external evaluators who assess the programs' compliance with various preestablished criteria.

These processes are commonly known in academic circles as "MEC visits" and are guided by the Undergraduate Program Evaluation Instruments. These instruments support the authorization, recognition, and renewal of recognition for on-campus and distance learning programs at all levels (INEP, 2024).

For this study, the relevant document is the "Evaluation Instrument for On-Campus and Distance Undergraduate Courses: Recognition and Renewal of Recognition," divided into three sections: "1. Didactic-Pedagogical Organization," "2. Faculty and Tutoring Staff," and "3. Infrastructure." Each dimension contains indicators for which a quantitative score ranging from 1 to 5 is assigned based on the evaluation, with 1 being the minimum and 5 the maximum. The final weighted average of these scores, with greater weight assigned to dimension two, represents the course score (INEP, 2017a).

Indicators 3.6 and 3.7 are particularly important for libraries because they refer to the basic and supplementary bibliographies of the courses. Depending on the context and perspective, the term bibliography can refer to the physical and material description of the book as an object (the analytical or material branch), the systematization of repertoires and inventories intended for the control and location of works on a given subject (the enumerative or systematic branch), or the scientific field that studies books and other graphic materials in their intellectual, historical, and technological dimensions (Alentejo, 2023).

Due to the polysemy of the term, it is important to note that the bibliographies referenced in this work are specifically the lists of essential and supplementary references indicated by faculty members in the Teaching Plans for undergraduate programs, as listed in the Course Pedagogical Project (PPC). These bibliographies align with the enumerative branch, which focuses on the intellectual function of organizing and locating works according to the needs of a specific user community.

The collection, verification, and adaptation of bibliographies for assessment pose challenges for various stakeholders.

Among the purposes of the bibliographic references in the Course Syllabus, the following stand out: to serve as the official theoretical framework adopted by the instructor for developing the course curriculum to be taught during the academic term; to serve as a source of information for students in their daily studies, guiding them on the topics covered in class; to inform the library regarding which basic and supplementary bibliographies should be acquired and their respective quantities; to assist the Course Committee in verifying whether there is consistency between the recommended bibliography and the requirements of the Course Curriculum Plan (PPC); guide other faculty members in planning classes for advanced courses, as there is a sequence of topics to be mastered by students (Sousa, 2018, p. 81, emphasis added)

In addition to the stakeholders identified by Sousa (2018), it is important to note the involvement of administrators, particularly the Core Teaching Faculty (Núcleo Docente Estruturante - NDE). Administrators are involved because the results of the bibliography review often necessitate new acquisitions. The NDE is involved because it is responsible for ensuring that the bibliographies align with the course objectives before the MEC evaluation committees (INEP, 2017a). The NDE is responsible for monitoring and actively participating in the design, consolidation, and continuous updating of the course's pedagogical project. It is composed of course faculty members who work on its development (Brazil, 2010).

The literature highlights the need for collaboration among various institutional actors in constructing and verifying bibliographies for the MEC evaluation, as well as the need for tools to assist in this process. Sousa (2018, p. 184) recommends constructing the bibliography jointly with the library and faculty members. He also indicates the need for "[...] the creation of new means of communication to share knowledge regarding the construction of basic and complementary bibliographies [...]," while Moro (2022, p. 73) proposed a checklist to help libraries and faculty diagnose and adjust bibliographies for a favorable MEC evaluation. However, he did not discuss its practical implementation.

In light of this scenario, this study aimed to propose a tool to help fill the gaps. From the perspective of the university library, and in collaboration with other institutional actors involved in MEC evaluations, the tool supports NDEs in verifying and adapting the basic and complementary bibliographies for undergraduate programs. This proposal aims to make the process more agile, visible, and reliable, and to automate it as much as possible, since it is lengthy and laborious for everyone involved.

2 METHODOLOGY

This paper is structured as an experience report, following the definition and methodological framework proposed by Mussi, Flores, and Almeida (2021). According to the authors, this type of work involves presenting an academic or professional experience and describes an intervention carried out within a specific institutional context.

The aforementioned framework suggests that the "Introduction" section presents the key concepts of the topic as well as the report's importance and objective. The "Methodological Procedures" section should detail the time period, location, activity, type of experience, target audience, resources, actions carried out, instruments used, and analysis criteria. The "Results" section should present the results obtained and the main experiences encountered. The "Discussion" section should present a dialogue with the literature, commentary on the report's information, a critical analysis of the information, and identification of difficulties and potentialities. Finally, the conclusion should state whether the original objective was achieved and present future proposals. For this article, only elements and guiding questions relevant to the reported experience were incorporated. The results and discussions were combined into a single section, adapting the structure to the research's needs.

Two artificial intelligence tools, ChatGPT and DeepSeek, were used to review parts of the text for conciseness, clarity, spelling, and grammatical correctness. No tools were used to support the project's execution, the analysis of the results, or the writing of the original text of this experience report.

This experience was conducted at the Health Sciences Center Library (Biblioteca do Centro de Ciências da Saúde - BIBLIOCCS), which is part of the Integrated Library System (Sistema Integrado de Bibliotecas - SIB) of the Universidade Federal do Espírito Santo (UFES). SIB-UFES is a supplementary body directly linked to the Office of the President. Through the Central Library (Biblioteca Central - BC), it coordinates technical and administrative procedures for activities that support teaching, research, extension, and the management of the institution’s libraries.

BIBLIOCCS currently employs four librarians and five technical and administrative staff members. The library also relies on the collaboration of scholarship recipients and interns on a rotating basis. The library is located on the Maruípe campus in Vitória, ES, and its primary audience is the academic community of the Center for Health Sciences (Centro de Ciências da Saúde - CCS). The library serves the undergraduate programs in nursing, pharmacy, physical therapy, speech-language pathology, medicine, nutrition, dentistry, and occupational therapy. It also offers support to faculty, students, and researchers in graduate programs, as well as to the external community and professionals affiliated with the Hospital Universitário Cassiano Antônio Moraes (HUCAM).

In 2020, a joint request from the administrations of SIB-UFES and CCS revealed the need to improve the efficiency of verifying bibliographies used in undergraduate programs, particularly for INEP/MEC evaluations. To address this issue, a working group was formed and coordinated by BIBLIOCCS. Departmental representatives from the affected programs participated in the group.

BIBLIOCCS developed the methodological procedures for carrying out this initiative at two complementary levels. The first level focused on defining the research problem and comprised two parts. The first part was documentary research involving the reading and comparison of the INEP/MEC Undergraduate Program Evaluation Instruments (2015 and 2017 editions) and UFES's institutional regulations on the management of bibliographies and teaching plans. The second part was bibliographic research involving a review of the scientific literature on the verification and management of bibliographies in university libraries, especially in the context of MEC evaluation processes.

The second level focused on assessing the institutional situation and developing the instrument proposed in this report. This level involved a) brainstorming with library staff based on their experience monitoring course evaluations, and b) a group discussion with department representatives about the difficulties they face with bibliographies in their work.

Additionally, direct observation of the library’s collection verification workflow and its interactions with faculty and administrators complemented the data collection. This observation enabled the identification of operational and communication bottlenecks, collective challenges, and the demands of different institutional stakeholders.

Combining all this data made it possible to list problems and potential solutions. The BIBLIOCCS team subsequently systematized these solutions, leading to the development, description, and implementation of the bibliography verification tool presented in the results below.

3 RESULTS AND DISCUSSION

As previously mentioned, the document analysis conducted in this case study focused on two sets of documents: the 2015 and 2017 editions of the INEP/MEC Undergraduate Program Evaluation Tools and the UFES institutional guidelines on the management of reading lists and syllabi. Comparing these documents revealed changes in the evaluation criteria and helped us identify gaps and shared responsibilities, forming the basis for the proposed instrument.

A recurring point of uncertainty in the evaluation of bibliographies is their quantitative analysis. The 2015 Evaluation Instrument for On-Campus and Distance Learning Undergraduate Programs provided quantitative criteria for calculating grades. This instrument indicated the ideal number of titles per discipline and the number of physical copies required per available spot. The availability of a digital collection reduced the number of physical copies required. A grade of 5, for example, was awarded when:

[...] the collection of the core bibliography, comprising at least three titles per course unit, is available at an average ratio of one copy for every fewer than 5 annual slots sought/authorized for each course unit across all programs that actually use the collection; furthermore, it is digitized and registered as part of the institution’s assets (INEP, 2015, p. 36, emphasis added).

[...] the supplementary bibliography collection contains at least five titles per course unit, with two copies of each title or virtual access (INEP, 2015, p. 37, emphasis added).

However, the guidelines published by INEP/MEC in 2017 and currently in effect specify less objective criteria. To earn a grade of 5 in both the core and supplementary bibliographies, the requirements are now as follows:

The physical collection is cataloged and digitized; the virtual collection is covered by a contract guaranteeing uninterrupted access for users; and both are registered in the institution's name. The bibliographic collection [...] is appropriate for the course units and the content described in the Course Syllabus and is up to date, given the nature of the course units. Similarly, it is endorsed by a suitability report, signed by the NDE, confirming the compatibility, in each course unit’s basic bibliography, between the number of authorized seats (for the course itself and others that use the titles) and the number of copies per title (or access subscriptions) available in the collection. For digital titles, physical access is guaranteed at the institution, with facilities and technological resources that meet demand and ensure uninterrupted online availability, as well as accessibility tools and support solutions for reading, studying, and learning. The collection contains copies or virtual access subscriptions to specialized journals that supplement the content taught in the UCs.The collection is managed to update the number of copies and/or access subscriptions for the titles in highest demand, with a contingency plan in place to guarantee access and service (INEP, 2017a, p. 33, emphasis added).

The lack of objective criteria creates some difficulties regarding what should be considered “adequate.” Nevertheless, it is up to the NDE to determine this adequacy, including the number of titles and copies (INEP, 2017b). Sousa (2018, p. 102) frames the dilemma as follows:

Even if the body responsible for evaluating undergraduate programs is tasked with validating the number of credits, the question remains: how will the NDE evaluate and/or justify the number of core and elective courses based on the number of seats in a curriculum component, without parameters that account for the time students need to master the course material?

Therefore, the 2017 update to the MEC’s evaluation tool signals a shift from quantitative to qualitative evaluation and an increased emphasis on the value of digital collections. The update also introduces collection management as a criterion and specifies the NDE’s responsibility for certifying the adequacy of bibliographies. While the library and its staff are not explicitly mentioned, their active involvement in the process is essential. Collection management should not be carried out solely with MEC visits in mind.

As a tangential reflection on the scope of this report, it is important to reinforce the distinction between the verification and diagnosis of collections. According to Figueiredo and Cunha (1967), the work developed here, which comprises identifying titles indicated in teaching plans, fundamentally corresponds to bibliographic verification rather than strictly to collection diagnosis. This is a predominantly quantitative analysis that focuses on the didactic dimension of the collection (Miranda, 1980), while also providing input for more qualitative analyses.

On the other hand, the nature of diagnosis is broad and multidimensional. It focuses on evaluating the collection as a whole and considers criteria such as thematic representativeness, depth, timeliness, and coverage in relation to the needs of the community it serves. These needs are not necessarily reflected in teaching plans. For example, they may not include activities that support the external community through outreach, preserve institutional memory, or promote accessibility. Therefore, limiting collection management to evaluative processes would neglect essential dimensions for fulfilling a university library’s mission.

Nevertheless, verifying and validating teaching plan bibliographies is costly and complex. Therefore, we developed an approach to analyze this information more rationally, quickly, and efficiently in light of the indicators in the INEP/MEC Undergraduate Course Evaluation Instrument.

It was necessary to automate stages of the process that did not require subjective judgment or in-depth qualitative analysis. Until the project's conclusion, the compilation of bibliographies was predominantly manual and based on Word documents. This made it impossible to extract and process the data provided by faculty members automatically. While the initial collection of relevant faculty information (titles, publication years, publishers, and number of copies) could not be automated, it was recognized that this collection could be carried out more efficiently and that a significant portion of the subsequent analyses could be automated.

Based on data collected from library staff, the primary objective was to develop a tool that would visualize the bibliographic situation while requiring minimal effort from various users, including librarians, faculty members, NDE staff, and administrators, and facilitate collective adoption throughout the process. The tool was designed to be dynamically updated to enable simultaneous and asynchronous work, eliminating the need for new investments in human or technological resources. Thus, a viable solution was sought within the existing institutional infrastructure without expanding the team or contracting software development services.

Based on these premises, guiding questions were formulated that could potentially be automated. Together, these questions guided the development of the proposed tool.

  • a) If the program were evaluated today, what grade would the Ministry of Education (MEC) assign to the bibliographies?

  • b) Do all courses meet the minimum requirements for the number of titles and copies? If not, which ones fail to meet these requirements?

  • c) How many and which titles are outdated, how many are available online, and which ones need to be purchased?

  • d) What are the “emergencies”-that is, the courses and titles that could negatively impact the course’s grade and should therefore be prioritized?

  • e) Which are the main publishers and databases responsible for the titles listed in the syllabi?

  • f) When was the last review of the course bibliographies conducted?

These questions formed the basis for structuring the instrument and defining the analytical parameters to be incorporated into the project’s methodology. This aligned the technical development with the evaluation framework of INEP/MEC and the day-to-day needs of university libraries.

In a second stage, based on additional feedback from other institutional stakeholders, the main difficulties related to the bibliography verification process were identified. The purpose of this phase was to identify workflow challenges and propose solutions, even if only conceptual, to support the development of procedures and tools better suited to the institutional reality.

Chart 1 summarizes the main issues identified during discussions with different groups of participants, including administrators, program coordinators, NDE members, and librarians. It highlights the difficulties raised and possible solutions suggested through collective dialogue.

Thumbnail
Chart 1
Key challenges in verifying bibliographies from a collective perspective.

Various members of the institution believed it was the library’s responsibility to verify the number of titles and copies in the bibliographies and assess their appropriateness. However, during the second phase of the project, discussions revealed several issues regarding the quality and consistency of the bibliographic references in the course syllabi.

The library repeatedly identified errors, repetitions, and inconsistencies in the references submitted by faculty members. This made it difficult to locate and verify the corresponding titles in the collection and their quantities. These errors made the verification process more time-consuming and prone to misinterpretation.

Another issue was that certain works cited by faculty members were unavailable at the institution, though equivalent or similar titles were available as substitutes for the original references. However, the library lacks the autonomy and disciplinary expertise to determine the pedagogical appropriateness of such substitutions, especially in highly specialized fields, where only the faculty member responsible for the course or the NDE could assess the academic relevance of the works.

In light of this finding, the library presented an argument that the other participants accepted without objection. The argument was that the need to consult with faculty members and the NDE individually to validate or correct missing references led to rework and process inefficiencies. The library argued that it would be more productive and reliable if faculty members (or the NDE, depending on the case) entered this data directly into a unified tool containing information on the bibliographies for their courses. This would eliminate the need to send bibliography lists to the library for verification via email or another format. Faculty members already possess the technical expertise and formal responsibility for their content, so this would be a more efficient solution. This practice would reduce redundancies, increase data accuracy, and strengthen the reliability of bibliography verification.

This proposal aligns with UFES's institutional regulations, which assign responsibility for verifying and updating references to faculty members and academic management bodies.

When proposing and approving changes to basic and/or supplementary bibliographic references, the bodies, sectors, and/or entities involved must take into account the availability at the Central and/or Departmental Libraries, both regarding the existence of the material and the quantities necessary to meet the needs of the students in the courses that use the same bibliography (UFES, 2017a, Art. 3, § 3, emphasis added).

At UFES, Normative Instruction No. 1/2017-Prograd, amended by IN No. 03/2017, establishes the inclusion of the Teaching Plan on the Faculty Portal under the faculty member’s responsibility. [...] It is the faculty member’s responsibility to organize their Teaching Plan on a semesterly basis with the appropriate schedule, following the Course Description, basic and supplementary bibliographies listed in the PPC, and to submit it for approval by the academic department responsible for offering the course, as determined by the UFES Regulations, in Articles 30 and 91, and subsequently to disseminate and discuss it with the students enrolled in the course. Any changes to the schedule agreed upon with the student classes must be reported to the Department for record-keeping and to the Collegiate Body for monitoring (UFES, 2017b, topic “Teaching Plan for Undergraduate Course Courses,” emphasis added).

Based on the described scenario, a proposal was developed to implement the bibliography verification process. This proposal is grounded in the systematization of identified needs and the search for practical, sustainable solutions within the institutional context. The proposal consisted of creating support tools and defining a standardized workflow that could integrate the various stakeholders in a coordinated and efficient manner. This would strengthen communication between departments, promote greater transparency and traceability, and improve the acquisition of new bibliographies.

Thus, the proposed model sought to balance operational simplification with regulatory compliance, enabling the library to perform its technical function more efficiently. At the same time, faculty and academic departments shared academic responsibility for the appropriateness of the bibliographies.

Considering the library staff's expertise and the institution's available resources, we chose to use Microsoft Excel to create a structured spreadsheet. We chose Excel because of its broad accessibility and versatility in using formulas and automation. These features allow us to create dynamic solutions without developing specific software or investing in additional technological infrastructure.

The spreadsheet is designed to be accessed via a web address, which enables remote work for all interested parties. Thus, data entry for bibliography titles and the number of works, as well as analysis of their suitability, can occur both synchronously and asynchronously, facilitating collaborative work among participants. Two distinct access modes were also implemented: one with editing permissions for faculty members and academic departments responsible for entering and updating data, and one with view-only access for audiences such as students, institutional administrators, and evaluation committees. This increases transparency regarding bibliography management.

The spreadsheet consists of seven main tabs, each with functions related to collecting, verifying, and analyzing bibliographies. Below, in subsections 3.1 through 3.8, the names and functions of each tab are presented, along with screenshots of the respective screens. A completed sample spreadsheet is available for download and testing on the Zenodo repository1. Any institution may use it, if it wishes, by adapting it to its specific context.

Additionally, a website was developed to document how to use the spreadsheet. It brings together support content in text and multimedia formats to centralize information, guidelines, and tutorials specifically for faculty, program coordinators, and NDEs. Subsection 3.9 presents the publicly available website that supports the process2.

3.1 Spreadsheet - “Bibliographies” Tab - Process Operations Center

This specific tab is the only interface in the spreadsheet that requires manual data entry, making it the operational core of the entire workflow. The information in the other tabs of the tool is automatically derived and updated based on the data entered in this master tab.

In this interface, the instructors responsible for each course enter bibliographic data into a structured table. The columns of this table include a set of essential metadata about each work in the bibliographies, including, but not limited to: complete bibliographic reference in ABNT format, publisher, purchase price, classification of the bibliography type (basic or supplementary), number of physical copies available in the collection, and indication of the work’s digital availability in subscribed databases. Figure 1 shows an example of the column structure in the “Bibliographies” tab.

thumbnail of Figure 1
Figure 1
Screenshots of the Bibliographies tab

The table contains several columns that are calculated automatically based on data provided by faculty members. For instance, one column shows the predicted grade for each assignment using a predefined algorithm (see Section 3.8 for methodological details). There are also columns designed to guide the NDE. For instance, these columns calculate the ideal number of physical copies needed to achieve a maximum grade of 5 and the number of copies to acquire to reach that ideal. Faculty members can then critically analyze the works in their course syllabus in real time, and the NDE has access to information intended to consolidate acquisition needs and their respective technical and pedagogical justifications.

To ensure data integrity and completeness, the table is equipped with automatic visual validation mechanisms. These features highlight mandatory fields that have not been filled out and flag logical inconsistencies in the records. For example, they indicate the availability of electronic access without properly specifying the corresponding database. These features aim to minimize errors and ensure the quality of information collected from faculty members.

3.2 Spreadsheet - “Dashboard” Tab - Automated Analysis and Overview of Bibliographies

The tab titled “Dashboard” is a module responsible for the consolidation and automated presentation of key metrics for bibliographic management. This feature aggregates and analyzes data entered in the manual entry tab (“Bibliographies”), transforming it into strategic information to support decision-making by the NDEs and the library itself. Figure 2 shows examples of the “Dashboard” tab screen.

thumbnail of Figure 2
Figure 2
Screen examples from the Panel tab

The information is organized and displayed in the following main categories:

  • a) Compliance and consistency indicators: Provides quantitative data on outstanding issues, such as the number of works with unfilled required fields (e.g., title, publisher, year of publication). Additionally, it identifies logical inconsistencies in the records, such as indicating online availability without specifying the database where the work can be accessed.

  • b) Educational suitability indicators: Highlights points of attention regarding bibliographic composition, such as the number of courses that have fewer or more titles than the minimum recommended for each type of bibliography (core and supplementary).

  • c) Projections and estimates: Provides automatic grade estimates based on a predefined algorithm and cost projections for acquisitions, enabling preliminary budget planning.

  • d) Time analysis (age of the collection): Calculates and displays the temporal distribution of titles, categorizing them as works published before or after a specific cutoff year, which allows for an assessment of the collection’s currency. Consolidated general data: Summarizes the overall totals for the program, including the total number of titles, the number of courses, and the volume of works with available digital access, offering a macro view of the bibliographic situation.

In short, the “Dashboard” tab serves as a control dashboard, centralizing diagnostic analysis and bringing efficiency, objectivity, and transparency to the process of evaluating and managing bibliographies.

3.3 Spreadsheet - “Filters” Tab - From macro analysis to micro intervention

While the “Dashboard” tab provides a broad, quantitative overview, allowing users to identify how many items in the bibliographies may require attention from the NDE and the library, the ‘Filters’ tab enables corrective action by identifying which items these are. Figure 3 shows an example of the “Filters” tab screen.

thumbnail of Figure 3
Figure 3
Example of the “Filters” tab screen

With the dynamic, multidimensional filtering interface, you can apply selection criteria to any field in the bibliography table. For example, the bibliography list can be filtered by anticipated concept to prioritize works in critical condition, by publisher or year of publication to analyze representativeness and timeliness, by discipline to focus on specific course units, or by access type to list titles lacking digital access.

The tool's analytical potential lies in its ability to create combined filters to identify complex scenarios. For instance, you can generate an instant list of "titles without digital access and with fewer physical copies than ideal" or "titles published more than ten years ago and classified as core bibliography." These combinations allow for precise, targeted screening, moving beyond realizing that "there are problems" to clearly identifying "exactly what the problems are and where they are located." When the filters are activated, the Bibliographies tab displays only the titles that meet the filters' criteria, hiding the rest. When the filters are deactivated, all titles are displayed normally again.

Thus, an efficient workflow is established: the "Dashboard" tab signals the existence and scale of issues to be resolved, and the "Filters" tab facilitates their identification. This allows interventions to be based on concrete, detailed evidence, optimizing the time and effort of the NDEs and the library.

3.4 Spreadsheet - “Number of Titles per Subject” Tab - Alignment of Titles

This tab automatically displays a summary of the number of titles in the core and supplementary bibliographies for each course subject. The interface uses visual validation mechanisms, such as color coding, to highlight deviations from predefined quantitative parameters. Figure 4 shows an example of the “Number of Titles per Subject” tab screen.

thumbnail of Figure 4
Figure 4
Example of the “Number of Titles per Subject” tab screen

In this tab, titles with quantities below the recommended minimum are highlighted in orange, indicating courses whose insufficient bibliographic composition may negatively affect the grade during evaluation. On the other hand, titles exceeding the minimum requirements are highlighted in green, indicating situations that, while not detrimental to the evaluation score, may suggest redundancy or generate non-essential acquisition demands, with implications for the efficient allocation of financial resources. The feature thus enables a quick and targeted analysis, focusing the NDE and library’s review efforts on the cases that most require intervention, whether to address gaps or to optimize investments.

3.5 Spreadsheet - “AverageConceptsPerSubject” Tab - Adjustment of copies

The primary function of this tab is to automatically consolidate and display the average rating for the core reading list for each course. While the “Reading Lists” tab assigns an individual rating to each title, this module synthesizes that information into an overall indicator for each course. Figure 5 shows an example of the “AverageRatingsByCourse” tab screen.

thumbnail of Figure 5
Figure 5
Example of the “Average Grades by Course” tab

The mechanism calculates the arithmetic mean of the ratings for the titles in the core bibliography of each course. To help identify priorities and critical issues, the interface uses a color-based alert system. Averages that fall below the predefined minimum desirable threshold are automatically highlighted in orange.

This color-coded system quickly and efficiently identifies courses with insufficient copies in their core bibliographies, thereby requiring further analysis by the NDE and corrective action by the library. Thus, the tool aims to optimize the management process by directing efforts where they are most needed.

3.6 Spreadsheet - “Print” Tab - Automatic formatting for academic documents

Because the data in the "Bibliographies" tab is in a tabular layout, transferring this information to institutional documents, such as course syllabi and teaching plans, requires significant reformatting. Because of their formal nature, these documents require references to be presented in continuous text and in a layout optimized for A4 format. The purpose of the "Print" tab is to address this formatting mismatch and eliminate the need for manual adjustments.

This feature serves as an automatic restructuring module that reorganizes bibliographic references from raw spreadsheet data into continuous text, properly segmented by period, discipline, and bibliography type (core or supplementary). Figure 6 shows an example of the "Print" tab screen.

thumbnail of Figure 6
Figure 6
Example of the Print tab screen

The output generated in this tab most closely matches the formatting standards required for the aforementioned academic documents. As a result, the content can be copied and pasted directly into the final documents, preserving the proper structure. This automation represents a substantial productivity gain, ensuring standardization and reducing errors inherent in manual transcription and formatting.

3.7 Spreadsheet - “Basic Concept” Tab - Estimate of the final score for the Basic Bibliography

The MEC scores calculated and presented in the other tabs of the spreadsheet-whether at the level of individual works or as averages by subject-serve primarily as indicators of the collection's strengths and weaknesses. However, they do not accurately reflect the methodology prescribed by the INEP/MEC assessment tools. The “Basic Score” tab aims to generate an estimated consolidated score for the entire Basic Bibliography that more accurately aligns with the official evaluation logic. Figure 7 shows an example of the “Basic Score” tab screen.

thumbnail of Figure 7
Figure 7
Example of the “Basic Concept” tab screen

The specific criteria that the library used to determine the “closest approximation” are detailed in section 3.8. To facilitate interpretation of the results, the interface displays a visual alert that highlights in yellow the final scores that fall below the minimum acceptable threshold. In this way, the tab provides managers with a more reliable projection of expected performance regarding bibliographies, supporting informed corrective actions even before the formal evaluation process begins.

3.8 Notes on the calculation of the items

As mentioned in the introduction to this paper, the INEP/MEC Undergraduate Program Evaluation Tool, which has been in effect since 2017, employs less objective criteria for assessing the quality of required and supplementary reading lists than the previous tool, introduced in 2015. The current model does not explicitly reference minimum numbers of titles or copies, leaving it to the NDE to define what constitutes "adequate."

After discussions with the UFES Institutional Evaluation Secretariat, a hybrid model for calculating bibliography-related scores was developed to reconcile the objectivity of the 2015 instrument with the interpretive flexibility of the 2017 instrument.

Accordingly, the spreadsheet was configured to automatically calculate scores based on quantitative values and parameters established in the 2015 instrument, such as the ratio of titles to copies to available spots. However, under the 2017 instrument, an exception was made for works available in digital format, which automatically receive a score of 5. The same weighting logic was applied to the calculation of scores by discipline.

Please note that these parameters are intended as guidelines, not regulations. They are designed to support the critical analysis of NDEs by offering an objective starting point for verifying and adapting bibliographies. Different interpretations may be adopted, provided they are justified by the specifics of the course and supported by arguments identified during curriculum planning. Thus, the hybrid model balances technical rigor and academic autonomy to contribute to a more transparent, coherent, and institutionally adapted evaluation process.

Based on the experience reported here, one aspect worthy of reflection concerns the need for a collection development policy as a formal document that grounds and legitimizes the library's decisions. As previously presented, the INEP/MEC evaluation instrument adopted more subjective and qualitative criteria in place of the explicit quantitative parameters of the previous instrument, which has been in effect since 2017. Subjectivity poses a challenge but also presents an opportunity for the library to establish its own guiding parameters with the support of a formally established policy.

In this sense, a collection development policy can grant the library greater technical autonomy. It can transform the subjectivity inherent in evaluation criteria into an opportunity to demonstrate the rigor and consistency of collection management. This demonstrates that collection development is grounded in the institutional mission, the needs of the university community, and the specific characteristics of the programs served.

3.9 The website and library support

The library is committed to providing faculty members and academic departments with technical and informational support at every stage of the bibliography verification and updating process. Support is provided via email and the institutional phone line, with a maximum response time of two business days. Questions submitted by faculty members and their respective answers are systematically recorded, reviewed, and compiled into a knowledge base accessible on a website developed specifically for this purpose.

The website, bibliografiasccs.wordpress.com, was designed as a central repository of information, guidelines, and support materials for this process. The main sections are accessible via the menu at the top and include the following:

  • a) Frequently Asked Questions: a compilation of questions raised by faculty members throughout the process and their respective answers;

  • b) Tutorials for Faculty: demonstrations on how to fill out the spreadsheet and search the library catalog and databases;

  • c) Tutorials for the NDE: video demonstrations of the spreadsheet’s filtering and analysis capabilities;

  • d) Glossary of spreadsheet fields: describes the type of information that must be entered in each field so that analyses are generated automatically without errors;

  • e) Bibliographic adequacy report: provides a template for the document to be prepared, approved, and submitted by the NDE to the evaluation committees, containing information about the library, the bibliography verification process, and space for the final list of titles and any acquisition needs. To avoid rework and contribute to the NDE’s productivity, the library has also developed a tool that, based on the original spreadsheet, generates bibliography and acquisition lists in A4 format, ready to be attached to the aforementioned report.

The development of the spreadsheets and supporting website marked the final stage of the library's proposed process, both theoretically and practically. From that point on, we could move on to the implementation phase, during which we applied the developed solutions to undergraduate courses at the CCS. This stage enabled us to evaluate the functionality of the tools and support provided and to identify the concrete impacts on the daily routines of those involved, the difficulties encountered, and the lessons learned throughout implementation. The next section presents the results obtained and reflections arising from applying the process in a real-world context.

3.10. Implementation of the process

To validate the project's applicability, relevance, and tools, the team decided to present to all stakeholders consulted during the diagnostic and requirements-gathering phases. To facilitate a clearer understanding of the process, we opted for a demonstration using near-final versions of the spreadsheet and website while maintaining the initiative's dialogic and collaborative nature. Thus, eight spreadsheets were created, one for each CCS course. Each spreadsheet contains the bibliographies extracted from the Course Pedagogical Projects available on the respective institutional websites. The bibliographies are accompanied by a check of the number of existing copies and verification of availability in digital format.

3.10.1 Initial implementation by the library staff

To carry out the bibliography verification phase, staff and fellows affiliated with BIBLIOCCS at the time participated in an initial synchronous training session. This session focused on standardizing procedures and using the developed tools. The participants included four librarians, four interns, and three administrative technicians. They also received links to the spreadsheets and video tutorials produced to guide them on correctly filling out the fields and using the available resources. This enabled future reference and ensured greater autonomy in the process. Each team member's responsibilities were explicitly defined and assigned based on four operational stages that guided the team’s workflow.

  • a) Spreadsheet: transcribe data found in the course syllabi (PPCs) on the course websites into the spreadsheet fields;

  • b) Check the catalog: access the library system, Pergamum, to identify and record in the spreadsheet the number of physical copies available whose publication year is the same as or later than the year specified in the syllabus;

  • c) Check databases: access the library’s e-book platform to check whether each title is available in digital format; Review/resolve notes: the spreadsheet contains a column titled “Notes,” whose initial purpose was to allow faculty to record information in a more open-ended format that might be of interest to the NDE or the library during analysis (for example, digital books are not accepted in the anatomy lab, making it necessary to purchase physical books even if the library offers digital access). Since the work at this stage was not carried out exclusively by librarians or faculty members, the guideline was that any questions from scholarship recipients and administrative assistants should be recorded in this field for later resolution by the librarians.

During this phase, relevant quantitative data were collected and subsequently informed the project's final presentation. The bibliographies for the eight courses offered by the Center for Health Sciences (CHS) totaled approximately 4,800 titles distributed across 425 courses. Based on the minimum parameters established by the INEP/MEC evaluation instrument (three titles for the basic bibliography and five for the supplementary), the ideal number of references per course would be eight, when pedagogically feasible. The analysis revealed that 38 courses had 20 or more titles, while 79 courses had fewer than eight. Nevertheless, an average of approximately 12 titles per course was observed, indicating that courses generally maintain several titles slightly above the required minimum.

Based on the average number of titles identified, it was possible to estimate the time the BIBLIOCCS team spent carrying out the four operational steps described above. The survey indicated that transcribing the references into the spreadsheet took an average of eight minutes per course, while verifying the physical copies in the Pergamum system took approximately 18 minutes. Meanwhile, checking the availability of digital titles required about nine additional minutes. Thus, the total average time invested per course was 35 minutes.

Based on these data, it was assumed that a faculty member could record a course bibliography containing 12 titles in 35 minutes. Assuming this faculty member was responsible for four courses, it would theoretically be possible to verify an entire program's bibliographies in approximately two hours (assuming all faculty members fill out the spreadsheet simultaneously) and then resolve any discrepancies as needed.

In practice, however, this estimate proved impractical due to the specific characteristics of each course, the diversity of fields of study, and the varying availability of faculty members, whose schedules rarely align. Nevertheless, the adoption of the proposed tool was observed to significantly improve efficiency and productivity, especially compared to previous methods characterized by manual processes, scattered information, and a lack of integration among the agents involved.

3.10.2 Submission of the proposal to NDEs, program coordinators, and administrators

A PowerPoint presentation was developed based on the arguments and findings from the previous stages. The presentation was designed to consolidate participants’ understanding of how the spreadsheet and the website work and establish guidelines for effectively adopting these tools. The presentation outlined several key principles essential to the process's success.

First, the spreadsheet was proposed as a management and transparency tool to be used systematically, not just during periods leading up to MEC evaluations. Second, the implementation of the new model was emphasized as entailing a shift in institutional culture. This shift would entail initial missteps, which would be corrected collaboratively, yielding significant benefits in the medium term. Third, the spreadsheet was proposed as the sole official means of recording and communicating bibliographies, exclusively in its online version, thus eliminating the use of emails and parallel documents.

Furthermore, requests to acquire works were decided based on analyses recorded in the spreadsheet and on dialogue between the NDE and the library, to avoid individual requests that are disconnected from the PPCs. The spreadsheet would also serve as the primary data source for generating all institutional documents related to bibliographies, including compliance reports, course syllabi, and PPCs. The CCS administration would be responsible for formalizing the agreements and procedures arising from this new proposal.

In short, the presentation argued that adopting the new proposal would provide the library with an immediate overview of the bibliographies' status. This would improve decisionmaking regarding acquisition methods, increase the predictability of the MEC concept, and enhance collection management. It would also reduce rework and increase information transparency. Furthermore, it would facilitate identification of necessary acquisitions and cost estimates, save coordinators and department representatives time, and provide an immediate overview of pending issues and easy contact with those responsible for them.

The presentation was conducted in person with the leadership of SIB and CCS, the course coordinators, the NDE presidents, the department representatives at the library, and the Institutional Evaluation Office. These meetings were held at different times to accommodate the various stakeholders involved in the process.

Although minor adjustments were requested, there was no opposition to adopting the proposed process. Chart 2 summarizes what was agreed upon.

Thumbnail
Chart 2
Proposed allocation of responsibilities
3.10.3 Adoption of the proposal by the programs, challenges, opportunities, and next steps

Implementation of the bibliographic management proposal began with providing working tools, such as links to spreadsheets, for the CCS programs. Five of the eight programs initially participated by establishing ongoing communication with the library and submitting bibliographic mappings that were either finished or in the final stages of completion. The remaining three programs justified their decision not to prioritize the activity by citing the absence of imminent on-site evaluations or processes to revise their PPCs. They expressed their intention to resume contact at a later date.

A significant finding from the process was that most courses preferred to receive a blank template rather than rely on the library's preliminary review. This preference was based on the need to develop new PPCs, which would require substantial changes to the disciplinary composition and, consequently, the corresponding bibliography. Additionally, it was noted that programs adopted various strategies for completing the form, such as delegating the task to student scholarship recipients or outsourcing the service for a fee. Regardless of the method employed, the final stage of data validation is carried out collaboratively by the NDE and library staff.

Based on the pilot experience, a standard protocol was established for making the spreadsheets available. This protocol limits the number of rows per discipline to eight (three for the basic bibliography and five for the supplementary bibliography). It includes a lock that prevents the insertion of new rows. Any need to add titles requires prior contact between the faculty member and the NDE. This procedure aims to promote critical dialogue regarding the relevance of new acquisitions before they are formalized, thereby fostering a more rational and judicious process.

In addition to providing digital tutorials, the library participated in pedagogical meetings for certain courses, demonstrating the processes described in this project and the use of the catalog and digital book databases. These in-person meetings provided a valuable opportunity to promote other library resources and services, including in-person and remote training sessions and a WhatsApp channel for sharing tips and announcements. This initiative was viewed as a positive development that strengthened the relationship between the library and the faculty.

Reconfiguring the bibliography verification process to return primary responsibility for checking the availability of physical and digital copies to faculty members and eliminating the submission of separate acquisition lists resulted in significant efficiency gains. There was a reduction in the rework previously performed by the library when screening requests that included titles already in the collection.

Due to these positive results, the SIB-UFES administration decided to present the methodology to librarians across the system to enable replication at other centers. Since then, the Institutional Evaluation Office has generated 41 additional spreadsheets for courses with scheduled evaluations.

Identified challenges

Two main challenges were identified. The first concerns the high turnover rate among faculty members on the NDEs and departmental representatives at the library. During these transitions, a lack of continuity was observed in the transfer of information regarding bibliographic standardization procedures, even among those succeeding the NDE chair. To mitigate this risk, the library maintains a centralized document containing a history of interactions and links to spreadsheets. The second challenge, which had a lesser impact, was the occasional failure of some faculty members to complete the spreadsheet. However, the tool’s design includes visual indicators of omissions, enabling rapid identification and timely follow-up with those responsible.

Potential and Recognition

Contrary to the BIBLIOCCS team’s initial expectation of resistance due to the redefinition of roles - bibliographic verification had historically been viewed as the library’s responsibility, though not required by regulations - the initiative was widely endorsed by the academic programs. The library was repeatedly cited as a strategic partner in an acknowledged complex process. One program coordinator, who is also an MEC evaluator, suggested patenting the spreadsheet. The chair of one of the NDEs invited the library to develop a similar process for collecting data for the Sucupira Platform to evaluate the program's graduate program.

The main positive outcomes include increased visibility of the library as a proactive agent in institutional evaluation processes, an increase in its perceived value, strengthened ties with the NDEs and programs, an expansion of the solution’s scope of application beyond the CCS, the rationalization of acquisition demands, and the establishment of partial predictability regarding financial impacts and MEC evaluation indicators related to the bibliography.

Next Steps and Future Outlook

A unified, automated dashboard is currently being developed to consolidate bibliographic information from the eight CCS programs. The existence of a common core of courses across the Center's programs justifies this initiative, which aims to provide a broader overview. The dashboard will enable advanced analyses such as calculating the individual cost per course, identifying recurring titles without digital access, and determining the added value of database subscriptions. However, the tool's full effectiveness depends on all courses completing their individual spreadsheets simultaneously, which is challenging given the particularities of each course. Nevertheless, the possibility of instantly visualizing the center's bibliographic landscape and potential gaps is considered an extremely promising prospect for academic and collection management.

Additionally, the website content is being migrated to an institutional platform belonging to SIB-UFES since the project has been validated and expanded beyond BIBLIOCCS.

5 CONCLUSION

The objective of this study was fully achieved. The tool was widely adopted within the institution, providing greater efficiency and automating processes to a large extent. It also fostered closer ties between the library and the academic community involved in the evaluation process.

This experience is a concrete initiative toward participatory collection management, as recommended by Sousa (2018). Furthermore, the automation capabilities of the proposed process improve upon the checklist proposed by Moro (2022), offering greater efficiency and scalability.

The most recent version of the INEP/MEC evaluation tool favors a more qualitative and subjective approach to bibliographies than previous versions. While this perspective is valuable, quantitative parameters and objective information are indispensable for efficient collection management, particularly in contexts involving multiple actors and sometimes conflicting interests. In this scenario, the proposed solution has demonstrated significant support and gains in productivity and transparency.

A gap in the regulatory frameworks has been identified. Although the library is called upon to participate in evaluation processes, neither the librarian nor the information unit is explicitly mentioned in INEP/MEC’s “Evaluation Instrument for On-Campus and Distance Undergraduate Programs.” The library and bibliographies are often considered infrastructure issues, which does not reflect their strategic role in the complex university educational environment. Despite lacking a formal seat on the NDE, this study shows that libraries can play a proactive role by developing and proposing support tools and processes that directly inform the evaluation.

One of the persistent challenges identified in this experience is maintaining up-to-date spreadsheets as an ongoing practice, rather than only updating them in the run-up to evaluations or PPC revisions. For the tool to transcend its role as merely an aid for reports and establish itself as a dynamic instrument for collection management, all bibliographic changes in the teaching plans must be recorded immediately. This practice has not yet been fully institutionalized.

Despite formal agreements, the natural turnover of teaching staff requires the library to reinforce communication to consolidate relationships continuously. Full adherence by all courses remains a challenge. However, it is worth noting that courses that choose not to adhere to the agreed-upon process exempt the library from responsibility for any bibliographic inadequacies since a formal solution was made available and approved by all parties. Conversely, adherence to the process enables the clear identification of communication failures and directs the search for solutions in collaboration with the responsible parties.

It should be emphasized that the operational context of BIBLIOCCS, in which the library predominantly facilitates and supports coursework, may not be replicable across all institutions. Nevertheless, for libraries that still perform full inventory checks, the tools described here are useful for identifying issues, suggesting adjustments to faculty members, and preparing assessments for administrators.

It should be noted that the solution presented in this report can be applied to diverse contexts. With relatively little effort, the project can be adapted to manage school library collections or meet the specific demands of graduate programs, for example. However, the most significant aspect transcends the tool itself: establishing dialogue with the served community, fostering mutually beneficial relationships, and offering agile, assertive support. A process based on an integrated vision of the academic community is essential to the initiative’s success and grants the library the strategic leadership role it deserves in these matters.

  • 1
  • 2
  • Article submitted to the similarity checker
  • Funding:
    Not applicable.
  • Ethical approval:
    Not applicable.
  • AI Statement:
    The authors mention in the methodology the use of AI tools (ChatGPT and DeepSeek) for the sole purpose of reviewing parts of the text for conciseness, clarity, and spelling and grammatical correctness.
  • Image:
    Retrieved from the Lattes Platform
  • Translation:
    S. Iacovacci Translation Service
  • JITA:
    DD. Academic libraries
  • SDG:
    4. Quality education

Acknowledgments:

Not applicable.

Availability of data and materials:

https://zenodo.org/records/17360156

References

Edited by

Publication Dates

  • Publication in this collection
    07 Aug 2026
  • Date of issue
    2026

History

  • Received
    22 Sept 2025
  • Accepted
    03 Oct 2025
  • Published
    30 Mar 2026
Creative Common - by 4.0
Este é um artigo publicado em acesso aberto sob uma licença Creative Commons.
location_on
Universidade Estadual de Campinas Rua Sérgio Buarque de Holanda, 421 - 1º andar Biblioteca Central César Lattes - Cidade Universitária Zeferino Vaz - CEP: 13083-859 , Tel: +55 19 3521-6729 - Campinas - SP - Brazil
E-mail: rdbci@unicamp.br
rss_feed Stay informed of issues for this journal through your RSS reader
Report error