Method for direct evaluation of automatic indexing through assessment by indexers

ABSTRACT

Objective:

In order to evaluate an automatic indexing system applied to e-books, this paper proposes and applies a method for direct evaluation of automatic indexing through assessment by indexers of the quality of the index terms automatically assigned to the documents.

Method:

A methodological research of descriptive and applied nature was carried out, using technical procedures of bibliographic research and empirical research. Initially, by means of a literature review, the steps of the proposed method of direct evaluation of automatic indexing through assessment by indexers were delimited, to then proceed with the construction of an instrument for data collection and application of the proposed method in the evaluation of automatic indexing of the SISTRA system in the indexing of technical-scientific e-books.

Result:

The proposed evaluation method is described by means of a diagram and systematic description. The method consists first in the judgment by indexers of the quality of terms assigned by the automatic indexing system to a sample of digital documents, and then in the analysis of values calculated for quality metrics of automatic indexing. The application of the proposed method proved useful in a first evaluation of an automatic indexing system.

Conclusions:

We conclude that the proposed method of direct evaluation of automatic indexing through assessment by indexers provides the standardization of evaluation and its practice by information professionals, and that direct evaluation is a necessary activity for the application and adoption of automatic indexing in indexing by subject of digital documents in the scope of information units.

KEYWORDS:
Automatic indexing; Automatic indexing system; Evaluation of automatic indexing; Direct evaluation of automatic indexing; Methodological pathway