Computerized protocol of orofacial myofunctional evaluation with scores: usability and validity

Abstracts

PURPOSE:  To test the usability of Computerized Orofacial Myofunctional Evaluation (OMES) protocol and analyze its validity.

METHODS:  The study was divided into three stages: the first stage, production of the computerized version of OMES. The second stage was the validation of the user's interface, in which 100 OMES protocols of a database, filled in printed version, were transferred using the computerized instrument. Necessary changes to the system have occurred at this stage. In the third stage, usability of the OMES protocol in multimedia version, three evaluators transferred data from other 25 printed protocols from database for the computerized version, and the time to transfer the data of each protocol was computed and compared between examiners by one-way ANOVA. Moreover, these evaluators analyzed the usability of computerized protocol according to the "Ten principles of Heuristics usability" as described in the literature.

RESULTS:  The computerized protocol satisfied the principles of heuristics usability, according to the evaluation of the three Speech-Language Pathology evaluators, and the average time spent by the evaluators to transpose the data of each protocol to the software ranged from 3.1±0.75 to 3.83±0.91 minutes.

CONCLUSION:  The Computerized AMIOFE protocol is valid and had its usability/functionality confirmed.

Evaluation; Software Validation; Diagnosis, Computer-Assisted; User-Computer Interface; Speech, Language and Hearing Sciences; Stomatognathic System

OBJETIVO:  Testar a usabilidade do protocolo de Avaliação Miofuncional Orofacial com Escores (AMIOFE) Informatizado e analisar a validade do mesmo.

MÉTODOS:  Estudo dividido em três etapas: a primeira, produção da versão informatizada do AMIOFE. A segunda etapa consistiu na validação da interface do usuário, na qual 100 protocolos AMIOFE de um banco de dados, preenchidos em versão impressa, foram transferidos empregando o instrumento informatizado. Alterações necessárias no sistema ocorreram nessa etapa. Na terceira etapa, usabilidade da versão multimídia do protocolo AMIOFE, três avaliadoras transferiram os dados de outros 25 protocolos do banco de dados para a versão informatizada, sendo que o tempo para a transferência dos dados de cada protocolo foi computado e comparado entre os examinadores pelo teste ANOVA one-way. Além disso, essas avaliadoras analisaram a usabilidade do protocolo informatizado de acordo com os "Dez princípios de usabilidade Heurística", como descritos na literatura.

RESULTADOS:  O protocolo informatizado satisfez aos princípios de usabilidade heurística, de acordo com a avaliação das três avaliadoras fonoaudiólogas, e o tempo médio despendido pelas avaliadoras para a transposição dos dados de cada protocolo para o software variou de 3,1±0,75 a 3,83±0,91 minutos.

CONCLUSÃO:  O protocolo AMIOFE Informatizado é válido e teve sua usabilidade/funcionalidade confirmada.

Avaliação; Validação de Programas de Computador; Diagnóstico por Computador; Interface Usuário-Computador; Fonoaudiologia; Sistema Estomatognático

INTRODUCTION

Technological advancement and qualification of professionals enabled the construction of electronic protocols. Various health services have implemented them, or are in the implementation phase, for clinical application and scientific research. Therefore, Speech-Language Pathology and Audiology must keep up with this moment of transition and participate in it.

Electronic protocols provide better access to information, greater security and electronic exchange of data between institutions, as well as facilitate collective research, with the possibility of retrieval and cross-checking of this information^{( 1 )}.

Previously, its use was limited due to the cost of the equipment, its maintenance, and the lack of skilled labor or the possible resistance of people to computers. However, it is possible to create these protocols today, increasing the rate of accuracy of records, with low cost, reduced physical space, and minimal training of personnel^{( 2 - 6 )}.

These tools can facilitate administrative and financial organization of consultations; staff time in handling procedures; retrieval of patient information, knowledge, and availability of this knowledge where and when it is necessary for adequate decision-making; and, in some cases, the generation of diagnosis and therapeutic guidance^{( 1 , 7 , 8 )}.

On the basis of this, we developed a computerized version of the orofacial myofunctional evaluation with scores (OMES)^{( 9 )} protocol to optimize the records for clinical use and research.

Briefly, the OMES protocol was designed to provide sufficient data for detection and grading of orofacial myofunctional disorders, without being too extensive and comprehensive. Previously, it has been validated for children^{( 9 )}, youth, and adults, with good sensitivity and specificity^{( 10 )}.

For a software to be considered valid and for its usage to be proper, it must go through a stage known as usability (functionality) inspection, which is a way of evaluating user interfaces^{( 11 )}.

Usability is defined in ISO 9241-11 as: "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (p. 3)^{( 12 )}.

Therefore, it concerns man-machine interaction. To be easily accepted, the instrument should be user-friendly, easy to use^{( 13 )}, and its validity is related to the perceived satisfaction and usefulness by users^{( 14 )}.

This study aimed to determine the usability of the OMES computerized protocol and analyze its validity.

METHODS

The project was approved by the research ethics committee of Hospital das Clínicas of Faculdade de Medicina de Ribeirão Preto, Universidade de São Paulo (USP-HCFMRP) according to the HCRP protocol no. 15602/2-12. Participants (evaluators) were informed about the objectives and methods of the study and were asked to sign a free and informed consent.

Production of the computerized version of orofacial myofunctional evaluation with scores

In the OMES computerized protocol, the characteristics of the original version were maintained, and consequently its psychometric properties^{( 9 )}. Software was developed by an undergraduate student of the Biomedical Informatics course under the guidance of a teacher in the area. For its creation, the Java programming language, executable in the Windows operating system, was used in the following order:

1. entry in the system and selection of existing or new protocol;

2. identification data of the patient;

3. evaluation data on appearance and posture;

4. mobility assessment data;

5. data on functions;

6. data from the functional evaluation of the occlusion;

7. placeholder for final comments.

Validation of the user interface

In this pretest stage of the instrument, 100 OMES protocols, completed in hard copy and taken from the prior database of the research team, were used. All protocols were transferred to the computerized version by an undergraduate student.

Twenty-five printed protocols were randomly selected from the total and transferred to the computerized OMES by another team member. In order not to create duplicates, the records were entered into the system with a code before their identification.

Subsequently, each user listed the possible changes in the software related to its operation and/or errors detected. This information was cross-checked and discussed, and then presented, so that the necessary changes were made in the area of Biomedical Informatics, before the next step.

Usability of the multimedia version of the orofacial myofunctional evaluation with scores protocol

The corrected version of the OMES computerized protocol was tested for its validity as follows:

A. Three Speech-Language therapists (mean age: 25±0.8 years), with prior training in the area of orofacial motricity and with different levels of training (from 30 to 66 months, average: 46 ±18.3 months) for the use of the OMES protocol (printed version), participated as evaluators of usability. They scanned the data independently and did not exchange information.

Data from other 25 printed protocols, different from the ones in the previous step, were transferred from the database to the computerized version. In order not to create duplicates in the system, each evaluator entered a different code into the system to identify each protocol.

B. Time for the data transfer of each protocol was computed.

C. The three evaluators also independently analyzed the usability of the system in accordance with the "Ten Usability Heuristics" proposed by Nielsen^{( 11 )}. For each of the items described, each evaluator responded to one of the alternatives: does not satisfy (score 1), partially satisfies (score 2), and satisfies (score 3). The instrument to evaluate usability, containing the heuristics and their descriptions, is presented in Chart 1.

Chart 1
Usability Heuristics Evaluation conducted by evaluators with regard to the protocol Orofacial Myofunctional Evaluation with computerized scores

Data analysis

Descriptive statistics were performed for the variables involved. The examiners were compared in terms of time spent for the transfer of information by one-way analysis of variance test.

RESULTS

Validation of the user interface

During the pretest of the OMES computerized protocol, problems were found and changes needed were proposed. In general, the main errors found were related to the standardization of markers in the protocol; buttons that were not performing their functions correctly, or even that were missing; absence of items from the printed protocol; and overlapping of some data when opening an already filled protocol.

Before starting the usability test step of multimedia version of the OMES protocol, adjustments were made and problems already listed were solved. Thus, the computerized protocol, as previously mentioned, followed the pattern of the printed protocol, as shown in Figures 1 and 2.

Figure 1
Example screenshot of the orofacial myofunctional evaluation with scores computerized protocol regarding the assessment of mobility

Figure 2
Example screenshot of the orofacial myofunctional evaluation with scores computerized protocol regarding the functional evaluation of the occlusion

Usability of the multimedia version of the orofacial myofunctional evaluation with scores protocol

The computerized protocol complied with the usability heuristics, according to the evaluation of three Speech-Language therapists, with scores ranging from 28 to 29, in a total of 30 points. The principle evaluated with the lowest score was^{( 5 )} "error prevention". The data and amounts are shown in Table 1.

Table 1
Evaluation of usability heuristics of the orofacial myofunctional evaluation with scores computerized protocol, according to the principles of Nielsen(11)

Principle	Scores attributed per evaluator			Total
	Evaluator 1	Evaluator 2	Evaluator 3
1	3	3	3	9
2	3	3	3	9
3	2	3	3	8
4	3	3	3	9
5	2	3	2	7
6	3	2	3	8
7	3	3	3	9
8	3	3	3	9
9	3	2	3	8
10	3	3	3	9
Total per evaluator	28	28	29	85

The average time spent by the evaluators for the transposition of data of each protocol to the software ranged from 3.1 to 3.83 minutes. The time spent by evaluator 2 was significantly higher than that by evaluator 3 (p<0.01). Statistical comparisons are presented in Table 2 and Graph 1.

Table 2
Total time spent per evaluator for the transfer of 25 orofacial myofunctional evaluation with scores protocols from hard copies to the computerized version

	Evaluator 1	Evaluator 2	Evaluator 3	p-Value
Total	83.1a	95.93ª.b	77.77ª.c	0.01
Mean	3.32	3.83	3.10
SD	0.73	0.91	0.75

• Means and standard deviations, in minutes; Means with different letters indicate differences in the Tukey post test
• ^Caption
  : p = probability in the ANOVA test; SD = standard deviation

Graph 1
Average total time spent per evaluator for the typing of the protocols, in minutes, with respective standard deviations

DISCUSSION

In this study, heuristic usability of the OMES computerized protocol was determined with excellent results in the ratings of the three users.

A usability problem can be defined as any characteristic, observed in a given situation, which may delay, hinder, or prevent the completion of a task, annoying, embarrassing, or traumatizing the user^{( 15 )}.

In the case of the computerized OMES protocol, only the item regarding the possibility of "error prevention" was rated by two evaluators as partially satisfying. In fact, the program does not inform the evaluator if, for example, they forgot to enter any data from the evaluation. Therefore, there is no lock that can prevent continuity. But, however, if a given piece of information cannot be obtained, this does will not prevent the continuity of the evaluation and registration.

The average time spent for the transposition of the data was brief, no more than 3.83±0.91 minutes, and the difference between two of the evaluators, although statistically significant, did not exceed 1 minute. This time does not concern the evaluation of patients with simultaneous input of data in the electronic protocol, but, as explained, only the transposition of the printed protocols to the program.

The objective regarding the computerized OMES protocol was to make it functional. Following the principles proposed by Nielsen^{( 11 )}, the information appears in a natural and logical order, with a user-friendly language, as already outlined in the original protocol, facilitating its management.

Electronic protocols present many conveniences to the user and ensure improved information management and quality of research^{( 1 )}. In clinical terms, the computerized version of the protocol in question will add convenience, speed, and ease of visualization of results: with just one command ("click"), you can enter the result of the evaluated item. For each category of the protocol, such as appearance, posture, mobility of the stomatognathic system components, and functions (breathing, chewing, and speaking), the software presents the sum as soon and the evaluation is completed. When the assessment is complete, the total score is informed and corresponds to the orofacial myofuncional condition of the individual evaluated.

From this, the professional can define the need for orofacial myofuncional therapy for a given patient, comparing the numerical results of their assessment to the normal parameters previously described^{( 16 , 17 )}.

It is noteworthy that the use of the computerized OMES protocol does not eliminate the need for knowledge in the area of orofacial motricity and the need for training in evaluation.

A careful orofacial myofuncional evaluation, especially when the instrument had been tested for validity and has good levels of sensitivity and specificity, favors the correct diagnosis and proper decision on therapy^{( 18 )}.

The usability of the OMES computerized protocol for the evaluation of patients is feasible and a digital database is generated with all the information. Therefore, no more data entry is necessary after the evaluation, which will reduce the time to organize these, as well as improve information quality and accuracy of records^{( 4 )}. The data relating to patients and the results can be retrieved quickly, clearly, without generating doubts^{( 3 )}, and with reduced costs^{( 5 )}.

The need for computerization in various areas, including in health, seems increasingly indispensable, because its advances have opened many possibilities for the use of information technology in clinical and scientific research^{( 4 )}. Scientific research has especially grown, both qualitatively and quantitatively^{( 3 )}.

According to our knowledge, the OMES computerized protocol is the first instrument of orofacial myofunctional evaluation with an electronic version, with proven construct and criteria validity^{( 9 , 10 )}, as well as usability heuristics, developed in the area of orofacial motricity, in a digitalized version. On the basis of our experience, we believe that it has potential to foster advances in clinical practice and in scientific research in the area.

CONCLUSION

The OMES computerized protocol had its usability/functionality confirmed and proved useful for the storage and retrieval of orofacial myofuncional evaluation data.

ACKNOWLEDGMENT

We thank the Speech-Language Pathologists who participated as evaluators in this work for the collaboration.

REFERENCES

• ¹ Cano DVB, Malafaia O, Alves VLS, Avanzi O, Pinto JSP. Protocolo eletrônico de fisioterapia respiratória em pacientes com escoliose idiopática do adolescente. Rev Col Bras Cir. 2011;38(3):177-80.
• ² McAfee A. Mastering the three worlds of information technology. Harv Bus Rev. 2006;84(11):141-9.
• ³ Garcia PN, Goldchmit M, Souza-Dias CR. Projeto piloto de informatização dos prontuários do setor de motilidade ocular extrínseca da Santa Casa de São Paulo. Arq Bras Oftalmol. 2008;71(4):504-8.
• ⁴ Zago-Filho LA, Malafaia O, Moreira ATR, Matias JEF. Base eletrônica de dados clínicos e cirúrgicos das doenças da retina e vítreo. Arq Bras Oftalmol. 2008;71(3):357-64.
• ⁵ Cavalcanti RF, Studart LM, Kosminsky M, Goes PSA. Validation of the multimedia version of the RDC/TMD axis II questionnaire in Portuguese. J Appl Oral Sci. 2010;18(3):231-6.
• ⁶ Fernandes GL, Santos RE, Malafaia O, Aoki T. Criação de protocolo eletrônico em câncer do colo do útero. Rev Col Bras Cir. 2012;39(1):28-32.
• ⁷ Wechsler R, Anção MS, Campos CJR, Sigulem D. A informática no consultório médico. J Pediatr (Rio J). 2003;79(supl. 1):S3-S12.
• ⁸ Nykänen P, Brender J, Talmon J, Keizer N, Rigby M, Beuscart-Zephir MC, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011;80(12):815-27.
• ⁹ Felício CM, Ferreira CLP. Protocol of orofacial myofunctional evaluation with scores. Int J Pediatr Otorhinolaryngol. 2008;7(3):367-75.
• ¹⁰ de Felício CM, Medeiros AP, de Oliveira Melchior M. Validity of the 'protocol of oro-facial myofunctional evaluation with scores' for young and adult subjects. J Oral Rehabil. 2012;39(10):744-53.
• ¹¹ Nielsen J. Heuristic evaluation. In: Nielsen J, Mack RL, editors. Usability inspection methods. New York: John Wiley & Sons; 1994.
• ¹² Associação Brasileira de Normas Técnicas. NBR ISO 9241-11/1998 - Requisitos Ergonômicos para Trabalho de Escritórios com Computadores: Orientações sobre Usabilidade. Rio de Janeiro: ABNT; 2000.
• ¹³ Carvalho AAA. Testes de usabilidade: exigência supérflua ou necessidade? In: Congresso da Sociedade Portuguesa de Ciência da Educação; 2002. Lisboa. Atas da Sociedade Portuguesa de Ciências da Educação; Lisboa; 2002. p. 235-42.
• ¹⁴ Sun J, Teng JTC. Information Systems Use: construct conceptualization and scale development. Comput Human Behav. 2012;28:1564-74.
• ¹⁵ Dias C. Usabilidade na Web: criando portais mais acessíveis. Rio de Janeiro: Alta Books; 2003.
• ¹⁶ de Felício CM, Sidequersky FV, Tartaglia GL, Sforza C. Electromyographic standardized indices in healthy Brazilian young adults and data reproducibility. J Oral Rehabil. 2009;36(8):577-83.
• ¹⁷ Giglio LD. Biomecânica orofacial e a eficiência mastigatória em adultos jovens. [dissertação]. Ribeirão Preto: Universidade de São Paulo; 2013.
• ¹⁸ Felício CM, Folha GA, Ferreira CLP, Medeiros APM. Expanded protocol of orofacial myofunctional evaluation with scores: validity and reliability. Int J Pediatr Otorhinolaryngol. 2010;74(11):1230-9.

• Financial support: Pro-rector of Research from Universidade de São Paulo.
• Study carried out at the Center for Research Support in Morphophysiology, Craniofacial Complex, Research Laboratory of the Stomatognathic System, School of Medicine of Ribeirão Preto, Universidade de São Paulo - USP - Ribeirão Preto (SP), Brasil.
• *
  CMF carried out the conception and study design, statistical analysis, interpretation of results, revision of the article and approval of the version to be published; GAF carried out the study design, data collection, statistical analysis, drafting of the article, interpretation of results and approval of the version to be published; ASG performed the collection and tabulation of data, drafting of the article and interpretation of results; MMMD carried out the construction of the computerized instrument, revision of the article and approval of the version to be published; PMAM carried out the construction of the computerized instrument, review of the article and approval of the version to be published.

Publication Dates

• Publication in this collection
  July-Aug 2014

History

• Received
  06 Feb 2014
• Accepted
  03 July 2014