Assessment of the usability of a nutritional epidemiology computerized system

Abstracts

INTRODUCTION:

Epidemiological research has been adhering to new technologies, such as computer systems, and using the Internet as a tool. Usability is a characteristic of a specific product concerning the facility to use it, its speed and the facility to learn how to use it. It should also not present errors, or these must be easy to solve, in case they occur, thus providing high satisfaction to users.

OBJECTIVE:

To evaluate the usability of the "System of health and nutrition monitoring - nutrition of school children" (NUTRISIM).

METHODS:

A sample of 17 Information Technology professionals evaluated the system and answered the "Questionnaire for System Usability", which determines the level of usability of systems by the Fuzzy Logic. The questionnaire contains 30 questions, which are divided into six metrics. The usability of the system determines six usability criteria in a large Fuzzy scale.

RESULTS:

With the exception of the metric "error control", all metrics were analyzed as "very good". The metrics "error control", "efficiency" and "satisfaction" presented medium amplitude, which is a better result in relation to the metrics "easy to learn", "easy to remember" and "effectiveness", which was assessed as "high".

CONCLUSION:

The study showed that the system is easy to be learned and used, but the answers are scattered. The instrument proved to be a useful tool to monitor and evaluate health and dietary intake in epidemiologic studies.

Epidemiology; Nutritional epidemiology software; Food consumption; Computer Systems; Fuzzy Logic

INTRODUÇÃO:

As pesquisas epidemiológicas vêm aderindo a novas tecnologias, tais como sistemas computadorizados, e utilizando a Internet como ferramenta. A usabilidade é uma característica de um determinado produto relativo à facilidade de uso, à rapidez para aprender a usá-lo, ao fato de não gerar erros ou à facilidade de resolução dos mesmos, caso ocorram, oferecendo um alto grau de satisfação para seus usuários.

OBJETIVO:

Avaliar a usabilidade do "Sistema de monitoramento da saúde e alimentação - nutrição do escolar" (NUTRISIM).

METÓDOS:

A avaliação do sistema foi realizada pela Lógica Fuzzy, através do "Questionário para Usabilidade de Sistemas", composto por 30 questões que se dividiam em seis métricas, respondido por 17 profissionais de Tecnologia da Informação. A avaliação da usabilidade do sistema consiste em determinar, para cada um dos seis critérios de usabilidade, um conceito na escala Fuzzy.

RESULTADOS:

Todas as métricas obtiveram avaliação "muito boa" com exceção da relacionada com o "controle de erros". As amplitudes referentes às métricas "controle de erros", "eficiência" e "satisfação" foram consideradas com designação "média", melhor resultado do que para as métricas "facilidade de aprender", "facilidade de relembrar" e "eficácia", que apresentaram classificação "alta".

CONCLUSÃO:

Pode-se perceber que o sistema é fácil de aprender e de usar, porém apresenta dispersão entre as respostas. O instrumento mostrou ser uma ferramenta útil para a avaliação e monitoramento da saúde e do consumo alimentar em pesquisas epidemiológicas de pequeno a grande porte.

Epidemiologia; Epidemiologia nutricional; Software; Consumo de alimentos; Sistemas de Computação; Lógica Fuzzy

Introduction

Computerized systems and the digital mean are more and more used in the field of research related to the health of the population. Documents, data and images can be transferred simultaneously in a digital circuit, therefore enabling the development of applications for primary health promotion and to follow-up the diet of individuals^{1 , 2}.

At the same time, the internet has provided epidemiology with alternatives for data collection. New supporters of epidemiological research have been adhering to these new technologies². However, it is clear that, at this stage, it is important to investigate the benefits and the traps involving the use of the internet and new instruments developed to collect epidemiological data.

Throughout the years, numberless computer programs have been created, and literature shows it is necessary to evaluate the relationship user-system. "System" is understood as any software presenting an interaction between man and machine, by means of an interface, therefore not being restricted to the programs that are frequently used in microcomputers, but also related to operational systems, text editors and electronic spreadsheets³.

In order to assess the quality of the systems, usability tests have been created in order to describe characteristics of a specific product, that is, if such a product si easy to use, fast and quick to learn, if it does not cause errors (and, in case there are errors, if they are easily solved), and if there is a high level of satisfaction among the users. Also, usability determines the success of a system. Poorly designed interfaces, especially for system working with information, can be responsible for the disinterest or discredit of the user, leading to damage and losses, among other factors^{4 , 5}.

Marchionini⁶ and Nielsen⁷ demonstrate in their studies that two types of evaluators should assess the usability of a system: experts in man-machine interaction and subjects coming from the same population of future users. According to Rubin⁸, an expert that assesses the system can easily detect inconsistencies in the system, confusing interfaces, among others. The user, on the other hand, can verbalize and register the difficulties in relation to the task to be performed, so that hindrances in the system can be identified, including performance and preferences of the user.

Therefore, usability studies are used to describe or classify, in detail and gradually, responses referring to the ability of the users in relation to the use of an operational system. Just like in human communication, the responses of the users are imprecise and vague verbal expressions, therefore, it is necessary to translate them into numerical values. The fuzzy logic has been used in order to better interpret verbal language.

With this approach, it is possible to solve the paradoxes created from the classification of "true or false". In the fuzzy logic, a premise ranges in true levels, from 0 to 1, which can be partially true or partially false³. Also, the linguistically expressed values of "true/false" are interpreted as a finite fraction subset, from zero to one⁹. In this sense, the fuzzy logic expresses the principle of duality by establishing that two opposite events can coexist, that is, to a certain degree one element can belong to a set and, to another degree, it belongs to another set, therefore being represented by a fraction value inside a numerical interval. The application of this concept can be observed in daily activities, especially when it comes to abstract concepts, for instance, defining if a portion of food is large or medium³.

Facing the exposed, this study aims at assessing the usability of a computerized system developed by researchers in the Public Health School of Universidade de São Paulo, which evaluates and monitors the health and the dietary intake of Brazilian children and adolescents.

Methods

This study assessed the usability of the software "System of health and nutrition monitoring - nutrition of school children" (NUTRISIM)¹⁰, a computerized system used for data collection, assessment and monitoring of the health and nutritional status of students. The system gathers different thematic blocks, such as dietary intake, which is assessed by means of a structured 24-hour recall method, a self-evaluation form and a photographic atlas. NUTRISIM is available in the website: http://www.fsp.usp.br/nutrisim.

Professionals from the information technology (IT) field, such as webdesigners, system analysts, interface designers, among others, were invited to assess the system. At first, there were 20 professionals, but after the exclusion process, 17 participants remained, and such number is in accordance with the recommendations by Nielsen¹¹. The access to the system was conducted through a link sent to professionals by electronic mail, and each of them received an individual login (user name) and password, which had been previously defined. The professionals were monitored during access by the developers of the system in order to make sure the process was concluded.

After fulfilling all of the stages proposed for the test, the professionals fulfilled the "Questionnaire for System Usability", developed by Santos³.

The first part of the questionnaire corresponds to questions that define the profile of the user (gender, age, schooling, time of experience of using a computer), and the second part is composed of 30 questions that assess the usability of the studied system, which are divided into six metrics: "easy to learn", "easy to remember", "error control", "efficiency", "efficacy" and "satisfaction". Therefore, each metrics has several questions, and each question has five response options: "very unsatisfied, unsatisfied, indifferent, satisfied and very satisfied.

The usability assessment consists of determining a concept in the fuzzy scale for each one of the six criteria: "very bad", "bad", "medium", "good", and "very good", associated to support with values of 0, 2, 4, 6 and 8, respectively (Figures 1 and 2). In this approach, it is necessary to find the Fuzzy Number (FN) and the Normal Triangular Fuzzy Set that are closer to the modal frequency response of the users. After assessing the Fuzzy Triangular Numbers (FTN), the Fuzzy Mean Number (FMN) is calculated for each question of the metrics, which is given by a specific formula. After finding the FMN, the number should be normalized by dividing all of the pertinence µ values by the maximum pertinence of this same fuzzy number, therefore finding the normalized mean fuzzy number (NMFN). The NMFN is compared to the FTNs of the same mode, leading to the final FTN, which determines the classification of usability for the metrics as a whole, with the use of the same support and classification³ (Figures 3 and 4).

Figure 1
Chart of the questions 1, 5, 6, 9 e 10 of the metric "ease to learn".

Figure 2
Chart of the questions 2, 3 e 7 of the metric "ease to learn".

Figure 3
Chart of final result of the metric "ease to learn".

Figure 4
Chart of finals results of the metrics "error control", ","efficiency", ","effectiveness", ","ease of recall" and "satisfaction".

The lower the amplitude of the interval, the higher the data reliability. Amplitude should be interpreted with the adoption of 1, 2, 3 and 4 for Minimum, Medium, High and Very High, respectively. When the amplitude found in the final result of the metrics is equal to or higher than two, it is recommended that the system be reassessed under the point of view of this metrics, and that the number of invited experts be increased. The amplitude of this final FTN will determine the dispersion of opinions of the users and the quality of the final result.

For the characterization of the studied groups, a descriptive analysis of personal information was conducted by means of frequency, and by calculating the mean and the standard-deviation. The results of the usability test were graphically generated by the MatLab software.

The procedures to develop the research were in accordance with the guidelines of resolution n. 196, from October 10, 1996, of the National Health Council, considering the ethical principle of respect for the autonomy of people, as well as the ethical obligation to increase as much as possible the benefits and reduce damage to the least. This research project was forwarded to the Ethics Committee of the Public Health School of Universidade de São Paulo, protocol n. 014/11.

Results

The mean age of the expert professional corresponds to 33 ± 8.7 years, being 70.5% male participants, graduated in the field of Information Technology, and mean time of experience of 16 ± 8.5 years.

In this study, the results of the metrics "Easy to learn" will be emphasized, and the information of the other metrics will be presented in summary by means of the aforementioned "final result" graphs.

The metrics "easy to learn" has ten questions, being five of them about the FTN 8/2 (Figure 1). Mode eight represented the "very good" response, with amplitude 2, which indicated a dispersion that can be classified as "medium".

Figure 2 represents the graph resulting of three questions that obtained FTN 6/2, which represents the "good" response, with amplitude 2, indicating a "medium" dispersion in the opinion of the users.

Question four presented FTN 8/1, with amplitude 1, and such opinion is classified as "very good", with "minimum" dispersion, which can lead to the interpretation that the opinions of the users were more agreeing. Question eight was categorized as "good", translated by FTN 6/1 with amplitude 1, "minimum" dispersion among the responses.

The final result for the metrics "easy to learn" presented FTN 8/3 as response, which, as proposed by the methodology, represents the convergence of all of the results for the ten questions composing this same metrics. Fuzzy number 8/3 represents a "very good" evaluation for the metrics "easy to learn" of the analyzed system, however, with amplitude 3, which means "high" dispersion among the opinions of the users (Figure 3).

The final graphs for the other metrics are represented in Figure 4. The metrics "error control" presented FTN 6/2 as response, which represents "good' opinion, with "medium" dispersion.

The metrics "efficiency" and "easy to remember" presented FTNs 8/2 and 8/3 responses, representing a "very good" evaluation with "medium" and "high" dispersion, respectively, among the opinion of the users.

Similarly to the metrics "easy to remember", "efficacy" presented FTN 8/3, which reflects a "very good" evaluation, however, with "high" dispersion.

The final result for the metrics "satisfaction" obtained FTN 8/2, representing a "very good" assessment with "medium" dispersion.

Table 1 synthetizes the results, and the study pointed out to "good" and "very good" evaluations. The system is easy to learn and satisfies the user. However, there are amplitudes that indicate a low confidence level. This dispersion could be corrected by inviting more experts; if dispersion persisted, these metrics should be reconsidered.

Table 1
Metrics Results.

Metrics	Evaluation	Amplitude
Easy to learn	8 – Very Good	3 – High
Error Control	6 – Good	2 – Medium
Efficiency	8 – Very Good	2 – Medium
Easy to Remember	8 – Very Good	3 – High
Efficacy	8 – Very Good	3 – High
Satisfaction	8 – Very Good	2 – Medium

Discussion

Currently, the Information and Communication Technology (ICT) offers great potential to develop instruments of data survey for studies related to the health of the population. The use of ICT provides elements that allow interactive relations, that is, relationships with equal components inside an ideal environment for interactivity and interaction¹². Recent studies state that the future of biomedical research will involve the participation of many collaborators at different locations, all of which will be interconnected through high-speed computer networks, therefore, able to present, analyze and share data simultaneously. So, this study showed that the NUTRISIM system is easy to understand and use by means of the usability analysis.

The dispersion of the metrics "easy to learn" can be related to the difficulty or long time to conclude a task. The dispersion of the metrics "efficacy" can be attributed to the number of produced errors and also to the development of the system in relation to the speed of the operation. Together, both metrics can be related to time of execution, and it is believed that when users are being assessed, they may not conclude tasks with the expected productivity.

Another metrics that presented high dispersion was "easy to remember". Memorization is related to training and/or repetition, and the system was presented only once to the users, which makes the possibility of remembering more difficult. With this hypothesis, it is important that the user know the system, screens and commands prior to the evaluation.

One limitation that should be mentioned is that there are still few studies presenting the usability of systems addressed to the assessment of dietary intake (or even to the health field), especially directed to population groups in the analyzed age group. The observed studies are generally international and are more focused on the description of the development and functionalities of the system than on the detailing of the method used to assess its usability^{13 - 21}.

Among the found studies, it is possible to mention the one by Foster16, which developed and tested a 24-hour computerized recall method (SCRAN24), fulfilled by the interviewee and addressed to the adolescent population aged between 11 and 16 years old. Like the structured 24-hour recall method, which composes one of the modules of NUTRISIM, the mentioned software contained divisions according to main meals and snacks, spaces to fulfill the time of the meals and photographs of the portions of food to assist quantification. One difference between systems concerns the list of food; in SCRAN24, the interviewee had free typing space to search for the food, while in NUTRISIM foods were organized in a pre-list, although there was also a free typing research field for less common foods that were not inserted in the pre-list. According to the authors, SCRAN24 was nearly as accurate and precise as similar systems, used in dietary intake studies, and it was well received by the user population, with relatively fast fulfilling.

Another tool addressed to individuals aged between 8 and 11 years old is the Web-based Dietary Assessment Software for Children (WebDASC), developed with the objective of measuring dietary changes resulting from interventions in the school environment. The system was developed to be fulfilled by children being assisted by their parents. Like the recall method of NUTRISIM, the referred system brings the food items organized in a list, or they can be accessed by a free typing field, however, based on 1,300 food items, which is more extensive than the data base used in NUTRISIM. There are also images of food portions in four different sizes¹⁴. The mentioned authors describe that, in preliminary tests, the system was well accepted by the interviewees in the referred age group.

The study by Riiser¹⁹ tested the usability of an internet program addressed to the stimulation of physical activity among adolescents with excessive weight, the Young & Active. As in NUTRISIM, by means of the results of the usability tests, adjustments were performed to make the content and the objectives more visible and explicit. In this study, a second usability test was conducted and assessed the program with the adjustments of the first test. It revealed that the program was well accepted by the participants, and only minor aesthetic adjustments had to be done in order to complete the final version of the program.

Besides the limitation concerning the external comparability due to the lack of similar studies, it is important to mention the fact that the characteristics of the studied group in this analysis may have interfered with the results. The positive results that were observed can be related to the fact that the users are adults, with high schooling, and especially used to handling computerized tools, thus presenting with more facility to operate the system.

It is worth to mention that the presented paper represents the first step concerning the tests with the referred system, once there has to be a technical evaluation addressed to the observation of fulfilling flows, existence and quantification of errors during execution, among other technical aspects; therefore, in continuity, it can be tested by the target-audience. Tests involving the use of the system by school students and by the researchers themselves (in relation to the environment addressed to them in the software) are in the phase of planning and short term execution.

Conclusion

The usability study with IT professional had good results, however, some metrics related to the time of use and training should be reassessed in a larger sample in order to obtain more accurate results.

Final considerations

The instrument proved to be a useful tool to assess and monitor health and the dietary intake in small and large epidemiological studies.

References

• ¹
  Wang DH, Kogashiwa M, Kira S. Development of a new instrument for evaluating individuals' dietary intakes. J Am Diet Assoc 2006; 106(10): 1588-93.
• ²
  Smith B, Smith TC, Gray CG, Ryan MAK, Millennium Cohort Study Team. When epidemiology meets the Internet: Web- based surveys in the Millennium Cohort Study. Am J Epidemiol 2007; 166(11): 1345-54.
• ³
  Santos RC. Desenvolvimento de uma metodologia para avaliação de usabilidade de sistemas utilizando a Lógica Fuzzy baseado na ISO. 2007. 115 p [dissertação de mestrado]. Rio de Janeiro: Faculdade de Economia e Finanças IBMEC; 2007.
• ⁴
  Nielsen J. Usability Engineering. Boston: Academic Press; 1993.
• ⁵
  Shneiderman B. Designing the User Interface: Strategies for Effective Human- Computer Interaction. EUA: Addison Wesley; 1998.
• ⁶
  Marchionini G. Evaluating Hypermedia-Based Learning. In: Jonassen DH, Mandl H. Designing Hypermedia for Learning. Berlin: Springer-Verlag; 1990. p. 355-73.
• ⁷
  Nielsen J. Multimedia and Hypertext: the Internet and beyond. Boston: AP. Professional; 1995.
• ⁸
  Rubin J. Handbook of Usability Testing. New York: John Wiley and Sons; 1994.
• ⁹
  Gomide FAC, Gudwin RR. Modelagem, controle, sistemas e lógica fuzzy. SBA Controle & Automação 1994; 4(3): 97-115.
• ¹⁰
  NutriSim - Sistema de Monitoramento da Saúde-Nutrição e alimentação do escolar. [software na internet]. versão 1. São Paulo (SP): Departamento de Nutrição da Faculdade de Saúde Pública - USP; 2011. Disponível em http://www.fsp.usp.br/nutrisim.
  » Disponível em http://www.fsp.usp.br/nutrisim
• ¹¹
  Nielsen J. Why you only need to test with 5 users. Jakob Nielsen's Alertbox, 2000. Disponível em http://www.useit.com/alertbox/20000319.html (Acessado em 20 de janeiro de 2011).
  » Disponível em http://www.useit.com/alertbox/20000319.html
• ¹²
  Skinner HA, Maley O, Norman CD. Developing internet-based eHealth promotion programs: the Spiral Technology Action Research (STAR) model. Health Promot Pract 2006; 7(4): 406-17.
• ¹³
  Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc 2009; 16(5): 660-9.
• ¹⁴
  Biltoft-Jensen A, Trolle E, Christensen T, Islam N, Andersen LF, Egenfeldt-Nielsen S, Tetens I. WebDASC: a web-based dietary assessment software for 8-11-year-old Danish children. J Hum Nutr Diet 2012; [Epub ahead of print].
• ¹⁵
  Thompson DA, Joshi A, Hernandez RG, Jennings JM, Arora M, Ellen JM. Interactive nutrition education via a touchscreen: is this technology well received by low-income Spanish-speaking parents? Technol Health Care 2012; 20(3): 195-203.
• ¹⁶
  Foster E, Hawkins A, Delve J, Adamson AJ. Reducing the cost of dietary assessment: Self-Completed Recall and Analysis of Nutrition for use with children (SCRAN24). J Hum Nutr Diet 2013; [Epub ahead of print].
• ¹⁷
  Lang AR, Martin JL, Sharples S, Crowe JA. The effect of design on the usability and real world effectiveness of medical devices: a case study with adolescent users. Appl Ergon 2013; 44(5): 799-810.
• ¹⁸
  Lee MK, Park HA, Yun YH, Chang YJ. Development and formative evaluation of a web-based self-management exercise and diet intervention program with tailored motivation and action planning for cancer survivors. JMIR Res Protoc 2013; 2(1): e11.
• ¹⁹
  Riiser KP, Løndal K, Ommundsen Y, Sundar T, Helseth S. Development and usability testing of an internet intervention to increase physical activity in overweight adolescents. JMIR Res Protoc 2013; 2(1): e7.
• ²⁰
  Vandelanotte C, Caperchione CM, Ellison M, George ES, Maeder A, Kolt GS, et al. What kinds of website and mobile phone-delivered physical activity and nutrition interventions do middle-aged men want? J Health Commun 2013; 18(9): 1070-83.
• ²¹
  Wyatt TH, Li X, Huang Y, Farmer R, Reed D, Burkhart PV. Developing an interactive story for children with asthma. Nurs Clin North Am 2013; 48(2): 271-85.

Publication Dates

• Publication in this collection
  Dec 2013

History

• Received
  24 May 2012
• Reviewed
  21 May 2013
• Accepted
  02 Aug 2013