Configuration of evaluation methods for mobile digital maps

Abstract:

Map evaluation is a fundamental and challenging aspect of cartography. This is especially true for digital maps that can be accessed via mobile devices (hereafter referred to as «mobile digital maps»). Researchers often use a combination of methods adapted from computer science to evaluate these maps in order to obtain comprehensive and reliable information. However, the evaluation of digital maps is a complex task that requires a multidimensional approach to understand the different perspectives and needs of users in different usage contexts. By analyzing scientific articles, this study aims to explore and identify the most commonly used methods for evaluating mobile digital maps. The goal is to provide an analytical review of the methods, test configurations, and application locations of mobile digital map evaluations. For this research, 195 scientific articles published in international journals between 2010 and 2023 were evaluated. Of these, 53 articles were selected, all of which dealt with the evaluation of digital maps accessed by mobile devices. The study answers the question of which methods are recommended depending on the context. The types of data collected cover both qualitative and quantitative aspects, while the evaluations tend to be moderate in nature.

Keywords:
Methodologies; Tests; Users; Smartphones; Cartography

1. Introduction

Digital maps are designed for specific usage contexts and user groups with different characteristics, each with different goals and tasks (Dick, Gonçalves, and Vitorino, 2017; Ricker and Roth, 2018; Krassanakis and Cybulski, 2019). However, the user experience of digital maps accessed through mobile devices-such as smartphones-can be affected by a variety of environmental, technical, physical, and social conditions (Vertzberger and Klein, 2021; Wang et al., 2022). According to Roth et al. (2017), the introduction of new hardware and software technologies can lead to new requirements for users of digital maps related to: symbology, learning, fatigue, privacy, adaptation of map evaluation methods, and, above all, the need for ecological validation of the collected data, which consists of collecting data obtained in real-world conditions with the aim of reflecting the user’s experience in practical situations when using mobile digital maps (Robinson et al., 2023).

The context of use is a variable that should be considered when evaluating digital maps, as it plays an important role in identifying specific problems and proposing solutions to improve the user experience (Bartling et al., 2022; Dutta et al., 2022). The environment in which the user is immersed, such as when using maps for navigation in cities, can have variations in lighting, noise, obstructions to the line of sight, vehicle and pedestrian traffic, divided attention, variations in Internet connection speed, and other factors, especially when it comes to digital maps accessed through smartphones (He and Chan, 2015). All of these elements can have a significant impact on the usability and readability of the map, and should therefore be considered as separate dimensions in the evaluation process. In this article, the term “mobile digital map” is used to refer to maps accessed via mobile devices, which mainly includes the category of devices classified as smartphones, but can also refer to tablets, smartwatches, and smart glasses.

Ren et al. (2019) state that it is essential to understand how users actually use mobile digital maps. This includes aspects such as the physical and social environment in which they are used, users’ tasks, goals, and preferences, and the devices used to access the maps (Kapaj et al., 2021). Understanding these aspects is key to designing mobile digital maps that effectively meet users’ needs and efficiently accomplish the desired tasks. In the context of mobile digital maps, evaluation is very important to understand how users interact with these tools and to identify user needs and interface shortcomings (Savino et al., 2019). Evaluation not only enables the identification of specific challenges faced by users, but also provides data on which solutions can be based and adaptations can be made in response to the dynamic demands of users (Basiri et al., 2017).

There are several criteria for evaluating maps, one of the most popular being usability (Maramba et al., 2019). User testing is one of the main ways to evaluate aspects of map use, including usability, and provides valuable information about how users actually use maps and what usage problems occur during interaction (Sugimoto et al., 2022; Xu et al., 2022). Researchers involved in the evaluation of mobile digital maps are often faced with a series of questions, such as: “What methods should be used to evaluate maps accessed via smartphones?” or “What changes when these methods are applied in different contexts of map use?

The need for a ranking of methods used in mobile digital map assessments lies in the ability to provide a systematic and comparative framework that allows stakeholders to have a comprehensive view of the prevailing approaches (Zein et al., 2016). The lack of such a compilation of methods with their respective classifications can obscure best practices and hinder the appropriate selection of methods for the evaluation of mobile digital maps (Robinson et al., 2023). The contribution of this article is to identify, organize, and categorize evaluation methodologies and configurations to provide a basis for supporting mobile digital map evaluations.

Currently, there is no clear consensus on which method or combination of methods is most effective for collecting data when evaluating maps accessed by smartphones in the field (Roth et al., 2017). For this reason, this study adopts the empirical hypothesis that by analyzing publications reporting studies on mobile digital maps, it is possible to determine which methods are most commonly used in this type of evaluation. Therefore, the aim of this article is to identify, in a selection of publications, the evaluation methods used, the test configurations applied, the criteria answered, and the types of data collected in evaluations of mobile digital maps.

1.1 Map task and evaluation methods

The relationship between the map task and the evaluation method is very important in the context of mobile digital map evaluation (Horbinski et al., 2019). Different types of tasks require different evaluation approaches in order to obtain relevant results. For instance, map reading tasks pertaining to the identification of points of interest may be enhanced by the utilisation of qualitative methodologies, such as interviews or the think-aloud protocol. These techniques facilitate the expression of participants’ perceptions and navigation strategies, thereby enriching the data collected. (Rehrl et al., 2012; Fogli et al., 2020; Swobodzinski et al., 2021; Nair et al., 2022). On the other hand, tasks involving the collection of quantitative data, such as navigation time efficiency or destination identification error rate, can be evaluated using questionnaires, which can be combined with the data logger to obtain additional quantitative information.

The environment in which the map reading task is performed also influences the choice of assessment method (Kratz et al., 2010; Liang et al., 2012; Einfeldt and Degbelo, 2021; Schirmer et al., 2015). Tasks in environments such as laboratories can benefit from more structured and controlled testing (Chu et al., 2017; Yang et al., 2019; Matsuo et al., 2020), while tasks in uncontrolled environments such as streets or parks may require more flexible and adaptive approaches, such as the use of verbal protocols during navigation (Delikostidis and Van Elzakker, 2011; Maly et al., 2013; Delikostidis et al., 2016; Lu et al., 2021; Vincent et al., 2022).

In addition to considering the type of map reading task and the assessment method, it is essential to consider the critical characteristics of spatial activities in relation to the assessment method. In this regard, it is advisable to consider the utilization of evaluation methods and techniques that facilitate the documentation of strategies employed in the utilization of maps and spatial interactions. For example, the use of technologies such as audio and video recordings during navigation can provide information about participants’ perceptions and behaviors in relation to the spatial environment (Swobodzinski et al., 2021; Vincent et al., 2022). Therefore, when evaluating mobile digital maps in relation to these types of tasks, it is essential to consider both the nature of the task and the critical characteristics of the spatial environment, and to select the most appropriate evaluation method in order to obtain accurate and relevant results.

The process of map reading and cartographic Interaction is structured around a series of geographic inquiries that assist users in navigating while resolving spatial tasks (Van Elzakker, 2004). In his 1984 study, Board presented a list of these questions, which involve location, spatial association, spatial interaction, and spatial change in analog maps. These questions can be applied to the digital context as well. These questions can be associated with map reading purposes, including identification, comparison, ordering, association, and delimitation. To illustrate, in elementary tasks:

• What constitutes the subject matter under examination? The differentiation and recognition of objects, as well as external identification, are key processes in this area.

• What entities are present in a specific location? The processes of object identification and internal identification.

• In a given location, what is the extent of the phenomenon? It is necessary to estimate quantities.

• The objective is to determine the location of the given geographic object. Identify the location of an object.

2. Methodology for identifying and analyzing methods

A selection of publications was carried out, and the evaluation methods, test configurations, criteria, and types of data collected in evaluations of mobile digital maps were evaluated. A bibliometric/scientometric analysis was also conducted. In the context of scientometrics, qualitative analysis entailed an examination of various aspects of scientific production. A content analysis of scientific articles and an analysis of the number of citations in relation to map evaluation methods were conducted anew.

The initial phase of implementing the methodology utilized in this study entailed the identification of pertinent articles within the domain of cartography that had been published in international journals. The searches were carried out in the Scopus and Web of Science databases. The criterion for the selection of articles was the need to deal with evaluations involving users of mobile digital maps. Publications relevant to the research context were analyzed with a focus on the methodology used to conduct studies with map users, allowing a comparison between the types of tests, their configurations and the criteria addressed. The search was limited to scientific articles published in journals between 2010 and 2023. Beyond this period, only papers written in English were selected. A combination of words and Boolean operators was used: TS=(usability AND maps AND navigation AND mobile). TS means “Search Topic,” and it can be found in the “Title,” “Summary,” and “Keywords” fields.

Specific keywords were used to optimize the search: “usability”, “maps”, “navigation” and “mobile”. These keywords were chosen with the aim of limiting the topic to some relevant dimensions for the analysis of the evaluation of mobile digital maps. The term “mobile” refers to the technology used, distinguishing maps accessed on mobile devices such as smartphones from maps accessed on computers such as desktops and laptops. The term “navigation” refers to the task of using maps and is one of the main functionalities of this type of application. The word “maps” represents the specific product being evaluated (Savino et al., 2021). Finally, “usability” is the dimension related to user testing, which is one of the main approaches used to evaluate the usability of digital maps.

To standardize the verification of “relevance to the topic under study” in each manuscript and minimize bias, a systematic process was implemented. The articles retrieved from the database were initially evaluated based on their titles and abstracts to ascertain whether they pertained to the assessment of digital maps accessed via mobile devices. Those that did not meet this criterion were subsequently excluded. Articles deemed potentially relevant underwent a more comprehensive evaluation, entailing the reading of their introductions and methodological procedures. During this process, general information was extracted regarding the evaluation methods employed, and objective data was collected concerning the configurations of the evaluation procedures applied, in accordance with predefined parameters, and recorded:

• The methods used to collect data.

• The moderation of the experiments, according to the physical presence of the evaluators.

• And the location where the evaluations were conducted.

• The number of citations is contingent upon the evaluation methodology employed.

2.1 Organizing and analyzing results

The methods used to evaluate mobile digital maps were organized into four groups, following the structure proposed by Rohrer (2014) and used and improved by Martins et al. (2023), as shown in the figure below (Figure 1). This organization was adapted from Rohrer’s (2014) proposal to facilitate the organization and analysis of the methods, which were represented by letters (A, B, C and D) according to their similar characteristics. Figure 1 shows the axes “participant responses vs. participant actions” (behavioral axis) and “qualitative vs. quantitative” (axis of classification of the data collected), which make it possible to distinguish the study methods according to the questions answered and the purposes for which they are best suited.

Methods aimed at evaluating verbal expressions allow a better understanding of the beliefs expressed by users (evaluation of responses), while methods aimed at behavior allow an understanding of the practices that users carry out in relation to the product or service in question (evaluation of actions). Qualitative studies generate data based on direct observations of test participants and may be more appropriate for answering questions about why or how a particular problem is solved, while quantitative methods respond to quantifiable measures and collect data by counting the number of observations.

Figure 1:
General structure for organizing evaluation methods.

Figure 1 shows that qualitative methods tend to be more appropriate for answering questions about how to solve problems, while quantitative methods tend to be derived from mathematical analysis. The «Participant Responses vs. Participant Actions» axis makes it possible to understand the relationship between what map users say or do. After organizing the data according to the criteria established in Figure 1, the results were tabulated and subjected to the analysis established in this study.

The organization of the information collected in the articles was based on the methods used to collect data in the map evaluations, which were associated with the corresponding groups shown in Figure 1. This organization took into account: i) the type of data collected, which varied between qualitative and quantitative data; ii) the way in which the evaluation methods were applied, distinguishing between moderated evaluations - in which the moderator was physically present during the evaluation - and unmoderated evaluations; iii) the environments in which the evaluations were conducted, classifying them as controlled, uncontrolled, or combined - corresponding to data collected in both laboratory and non-laboratory environments; and iv) the location in which the tests were conducted, classifying them as indoor, outdoor, or both (indoor/outdoor).

First, an exploratory analysis of the results was performed, using the frequency of the most common words in the set of nominal qualitative data collected in the articles. After this stage, Multiple Correspondence Analysis (MCA) was applied, which is an extension of Principal Component Analysis (PCA), but applied to the analysis of categorical data, with the aim of ordering the qualitative observations and analyzing the percentage contribution of each variable present in the MCA. Finally, the frequency of combinations of the most and least used methods of data collection in mobile digital map evaluation was analyzed. All the results obtained through the analysis were discussed and explored in order to understand the implications and contributions of the different methods.

3. Analysis of methods for the evaluation of mobile digital maps

This article consulted 115 articles in the Scopus database and 80 articles in the Web of Science database, for a total of 195 articles found from the query created within the established time frame. Of these, only 53 articles were strictly related to the topic of «evaluation of digital maps accessed by mobile devices» and were therefore selected for the study. Of the 53 articles selected, 40 used up to two combined methods for data collection in user evaluations, while the other 13 used more than two combined methods.

In general, the evaluations mentioned in the 53 articles aimed to assess the usability and effectiveness of maps used to perform spatial tasks. The articles analyzed made it possible to identify existing gaps in the understanding of the evaluation methods used, especially the clear lack of consensus on best practices. This approach provides a context for understanding the results and establishes the necessary basis for understanding the purpose and contribution of the study in identifying and categorizing the methodologies used to evaluate mobile digital maps.

Table 1 shows the results of the analyzed articles. Its columns show the names of the authors, followed by brief thematic summaries. Next, details of the methods used for data collection are presented, along with their correspondence to the groups organized in Figure 1: Group (A) includes the Think Aloud Protocol and Interview methods; Group (B) refers to the User Observation and Cognitive Map methods; Group (C) adopts the Data Logger, A/B Test, and Eye Tracking methods; and Group (D) includes the Questionnaire and Heuristic Evaluation methods. Table 1 provides an exploratory view of the configuration of mobile digital map evaluation methods. It also includes a column indicating whether the experiment was conducted with or without moderation, meaning whether a moderator was present while the participants performed the tasks, and whether the experiment location was conducted indoors, outdoors, or a combination of both. The Table 1 presents a column indicating the number of citations received by each publication at the time the article was written. The citations were verified through an online search on Google Scholar Citations.

Table 1:
Selected articles, data collection methodology and evaluation context.

Article(s)	Article topics	Method(s)	N° Citations
(Lu et al., 2021); (Noguera et al., 2012); (Brade et al., 2017); (Sakpere et al., 2017)	Evaluation of the usability and effectiveness of mobile digital maps with moderated experiments, mainly conducted in indoor environments.	Questionnaire (D)	567
(Dirin et al., 2018)	Evaluation of emotional needs in a mobile application for tourists. The experiments were moderated and the evaluation location was indoor.	Interview (A)	21
(Yang et al., 2019); (Link et al., 2013)	Evaluation of the usability and accuracy of indoor positioning with real-world experiments, including both unmoderated and moderated evaluations. The locations were indoor and outdoor.	Data logger (C)	33
(Darvishy et al., 2020)	Evaluation of a digital map designed for visually impaired people. The experiments were moderated and the evaluations took place indoors.	User observation (B)	6
(Maly et al., 2013); (Montuwy et al., 2019)	Evaluation of maps for visually impaired and elderly people, taking into account the ability to remember routes and stress during navigation. Experiments were moderated and evaluations were conducted in indoor and outdoor environments.	Interview and data logger (A-C)	36
(Brock et al., 2015)	Usability evaluation between classic tactile maps and digital maps for visually impaired people. The evaluations were moderated and conducted in an indoor environment.	User interview and user observation (A-B)	222
(Liang et al., 2012); (Kratz et al., 2010); (Einfeldt and Degbelo, 2021); (Schirmer et al., 2015)	Usability and the use of the map during navigation were evaluated, with predominantly moderate evaluations and carried out in indoor and outdoor environments.	Questionnaire and data logger (D-C)	139
(Rehman and Cao, 2016); (Rehman and Cao, 2015)	Comparison of performance, workload and perceived usability of mobile devices for internal navigation. In general, the evaluations were moderate and carried out in indoor environments.	Questionnaire and cognitive map (D-B)	172
(Vincent et al., 2022); (Brata and Liang, 2019); (Brata et al., 2017); (Drewlow et al., 2022)	Evaluation of the usability and effectiveness of navigation technologies such as 2D maps and augmented reality. The evaluations were conducted in a moderate manner and covered both outdoor and indoor environments.	Questionnaire and user observation (D-B)	60
(Abreu and Moraes, 2012); (Delikostidis and Van Elzakker, 2011); (Bernelind, 2015); (Nadzir et al., 2019); (Zahabi and Kaber, 2018); (Schnitzler and Holscher, 2015)	Usability evaluation and identification of problem solutions in mobile navigation applications for different target groups. Evaluations were conducted in a moderate manner, covering both indoor and outdoor environments.	Questionnaire and Think Aloud (D-A)	52
(Fogli et al., 2020); (Chu et al., 2017); (Swobodzinski et al., 2021); (Goh et al., 2016); (Rehrl et al., 2012); (Nair et al., 2022); (Sanjaya et al., 2020); (Noh et al., 2015); (Matsuo et al., 2020); (Sharin et al., 2020); (Brata and Liang, 2020); (Swobodzinski et al., 2021); (Brata et al., 2021)	Data was collected on the usability and perceived usefulness of different maps. Participants with different profiles and needs were included, including people with disabilities. All evaluations were moderated and conducted in a variety of environments, both indoor and outdoor.	Questionnaire and interview (D-A)	214
(Ramsay et al., 2010); (Skulimowski et al., 2019); (Rehrl et al., 2012); (Qiu et al., 2023)	Evaluation of the usability of mobile maps, including the use of sound, depth images, and augmented reality maps in orientation and spatial knowledge tasks. All evaluations were conducted in a moderated manner and took place in both indoor and outdoor environments.	Questionnaire, interview and data logger (D-A-C)	53
(Grubert et al., 2015); (Kim and Song, 2014)	Data were collected to evaluate users during navigation, taking into account touch gestures in map operation under visual occlusion. The experiments were conducted in a moderate manner and took place in a variety of environments, both indoor and outdoor.	Questionnaire, interview and user observation (D-A- B)	110
(Delikostidis et al., 2016)	The usability and navigation of pedestrians was evaluated, with experiments conducted in a moderate and outdoor environment.	Questionnaire, interview, Think Aloud and map cognitive (D-A-B)	46
(Wenig et al., 2016)	Navigation, time measurement, navigation errors, usability and workload were evaluated, with experiments conducted in a moderate indoor environment.	Questionnaire, interview, Think Aloud and user observation (D-A-B)	16
(VanElzakker and Delikostidis, 2010)	Evaluation of the use of landmarks for orientation and navigation in an outdoor environment. The experiments were carried out moderately.	Questionnaire, interview, Think Aloud, cognitive map and user observation (D-A-B)	5
(Rehrl et al., 2014); (Diao and Shih, 2018); (Aditya et al., 2018)	Usability was evaluated during pedestrian navigation in different environments, with experiments conducted in a moderate manner, both outdoors and indoors.	Questionnaire, user observation and Data logger (D-B-C)	91
(Dong et al., 2021)	Participants’ navigation, visual attention, and route memory were assessed in experiments conducted in a moderate outdoor environment.	Questionnaire, interview, eye tracking and cognitive mapping (D-A-B-C)	63

• Source: The authors.

As shown in Table 1, the most commonly used data collection methods in the selected studies, which were generally used together to collect qualitative and quantitative data, were: Questionnaire; Data Logger; Interview; or Think Aloud Protocol. Although many authors do not explicitly mention user observation in their methodology, the studies that present moderate evaluations tend to adopt this practice indirectly. Most of the studies were conducted in indoor and outdoor environments. The studies have different focuses of analysis, such as navigation in different environments, accessibility of digital maps for people with disabilities, effectiveness and efficiency of classic maps compared to interactive maps, use of mobile devices to access digital maps, and especially usability evaluation of mobile digital maps.

Of the total of 53 selected articles, 8 used only one evaluation method, while the majority (45) employed two or more combined methods. These combined methods provide a comprehensive approach to data collection, allowing for a more complete view of the user experience, and thus the collection of larger amounts of quantitative data, such as time, device data, navigation errors, and others. However, using many methods simultaneously can increase the complexity of the study, require more resources to run the experiments, and generate redundant data, so the relationship between the resources used and the efficiency of the desired return must be considered.

Most of the experiments have been conducted sparingly and in outdoor locations in an attempt to simulate real navigation situations. The positive points of conducting experiments in outdoor contexts are that they can simulate real navigation situations, allowing for a more accurate assessment of user performance and the effectiveness of navigation technologies. In addition, these experiments can provide a broader view of how users use navigation technologies in different contexts. However, conducting experiments outdoors can have drawbacks, such as the lack of control over environmental variables that can affect test results, such as outdoor noise and weather changes. On the other hand, conducting tests in controlled environments can limit the generalizability of results to real-world situations, as well as the ability to account for environmental and contextual variables that may affect user navigation.

The articles with the highest number of citations employed group D evaluation methods, with a total of 567 citations. The combination of methods from the DA group occupies the second position, with a total of 266 citations. The AB group methods occupy the third position with 222 citations. These data indicate that the primary methodological approaches utilized in the evaluated studies are directly correlated with the number of citations, which serves as an additional indicator of the quality and relevance of these methods in the context of evaluating mobile digital maps. The preponderance of citations associated with Group D may be indicative of a greater degree of acceptance of these methods within the academic community. Moreover, the combination of methods, as observed in the DA group, also demonstrates a greater focus from the scientific community, which can enrich the analysis and provide more comprehensive results. The third position of the AB group, despite being less cited than the first two, still signifies a notable degree of methodological importance, contributing substantially to the existing body of knowledge. The correlation between the number of citations and the methods employed underscores the necessity of judiciously selecting and implementing suitable methodologies in scientific research.

3.1 Context and methods for applying mobile digital map assessments

The way in which moderated or unmoderated tests are used is also analyzed, as well as the type of data collected, classified as qualitative or quantitative. Figure 2 shows the results of an exploratory analysis of the main keywords in the total data set collected in the articles, which allows: i) identify the most used methods to evaluate mobile digital maps; ii) identify the most common location where tests were conducted (indoor, outdoor, or both); iii) identify the types of data collected (qualitative or quantitative); and iv) observe how the tests were applied (moderated and unmoderated). The graph in Figure 2 highlights the most common words in the data set.

Figure 2:
Most frequent words in the set of nominal qualitative data collected.

The exploratory analysis of the nominal qualitative results collected in the articles showed that moderate evaluations were the most frequently cited, appearing in 50 of the observed results. The “questionnaire” method, included in group (D), was mentioned in 47 results, indicating a wide use of this technique for data collection. Data collection with both qualitative and quantitative aspects appeared in 41 of the results analyzed. The “interview” data collection method, group (A), was mentioned 33 times, showing its relevance in carrying out evaluations. The “indoor” environment was mentioned 28 times.

The “Think Aloud Protocol” (A) is mentioned 25 times, confirming the importance of these methods in the evaluation of mobile digital maps. The “Outdoor” environment appears 19 times, indicating a more limited use of this approach. The application of the “user observation” method, group (B), appears 17 times, followed by the appearance of the “data logger” method, group (C), which is mentioned 15 times, and evaluations made only with the collection of quantitative data, which appear 10 times. After carrying out an exploratory analysis of the data collected in the articles, it is possible to obtain a more precise understanding of the main trends, contexts and methods for evaluating mobile digital maps through the frequency of the most recurring words. In addition, the results were subjected to Multiple Correspondence Analysis (MCA), which made it possible to assess the contribution of each variable to the variation in the results. The contribution of each variable to the total variation in the data was then expressed as a percentage. The results of this analysis are presented in Figure 3.

Figure 3:
MCA - Assessment methods and context of application.

The MCA analysis shows the proportion of the variation in the data that each observation represents. A high contribution means that the variable accounts for a significant amount of the total variation in the data set, while a low contribution means that this variable has less impact on the variation in the data. An MCA chart is a visual representation of the relationships among the categorical variables being analyzed. It displays the categories of variables as points in the two-dimensional space defined by the first two axes of the analysis. The position of the points on the graph indicates the relationship between the categories. Observations that are close together on the graph tend to have similar profiles and are more likely to occur together. Categories that are far apart, on the other hand, have different profiles and are less likely to occur together. The «Dim1» and «Dim2» axes represent the two principal components resulting from the MCA. These dimensions are linear combinations of the original variables and help to simplify the information in complex data.

The results shown in Figure 3 help to identify the variables that are most correlated with each dimension. The correlations between the variables and the dimensions are used as coordinates and can be interpreted on the basis that nearby points are more relevant to their respective components. Interpretation of the Dim1 and Dim2 axes can be aided by analyzing the weights of the variables, or their contributions to the total variation, which can be used to explore trends and groupings in the data. In Figure 3, «Dim1» explains 14% of the variability in the data, making it the most relevant, while «Dim2» explains 12.9% of the variability in the data.

According to Figure 3, indoor testing (red ellipse) tends to be moderate, as in laboratories, which ensures that the environmental conditions are as controlled and constant as possible during the experiment. Indoor assessment methods are very diverse and are usually used in combination. In the indoor and outdoor tests, the methods were categorized according to Figure 1 (Group A: Think Aloud Protocol and Interview; Group B: User Observation and Cognitive Map; Group C: Data Logger, Eye Tracking and A/B Test; and Group D: Questionnaire and Heuristic Evaluation). The most commonly used methods are quite broad and include groups D-B, A-B, D-A, D-B-C, D-A-B, A-C, D-A-C, D-A-B-C, D-A-B-C, D, and D-C.

Tests in combined indoor/outdoor environments (green ellipses) also tend to be moderate. In these conditions, the environment is not controlled, which means that the environmental conditions may change during the course of the experiment. The most commonly used methods for collecting data in these conditions include A-C, D-A-B, and D-A-C combinations. The testing methods used in outdoor environments (blue ellipses) are similar to those used in mixed environments (green ellipses), which also tend to be moderate. The most commonly used data collection methods in these tests include A-C, D-A-C, D-A-B, D-B-C, and D-A combinations.

It is important to note that indoor experiments tend to be broader in terms of the methods used for evaluation. In addition, it is common for experiments to be moderated and to be conducted using combined methods that allow for the simultaneous collection of qualitative and quantitative data. The results not included in the ellipses correspond to evaluations carried out without moderation, which tend to collect only quantitative data using the methods in groups C and D. On the other hand, studies aimed at collecting qualitative data generally use the isolated methods in groups A or B. Figure 4 shows the contribution values of the columns resulting from the MCA analysis, presented in order with the variables and their respective contributions to the variation in the data.

Figure 4:
Contributions of the variables in the analysis of data variation.

As can be seen in Figure 4, the variables with the largest contribution to the variation of the data correspond to the group “C” (46.30%), “Indoor/Outdoor” (44.38%) and experiments in the category “Qualitative” (42.73%). These variables are fundamental to explain the differences observed in the results of experiments with mobile digital maps and are less frequent in the configurations of these experiments. On the other hand, the other variables such as “D-B”, “Qualitative and Quantitative”, “D”, “D-B-C” and “D-A-B” have smaller but still relevant contributions, ranging from 8.87% to 2.77%. This indicates that these variables also play an important role in explaining the data patterns, but occur less frequently compared to the main variables. Finally, the “A-C” and “Moderate” variables had the smallest contributions to data variation, with values of 2.33% and 0.82%, respectively.

3.2 Spatial Distribution of Methods

Figure 5 illustrates the exploratory analysis of the data collected in the articles, relating the spatial location of each article’s lead researcher to the combination of methods used to evaluate the mobile digital maps. The MCA analysis highlights the proportion of variation in the data represented by each observation, indicating that nearby observations on the graph tend to have similar profiles and are more likely to occur together. Figure 5 illustrates that “Dim1” accounts for 10.7% of the variability in the data, while “Dim2” accounts for 9.8%.

Figure 5:
A comparison of the spatial distribution of methods in relation to continents.

The results presented in Figure 5 demonstrate a geographic distribution of the groups of methods evaluated. The evaluations carried out on the American continent, with emphasis on the United States, followed by Canada and Brazil, are shown in the figure. The most prevalent method group on the American continent is D-B, which is indicated by a salmon-colored ellipse. The methods employed on this continent are analogous to those utilized on the Asian continent, as represented by a green ellipse. It is notable that countries such as Taiwan, China, Indonesia, Malaysia, Korea, Japan, and Singapore stand out as locations where the assessments were conducted. This geographic diversity reflects the growing prominence of the Asian region as a research center, corresponding to a wide range of groups of methods employed, including D-A, D-B, D-B-C, and their combinations. Both continents are represented in close proximity to the origin of the graph, with slightly flattened ellipses, which suggests a relatively low variation in relation to the analyzed data set, particularly in comparison to the European continent.

The European continent is distinguished by the prevalence of a particular approach, as illustrated by the blue ellipse in Figure 5. The results indicate a broad distribution of groups of methods across Europe, encompassing countries such as Italy, the United Kingdom, the Czech Republic, Poland, Austria, France, the Netherlands, Sweden, Switzerland, Spain, Germany, and Finland. This geographic diversity serves to illustrate the relevance and scope of assessments conducted across the European region. Among the various groups of methods identified, the frequency of use of the D-B-C, C, D-A-C, and D-C methods and their combinations is particularly noteworthy. Conversely, method groups D, A, B, A-B, A-C, and D-A-B-C tend to exhibit greater variability and lower usage rates in comparison to other countries on other continents. Finally, Oceania, represented mainly by New Zealand, exhibits a proclivity to adopt the D-A-B combination of methods, although the analysis was limited to articles from New Zealand.

3.3 Application of tests and combination of assessment methods

Figure 6 shows the frequency of combinations of the most and least used methods for data collection in mobile digital map evaluation. Each letter represents the groups of methods organized in Figure 1. This analysis highlights the main methodological approaches used in the evaluated studies.

Figure 6:
Frequency of methods used to evaluate mobile digital maps.

The results presented in Figure 6 show that the combination of methods most often used to evaluate mobile digital maps was the one presented in Group D-A, corresponding to the questionnaire and interview or questionnaire and think aloud protocol methods. These combinations allow evaluators to obtain a more complete and detailed view of the user experience of mobile digital maps (CHO and CASTANEDA, 2019; BARTLING et al., 2021). The quantitative information obtained through questionnaires is complemented by qualitative information obtained through interviews and think aloud. Other frequently used methods, according to the data analyzed, are the combination of questionnaire, interview, think aloud, data logger, user observation and cognitive map, associated with the groups D-A-B, D-B and D-C.

The experiments conducted to evaluate mobile digital maps accessed by smartphones are generally moderate. In addition, the test site can vary between indoor and outdoor, depending on the type of map being evaluated and the purpose of the study. The choice of test application context and the type of methodology can have a significant impact on the results obtained, so it is important to consider these factors when designing and conducting these evaluations. However, there is a subtle bias toward conducting evaluations in uncontrolled environments and indoors. This change in the location of the experiment can influence the methodology used to collect data.

3.4 Trends, challenges, and future directions

The analyses presented in the article reveal fundamental trends and challenges in the evaluation of mobile digital maps, emphasizing the predominance of specific methods and the variation in methodological approaches. It was observed that evaluations of mobile digital maps often combine qualitative and quantitative methods, such as questionnaires, interviews, and the Think Aloud protocol. Additionally, the use of technological methods, such as data loggers and eye tracking, was also revealed. This pattern reflects a growing attention to collecting data related to understanding user interaction with digital maps, with the aim of optimizing navigation and the user experience.

However, a citation analysis revealed that combined methods, particularly those incorporating qualitative assessments, are more frequently recognized and cited within the academic community. This suggests a preference for methodological approaches that offer a comprehensive perspective on users’ behaviors and attitudes. This finding corroborates the proposition put forth by Roth et al. (2017), namely that there is an opportunity to conduct empirical research on the design and use of interactive maps and visualizations. It is recommended that qualitative and mixed methods research be expanded in order to confirm and enrich quantitative research in cartography.

The observed trends indicate a growing tendency towards moderated experiments conducted in controlled environments, such as laboratories, which allow for greater control over experimental variables. This corroborates the trends highlighted by Fairbairn and Hepburn (2023). Nevertheless, there are still obstacles to be overcome in order to accurately simulate the actual conditions under which mobile digital maps are used in real-life scenarios. The evaluation of mobile digital maps in external and mixed environments (indoor and outdoor) is less frequent due to the difficulty in controlling environmental variables. However, such studies are essential for a more accurate understanding of the performance and usability of mobile digital maps in real navigation situations. It is crucial to capture details about users’ interaction with maps in real usage situations, as these conditions are important.

In terms of geographical distribution of evaluations, the analysis revealed a notable concentration of studies on the American continent, with a particular focus on the United States. Additionally, there has been a discernible increase in contributions from Asia, particularly from countries such as Taiwan, China, and Japan. In Europe, a variety of methodological approaches were observed, reflecting a diverse range of techniques employed in countries such as Italy, the United Kingdom, and Germany. The analysis of the spatial distribution of methods demonstrated that the methodological choice is subject to variation depending on the geographic context. This suggests the existence of technological, financial, and also cultural differences in research approaches, which present a challenge that warrants further study and understanding.

It is therefore pertinent to investigate the evaluation methods employed in cartography, with particular focus on mobile digital maps. This should be done with due consideration of the profile of users and the aspects related to the interaction and use of maps by individuals or groups of people. This demand is corroborated by the International Cartographic Association (ICA), which initiated the Commission on Maps and Use of Spatial Data in 1987 under the direction of Judy Olson. The formal reestablishment of a successor Commission on Use & Usability (2007-2015) subsequently underwent a renaming to the Commission on Use, User, and Usability Issues (2015-2019) before finally becoming the Commission on User Experience (UX). This resulted in an expansion of the scope of issues related to the use of maps and the user experience. This expansion encompasses the utilization of hardware, software, information systems, interfaces, geographic data, and databases. Moreover, novel advancements in user experience design pertaining to information utilization and visualization, human-computer interaction, usability engineering, and web design have been integrated.

4. Conclusion

This study provides a contribution to the understanding of the methods used to evaluate mobile digital maps, highlighting the methods used and the configuration associated with each evaluation. Based on the results obtained, the most commonly used method is the questionnaire, often combined with an interview or think aloud protocol. This combination of methods makes it possible to collect qualitative and quantitative data simultaneously.

Evaluation sites vary between indoor and outdoor environments, depending on the type of map and the objectives of the study, which may influence the data collection methodology. However, there is a slight bias in favor of conducting evaluations in indoor environments. It is important to note that the choice of application context and testing method can have a significant impact on the results obtained. It is therefore important to consider these factors when planning and conducting evaluations of mobile digital maps.

Finally, we hope that this article will contribute to the advancement of cartographic research by providing a better understanding of the test methods used to evaluate mobile digital maps. As a recommendation for future studies, we suggest further investigation into which of the qualitative data collection methods (interview or think aloud protocol) is more effective in evaluating mobile digital maps in different environments, thus comparing the use of these methods in combination with other methods that optimize data collection.

ACKNOWLEDGMENT

The first author would like to express gratitude to CNPq (National Council for Scientific and Technological Development) for the grant number 174815/2023-9 for Junior Post-Doctorate - PDJ, which was granted in 2023. The first author would also like to express gratitude to the Coordination for the Improvement of Higher Education Personnel (CAPES) for their encouragement of research and financial support, process 88887.601430/2021-00, granted between 2021 and 2023. The third author would like to thank the National Council for Scientific and Technological Development - CNPq, for the funding (Process 310312/2017-5). The authors would also like to thank the Academic Publishing Advisory Center (CAPA - http://www.capa.ufpr.br) of the Federal University of Paraná (UFPR), for assistance with the English translation and developmental editing.

REFERENCES

• Abreu, L., and Moraes, A. (2012). Point Navigation Map method to track low-income mobile user’s interaction. Work, 41(1), 3438-3442.
• Aditya, T., Laksono, D., Sutanta, H., Izzahudin, N., and Susanta, F. (2018). A usability evaluation of a 3D map display for pedestrian navigation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
• Bartling, M., Resch, B., Reichenbacher, T., Havas, C. R., Robinson, A. C., Fabrikant, S. I., and Blaschke, T. (2022). Adapting mobile map application designs to map use context: a review and call for action on potential future research themes. Cartography and Geographic Information Science, 49(3), 237-251.
• Bartling, M., Robinson, A. C., Resch, B., Eitzinger, A., and Atzmanstorfer, K. (2021). The role of user context in the design of mobile map applications. Cartography and Geographic Information Science , 48(5), 432-448.
• Basiri, A., Lohan, E. S., Moore, T., Winstanley, A., Peltola, P., Hill, C., ... and e Silva, P. F. (2017). Indoor location based services challenges, requirements and usability of current solutions. Computer Science Review, 24, 1-12.
• Bernelind, S. (2015). Navigation in Augmented Reality, Navigation i Augmented Reality. Media And Interaction Design (MID), Second Level Stockholm, Sweden.
• BOARD, C. Higher order map-using tasks: Geographical lessons in danger of being forgotten. Cartographica. v.11, n.1, p.85-97, 1984.
• Brade, J., Lorenz, M., Busch, M., Hammer, N., Tscheligi, M., and Klimant, P. (2017). Being there again-Presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task. International Journal of Human-Computer Studies, 101, 76-87.
• Brata, K. C., and Liang, D. (2019). An effective approach to develop location-based augmented reality information support. International Journal of Electrical and Computer Engineering, 9(4), 3060.
• Brata, K. C., and Liang, D. (2020). Comparative study of user experience on mobile pedestrian navigation between digital map interface and location-based augmented reality. International Journal of Electrical and Computer Engineering , 10(2), 20-37.
• Brata, K. C., Pinandito, A., Ananta, M. T., and Priandani, N. D. (2017). Design of public transportation navigation system on Android Wear device. In 2017 International Conference on Sustainable Information Engineering and Technology (SIET) (pp. 385-391).
• Brata, K. C., Pinandito, A., Priandani, N. D., and Ananta, M. T. (2021). Usability improvement of public transit application through mental model and user journey. TELKOMNIKA-Telecommunication Computing Electronics and Control, 19(2), 397-405.
• Brock, A. M., Truillet, P., Oriola, B., Picard, D., and Jouffrais, C. (2015). Interactivity improves usability of geographic maps for visually impaired people. Human-Computer Interaction, 30(2), 156-194.
• Cho, M. H., and Castaneda, D. A. (2019). Motivational and affective engagement in learning Spanish with a mobile application. System, 81, 90-99.
• Chu, C., Wang, S., and Tseng, B. (2017). Mobile navigation services with augmented reality. IEEJ Transactions on Electrical and Electronic Engineering, 12, 95-103.
• Darvishy, A., Hutter, H. P., Grossenbacher, M., and Merz, D. (2020). Touch explorer: exploring digital maps for visually impaired people. In Computers Helping People with Special Needs: 17th International Conference, ICCHP 2020, Lecco, Italy, September 9-11, 2020, Proceedings, Part I 17 Springer International Publishing, (pp. 427-434).
• Delikostidis, I., Van Elzakker, C. P. J. M. (2011). Usability testing of a prototype mobile navigation interface for pedestrians. Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente
• Delikostidis, I., Van Elzakker, C., and Raak, M. (2016). Overcoming challenges in developing more usable pedestrian navigation systems. Cartography and Geographic Information Science , 43(3), 189-207.
• Diao, P., and Shih, N. (2018). MARINS: A mobile smartphone AR system for pathfinding in a dark environment. Sensors, 18(10), 34-42.
• Dick, M. E., Gonçalves, B. S., and Vitorino, E. V. (2017). Design da informação e competência em informação: relações possíveis| Information design and information literacy: possible relationships. InfoDesign-Revista Brasileira de Design da Informação, 14, 1-13.
• Dirin, A., Laine, T., and Alamäki, A. (2018). Managing emotional requirements in a context-aware mobile application for tourists. International Journal of Interactive Mobile Technologies, 12(2).
• Dong, W., Wu, Y., Qin, T., Bian, X., Zhao, Y., He, Y., and Yu, C. (2021). What is the difference between augmented reality and 2D navigation electronic maps in pedestrian wayfinding?. Cartography and Geographic Information Science , 48(3), 225-240.
• Drewlow, J., Däppen, M., and Lehmann, M. (2022). Navigation with Augmented Reality in a Hospital. Studies in Health Technology and Informatics, 292, 111-114.
• Dutta, R., Mantri, A., and Singh, G. (2022). Evaluating system usability of mobile augmented reality application for teaching Karnaugh-Maps. Smart Learning Environments, 9(1), 6.
• Einfeldt, L., and Degbelo, A. (2021). User interface factors of mobile UX: A study with an incident reporting application. In: 5th International Conference on Human Computer Interaction Theory and Applications
• Fairbairn, D., & Hepburn, J. (2023). Eye-tracking in map use, map user and map usability research: what are we looking for?. International Journal of Cartography, 9(2), 231-254.
• Fogli, D., Arenghi, A., and Gentilin, F. (2020). A universal design approach to wayfinding and navigation. Multimedia Tools and Applications, 79, 33577-33601.
• Goh, D. H., Lee, C. S., and Razikin, K. (2016). Interfaces for accessing location-based information on mobile devices: An empirical evaluation. Journal of the Association for Information Science and Technology, 67(12), 2882-2896.
• Grubert, J., Pahud, M., Grasset, R., Schmalstieg, D., and Seichter, H. (2015). The utility of magic lens interfaces on handheld devices for touristic map navigation. Pervasive and Mobile Computing, 18, 88-103.
• He, S., and Chan, S. H. G. (2015). Wi-Fi fingerprint-based indoor positioning: Recent advances and comparisons. IEEE Communications Surveys and Tutorials, 18(1), 466-490.
• Horbiński, T. (2019). Progressive evolution of designing internet maps on the example of Google Maps. Geodesy and Cartography, 68(1).
• Kapaj, A., Lanini-Maggi, S., and Fabrikant, S. I. (2021). The influence of landmark visualization style on expert wayfinders’ visual attention during a real-world navigation task. GIScience 2021 Short Paper Proceedings
• Kim, H., and Song, H. (2014). Evaluation of the safety and usability of touch gestures in operating in-vehicle information systems with visual occlusion. Applied ergonomics, 45(3), 789-798.
• Krassanakis, V., and Cybulski, P. (2019). A review on eye movement analysis in map reading process: The status of the last decade. Cognitive Processing, 20(1), 191-209.
• Kratz, S., Brodien, I., and Rohs, M. (2010). Semi-automatic zooming for mobile map navigation. In: Proceedings of the 12th international conference on Human-computer interaction with mobile devices and services, 63-72.
• Liang, H. N., Trenchard, J., Semegen, M., and Irani, P. (2012). An exploration of interaction styles in mobile devices for navigating 3D environments. In: Proceedings of the 10th Asia Pacific conference on Computer Human Interaction, 309-313.
• Link, J. Á. B., Smith, P., Viol, N., and Wehrle, K. (2013). Accurate map-based indoor navigation on the mobile. Journal of Location Based Services, 7(1), 23-43.
• Lu, F., Zhou, H., Guo, L., Chen, J., and Pei, L. (2021). An ARCore-Based Augmented Reality Campus Navigation System. Applied Sciences, 11(16), 7515.
• Maly, I., Mikovec, Z., Vystrcil, J., Franc, J., and Slavik, P. (2013). An evaluation tool for research of user behavior in a realistic mobile environment. Personal and Ubiquitous Computing, 17, 3-14.
• Maramba, I., Chatterjee, A., and Newman, C. (2019). Methods of usability testing in the development of eHealth applications: a scoping review. International Journal of Medical Informatics, 126, 95-104.
• Martins, V. B., Amorim, F. R., Schmidt, M. A. R., and Delazari, L. S. (2023). Study about the appropriate number of participants in map user studies. International Journal of Cartography , 1-14.
• Matsuo, M., Miura, T., Ichikari, R., Kato, K., and Kurata, T. (2020). OTASCE Map: A mobile map tool with customizable audio-tactile cues for the visually impaired. Journal on Technology and Persons with Disabilities, 82-103.
• Montuwy, A., Cahour, B., and Dommes, A. (2019). Using Sensory wearable devices to navigate the city: effectiveness and user experience in older pedestrians. Multimodal Technologies and Interaction, 3(1), 17.
• Nadzir, M., Hussain, A., Mkpojiogu, E. O., Faromiki, J. O., and Abdusalam, E. M. (2019). The effectiveness and efficiency of a GPS route and voice navigation app. International Journal of Innovative Technology and Exploring Engineering, 8, 425-427.
• Nair, V., Olmschenk, G., Seiple, W. H., and Zhu, Z. (2022). ASSIST: Evaluating the usability and performance of an indoor navigation assistant for blind and visually impaired people. Assistive Technology, 34(3), 289-299.
• Noguera, J. M., Barranco, M. J., Segura, R. J., and Martínez, L. (2012). A mobile 3D-GIS hybrid recommender system for tourism. Information Sciences, 215, 37-52.
• Noh, G., Jeong, D., Park, S., and Cho, J. (2015). Design Guideline on Location Based User Emotion Sharing Map Service. In HCI International 2015-Posters’ Extended Abstracts: International Conference, HCI International 2015, Los Angeles, CA, USA (pp. 514-518). Springer International Publishing
• Qiu, X., Yang, Z., Yang, J., Wang, Q., and Wang, D. (2023). Impact of AR Navigation Display Methods on Wayfinding Performance and Spatial Knowledge Acquisition. International Journal of Human - Computer Interaction, 1-21.
• Ramsay, A., Mcgee-Lennon, M., Wilson, G. A., Gray, S. J., Gray, P., and De Turenne, F. (2010). Tilt and go: exploring multimodal mobile maps in the field. Journal on Multimodal User Interfaces, 3, 167-177.
• Rehman, U., and Cao, S. (2015). Experimental evaluation of indoor navigation devices. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 1783-1787). Sage CA: Los Angeles, CA: SAGE Publications
• Rehman, U., and Cao, S. (2016). Augmented-reality-based indoor navigation: A comparative analysis of handheld devices versus google glass. IEEE Transactions on Human-Machine Systems, 47(1), 140-151.
• Rehrl, K., Häusler, E., Steinmann, R., Leitinger, S., Bell, D., and Weber, M. (2012). Pedestrian navigation with augmented reality, voice, and digital map: results from a field study assessing performance and user experience. In Advances in Location-Based Services: 8th International Symposium on Location-Based Services Springer Berlin Heidelberg, 3-20.
• Rehrl, K., Häusler, E., Steinmann, R., Leitinger, S., Bell, D., and Weber, M. (2014). Pedestrian navigation with augmented reality, voice, and digital map: final results from an in situ field study assessing performance and user experience. Journal of Location-Based Services, 8(2), 75-96.
• Ren, R., Castro, J. W., Acuña, S. T., and Lara, J. (2019). Evaluation techniques for chatbot usability: A systematic mapping study. International Journal of Software Engineering and Knowledge Engineering, 29(12), 1673-1702.
• Ricker, B. A., and Roth, R. E. (2018). Mobile maps and responsive design. Geographic Information Science and Technology Body of Knowledge, 40.
• Robinson, A. C., Kettunen, P., Delazari, L., and Çöltekin, A. (2023). New directions for the state of the art and science in Cartography. International Journal of Cartography , 1-7.
• Rohrer, C. (2014). When to use which user-experience research methods. Nielsen Norman Group, 12.
• Roth, R. E., Çöltekin, A., Delazari, L., Filho, H. F., Griffin, A., Hall, A., and Van Elzakker, C. P. (2017). User studies in cartography: opportunities for empirical research on interactive maps and visualizations. International Journal of Cartography , 3, 61-89.
• Roth, R. E., Çöltekin, A., Delazari, L., Filho, H. F., Griffin, A., Hall, A., and Van Elzakker, C. P. (2017). User studies in cartography: opportunities for empirical research on interactive maps and visualizations. International Journal of Cartography , 3, 61-89.
• Sakpere, W. E., Mlitwa, N. B. W., and Oshin, M. A. (2017). Towards an efficient indoor navigation system: a near field communication approach. Journal of Engineering, Design and Technology
• Sanjaya, M. R., Kurniawan, D., and Saputra, A. (2020). GIS Android mobile based Software Development for Tourism Objects, Public Places list, Transportations list, and Culinary Places list Using Usability Measurement. In Journal of Physics: Conference Series (Vol. 1500, No. 1, p. 012117).
• Savino, G. L., Emanuel, N., Kowalzik, S., Kroll, F., Lange, M. C., Laudan, M., ... and Schöning, J. (2019, October). Comparing pedestrian navigation methods in virtual reality and real life. In 2019 International Conference on Multimodal Interaction (pp. 16-25).
• Savino, G. L., Sturdee, M., Rundé, S., Lohmeier, C., Hecht, B., Prandi, C., ... and Schöning, J. (2021). MapRecorder: analysing real-world usage of mobile map applications. Behaviour and Information Technology, 40(7), 646-662.
• Schirmer, M., Hartmann, J., Bertel, S., and Echtler, F. (2015). Shoe me the way: A shoe-based tactile interface for eyes-free urban navigation. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 327-336).
• Schnitzler, V., and Holscher, C. (2015). User Experience and Strategy Choices During Navigation: A Content Analysis of Navigators Using Different Types of Wayfinding Devices. In EAPCogSci
• Sharin, N. A., Norowi, N. M., and Abdullah, L. N. (2020). A user requirements study for developing mobile-based augmented reality indoor map locator application. International Journal of Advanced Research in Engineering and Technology (IJARET), 11(12), 1361-1373.
• Skulimowski, P., Owczarek, M., Radecki, A., Bujacz, M., Rzeszotarski, D., and Strumillo, P. (2019). Interactive sonification of U-depth images in a navigation aid for the visually impaired. Journal on Multimodal User Interfaces , 13, 219-230.
• Sugimoto, M., Kusumi, T., Nagata, N., and Ishikawa, T. (2022). Online mobile map effect: How smartphone map use impairs spatial memory. Spatial Cognition and Computation, 22(1-2), 161-183.
• Swobodzinski, M., Parker, A. T., Wright, J. D., Hansen, K., and Morton, B. (2021). Seamless Wayfinding by a Deafblind Adult on an Urban College Campus: A Case Study on Wayfinding Performance, Information Preferences, and Technology Requirements. Frontiers in Education, 723098.
• VAN ELZAKKER, C. P. J. M. The use of maps in the exploration of geographic data Doctoral thesis. Geo-Information Science and Earth Observation Koninklijk Nederlands Aardrijkskundig Genootschap. Faculteit Geowetenschappen, Universiteit Utrecht. International Institute for. Utrecht, Netherlands, 2004.
• Van Elzakker, C. P., and Delikostidis, I. (2010). Use, user, and usability research for mobile geo-applications for personal orientation and navigation. In 2010 IEEE International Professional Communication (pp. 121-130).
• Vertzberger, E., and Klein, I. (2021). Attitude adaptive estimation with smartphone classification for pedestrian navigation. IEEE Sensors Journal, 21(7), 9341-9348.
• Vincent, C., Girard, R., Dumont, F., Archambault, P., Routhier, F., and Mostafavi, M. A. (2022). Evaluation of satisfaction with geospatial assistive technology (ESGAT): a methodological and usability study. Disability and Rehabilitation: Assistive Technology , 17(2), 134-151.
• Wang, Q., Luo, H., Wang, J., Sun, L., Ma, Z., Zhang, C., and Zhao, F. (2022). Recent advances in pedestrian navigation activity recognition: a review. IEEE Sensors Journal , 22(8), 7499-7518.
• Wenig, D., Steenbergen, A.,Schöning, J. , Hecht, B., and Malaka, R. (2016). ScrollingHome: Bringing image-based indoor navigation to smartwatches. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 400-406).
• Xu, Y., Qin, T., Wu, Y., Yu, C., and Dong, W. (2022). How do voice-assisted digital maps influence human wayfinding in pedestrian navigation?. Cartography and Geographic Information Science , 49(3), 271-287.
• Yang, F., Xiong, J., Liu, J., Wang, C., Li, Z., Tong, P., and Chen, R. (2019). A pairwise SSD fingerprinting method of smartphone indoor localization for enhanced usability. Remote Sensing, 11(5), 566.
• Zahabi, M., and Kaber, D. (2018). Identification of task demands and usability issues in police use of mobile computing terminals. Applied Ergonomics, 66, 161-171.
• Zein, S., Salleh, N., and Grundy, J. (2016). A systematic mapping study of mobile application testing techniques. Journal of Systems and Software, 117, 334-356.

ADDITIONAL INFORMATION

• Data Availability

  The dataset is available at: https://github.com/Data92repository/Data_01.git

Publication Dates

• Publication in this collection
  30 Aug 2024
• Date of issue
  2024

History

• Received
  12 Mar 2024
• Accepted
  31 July 2024