Abstract
The “International Group for Lean Construction” aims to adopt a new management philosophy for the Architecture, Engineering, and Construction (AEC) industry. Learning this philosophy can be supported by using active methodologies, especially serious games. This work aimed to analyse the evolution of using serious games as a teaching technique based on problem classes, in the context of the Lean Construction community. A literature review and an analysis of an empirical experience were carried out in an architecture and urbanism undergraduate course. The results demonstrate the evolution of the use of these techniques, as well as their implications for the education and training of AEC professionals.
Keywords: Serious Games; Lean Construction; Teaching; Active methodologies; Problem classes
Resumo
O “International Group for Lean Construction” busca adaptar à indústria de Arquitetura, Engenharia e Construção (AEC) uma nova filosofia gerencial. O aprendizado dessa filosofia pode ser apoiado pela utilização de metodologias ativas, especialmente dos jogos didáticos (do inglês “serious games”). Este trabalho teve como objetivo analisar a evolução da aplicação de jogos didáticos como técnica de aprendizagem baseada em classes de problemas, no contexto da comunidade da Construção Enxuta.Foi realizada uma revisão sistemática de literatura e a análise de uma experiência empírica, em uma disciplina de graduação de arquitetura e urbanismo. Os resultados demonstram a evolução do uso dessas técnicas, bem como suas implicações para a formação e treinamento de profissionais da AEC.
Palavras-chave: Jogos Didáticos; Construção Enxuta; Ensino; Metodologias Ativas; Classes de problemas
Introduction
Lean Construction is a production philosophy that focuses on continuous improvement by identifying and eliminating waste (activities that do not add value), understanding processes as flow rather than just conversion, in contrast to traditional construction management (Koskela, 1992). Research and practice efforts in Lean Construction have been carried out by the International Group for Lean Construction (IGLC)1. Founded in 1993, the IGLC is an international network of researchers and professionals who work in research, education and practice in Architecture, Engineering and Construction (AEC). By proposing new concepts, principles, and methods for project management and development, the IGLC aims to adapt successful Lean Production innovations (from the Toyota Production System) in manufacturing to the AEC industry.
Although Lean Construction principles, concepts, and practices have been widely disseminated, the underlying theories are not commonly known (Koskela; Tezel; Tzortzopoulos, 2018). Therefore, learning related to this philosophy can be supported by teaching approaches such as active methodologies.
According to Mota and Werner Da Rosa (2018), active methodologies encourage students to take responsibility for their learning autonomously, providing an alternative to the traditional teaching strategy of passive learning through oral and unilateral content transmission by the teacher. In fact, active methodologies allow students to construct meaning through relevant learning activities (Sasaki; Mällinen, 2018), involving collaborative environments with methods, techniques, and resources that stimulate interaction among students and between students and the teacher (Mota; Werner Da Rosa, 2018).
Predict-Observe-Explain, Jigsaw and Six Thinking Hats are some well-known active methodology techniques (Sasaki; Mällinen, 2018), which can also involve interactive serious games, making learning enjoyable, more natural, engaging, and fun (Moran, 2009). Serious games have been researched in various fields of knowledge to address the challenge of attracting students' attention and motivating them to learn. In the built environment, and particularly in the education of future civil engineers, architects, and urban planners, there is also significant potential for research on using games in terms of these professionals’ learning and training (Miron; Moura; Brito, 2012). This is because when people experience something in practice using all their senses, they assimilate theoretical concepts better (Moran, 2009).
Furthermore, a series of changes are currently occurring in teaching methods at Universities, especially as a result of the COVID-19 pandemic, which required a rapid development of remote teaching activities. Additional challenges arose to maintain students' attention considering distance learning, virtual reality and interaction using electronic equipment. Thus, using serious games as a teaching tool seems to be an effective way of transmitting content, especially in the context of construction management, in which games often present a problem and subsequently the principles, concepts and methods to solve it (Rocha; Miron, 2018). This problem-based learning is motivating for knowledge construction, making the process more engaging and participatory (Lorenzi; Miron, 2017). Thus, an approach structured around problem classes can assist in planning and selecting active methodologies.Problem classes are understood as the organisation of a set of practical or theoretical problems that contain useful artifacts (e.g., serious games) for action within the organisation (Dresch; Lacerda; Antunes Júnior, 2015), in this case, the educational institution.Both the problems and the artifacts that generate satisfactory solutions for these problems may share common characteristics that allow the knowledge to be organised into problem classes, thereby enabling the generalisation and advancement of knowledge in the field (Dresch; Lacerda; Antunes Júnior, 2015). Based on this problematisation, the following research question was proposed: How has problem-based learning using problem classes, particularly through serious games, evolved in the context of the Lean Construction community? Thus, the aim of this study was to analyse the evolution of using serious games as a problem-based learning technique within the context of the Lean Construction community.
Active methodologies and problem classes
Active methodologies are innovative approaches that allow for a more effective way to disseminate Lean Production principles, concepts, methods, and practices. Many simulations and serious games have been used as active methodologies to enhance understanding in the field of construction management, designed as active learning tools for both students and construction professionals (Bhatnagar et al., 2023; Forcael; Glagola; González, 2012). In simulations, an artificial environment is created that replicates reality (Lean et al., 2006; Narayanasamy et al., 2006), aiming to train and develop the user's skills (Bhatnagar et al., 2023). In serious games, participants engage in various interaction modes under a predefined context and with clear procedures (Bhatnagar et al., 2023), resulting in a goal-oriented activity (Narayanasamy et al., 2006) toward solving a problem or achieving the objective of the game. The objective is to instruct, educate and entertain users in physical or computer-based formats (Bhatnagar et al., 2023). Serious games are a useful tool in the construction industry, as they transfer knowledge clearly, realistically and simply (Hamzeh et al., 2017), playing a critical role in bridging the theory and practice of Lean Construction (Bhatnagar et al., 2023).
Serious games can also be understood as artifacts, artificial phenomena that serve human purposes (March; Smith, 1995). An artifact can be considered as an interface between an internal environment (the substance and organisation of the artifact itself) and an external environment, where it operates and functions (Dresch; Lacerda; Antunes Júnior, 2015). Serious games (artifacts) carried out in a real environment are instantiations that operationalise constructs, models and methods (March; Smith, 1995). Therefore, instantiation can play a relevant role in understanding principles, concepts, methods and practices, as well as problems and their possible solutions. Understanding the problem classes for which these artifacts (serious games) present an answer, for a certain specific problem, will also allow a parameter to compare competing satisfactory solutions that can be achieved in similar simulations (Dresch; Lacerda; Antunes Júnior, 2015). Three problem classes are explored in this research: “Lean Construction principles”, “value generation” and “production planning and control”.
Aiming at finding a structure for a production theory based on the Lean Production phenomenon, researchers have listed principles (Koskela, 1992) as propositions that guide the studies, approaches and practices of the Lean Construction community. A principle can be understood as a guiding rule for action, creating a bridge between theory and practice (Skaar et al., 2020). Among the “Lean Construction principles” used, three groups of principles from different authors can be mentioned:
-
Koskela (2000): reducing lead time, reducing variability, minimising the number of steps or parts;
-
Womack and Jones (1996): pull and create flow; and
-
Kenley and Seppänen (2010): balancing activities.
Serious games related to the “principles of Lean Construction” can cover this problem class more generally, addressing several related concepts, such as the Parade of Trades game (Tommelein, 2020), or they can cover specific concepts of this class, such as the 5S Puzzle Game (Obulam; Rybkowski, 2021), which explores Visual Management using the 5S tool.
The "value generation" problem class is one of the most important research topics for IGLC researchers. In this class, the client (interested party, consumer, user) is highlighted as the focus of the development process and use of the built environment. Value generation was illustrated by Koskela (2000) based on a value generation cycle, explaining customer-supplier interactions associated with 5 principles that establish propositions for managing customer requirements. Examples of serious games that are more focused on generating value are House Factory, a simulation to understand mass customisation in the house constructions (Rocha; Miron, 2018), as well as games that simulate Target Value Design (Munankami, 2016), a project approach based on maximising value (benefits) and target cost (Macomber; Howell; Barberio, 2007), and Choose by Advantages, a multi-criteria decision-making method (Arroyo et al., 2018).
Finally, the problem class “production planning and control” (PCP) refers to the decision-making processes that define the goals, means and controls of the development process of a work or project, usually subdivided into three levels: long, medium and short term. The planning technique that is most investigated by the IGLC community is the Last Planner System - LPS (Ballard, 1994, 1997; Ballard; Howell, 1998), a production system that creates a predictable workflow among several parties to achieve reliable results (Ballard, 2000). Examples of games related to the PCP problem class are LEBSCO, proposed by González et al. (2015), which simulates some aspects of the principles of lean production and LPS, and the game called Villego® (Warcup; Reeve, 2014), which also simulates LPS.
Method
To achieve the proposed objective, a systematic literature review and analysis of an empirical experience were carried out in an undergraduate course in Architecture and Urbanism.
Systematic literature reviews are secondary studies used to map, critically evaluate and consolidate results from relevant primary studies on a specific topic (Dresch; Lacerda; Antunes Júnior, 2015). The systematic literature review was considered as a basis for searching articles in the annals of the Lean Construction community’s main event, the Annual Conference of the International Group for Lean Construction (IGLC)2, which was established in 1993. Using "game" as the search word, 51 articles published until 2022 were initially retrieved. After a more detailed analysis, 37 articles were selected as they addressed the application of serious games. The following data were considered: authors and year of publication, materials used, which problem class the game was related to and whether the game was used in person or remotely (online).
Compared to publications in the literature, the experience of the Construction Economics and Management course of the Architecture and Urbanism Program at the Federal University of Rio Grande do Sul (UFRGS) was analysed, which adopts active methodologies to provide problem-based learning to students. Thus, the professor in charge, as well as the researchers from the Construction Economics and Management Group of the Postgraduate Program in Civil Engineering: Construction and Infrastructure has been using serious games in classes since 2011, initially in an in-person format and recently adapted to an online format as a result of emergency remote teaching (ERT) during the COVID-19 pandemic.
From the aforementioned course, all 22 semester schedules from 2011/1 to 2021/2 were collected to carry out an ex post facto study (investigation of a phenomenon that occurred in the past). Of these 22 semesters, the following data were considered: the semester, the number of students enrolled, the number of games used, the materials used, which problem class each game is related to, the concepts and techniques illustrated by the game and whether the game was used in person or remotely (online). From the graphs and tables prepared, a comparative analysis of IGLC publications with the experience of the course was developed. The professor and researcher’s accounts regarding the differences between in-person serious games compared to online games was also considered.
Serious games in lean construction
The use of games to demonstrate the theory and practice of Lean Construction has been reported in the literature as a way of facilitating the understanding of principles, concepts, approaches and techniques, in addition to allowing experimentation of possible results. The need for isolation caused by the COVID-19 pandemic encouraged the development of games that can be played on digital platforms. Figure 1 shows that most of the online games identified were reported in articles published from 2020 onwards (2 articles from 2008 to 2018 and 11 articles from 2020 to 2022).
It is worth noting that the same game can be played both in person and online. This is the case of the “Airplane Game” (Rybkowski et al., 2008) which is known for its Lego System® version but also has a version in specific software. Other serious games for teaching Lean Construction practices also used, for example, virtual reality as a learning tool (Jacobsen; Strange; Teizer, 2021; Jara; Alarcón; Mourgues, 2009). This type of tool can be adapted for both in-person and online applications. It is interesting to note that the main methods used are focused on interaction, allowing participants to be proactive, as is the case with Lego System®, virtual reality and different materials developed specifically for playing a game.
Figure 2 demonstrates the 37 articles selected from the IGLC conferences, 54.04% (totaling 20) are related to the “principles of Lean Construction” problem class; 27.03% (totaling 10) are related to the “value generation” problem class and 18.92% (totaling 7) are related to the “production planning and control” problem class. Furthermore, the “principles of Lean Construction” problem class was distributed evenly from 1998 to 2022. The “value generation” problem class emerged in 2016 and the “production planning and control” problem class was distributed from 2010 to 2020. Apparently, the learning and dissemination of broader classes (“Lean Construction principles” and “value generation”) require continuous effort in the search for active methodologies, such as serious games. The more specific techniques related to “production planning and control” such as the Last Planner System and Line of Balance (LOB) are less diverse in the number of serious games.
Experience of the course compared to the literature
Construction Economics and Management, a course from the 7th semester of the Architecture and Urban Planning Course at UFRGS, has been taught since the first semester of 2011.This course has theoretical and practical modules and covers topics related to architects' professional practice, including project planning and management, project management, cost management and quality management. It also aims to introduce principles and concepts aligned with management philosophies such as Lean Construction. Throughout the eleven years of the course, only in the first semester were there no games. Since the second semester of 2011, serious games have been introduced in the course, so that students can learn (through simulations) some principles, concepts and techniques related to the practice of managing AEC companies (production modes, value generation, planning and control techniques, cost management and quality management).
Figure 3 shows a summary of the in-person and online serious games used in the course from 2011 to 2021, considering the total number of participants. It is worth highlighting the high number of students participating in the dynamics of the games, which contributed to the consolidation of games in the course.
In the first year of the subject, only one game was taught, and different games were gradually included, helping students to learn. As the subject takes place twice a week, there are around 30 to 36 meetings throughout an academic semester (due to the COVID-19 pandemic, the number of weeks in some semesters was reduced from 18 to up to 15 weeks). It was observed that 5 games were a suitable amount to meet the objectives of the course, interspersing presentations in which the student passively learns the content, as well as games and Powerpoint presentations in which a more active stance is required.
From 2011 to 2019, the sequence of game themes was developed. Figure 3 shows that the types of games used in the analysed course are in accordance with what was reported in the literature of the Lean Construction community, especially in the change from in-person games to online games.
Table 1 shows the serious games used in the course from 2011 to 2021, indicating: the problem classes addressed, the names of the games, the concepts and techniques that the games help to illustrate and whether the game format is online (O) or in person (P).
Of the 5 in-person games listed (Table 1), one is for the “principles of Lean Construction” problem class, two for the “value generation” problem class and two for the “production planning and control” problem class, and the serious games for “creating value” were used at the beginning and end of the course. When adapting classes to the remote format, of the five online games used from 2020 to 2021 (Table 1), two are related to the “principles of Lean Construction” problem class, two for the “value generation” problem class and one for the “production planning and control” problem class, and in most semesters only four games were used (the Parade Game was used only once). Short explanations of each of the games are listed in Table 1, as well as the instantiations (operationalisation of the artifact in its environment) and the related problem classes:
-
the “ire Funnel Game”4 simulates a paper funnel production line (Post It). Eight representatives of the production process participate (customer, transporter, storer, marker, cutter, perforator, assembler and quality control) (Miron; Moura; Brito, 2012). Spreadsheets and a timer help record the performance. Also normally used in the initial phase of the discipline, this game is for the “principles of Lean Construction” problem class. Preceded by an explanation of the eleven principles of Lean Construction, originally published in Report 92 (Koskela, 1992), the three rounds of the game (10 minutes each) identify problems related to excessive flow activities (transport), production bottlenecks (overload on a workstation), work in progress (incomplete), variability (lack of standardisation), long cycle time (from order to delivery) and the reduced number of product deliveries compared to the number of orders. At the end of round 1, students are encouraged to identify the problems that occurred and propose solutions that are aligned with the application of Lean Construction principles. At the end of round 2, the improvements achieved are compared and new improvements are proposed. At the end of round 3, the improvements achieved are compared again, the main problems that occurred are identified and all the principles used are discussed. It is worth noting that in this game, students identify the problems and proposals for solutions to these problems in each round;
-
the “Parade of Trades” or “Parade Game”5 consists of a game to demonstrate the impacts of workflow variability on a construction project, in which several teams work independently (Tommelein; Riley; Howell, 1998). According to González et al. (2014), the main aim of the game is to illustrate how productivity is reduced, followed by project delay and the increase in losses as a result of increased flow variation. This game is related to the “Lean Construction principles” problem class, illustrating the impact that workflow variability has on the performance of construction work packages and their successors. This is achieved by simulating the construction process in which the resources produced by one activity are prerequisites for the work performed by the next work package (González et al., 2014);
-
due to the COVID-19 pandemic, researchers and academics from the Lean Construction community developed online versions of this game. The online Parade Game is played in two rounds so that the concepts presented can be understood (production capacity, buffers, variability, workflow), and is based on data probabilities, using cubic data (physical or virtual) with six faces (Biotto et al., 2021). In its online version in the course, the Parade Game did not arouse the same interest nor the expected level of discussion. Therefore, it was replaced by Batch, Pull, Balance, which deals with variability indirectly;
-
the online “Batch, Pull, Balance Game”7 was developed to illustrate the impacts of Little's Law (Hopp; Spearman, 2000), Theory of Constraints (Goldratt, 1990) and pull production (Womack; Jones, 1996). The objective of the game is to produce a set of quality houses as quickly and efficiently as possible. This game is related to the “principles of Lean Construction” problem class. Three rounds of the game are played. Files shared on Google Sheets make the game easier. Production times throughout the rounds are recorded. The difference between the first round and the second is the size of the lot of houses, that is, the lot for the first is larger than for the second round (unitary lot). Furthermore, in the first round, employees must not communicate with each other or inspect the quality of the houses. This activity is exclusive to the inspector. Finally, the third round has the same rules as the second, except any employee can inspect and correct errors made. At the end of each round, participants have to answer questions about the problems they faced (tunnel vision, bottlenecks, idle times) and make suggestions for improvements for the next round. For each round, the number of completed houses, the number of rejected houses, the time on the first house and the work in progress are recorded. Solutions are discussed to avoid bottlenecks, reduce work in progress and balance work between teams;
-
The "House Factory Game"8 (Rocha; Miron, 2018) aims to improve the intuitive understanding of mass customisation by simulating different modes of production (craft production, mass production, mass customisation and mass customisation with differentiation product delay) to illustrate the trade-off between product variety and production efficiency. House production simulations, in different production modes, are made with simple materials (graphed paper, colored paper, glue, scissors, pencil). Four rounds are simulated, one for each production mode, aiming to illustrate the main differences in production times, product diversity and process standardisation. Spreadsheets and a timer help record the performance. his game was associated with the “value generation” problem class due to its focus on simulating the mass customisation mode, which increases the product value without proportionally increasing the cost or delivery time (Rocha; Miron, 2018). However, by simulating in four rounds the differences between the modes of craft production, mass production, mass customisation and mass customisation with late product differentiation, the game allows a discussion about the evolution of the industry throughout the 20th century, the main differences between the different modes of production and the understanding of the coexistence of these modes today. Thus, craft products, massively standardised products or customised products coexist, presenting different production volumes, different sales values, and are therefore suitable for specific contexts (e.g.: craft products tend to be produced in small quantities and for higher prices but are appreciated by customers looking for unique and differentiated products). Thus, this game has been used in the initial phase of the course, helping to justify the study of new production philosophies, as well as related principles, concepts and techniques. It is worth considering that in this game there is not yet an actual problem to be solved, but rather the understanding of the phenomena that are the different modes of production.
-
the “TVD Game” in turn, was developed by Munankami (2016) to illustrate the principles of Target Value Design (Macomber; Howell; Barberio, 2007). This game was the last in-person game to be implemented in the course and is related to the “value generation” problem class. The game aims to simulate the basic principles of IPD (Integrated Project Delivery) and Target Value Design in two rounds. The game simulates the participation of three groups of stakeholders in a project: the client, designers and the production team. The project to be built is a 60cm tower (constructed with plastic straws and wooden sticks), which will hold up a marshmallow at the top. In the first round, the game aims to simulate the traditional Design Bid Build (DBB) contracting mode, in which the cost is determined based on the executive project and there is little integration between those involved, with teams working separately and communicating only through project documents. In the second round, a new tower is built in a context of simulation of the basic principles of IPD, a contracting method that brings together all parties (client, designers and executors) at the beginning of the project in a collaborative and integrated process of sharing data between teams and TVD, a management practice in which the project delivers value to customers and develop the project even with project cost restrictions. Spreadsheets and a timer help record the performance. The TVD approach requires greater integration between the project and the building budget, demanding client involvement and a collaborative approach between the teams involved in design, planning and doing;
-
the online “Customisation Game”9, aims to illustrate the process of choosing and purchasing housing. This game is related to the “value generation” problem class by simulating the concept of mass customisation as an organisational strategy for meeting the specific requirements of each client. The differences between the modes of craft production, mass production and mass customisation in terms of diversity and production volume are exemplified. Players impersonate clients and architects. The focus of the three rounds of the game is related to communication problems between architects and clients and how this can affect the management of client requirements and, consequently, the value generation. Using simplified questions and answers, communication difficulties between architects and their clients when choosing apartment standards are simulated (standard plans are presented for both groups of players). Files shared on Google Sheets make the game easier. As the game progresses in rounds, communication between these two groups of players improves and, in the same way, the percentage of success between the choices of clients and architects increases. At the end of the game, a discussion is held that emphasises the importance of dialogue between architects and clients, knowing how to listen and knowing how to ask the right questions to establish a relationship that enables the value generation;
-
the online “CBA Game”10, which illustrates the Choosing By Advantages technique (Arroyo et al., 2018; Parrish; Tommelein, 2009), aims to present a structured tabular method for decision-making based on advantage-maximising criteria. This game is related to the “value generation” problem class. In this method, players go through all the steps of a multi-criteria decision based on maximising advantages: 1. Identify alternatives of the same type of product or solution; 2. Define factors (relevant product characteristics); 3. Define essential criteria for evaluating each factor; 4. Describe the attributes of each alternative; 5. Decide the advantages of each alternative; 6. Decide the importance of each advantage; 7. Evaluate cost data. It is worth considering that in this decision-making process, first, all possible advantages (benefits) of each alternative are evaluated, and only at the end, is the cost considered. Thus, we aim to understand how to evaluate and consider the value generation, as opposed to the usual decision-making based on the lowest cost. The objectives and desired results for the game are: understanding the importance of good decision making; learn the basic definitions, principles and concepts of CBA; understand the decision making phases; learn to use basic CBA methods for decisions. Files shared in Google Sheets facilitate the dynamics of the game;
-
the “Line of Balance Game”11, developed by a team of researchers from UFRGS (Miron; Moura; Brito, 2012) illustrates the planning technique, called Line of Balance, which is based on location and allows planning the rhythm (time that each team stays in a base unit) and flow control of teams specialised in the production of repetitive elements (base units) throughout the entire project. This game is usually used in the second phase of the discipline, course when the focus becomes the “production planning and control” problem class. The game was created based on a schematic project of a housing unit to be assembled using the Lego System®. Six base units (repeating units) are constructed in each of the three rounds. Six specialised teams (foundations, marking, masonry, slab, roof and chimney) carry out their activities repeatedly in the six base units throughout the rounds (variable time - up to 9 min in the first round). Spreadsheets and a timer help record the performance. At the end of each round, based on the time records, the LOB is constructed to represent the house production, as it occurred. After each round, students are encouraged to identify the problems that occurred based on their perceptions, as well as by analysing the LOB. Based on this analysis, students propose solutions aligned with the LOB technique: uninterrupted flow of activities, balancing teams, use of time off, learning effect from repetitiveness, understanding the times of production batches of specialised activities. At the end of rounds 2 and 3, the LOB are compared and improvements are usually achieved that tend to the uninterrupted flow of specialised activities and the reduction of production times, which, consequently, reduces the time of rounds 2 and 3;
-
“ Villego® ”12 is a game that simulates the Last Planner® Planning System (Ballard; Howell, 1998) and is related to the “production planning and control” problem class. The Last Planner® System (LPS) is a planning and control methodology, based on the principles of the Lean Construction philosophy, structured into three levels (long, medium and short term) aimed at reducing uncertainty and variability and increasing planning reliability. The game uses the Lego System® to build two basic types of houses. The players (different execution teams) receive the project and spreadsheets for execution planning. According to González et al. (2014) the game has limitations (time of about 5 hours, minimum of 14 players, and high level of house complexity and LPS detailing) that may make it unsuitable for use in a university class environment in a short space of time. However, researchers from GEC/PPGCI/UFRGS adapted a simplified version of the game that allows it to be played during two rounds of around 2 hours each, also allowing for better discussions between each phase of the game. In the initial round, players build the house based on the project without planning. The execution time of the first round, recorded with a timer and spreadsheets, serves as a reference for defining a deadline for the second round, which is carried out using planning (long and short term). Throughout the game, techniques and concepts in the LPS are illustrated, among which we can highlight: the exercise of planning long and short-term plans, team commitment, understanding the PPC indicator (percentage of complete work packages), reducing waiting times and reducing total construction time; and
-
the online “PDCA Game”14 simulates the PDCA Cycle (Plan, Do, Check, Act) in house construction, illustrating concepts such as: variability, standardisation and continuous improvement. This game is related to the “production planning and control” problem class and the topic of quality management. The game is played in two rounds in which teams have to carry out plan, do, check, act activities. Files shared on Google Sheets make the game easier. At the end of the first round, participants must identify their performance, what they learned and how to use this learning in the next round. At the end of the second round, participants have to identify in which round they performed best, how they managed to apply PDCA and what the solutions would be to improve productivity, reduce variability and so that the learning curve is not lost over time. Finally, the concepts of standardisation and continuous improvement are discussed.
During 2020 and 2021, there was a need to change the type of games due to the COVID-19 pandemic. We sought to adapt the games so we could work with the same concepts. However, the dynamics of the online class presented other challenges to encourage engagement, and choosing the games was difficult.
Considering the professor and researcher’s accounts in relation to in-person serious games compared to online games, in-person games created an environment that facilitated participants’ engagement and “learning by doing”. Despite generating a more stimulating environment for remote teaching classes, online games were more subject to technical problems with each participant's internet networks. Thus, engagement and understanding of the game were sometimes affected by delays and lack of synchronisation in online connections. It is worth considering that online games are less mature compared to in-person games and that in the future they will be able to achieve all the qualities relevant to active methodologies.
Discussion
Serious games understood as useful artifacts for addressing problem classes appear implicitly in the evolution of IGLC conference publications, although they do not represent a discussion that is clearly explained in the articles. Selecting serious games to use in the course studied demonstrates a certain alignment with the evolution of using these methodologies by the Lean Construction community. However, it is worth considering that in the course there is a greater emphasis on the “value generation” problem class because it is an Architecture and Urbanism course, whose central activity is design, and in essence strongly related to the value generation cycle and its principles (Koskela, 2000).
Educational institutions constantly address the same problems with their students and both the problems and the serious games that lead to satisfactory solutions for these problems share common characteristics that allow the advancement of knowledge in the area (Dresch; Lacerda; Antunes Júnior, 2015). Particularly “learning by doing”, which experiences something in practice with all the senses, helps to better understand and apply theoretical concepts (Moran, 2009).
As described in each of the games used in the course, each simulation covers a considerable number of practical problems, concepts and techniques that they illustrate. After the rounds, students are encouraged to deepen their understanding of the problem and find solutions aligned with the problem classes addressed. Considering serious games as instantiations, the operationalisation of an artifact in an environment, instantiation sometimes precedes a complete articulation of the conceptual vocabulary and principles it incorporates (March; Smith, 1995). Therefore, the instantiation of serious games in teaching environments has effectively contributed to understanding Lean Construction concepts, principles, methods and techniques. Furthermore, the instantiation of serious games also allows them to be constantly improved and adapted to increase student learning.
These results indicate that serious games have functioned as active methodologies that stimulate self-construction of meanings in relevant learning activities (Sasaki; Mällinen, 2018), involving specific problem classes, collaborative environments with methods, techniques and resources that stimulate interaction, therefore, it is an alternative to the traditional teaching strategy of passive learning (Mota; Werner Da Rosa, 2018).
On the other hand, the instantiation of serious games with technical problems, as occurred during emergency remote teaching with some online games, can compromise the participants' engagement and, ultimately, the learning of the content. Thus, it can be observed that analysing serious games as instantiations related to problem classes shows a gap in knowledge to be investigated regarding the expansion of active methodologies in Lean Construction.
Conclusions
This study aimed to analyse the evolution of serious games as a learning technique based on problem classes in the context of the Lean Construction community. By analysing publications from the IGLC annual conferences, we were able to identify the increase in the use of serious games and, especially from 2020, the intensification of the use of online games in view of the need for social isolation resulting from the COVID-19 pandemic.
The analysis of the games used in the course corroborates the tendency to use serious games as active methodologies as an alternative to the traditional passive learning strategy. Furthermore, the instantiation of serious games proved to have a relevant role in understanding problem classes and their possible solutions through Lean Construction principles, concepts, methods and practices. However, although the problem classes are contained in the evolution of IGLC conference publications related to serious games, they still represent a knowledge gap to be explored.
The effort of research groups to quickly translate and adapt games to the online format also indicates that, even in a situation of social isolation, such as what occurred in Emergency Remote Teaching, games enable greater student engagement in their learning process. Thus, the field of investigation into the use of serious games can be expanded both in the area of training professionals and also engaging users to select, consume and manage the built environment.
Referências
- ALARCON, L. F.; ASHLEY, D. B. Playing games: evaluating the impact of lean production strategies on project cost and schedule. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 7., Berkeley, 1999. Proceedings […] Berkeley, 1999.
- ALVES, S.; MELZNER, J.; HOLLERMANN, S. Lean simulation game with BIM-Based progress monitoring for takt control. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 30., Edmonton, 2022. Proceedings […] Edmonton, 2022.
- ALVES, T. The Silo Game: a simulation on interdisciplinary collaboration. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 30., Edmonton, 2022. Proceedings […] Edmonton, 2022.
- ARROYO, P. et al A new method for applying choosing by advantages (CBA) multicriteria decision to a large number of design alternatives. Energy and Buildings, v. 167, p. 30-37, 2018.
- ASLESEN, S.; TOMMELEIN, I. D. What “makes” the Last Planner? a typology of behavioral patterns of Last Planners. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 24., Boston, 2016. Proceedings […] Boston, 2016.
- BALLARD, G. Lookahead planning: the missing link in production control. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 5., Gold Coast, 1997. Proceedings […] Gold Coast, 1997.
- BALLARD, G. The Last Planner. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 2., Santiago, 1994. Proceedings […] Santiago, 1994.
- BALLARD, G.; HOWELL, G. Shielding production: essential step in production control. Journal of Construction Engineering and Management, v. 124, p. 11-17, 1998.
- BALLARD, G. The Last Planner system of production control. Birmingham, 2000. 192 f. Doctor of Philosophy, School of Civil Engineering Faculty of Engineering, University of Birmingham, 2000.
- BHATNAGAR, S. et al A systematic review of lean simulation games in the construction industry. Architectural Engineering and Design Management, v. 19, n. 6, p. 701-719, 2023.
- BHATNAGAR, S.; DEVKAR, G. Development and testing of a simulation game on waste elimination using lean practices. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- BHATT, Y. et al Trainathon Lean Simulation Game: Determining Perceptions of the Value of Training Among Construction Stakeholders. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 24., Boston, 2016. Proceedings […] Boston, 2016.
- BINNINGER, M. et al Learning Simulation Game for Takt Planning and Takt Control. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 25., Heraklion, 2017. Proceedings […] Heraklion, 2017.
- BIOTTO, C. N. et al Virtual parade game for lean teaching and learning in students from Brazil and Chile. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- BRANDALISE, F. M. P. et al Exploring visual management purposes in construction projects. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- DRESCH, A.; LACERDA, D. P.; ANTUNES JUNIOR, J. A. V. A. Design science research: método de pesquisa para avanço da ciência e tecnologia. Porto Alegre: Bookman, 2015.
- ESQUENAZI, A.; SACKS, R. Evaluation of Lean improvements in residential construction using computer simulation. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 14., Santiago, 2006. Proceedings […] Santiago, 2006.
- FORCAEL, E.; GLAGOLA, C. R.; GONZÁLEZ, V. In corporation of computer simulations into teaching linear scheduling techniques. Journal of Professional Issues in Engineering Education and Practice, v. 138, n. 1, p. 21-30, 2012.
- FRIBLICK, F.; AKESSON, A.; LEIGARD, A. Learning Lean through lean game: a case from the infrastructure industry. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 15., East Lansing, 2007. Proceedings […] East Lansing, 2007.
- GOLDRATT, E. M. What is this thing called theory of constraints and how should it be implemented? Great Barrington: North River Press, 1990.
- GONZÁLEZ, V. A. et al LEBSCO: Lean-based simulation game for construction management classrooms. Journal of Professional Issues in Engineering Education and Practice , v. 141, n. 4, 2015.
- GONZÁLEZ, V. A. et al Simulating lean production principles in construction: a Last Planner-driven game. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 22., Oslo, 2014. Proceedings […] Oslo, 2014.
- HAMZEH, F. et al Application of hands-on simulation games to improve classroom experience. European Journal of Engineering Education, v. 42, n. 5, p. 471-481, 2017.
- HEYL, J. Lean simulation in road construction: teaching of basic Lean principals. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 23., Perth, 2015. Proceedings […] Perth, 2015.
- HOPP, W.; SPEARMAN, M. L. Factory Physics. 2nd ed. New York: McGraw-Hill/Irwin, 2000.
- HOWELL, G.; LIU, M. The Oops Game: how much planning is enough? In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 20., San Diego, 2012. Proceedings […] San Diego, 2012.
- JACOB, G. et al Target value design: development and testing of a virtual simulation. In: ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- JACOBSEN, E. L.; STRANGE, N. S.; TEIZER, J. Lean Construction in a Serious Game Using a Multiplayer Virtual Reality Environment. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- JARA, C.; ALARCÓN, L. F.; MOURGUES, C. Accelerating interactions in project design through extreme collaboration and commitment management: a case study. In : ANNUAL CONF. OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 17., Taipei, 2009. Proceedings […] Taipei, 2009.
- KENLEY, R.; SEPPÄNEN, O. Location-based management for construction: planning scheduling and control. Abington: Spon Pres., 2010.
- KOSKELA, L. An exploration towards a production theory and its application to construction. Espoo, 2000. 296 f. PhD. Thesis, Construction and Facility Management, Technical Research Centre of Finland, Espoo, 2000.
- KOSKELA, L. Application of the new production philosophy to construction. Stanford: Center For Integrated Facility Engineering, 1992. Technical report.
- KOSKELA, L.; TEZEL, A.; TZORTZOPOULOS, P. Why visual management? In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 26., Chennai, 2018. Proceedings […] Chennai, 2018.
- LEAN, J. et al Simulations and games: use and barriers in higher education. Active Learning in Higher Education, v. 7, n. 3, p. 227-242, 2006.
- LIDELÖW, H. The ER Design simulation game: experience and reflect. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 25., Heraklion, 2017. Proceedings […] Heraklion, 2017.
- LIU, C. et al Accelerating the Last Planner System® (LPS) uptake using virtual reality and serious games: a sociotechnical conceptual framework. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 28., Berkeley, 2020. Proceedings […] Berkeley, 2020.
- LORENZI, L. S.; MIRON, L. Método inovador de integração entre os cursos de engenharia civil e arquitetura no ensino de graduação para resolução de problemas. In: CONGRESSO BRASILEIRO DE EDUCAÇÃO EM ENGENHARIA, Joinville, 2017. Anais […] Joinville, 2017.
- MACOMBER, H.; HOWELL, G.; BARBERIO, J. Target-value design: Nine foundational practices for delivering surprising client value. AIA Practice Management Digest, p. 1-4, winter 2007.
- MARCH, S. T.; SMITH, G. F. Design and natural science research on information technology. Decision Support Systems, v. 15, n. 4, p. 251-266, 1995.
- MIRON, L. I. G.; MOURA, P. M.; BRITO, J. Jogos didáticos utilizados como instrumentos no ensino de arquitetura e gestão da construção. In : CONGRESO ARQUISUR - ARQUITECTURA Y CIUDAD CON COMPROMISO SOCIAL Y AMBIENTAL, 16., Buenos Aires, 2012. Proceedings […] Buenos Aires, 2012.
- MORAN, J. M. Ensino e aprendizagem inovadores com apoio das tecnologias: caminhos para a aprendizagem inovadora. In : MORAN, J. M.; MASETTO, M. T.; BEHRENS, M. A. Novas tecnologias e mediação pedagógica. Campinas: Papirus, 2009.
- MOTA, A. R.; WERNER DA ROSA, C. T. Ensaio sobre metodologias ativas: reflexões e propostas. Revista Espaço Pedagógico, v. 25, n. 2, p. 261-276, 2018.
- MUNANKAMI, M. B. Development and testing of simulation (game) to illustrate basic principles of integrated project delivery and target value design: a first run study. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 24., Boston, 2016. Proceedings […] Boston, 2016.
- MUSA, M.; PASQUIRE, C.; HURST, A. Using TVD simulation to improve collaboration. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 27., Dublin, 2019. Proceedings […] Dublin, 2019.
- NARAYANASAMY, V. et al Distinguishing games and simulation games from simulators. Computers in Entertainment, v. 4, n. 2, p. 1-18, 2006.
- NG, C.; HALL, D. Teaching target value design for digital fabrication in an online game: overview and case study. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- OBULAM, R.; RYBKOWSKI, Z. K. Development and testing of the 5S puzzle game. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- PARRISH, K.; TOMMELEIN, I. Making design decisions using choosing by advantages. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 17., Taipei, 2009. Proceedings […] Taipei, 2009.
- POLLESCH, P. et al House of cards: a simulation of Lean construction principles. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 25., Heraklion, 2017. Proceedings […] Heraklion, 2017.
- PÜTZ, C. et al Potential of gamification for lean construction training: an exploratory study. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- RAGHAVAN, N. et al Simulation exercise for collaborative planning system / Last Planner System (COLPLASSE). In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 26., Chennai, 2018. Proceedings […] Chennai, 2018.
- ROCHA, C. G. da; MIRON, L. I. G. The house factory: a simulation game for understanding mass customization in house building. Journal of Professional Issues in Engineering Education and Practice , v. 144, n. 1, p. 1-8, 2018.
- RUSSMANN, E. et al Development of an educational game to teach integrated project delivery principles. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 30., Edmonton, 2022. Proceedings […] Edmonton, 2022.
- RYBKOWSKI, Z. K. et al Development and testing of a Lean simulation to illustrate key principles of target value design: a first run study. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 24., Boston, 2016. Proceedings […] Boston, 2016.
- RYBKOWSKI, Z. K. et al Using controlled experiments to calibrate computer models: the Airplane Game as a lean simulation exercise. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 16., Manchester, 2008. Proceedings […] Manchester, 2008.
- RYBKOWSKI, Z. K. Last Planner and its role as conceptual Kanban. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 18., Haifa, 2010. Proceedings […] Haifa, 2010.
- RYBKOWSKI, Z. K.; DA ALVES, T. C. L.; LIU, M. The emergence and growth of the on-line serious games and participatory simulation group “APLSO”. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 29., Lima, 2021. Proceedings […] Lima, 2021.
- SACKS, R.; GOLDIN, M.; DERIN, Z. Pull-driven construction of high-rise apartment buildings. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 13., Sydney, 2005. Proceedings […] Sydney, 2005.
- SASAKI, D. G. G.; MÄLLINEN, S. Developing student-centered assessment for a postgraduate course designed for Basic Education Teachers. Revista Ibero-Americana de Estudos em Educação, v. 13, n. esp 1, p. 520-525, 2018.
- SEPPÄNEN, O. A production control game for teaching of location-based management system’s controlling methods. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 20., San Diego, 2012. Proceedings […] San Diego, 2012.
- SKAAR, J. et al Principles as a bridge between theory and practice. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 28., Berkeley, 2020. Proceedings […] Berkeley, 2020.
- SMITH, J. P.; RYBKOWSKI, Z. K. The maroon-white game a simulation of trust and long-term gains and losses. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 21., Fortaleza, 2013. Proceedings […] Fortaleza, 2013.
- SOLHJOU KHAH, F. et al Development and testing of an innovative architectural programming simulation as a precursor to target value design. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 27., Dublin, 2019. Proceedings […] Dublin, 2019.
- TOMMELEIN, I. D. Takting the parade of trades: use of capacity buffers to gain work flow reliability. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 28., Berkeley, 2020. Proceedings […] Berkeley, 2020.
- TOMMELEIN, I. D.; RILEY, D.; HOWELL, G. A. Parade game: impact of work flow variability on succeeding trade performance. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 6., Guarujá, 1998. Proceedings […] Guarujá, 1998.
- TSAO, C. C. Y.; HOWELL, G. A. Development of simulations & pull planning for Lean construction learning and implementation. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 30., Edmonton, 2022. Proceedings […] Edmonton, 2022.
- WARCUP, R.; REEVE, E. Using the villego® simulation to teach the last planner® system. Lean Construction Journal, v. 2014, p. 1-15, 2014.
- WOMACK, J.; JONES, D. Lean thinking: banish waste and create wealth in your corporation. New York: Simon & Schuster, 1996.
- YAW, M. W.; RYBKOWSKI, Z. K. D. J. H. Reducing handoffs between sequential trades: a simulation. In : ANNUAL CONFERENCE OF THE INTERNATIONAL GROUP FOR LEAN CONSTRUCTION, 28., Berkeley, 2020. Proceedings […] Berkeley, 2020.
-
1
https://iglc.net.
-
2
https://iglc.net/Papers.
-
3
The selected IGLC articles are:Alarcon and Ashley (1999),Alves, Melzner and Hollermann (2022),Alves (2022),Aslesen, Tommelein and (2016),Bhatnagar and Devkar (2021),Bhatt et al. (2016),Binninger et al. (2017),Biotto et al.(2021),Brandalise et al. (2021),Esquenazi and Sacks (2006),Friblick, Akesson and Leigard (2007),González et al.(2014),Heyl (2015),Howell and Liu (2012),Jacob et al.(2021),Jacobsen, Strange and Teizer (2021),Lidelöw (2017),Liu et al. (2020),Musa, Pasquire and Hurst (2019),Ng and Hall (2021),Obulam and Rybkowski (2021),Pollesch et al. (2017),Pütz et al. (2021),Raghavan et al.(2018),Russmann et al.(2022),Rybkowski et al.(2016),Rybkowski et al. (2008),Rybkowski (2010),Rybkowski, Da Alves and Liu (2021),Sacks, Goldin and Derin (2005),Seppänen (2012),Smith and Rybkowski (2013),Solhjou Khah et al. (2019),Tommelein (2020),Tommelein, Riley and Howell (1998),Tsao and Howell (2022)andYaw and Rybkowski (2020).
-
4
The Pasquire Funnel Game, based on an original idea by Rubicon Associates.
-
5
https://p2sl.berkeley.edu/knowledge-center/parade-of-trades-game/.
-
6
Adapted version developed by ASKM & Associates, LLC and Navilean LLC copyright Creative Commons license - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).
-
7
Version developed by ASKM and Associates, LLC under Creative Commons license - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). Version translated and adapted by researchers from GEC/PPGCI/UFRGS: Fernanda M. P. Brandalise; Daniela D. Viana; Fernanda S. Bataglin.
-
8
https://www.ufrgs.br/housefactorygame/pb/.
-
9
Original game Architectural Programming (AP) Simulation. Created by Fatemeh Soljou Khah & Zofia Rybkowski, PhD. Department of Construction Science, Texas A&M University. Version translated and adapted by researchers from GEC/PPGCI/UFRGS: Cynthia Hentschke and Manoela Conte.
-
10
Developed by researchers Paz Arroyo and Annett Schöttle. Translation into Portuguese and adaptation by researchers GEC/PPGCI/UFRGS: Fernanda M. P. Brandalise, Fabrício Vargas and Daniela D. Viana.
-
11
https://www.ufrgs.br/jogolinhadebalanco/creditos/.
-
12
BOB bv trademark: https://www.villego.com.
-
13
Lean Construction Institute trademark: www.leanconstruction.org.
-
14
ASKM Associates, LLC under Creative Commons license - Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) Version translated and adapted by GEC/PPGCI/UFRGS researcher: Fabrício Vargas.
Publication Dates
-
Publication in this collection
11 Mar 2024 -
Date of issue
2024
History
-
Received
14 Apr 2023 -
Accepted
02 Oct 2023