A novel technological performance measurement indicator: a smart manufacturing approach

Um novo indicador de medição de desempenho tecnológico: uma abordagem de fabricação inteligente

Abstracts

Abstract

The implementation of digital manufacturing technologies (DMTs) represents the beginning of transforming a manufacturing system towards a smart manufacturing system (SMS). Assessing the performance of the DMTs implemented is essential to meet the objectives in a SMS and allows identifying their usefulness. However, estimating this performance is a challenging task due to the heterogeneous characteristics of the DMTs, such as the origin of information, capacity, connectivity, etc. Although some SMS performance measurement metrics are known, none are intended to identify the performance of DMTs. This article follows a methodology for the construction of technological performance indicators and proposes a novel indicator based on the individual characteristics of the DMTs and the smart factory concept of interoperability. The proposed indicator allows approaching the behavior of one or multiple DMTs implemented simultaneously and introduces a quantifiable measurement that can be applied to any industrial process. It is noteworthy, that such an indicator is not present in the literature and may be of great interest to enterprises currently implementing DMTs related to SMS. The applicability of the indicator considering multiple DMTs is validated through an illustrative test case.

Keywords:
Digital manufacturing technologies; Smart manufacturing; Indicator; Interoperability, Measurement

Resumo

A implementação de tecnologias de fabricação digital (DMTs) representa o início da transformação de um sistema de fabricação em um sistema de fabricação inteligente (SMS). Avaliar o desempenho dos DMTs implementados é essencial para cumprir os objetivos de um SMS e permite identificar a sua utilidade. No entanto, estimar esse desempenho é uma tarefa desafiadora devido às características heterogêneas dos DMTs, por exemplo, origem da informação, capacidade, conectividade etc. Embora algumas métricas de medição de desempenho de SMS sejam conhecidas, nenhuma é específica para identificar o desempenho dos DMTs. Este artigo segue uma metodologia para a construção de indicadores de desempenho tecnológico e propõe um novo indicador baseado nas características individuais dos DMTs e no conceito de interoperabilidade de fábrica inteligente. O indicador proposto permite abordar o comportamento de um ou vários DMTs implementados simultaneamente e apresenta uma medição quantificável que pode ser aplicada a qualquer processo industrial. É imortante destacar que tal indicador não está presente na literatura e pode ser de grande interesse para as empresas que estáo atualmente implementam DMTs relacionadas ao SMS. A aplicabilidade do indicador considerando vários DMTs é validada através de um caso de teste ilustrativo.

Palavras-chave:
Tecnologias de fabricação digital; Fabricação inteligente; Indicador; Interoperabilidade, Medição.

1 Introduction

Currently, many industries are transforming into a smart manufacturing system (SMS) by adopting advanced Information and Communication Technologies (ICT) to increase the level of automation and digitization of production, manufacturing, and industrial processes. The ICTs involved in this transformation are known as technological enablers, which are pillars of the industry 4.0 (I4.0) concept, such as emerging digital technologies, digital manufacturing technologies (DMTs), etc. The DMTs mainly mentioned in the literature include the Internet of things (IoT), cloud computing, integration systems, industrial robots, simulation, virtual reality, big data, cyber security, additive manufacturing, and various analytics techniques (Kamble et al., 2020).

From the academic point of view, some of the technologies examined are still rapidly transforming (Lu, 2017). This is due to the unprecedented increase in the number of technology sources and the heterogeneity in their characteristics (e.g., the origin of information, capacity, and connectivity). Integrating of the operation of the DTMs and their effective use allows achieving the SMS objectives (Wu, 2009). Therefore, analyzing the characteristics of DTMs is essential to discover their performance (Wu, 2009). Furthermore, DMTs can be implemented individually or in combinations, which reinforces the importance of exploring the interoperability feature.

Performance is an expression that compares the achievement of a process, device, or product to a given reference level and indicates a deficiency that can then be acted upon (Kamble et al., 2020). An exponential increase in the literature on DMTs can be seen in this direction. For example, the authors of Büchi et al. (2020) identify the main characteristics of I4.0 and its DMTs and verify the causal relationship between the degree of openness to I4.0 and performance. In Nara et al. (2021), the authors propose a model to analyze the impact of I4.0 technologies on several key performance indicators related to sustainable development. In Dalenogare et al. (2018), the authors discuss work contributes by examining the real expectations on the industry's future performance when implementing DMTs, providing a background to advance in the research on tangible benefits of I4.0. These studies mention the characteristics of DMTs, their positive relationship between them, their impact, and possible benefits for SMS (Büchi et al., 2020; Cugno et al., 2021). The literature reports performance measures of some DMTs, such as big data, evaluating characteristics such as volume of information, variety, and veracity (Ferrari et al., 2017). Furthermore, IoT performance by assessing features such as latency and unlimited throughput (Cappa et al., 2021). However, the literature lacks a measurement indicator where the performance is considered according to the interaction between different DMTs.

Therefore, this article proposes a performance indicator for DMTs to ascertain their usefulness, considering the individual characteristics of each DMT and the connection between them under the concept of interoperability. Compared to the available performance measurements of DMTs, the proposed indicator considers the state of operation from a technical point of view with a quantitative perspective. This paper uses the performance indicator design methodology proposed in Ibarguen-Valverde et al. (2017) as a guide, starting with a diagnostic to identify the testing features. Next, it compares of how interoperability between DMTs has been measured in the literature, followed by the proposal of the mathematical model. The development of the mathematical model was inspired by a structure presented in the Society of Automotive Engineers (SAE) J4000 standard developed by the SAE, establishing a numerical measure of the expected level and the actual value of a specific element (SAE, 1999). Finally, the proposed indicator is tested in an application case. It is noteworthy, that such an indicator is not present in the literature and may be of great interest to enterprises currently implementing DMTs related to SMS.

The rest of the paper will be structured is as fallows. In Section 2, an overview of the existing literature is presented, where we investigate available approaches to measure the performance of DMTs. In Section 3, we present the methodology for the construction of the technological performance indicators. Furthermore, we introduce the proposed novel indicator based on the individual characteristics of the DMTs and the smart factory concept of interoperability.

2 Theoretical background

The expected effects of the adoption of DMTs in SMS have been a topic of interest during the last years. The positive effects of adopting the DMTs in SMS are associated with their correct implementation, prepared human resources, the functionality of the technologies, among others (Dalenogare et al., 2018; Nara et al., 2021; Büchi et al., 2020; Zeid et al., 2019; Silva et al., 2022; Cugno et al., 2021).

The performance measures have been considered according to the characteristics of the DMTs. Some of these characteristics are analyzed from a qualitative point of view and others from a quantitative point of view. Quantitatively, the performance of DMT Big data has been addressed in (Ferrari et al., 2017), analyzing these three characteristics: volume, variety, and veracity. Table 1 shows the proposed measure for each feature.

Table 1
Definitions and Measures for Big Data Volume, Variety, and Veracity from Cappa et al. (2021).

Big Data Dimensions	Definition	Measure
Volume	Amount of data collected	Number of mobile device applications downloaded
Variety	Assortment of data collected per observation	Number of types of data collected per application
Veracity	Reliability and insightfulness of data	Percentage of employees devoted to big data analysis

Similarly, DMT IoT performance has been addressed in Cappa et al. (2021) through latency estimation using the initiator-to-partner round-trip latency indicator (RTL IP) and the time offset indicator between Initiator and Partner (OFF IP). The performance of DMT Virtual reality has been analyzed based on quantitative characteristics such as projection efficiency, which considers the relationship between the spherical surface area and the calibrated projection area. Also, qualitative characteristics such as subjective quality have been examined (Hu et al., 2021).

In the vision of DMT functionality, one of the main requirements is to achieve interoperability across diverse technologies (Zeid et al., 2019; Frederico et al., 2021). Interoperability is “the ability for two systems to understand one another and to use the functionality of one another” (Chen et al., 2008, p. 648). In other words, interoperability is the ability of two systems to interchange data, information, and knowledge (Lu, 2017), (Guédria et al., 2015). The Institute of Electrical and Electronic Engineers (IEEE) defines interoperability as “The ability of two or more systems or components to exchange information and use the information that has been exchanged” (Geraci, 1991, p. 42. The stated role of interoperability within the needs of SMS is to synthesize software components, business processes and application solutions through a diversified heterogeneous and autonomous process. To achieve a level of interoperability, establishing global standards and architecture guidelines is necessary for the development of SMS (Burns et al., 2019; Pedone & Mezgár, 2018). For example: Reference Architecture Model for Industry 4.0 (RAMI 4.0), Standards Landscape for Smart Manufacturing Systems, and National Smart Manufacturing Standards Architecture Construction Guidance. In Saturno et al. (2017) an analysis of the level of interoperability between systems within an existing automation platform (ISA-95) is presented. This work uses the AHP method for this evaluation, extracting criteria from the literature and the experience of experts.

Interoperability measures can be approached from both qualitative and quantitative perspectives. Qualitative measures use a rating scale made up of linguistic variables (for example, “Good,” “Optimized,” and “Adaptive”) to rate a system. Qualitative measures are commonly used in maturity models. Quantitative measures define numerical values to characterize interoperability. In general, the rating scale is from 0 to 100%. For example, some approaches use equations to determine interoperability based on the “actual/expected” relationship (Pedone & Mezgár, 2018). Another approach is the evaluation of interoperability from two criteria: independent and dependent. Independent criteria include cost, time, and quality. Dependent criteria include the degree of coupling or compatibility, as well as who evaluate the system (Neghab et al., 2015). Formal interoperability measures based on the semantic relations between two information systems are presented in Table 2 (Yahia et al., 2012).

Table 2
Interoperability measure from Yahia et al. (2012).

Indicator	Measure
Maximal potential interoperability	$v_{1\to 2}=\frac{\left|R_c^2\right|}{\left|R_{cexpected}^2\right|}$, where $R_c^2$ is the semantic relationships and $R_{cexpected}^2$ is the total number of the expected semantic relationships to fully interoperate.
Minimal effective interoperability	$v_{1\to 2}^{\mathrm{e}}=\frac{\left|R_c^{2\mathrm{e}}\right|}{\left|R_{cexpected}^2\right|}$, where $R_c^{2\mathrm{e}}$ is the returned effective semantic relationships and $R_{cexpected}^2$ is the total number of the expected semantic relationships to fully interoperate.

Although the studies discussed above established the relevance of the functionality of the DMTs, the measure of this functionality, i.e., performance, for each DMTs is not mentioned. Moreover, the measure of interoperability between them is disregarded. The cited studies can be seen as a first approach to the study of the performance of DMTs. While studies have extensively examined measures of functionality of individual DMTs, there is a significant research gap in understanding the overall functionality of DMTs through interoperability. Figure 1 illustrates this research gap, highlighting the need for more research in this area.

Figure 1
Research gap.

3 Performance indicator of digital manufacturing technologies: design and evaluation

The methodology proposed in Ibarguen-Valverde et al. (2017) for the construction of energy performance indicators was adapted for the design of the technological performance indicator, as shown in Figure 2. The methodology begins with a diagnostic phase that identifies the characteristics of the DMTs with which self-performance will be evaluated. In addition to identifying the interoperability characteristics with which mutual-performance will be evaluated. This step is followed by establishing the mathematical model that quantifies the performance of DMTs against their expected performance. The final phase involves the analysis and monitoring of the proposed indicator; here the numerical result and its significance is established.

Figure 2
Methodology for the construction of technological performance indicators.

3.1 Diagnostic phase

A performance indicator is a quantitative expression that compares the achievement of a process, device, or product to a given reference level and indicates a deficiency that can then be acted upon (Klovienė & Uosytė, 2019). That is why a performance indicator is proposed to measure DMTs to identify functionality. The performance indicator considers the individual characteristics (self-performance) and the connection between them (mutual-performance) under the concept of interoperability.

In studies such as Büchi et al. (2020), Bigliardi et al. (2020), Ardito et al. (2018) and Mabkhot et al. (2021), the list of DMTs is defined as: advanced manufacturing; additive manufacturing; augmented reality; simulation; cloud computing; industrial internet of things; cyber security; and big data analysis. These studies also define the characteristics of each DMT. A summary of the findings on DMTs is show in Table 3, where the characteristic identified in each DMT is evaluated according to the performance.

Table 3
Summary DMTs.

i	DMT	Definition	Characteristic
1	Industrial robots	Intelligent machines capable of performing tasks in the world by themselves, without explicit human control	Autonomy
			Perception
			Deliberation
			Productivity
2	Simulation	Technologies that reproducing the physical world in virtual models and allowing operators to test and optimize the settings to obtain materials, productive processes (discrete elements), and products (finished or distinct elements)	Representation level
3	IoT	This corresponds to a set of devices and intelligent sensors that facilitate communication between people, products, and machines	Latency
			Unlimited throughput
4	Cyber security	Security measures designed to protect the flow of information over interconnected corporate systems	Identification and detection
5	Cloud computing	Technologies facilitate the archiving and processing of large quantities of data with high performance in terms of speed, flexibility, and efficiency	Store capacity
			Secure
			Reliable
6	Additive manufacturing	This additive production process allows for complex products by creating layers of materials, including such different types of materials as plastics, ceramics, metals, and resins, thus eliminating the need to assemble the material	Higher speed in prototype
			Productivity
7	Virtual reality	This involves a series of devices that enriches (or lessens) human sensory perception through the access to virtual environments	Perception of reality
			Latency
			Positioning accuracy
8	Big data and Analytics	Technologies that capture, archive, analyze, and disseminate large quantities of data derived from the products, processes, machines, and people interconnected in a company	Volume
			Velocity
			Variety
9	Integration systems	Integration in two dimensions: internal versus external. The first (horizontal integration) concerns the integration and exchange of information among the different areas in the company. The second (vertical integration) concerns the company's relationships with its suppliers and customers	Reference Architecture

From Zeid et al. (2019), Sun et al. (2020) and Burns et al. (2019) it is evident that the concept of interoperability is highly related to the performance of DMTs. Although some tools measure the interoperability, none of them is directly designed for its use with DTMs from a technical point of view. Instead, they focus on a semantic vision of communication between systems, which results in qualitative characterizations. However, in Saturno et al. (2017), an analysis is carried out under the concept of interoperability of I4.0 in the areas shown in Table 4. Each area is assigned a value of three levels according to its degree of maturity.

Table 4
Interoperability I4.0 areas from Saturno et al. (2017).

Area	Definition
Infrastructure	Define a single standard for networks and communication protocols between the interoperable systems facilitating the communication between different suppliers for the same solution
Architecture standard	An architecture that meets the international standards, with the application of open protocols and ease of access to decrease the complexity and barriers in the integration of this architecture
Software platform	Flexible software platforms with easy remote access and availability of access by Web Services
Technology update	Define the potential of future integration with other systems. Software updates facilitated and hardware components exchange occurs in a modular way.

Hence, there is a gap in the literature for a quantitative indicator measuring the performance of DTMs based on the concept of interoperability.

3.2 Mathematical model

The proposed indicator is inspired by a structure presented in the Society of Automotive Engineers (SAE) J4000 standard developed by the SAE, which was developed to identify and measure companies' best lean manufacturing practices (SAE, 1999). The standard establishes a numerical measure of the expected level and the actual value of a specific element. In Lucato et al. (2019), the same standard was used to define the degree of general readiness for the adoption of I4.0 by a company considering all DMTs. However, neither SAE (1999) nor Lucato et al. (2019) measure performance directly.

This paper proposes an indicator based on the parameter introduced in the SAE J4000 standard, that provides a performance rate of each DMTs and interaction performance rate between pairs of DTMs.

The proposed indicator, here referred to as $MpT$, includes a contribution from the arithmetic mean of the performance evaluation of self-performance and the mutual-performance evaluation of all pairs of technologies, Equation 1:

$MpT=\frac{\sum _{i=1}^n\sum _{j=1}^n{r}_{pij}}{h}:\mathrm{}i\le j$,(1)

where,

- $n$ is the number of technologies to analyze and $i,j\in \left\{1,2,\dots ,n\right\}$,

- $r_{pij}$ is the rate of performance of the DMT $i$ relative to DMT$j$,

- and $h$ is the number of elements in the sum.

It should be noted that the rate of performance can be expressed as the matrix $MMpT$ consisting of all possible values of $r_{pij}$, Equation 2:

$MMpT=\left[\begin{array}{c}\begin{array}{ccc}{r}_{p11}& r_{p12}& \dots \\ r_{p21}& r_{p22}& \dots \\ ⋮& ⋮& \ddots \end{array}\begin{array}{c}{r}_{p1j}\\ r_{p2j}\\ ⋮\end{array}\\ \begin{array}{ccc}{r}_{pi1}& r_{pi2}& \begin{array}{cc}\dots & r_{pij}\end{array}\end{array}\end{array}\right]$ (2)

The sum of all the elements $MMpT$ is given by $\sum _{i=1}^n\sum _{j=1}^n{r}_{pij}$; the condition $i\le j$ in Equation 1 removes the elements located above the main diagonal from the sum; this is done because the matrix is symmetric ($r_{pij}=r_{pji}$). The number of elements in the summation is given by $h=\frac{n(n+1)}{2}$. Note that the elements that correspond to $i=j$ describes the self-performance of each technology.

Formally, $r_{pij}$is represented as the ratio between the length of the projection of the vector $r_{eij}$ and the length of $r_e´$, Equation 3. The vector $r_{eij}$ is referred to as the result vector and the vector $r_e´$ is termed the standard vector, as shown in Figure 3.

Figure 3
Ratio vector result vs vector standard.

${\mathit{r}}_{\mathit{p}\mathit{i}\mathit{j}}=\frac{\mathit{L}{\mathit{r}}_{\mathit{eij}}\mathrm{}\mathbf{cos}\mathit{\theta }}{\mathit{L}{\mathit{r}}_{\mathit{e}}´}$ (3)

The vector $r_{eij}$ is composed of the measurements related to each characteristic evaluating the performance between DMT $i$ and DMT$j$, Equation 4. This evaluation is formally defined as $e_m$, with $m$ representing the number of characteristics considered,

${\mathit{r}}_{\mathit{e}\mathit{i}\mathit{j}}=\left[\begin{array}{c}{\mathit{e}}_1\\ \begin{array}{c}{\mathit{e}}_2\\ ⋮\end{array}\\ {\mathit{e}}_{\mathit{m}}\end{array}\right]$ (4)

A characteristic (capacity) has been defined based on the properties of DMTs (Büchi et al., 2020; Bigliardi et al., 2020; Ardito et al., 2018; Mabkhot et al., 2021; Rubio et al., 2018; Rajan, 2013) for assessing the self-performance of a technology (corresponding to the elements ${MpT}_{ij}$ for $i=j$):

• a. Capacity: This measure establishes the degree of fulfilment of the principal characteristic of the given technology. These characteristics are shown in Table 5 related to various DTMs.

  Table 5
  Self-performance DMTs characteristic.
  

  i	DMT	Characteristic	Measure
  1	Industrial robots	Productivity	Short cycle time
  2	Simulation	Representation level	Product and processes
  			Production lines, workstations, internal logistics
  			Enterprise, operational environment
  3	IoT	Latency	Time transferring data (Cappa et al., 2021)
  4	Cyber security	Detection	Physical
  			Virtual (Rubio et al., 2018)
  5	Cloud computing	Store capacity	Substantial quantities of storage (Rajan, 2013)
  6	Additive manufacturing	Higher speed in prototype	Time prototyping (Wong & Hernandez, 2012)
  7	Virtual reality	Perception of reality	Mean Opinion Score (MOS) (Hu et al., 2021)
  8	Big data and Analytics	Volume	Amount of data collected (Ferrari et al., 2017)
  9	Integration systems	Reference Architecture	Architecture application level

There are four possible responses to each statement that have been defined for measuring the characteristics described above. Each response is associated with a certain number of points indicating the observed degree of compliance.

• Level 0 (L0): 0 points, characteristic $m$ is low in DMT i.

• Level 1 (L1): 1 point, characteristic $m$ is average in DMT i.

• Level 2 (L2): 2 points, characteristic $m$ is high in DMT i.

• Level 3 (L3): 3 points, characteristic $m$ is very high in DMT i.

Similarly, we proposed the analysis of the mutual-performance of DMTs based on the definition of interoperability considering four defining characteristics (i.e., $m=4$). These four characteristics have been previously used in Saturno et al. (2017) for analyzing interoperability; however, it must be noted that it is a general approach and can be tailored according to specific requirements:

• a. Infrastructure: Network infrastructure level. For example: connection modules, smart network.

• b. Standard architecture: Architecture implementation level. For example: Industrial Internet Reference Architecture (IIRA), Reference Architecture Model for Industry 4.0 (RAMI 4.0), ISA 95, etc.

• c. Software platform: Software implementation level. For example: SCADA, Communication between architectures levels, Remote communication between architecture levels.

• d. Technological upgradability: Update level. For example: update between current devices, update adding new devices.

Four possible responses to each statement can be defined to measure each of these characteristics ($L=4$). Each response is associated with a certain number of points:

• Level 0 (L0): 0 points, DMT $i$ does not share the characteristic $m$ with DMT j.

• Level 1 (L1): 1 point, DMT $i$ partially shares the characteristic $m$ with technology j.

• Level 2 (L2): 2 points, DMT $i$ almost completely shares the characteristic $m$ with DMT j.

• Level 3 (L3): 3 points, DMT $i$ completely shares the characteristic $m$ with DMT j.

The standard vector, $r_e´$, is made up of the values that describe the optimal performance of the process under evaluation, Equation 5. This vector represents the maximum possible number of points that can be obtained. In this case, the maximum is 3, thus,

${\mathit{r}}_{\mathit{e}}´=\left[\begin{array}{c}3\\ \begin{array}{c}3\\ ⋮\end{array}\\ 3\end{array}\right]$ (5)

Noting that $\cos \theta =\frac{\mathrm{}{r}_{eij}\bullet r_e´}{L{r}_{eij}L{r}_e´}$, and ${L{r}_e´}^{\mathrm{}}=\mathrm{}\sqrt{3^{2}m}$, we see that Equation 3 can be rewritten as Equation 6,

${\mathit{r}}_{\mathit{p}\mathit{i}\mathit{j}}=\frac{\mathrm{}{\mathit{r}}_{\mathit{e}\mathit{i}\mathit{j}}\bullet {\mathit{r}}_{\mathit{e}}´}{9\mathit{m}}$ (6)

3.3 Indicator analysis and monitoring phase

The values that the elements of $MpT$ can take vary between 0 and 1, where zero denotes the lowest level of performance and one the highest. $MpT$ depends on the range of values that $r_{pij}$ can achieve, which determines the possible vectors of$r_{eij}$.

The mutual-performance monitoring of interoperability offered by the performance indicator indicates the least robust relationships between DMTs, presenting an opportunity to improve technologies efficiency. Similarly, individual DMT evaluation enables the identification of strategic actions to enhance their performance levels.

The $MpT$ indicator has been designed to measure the performance of a set of technologies and identify areas for improvement in a production system, which can ultimately lead to greater efficiency and productivity. This type of indicator is a crucial tool for making data-driven decisions and improving overall performance in any industry.

4 Test case

In this section, we demonstrate how the proposed indicator is implemented. Considering a small and medium-sized enterprise manufacturing system for which the ISA95 standard has been identified as architecture; a DeviceNet network used to connect devices at the plant level; a monitoring and simulation process with online services that have the option to connect to the DeviceNet network. In addition to standard manufacturing devices such as milling machines, jointers, conveyors, and a manual station. The manufacturing system has an industrial robot in charge of high-precision tasks, adjustable production speed, and a DeviceNet communication module.

We identify the DMTs currently available in the process: Industrial Robots and Simulation. Therefore, $n=2$. Each one of the evaluation characteristics self-performance are identified. First, we identify the following evaluation characteristics: short cycle time for Industrial Robots, and representation level for Simulation.

The characteristics mentioned above are then graded using the four-point evaluation procedure defined in Section 3.2. Here we focus on the process used to obtain $r_{e11}$ related to the Industrial Robots performance.

• $e_1$ (L3) Short cycle time is very high in the Industrial Robots DMT due to the characteristics of the installed robot.

Similarly, we focus on the process used to obtain $r_{e22}$ associated with the Simulation performance.

• $e_1$ (L1) Representation level is average in the Simulation DMT due to simulation being only of the process.

Then, each one of the evaluation characteristics mutual-performance are identified: DeviceNet communication network for Infrastructure, ISA 95 for Standard architecture, and Platform online supervision for Software platform.

The characteristics mentioned above can then be graded using the four-point evaluation procedure, as defined in Section 3.2. Particularly, the procedure used to obtain the value of the parameter $r_{e21}$, which is related to the mutual-performance between Industrial Robots and Simulation, is shown below:

• $e_1$ (L3) Simulation DMT fully shares the DeviceNet network infrastructure with the Industrial Robots DMT. Both the simulation software and the industrial robot have a DeviceNet network communication module.

• $e_2$ (L2) Simulation DMT almost entirely shares the ISA 95 standard architecture with the Industrial Robots DMT. The architecture defined by the ISA95 standard does not directly mention any level of simulation. It can thus be integrated at the planning level.

• $e_3$ (L3) Simulation DMT fully shares a software platform with the Industrial Robots DMT. Both are compatible with this platform, and it is possible to communicate through the initially defined network infrastructure.

• $e_4$ (L1) Simulation DMT partially shares Technological upgradability with the Industrial Robots DMT. Due to the simulation tool having a finite number of inputs to simulate.

When all the elements $r_{eij}$ have been evaluated, the values of $r_{pij}$ can be obtained using Equation 6. Table 6 illustrates this process.

Table 6
Obtained values of $r_{eij}$ and $r_{pij}$.

$r_{eij}$	$r_{pij}$
$r_{e11}=\left(3\right)$	1
$r_{e12}=r_{e21}=(\mathrm{3,2},\mathrm{3,1})$	0.75
$r_{e22}=\left(1\right)$	0.33

Finally, using Equation 1 the total performance value can be obtained as in Equation 7

$MpT=\frac{\sum _{i=1}^2\sum _{j=1}^2{r}_{pij}}{3}:\mathrm{}i\le j=\frac{2.08}{3}\mathrm{}=0.69\mathrm{}$ (7)

This means that the performance of the DMTs in this manufacturing system is 69%.

5 Discussion

It has been found in the literature that performance can be evaluated from differing perspectives using various metrics. One of these perspectives concerns the concept of interoperability of systems; we focused on this characteristic in this paper due to the diversity of the DTMs analyzed and the impossibility of using other performance metrics.

When compared to other indicators such as maximal potential interoperability and minimal effective interoperability, the proposed indicator was found to exhibit similar behavior to that observed when measuring interoperability between two elements. However, the known indicators related to interoperability measure the interoperability of systems in computational terms, whereas in the proposed parameterization, the interoperability between two technologies is measured in terms of compatibility. It is worth noting that the indicator proposed in this work measures the interoperability between all possible pairs of DTMs that are present in the process rather than individually, as it is done in other methodologies.

One possible deficiency of the proposed parameterization is that the evaluation of the defined characteristics (capacity, digitization, and technological upgradability) may be subjective when assessing the self-performance of each individual technology.

We note that the proposed methodology evaluates the overall performance of the DTMs in a SMS and makes it possible to separate the $r_{pij}$ data group; this permits the evaluation of individual metrics related to the process. This kind of indicator is intended for use in strategies related to continuous improvement and the adoption of DTMs in a SMS.

6 Conclusion

Monitoring systems and processes using well-defined metrics can reveals the gap between reality and expectations; this is how performance metrics can serve as a feedback tool, helping identify and correct potential problems. In this work, performance measures were investigated within the field of SMS; we identified performance measures of technologies from different perspectives; as a business model, the semantic vision of communication and value chain. An indicator was defined as a measurement pattern resulting from the observations undertaken in this methodology.

This work presented a practical approach to the development of a quantifiable indicator that can be adjustable to a given environment and manufacturing process; this work is based on the premise that the performance of DTMs can be characterized by the interoperability of the various technologies present within the process.

From a theoretical approach, this indicator starts with the identification and analysis of existing indicators and measures to estimate the functionality of DMTs. In this analysis, the heterogeneous capacity characteristics for each DMT are highlighted, such as latency for IoT, volume for big data, and storage capacity for cloud computing.

Measuring the performance of DMTs through a quantifiable indicator can have a significant impact on understanding how the potential of these technological tools is being used and identifying ways to increase their usage. The information obtained through the indicator can guide future DMT implementation processes in production systems, leading to system improvement and innovation.

In practical terms, the proposed indicator could aid in the decision-making process regarding which DMTs to implement in a production system. This would be done by evaluating the capacity characteristics of each DMT and the interoperability characteristics of the DMTs as a whole, such as communication architecture and standard platform. Furthermore, using the indicator could improve cost evaluation during the DMT implementation process by analyzing the interoperability characteristics of different devices in the market, enabling the identification of compatible devices before installation and potentially reducing technological misusage.

In future research, the proposed performance indicator could be incorporated into a decision-making model for the implementation of DMT in manufacturing systems. Evaluating the performance of a DMT assembly can determine its suitability in a specific manufacturing system. In addition, it is possible to extend the use in case studies with data from real manufacturing systems, performing simulations of the manufacturing system to identify the behavior of the process and how it is affected by the implementation of DMTs.

• Financial support: None.
• How to cite: Tumbajoy Cardona, L. M., & Muñoz-Añasco, M. (2023). A novel technological performance measurement indicator: a smart manufacturing approach. Gestão & Produção, 30, e9622. https://doi.org/10.1590/1806-9649-2023v30e9622

References

• Ardito, L., Petruzzelli, A. M., Panniello, U., & Garavelli, A. C. (2018). Towards Industry 4.0: mapping digital technologies for supply chain management-marketing integration. Business Process Management Journal, 25(2), 323-346. http://dx.doi.org/10.1108/BPMJ-04-2017-0088
  » http://dx.doi.org/10.1108/BPMJ-04-2017-0088
• Bigliardi, B., Bottani, E., & Casella, G. (2020). Enabling technologies, application areas and impact of industry 4.0: a bibliographic analysis. Procedia Manufacturing, 42, 322-326. http://dx.doi.org/10.1016/j.promfg.2020.02.086
  » http://dx.doi.org/10.1016/j.promfg.2020.02.086
• Büchi, G., Cugno, M., & Castagnoli, R. (2020). Smart factory performance and Industry 4.0. Technological Forecasting and Social Change, 150, 119790. http://dx.doi.org/10.1016/j.techfore.2019.119790
  » http://dx.doi.org/10.1016/j.techfore.2019.119790
• Burns, T., Cosgrove, J., & Doyle, F. (2019). A review of interoperability standards for Industry 4.0. Procedia Manufacturing, 38, 646-653. http://dx.doi.org/10.1016/j.promfg.2020.01.083
  » http://dx.doi.org/10.1016/j.promfg.2020.01.083
• Cappa, F., Oriani, R., Peruffo, E., & McCarthy, I. (2021). Big data for creating and capturing value in the digitalized environment: unpacking the effects of volume, variety, and veracity on firm performance. Journal of Product Innovation Management, 38(1), 49-67. http://dx.doi.org/10.1111/jpim.12545
  » http://dx.doi.org/10.1111/jpim.12545
• Chen, D., Doumeingts, G., & Vernadat, F. (2008). Architectures for enterprise integration and interoperability: Past, present and future. Computers in Industry, 59(7), 647-659. http://dx.doi.org/10.1016/j.compind.2007.12.016
  » http://dx.doi.org/10.1016/j.compind.2007.12.016
• Cugno, M., Castagnoli, R., & Büchi, G. (2021). Openness to Industry 4.0 and performance: the impact of barriers and incentives. Technological Forecasting and Social Change, 168, 120756. http://dx.doi.org/10.1016/j.techfore.2021.120756
  » http://dx.doi.org/10.1016/j.techfore.2021.120756
• Dalenogare, L. S., Benitez, G. B., Ayala, N. F., & Frank, A. G. (2018). The expected contribution of Industry 4.0 technologies for industrial performance. International Journal of Production Economics, 204, 383-394. http://dx.doi.org/10.1016/j.ijpe.2018.08.019
  » http://dx.doi.org/10.1016/j.ijpe.2018.08.019
• Ferrari, P., Sisinni, E., Brandão, D., & Rocha, M. (2017, September). Evaluation of communication latency in industrial IoT applications. In 2017 IEEE International Workshop on Measurement and Networking (M&N) (pp. 1-6). Naples, Italy: IEEE. http://dx.doi.org/10.1109/IWMN.2017.8078359
  » http://dx.doi.org/10.1109/IWMN.2017.8078359
• Frederico, G. F., Garza-Reyes, J. A., Kumar, A., & Kumar, V. (2021). Performance measurement for supply chains in the Industry 4.0 era: a balanced scorecard approach. International Journal of Productivity and Performance Management, 70(4), 789-807. http://dx.doi.org/10.1108/IJPPM-08-2019-0400
  » http://dx.doi.org/10.1108/IJPPM-08-2019-0400
• Geraci, A. (1991). IEEE standard computer dictionary: compilation of IEEE standard computer glossaries. New York, USA: IEEE Press.
• Guédria, W., Naudet, Y., & Chen, D. (2015). Maturity model for enterprise interoperability. Enterprise Information Systems, 9(1), 1-28. http://dx.doi.org/10.1080/17517575.2013.805246
  » http://dx.doi.org/10.1080/17517575.2013.805246
• Hu, M., Luo, X., Chen, J., Lee, Y. C., Zhou, Y., & Wu, D. (2021). Virtual reality: a survey of enabling technologies and its applications in IoT. Journal of Network and Computer Applications, 178, 102970. http://dx.doi.org/10.1016/j.jnca.2020.102970
  » http://dx.doi.org/10.1016/j.jnca.2020.102970
• Ibarguen-Valverde, J. L., Angulo-López, J. E., Rodríguez-Salcedo, J., & Prías-Caicedo, O. (2017). Indicators of energetic performance: a path to sustainability.“A case study of a high-roasting industry of coffee. Dyna, 84(203), 184-191. http://dx.doi.org/10.15446/dyna.v84n203.65336
  » http://dx.doi.org/10.15446/dyna.v84n203.65336
• Kamble, S. S., Gunasekaran, A., Ghadge, A., & Raut, R. (2020). A performance measurement system for industry 4.0 enabled smart manufacturing system in SMMEs: a review and empirical investigation. International Journal of Production Economics, 229, 107853. http://dx.doi.org/10.1016/j.ijpe.2020.107853
  » http://dx.doi.org/10.1016/j.ijpe.2020.107853
• Klovienė, L., & Uosytė, I. (2019). Development of performance measurement system in the context of industry 4.0: a case study. Inžinerinė Ekonomika, 30(4), 472-482.
• Lu, Y. (2017). Industry 4.0: A survey on technologies, applications and open research issues. Journal of Industrial Information Integration, 6, 1-10. http://dx.doi.org/10.1016/j.jii.2017.04.005
  » http://dx.doi.org/10.1016/j.jii.2017.04.005
• Lucato, W. C., Pacchini, A. P. T., Facchini, F., & Mummolo, G. (2019). Model to evaluate the Industry 4.0 readiness degree in Industrial Companies. IFAC-PapersOnLine, 52(13), 1808-1813. http://dx.doi.org/10.1016/j.ifacol.2019.11.464
  » http://dx.doi.org/10.1016/j.ifacol.2019.11.464
• Mabkhot, M., Ferreira, P., Maffei, A., Podržaj, P., Mądziel, M., Antonelli, D., & Lohse, N. (2021). Mapping industry 4.0 enabling technologies into united nations sustainability development goals. Sustainability, 13(5), 2560. http://dx.doi.org/10.3390/su13052560
  » http://dx.doi.org/10.3390/su13052560
• Nara, E. O. B., da Costa, M. B., Baierle, I. C., Schaefer, J. L., Benitez, G. B., do Santos, L. M. A. L., & Benitez, L. B. (2021). Expected impact of industry 4.0 technologies on sustainable development: a study in the context of Brazil’s plastic industry. Sustainable Production and Consumption, 25, 102-122. http://dx.doi.org/10.1016/j.spc.2020.07.018
  » http://dx.doi.org/10.1016/j.spc.2020.07.018
• Neghab, A. P., Etienne, A., Kleiner, M., & Roucoules, L. (2015). Performance evaluation of collaboration in the design process: using interoperability measurement. Computers in Industry, 72, 14-26. http://dx.doi.org/10.1016/j.compind.2015.03.011
  » http://dx.doi.org/10.1016/j.compind.2015.03.011
• Pedone, G., & Mezgár, I. (2018). Model similarity evidence and interoperability affinity in cloud-ready Industry 4.0 technologies. Computers in Industry, 100, 278-286. http://dx.doi.org/10.1016/j.compind.2018.05.003
  » http://dx.doi.org/10.1016/j.compind.2018.05.003
• Rajan, A. P. (2013). Evolution of cloud storage as cloud computing infrastructure service. Retrieved in 2023, April 5, from https://arxiv.org/abs/1308.1303
  » https://arxiv.org/abs/1308.1303
• Rubio, J. E., Roman, R., & Lopez, J. (2018). Analysis of cybersecurity threats in industry 4.0: the case of intrusion detection. In Critical Information Infrastructures Security: 12th International Conference, CRITIS 2017 (pp. 119-130). Lucca, Italy: Springer International Publishing. http://dx.doi.org/10.1007/978-3-319-99843-5_11
  » http://dx.doi.org/10.1007/978-3-319-99843-5_11
• Saturno, M., Ramos, L. F. P., Polato, F., Deschamps, F., & Loures, E. D. F. R. (2017). Evaluation of interoperability between automation systems using multi-criteria methods. Procedia Manufacturing, 11, 1837-1845. http://dx.doi.org/10.1016/j.promfg.2017.07.321
  » http://dx.doi.org/10.1016/j.promfg.2017.07.321
• Silva, J. F. D., Silva, F. L. D., Silva, D. O. D., Rocha, L. A. O., & Ritter, Á. M. (2022). Decision making in the process of choosing and deploying industry 4.0 technologies. Gestão & Produção, 29, 29. http://dx.doi.org/10.1590/1806-9649-2022v29e163
  » http://dx.doi.org/10.1590/1806-9649-2022v29e163
• Society fof Automotive Engineers - SAE. (1999). J4000: Identification and measurement of best practice in implementation of lean operation Warrendale, PA: SAE.
• Sun, S., Zheng, X., Villalba-Díez, J., & Ordieres-Meré, J. (2020). Data handling in industry 4.0: interoperability based on distributed ledger technology. Sensors (Basel), 20(11), 3046. http://dx.doi.org/10.3390/s20113046 PMid:32471234.
  » http://dx.doi.org/10.3390/s20113046
• Wong, K. V., & Hernandez, A. (2012). A review of additive manufacturing. International Scholarly Research Notices, 2012, 208760.
• Wu, D. (2009). Measuring performance in small and medium enterprises in the information and communication technology industries (Doctoral dissertation). RMIT University, Australia.
• Yahia, E., Aubry, A., & Panetto, H. (2012). Formal measures for semantic interoperability assessment in cooperative enterprise information systems. Computers in Industry, 63(5), 443-457. http://dx.doi.org/10.1016/j.compind.2012.01.010
  » http://dx.doi.org/10.1016/j.compind.2012.01.010
• Zeid, A., Sundaram, S., Moghaddam, M., Kamarthi, S., & Marion, T. (2019). Interoperability in smart manufacturing: research challenges. Machines, 7(2), 21. http://dx.doi.org/10.3390/machines7020021
  » http://dx.doi.org/10.3390/machines7020021

Publication Dates

• Publication in this collection
  21 July 2023
• Date of issue
  2023

History

• Received
  05 Apr 2023
• Accepted
  18 Apr 2023