New Trends and Challenges in Condition Monitoring Strategies for Assessing the State-of-charge in Batteries

*Juan Jose Saucedo-Dorantes, David Alejandro Elvira-Ortiz, Carlos Gustavo Manriquez-Padilla, Arturo Yosimar Jaen-Cuellar and Angel Perez-Cruz*

#### **Abstract**

Condition monitoring strategies play an important key role to ensure the proper operation and/or working conditions in electrical, mechanical, and electronic systems; in this sense, condition monitoring methods are commonly implemented aiming to avoid undesired breakdowns and are also implemented to extend the useful life of the evaluated elements as much as possible. Therefore, the objective of this work is to report the new trends and challenges related to condition monitoring strategies for assessing the state-of-charge in batteries under the Industry 4.0 framework. Specifically, this work is focused on the analysis of those signal processing and artificial intelligence techniques that are implemented in experimental and model-based assessing approaches. With this work, important aspects may be highlighted as well as the conclusions and prospects may be included for the development trend of condition monitoring strategies to assess and ensure the state-of-charge in batteries.

**Keywords:** condition monitoring, state-of-charge, battery

#### **1. Introduction**

Condition monitoring strategies have been successfully implemented as a part of Condition-Based Maintenance (CMB) programs for several decades with the aim of preventing the occurrence of malfunction problems. Although CBM programs have been effectively implemented, in the last years, Industry 4.0 is changing the landscape in different sectors with the rise of the smart factory and the use of data, such changes have been possible through the digitization of value chains, where the aim is to improve the efficiency, sustainability, and flexibility of operations. These new trends present new tools to further maximize the value of the data that are collected during the equipment operation to coordinate tasks in a predictive environment (before a functional failure occurs). However, it is important to clearly define what information or data to collect will represent a meaningful value to the decision-making process. In this regard, it should be noted that the proper implementation/conversion to the Industry 4.0 may lead to numerous advantages to any process [1, 2]. Thus, **Figure 1** shows the most important benefits that may be reached by the implementation of Industry 4.0, the order of importance may differ according to the process and/or application where Industry 4.0 is implemented.

Thus, the most important profits that are taken into account focused on monitoring strategies applied to assess the condition of a specific system are described below:


Under this framework, the term Industry 4.0 can be interpreted as the hyperconnection, where all systems are connected between them and can send,

#### **Figure 1.**

*Most important advantages and benefits reached through the implementation of the Industry 4.0, where such advantages may be found in several research papers focused on the Industry 4.0 [1, 2].*

#### *New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

receive, and analyze data and are no longer a novelty. Thus, this new concept is currently used during the collection and monitoring of the control parameters of the equipment to optimize its operation. It is well known that the Industry 4.0 has profoundly changed industrial processes, in fact, in a constant optimization process, also, the Industry 4.0 would reduce energy and resource consumption while improving production. Accordingly, many problems that have been and are still faced by our planet are the product of industrialization, such as climate change, unsafe levels of air pollution, the depletion of resources, or the loss of biodiversity are some examples of the impact of our activity in the world [2, 3]. Also, the implementation of the Industry 4.0 has impacted different subject areas, and it should be noted that engineering and data science applications have been significantly benefited and other areas such as energy have not been widely studied, this statement is supported by the percentage of published papers related to Industry 4.0 for the different subjected areas as **Figure 2** depicts.

As stated, condition monitoring strategies have been extensively applied and its implementation as a CMB program has benefited the industry sector since major of the procedures are accomplished by electronic, electrical, and mechanical elements, where its combination leads to electromechanical systems [4–6]; moreover, it is worth noting that condition monitoring is also a very active area of research in aerospace and civil engineering where the objective also remains to ensure its functionality. In this regard, CBM programs may be implemented with different aims, for example, by analyzing the remaining useful life (RUL), it is possible to predict the occurrence of faults that may affect the functionality of the whole system in a near future, as well as the detection and isolation of faults that have been occurred and are present by analyzing the state-of-health (SOH), and the detection and identification of multiple and combined faults that may occur simultaneously. As stated, despite most of the condition monitoring strategies being developed under a particular framework, i.e., RUL and/or SOH, the principal aim remains to identify abnormal states and/or operations that tend to present deviations from an optimal condition or state of operation; therefore, the most appropriated way of implementing such condition monitoring practices will depend on the application or problem being addressed [7, 8].

#### **Figure 2.**

*Percentages of papers that have been published and focused on the Industry 4.0 for different subject areas.*

On the other side, the Industry 4.0 framework also aims to face pollution problems through the proposal of green solutions and by the implementation of renewable energy systems. In this context, environmental pollution, which is one of the most critical global problems affecting today's world, has attacked the attention of many scientists aiming to provide successful solutions. Certainly, it is well known that the world's pollution (measured in terms of air quality) is in general produced due to the effects associated to the global greenhouse gas emissions, where carbon dioxide (CO2) is the most dangerous gas produced by the use of fossil fuel and also produced in industrial processes, which has a concentration of about 65% only for the global greenhouse gas emissions; meanwhile, the remaining 35% of gases are composed by carbon dioxide, methane, nitrous oxide, among others. Therefore, cars, trucks, and/or industrial processes that are based on the use of fossil fuel are the main sources that contribute the environmental pollution, specifically, to the pollution of the air. In this sense, in the most recent decades, it has been noticed that electrification may be the key solution that can lead to the reduction of those high percentages of gas concentrations that increase the world's pollution and that endanger human health [9–11]. Accordingly, since electrification can be considered the most adequate solution to the reduction of environmental pollution, it may be understood as the reconversion of those traditional systems that are dependent on fossil fuel to new systems that only use electric drives. Hence, nowadays, new scientific and technological advances have made it possible to innovate as the readily technology is scalable; in this regard, the new trends are toward the manufacturing of electric vehicles if possible and/or hybrid vehicles to reduce the emission of polluting gases. Although the manufacturing of electric or hybrid vehicles has been promoted by technologically developed countries, some challenges must be faced; thereby, the energy storage and management are probably the most critical issues that are recently addressed. Certainly, the monitoring of the state-of-charge in batteries may be the key point that allows the characterization of the efficiency and/or autonomy in electric and hybrid vehicles [12–15].

In fact, the Industry 4.0 can be the solution to face actual problems and to overcome challenges that have not been addressed, thus, it should be highlighted that the Industry 4.0 is adaptable to a specific application. For example, for electromobility and electric vehicles, the most critical challenges are the range, charging time, and charging infrastructure. Consequently, most of the recent research has been focused on the condition assessment of the state-of-charge in batteries under the Industry 4.0 framework, which involves the general terms of automation, big data, cloud computing, autonomous Internet of Things (IoT), and data management. Moreover, the efficiency of electric vehicles is intuitively in terms of installed monitoring and diagnostic systems and depends on the number of available variables that can be acquired to assess the vehicle parameters. As illustrated below (**Figure 3**), under the Industry 4.0 framework, it is shown a general scheme where are presented different problems (challenges) to be solved under the Industry 4.0 framework.

Therefore, this work presents a systematic report related to the new trends and challenges that are associated with condition monitoring strategies used for assessing the state-of-charge in batteries under the Industry 4.0 framework. Precisely, in this work are presented those classic and significant techniques of analysis that have led to high-performance signal processing, as well as those artificial intelligence techniques that are implemented in experimental and model-based assessing approaches. Additionally, in this work are included the most important aspects that have to be theoretically considered whether a condition monitoring strategy is intended to be implemented for the assessment of the battery's condition.

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

**Figure 3.**

*Challenges to be faced under an Industry 4.0 framework presented in some published research works ([3]).*

#### **2. Theoretical background**

In this section, a summary of the most common battery technologies nowadays, as well as an overview of the main components and functions that must be accomplished by a BMS (Battery Management System) to guarantee the proper operation of any battery system, is presented.

#### **2.1 Different battery technologies**

Batteries are electrochemical devices that can receive and store energy to be used at a later moment. Although there are more energy-storage devices, batteries have gained popularity due to their capability of providing high power and energy efficiency at a relatively low cost with a long life cycle and a rapid response [13]. **Figure 4** illustrates the general construction of any type of battery. It is composed of two electrochemical cells that can turn chemical energy into electricity. Each cell consists of a positive electrode, or cathode, a negative electrode or anode, and an electrolyte that is commonly a fluid that allows the flow of the ions (i+) from one electrode to another. This way, the electric current flows outside, and it can be used to feed any load.

**Figure 4.** *Representation of the basic internal composition of a battery.*

#### **Figure 5.**

*Main technologies used in the internal composition of a battery.*

Due to the key role that batteries play in important emergent technologies such as electric vehicles and renewable generation sources, a big effort is put into the development of a wide variety of batteries with different characteristics. This situation is achieved by using different chemical elements and construction strategies resulting in a wide variety of battery technologies. Next, the main technologies used in batteries are shown in **Figure 5** and also addressed and briefly described. Although there exist several battery technologies, the ones that are presented in this section represent the most used in applications such as renewable generation and electric vehicles.

#### **2.2 Lead-acid batteries**

This is one of the oldest technologies used for the development of batteries. Therefore, the lead-acid technology for batteries is mature and widely spread. These types of batteries are characterized to be low cost and very reliable; thus, it is a proficient technology for applications that require an uninterrupted power supply with high quality [16]. In lead-acid batteries, the positive electrode (cathode) is composed of lead dioxide (PbO2) and a negative electrode (anode) of metallic lead (Pb). Additionally, they consider a sulfuric acid solution (H2SO4) as an electrolyte. At the anode, the Pb reacts with a sulfate ion to obtain lead sulfate (PbSO4) as shown in Eq. (1):

$$\text{Pb} + \text{SO}\_4^{2-\text{-}2e^{-+}} \text{PtSO}\_4 \tag{1}$$

It is observed in Eq. (1) that two electrons are released at the lead electrode conferring it the negative charge. On its part, the PbO2 of the cathode reacts with the electrolyte yielding PbSO4 and water according to Eq. (2):

$$PbO\_2 + 4H^{++\text{SO}\_4^{2-+2r}} \tag{2}$$

Finally, the total reaction can be expressed with Eq. (3):

$$Pb + PbO\_2 + 2H\_2SO\_4 \Leftrightarrow 2PbO\_2 + H\_2O\tag{3}$$

Eq. (3) shows that the reaction is reversible allowing the battery to be repeatedly charged and discharged. Commonly, a lead-acid battery is composed of several pairs of electrodes that are placed in separate compartments. Each one of these compartments is called a cell. The negative electrode of each cell is connected with the positive electrode of the next cell leaving free the cathode of the first cell and the anode of the last cell, and the result is a battery whose voltage is the sum of the individual voltages of each cell. It is important to mention that each cell of a lead-acid battery handles

typical voltages of *E*0≈2*:*048*V* and typical configurations consider three, six, and 12 cells for a complete battery [17].

#### **2.3 Lithium-ion batteries**

This technology is more recent, it was first introduced in the 1990s, but it is recently widely used in electronic devices, smart grids, and electric vehicles [18]. Lithium-ion batteries have gained a lot of popularity because they are the main type of storage system used by all mobile devices as smartphones and tablets. Notwithstanding, they are also highly used in electric vehicle applications as well as in grids containing renewable energy generation. These types of batteries can provide a higher energy density than most of the other available technologies since they operate at voltages around 4 V per cell, while other systems operate at 2 V per cell [19]. Lithium-ion batteries use anodes and cathodes based on insertion-compound materials. In the case of the anode, a carbonaceous material [20] is required; therefore, the preferred compound is graphite formed by one lithium atom per six carbon atoms *LiC*6. On its part, for the construction of the cathode, it is used a metal oxide and the available materials are mainly three: the layered *lim O*<sup>2</sup> (*M* ¼ *Mn*,*Co* and) [21], spinel *lim n*2*O*<sup>4</sup> [22] and olivine *LiFePO*<sup>4</sup> [23]. Additionally, these batteries use water-free organic liquid electrolytes such as *LiPF*<sup>6</sup> salt dissolved in a mixture of ethylene carbonate (EC). In fact, the use of this type of electrolytes is the reason why lithium-ion batteries are capable of handling 4 V per cell. Finally, this technology incorporates a separator that allows only the lithium ions to flow from one side to another in the battery. During the charging process, some of the lithium ions leave the positive electrode and flow through the electrolyte to the negative electrode. When the lithium ions reach the graphite, they are inserted between the atomic layers of that material, where they recombine with the electrons, leaving the lithium deposited there. When the ions stop flowing, the battery is completely charged. On the other hand, when the battery is discharging, the lithium ions flow back through the electrolyte from the graphite anode to the cathode.

#### **2.4 Niquel-Cadmium (Ni-Cd) batteries**

This is another technology that has been on the market for many years. These batteries use a cathode of nickel hydroxide and an anode of cadmium hydroxide. In this case, the electrolyte is an alkaline substance and the charge and discharge process can be described by Eq. (4):

$$\text{2NiOOH} + \text{2H}\_2\text{O} + \text{Cd} \Leftrightarrow \text{2} \left(\text{OH}\right)\_2 + \text{Cd} (\text{OH})\_2\text{E}^0 = \text{1.29V} \tag{4}$$

where *E*<sup>0</sup> represents the voltage of a single Ni-Cd cell.

These batteries are famous because they can operate at a wide temperature range and they are easy to maintain. However, their manufacturing is complex, making these batteries expensive. But probably the biggest issue related to this technology relays in the fact that it contains cadmium, which is a heavy metal well known for its toxicity [24].

#### **2.5 Nickel-metal hydride (Ni-MH) batteries**

This type of battery operates in a way similar to the Ni-Cd one, and this technology is preferred in hybrid electric vehicles (HEV) due to its high-power density and

tolerance to overcharge/over-discharge processes [25]. In this case, the Ni-MH technology considers that the active material of the positive terminal is nickel oxyhydroxide (NiOOH) and the active material that constitutes the negative terminal is hydrogen in the form of a metal hydride, which allows the hydrogen produced during the charging process to be stored and released during the discharge process [24]. This type of electrode is responsible for providing greater capacity per volume unit compared to a Ni-Cd battery. A common metal alloy (M) in Ni-MH batteries is an alloy made up of a mixture of zirconium or titanium hydride with another metal such as nickel, cobalt, or aluminum. And the electrolyte in these batteries is mainly made up of potassium hydroxide, which also makes it a type of alkaline battery. The chemical reaction that occurs inside these batteries is described by Eq. (5):

$$\text{MH} + \text{NiOOH} \Leftrightarrow \text{M} + (\text{OH})\_2 \\ E^0 = \text{1.35V} \tag{5}$$

Again, as in the Ni-Cd battery, the term *E*<sup>0</sup> refers to the voltage of a single battery cell. Compared to its cadmium counterpart, this technology is less harmful to the environment. However, its disposal at the end of its lifecycle must be cautious since it still uses corrosive salts.

#### **2.6 Flow batteries**

This is a technology that considers systems of two connected tanks, both containing electrolytic liquids: one with a positively charged cathode and the other with a negatively charged anode. Electricity passes from one electrolytic liquid to another through a membrane between the tanks. There are two main types of commercial flow batteries: Vanadium redox batteries (VRB) and Zinc-Bromine (Zn-Br). The VRB uses sulfuric acid containing V5+/V4+ and V3+/V2+ redox couples as the positive and negative half-cell electrolytes. The reaction that describes the charge/ discharge process is described by Eq. (6):

$$\text{VO}\_2^{+ + 2H^{++V^{2+\bullet}VO^{2++H\_2O+V^{3+E^0}}}}\tag{6}$$

In the case of the Zn-Br battery, its operation principle may be defined by Eq. (7) as follow:

$$Zn + Br\_3^{-\Leftrightarrow ZnBr\_2 + Br^{-E^0 = 1.8\\$}}\tag{7}$$

Despite this technology having technical advantages, such as potentially separable liquid reservoirs and almost unlimited longevity over most conventional rechargeable batteries, current implementations are relatively less powerful and require more sophisticated electronics [26].

#### **2.7 Battery management system (BMS)**

To ensure the safe and reliable operation of any battery, it is important to keep the operating conditions within a range known as the safe operating area (SOA). **Figure 6** shows a diagram of the different operating conditions that can be observed in a battery.

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

#### **Figure 6.**

*Common diagram of the SOA for a battery that depicts different states during the charging procedure.*

The SOA considers that the voltage and temperature of the battery must not exceed or fall below very specific values. These values are different for any battery, and they must be specified by the manufacturer. However, they can be addressed as the maximum voltage (Vmax), minimum voltage (Vmin), maximum temperature (Tmax), and minimum temperature (Tmin). If Vmax is exceeded, the battery presents an overcharge; when the battery reaches voltages lower than Vmin, it has reached the overdischarge state; for the case of a temperature superior to Tmax, the battery shows an over-temperature state; and finally, if the temperature is lower than Tmin and under temperature condition is achieved. All these last four conditions must be avoided because they can lead to severe damage to the battery, and they can result in safety risks for the final users. On the other hand and as observed, a single-cell battery delivers a small voltage value; therefore, a common battery is confirmed by a series of cells that can deliver a higher voltage together. This situation supposes some challenges, for instance, it is important to guarantee that all the cells perform the charge/discharge operations at the same rate so the complete system is balanced. Additionally, it is necessary to regulate the amount of current that is delivered or received by each cell to avoid damages associated with a misuse of the batteries. In this sense, the battery management system (BMS) plays an important role to keep the battery pack operating safely, reliably, and efficiently [27]. The BMS can be described as a black box model as depicted in **Figure 7**. To accomplish its purpose, the BMS takes the temperature (T), voltage (V), and current (I) of the battery pack and use them to perform different algorithms for controlling the operational conditions of the battery to extend its life and guarantee a safe operation. Additionally, the BMS provides an accurate estimation of the

**Figure 7.** *Black box diagram of a BMS.*

State of Charge (SOC) and the State of Health (SOH) of the battery pack, and based on all these parameters, the BMS can deliver information regarding the status of the battery pack and detect if a fault condition is present in the storage system.

The SOC is a parameter that can be defined both: in terms of the battery capacity or energy consumption. In renewable energy generation and EVs applications, it is more common to define the SOC as the ratio of the remaining energy (*Er*) and the total energy (*ET*) of the battery pack, and it is expressed as a percentage. The mathematical definition can be observed in Eq. (8):

$$\text{SOC}\_{E} = \frac{E\_r}{E\_T} \times \mathbf{100} \tag{8}$$

On the other hand, the SOH can be defined as the current total capacity that can be performed by the battery compared with the total capacity of the battery at the beginning of its life. As in the case of SOC, this parameter is defined as a percentage, and it is mathematically defined by Eq. (9):

$$\text{SOH} = \frac{\text{C}\_T}{\text{C}\_{\text{BOL}}} \times \mathbf{100} \tag{9}$$

Where *CT* is today's total capacity, and *CBOL* is the capacity at the beginning of life. In the following section, the most common approaches for the implementation of BMS are presented. A more detailed diagram of how a BMS is composed can be observed in **Figure 8**.

#### **3. Approaches and technologies for the implementation of BMS**

In order to ensure the reliable and safe operation of electric vehicles, the accurate application of fault diagnosis schemes over the battery system is mandatory, in which the most relevant elements are composed of the sensors, the systems and components, and the actuators. Hence, different methods have been reported in the literature to

**Figure 8.** *Detailed diagram of a BMS showing the main and minimal components.*

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

implement different tasks that must be performed by a BMS. In general, all the developed methodologies can be classified into two groups: experimental approaches and model-based approaches. The first one considers that several tests must be performed several times to obtain the information regarding the condition of the battery pack, whereas the second one considers that there exists a series of parameters that describe the battery state, and they focus on finding such parameters [28].

#### **3.1 Experimental approaches**

First, it is important to mention that most of the BMSs focus on performing an accurate estimation of the consumed capacity. If this task is correctly performed, it is possible to estimate the SOC and the SOH of a battery pack accurately and reliably. Therefore, most of the works reported in the literature pay special attention to this matter. The most common solution for this issue is the method known as the Coulomb counting [29], which considers the used capacity as the area behind the curve defined by the discharging current over time. When this value is subtracted from the total capacity, it is possible to know the remaining capacity in the battery pack. This method can be mathematically described by Eq. (10):

$$\text{SOC}(t) = \text{SOC}(t\_0) - \frac{1}{C\_T} \int\_{t\_0}^{t} i(t)dt \tag{10}$$

Where *SOC t*ð Þ is the current SOC; *SOC t*ð Þ<sup>0</sup> is the initial SOC that is commonly considered as 100%; *CT* is the nominal capacity of the battery; and *i t*ð Þ is the discharge current extracted from the battery. Accordingly, the implementation of the aforementioned method can be experimentally performed by means of following the flowchart of **Figure 9**, where the SOC starts by carrying out the real-time data acquisition,

**Figure 9.**

*General flowchart that may be followed to apply the assessment and achieve the SOC in batteries through experimental-based models.*

then in a second step, the model parameter identification is achieved, and subsequently, the SOC is estimated in terms of the collected data by applying Eq. (10). Despite this approach being preferred, the implementation of this method has a technical drawback that is related to the use of a sensor for the current measuring. The sensors used for this purpose are usually shunted resistors or Hall effect transducers. These types of instruments introduce an error in the estimation due to the drift. Therefore, the Coulomb counting must be complemented with another technique to compensate for this effect. In this sense, the use of the open circuit voltage (OCV) [30] allows the analysis of energy changes in the electrodes of the battery, and therefore, there exists a direct relationship between the OCV and the SOC of the battery. In the experimental approaches, the OCV is sometimes obtained from the specifications given by the manufacturers. Notwithstanding, the information given by the manufacturer is not as detailed as required to perform an accurate estimation of the SOC.

Thus, the use of methodologies such as the low current test and the incremental current test results is helpful to solve this issue. The low current test considers that the battery must be initially charged using a constant current rate of 1C, considering that 1C means that the complete energy of the battery is taken in intervals of 1 hour. Next, the battery is discharged at a constant rate of C/20, and then, recharging the battery uses this same last rate (C/20). In this test, the voltage between electrodes is constantly measured and recorded during the entire test. This process is repeated several times and the average of all the tests is taken as the OCV [31]. On its part, the incremental current test considers that the battery must be completely charged to represent a 100% SOC. Then, a negative pulse current relaxation is used to discharge the battery and the voltage between terminals is measured every 10% of the discharge. When the battery has been completely discharged, the process is applied in reverse, i.e., the battery is charged with a positive pulse current and the voltage between terminals is measured every 10% of charge. This process must be repeated several times and the OCV curve is obtained by linear interpolation [32]. These techniques provide a good approximation of the OCV that can be easily related to the SOC and SOH of the battery. However, they are considered aggressive tests that may cause damage to the batteries; moreover, they suffer from the polarization effect due to the constant current discharge. In this sense, another widely spread methodology for the estimation of the SOC and SOH in batteries is the use of the impedance measurement [28]. This method takes advantage of the fact that the internal resistance of a battery determines its power capacity. Thus, the internal resistance is calculated using Ohm's law considering the voltage drop over the electrodes when a current is demanded. The so far mentioned algorithms calculate the SOC and SOH directly using their definition stated by Eqs. (8) and (9), respectively. But there is also possible to perform an indirect estimation of the SOC and SOH of the battery using the incremental capacity analysis (ICA) and the differential voltage analysis (DVA). These techniques allow to find a curve coming from the gradient of charged/discharged capacity concerning the cell voltage according to Eq. (11) and another one derived as the gradient of the cell voltage for the battery capacity as shown by Eq. (12):

$$IC = \frac{\Delta Q}{\Delta V} = \frac{dQ}{dV} \tag{11}$$

$$DV = \frac{\Delta V}{\Delta Q} = \frac{dV}{dQ} \tag{12}$$

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

Where *IC* is the incremental capacity feature, *DV* is the differential voltage feature, Q is the cell capacity, and V is the cell voltage. These curves present peaks at specific values and locations, and as the battery degrades, the amplitude and location of the peaks change. This situation is used for determining the SOH of the battery accurately and reliably [33].

#### **3.2 Model-based approaches**

The experimental methods provide a good tool for BMS to perform its task. However, they present the disadvantage of requiring a repeated number of tests to deliver their results. Therefore, they are not recommendable for an online implementation since BMS is expected to monitor the condition of the battery in real time, the model-based solutions seem to be a more appropriate tool. These approaches consider that certain parameters as the capacity and resistance of the battery can be calculated based on a mathematical model. In this regard, batteries have been described using an equivalent circuit model (ECM). This methodology states that a battery can be described by three main parameters: resistance, inductance, and capacitance. By finding these parameters, it is possible to determine the SOC and SOH of the battery in the function of the variations in the nominal values of the parameters. Here, the Kalman filter algorithm turns out to be particularly good for the estimation of the parameters of the battery [34]. This model delivers good results; however, it does not consider what happens inside the battery and may lead to errors if parameters such as the temperature are not taken into account. To overcome this situation, some works propose the development of an electrochemical model (EM). This way, the operation principle of the battery and its dynamic are modeled getting a more reliable and accurate representation. But this increment in the accuracy is not for free, the complexity of the model and the number of parameters increase, making the proper parameter identification more difficult. For instance, in [35], the use of different types of parameters: geometric, transport, kinetic, and concentration is proposed. The result is a mathematical model that comprises a total of 26 parameters. With this model, the SOC and the SOH are calculated considering not only the electric performance but also the composition and internal reactions of the battery. Thereby, SOC and SOH are commonly proposed and/or designed as a condition monitoring scheme that accomplishes stages such as data acquisition or monitoring, feature extraction or signal processing, and the fault diagnosis task in which the fault detection, isolation, and estimation are executed. **Figure 10** shows the flowchart of a condition monitoring scheme used for the implementation of a SOC.

#### **Figure 10.** *Flowchart of a condition monitoring-based scheme used for performing the fault diagnosis in battery systems.*

Another approach that is gaining popularity is the use of data-driven methodologies based on machine learning. These methodologies model the battery as a black box and develop software that uses example data or past experiences for learning how to solve a problem [28]. Here, support vector machines (SVMs) have proven to be effective for the estimation of parameters such as the SOC of a battery. For instance, in [36], the authors use voltage, current, and cell temperature as inputs of an SVM and with a least square algorithm, they estimate the SOC based on the behavior of the input parameters. A similar implementation is carried out in [37], the difference is that in this work the use of an SVM and the least square approximation are replaced by a deep neural network that estimates the battery condition using as inputs the voltage, current, and temperature. On their part, the authors in [38] propose the use of an ECM, and they use a fuzzy logic system to perform the parameter estimation. At this point, it is important to mention that all the machine learning approaches can be appreciated as a hybrid of the experimental and the model-based methods because they require a series of previous experiments before being implemented; additionally, they use a mathematical model but the model does not describe the system but the conditions required for the system to meet a specific state.

#### **4. New applications and trends in BMS devices**

According to recent research works and studies, it has been determined that the BMS (Battery Management System) is the key element in applications such as electric vehicles and renewable energy, this assert is due to the BMS being responsible for managing the energy consumption totally or partially, and it is also responsible for managing the energy storage. Although there exist different types of BMS that allow achieving an effective energy exploitation, nowadays new trends are emerging aiming to contribute to the development of innovative solutions. In this regard, the trend of new research will continue to consider a general diagnostic framework, and these will be based on the flowchart of **Figure 11** as a common base, where the data monitoring, data processing, data analysis, and diagnosis comprise the four general steps.

Accordingly, regarding the *Data monitoring* step, the most accepted approaches are those that perform the assessment by means of experimental and/or model-based implementations, which are also known as data-driven approaches. Despite these proposals differing whether experimental data and/or simulated data are used, in both cases may exist similar aspects that are taken into account and that lead to new proposals. In case that data are acquired through experimental tests, the monitoring procedure consists of recording physical magnitudes such as voltage levels, current consumption, and reached temperature; in fact, these signals are commonly acquired for the whole battery

**Figure 11.** *General flowchart that may be followed to perform the assessment of batteries.* *New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

bank and are also individually acquired for each cell [39]. On the other side, equivalent circuit models are considered into account as the theoretical models when the data used are generated through simulation procedures, where the battery dynamics remain the most important aspect to be considered during the simulation [40].

Subsequently, the *Data processing* step may probably represent the most important stage since all the acquired data are processed with specific techniques, in this sense, the data processing may consider the simplest signal processing procedures such as the data normalization, data sub-sampling, data organization and may also consider the most complex signal processing procedures such as those techniques based on time domain, frequency domain, and time-frequency analysis [41]. The main objective of the Data processing step relies on the characterization and modeling of the acquired data, therefore, the processing of each acquired signal is performed in order to achieve a specific task, for example, the voltage signal may be processed aiming to give the current percentage or level of charge of the bank battery, the current signals are used to estimate the energy that may be supplied to all cells of the bank battery during the charging process and/or to estimate the energy consumption during the discharge procedure; and the temperature signals are taken into account as an additional variable that is implemented in most of the state-of-health monitor approaches to take care of the current state of the battery bank and to extend its useful life as much as possible [42].

Afterward, the *Data Analysis* and *Diagnosis* steps are commonly implemented as a part of the process that leads achieving the state estimation of the bank battery, as well as the remaining useful life, the level of charge, or in general is implemented to provide the SOH (state-of-health). Commonly, the *Data Analysis* stage includes Machine Learning techniques to process the available data [43], whereas the *Diagnosis* stage comprises intelligent algorithms to perform the automatic assessment task, in this regard, the most used techniques and algorithms are dimensionality reduction and/or feature extraction techniques, Support Vector Machines (SVM), Neural Networks such as Recurrent Neural Networks (RNN) and Fed-Forward Neural Networks (FNN), as well as regression models that may be based on Fuzzy algorithms; additionally, the use of genetic algorithms (GI) as a part of the assessing structures when the optimization of parameters is required [44]. On the other hand, it should be mentioned that for both stages, *Data analysis* and *Diagnosis*, most of the proposed approaches compute numerical values such as the Maximum Absolute Error (MAE), the Root Mean Square (RMSE), the Mean Square Error (MSE), and the goodness-of-fit R2, where these values are used as a quantitative measurement that depicts the effectiveness of the designed approaches [45]. An important aspect that must be also highlighted for the *Data analysis* stage is the estimation of the most representative set of features that allows a high-performance characterization of the processed signals. Finally, the use of Neural Networks is preferred in most of the designed predictors or SOH approaches due to their versatility and the low computational burden for their implementation in real-time applications. Thus, the selection of an appropriate signal processing technique, the use of Machine Learning techniques, and the implementation of Artificial Intelligence may represent the most important aspects to be considered during the proposal of novel strategies applied to assess the state-of-charge in batteries for multiple applications.

#### **5. Conclusions**

Modern society is undergoing an important transition toward new forms of transportation and energy generation that are sustainable and that allow reducing the

emission of gasses that cause the greenhouse effect and global warming. In this sense, batteries play an important role because they allow energy storage with high power and energy efficiency at a relatively low cost. However, to ensure their proper operation and to extend their lifecycle as much as possible, the use of a BMS is mandatory. BMS allows the battery pack to perform its task safely and reliably by estimating parameters that provide information regarding the condition of the batteries. Several methodologies have been developed to allow the BMS to fulfill its task reliably and accurately. The experimental approaches can provide an estimation of the battery status using a simple but effective method. However, they require the implementation of several tests to properly work becoming these techniques suitable mainly for offline implementations. On the other hand, the model-based approaches can perform the same task that the experimental techniques robustly and reliably can be implemented for online condition monitoring, at the cost of higher complexity. Finally, the machine learning techniques provide a hybrid between the experimental and the model-based methodologies that uses artificial intelligence techniques for identifying the condition of the batteries based on the behavior of some inputs that are commonly the electric parameters of the battery pack. These implementations require a set of experiments to be performed before they can be implemented; however, once they have been properly trained, they can operate in online systems. All the methodologies used for BMS deliver accurate and reliable results, and this work aims to be a tool for the readers to know different options so they can select the one that better fits their needs.

### **Acknowledgements**

This work has been partially supported by the FONDEC-UAQ2022.

### **Conflict of interest**

The authors declare no conflict of interest.

### **Author details**

Juan Jose Saucedo-Dorantes, David Alejandro Elvira-Ortiz, Carlos Gustavo Manriquez-Padilla, Arturo Yosimar Jaen-Cuellar and Angel Perez-Cruz\* Engineering Faculty, Autonomous University of Queretaro, San Juan del Rio, Mexico

\*Address all correspondence to: angel.perez@uaq.mx

© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

### **References**

[1] Athanasopoulou L, Bikas H, Papacharalampopoulos A, Stavropoulos P, Chryssolouris G. An industry 4.0 approach to electric vehicles. International Journal of Computer Integrated Manufacturing. 2022;**1**:1-25. DOI: 10.1080/ 0951192X.2022.2081363

[2] Bongomin O, Yemane A, Kembabazi B, Malanda C, Mwape MC, Mpofu NS, et al. Industry 4.0 disruption and its neologisms in major industrial sectors: A state of the art. Journal of Engineering. 2020;**2020**:1-25. DOI: 10.1155/2020/8090521

[3] Mastinu GRM, Cadini F, Gobbi M. Industry 4.0 and automotive 4.0: Challenges and opportunities for designing new vehicle components for automated and/or electric vehicles, SAE Technical Paper 2019-01-0504. 2019. DOI: 10.4271/ 2019-01-0504

[4] Catelani M, Ciani L, Fantacci R, Patrizi G, Picano B. Remaining useful life estimation for prognostics of lithium-ion batteries based on recurrent neural network. IEEE Transactions on Instrumentation and Measurement. 2021. Art no. 3524611;**70**:1-11. DOI: 10.1109/TIM.2021.3111009

[5] Iftikhar A, Lim MH, Ahmed MA, Syed AI, Leong MS. Scopes, challenges and approaches of energy harvesting for wireless sensor nodes in machine condition monitoring systems: A review. Measurement. 2021;**183**:1-13

[6] Qian L, Jun W, Rui X, Weixiang S, Hongwen H. Towards a smarter battery management system: A critical review on optimal charging methods of lithium ion batteries. Energy. 2019;**183**: 220-234

[7] Bektas O, Jones JA, Sankararaman S, et al. A neural network filtering approach for similarity-based remaining useful life estimation. International Journal of Advanced Manufacturing Technology. 2019;**101**:87-103. DOI: 10.1007/s00170-018-2874-0

[8] Zunya S, Abdallah C. A dual-LSTM framework combining change point detection and remaining useful life prediction. Reliability Engineering & System Safety. 2021;**205**:107257

[9] Zheng Y, He X, Wang H, et al. Well-to-wheels greenhouse gas and air pollutant emissions from battery electric vehicles in China. Mitigation and Adaptation Strategies for Global Change. 2020;**25**:355-370. DOI: 10.1007/ s11027-019-09890-5

[10] Ning Z, Huang J, Wang X, Rodrigues JJPC, Guo L. Mobile edge computing-enabled internet of vehicles: Toward energy-efficient scheduling. IEEE Network. 2019;**33**(5): 198-205. DOI: 10.1109/MNET.2019. 1800309

[11] Khan F, Ali Y, Khan AU. Sustainable hybrid electric vehicle selection in the context of a developing country. Air Quality, Atmosphere and Health. 2020; **13**:489-499. DOI: 10.1007/s11869-020- 00812-y

[12] Monteiro V, Afonso JA, Ferreira JC, Afonso JL. Vehicle electrification: New challenges and opportunities for smart grids. Energies. 2019;**12**:118. DOI: 10.3390/en12010118

[13] Chen H, Chen S, Jiang S. Optimal game-based energy management with renewable energy for secure electric vehicles charging in internet of things. Mathematical Problems in Engineering. 2021, Article ID 6078149, 2021:19. DOI: 10.1155/2021/6078149

[14] Gayathri MN. A smart bidirectional power interface between smart grid and electric vehicle. In: Vinoth Kumar B, Sivakumar P, Rajan Singaravel M, Vijayakumar K, editors. Intelligent Paradigms for Smart Grid and Renewable Energy Systems. Algorithms for Intelligent Systems. Singapore: Springer; 2021. DOI: 10.1007/978-981- 15-9968-2\_5

[15] Anna N, Cheng C, Bin l, Andrew B, Matthew S. Fully electrified land transport in 100% renewable electricity networks dominated by variable generation. Renewable Energy. 2022;**182**: 562-577

[16] Zou C, Zhang L, Hu X, Wang Z, Wik T, Pecht M. A review of fractionalorder techniques applied to lithium-ion batteries, lead-acid batteries, and supercapacitors. Journal of Power Sources. 2018;**390**:286-296

[17] Saminathan RK. Lead Acid Battery. Attacking Sulphate Passivation and Cyclability Problems. Munich: GRIN Publishing; 2017

[18] Xiong R, Zhang Y, Wang J, He H, Peng S, Pecht M. Lithium-ion battery health prognosis based on a real battery management system used in electric vehicles. IEEE Transactions on Vehicular Technology. 2018;**68**(5):4110-4121

[19] Manthiram A. An outlook on lithium ion battery technology. ACS Central Science. 2017;**3**(10):1063-1069

[20] Xiong R, Pan Y, Shen W, Li H, Sun F. Lithium-ion battery aging mechanisms and diagnosis method for automotive applications: Recent advances and perspectives. Renewable and Sustainable Energy Reviews. 2020; **131**:110048

[21] Luo D, Li G, Fu C, Zheng J, Fan J, Li Q, et al. LiMO2 (M= Mn, Co, Ni) hexagonal sheets with (101) facets for ultrafast charging–discharging lithium ion batteries. Journal of Power Sources. 2015;**276**:238-246

[22] Zhang S, Deng W, Momen R, Yin S, Chen J, Massoudi A, et al. Element substitution of a spinel LiMn 2 O 4 cathode. Journal of Materials Chemistry A. 2021;**9**(38):21532-21550

[23] Yu Z, Jiang L. Olivine LiFePO4 nanocrystals grown on nitrogen-doped graphene sheets as high-rate cathode for lithium-ion batteries. Solid State Ionics. 2018;**325**:12-16

[24] Fan X, Liu B, Liu J, Ding J, Han X, Deng Y, et al. Battery technologies for grid-level large-scale electrical energy storage. Transactions of Tianjin University. 2020;**26**(2):92-103

[25] Zhou W, Zhu D, Liu K, Li J, Wu C, Chen Y. Long-life Ni-MH batteries with high-power delivery at lower temperatures: Coordination of lowtemperature and high-power delivery with cycling life of low-Al AB5-type hydrogen storage alloys. International Journal of Hydrogen Energy. 2018; **43**(46):21464-21477

[26] Lourenssen K, Williams J, Ahmadpour F, Clemmer R, Tasnim S. Vanadium redox flow batteries: A comprehensive review. Journal of Energy Storage. 2019;**25**:100844

[27] Rahimi-Eichi H, Ojha U, Baronti F, Chow MY. Battery management system: An overview of its application in the smart grid and electric vehicles. IEEE Industrial Electronics Magazine. 2013; **7**(2):4-16

*New Trends and Challenges in Condition Monitoring Strategies for Assessing… DOI: http://dx.doi.org/10.5772/intechopen.109062*

[28] Xiong R, Li L, Tian J. Towards a smarter battery management system: A critical review on battery state of health monitoring methods. Journal of Power Sources. 2018;**405**:18-29

[29] Liu K, Li K, Peng Q, Zhang C. A brief review on key technologies in the battery management system of electric vehicles. Frontiers of Mechanical Engineering. 2019;**14**(1):47-64

[30] Zhang R, Xia B, Li B, Cao L, Lai Y, Zheng W, et al. A study on the open circuit voltage and state of charge characterization of high capacity lithium-ion battery under different temperature. Energies. 2018; **11**(9):2408

[31] Lipu MH, Hannan MA, Hussain A, Hoque MM, Ker PJ, Saad MM, et al. A review of state of health and remaining useful life estimation methods for lithium-ion battery in electric vehicles: Challenges and recommendations. Journal of Cleaner Production. 2018;**205**: 115-133

[32] Zheng F, Xing Y, Jiang J, Sun B, Kim J, Pecht M. Influence of different open circuit voltage tests on state of charge online estimation for lithium-ion batteries. Applied Energy. 2016;**183**: 513-525

[33] Zheng L, Zhu J, Lu DDC, Wang G, He T. Incremental capacity analysis and differential voltage analysis based state of charge and capacity estimation for lithium-ion batteries. Energy. 2018;**150**: 759-769

[34] Park J, Lee M, Kim G, Park S, Kim J. Integrated approach based on dual extended Kalman filter and multivariate autoregressive model for predicting battery capacity using health indicator and SOC/SOH. Energies. 2020;**13**(9): 2138

[35] Li W, Cao D, Jöst D, Ringbeck F, Kuipers M, Frie F, et al. Parameter sensitivity analysis of electrochemical model-based battery management systems for lithium-ion batteries. Applied Energy. 2020;**269**:115104

[36] Li J, Ye M, Meng W, Xu X, Jiao S. A novel state of charge approach of lithium ion battery using least squares support vector machine. IEEE Access. 2020;**8**: 195398-195410

[37] How DN, Hannan MA, Lipu MSH, Sahari KS, Ker PJ, Muttaqi KM. State-ofcharge estimation of li-ion battery in electric vehicles: A deep neural network approach. IEEE Transactions on Industry Applications. 2020;**56**(5): 5565-5574

[38] Zheng W, Xia B, Wang W, Lai Y, Wang M, Wang H. State of charge estimation for power lithium-ion battery using a fuzzy logic sliding mode observer. Energies. 2019;**12**(13):2491

[39] Abd EB, Farid M, Belkacem OB, Saad M. Bond graph modeling, design and experimental validation of a photovoltaic/fuel cell/electrolyzer/ battery hybrid power system. International Journal of Hydrogen Energy. 2021;**46**(47):24011-24027

[40] Dao VQ, Dinh M-C, Kim CS, Park M, Doh C-H, Bae JH, et al. Design of an effective state of charge estimation method for a lithium-ion battery pack using extended Kalman filter and artificial neural network. Energies. 2021; **14**:2634. DOI: 10.3390/en14092634

[41] Panwar NG, Singh S, Garg A, Gupta AK, Gao L. Recent advancements in battery management system for li-ion batteries of electric vehicles: Future role of digital twin, cyber-physical systems, battery swapping technology, and nondestructive testing. Energy

Technology. 2021;**9**:2000984. DOI: 10.1002/ente.202000984

[42] Long Z, Long H, Yuejiu Z, Xin L, Minggao O, Languang L. Massive battery pack data compression and reconstruction using a frequency division model in battery management systems. Journal of Energy Storage. 2020;**28**:101252

[43] Samanta A, Chowdhuri S, Williamson SS. Machine learning-based data-driven fault detection/diagnosis of lithium-ion battery: A critical review. Electronics. 2021;**10**:1309. DOI: 10.3390/ electronics10111309

[44] Qaisar SM, AbdelGawad AEE. Prediction of the Li-Ion Battery Capacity by Using Event-Driven Acquisition and Machine Learning, 2021 7th International Conference on Event-Based Control, Communication, and Signal Processing (EBCCSP). 2021. pp. 1-6. DOI: 10.1109/ EBCCSP53293.2021.9502399.

[45] Surya S, Rao V, Williamson SS. Comprehensive review on smart techniques for estimation of state of health for battery management system application. Energies. 2021;**14**:4617. DOI: 10.3390/en14154617

#### **Chapter 5**

## Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater Objects

*Meng Joo Er, Jie Chen and Yani Zhang*

#### **Abstract**

Underwater marine robots (UMRs), such as autonomous underwater vehicles, are promising alternatives for mankind to perform exploration tasks in the sea. These vehicles have the capability of exploring the underwater environment with onboard instruments and sensors. They are extensively used in civilian applications, scientific studies, and military missions. In recent years, the flourishing growth of deep learning has fueled tremendous theoretical breakthroughs and practical applications of computer-vision-based underwater object detection techniques. With the integration of deep-learning-based underwater object detection capability on board, the perception of underwater marine robots is expected to be enhanced greatly. Underwater object detection will play a key role in Marine Robotics 4.0, i.e., Industry 4.0 for Marine Robots. In this chapter, one of the key research challenges, i.e., real-time detection of underwater objects, which has prevented many real-world applications of object detection techniques onboard UMRs, is reviewed. In this context, state-of-theart techniques for real-time detection of underwater objects are critically analyzed. Futuristic trends in real-time detection techniques of underwater objects are also discussed.

**Keywords:** underwater marine robots, deep learning, real-time object detection

#### **1. Introduction**

In the age of Industry 4.0, revolutions based on artificial intelligence have increased by leaps and bounds in various sectors [1–3]. In the community of marine science and engineering, many underwater exploration tasks are usually executed by Underwater Marine Robots (UMRs), such as Remotely Operated Underwater Vehicles (ROVs) and Autonomous Underwater Vehicles (AUVs), as shown in **Figure 1**. These marine robots have significantly overcome many difficulties in underwater exploration tasks thanks to their distinct capability of operating round the clock. As a matter of fact, they have been widely used in the community of marine science and engineering extensively.

These UMRs, which are available in different shapes and sizes, are capable of performing a wide variety of tasks and are widely employed in many sectors. In the

**Figure 1.**

*Underwater marine robots: (a) remotely operated underwater vehicle (ROV), (b) autonomous underwater vehicle (AUV). (images from the internet).*

civilian sector, UMRs are used for aquaculture, such as providing important information for feeding, surveillance and security, and early warning of diseases [4]. Furthermore, UMRs have been exploited in seafood collection, e.g., picking holothurian, sea urchin, scallop, and other marine products, and have made significant contributions to the economy [5]. UMRs are also a promising choice to perform maintenance and cleaning works on underwater hulls, which is important to maintaining health conditions of a ship [6]. In other applications, UMRs have been employed for scientific research of the ocean, including ocean observation, underwater inspection, and monitoring of marine ecosystems. Furthermore, UMRs have been employed in the military and security sector for specific missions, such as surveillance, underwater monitoring, mine detection, and countermeasure [7].

Superior perception is highly desired for UMRs to perform assigned tasks successfully. Cameras and sonars are two kinds of sensors that UMRs typically rely on for environmental perception. There are distinct advantages and disadvantages in employing cameras and sonars for exploration tasks. However, it should be noted that both optical and sonar images share the same technology stack of processing. In recent years, flourishing development of artificial intelligence, especially deep learning, has fueled tremendous theoretical breakthroughs and practical applications [8, 9]. On one hand, development of deep learning is inseparable from exponential growth of data, which has spawned a lot of research works related to data mining [10–12]. On the other hand, artificial intelligence has been successfully applied to various fields, such as smart city [13, 14] and intelligent transportation [15, 16]. However, to our knowledge, most of these applications are on the land; underwater applications with artificial intelligence have not been fully explored yet. In the age of Industry 4.0, underwater object detection is one of the important applications that employ artificial intelligence techniques. Object detection is crucial for environmental perception which resolves around "what objects are located at where". With the adoption of deep-learning-based underwater object detection techniques on board, the perception capability of UMRs is expected to be enhanced greatly.

However, due to the constraints of existing technology, UMRs can only be equipped with embedded computing platforms, such as the Raspberry Pi, as shown in **Figure 2**-(a), which has extremely limited computing power. A more high-end computing platform is the NVIDIA Jetson, provided by NVIDIA Corporation, and is shown in **Figure 2**-(b). However, it also has limited computing power.

In order to circumvent the scarcity of limited computing resources, programs executed on such platforms must be significantly lightweight and efficient. However, existing deep learning models are usually computationally expensive. According to [17], *Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

**Figure 2.** *Embedded computing platforms for UMRs: (a) raspberry pi, (b) NVIDIA Jetson. (images from the internet).*

a standard ResNeXt-50 has about 25*:*<sup>0</sup> � <sup>10</sup><sup>6</sup> parameters and 4*:*<sup>2</sup> � <sup>10</sup><sup>9</sup> FLOPS on 8 GPUs of NVIDIA M40. This demonstrates that deep learning models are not suitable for deployment on embedded platforms, and they pose a critical research challenge for underwater object detection. In order to circumvent this limitation, deep-learningbased underwater object detection algorithms should be efficient so that they are implementable. As such, viable real-time detection techniques of underwater objects are highly desired.

Real-time detection of underwater objects, as one of the key challenges in Marine Robotics 4.0, i.e., Industry 4.0 for Marine Robots, is critically reviewed in this chapter. To facilitate a full understanding of the subject matter, we have comprehensively and systematically reviewed and analyzed related techniques for real-time detection of underwater objects. Futuristic trends in real-time detection of underwater objects are also discussed.

### **2. Preliminaries**

Underwater object detection not only needs to recognize all objects of interest, but also locate their positions in underwater images. As shown in **Figure 3**, position

#### **Figure 3.**

*Underwater object detection. The detection result is presented by a bounding box with a label on it, where xi*, *yi denotes the coordinates of i-th object, and wi* ð Þ , *hi denotes the width and height of box. x*ð Þ , *y is the frame of axes for detection results, with origin at the top left corner of the image (image from the DUO dataset [18]).*

information is generally represented by a rectangular bounding box defined by *xi*, *yi* , *wi*, *hi* , where *xi*, *yi* denotes center-point coordinates of *<sup>i</sup>* � *th* object, and *wi* ð Þ , *hi* is the width and height of the box. The frame of axes ð Þ *x*, *y* for the detection result is presented in yellow with the origin (0 � *indexed*) at the top left corner of the image. In addition, category label of the object is attached to the bounding box.

The underwater object detection problem can be formulated as follows:

$$X \stackrel{f(\theta)}{\rightarrow} \left\{ (p\_i, \ c\_i, \ x\_i, \ y\_i, \ w\_i, \ h\_i) \: \mid \: i \in (\mathfrak{1}, \ldots, N) \right\} \tag{1}$$

where *f*ð Þ*θ* indicates an object detector that is based on any neural networks parameterized by *θ*. The function *f*ð Þ*θ* takes an image *X* as its input, and outputs *N* predictions for objects in that image. The term *N* denotes the number of objects detected in that image. Each prediction contains a confidence indicator *pi* , the category label *ci* that the object belongs to, and the position information encoded in the bounding box *xi*, *yi* , *wi*, *hi* . It is well-known that underwater object detection can provide valuable information for semantic understanding of the underwater environment, and it is a fundamental research topic in the community of marine science and engineering.

#### **3. State of the arts using deep learning**

Deep-learning-based object detection methods are typically associated with large model sizes, are usually sophisticated, and cannot match real-time requirements when applied on UMR platforms. However, as far as actual use of underwater object detection in shallow water for mission execution is concerned, real-time detection is the most important prerequisite. As such, deep-learning-based detectors for UMR platforms must be as efficient as possible. The key idea that underpins the lightweight model is to create an elegant and practical lightweight network architecture while achieving excellent performance. In the field of object detection, this is a neverending quest for research excellence.

The development of underwater object detection techniques suitable for real-time performance has a long history. In this context, we will review representative literatures on real-time detection techniques, which can be categorized into three categories, namely two-stage detectors, one-stage detectors, and anchor-free detectors.

#### **3.1 Two-stage detectors**

The R-CNN (Regions with CNN features) for object detection [19] is the first successful two-stage deep learning object detector developed in the object detection community, but it is not suitable for real-time detection. As illustrated in **Figure 4**, it begins with a selective search [20] to extract a collection of object candidates (region proposals). Next, to extract features, each proposal is re-scaled to a fixed-size picture and input to a Convolutional Neural Network (CNN) which is pre-trained on ImageNet [21]. Finally, linear SVM classifiers are utilized to predict the existence of an object and to distinguish object types inside each region based on the features extracted by CNN.

However, the R-CNN applies CNN to each potential region for extracting features. There are a lot of overlaps, resulting in many redundant computations and resulting in very sluggish detection speed. In order to alleviate this problem, the Fast R-CNN [22]

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

**Figure 4.**

*Network architecture of R-CNN, where CNN features extraction is applied on each candidate region (image from [19]).*

**Figure 5.**

*Network architecture of fast R-CNN, where features extraction is applied to the entire image only once (figure from [22]).*

employ CNN to extract features from the entire picture only once and obtains features for each candidate region via a Region of Interest (ROI) pooling operation, as illustrated in **Figure 5**. In comparison with the R-CNN, it achieves superior accuracy on various benchmark datasets but improve image processing speed by 146 times under the same conditions and reduces the training time by 9 times.

In [23], Fast R-CNN is trained to detect underwater objects in sonar images. By using Bayesian optimization, which follows the Automated Machine Learning (AutoML) principle, the hyperparameter configuration of Fast R-CNN was set to be optimum. In [24], encouraged by the powerful detection performance obtained by CNNs on generic datasets, Fast R-CNN is applied to a domain-specific underwater environment for accurate identification and recognition of fish. At the time, Fast R-CNN was widely used in underwater object detection.

However, Fast R-CNN continues to employ complicated selective search approach for the generation of candidate region proposals, which turns out to be timeconsuming. Ren *et al.* [25] propose a Region Proposal Network (RPN) that predicts candidates directly from the shared feature maps, as illustrated in **Figure 6**. This new

#### **Figure 6.**

*Network architecture of faster R-CNN, where region proposal network (RPN) is proposed for extraction of region candidates based on the shared feature maps (figure from [25]).*

**Figure 7.** *Network architecture of mask R-CNN (figure from [33]).*

architecture is dubbed Faster R-CNN. Typical processing time of each picture in selective search is around 1 2*s*, but the RPN requires only approximately 10*ms*, resulting in tremendous increase in detection speed.

In [26], for faster detection and recognition of fishes by sharing CNNs with objectness learning, the backbone of Faster R-CNN is substituted with a pre-trained ZFNet [27]. In [28], the Faster R-CNN is enhanced to detect underwater organisms, which is exposed to many challenges, such as low-quality images, varying sizes or forms, and overlapping or occlusion objects. The backbone is replaced with ResNet [29]. For multi-scale feature fusion, the BiFPN architecture proposed in [30] is adopted. Finally, to minimize the amount of redundant bounding boxes in the training data, the EIoU (Effective IoU) [31] is utilized to replace IoU. On the URPC2018 dataset [32], the accuracy of the modified Faster R-CNN is 8.26% higher than the original version of Faster R-CNN. Faster R-CNN has dominated underwater object detection for a long time.

After that, the Faster R-CNN is extended by Mask R-CNN, which adds a branch for predicting an object mask in parallel with the current branch for bounding box identification [33], as illustrated in **Figure 7**. It can recognize objects in a picture quickly while also creating a high-quality segmentation mask for each instance. Thanks to the benefits of multi-task learning, Mask R-CNN outperforms all existing single-model entries on a wide range of computer vision tasks by adding only a minor overhead to Faster R-CNN.

In [34], to identify and separate underwater objects from forward-looking sonar pictures, a modified Mask R-CNN is proposed by replacing the Resnet backbone. The modified Mask R-CNN reduces the number of network parameters significantly while maintaining the detection performance. It is suitable for real-time detection. The Mask R-CNN is also utilized to identify common fishery species (yellowfin bream, *Acanthopagrus australis*) for animal movement studies to assess ecosystem health, comprehend ecological dynamics, and address management and conservation problems [35].

In this section, we have reviewed several representative two-stage detectors. By discarding the complicated module with high computational complexity, the detection speed improves significantly.

#### **3.2 One-stage detectors**

The aforementioned detectors are members of the R-CNN family of two-stage algorithms, which frame the detection as a "coarse-to-fine" process [36]. They are well-known for their excellent detection precision but low detection speed [37].

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

Another family of detectors, the YOLOs (You Only Look Once) [38–40] foregoes extraction of candidate region proposals and predicts detection outcomes directly from shared feature maps of CNN. These approaches are also known as one-stage detectors. The inference time is reduced to 50 *ms* by using a one-stage approach while maintaining relatively high accuracy, whereas other competitive models need more than 200 *ms*. This is a bigger leap forward in terms of real-time detection.

In 2015, R. Joseph *et al.* proposed the YOLO detector [38]. The key idea of the YOLO detector is to split the picture into grids and predict bounding boxes and probabilities for each cell by using a CNN directly. As illustrated in **Figure 8**, it splits the picture into a *S* � *S* grid, and predicts *B* bounding boxes with 1 confidence per box, and *C* class probabilities for each grid cell. The final predictions are encoded in a *S* � *S* � ð Þ *B* ∗ 5 þ *C* output tensor directly by the convolutional network.

In [41], a YOLO detector is trained on generating realistic sonar pictures by GANs [42] for underwater object recognition, which is required to automate activities like shipwreck investigation, mine clearance, and landmark-based navigation. Later, R. Joseph produced a series of enhancements to YOLO and offered its v2 and v3 editions [39, 40], which improved detection accuracy while maintaining fast detection speed.

YOLO v2 is an enhanced version of YOLO, with batch normalization [43], removal of fully connected layers, and the use of excellent anchor boxes acquired using kmeans and multiscale training, in addition to the custom GoogLeNet network [44] being replaced by the simpler DarkNet19 network. In [45], YOLO v2 is presented as a coarse pre-detection module in the pipeline of rotational object detection using forward-looking sonar in underwater applications, where detection results of YOLO v2 are clipped from the sonar picture and fed to a more fine-grained detector.

The most extensively utilized approach in the industry is YOLO v3, where the Darknet-53 backbone harvests features, and three detection heads fuse different scale feature maps for object detection with different sizes. In [46], experiments to detect and classify sea cucumber, scallop, and sea urchin from underwater photos were carried out, and the results demonstrate that the YOLO v3 algorithm has a *mAP* value 6.4% higher and a recall rate 13.9% higher than Faster R-CNN. Furthermore, YOLO v3 has a detection speed of 20 frames per second, which is 12 frames per second faster

#### **Figure 8.**

*The YOLO detector is depicted as a regression issue in this picture. It splits the picture into a S* � *S grid, predicting B bounding boxes with 1 confidence per box, and C class probabilities for each grid cell. The tensor S* � *S* � ð Þ *B* ∗ 5 þ *C encodes these predictions.*

than Faster R-CNN. In [47], YOLOv3 is integrated into an underwater manipulator (BlueROV2) to identify objects for grabbing.

YOLO v4 [48] has put to the test a large variety of strategies that are supposed to enhance accuracy of a CNN. Finally, it combines techniques such as Weighted-Residual-Connections [30], Cross-Stage-Partial-Connections [49], Cross mini-Batch Normalization [50], Self-adversarial-training [51], Mish activation [52], Mosaic data augmentation, DropBlock regularization [53], and CIoU loss [54] to achieve optimal object detection speed and accuracy. In [55], to construct a lightweight underwater object detector, YOLO v4 is combined with a multi-scale attentional feature fusion module. For real-time performance, it also replaces the CSPDarknet53 backbone [49] with MobileNet [56].

From two-stage detectors to one-stage detectors, the YOLO series has gained a qualitative leap in real-time underwater object detection. Leveraging meticulous design in the network architecture, one-stage detectors will improve performance and detection speed significantly.

#### **3.3 Anchor-free detectors**

Another significant paradigm shift in real-time object detection is from anchorbased to anchor-free techniques. The majority of the aforementioned approaches are anchor-based, whereby anchors of various sizes and aspect ratios are established on the picture, allowing object detection to predict related offsets. The usage of anchor boxes has long been thought to be a secret to successful detection [57].

Thousands of pre-defined anchor boxes are placed on the picture in anchor-based techniques, and the model predicts which anchor box will respond to the groundtruth. However, the generation of anchors via region proposal network [25] or kmeans clustering [40] is a time-consuming process. Undoubtedly, anchor-based approaches will also result in duplicate predictions, necessitating the use of a nonmaximum suppression algorithm [58] to eliminate undesirable outcomes. Unfortunately, non-maximum suppression is also an expensive operation, which slows down the speed of object detection significantly.

Anchor-free detectors aim to eliminate expensive operations that are related to anchor mechanism. Without the necessity for non-maximum suppression, anchorfree techniques remove the computation load raised by anchors and regress the category and position of the object directly by convolutional networks [57, 59]. They remove anchor-related computations like anchor clustering, allowing for even more real-time efficiency.

One of the most canonical anchor-free detectors, CenterNet [59], represents an object as a single point – the center-point of its bounding box. As illustrated in **Figure 9**, the neural network predicts the center-point heatmaps *Y*^, offsets *O*^ and sizes ^ *S* of bounding boxes. By using key point estimation, CenterNet determines the center point of objects and regresses all other object parameters, such as size. The bounding box at position *xi*, *yi* may be generated from predictions at inference as follows:

$$\left(\hat{\mathfrak{x}}\_{i} + \delta\hat{\mathfrak{x}}\_{i} - \hat{w}\_{i}\hat{\mathfrak{y}}\_{i} + \delta\hat{\mathfrak{y}}\_{i} - \hat{h}\_{i}\hat{\mathfrak{x}}\_{i} + \delta\hat{\mathfrak{x}}\_{i} + \hat{w}\_{i}\hat{\mathfrak{y}}\_{i} + \delta\hat{\mathfrak{y}}\_{i} + \hat{h}\_{i}\right) \tag{2}$$

where *δx*^*i*, *δ*^*yi* <sup>¼</sup> *<sup>O</sup>*^ *<sup>x</sup>*^*i*,^*yi* is the offset prediction and *<sup>w</sup>*^*i*, ^ *hi* <sup>¼</sup> ^ *Sx*^*i*,^*yi* is the size prediction. Without the use of IoU-based non-maxima suppression or other postprocessing operations, all outputs are generated directly from key point estimations. *Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

**Figure 9.** *Architecture of CenterNet (figure from [59]).*

Contrary to complicated computation experienced in anchor mechanism, the detection speed of anchor-free models is improved over one-stage detectors significantly while maintaining superior detection accuracy. Anchor-free models have become the de facto solution for real-time detection [57]. For example, the AquaNet [5] and MRF-Net [60] are improved based on the anchor-free model termed CenterNet for underwater detection, and the efficiency and effectiveness are both verified by comprehensive experiments.

#### **4. Futuristic trends**

The limited computing resources of UMRs is the main factor that prevents the deployment of deep-learning-based models for real-time detection in underwater environment. Meanwhile, difficulties of communication in underwater environment prevent the possibility of exploring other cloud computing solutions. As a consequence, reducing model size seems to be the only feasible method moving forward.

In the literatures, the two strategies to achieve real-time underwater object detection, namely lightweight network design and model compression, have been proposed. Lightweight network design aims at developing some effective low-complexity network architecture, while model compression attempts to remove redundant parameters of a pre-trained model.

#### **4.1 Lightweight network design**

In the development of deep learning algorithms, by discarding or replacing the most complicated module in a model, both accuracy and inference speed in deeplearning-based object detection have been improved [44, 56, 61]. Re-designing fundamental components in the neural network architecture is another option for achieving light-weighting model.

GoogLeNet [44] presented an Inception block made up of 4 convolution paths in various configurations. Convolution with 1 1 kernel is extensively utilized in the Inception block to minimize the computational complexity. The network becomes more efficient by approximating the predicted ideal sparse structure using conveniently accessible dense construction pieces. In SqueezeNet [62], 1 1 convolutions are also utilized to replace 3 3 convolutions. It reduces the number of input channels to 3 3 convolutions and postpones the down-sampling operations in the network architecture. Finally, with the same detection accuracy, SqueezeNet is 50 smaller than AlexNet [63] in size, resulting in higher detection speed.

**Figure 10.** *Illustration of depth-wise separable convolution.*

In contrast with conventional convolution, MobileNet [56] proposed depth-wise separable convolutions, which are a type of factorized convolution that factorizes a standard convolution into a depth-wise convolution and a 1 1 point-wise convolution, as shown in **Figure 10**, saving a significant amount of mult-adds operations and parameters while reducing accuracy by only 1%. ShuffleNet [64], on the other hand, makes use of two novel operations, point-wise group convolution and channel shuffle, to drastically reduce computational costs while preserving moderate detection accuracy. Xception [65], ResNeXt [17], and ChannelNet [66] are also wonderful works that adopt depth-wise separable convolution.

Depth-wise separable convolution, 1 1 convolution, and Max-pooling procedure are all employed extensively in the deep neural network presented in [61] to reduce computational complexity and model size. They also constructed an efficient receptive module inspired by Inception v3 architecture [67] to compensate for the inadequately retrieved features, as illustrated in **Figure 11**. Taking advantage of lightweight design, the proposed method outperforms or is comparable to state-of-the-art methods in terms of the *mAP* metric, and it significantly outperforms existing methods in terms of detection speed metrics, such as *GFLOPs*, processing time per image, and *FPS*. Experimental results demonstrate that the proposed algorithm can be executed on *RaspberryPi*, achieving real-time underwater object detection.

The underlying theory of lightweight network design is low-rank approximation. When information is encoded in data matrix *X*, a full-rank matrix *X*^, which is constructed by the linearly independent columns (or rows) of matrix *X*, can be obtained. It is quite conceivable (and rather frequent) for the rank of a matrix to be smaller than the total number of column vectors in the matrix. This means there are some redundant columns that can be generated by scaling and concatenating multiple

**Figure 11.** *Receptive module inspired by inception block (figure from [61]).*

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

columns from the full-rank matrix *X*^. In other words, when a matrix contains redundant information, it can be represented by using fewer bits with little-to-no loss in information. Based on the theory of low-rank approximation, different effective and efficient techniques can be employed to design lightweight network architectures. Undoubtedly, it will play a pivotal role in realizing real-time detection of underwater objects.

#### **4.2 Model compression**

Another key method to achieve real-time detection is model compression [68], which aims to remove redundant parameters (or neurons) in the pre-trained models. Existing research has shown that deep networks exhibit parameter redundancy, which is useless for final prediction [69]. This serves as a theoretical foundation for compressing of deep learning models. Various compression methods have been proposed over the years, each of which has its pros and cons. Network pruning [70], knowledge distillation [71], and parameter quantization [72] are some of the most prominent strategies used to reduce network complexity.

Neural networks are typically over-parameterized, i.e., there are significant redundant parameters or neurons [73]. Based on this observation, we can reduce redundancy without compromising substantial performance degradation. In network pruning, the importance of neurons (or parameters) is first evaluated based on some metrics, such as the number of times it was not zero on a given dataset, the absolute values or the lifetime of the neurons, etc. Next, neurons that are of less importance will be removed.

With pruning, the model's performance is expected to drop. In general, performance degradation can be recovered by fine-tuning using the training dataset [74]. Network pruning can be applied at multiple granularities by different implementations, such as weight pruning, neuron pruning, kernel pruning, channel pruning, etc. By removing redundancy in the network, model complexity can be reduced, and generalization can be improved. Based on network pruning, even over 90% of the model size can be removed with little-to-no performance loss, and the computation speed of the model is improved significantly. In fact, network pruning has become a prerequisite for the deployment of deep learning on edge devices.

Knowledge distillation is another important technique for model compression. In general, training multiple distinct models on the same dataset and then averaging their predictions is a fairly easy technique to enhance the performance of almost any machine learning algorithm [75]. It is also widely believed that a large neural network usually outperforms a small one before over-fitting. Knowledge distillation compresses the knowledge in an ensemble (or a large model, known as a "Teacher Model") into a single small model (known as a "Student Model"), which is much easier to deploy on edge devices that are limited in computing resources [71]. It is achieved by minimizing the distance of predictive distribution between the Teacher Model and Student Model. The predictive distribution output by Teacher Model usually contains some implicit knowledge from the training dataset, which is helpful to guide model learning, easing out the optimization process. Through knowledge distillation, we can maintain superior performance of the larger model while reducing model size and consumption of computing resources.

Parameter quantization is concerned with re-organization of network parameters. The main objective of parameter quantization is to represent the neural network with fewer bits [76]. For example, by compressing the 16-bit float parameters into 8-bit integers, one can halve the memory cost with little loss in performance [77]. However, the most commonly used quantization technique is parameter clustering [78, 79], where the parameters in a network are first clustered by clustering methods (e.g., kmeans), and then every parameter is represented by the centroid of the corresponding cluster. Based on parameter clustering, the entire neural network can be represented by a cluster index table and the centroids. Each index is denoted by 2-bit unsigned integers. Hence, the deep learning model can be compressed significantly. In the extreme case, we can convert a network to a binary connect model, where all parameters are þ1 or �1 [80]. Last but not least, some information encoding methods, such as Huffman encoding, that represents frequent clusters with fewer bits and rare clusters with more bits [81], can also be used as quantization techniques, since they are efficient encoding strategies.

In this section, we have reviewed two key techniques that help to reduce the model size but maintain moderate performance with only slight degradation. Through model reduction, the memory cost and computational complexity are reduced significantly, which makes real-time detection on resource-constrained devices more feasible. Indeed, lightweight network design and model compression are complementary and should be applied iteratively to obtain a more elegant model.

#### **5. Conclusions**

UMRs play a significant and pivotal role in ocean exploration in the era of Industry 4.0. Real-time object detection will equip UMRs with superior perception capabilities. In this chapter, we have identified real-time object detection as a key challenge of ocean exploration while using UMRs. Towards this end, crucial techniques pertaining to real-time detection of underwater objects have been critically reviewed and systematically analyzed based on the evolution in deep learning techniques. Three categories of detectors, namely two-stage detectors, one-stage detectors, and anchor-free detectors, have been reviewed and analyzed. Furthermore, futuristic trends of realtime detection, including lightweight network design and model compression, have been proposed and intensively discussed. It is hoped that readers will find this survey informative and useful in helping them to understand recent advancements in realtime detection of underwater objects, and will guide them in research in this exciting area, which will have a long-lasting impact to the mankind.

#### **Acknowledgements**

The authors would like to acknowledge the support of Fundamental Research Funds for the Central Universities under Grant 3132019344 and Leading Scholar Grant, Dalian Maritime University under Grant 00253007.

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

### **Author details**

Meng Joo Er\*, Jie Chen and Yani Zhang Institute of Artificial Intelligence and Marine Robotics, College of Marine Electrical Engineering, Dalian Maritime University, Dalian, China

\*Address all correspondence to: mjer@dlmu.edu.cn

© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Gordan M, Razak HA, Ismail Z, Ghaedi K. Recent developments in damage identification of structures using data mining. Latin American Journal of Solids and Structures. 2017;**14**: 2373-2401

[2] Ghaedi K, Gordan M, Ismail Z, Hashim H, Talebkhah M. A literature review on the development of remote sensing in damage detection of civil structures. Journal of Engineering Research and Reports. 2021;**20**(10): 39-56

[3] Gordan M, Sabbagh-Yazdi S-R, Ismail Z, Ghaedi K, Carroll P, McCrum D, et al. State-of-the-art review on advancements of data mining in structural health monitoring. Measurement. 2022:**193**:110939

[4] Li J, Xu C, Jiang L, Xiao Y, Deng L, Han Z. Detection and analysis of behavior trajectory for sea cucumbers based on deep learning. IEEE Access. 2019;**8**:18832-18840

[5] Liu C, Wang Z, Wang S, Tang T, Tao Y, Yang C, et al. A new dataset, poisson Gan and aquanet for underwater object grabbing. IEEE Transactions on Circuits and Systems for Video Technology. 2022;**32**:2831-2844

[6] Song C, Cui W. Review of underwater ship hull cleaning technologies. Journal of Marine Science and Application. 2020;**19**(3): 415-429

[7] Hoadley DS and Lucas NJ. Artificial intelligence and national security. Congressional Research Service. 2018

[8] Gordan M, Chao OZ, Sabbagh-Yazdi S-R, Wee LK, Ghaedi K, Ismail Z. From cognitive bias toward advanced

computational intelligence for smart infrastructure monitoring. Frontiers in Psychology. 2022;**13**:846610-846610

[9] Talebkhah M, Sali A, Marjani M, Gordan M, Hashim SJ, Rokhani FZ. Iot and big data applications in smart cities: Recent advances, challenges, and critical issues. IEEE Access. 2021;**9**:55465-55484

[10] Gordan M, Razak HA, Ismail Z, Ghaedi K, Tan ZX, Ghayeb HH. A hybrid ann-based imperial competitive algorithm methodology for structural damage identification of slab-on-girder bridge using data mining. Applied Soft Computing. 2020;**88**:106013

[11] Gordan M, Ismail Z, Razak HA, Ghaedi K, Ibrahim Z, Tan ZX, et al. Data mining-based damage identification of a slab-on-girder bridge using inverse analysis. Measurement. 2020;**151**:107175

[12] Gordan M, Ismail ZB, Razak HA, Ghaedi K, Ghayeb HH. Optimizationbased evolutionary data mining techniques for structural health monitoring. Journal of Civil Engineering and Construction. 2020;**9**(1):14-23

[13] Tan ZX, Thambiratnam DP, Chan TH, Gordan M, Abdul Razak H. Damage detection in steel-concrete composite bridge using vibration characteristics and artificial neural network. Structure and Infrastructure Engineering. 2020;**16**(9):1247-1261

[14] Gordan M, Sabbagh-Yazdi S-R, Ghaedi K, Thambiratnam DP, Ismail Z. Introduction to monitoring of bridge infrastructure using soft computing techniques. In: Applied Methods in Design and Construction of Bridges, Highways and Roads - Theory and Practice. London, UK: IntechOpen; 2022 ch. 4

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

[15] Prakash A, Behl A, Ohn-Bar E, Chitta K, Geiger A. Exploring data aggregation in policy learning for visionbased urban autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. pp. 11763-11773. DOI: 10.1109/ CVPR42600.2020.01178

[16] Kendall A, Hawke J, Janz D, Mazur P, Reda D, Allen JM, et al. Learning to drive in a day. In: Proceedings of the International Conference on Robotics and Automation (ICRA). 2019. pp. 8248-8254. DOI: 10.1109/ICRA.2019.8793742

[17] Xie S, Girshick R, Dollár P, Tu Z, He K. Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. pp. 1492-1500. DOI: 10.48550/ arXiv.1611.05431

[18] Liu C, Li H, Wang S, Zhu M, Wang D, Fan X, et al. A dataset and benchmark of underwater object detection for robot picking. In: Proceedings of the IEEE International Conference on Multimedia & Expo Workshops. 2021. pp. 1-6. DOI: 10.1109/ ICMEW53276.2021.9455997

[19] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014. pp. 580-587. DOI: 10.1109/ CVPR.2014.81

[20] Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW. Selective search for object recognition. International Journal of Computer Vision. 2013;**104**(2):154-171

[21] Jia D, Wei D, Richard S, Li JL, Kai L, and Li FF. Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009. pp. 248-255. DOI: 10.1109/CVPR.2009.5206848

[22] Girshick R. Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. 2015. pp. 1440-1448. DOI: 10.1109/ ICCV.2015.169

[23] Karimanzira D, Renkewitz H, Shea D, Albiez J. Object detection in sonar images. Electronics. 2020;**9**(7):1180

[24] Li X, Shang M, Qin H, Chen L. Fast accurate fish detection and recognition of underwater images with fast r-cnn. In: Proceedings of the IEEE Conference on OCEANS. 2015. pp. 1-5. DOI: 10.23919/ OCEANS.2015.7404464

[25] Ren S, He K, Girshick R, Sun J. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016;**39**(6): 1137-1149

[26] Li X, Shang M, Hao J, Yang Z. Accelerating fish detection and recognition by sharing cnns with objectness learning. In: Proceedings of the IEEE Conference on OCEANS. 2016. pp. 1-5. DOI: 10.1109/ OCEANSAP.2016.7485476

[27] Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. In: Proceedings of the European Conference on Computer Vision. 2014. pp. 818-833. DOI: 10.1007/978-3-319- 10590-1\_53

[28] Shi P, Xu X, Ni J, Xin Y, Huang W, Han S. Underwater biological detection

algorithm based on improved fasterrcnn. Water. 2021;**13**(17):2420

[29] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. pp. 770-778. DOI: 10.1109/CVPR.2016.90

[30] Tan M, Pang R, Le QV. Efficientdet: Scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. pp. 10781- 10790. DOI: 10.48550/arXiv.1911.09070

[31] Zhang YF, Ren W, Zhang Z, Jia Z, Wang L, Tan T. Focal and efficient iou loss for accurate bounding box regression. Neurocomputing. 2022;**506**: 146-157

[32] D. U. o. T. National Nature Science Foundation of China. China Underwater Robot Predessional Contest [Online]. Available: http://www.urpc.org.cn/

[33] He K, Gkioxari G, Dollár P, Girshick R. Mask r-cnn. In: Proceedings of the IEEE international Conference on Computer Vision. 2017. pp. 2961-2969. DOI: 10.1109/ICCV.2017.322

[34] Fan Z, Xia W, Liu X, Li H. Detection and segmentation of underwater objects from forward-looking sonar based on a modified mask rcnn. Signal, Image and Video Processing. 2021;**15**:1135-1143

[35] Lopez Marcano S, Jinks EL, Buelow CA, Brown CJ, Wang D, Kusy B, et al. Automatic detection of fish and tracking of movement for ecology. Ecology and Evolution. 2021;**11**(12): 8254-8263

[36] Zou Z, Shi Z, Guo Y, Ye J. Object detection in 20 years: A survey. arXiv preprint arXiv:1905.05055. 2019

[37] Yan D, Li G, Li X, Zhang H, Lei H, Lu K, et al. An improved faster r-cnn method to detect tailings ponds from high-resolution remote sensing images. Remote Sensing. 2021;**13**(11):2052

[38] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. pp. 779-788. DOI: 10.1109/CVPR.2016.91

[39] Redmon J, Farhadi A. Yolo9000: Better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. pp. 7263-7271. DOI: 10.1109/ CVPR.2017.690

[40] Redmon J, Farhadi A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018

[41] Sung M, Kim J, Lee M, Kim B, Kim T, Kim J, et al. Realistic sonar image simulation using deep learning for underwater object detection. International Journal of Control, Automation and Systems. 2020;**18**(3): 523-534

[42] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde Farley D, Ozair S, et al. Generative adversarial networks. Communications of the ACM. 2020;**63** (11):139-144

[43] Ioffe S and Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the International Conference on Machine Learning. 2015. pp. 448-456. DOI: 10.5555/3045118.3045167

[44] Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. In:

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015. pp. 1-9. DOI: 10.1109/ CVPR.2015.7298594

[45] Neves G, Ruiz M, Fontinele J, Oliveira L. Rotated object detection with forward-looking sonar in underwater applications. Expert Systems with Applications. 2020;**140**:112870

[46] Yang H, Liu P, Hu Y, Fu J. Research on underwater object recognition based on yolov3. Microsystem Technologies. 2021;**27**(4):1837-1844

[47] Haugaløkken BOA, Skaldebø MB, Schjølberg I. Monocular vision-based gripping of objects. Robotics and Autonomous Systems. 2020;**131**:103589

[48] Bochkovskiy A, Wang C-Y, Liao H-YM. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv: 2004.10934. 2020

[49] Wang CY, Liao HYM, Wu YH, Chen PY, Hsieh JW, Yeh IH. Cspnet: A new backbone that can enhance learning capability of cnn. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020. pp. 390-391. DOI: 10.1109/ CVPRW50498.2020.00203

[50] Yao Z, Cao Y, Zheng S, Huang G, Lin S. Cross-iteration batch normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. pp. 12331-12340

[51] Chen K, Chen Y, Zhou H, Mao X, Li Y, He Y, et al. Self-supervised adversarial training. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 2020. pp. 2218-2222. DOI: 10.1109/ ICASSP40776.2020.9054475

[52] Misra D. Mish: A self regularized non-monotonic neural activation function. arXiv preprint arXiv: 1908.08681. 2019;**4**:2

[53] Ghiasi G, Lin TY, Le QV. Dropblock: A regularization method for convolutional networks. Advances in neural information processing systems. 2018;**31**

[54] Zheng Z, Wang P, Liu W, Li J, Ye R, Ren D. Distance-iou loss: Faster and better learning for bounding box regression. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020. pp. 12993-13000. DOI: 10.1609/ aaai.v34i07.6999

[55] Zhang M, Xu S, Song W, He Q, Wei Q. Lightweight underwater object detection based on yolo v4 and multiscale attentional feature fusion. Remote Sensing. 2021;**13**(22):4706

[56] Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861. 2017

[57] Tian Z, Shen C, Chen H, He T. Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. pp. 9627-9636. DOI: 10.1109/ ICCV.2019.00972

[58] Hosang J, Benenson R, and Schiele B. Learning non-maximum suppression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. pp. 4507-4515. DOI: 10.1109/CVPR.2017.685

[59] Zhou X, Wang D, Krähenbühl P. Objects as points. arXiv preprint arXiv: 1904.07850. 2019

[60] Qin R, Zhao X, Zhu W, Yang Q, He B, Li G, et al. Multiple receptive field network (mrf-net) for autonomous underwater vehicle fishing net detection using forward-looking sonar images. Sensors. 2021;**21**(6):1933

[61] Yeh CH, Lin CH, Kang LW, Huang CH, Lin MH, Chang CY, et al. Lightweight deep neural network for joint learning of underwater object detection and color conversion. IEEE Transactions on Neural Networks and Learning Systems. 2021:1-15. DOI: 10.1109/TNNLS.2021.3072414

[62] Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 1602.07360. 2016

[63] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 2012;**25**:1097-1105

[64] Zhang X, Zhou X, Lin M, Sun J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. pp. 6848-6856. DOI: 10.1109/ CVPR.2018.00716

[65] Chollet F. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. pp. 1251-1258. DOI: 10.1109/CVPR.2017.195

[66] Gao H, Wang Z, Cai L, Ji S. Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021;**43**(08): 2570-2581

[67] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. pp. 2818- 2826. DOI: 10.1109/CVPR.2016.308

[68] Cheng Y, Wang D, Zhou P, Zhang T. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine. 2018;**35**(1):126-136

[69] Cheng Y, Yu FX, Feris RS, Kumar S, Choudhary A, Chang S. An exploration of parameter redundancy in deep networks with circulant projections. In: Proceedings of the IEEE International Conference on Computer Vision. 2015. pp. 2857-2865. DOI: 10.1109/ ICCV.2015.327

[70] Liu Z, Sun M, Zhou T, Huang G, Darrell T. Rethinking the value of network pruning. arXiv preprint arXiv: 1810.05270. 2018

[71] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02531. 2015

[72] Han S, Mao H, Dally WJ. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. 2015

[73] LeCun Y, Denker J, Solla S. Optimal brain damage. Advances in Neural Information Processing Systems. 1989;**2**: 598-605

[74] Liang T, Glossner J, Wang L, Shi S, Zhang X. Pruning and quantization for

*Marine Robotics 4.0: Present and Future of Real-Time Detection Techniques for Underwater… DOI: http://dx.doi.org/10.5772/intechopen.107409*

deep neural network acceleration: A survey. Neurocomputing. 2021;**461**: 370-403

[75] Dietterich TG. Ensemble methods in machine learning. In: Proceedings of the International Workshop on Multiple Classifier Systems. 2000;**1578**:1-15

[76] Cheng Y, Wang D, Zhou P, Zhang T. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282. 2017

[77] Vanhoucke V, Senior A, Mao MZ. Improving the speed of neural networks on cpus. In: Proceedings of the Deep Learning and Unsupervised Feature Learning Workshop. Granada Spain: NIPS; 2011

[78] Gong Y, Liu L, Yang M, Bourdev L. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. 2014

[79] Wu J, Leng C, Wang Y, Hu Q, Cheng J. Quantized convolutional neural networks for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. pp. 4820-4828. DOI: 10.48550/arXiv.1512.06473

[80] Courbariaux M, Bengio Y, David J-P. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems. 2015; **28**:3123-3131

[81] Erdal E, Ergüzen A. An efficient encoding algorithm using local path on huffman encoding algorithm for compression. Applied Sciences. 2019; **9**(4):782

#### **Chapter 6**

## Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications

*Maddi Etxegarai, Marta Camps, Lluís Echeverria, Marc Ribalta, Francesc Bonada and Xavier Domingo*

#### **Abstract**

The current digitalisation revolution demonstrates the high importance and possibilities of quality data in industrial applications. Data represent the foundation of any analytical process, establishing the fundamentals of the modern Industry 4.0 era. Data-driven processes boosted by novel Artificial Intelligence (AI) provide powerful solutions for industrial applications in anomaly detection, predictive maintenance, optimal process control and digital twins, among many others. Virtual Sensors offer a digital definition of a real Internet of Things (IoT) sensor device, providing a smart tool capable to face key issues on the critical data generation side: i) Scalability of expensive measurement devices, ii) Robustness and resilience through real-time data validation and real-time sensor replacement for continuous service, or iii) Provision of key parameters' estimation on difficult to measure situations. This chapter presents a profound introduction to Virtual Sensors, including the explanation of the methodology used in industrial data-driven projects, novel AI techniques for their implementation and real use cases in the Industry 4.0 context.

**Keywords:** virtual sensors, artificial intelligence, machine learning, innovative sensing strategies, internet of things, industry 4.0

#### **1. Introduction**

Digitalisation and data exploitation are two of the fundamental driving forces of the new paradigm defined by the Industry 4.0 (I4.0) revolution. Recent developments in sensors, Cyber-Physical Systems (CPS), automation, and quality inspection, among others, are motivating the digitalisation of the manufacturing and nonmanufacturing industries, making available large amounts of data that may capture the nature of the process and its variability. These data streams become of utmost importance when targeting enhanced productivity, flexibility, competitiveness, and environmental impact. Hence, these large data streams not only represent a valuable opportunity but also introduce a substantial challenge for industries to digest and extract value from them, without losing focus on their day-to-day operations. Data-driven solutions, including Data Mining, Big Data, or Artificial Intelligence

(AI), provide the right tools and functionalities to digest these large amounts of data, create value, and impact manufacturing operational Key Performance Indicators (KPIs). Moreover, AI-based solutions can also support knowledge discovery actions and enrich experts' industrial knowledge by discovering previously unknown process parameter correlations that can have a big impact on industrial operations.

The perceived value of data exploitation techniques, mainly powered by AI solutions, has increased in line with the growing available data in nearly all processes and sectors. The development of data-centred and data-driven solutions has become a crucial element as a tool for not only managing but also taking advantage of the incoming process data. Nevertheless, an important issue must be considered: do available or captured data accurately represent the scenario, the process, and the environment? In most cases, the answer is no. Not all relevant or key process parameters can be physically measured, or the associated cost for direct measurement is not sustainable. Thus, the need for computing or estimating these key process parameters based on measured data has become more relevant as data availability has increased and production excellence has become progressively more demanding.

Traditionally, relying on the data provided by physical sensors has been a recurring challenge due to several limitations: the cost of the sensor, accuracy, stability, and impossibility to measure specific parameters due to physical, spatial, or environmental constraints. These challenges have commonly been addressed by using analytical approaches based on physical and mathematical expressions. While this strategy can increase the underlying physics knowledge of the process and provides a general solution, it also requires extensive experimental validation and the definition of accurate assumptions and boundary conditions. Recent AI and Machine Learning (ML) advances allow for novel data-driven approaches to estimate key process parameters. The so-called Virtual Sensors (VS), also known as Soft Sensors or Software Sensors, represent a software layer that provides indirect measurements of a process variable based on the data gathered by physical (or other virtual) sensors leveraging a fusion function [1]. The exponential growth of data during the last decade has entailed the rise of data-driven solutions powered by AI and ML algorithms, correlating input data (measurable parameters) with output targets by heuristic and probabilistic models.

This chapter aims to explain the potential of Virtual Sensors for industrial process monitoring and provide an introduction to their development. Two real industrial use cases are presented, focused on High Pressure Die Casting and wastewater treatment, to illustrate and highlight the capabilities of this technology.

#### **2. Industrial applications of Virtual Sensors**

The decision-making process in industrial applications (logistics, planning, quality control, predictive maintenance, etc.) is driven and influenced by the evolution of key parameters along the production process value chain. In most cases, this set of key parameters is obtained by deploying sensors along the process chain. For instance, placing a thermocouple sensor to monitor the temperature in a foundry or a flow meter in a complex water distribution system pipe. The monitoring data captured along the production line is used to compute KPIs that measure the operational performance of the industrial process or equipment over time. Industrial KPIs are of utmost importance for informed decision-making, as well as for measuring and targeting an objective accomplishment. Some of the most relevant KPIs are:

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*


Data accuracy and reliability are of great importance since the decision-making process relies on KPIs computed from data gathered at the production chain. In case of sensor failure, malfunction, drift, or need for recalibration, industrial KPIs may not confidently represent the process performance anymore. This situation could lead to two non-optimal scenarios: poor decision-making due to the lack of reliable information or production breakdowns due to equipment failure. Furthermore, equipment, infrastructure, material, or even people involved (technicians, staff, etc.) may be threatened due to the malfunction of the monitoring systems. Thus, mitigation strategies should be considered to reduce this risk. Robust and accurate data-driven solutions leveraging production data can provide resilience capabilities to operate in non-optimal conditions. Virtual Sensor offers an appropriate solution since they increase the reliability and agility of the system at a low operational cost, providing an indirect measurement for non-measurable physically properties.

AI and data-driven Virtual Sensor can significantly impact industrial applications by providing valuable process insights that support and enrich informed decisionmaking processes, as shown in **Figure 1**, where the schematic design of two Virtual Sensors is introduced.

Within the industrial applications, the three key objectives of Virtual Sensors are:

• **Expand knowledge:** To compute extra parameters derived from real sensors that are impossible or not sustainable to measure (at full-scale), thus contributing to a better understanding of the process.

**Figure 1.** *Virtual sensors applications in industry.*


Industrial applications can benefit from the Virtual Sensor functionalities: increasing the knowledge of the process, reducing the operational costs of the monitoring strategy, and offering a cost-effective solution enhancing monitoring system resilience.

Even though Virtual Sensors are a relatively recent research topic, their industrial applications are becoming increasingly relevant. A promising example is the usage of Virtual Sensor in Smart Factories and digitalised manufacturing facilities where devices, machinery and production systems are interconnected to enhance decisionmaking and management [2]. Dobrescu et al. [3] presented the development of services and computing resources for hybrid simulation of Virtual Manufacturing systems, providing a sensor-cloud interface where the end-user can virtualise multiple Virtual Sensors. The adoption of robots and their interactions with humans in Smart Factories was studied by Indri et al. [4], where Virtual Sensors were used to enhance the knowledge of the robot operation.

The applications of Virtual Sensor in the manufacturing industry are very heterogeneous. Maschler et al. [5] estimated the combustion duration on a large gas engine using just the rotational speed as input data. They studied in this work the importance of pre-processing the data for greater accuracy, showing different results for the use of Principal Component Analysis, Fast Fourier Transformation, or just a simple smoothing of the measured rotational speed. Alonso et al. [6] aimed to calculate the cooling power estimation to enable the replacement of the expensive portable measuring system. They used a model based on a Deep Learning architecture that involved data from the chiller's thermodynamic variables (temperature and pressure) and data from the refrigeration circuit (pressure power).

Other studies focus on the malfunctioning of the system instead. Zenisek et al. [7] presented an approach to stabilise and optimise the metal deposition process, merging information from various sources. The ML-based method generates a valid data stream from heterogeneous sources and can mitigate the problem of data merging through the knowledge of domain experts. Finally, they presented a real use case where they estimated the current weld bead height, one of the principal performance indicators of the process. Aware of the problems that could generate a sensor failure and the consequent interruption of information flow, Ilyas et al. [8] introduced a framework capable of finding Internet of Things (IoT) sensors in the surrounding environment and replacing faulty sensors in an automated way. The framework selects the data source based on metadata description, pre-processes historical data, and trains and ranks machine learning algorithms with great results without human intervention. They tested the model predicting the output of a solar power plant.

Tegen et al. [9] proposed Dynamic Intelligent Virtual Sensors (DIVS). The idea was to combine a broader (and not fixed) set of heterogeneous data sources based on Machine Learning to involve the user in the loop. The dynamic part of the concept can be interesting for industrial applications: evaluating the inputs of the Virtual Sensor in terms of information quality (for instance, noise, entropy, etc.) and deciding whether a data source (physical sensor) should be removed or added to the Virtual Sensor. Moreover, the online incremental learning concept was also applied, looking

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

for a Virtual Sensor that relies not only on traditional batch learning but can be dynamically adjusted involving user labelling.

Virtual Sensor can also be applied in multiple areas of the industrial water domain covering the whole water cycle. Djerioui et al. [10] implemented a Virtual Sensor of the chlorine parameter in water treatment plants using the conductivity, dissolved oxygen, suspended solids, and pH variables as input data. The study compares the performance of a Support Vector Machine (SVM) and an Extreme Learning Machine (ELM) ML algorithm, showing better behaviour using ELM. Pattanayak et al. [11] developed a Virtual Sensors to predict in real-time the Chemical Oxygen Demand (COD) of the river Ganga using the input quality parameters of ammonia, total suspended solids, nitrate, pH, and dissolved oxygen. They evaluated different algorithms, finally building a predictive model based on K-Nearest Neighbours, which was used to predict the water quality at the treatment plant's discharge point.

Wastewater treatment is a process where factors such as energy cost or climate footprint are directly related to the process optimisation. Virtual Sensor enables monitoring key parameters in situations where the physical sensors may lead to error due to the constant contact with wastewater. Foschi et al. [12] proposed a Virtual Sensor for the *E. Coli* value for wastewater disinfection using the data from conventional wastewater physical and chemical indicators (such as COD, nitrate, and ammonia). Their research obtains a predictive model trained using an artificial neural network, which could save up to 57% of disinfectant. Pisa et al. [13] showed a Virtual Sensor to predict ammonium and total nitrogen to control effluent violations at the treatment plant using input flow, input ammonium, temperature, and internal recycle flow data. They accomplished the generation of a predictive model using a deep neural network with Long-Short Term Memory neurons, capable of predicting the nitrogen-derived parameters with good accuracy.

#### **3. Methodology**

In this chapter, we focus on the Machine Learning domain, currently one of the most trending areas under the Artificial Intelligence umbrella. Machine Learning aims to develop smart models based on data-driven algorithms that can accurately generate predictions without the explicit necessity to program them for that objective. It can be seen as learning (or training) a function (f) that maps input variables (X) to output variables (Y). Once defined, function f can be used to generalise the learned behaviour and make predictions (Y′) given a new unseen instance of input variables X'. Here, a data-driven approach depends on existing data sets to infer the unknown function f based on parametric or non-parametric algorithms.

More specifically, we propose the use of the regression-type of the Supervised Learning family of algorithms for the Virtual Sensor implementation, as shown in **Figure 2**. These algorithms rely on labelled datasets providing both input variables X

**Figure 2.** *Supervised learning paradigm.* and output variables Y to infer the function f. Moreover, in regression problems, the output variables Y are continuous values instead of the categorical data type required in classification problems.

In this scenario, to successfully conduct a data-driven project, it is of utmost importance to follow a standardised method to translate business problems into tasks, suggest data transformations, or provide means for evaluating the final results and reporting the process, among other objectives. The Cross Industry Standard Process for Data Mining (CRISP-DM) methodology provides this flexible framework [14] and it is organised into seven well-differentiated phases, as shown in **Figure 3**. In this sense, the data mining process is generally cyclic, since it is usually necessary to go back and forth between stages until a valid solution that meets the quality criteria is found. At this point, it is usually a common misunderstanding across the community to consider that the work is finished. Even when the solution is finally deployed and integrated into a production environment, the performance of the underlying models needs to be continuously checked. This is due to the data-driven nature of the concept, which could make a model unfeasible, for example, in those cases where the baseline conditions of the studied process change or evolve over time. This effect makes the learned function f not valid for new scenarios since the relation between input variables X and output variables Y has changed. An innovative solution in this scenario considers an online CRISP-DM model to retrain and validate the predictive models periodically over time.

The CRISP-DM process starts with understanding the business perspective, objectives, and requirements to design the project plan together with the field expert. Once the goals are defined, the initial data are collected and processed with activities channelled to familiarise with them. This first analysis can help identify data quality problems or detect interesting subsets to enable hypotheses for hidden information. Next, the data preparation phase aims to construct the final dataset, which will be used to feed and validate the algorithms. Usually, a significant amount of effort is devoted to this task since it is the most time-consuming and delicate stage that generates one of the most critical outcomes, the training dataset. This is important because data must be consistent and reliable in the Data Science domain since it defines the basis of all the solutions. Data cleaning, feature engineering, feature selection, or data scaling are some of the common processes carried out in this stage and require experienced and creative data scientists for a successful implementation.

**Figure 3.** *Phases of the CRISP-DM process.*

#### *Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

Different Machine Learning algorithms are selected, trained, and calibrated in the modelling phase to achieve optimal performance. The reason behind trying different algorithms is that each one is based on several techniques, has different mathematic fundamentals, and makes different assumptions. Thus, it does not exist one general solution to all the problems and each case needs to be analysed independently, given the fact that underlying data and patterns are different [15].

Then, even though the algorithms are independently evaluated in the modelling phase, during the evaluation phase, the whole model, all the stages, and all the algorithms that appear should be thoroughly assessed and reviewed, as well as the business objectives defined in the initial business understanding phase. Furthermore, a comparison across different models is required to identify the most successful ones. Finally, for the deployment, the model and the knowledge gained are organised and presented in a way that the final customer can understand, use, and maintain.

Usually, the model training, selection and evaluation stages follow a well-established methodology in the Data Science domain, as shown on the left in **Figure 4**. First, in case of parametric Machine Learning algorithms, the model training step is aimed at learning and validating the parameters of the function f (e.g., the coefficients in regression or the weights of a neural network). Separate training and testing datasets must be used across these phases. Otherwise, the model would suffer from overfitting. In this case, a model that reproduces training labels Y during the validation would present a perfect score but would not be able to make good predictions on new data X', since it would not have learned the authentic data patterns.

Splitting data into training and test datasets is known as the Cross-Validation (CV) process, and K-fold CV defines its most basic implementation. The idea is to split the training dataset into k folds, train k models using k-1 different folds (as training datasets) and validate them on the remaining fold. Several methodology variations have been proposed depending on the data type, the basic idea of this concept is shown on the right in **Figure 4**. The final training performance corresponds to the average of the k individual models' performance.

The training step also considers the evaluation and search of the most optimal algorithm hyperparameters given a training dataset. In this context, the hyperparameters are those algorithm parameters that by changing their value, are used to manage the learning process (e.g., the learning rate in Gradient Descent-based approaches). Similar to the CV procedure, several methodologies were used to this end [16]. To mention some, Grid Search proposes an exhaustive search on all hyperparameter combinations given a set of predefined values, while Randomised Search samples any given number of candidates from a parameter space following a specified distribution.

#### **Figure 4.**

*Left: Cross-validation flow in ML models training. Right: K-folds CV approach.*

Finally, to correctly understand the presented Virtual Sensor case study's performance, it is also essential to introduce the validation metrics used to evaluate and compare the models. The following regression metrics are proposed:

• **Mean Absolute Error (MAE) regression loss**: computes the averaged absolute difference (error) between the ground truth and the model predictions.

$$MAE = \frac{1}{N} \sum\_{i=1}^{N} \left| \boldsymbol{\nu}\_i - \boldsymbol{\dot{\nu}}\_i \right| \tag{1}$$

where *N* is the test dataset size, *<sup>i</sup> y* is the ground truth value of the *i*st entry in the test dataset and *<sup>i</sup> y* is the model prediction of the *i*st entry in the test dataset.

• **<sup>2</sup>** *R* **or the coefficient of determination:** computes the proportion of variance explained by the independent variables in the model.

$$R^2\left(\boldsymbol{\mathcal{Y}}\_i, \boldsymbol{\dot{\mathcal{Y}}}\_i\right) = \frac{\sum\_{i=1}^N \left(\boldsymbol{\mathcal{Y}}\_i - \boldsymbol{\dot{\mathcal{Y}}}\_i\right)^2}{\sum\_{i=1}^N \left(\boldsymbol{\mathcal{Y}}\_i - \overline{\boldsymbol{\mathcal{Y}}}\right)^2} \tag{2}$$

where *N* is the test dataset size, *<sup>i</sup> y* is the ground truth value of the *i*st entry in the test dataset, *<sup>i</sup> y* is the model prediction of the *i*st entry in the test dataset, and *y* is the average ground truth value.

#### **4. Case studies**

Virtual Sensors are a flexible and versatile technology that can be found in multiple sectors of the industry. In this section, two real use cases are introduced. The first case is related to mould injection of metallic pieces in the manufacturing industry. The second case is related to the wastewater treatment industry.

#### **4.1 Aluminium mould injection use case**

High-Pressure Die Casting (HPDC) is a process in which a molten metallic alloy is forced under pressure into a locked metal mould cavity, formed by the cover die half and the ejector die half, where a powerful press holds it until the metal solidifies. After solidifying, the ejector die half opens, and the piece is ejected. Finally, the dies are closed again, ready for the next cycle. The casting process is composed of 3 stages:


*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

#### **Figure 5.**

*HPDC Bühler machine with the three machine sensors (counter pressure, head pressure, plunger position) and an image of the mould with the holes of the three mould sensors (temperature, pressure 1 and pressure2).*

The HPDC machine incorporates many sensors to track its activity. However, the mould, which must be redesigned for each new piece or batch, should include additional sensors if its sensorization is needed. As mould sensors are expensive, difficult to instal, and their integration may affect the product's finish, the proposed solution is to replace the in-mould sensors with Virtual Sensor, inferred using external machine sensors data. The Virtual Sensors allow to monitor the process and to apply corrective and preventive actions. These Virtual Sensors are developed using AI and ML methods, enabling a richer and more profound understanding of the process. The HPDC machine used for these experiments, the mould and the sensors are shown in **Figure 5**.

#### *4.1.1 Data*

The experimental campaign is carried out in the Bühler Evolution D53 machine, where aluminium L2630 is injected into tray-shaped moulds. During the lapse of two days, 256 pieces are cast at 13 different machine configurations. For each machine configuration at least 10 samples are manufactured. During each batch, the data from six sensors is recorded at a 2 kHz frequency. The change of the three in-mould sensors, two for the cavity pressure and one for the cavity temperature is shown on the left graphic in **Figure 6**. The temporal evolution of the three machine sensors: plunger position, head pressure and counter pressure is shown on the right plot in **Figure 6**.

The dataset recorded during the first day (132 tests) is used to train the model while the dataset of the second day (126 tests) is used for the test phase.

#### *4.1.2 Methods*

As previously explained, the in-mould sensors are expensive and may affect the shape and result of the final piece. Therefore, in this section, the in-mould sensors: a temperature sensor and two pressure sensors, from now on referred to as pressure 1 and pressure 2, are predicted using machine sensors: plunger position, velocity, head pressure and counter pressure. This part exemplifies the Virtual Sensor forecasting.

#### **Figure 6.**

*Schematic representation of an HPDC shot sleeve and an injection curve with the 3 different phases: Prefill, fill and consolidation.*

Following the CRISP-DM methodology, the following phase is the data preparation, essential to arrange the input data that is later fed to the algorithms. The Pearson's correlation analysis [17] demonstrates that the plunger position, counter pressure, and head pressure are highly correlated. Thus, the products of these three sensors, in pairs, are added as variables: counter pressure x head pressure, counter pressure x plunger position and head pressure x plunger position. This technique enables the use of highly related variables while preserving their influence.

To predict an instant value of any of the three virtual samples defined, the three original sensors, the three products explained above, the derivative of the position (the velocity), and 2 or 5 back samples are given as input, iterating through the results to find the most suitable input parameters for this use case.

The training dataset is split randomly in a stratified way, keeping the same percentage of machine configurations in each. The 80% of data are used for the training dataset, and the remaining 20% are employed in the validation dataset. Finally, all the data are scaled using the *MinMaxScaler,* which transforms the data into the 0−1 range. Training data are first fitted and, afterwards, train and validation datasets are converted.

The CV grid search methodology with the aforementioned K-fold split is implemented to train an evaluate different models based on the following ML algorithms:


*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

The most important parameters are the number of neighbours used to predict each data point and the weight function used to determine the importance given to the neighbour data [20].

• **SVR:** Support Vector Regression algorithm is a variation of the classifier Support Vector Machine but adjusted for regression problems. Instead of separating data into classes by means of a hyperplane, the data are adjusted to the mentioned hyperplane with a certain degree of tolerance given (ε), where the best fit is the hyperplane with the maximum number of points. Therefore, the hyperparameter ε needs to be tuned, together with the C parameter, which determines de regularisation applied to the algorithm [21].

#### **4.2 Wastewater treatment plant use case**

The Activated Sludge Process (ASP) [22] is usually a critical stage in a Wastewater Treatment Plant (WWTP) and has a direct impact on the effluent water quality as well as on the greenhouse gas (GHG) emissions, demanding considerable quantities of energy. Specifically, the ASP Nitrification step is the biological process of converting ammonia to nitrate in wastewater tanks using aerobic autotrophic bacteria. The process requires proper working conditions such as enough biomass concentrations, specific environmental conditions, a minimum residence time to process the water, and a great amount of oxygen. Any variation in these conditions directly affects the amount of ammonia being treated, thus in the effluent water quality.

In this scenario, the airflow system controls the oxygen injection, one of the key processes with the highest resource consumption and impact in the treatment plant. The water operators manage the air blowers to optimise the process (i.e., the effluent water quality reaches the expected criteria, while energy consumption and GHG emissions are minimised), thus the use of sensors to monitor in real-time these quality parameters enable an online control. Ammonia is another key parameter that needs to be adequately treated. In case its monitoring gathers non-real values, the blower's management is directly influenced, resulting in elevated costs, climate impacts and issues in the effluent water quality. Implementing a Virtual Sensor enables continuous monitoring of the ammonia parameter which enables the: i) detection of sensors' malfunction or drift in measurements (due to the constant contact with wastewater), and ii) implementation of maintenance actions without the need to stop dependent systems, therefore ensuring correct and continuous WWPT operations.

This use case focuses on the WWTP ASP treatment tanks within its corresponding lanes. It operates in the following manner:


#### *4.2.1 Data*

The data available comprises historical information on three sensors located inside the treatment lane, as shown in **Figure 7**:


These sensors extract the information every 5 minutes, and the dataset spans two years of registers, with regular and irregular values that need to be checked and filtered. Furthermore, due to the sensors being located at different parts of the reaction tank and the water taking time to flow between the inner tanks, it is required to study the time correlation between sensors.

The train set contains 80% of the data, and the test set the remaining 20%. This second set includes the latest data gathered.

#### *4.2.2 Methods*

The first phase of the CRISP-DM cycle (Business Understanding) covers the analysis of the problem and the definition of the data-driven approach. The approach focuses on predicting the real-time value of the ammonia parameter using the past and real-time values of the DO and water flow variables, and the past values of ammonia.

Following the CRISP-DM methodology, data are preprocessed, cleaned and new variables are created. To decide which timestamps are used as input features for the model, it is crucial to understand the correlation between them and the objective variable. Pearson's correlation, autocorrelation and cross-correlation techniques [23] are applied to decide the features.

The autocorrelation plot for ammonia is shown on the left graphic in **Figure 8**. The most important lags (previous values) are the ones nearest to the present time, and past hour lags are used as input variables for the model. The cross-correlation among the sensors' data also shows the most important lags. The cross-correlation between the ammonia and water flow variables is shown on the right graph in **Figure 8**, indicating the correlation of any lag from the water flow sensor with the present value of the ammonia sensor. The most important lags are from the previous three hours (−30 lags \* 5 minutes per lag), which coincides with the time the water spends moving inside the

**Figure 7.** *Wastewater treatment plant lane. Visual sensor location.*

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

#### **Figure 8.**

*Left: Autocorrelation for the ammonia parameter. Right: Cross-correlation between the ammonia and water flow parameters.*

reaction tank. The DO lag selection follows the same strategy, but in this case, the present values are the most related.

To use the data of the different water parameters, the registers need to have a similar scale of values, so the weight assigned to a feature by the predictive model is not affected by higher or lower values. In this case, the standard score (or Z-score) [24] is used, setting the mean to 0 and scaling the variance to 1.

In the final iterations of the CRISP-DM process, a Long-Short Term Memory (LSTM) [25]. Artificial Neural Network [26] algorithm has been used to deal with the process nonlinearities and multiple input time series data, and ultimately, to implement the Ammonia Virtual Sensor. LSTM is a Recurrent Neural Network (RNN) [27] that has feedback connections and can process data sequences such as videos, text, or time series. The inner structure of the LSTM stores the output activations from the different layers of the network. Then, the next time an input is fed, the previously obtained outputs are used as inputs, concatenating the stored information with the new input thus simulating some kind of memory system. The LSTM differentiates from other types of RNNs in the capability of storing multiple iterations of output activations without losing information through time, being the best reason to use this architecture when numerous lags are used. To generate an LSTM architecture, several parameters need to be considered and iterated over. The most important ones are:


To decide the best algorithm hyperparameters (e.g. neural network layer and neurons per layer), several training iterations are done using the Cross-Validation grid search technique over the training dataset to ensure the model is not overfitting. Afterwards, several combinations are compared to find out which combination obtains better results in the test set. The scoring metrics used are the MAE and R<sup>2</sup> .

#### **5. Results and discussion**

#### **5.1 Aluminium mould injection use case**

Using 2 and 5 back samples as additional input variables for the algorithms does not improve the results. Neither the R<sup>2</sup> score nor the MAE score nor the performance improve, but the additional samples hugely increase the prediction time and computational power needed. Therefore, only the same instant sensors' samples, their interactions and the velocity are considered as input variables of the final model.

The predictions of all ML algorithms used in each Virtual Sensor development compared with the real sensor values (black lines) are shown in **Figures 9**–**11**. For better visual clarity, only four cycles are depicted for each sensor. The prediction and the real values of the first pressure sensor are shown in **Figure 9**. The SVR algorithm (light blue line) and KNN (yellow line) are the algorithms with the lowest R2 error and highest MAE error for all three sensors. On the contrary, the other two algorithms, Decision Tree (red line) and Random Forest (dark blue line) both present higher R2 errors and lower MAE errors for all three sensors. These metrics can be seen in **Table 1**.

The pressure 2 virtual sensor predictions compared with the real values are shown in **Figure 10**. The results are generally worse in this case than in the pressure 1 sensor. Even though the Random Forest algorithm adjusts more closely to the real sensor, the third and fourth cycle predictions show an example of a fair disparity in the results. It should be kept in mind that the graphic only depicts 4 sample cycles and not the totality of the data predicted. The metrics of the predictions can be found in **Table 2**.

#### **Figure 9.** *VS performance comparison for pressure 1 variable simulation.*

#### **Figure 10.**

*VS performance comparison for pressure 2 variable simulation.*

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

#### **Figure 11.**

*VS performance comparison for temperature variable simulation.*


#### **Table 1.**

*Performance of each algorithm for the temperature sensor in both datasets.*

The results for the temperature sensor are shown in **Figure 11**. In this case also, the SVR predicts almost constant values during the consolidation stage. Unlike the other Virtual Sensor, the prediction during the prefill stage fails to fit closely to the real sensor in all the algorithms.

**Tables 1**–**3** show the main results in R2 and MAE for the three Virtual Sensor models and for each algorithm employed. The algorithm with the highest R<sup>2</sup> error and the lowest MAE error is the Random Forest regressor for all three in-mould sensors. Therefore, Random Forests with the mentioned fine-tuned hyperparameters is chosen as the best algorithm. Following the train/test methodology explained beforehand, the performances of the test dataset are also shown in **Tables 1**–**3**.

Both the temperature and pressure 1 sensors obtain high R2 errors and low MAE errors for the Virtual Sensors predictions. Pressure 2 also gets a high R<sup>2</sup> , but high overfitting behaviour can be assumed due to the lower values in the validation and test dataset contrary to the train errors. To illustrate the distribution of the predicted VS values, the counts of the real values versus the predicted ones for the three in-mould


#### **Table 2.**

*Performance of each algorithm for the pressure 1 sensor in both datasets.*


#### **Table 3.**

*Performance of each algorithm for the pressure 2 sensor in both datasets.*

sensors are shown in **Figure 12**, using the test dataset. The prediction of pressure 1 is more accurate than pressure 2. In this figure, it can also be observed that although the prediction of pressure 2 is far from making a good prediction, it is worth noting that

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

#### **Figure 12.**

*Heat map of the predicted value vs. real values of each virtual sensor. The colourmap indicates the frequency of repetition.*

its errors are mostly due to an erroneous prediction around 0 values, the 'stand-by' value. For the temperature sensor, most values are inside an error of 50 degrees.

#### **5.2 Wastewater treatment plant use case**

The train and test processes resulted in the three LSTM architectures outputting the best results are shown in **Table 4**, displaying the scores for the final test set. Similar performances are achieved, but the third model presents the highest R2 . Therefore, the selected architecture is the last one, with 3 hidden layers and 25 neurons on each layer.

The model's response also needs to be validated in situations with a high increase in the ammonia parameter. The Virtual Sensor acting in two cases where the predictions correctly follow the increase of ammonia is shown in **Figure 13**. As it can be seen, the error also increases in these situations since the model is predicting unusual conditions.

To detect possible flaws in the model at a more individual level, the evaluation of registers is done by means of a scatter plot, as shown in **Figure 14**. It compares the predicted and real values, plotting the regression line of all the values to give a general perspective of the overall correlation. It can be observed that, within the predictions, there are no individual registers with a great error, but the general error detected previously is confirmed here. The predictions are lower than the real values, and that is a general flaw of the model trained.


**Table 4.**

*LSTM architectures and their scoring using the final test set.*

**Figure 13.**

*Real ammonia value, in blue, versus predicted virtual sensor value, in orange. Two cases of a sudden increase in ammonia.*

**Figure 14.** *Scatter plot of the predictions, comparing the individual predictions with the real values.*

#### **6. Conclusion**

Artificial Intelligence is becoming a key element in the 'must have' technology stack for industries that embrace the challenges and opportunities of the Industry 4.0 paradigm. Smart exploitation of the production chain parameters and data is key for informed decision-making that can impact relevant industrial Key Performance Indicators.

This chapter focuses on a novel approach that utilises Artificial Intelligence and data-driven solutions to expand the production process knowledge base and provide more resilient and robust monitoring systems. The so-called Virtual Sensors allow the creation of indirect measurements of process variables, creating virtual replicas of the real sensors that can detect and mitigate sensors drifts, malfunctions, inaccuracies, etc. Furthermore, new parameters that are difficult or impossible to measure can be estimated by combing inputs of different sensors by means of AI-driven models.

The use of standard methodologies and good practices is considered when describing how the Cross Industry Standard Process for Data Mining can be put in place for developing Virtual Sensor for industrial applications. Additionally, two use cases are presented and described: High Pressure Die Casting (HPDC) and Wastewater Treatment Plant. In the HPDC use case, three Virtual Sensors are implemented to predict two different pressures and the temperature inside the mould cavity. The final models based on Random Forest algorithms offer an R2 error of 0.903 for the temperature sensors, 0.820 for the pressure 1 sensor and 0.071 for the pressure 2 sensor. The predicted curves follow the real trend, especially for the pressure 1 and temperature sensors, positioning the Virtual Sensors as a trustworthy technology to avoid the implementation of cavity sensors that increase the cost and can affect the shape of the final piece.

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

In the Wastewater Treatment Plant case, a Virtual Sensors is implemented to improve and ensure the continuous monitoring of the Ammonia parameter in the Activated Sludge Process stage. In this way, the dependence on online real sensor measurements is considerably reduced, which enables an uninterrupted WWTP optimal control. Long-Short Term Memory Deep Neural Network architectures are introduced as algorithms capable to deal with non-linear process behaviours, showing a Deep Learning architecture that correctly adapts to the needs of time series data, which is a good match for Virtual Sensors development. The model benchmarks show a low predictive error, offering a R<sup>2</sup> score of 0.975, thus demonstrating the capacities of such technologies in these complex scenarios.

#### **Acknowledgements**

This work was financially supported by the Catalan Government through the funding grant ACCIÓ-Eurecat.

#### **Author details**

Maddi Etxegarai\*, Marta Camps, Lluís Echeverria, Marc Ribalta, Francesc Bonada and Xavier Domingo Eurecat, Centre Tecnològic de Catalunya, Unit of Applied Artificial Intelligence, Barcelona, Spain

\*Address all correspondence to: maddi.etxegarai@eurecat.org

© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### **References**

[1] Martin D, Kühl N, Satzger G. Virtual Sensors. Business and Information Systems Engineering. 2021;**63**:315-323. DOI: 10.1007/s12599-021-00689-w

[2] Pech M, Vrchota J, Bednář J. Predictive maintenance and intelligent sensors in smart factory: Review. Sensors. 2021;**21**:1470. DOI: 10.3390/s21041470

[3] Dobrescu R, Merezeanu D, Mocanu S. Process simulation platform for virtual manufacturing systems evaluation. Computers in Industry. 2019;**104**:131- 140. DOI: 10.1016/j.compind.2018.09.008

[4] Indri M, Lachello L, Lazzero I, Sibona F, Trapani S. Smart sensors applications for a new paradigm of a production line. Sensors. 2019;**19**(3):650. DOI: 10.3390/s19030650

[5] Maschler B, Ganssloser S, Hablizel A, Weyrich M. Deep learning based soft sensors for industrial machinery. Procedia CIRP. 2021;**99**:662- 667. DOI: 10.1016/j.procir.2021.03.115

[6] Alonso S, Morán A, Pérez D, Reguera P, Díaz I, Domínguez M. Virtual sensor based on a deep learning approach for estimating efficiency in chillers. In: International Conference on Engineering Applications of Neural Networks. Cham: Springer; 2019. pp. 307-319. DOI: 10.1007/978-3-030-20257-6\_26

[7] Zenisek J, Gröning H, Wild N, Huskic A, Affenzeller M. Machine learning based data stream merging in additive manufacturing. Procedia Computer Science. 2022;**200**:1422-1431. DOI: 10.1016/j.procs.2022.01.343

[8] Ilyas EB, Fischer M, Iggena T, Tönjes R. Virtual sensor creation to replace faulty sensors using automated machine learning techniques. In: 2020 Global Internet of Things Summit (GIoTS). Dublin, Ireland: IEEE; 2020. pp. 1-6. DOI: 10.1109/ GIOTS49054.2020.9119681

[9] Tegen A et al. Collaborative sensing with interactive learning using dynamic intelligent virtual sensors. Sensors. 2019;**19**(3):477. DOI: 10.339/s19030477

[10] Djerioui M, Bouamar M, Ladjal M, Zerguine A. Chlorine soft sensor based on extreme learning machine for water quality monitoring. Arabian Journal for Science and Engineering. 2019;**44**:2033-2044. DOI: 10.1007/ s13369-018-3253-8

[11] Pattanayak AS, Pattnaik BS, Udgata SK, Panda AK. Development of chemical oxygen on demand (COD) soft sensor using edge intelligence. IEEE Sensors Journal. 2020;**20**:14892-14902. DOI: 10.1109/JSEN.2020.3010134

[12] Foschi J, Turolla A, Manuela A. Soft sensor predictor of E. coli concentration based on conventional monitoring parameters for wastewater disinfection control. Water Research. 2021;**191**:116806. DOI: 10.1016/j. watres.2021.116806

[13] Pisa I, Santín I, Lopez J, Morell A, Vilanova R. ANN-based soft sensor to predict effluent violations in wastewater treatment plants. Water. 2019;**19**(6):1280. DOI: 10.3390/s19061280

[14] Wirth R, Hipp J. CRISP-DM: Towards a standard process model for data mining. In: Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining. Vol. 1. NY, USA: 2000. pp. 29-39

*Virtual Sensors for Smart Data Generation and Processing in AI-Driven Industrial Applications DOI: http://dx.doi.org/10.5772/intechopen.106988*

[15] Wolpert DH, Macready WG. No-free-lunch theorems for optimization. IEEE Transactions on Evolutionary Computation. 1995;**1**:67. DOI: 10.1109/4235.585893

[16] Claesen M, De Moor B. Hyperparameter search in machine learning. The XI Metaheuristics International Conference. 2015:1-4. DOI: 10.48550/ARXIV.1502.02127

[17] Benesty J et al. Pearson correlation coefficient. In: Noise Reduction in Speech Processing. Berlin, Heidelberg: Springer; 2009. pp. 1-4. DOI: 10.1007/978-3-642-00296-0\_5

[18] Quinlan JR. Induction of decision trees. Machine Learning. 1986;**1**(1): 81-106. DOI: 10.1007/BF00116251

[19] Ho TK. Random decision forests. In: Proceedings of 3rd International Conference on Document Analysis and Recognition. Montreal, QC, Canada: IEEE; 1995. pp. 278-282. DOI: 10.1109/ ICDAR.1995.598994

[20] Altman NS. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician. 1992;**46**(3):175-185. DOI: 10.1080/00031305.1992.10475879

[21] Vapnik V. The nature of statistical learning theory. Springer Science & Business Media. 1999. DOI: 10.1007/978-1-4757-2440-0

[22] Metcalf L, Eddy HP, Tchobanoglous G. Wastewater Engineering: Treatment, Disposal, and Reuse. Vol. 4. New York: McGraw-Hill; 1991

[23] Rabiner LR, Gold B, Yuen CK. Theory and application of digital signal processing. IEEE Transactions on Systems, Man, and Cybernetics.

Feb 1978;**8**(2):146-146. DOI: 10.1109/ TSMC.1978.4309918

[24] Gross E. Practical statistics for High Energy Physics. CERN Yellow Reports: School Proceedings. 2017;**4**:165-186. DOI: 10.23730/CYRSP-2017-004.165

[25] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation. 1997;**9**(8):1735-1780. DOI: 10.1162/neco.1997.9.8.1735

[26] McCulloch WS, Pitts W. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics. 1943;**5**(4): 115-133. DOI: 10.1007/BF02478259

[27] Rumelhart DE, Hinton GE, Williams RJ. Learning internal representations by error propagation. Parallel Distributed Processing: Explorations in the Microstructure of Cognition. 1986;**1**:318-362. DOI: 10.5555/104279.104293

### **Chapter 7**
