Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with Force Feedback

*Pingguo Huang and Yutaka Ishibashi*

## **Abstract**

By using remote robot systems with force feedback, we can largely improve the efficiency and accuracy of work among the systems over a network. In such a system, a human can operate a remote robot by manipulating a haptic interface device while monitoring the movement of the robot arm and perceiving force applied to an object touched/moved by an arm of the robot having a force sensor. The remote robot systems with force feedback are expected to be used in many areas such as outer space, deep sea, nuclear power plants, and disaster areas, which humans cannot enter easily. In these situations, three types of cooperation among humans and robots (that is, between humans and robots, between humans, and between robots) are highly demanded. In this chapter, we introduce our remote robot systems with force feedback and describe the three types of cooperation among humans and robots in the systems. We also explain QoS (Quality of Service) control and stabilization control as our challenges and solutions for effective cooperation. Furthermore, we discuss future directions of the cooperation.

**Keywords:** remote robot system, force feedback, haptic interface device, cooperation, QoS control, stabilization control

## **1. Introduction**

In recent years, many researchers pay attention to remote robot systems with force feedback [1–6]. In such a system, a human can operate a remote robot by manipulating a haptic interface device while monitoring movement of the robot arm with a video camera and perceiving force from an object touched/moved by an arm of the robot having a force sensor. Since the user can touch and feel the shape, weight, and softness of the object, the efficiency and accuracy of remote operation can be largely improved [7]. Therefore, the remote robot systems with force feedback are able to be used in various areas such as outer space, deep sea, nuclear power plants, and disaster areas, in which humans cannot enter easily. In these applications, it is important to make cooperation among humans and robots with force feedback efficiency. We focus on the cooperation among humans and robots with force feedback in this chapter.

When the control information (i.e., force and/or position) is transmitted over a network that does not guarantee QoS (Quality of Service) [8], owing to network problems such as network delay, delay jitter, and packet loss, the QoE (Quality of Experience) [9] may be seriously deteriorated. Also, unstable phenomena such as vibrations may occur [10–12], and cooperation may not be well. To solve the problems, we need to carry out QoS control and stabilization control. QoS control alleviates the influences of network delay, delay jitter, and packet loss, and stabilization control suppresses instability phenomena such as vibrations of the robot and haptic interface device. We mainly focus on the problems and their solutions for effective cooperation among humans and robots with force feedback.

In this chapter, first, we explain the cooperation among humans and robots with force feedback in Section 2. Next, we introduce the remote robot systems with force feedback in Section 3. Then, we outline the problems to be solved in the cooperation among humans and robots with force feedback in Section 4 and explain our solutions for effective cooperation in Section 5. Finally, we discuss the future directions of the cooperation among humans and robots in Section 6 and we conclude the chapter in Section 7.

## **2. Cooperation among humans and robots with force feedback**

In a remote robot system with force feedback, a human can operate a haptic interface device to control/support a remote robot while watching the video received from the remote terminal. From the haptic interface device, position information which is used to control/support remote robot is transmitted to the robot, and the position information of robot arm and force information sensing by force sensor attached to the robot is transmitted to the haptic interface device for outputting reaction force (see **Figure 1**, in which a toggle clamp hand is attached to the force sensor. We can attach various types of hands).

The remote robot systems with force feedback can be used for cooperation among humans and robots. **Figure 2** shows examples of cooperation between the two remote robots. In **Figure 2(a)**, the two remote robots collaboratively carry and move an object (wooden stick) from one position (initial position) to another position (destination). In **Figure 2(b)**, one robot (robot arm 1) moves an object (wooden stick) toward another robot (robot arm 2), and hands over the object to robot arm 2. In **Figure 2(c)**, the two remote robots carry an object (cardboard box) by putting the object from both sides and moving the object from one position to another position. Cooperation can be widely used in many areas, and several types of cooperation exist. As shown in **Figure 3**, the cooperation among humans and robots can be grouped into cooperation between human-human, humanrobot, and robotrobot. In this section,

**Figure 1.** *Remote robot system with force feedback.*

*Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

#### **Figure 2.**

```
Cooperation images between two robots.
```
we introduce expected applications using the remote robot systems with force feedback and explain the types of cooperation.

## **2.1 Expected applications**

The cooperation among humans and robots in the remote robot systems with force feedback is expected to be used in various areas such as dangerous areas in which humans cannot enter easily (for example, outer space, deep sea, and

#### **Figure 3.** *Cooperation among humans and robots.*

radioactively contaminated areas), disaster areas (disaster rescue and relief), and remote medicine.

## *2.1.1 Applications in dangerous areas*

It is dangerous for humans to work in dangerous areas such as outer space, deep sea, and radioactively contaminated areas. We can use robots to work in such areas instead of humans entering the areas while the humans control the robots in safe areas.

## *2.1.2 Disaster areas (rescue and relief)*

After the outbreak of disasters such as earthquakes and concentrated downpours, rescue and relief are the most important. However, it may be difficult for humans to enter the hard-hit areas. We can control remote robots to confirm disaster situations for making effective rescue plans. Also, humans in safe areas can control remote robots to enter the areas for rescue and relief.

## *2.1.3 Remote medicine*

There exist large health disparities in different regional/national areas. Using remote robot systems with force feedback may be an effective method to solve the problem. The systems can be used for remote surgery and remote rehabilitation. Also, they can be employed for remote surgery training for medical interns.

## *2.1.4 Other areas*

The remote robot systems with force feedback are also expected to be used for various applications such as industrial factories and home delivery in other areas. For *Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

industrial factories, humans can control remote robots to enter extreme environments (for example, high temperature and/or high-pressure environments) or limited working areas which are difficult for humans for production work or inspection and maintenance. For homedelivery, humans can remote control drones or movable robots to deliver packages.

In these applications, it is difficult to conduct work with only robots if the situations and environments are unknown in advance and may be always changing. Therefore, human's support is needed [6]. This means that we need robots to help humans, and robots also need human's support. That is, in the applications, cooperation between a human (local user) and a robot (remote robot) in each system, that between humans, that between robots (remote robots), and that among humans and robots (remote robots) are needed. We explain the three types of cooperation in the next subsection.

#### **2.2 Types of cooperation**

As described in the previous subsection, robots can help humans, and robots also need humans'support. We explain the cooperation among humans and robots in this subsection.

1.Cooperation between human and robot

In this type of cooperation, a human operates a haptic interface device to control/support a remote robot. Instead of the human, a robot does the work at a remote side and sends back the information about reaction force sensed by force sensor, and the information about a remote environment by other sensors such as video cameras and microphones. The information can help the human to know the remote situation. Also, it is difficult for the robot to do work in a complex environment independently. Therefore, human's support is needed for the robot. The human can control or adjust the movement of the remote robot to help the robot conduct the work.

2.Cooperation between robots

In dangerous areas or disaster areas, wireless communication can be an effective method for communication between humans and robots. However, wireless communication is largely affected by weather, obstacles, and distances; this means that large network delays or network troubles may occur. These problems may make humans difficult to control or support remote robots to deal with abrupt changes in robots' positions. In this case, since humans require time to support the remote robots, cooperation between robots using force sensors is needed for quick response.

3.Cooperation between humans

In this cooperation, multiple humans operate haptic interface devices to control remote robots to do work. In this case, to do the work smoothly and effectively, cooperation between humans is important. In cooperation, humans need to start to operate remote robots at the same time at the beginning of the cooperation work, keeping almost the same movement speed in the cooperation. Also, when multiple humans operate the remote robots to move an object, efficient cooperation may be needed for position's fine adjustment to the goal position.

Therefore, in cooperation, humans need to transmit their wills (e.g., the movement direction and movement speed) to each other for efficient cooperation.

4.Cooperation among humans and robots

For work that cannot be conducted by only one remote robot, multiple robots may be needed. The robots should be operated by multiple humans. In this case, we need the above three types of cooperation.

## **3. Remote robot systems with force feedback**

In this section, we introduce the remote robot systems with force feedback constructed in our study.

## **3.1 System configuration**

The configuration of the remote robot systems with force feedback is shown in **Figure 4**. Each system consists of a master terminal and a slave terminal. Each terminal has two PCs; that is, PC for haptic interface device and PC for video at the master terminal, and PC for industrial robot and PC for video (web camera) at the slave terminal. At the master terminal, a haptic interface device (3D Systems Touch [13]) having 3 DoF (Degree of Freedom) is connected to PC for haptic interface device. At the slave terminal, an industrial robot is connected to PC for industrial robot, and a web camera (produced by Microsoft Corp., and the video resolution is 1920 1080 pixels) is connected to PC for video. The industrial robot has a 6 DoF robot arm (RV-2F-D [14] by Mitsubishi Electric Crop.), a robot controller (CR750-Q [14]), and a force sensor (1F-FS001-W200 [15]) attached to the tip of the robot arm. The force sensor is connected to the robot controller through a force interface unit

**Figure 4.** *Configuration of remote robot systems with force feedback.* *Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

(2F-TZ561 [15]). Also, a toggle clam hand/a electric hand is further linked to the force sensor.

In each system, the master terminal inputs the position information of the haptic interface device and calculates and outputs the reaction force for the haptic interface device every millisecond. The master terminal also transmits the position information to the slave terminal by UDP (User Datagram Protocol). At the slave terminal, based the position information received from the master terminal, a position vector is calculated and sent to the industrial robot every 3.5 milliseconds by the real-time control function [16]. The force information sensed by the force sensor is transmitted to the master terminal, and the master terminal calculates the reaction force outputted by the haptic interface device based on the received force information every millisecond as described previously.

At the master terminal, the reaction force *F*ð Þ <sup>m</sup> *<sup>t</sup>* outputted through to the haptic interface device at time *t* (*t* ≥ 1) is calculated as follows:

$$\mathbf{F}\_{t}^{(\mathbf{m})} = \mathbf{K}\_{\text{scale}}^{(\mathbf{F})} \mathbf{F}\_{t-1}^{(s)} \tag{1}$$

where *K*ð Þ **<sup>F</sup> scale** is a force scale which is set to 1 in this chapter, and *<sup>F</sup>*ð Þ**<sup>s</sup>** *<sup>t</sup>*�**<sup>1</sup>** denotes the force received from the slave terminal (note that we use only the three axes (the *x*-, *y*-, and *z*- aexs) of force here). Furthermore, since the maximum force outputted through the haptic interface device is 3.3 N [13], the reaction force is set to 3.3 N when the calculated force is larger than 3.3 N.

At the slave terminal, the position vector *S<sup>t</sup>* of the industrial robot outputted at the time *t* (*t* ≥ 2) is calculated as follows:

$$\mathbf{S}\_{t} = \mathbf{K}\_{\text{scale}}^{(\text{P})} \mathbf{M}\_{t-1} + \mathbf{V}\_{t-1} \tag{2}$$

where *K*ð Þ **<sup>P</sup> scale** is a coefficient for mapping of workspace, *Mt* is the position vector of the haptic interface device received from the master terminal at time *t*, *Vt*ð Þ ¼ *St* � *St*�**<sup>1</sup>** is the velocity vector of the industrial robot, and *Vt* j j *V* **max** , where *V* **max** is the maximum movement velocity (*V* **max** = 5 mm/s [16] in this chapter) to operate the robot arm safely.

### **3.2 Cooperation methods**

The remote robot systems with force feedback are used for serval types of cooperation, and we here introduce the cooperation methods.

#### *3.2.1 Human-robot*

As described in Subsection 2.2, the cooperation between human and robot, a human operates a haptic interface device to control/support a remote robot. In this type of cooperation, a single remote robot system is used, and the human can feel the reaction force sensed by the force sensor attached to the remote robot arm. By feeling the reaction force and watching video received from remote side, the human can support/control the remote robot by sending position information which is used to move the remote robot.

We conduct the cooperation between human-robot in our previous work [12, 17]. In Ref. [12], the authors handle work of pushing a ball and identifying what kind of

ball is pushed according to the softness without watching video. In Ref. [17], a human can manipulate a robot arm with a pen to write characters by using a haptic interface device, and the authors make a comparison among different types of stabilization control which we will explain in Section 5.

#### *3.2.2 Robot-robot*

As described previously, the remote robot systems can be used for the area such as disaster areas and deep see, in which communication failure, and unforeseen events and abrupt change of position may occur frequently. In this case, robots cannot receive the support/control information from humans. The cooperation between robot and robot can be used to solve the problem by working independently without humans'support. Also, the cooperation can be used under the ordinary environment. In order to conduct the cooperative work independently, the robots can cooperate with each other according to the force information sensed by force sensors and/or by transmitting position information transmitted between the robots.

In Ref. [18], the authors deal with cooperation work in which an object (a wooden stick) is carried together by using the two remote robot systems with force feedback. In the cooperation, according to the force sensed by force sensor, one robot adjusts the position in the direction of reducing the force applied to the object to cooperate with another robot. We also suppose that mobile robots suddenly move up and down largely and employed the enhanced control of the robot position control using force information to against sudden and large position change in the vertical direction [19]. We further investigate the effects of the enhanced control by employing one axis among the six axes for the robot movement, and we regard the industrial robots as mobile robots (i.e., pseudo-mobile robots). In Ref. [18], the two remote robot systems are used to do collaborative work of carrying an object (a wooden stick) together. In the cooperation, each slave terminal (robot side) also sends the position information of robot arm to that of the other system (robot side). In Ref. [20], two humans operate the two remote robot systems to do cooperative work of carrying an object. In the cooperation, the position information is transmitted between robots to make the cooperation smooth.

#### *3.2.3 Human-human*

When multiple humans control remote robot systems with force feedback to do work, the cooperation between humans is important. In this type of cooperation, communication between humans is necessary. Humans can transmit their wills (e.g., the movement direction and movement speed) to each other by using sound, video, and/or force information (i.e., position information which can be used to calculate the reaction force outputted through a haptic interface device) and make the cooperation smooth.

In Ref. [21], one of two humans operate a remote robot to carry an object together with the other human by using a haptic interface device. In the cooperation, the position information of the haptic interface device is transmitted to calculate the reaction force outputted through haptic interface device. In Ref. [22], one human operates a haptic interface device to control a remote robot to carry and move an object (a wooden stick), and hand over the object to another remote robot which is controlled by another human.

*Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

#### *3.2.4 Humans-robots*

For work which cannot conduct successfully by only one remote robot system, multiple remote systems with force feedback and multiple humans may be needed. When multiple remote robot systems with force feedback are employed to do collaborative work, cooperation among humans and robots is needed. That is, communications between human and robot, between robot and robot, and between human and human are required.

## **4. Problems to be solved**

In order to achieve effective cooperation, many problems should be solved. In this section, we explain the problems by grouping them into QoS problems and stabilization problems.

### **4.1 QoS problems**

As described in Section 1, when the control information (i.e., force and/or position) is transmitted over a network which does not guarantee QoS like the Internet, owing to the network problems such as the network delay, delay jitter, and packet loss, the following problems (e.g., QoE deterioration and poor cooperation) may occur.

1.Fluctuation of reaction force

As the network delay increases, the reaction force outputted through the haptic interface device may become larger. That is, it is more difficult for a human to operate the device.

2.Out of temporal synchronization (known as media synchronization [23])

Owing to the network delay and delay jitter, media synchronization (i.e., the temporal relationship of media units (MUs), which are information units for media synchronization [24]) problems may occur. The problems of temporal synchronization can be grouped into problems of intra-stream synchronization, inter-stream synchronization, and inter-destination (or group) synchronization [23].

3.Out of spatial synchronization

In the cooperation, robots need to move/operate an object at the same altitude, same angle, and same speed. However, humans and robots are at different locations, out of spatial synchronization (i.e., robots move/operate the object in different altitude, different angle, and different speed) may occur, and cooperation may not be well. This is because if spatial asynchrony occurs, the force applied to the operating object becomes large, and the object may be damaged by the large force. That is, we need to keep temporal and spatial synchronization high.

To solve the problems, we need to carry out QoS control, and we will introduce the QoS control in Subsection 5.1.

#### **Figure 5.**

*Control loops in remote robot systems with force feedback.*

#### **4.2 Stabilization problems**

In the remote robot system with force feedback, as the network delay increases, the reaction force becomes larger, and instability phenomena such as vibrations of the robot and haptic interface device may occur more frequently [12]. Furthermore, when multiple remote robot systems are used for cooperation, there are multiple loops caused by communication and force sensors in the systems (see **Figure 5**), and there exist close relationships among loops. Therefore, instability phenomena may become more often and more complex.

## **5. Solutions for effective cooperation**

In this section, we introduce solutions which can be used to solve the problems described in Section 4 for effective cooperation.

#### **5.1 QoS control**

In order to solve the QoS problems, we need to carry out QoS control. Various types of QoS control have been proposed [5]. We explain serval types of QoS control which can be used to solve the QoS problems for the cooperation among remote robot systems.

#### 1.Temporal synchronization control

Temporal synchronization control can be used to solve the problems caused by the network delay and delay jitter [23] and following types of control are included. In the control, we suppose that the global synchronization clocks [25], which have the same value and advancement, are used at all the terminals for

simplicity. We can adjust the clocks by Network Time Protocol (NTP) [26] or Global Positioning System (GPS) easily.

a. Intra-stream synchronization control

The control is used to keep the timing relation between MUs such as video pictures and voice packets in a single media stream. Serval types of intrastream synchronization control such as Skipping [24], Virtual-Time Rendering (VTR) [24] have been proposed.

In the remote robot systems with force feedback, the control can be used to keep the timing relation of each media stream such as haptic/force media and video for each system.

b. Inter-stream synchronization control

The inter-stream synchronization control is used to preserve the temporal relationship among multiple media streams. VTR can also be used for inter-stream synchronization control. Under the control, one media stream is selected as the master stream, and the others are handled as slave streams. For the master stream, we carry out only the intra-stream synchronization control, and we exert the inter-stream synchronization control which adjusts the output timing of each slave stream to that of the master stream after carrying out the intra-stream synchronization control.

In the remote robot systems, haptic/force streams and video may be transmitted between the master and slave terminals in the cooperation between human-robot/humans-robots; the haptic stream and voice may be transmitted among the master terminals (i.e., humans'sides) in the cooperation of humans-humans. In these cases, multiple streams are transmitted among the terminals, and we need to carry out the interstream synchronization control. Normally, the most important stream such as the haptic/force stream can be selected as the master stream under the control.

c. Inter-destination synchronization control

The inter-destination synchronization control can be employed to output each MU simultaneously at different terminals (destinations). When multiple remote robot systems with force feedback are used to do collaborative work, if each MU cannot be outputted simultaneously at different terminals, the cooperation may not be well because humans may watch different displayed video images of cooperation work and feel different reaction force. Therefore, we need to carry out the interdestination synchronization control to improve the efficiency of cooperation work. Also, the control can be used for spatial synchronization.

2.Adaptive reaction force control (adaptive elasticity control)

In each remote robot system with force feedback, the reaction force of the haptic interface device is normally calculated from the force sensed by force sensor (see Eq. (1)). When the master terminal receives position information from the slave terminal, the spring-damper model [27] can be used to calculate the reaction force. In the spring-damper model, the reaction force includes the elasticity and viscosity. The elasticity is force exerted by deformation of a spring or rubber. For example, when a spring is pushed or pulled, the elasticity is proportional to the depth of the spring, and it is calculated by multiplying the depth by the elastic spring coefficient. The viscosity is force or resistance exerted by fluids. For example, when we move an object in the fluids (e.g., water and oil). The viscosity is proportional to the relative velocity (i.e., the velocity of the object relative to the fluids), and it can be calculated by multiplying the relative velocity by the viscosity (damper) coefficient. The reaction force is calculated as follows:

$$\mathbf{F}\_{t}^{(\mathbf{m})} = -K\_{\mathbf{s}} \left( \mathbf{P}\_{t-1}^{(\mathbf{m})} - \mathbf{P}\_{t-1}^{(\mathbf{s})} \right) - K\_{\mathbf{d}} \left( \boldsymbol{\nu}\_{t-1}^{(\mathbf{m})} - \boldsymbol{\nu}\_{t-1}^{(\mathbf{s})} \right) \tag{3}$$

where *K*<sup>s</sup> is the spring coefficient, *K*<sup>d</sup> is the damper coefficient, *P*ð Þ <sup>m</sup> *<sup>t</sup>*�<sup>1</sup> is the position vector of the haptic interface device at the master terminal, *P*ð Þ<sup>s</sup> *<sup>t</sup>*�<sup>1</sup> is that at the slave terminal, *v* ð Þ m *<sup>t</sup>*�<sup>1</sup> is the velocity of the haptic interface device at the master terminal, and *v* ð Þs *<sup>t</sup>*�<sup>1</sup> is that at the slave terminal. By using this model, a human can still control remote robot because the human can perceive the reaction force even if other parts of the robot with force sensors hit an object and do not move further. Note that the sensors do not sensed force in this case.

The adaptive reaction force control [7] includes the adaptive viscosity control [28], adaptive elastic control [29], and adaptive viscoelasticity control [30]. The adaptive elastic control dynamically changes the elastic coefficient according to network delay. The adaptive viscosity control dynamic changes the viscosity coefficient according to the network delay and the velocity of the haptic interface device. The adaptive viscoelasticity control exerts the two types of control.

3.Robot position control using force information

The robot position control using force information [31] is proposed to reduce the force applied to an object carrying by two remote robot systems in cooperative work of carrying the object. The control finely adjusts the position of the robot arm in the direction which leads to small reaction force.

4.Adaptive Δ-causality control

The adaptive Δ-causality control dynamically changes the value of Δ which is used for outputting each MU simultaneously (i.e., the MU is output at the time limit of generation time plus Δ seconds) according to the network load (i.e., the network delay and delay jitter) [7]. If the MU is received after the time limit, it is discarded as an obsolete MU. In the remote robot system with force feedback, the control can be used to output of the position information simultaneously among multiple terminals.

In [20], the authors propose the global Δ-causality control and compare the proposed control with the local adaptive Δ-causality control by experiment. The local adaptive Δ-causality control is partially applied to the remote robot systems *Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

(i.e., the adaptive Δ-causality control is carried out between the two robots, or the control is carried out in each remote robot system). To output each MU simultaneously at each system, we need to carry out the global adaptive Δcausality control in which the control is globally performed (i.e., the adaptive Δcausality control is carried out between the two robots and in each remote robot system). Experimental results show that the global control is more effective than the local control for cooperation.

The control can also be used for spatial synchronization.

#### 5.Force adjustment control

The force adjustment control is proposed for cooperative work (see **Figure 2(c)**) of carrying an object by putting the object from both sides between two remote robot systems [32], and the control is used to suppress large force applied to the object and to avoid dropping it because of too small force. When the force applied to the object carried by the two robot arms is larger than a threshold value, each robot arm moves in the direction to reduce the force by a certain distance. When the force is smaller than another threshold value, the robot arms adjust in the direction to increase the force.

Based on the characteristics of the above QoS control, we can summarize the relationship between QoS control and the three types of cooperation in **Table 1**.

For the temporal synchronization control, the intra-stream synchronization control is needed for each terminal in the cooperation of human-robot, robot-robot, and human-human; the inter-stream synchronization control is required for the cooperation of human-robot in which multiple streams are transmitted between the master and slave terminals; the inter-destination synchronization control is necessary for the cooperation of robot-robot and human-human, in which the same types of operation are needed at the terminals.

The adaptive reaction force control can be used to reduce the large reaction force caused by network delay in the cooperation of human-robot (i.e., position difference which is used to calculate the reaction force becomes large as network delay increases) and human-human.

The robot position control using force information can be used to reduce the force applied to an object carrying by two robots in the cooperation of human-robot and robot-robot.

The adaptive Δ-causality control can be employed to alleviate the influence of difference in network delay between the haptic interface device and robot in the


**Table 1.**

*Close relationship between QoS control and cooperation.*

cooperation of human-robot and can be used to reduce the influence of difference in network delay between the robots in the cooperation of robot-robot.

The force adjustment control can be used to adjust the force applied to the operating object in suitable range of values, and the control can be used for the cooperation of human-robot and robot-robot.

For the cooperation of humans-robots, since the cooperation including the three types of cooperation, all the above types of control can be utilized.

#### **5.2 Stabilization control**

In order to achieve stable and high quality of the cooperation using the remote robot systems with force feedback, we need to carry out stabilization control as well as QoS control. This subsection introduces serval types of stabilization control which are employed in our remote robot systems.

1.Stabilization control with filters

The stabilization control with filters uses the wave filter in combination with the phase control filter [12]. The stabilization control with filters makes the remote robot system with force feedback stable against large network delays.

2. Stabilization control by viscosity

In the stabilization control by viscosity [33], the instability phenomenon is suppressed by using viscosity. The control uses the coefficient to restrict the movement distance of the industrial robot to some extent compared with the movement distance of the haptic interface device.

3.Reaction force control upon hitting

In order to solve the vibration problem when the remote robot hits/touches a hard object, the reaction force control upon hitting [34] is proposed. The reaction force control upon hitting gradually increases the reaction force to prevent a robot arm from jumping when the arm hits a hard object (i.e., when the reaction force calculated based on the force information received from the force sensor attached to the remote robot arm is larger than a threshold value).

4. Stabilization control by viscosity

Since the viscosity can be used to suppress vibration, the adaptive viscosity control can also be used as stabilization control. In the control, viscosity is generated by decreasing the movement distance of the industrial robot by a certain amount proportional to the movement velocity [33].

## **6. Future directions**

In the previous section, we have introduced serval types of QoS control and stabilization control to solve the problems in the cooperation among humans and robots. However, there still exist many challenges. In this section, we discuss the future directions of QoS control and stabilization control for further stable and efficient cooperation.

*Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

## 1. Integrated QoS control

In the cooperation using multiple remote robot systems with force feedback, we need to carry out multiple types of QoS control for the systems. If we carry out the QoS control independently, QoE may be deteriorated owing to excessive or insufficient effects of the control [7]. Therefore, we need to carry out different types of QoS control in an integrated manner like the adaptive QoS control [35]; for example, we need to carry out error control [5] together with traffic control [5] to improve the efficiency. Note that the error control increases the network traffic volume. In such a way, if we perform the QoS control independently, excessive or insufficient effects of the control may occur.

## 2.Global QoS control

In the cooperation among multiple remote robot systems, it is important to carry out QoS control for each system and/or each terminal. However, it is also necessary to carry out QoS control for the multiple systems globally (i.e., we need to take account the whole systems when we carry out the QoS control). This is because multiple systems are used to conduct one work together. In Ref. [20], as described in Subsection 5.1 (4), we also found that the global adaptive Δ-causality control is more effective than the local adaptive Δ-causality control in the cooperation between the two remote robot systems.

3. Synthesis of QoS control and stabilization control

Since we need to carry out QoS control and stabilization control for remote cooperation using multiple remote robot systems, it is important to integrate the QoS control with stabilization control to achieve stable and high quality of systems. We can integrate QoS control into stabilization control for the system. For example, we can carry out QoS control in the loop of stabilization control [6].

4.Application of AI

Many factors such as contents of work, movement speed, network delay, room temperature, and wind may affect the effects of QoS control and stabilization control [36]. In order to achieve efficient cooperation (stable and high QoS), big data [37], cloud computing [38], and AI (Artificial Intelligence) [39] technologies such as neural network, fuzzy theory, and genetic algorithm can be useful methods for taking account of the factors.

## **7. Conclusions**

In this chapter, we focused on the cooperation among humans and robots in remote robot systems with force feedback. We explained the expected applications and the four types of cooperation by using the remote robot systems. We also introduced our remote robot systems with force feedback and how to cooperate among the systems. We further explained QoS (Quality of Service) control and stabilization control as our challenges and solutions for effective cooperation. Furthermore, we discussed future directions of the cooperation.

## **Acknowledgements**

The authors thank Prof. Takanori Miyoshi of Nagaoka University of Technology, Prof. Takashi Okuda of Aichi Prefectural University, Prof. Hitoshi Ohnishi of the Open University of Japan, and Prof. Hitoshi Watanabe of Tokyo University of Science for their valuable discussions.

## **Conflict of interest**

The authors declare no conflicts of interest associated with this chapter.

## **Author details**

Pingguo Huang<sup>1</sup> \* and Yutaka Ishibashi<sup>2</sup>

1 Gifu Shotoku Gakuen University, Gifu, Japan

2 Nagoya Institute of Technology, Nagoya, Japan

\*Address all correspondence to: huangpg@gifu.shotoku.ac.jp

© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

## **References**

[1] Ohnishi K. Real world haptics: Its principle and future prospects. The Journal of the Institute of Electrical Engineers of Japan. 2013;**133**:268-269 (In Japanese)

[2] Music S, Salvietti G, Dohmann P, Chinello F, Prattichizzo D, Hirche S. Human–robot team interaction through wearable haptics for cooperative manipulation. IEEE Transactions on Haptics. 2019;**12**(3):350-362

[3] Singh J, Srinivasan A, Neumann G, Kucukyilmaz A. Haptic-guided teleoperation of a 7-DoF collaborative robot arm with an identical twin master. IEEE Transactions on Haptics. 2020;**13**(1):246-252

[4] Haruna M, Kawaguchi N, Ogino M, Akino T. Comparison of three feedback modalities for haptics sensation in remote machine manipulation. IEEE Robotics and Automation Letters. 2021;**6** (3):5040-5047

[5] Ishibashi Y, Huang P. Improvement of QoS in haptic communication and its future. The IEICE Transactions on Communications. 2016;**J99-B**:911-925 (In Japanese)

[6] Huang P, Ishibashi Y. QoS control in remote robot operation with force feedback. In: Robotics Software Design and Engineering. London, UK: IntechOpen; 2020. pp. 1-13

[7] Huang P, Ishibashi Y. QoS control and QoE assessment in multi-sensory communications with haptics. The IEICE Transactions on Communications. 2013; **E96-B**:392-403. DOI: 10.1587/transcom. E96.B.392

[8] ITU-T Rec. I. 350 General Aspects of Quality of Service and Network Performance in Digital Networks. 1993

[9] ITU-T Rec. G. 100/P. 10 Amendment 1, New appendix I - Definition of quality of experience (QoE). 2007

[10] Miyoshi T, Maeda Y, Morita Y, Ishibashi Y, Terashima K. Development of haptic network game based on multilateral tele-control theory and influence of network delay on QoE. Transactions of the Virtual Reality Society of Japan, Special Issues on Haptic Contents. 2014; **19**:559-569 (In Japanese)

[11] Qian Q, Toyoda Y, Ishibashi Y, Huang P, Tateiwa Y. Effect of stabilization control on cooperative work between remote robot systems with force feedback. International Journal of Mechanical Engineering and Robotics Research (IJMERR). 2020;**9**(3):415-420

[12] Huang P, Miyoshi T, Ishibashi Y. Enhancement of stabilization control in remote robot system with force feedback. International Journal of Communications, Network and System Sciences (IJCNS). 2019;**12**(7):99-111

[13] https://www.3dsystems.com/haptic s-devices/touch

[14] http://www.mitsubishielectric.co.jp/ fa/products/rbt/robot/lineup/manual/f/ bfp-a8899k.pdf (In Japanese)

[15] http://dl.mitsubishielectric.co.jp/dl/ fa/members/document/manual/robot/bf p-a8940/bfp-a8940Z.pdf (In Japanese)

[16] http://dl.mitsubishielectric.co.jp/dl/ fa/members/document/manual/robot/bf p-a8080/bfp-a8080e.pdf (In Japanese)

[17] Ye R, Ishibashi Y, Huang P, Tateiwa Y. Comparison of stabilization control for writing characters in remote robot system with force feedback. In:

Proc. the 3rd International Conference on Computer Communication and the Internet (ICCCI), Nagoya, Japan. 2021

[18] Ito S, Ishibashi Y, Huang P, Tateiwa Y. Effect of robot position control using force information: Human versus robot with force sensor. In: Proc. the 9th International Conference on Information and Education Technology (ICIET), Okayama, Japan. 2021. pp. 257-261

[19] Ishibashi Y, Huang P, Psannis KE. Enhanced robot position control using force information for mobile robots: Influences of obstacles on cooperative work. In: Proc. 2021 IEEE 9th International Conference on Information, Communication and Networks (ICICN), the 4th World Symposium on Communication Engineering (WSCE) Held as Workshop of ICICN, Xi'an, China. 2021. pp. 555-559

[20] Hameedha N, Ishibashi Y. Effects of local and global adaptive Δ-causality control on cooperative work between remote robot systems with force feedback. ITE Transactions on Media Technology and Applications (MTA). 2022;**10**(1):1-7

[21] Ye R, Ishibashi Y, Huang P, Tateiwa Y. Comparison of collaboration methods between users in remote robot systems with force feedback. In: Proc. the 7th International Conference on Computer and Communications (ICCC), Chengdu, China. 2021. pp. 1052-1056

[22] Toyoda Y, Ishibashi Y, Huang P, Tateiwa Y, Watanabe H. Efficiency of cooperation between remote robot systems with force feedback: Comparison with cooperation between user and remote robot system. International Journal of Mechanical Engineering and Robotics Research (IJMERR). 2020;**9**(6):900-905

[23] Ishibashi Y, Tsuji A, Tasaka S. A group synchronization mechanism for stored media in multicast communications. In: Proc. 16th IEEE International Conference on Computer and Communications (INFOCOM), Kobe, Japan. 1997. pp. 693-701

[24] Ishibashi Y, Tasaka S, Hasegawa T. The virtual-time rendering algorithm for haptic media synchronization in networked virtual environments. In: Proc. the 16th International Workshop on Communications Quality and Reliability (CQR), Okinawa, Japan. 2002. pp. 213-217

[25] Huang P, Ishibashi Y. Simultaneous output-timing control in networked games and virtual environments. In: Handbook on Multimedia Synchronization. Switzerland: Springer International Publishing; 2018. pp. 149-166

[26] Mills DL. Internet time synchronization: The network time protocol. IEEE Transactions on Communications. 1991;**39**(10):1482-1493

[27] 3D systems, OpenHaptics toolkit programmer's guide. version 3.2, 2013

[28] Komatsu Y, Ohnishi H, Ishibashi Y. Adaptive control of viscosity in remote control system with force feedback. In: Proc. IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), Chinese Tapei. 2017. pp. 237-238

[29] Suzuki S, Matsunaga K, Ohnishi H, Ishibashi Y. Influences of network delay variation on haptic perception under adaptive reaction force control. In: Proc. the 3rd IEEE Global Conference on Consumer Electronics (GCCE), Tokyo, Japan. 2014. pp. 669-673

[30] Abe T, Komatsu Y, Onishi H, Ishibashi Y. QoE assessment of adaptive *Perspective Chapter: Cooperation among Humans and Robots in Remote Robot Systems with… DOI: http://dx.doi.org/10.5772/intechopen.106951*

viscoelasticity control in remote control system with haptic and visual senses. In: Proc. IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), Chinese Tapei. 2018. pp. 133-134

[31] Ishibashi Y, Taguchi E, Huang P, Tateiwa Y. Robot position control with force information in cooperation between remote robot systems. In: Proc. the 5th International Conference on Control, Automation and Robotics (ICCAR), Beijing, China. 2019. pp. 147-151

[32] Kato H, Ishibashi Y, Ohnishi H, Huang P. Force Adjustment Control for Carrying Object between Remote Robot Systems with Force Feedback. Fukui, Japan: IEICE Technical Report, CQ2022–13. 2022 (In Japanese)

[33] Rikiishi T, Ishibashi Y, Huang P, Miyoshi T, Ohnishi H, Tateiwa Y, et al. Effect of Stabilization Control by Viscocity in Remote Robot System. Chiba, Japan: IEICE Technical Report, MVE2017–19. 2017 (In Japanese)

[34] Arima R, Huang P, Ishibashi Y, Tateiwa Y. Softness Assessment of Objects in Remote Robot System with Haptics: Comparison between Reaction Force Control upon Hitting and Stabilization Control. Tokyo, Japan: IEICE Technical Report, CQ2017–98. 2018 (In Japanese)

[35] Ishibashi Y, Nagasaka M. An adaptive QoS control scheme of avatars in distributed virtual environments. The Journal of The Institute of Image Information and Television Engineers. 2006;**60**(11):1835-1839

[36] Wang X, Huang P, Ishibashi Y, Okuda T, Watanabe H. Influence of network delay on QoS control using neural network in remote robot systems with force feedback. In: Proc. 2020 the 9th International Conference on Networks, Communication and Computing (ICNCC), Tokyo, Japan. 2020

[37] Hashem IAT, Yaqoob I, Anuar NB, Mokhtar S, Gani A, Khan SU. The rise of 'big data' on cloud computing: Review and open research issues. Information Systems. 2015;**47**:98-115

[38] Memos VA, Psannis KE, Ishibashi Y, Kim BG, Gupta BB. An efficient algorithm for media-based surveillance system (EAMSuS) in IoT smart city framework. Future Generation Computer System. 2018;**83**:619-628

[39] Ramesh A, Kambhampati C, Monson J, Drew P. Artificial intelligence in medicine. Annals of the Royal College of Surgeons of England. 2004;**86**:334-338

## **Chapter 5**
