**2. Hardware design of a dual-arm robot**

*Service Robotics*

developed a cherry picking robot using a four-degree-of-freedom manipulator and a specially designed end effector with suction and shear functions. The visual part uses a light emitter, a photodetector, and a scanning device. The fruit was picked in 14 s, and the success rate was about 84%. The CROPS plan completed in 2014 was jointly completed by many European countries and units and aims to develop a modular picking robot system for different mission scenarios. The greenhouse bell pepper picking robot platform completed in the experiment in the Netherlands [5] used a 9-degree-of-freedom manipulator, two color CCD cameras and a depth-measuring camera, and its end effector was also equipped with a small camera to complete the picking with higher accuracy. Taqi et al. [6] have developed small household cherry tomato picking robots that can achieve

The continuous operation of the picking robot can make intelligent decisions on multi-task under multi-objective scenarios and plan the operation according to the picking needs. The research goal is that the picking robot system can intelligently select and pick fruits that meet the picking conditions, thereby greatly improving the degree of automation of the picking process and improving the quality of the harvested fruits. Because the picking robot is still in the laboratory research stage, the research on continuous operation is also very limited, and it is basically in its infancy. Japan's Nagata et al. [7] used the shape to judge and classify strawberry quality. The accuracy in the experiment is acceptable, but the speed is much slower than artificial. Zhao et al. [8] investigated the visual recognition of apple maturity, using multi-spectral laser beams to complete fruit identification and positioning and ripeness judgment. Guo [9] and others judged strawberry maturity based on HIS color space algorithm. In the field test, the accuracy of the goal of picking fruits with maturity of 80% or more was more than 90%. Wang [10] and Ling et al. [11] carried out research on selective harvest information acquisition and path planning of tomato picking robots. Multi-sensor information fusion method was used for tomato quality detection and classification and selective picking decision. The appearance maturity of the fruit is detected by the H-means in the computer image, the fruits are classified in real time according to the agricultural industry standards, and the selective harvesting decision is made through the progressive identification of feature information and the fusion decision. At harvest time, path planning is performed on multiple targets through an optimization algorithm and

This chapter mainly focuses on developing an intelligent software system for the continuous operation of the dual-arm picking robot in a plant factory. Typically, the semi-structure environment of the greenhouse poses challenges for autonomous operations of the robot, and the complex tasks mainly include identification and positioning under variable light conditions, selective picking in multi-cluster growth environment, and complex multi-task programming. For the difficulty of developing the software system, a hierarchical modular software system framework is designed. Moreover, a scheduling method of functional modules is designed based on the idea of finite state machine for the complex multi-task planning problem of tomato continuous picking process. The task scheduling design based on the Finite State Machine (FSM) reduces the difficulty of development work and

This chapter is organized as follows: Section 2 briefly describes the hardware structure of a dual-arm robot, and Section 3 presents the software framework with the highlights of deploying SMACH- and ViSP-based nodes in a ROS development environment. The experimental results are also included at the end

very accurate picking tasks in specific environments.

**110**

of Section 3.

then boxed by level.

improves the efficiency of development.

The semi-structured operating environment in the plant factory is relatively complicated, such as the occlusion of fruit branches and leaves, the challenging grasping shape of the fruit, the changing light, and the variety of tasks, which requires to consider the multi-tasking ability of the robot when designing the hardware of the picking robot. At the same time, the complexity of the picking environment requires that the execution of the robot be robust to the environment in which the target is located, only in this way can it pick fruits in different states. Based on the above two design goals, we designed a dual-arm picking robot to simulate human picking operations.

#### **2.1 Design of dual-arm robot body**

The mechanical structure of the robot body mimics human arms, and the left and right arms each have three joints: a vertical lift joint, a boom rotation joint, and a forearm rotation joint. The vertical lifting joint is driven by a servo motor to drive the roller screw. The actual effective stroke is about 300 mm. The big and small arm joints are driven by a servo motor connected to a harmonic reducer. There is a waist joint between the body and the base, which can provide 360° rotary motion. The single-arm movement of the robot is similar to that of the SCARA robot. The vertical positioning is achieved by the lifting joint, and the rotation of the large and small arms realizes the positioning in the plane. There are three degrees of freedom in motion. Each of the left and right forearms is designed with a mounting flange surface, and end effectors can be installed as required. The dual-arm robot base is installed on a mobile cart and is transported along the track to different picking points for picking operations.

Due to the limitation of the freedom of the robot body, for complex tomato picking environments, it is not enough to rely only on the freedom of the arms, so we can use the design of the end effector to increase the freedom of our robot and enable the robot to flexibly complete the picking operation. We designed a shearing end effector on the cutting hand of the dual-arm robot and designed a suction-type end sleeve on the auxiliary hand to fix the target tomato and assist the hand in picking (**Figure 1**).

**Figure 1.** *The structure of the robot body.*

### **2.2 Dual-arm robot coordinate system**

As shown in **Figure 2**, the world coordinate system (*xw*, *yw*, *zw*) of the dual-arm robot is built on the waist and coincides with the waist joint coordinate system. The left and right arm coordinate systems (*xl*, *yl*, *zl*) and (*xr*, *yr*, *zr*) use the right-hand principle, and the span direction at the zero position of the *x* axis is different from the *x* axis of the world coordinate system by +45° and −45°. The *z*-axis direction is vertically upward and is at the zero position at the lowest point. The *x–y* plane is parallel to the world coordinate system *x–y* plane. The binocular camera coordinate system is established at the intersection of the right-eye visual axis and the camera lens. The coordinate system adopted by the bumblebee2 camera is the left-hand principle. In order to be compatible with the entire system, the *y*-axis direction is reversed.

We use the tf function package [12] provided by ROS to maintain the transformation relationship between coordinate systems. After the robot model is built according to urdf, the system will automatically broadcast the transformation relationships between all the coordinate systems. The picking robot coordinate system is shown in **Figure 3**.

**Figure 2.** *Robot coordinate system.*

**113**

**Figure 5.**

*Picking robot communication architecture.*

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations*

Based on the above coordinate system, using the *rosrun tf view\_frames* command, we can view the tf tree of the dual-arm picking robot, as shown in

*DOI: http://dx.doi.org/10.5772/intechopen.92254*

**Figure 4**.

**Figure 4.** *The tf tree of robot.*

**Figure 3.** *The picking robot tf coordinate system under Rivz.*

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations DOI: http://dx.doi.org/10.5772/intechopen.92254*

Based on the above coordinate system, using the *rosrun tf view\_frames* command, we can view the tf tree of the dual-arm picking robot, as shown in **Figure 4**.

**Figure 4.** *The tf tree of robot.*

*Service Robotics*

reversed.

system is shown in **Figure 3**.

**2.2 Dual-arm robot coordinate system**

As shown in **Figure 2**, the world coordinate system (*xw*, *yw*, *zw*) of the dual-arm robot is built on the waist and coincides with the waist joint coordinate system. The left and right arm coordinate systems (*xl*, *yl*, *zl*) and (*xr*, *yr*, *zr*) use the right-hand principle, and the span direction at the zero position of the *x* axis is different from the *x* axis of the world coordinate system by +45° and −45°. The *z*-axis direction is vertically upward and is at the zero position at the lowest point. The *x–y* plane is parallel to the world coordinate system *x–y* plane. The binocular camera coordinate system is established at the intersection of the right-eye visual axis and the camera lens. The coordinate system adopted by the bumblebee2 camera is the left-hand principle. In order to be compatible with the entire system, the *y*-axis direction is

We use the tf function package [12] provided by ROS to maintain the transformation relationship between coordinate systems. After the robot model is built according to urdf, the system will automatically broadcast the transformation relationships between all the coordinate systems. The picking robot coordinate

**112**

**Figure 3.**

**Figure 2.**

*Robot coordinate system.*

*The picking robot tf coordinate system under Rivz.*

**Figure 5.** *Picking robot communication architecture.*

### **2.3 Communication architecture of the dual-arm picking robot**

The communication architecture of the dual-arm picking robot is shown in **Figure 5**. The motor driver is a Gold series motor driver produced by Elmo and is equipped with the same series of multi-axis controllers. The manufacturer has provided a complete motor driver to multi-axis controller communication protocol and communication protocol implementation and does not require customers to conduct secondary development. The multi-axis controller uses Modbus TCP communication as the communication between the lower computer and the industrial computer. It is connected to it through an Ethernet cable.

## **3. Software frame design**

In the process of programming, we generally manually divide the tasks of the robot. Once the tasks are effectively divided, we can stack the tasks with the smallest functional components, so the difficulty of task division is how to use human prior knowledge to divide the minimum granularity of robot skills, which is a learning process. At the same time, how to easily and effectively combine the divided components is also an important part of completing the task simulation.

The system architecture describes the functional structure of the subdivision and the topological relationship between them and a series of specifications that need to be set for subsequent development. The basic requirements of software engineering include modularity, code reuse, and function sharing. Using a common framework is helpful for decomposing development tasks and code migration. Robot software also follows the general rules of software engineering. Architecture is how you break up the robot's functions and organize your code. A clear architecture that matches the project directly determines your development efficiency and even the success or failure of the final function. There are two main approaches to robot system architecture: SPA architecture and behavior-based architecture.

#### **3.1 SPA**

The software system architecture is "Sense Plan Act"(SPA), as shown in **Figure 6**. The robot maps the external environment space through sensors and uses a certain modeling method to structure and model the perception information and then analyzes the model to plan the robot's actions. Finally, the action instructions are executed in the environment to achieve a complete interactive process.

The typical software architecture in a SPA robot system is a three-layer architecture: the perception layer, the planning layer, and the motion control layer, as shown in **Figure 7**. The perception layer receives and processes the sensor data, the planning layer plans the motion trajectory, and the motion control layer ensures the accurate execution of the movement. The SPA robot software system architecture

**115**

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations*

pays more attention to the perception and modeling of the world because this is the

The behavior-based software system architecture is a top-down software design. The small functions of each robot are packaged into individual small modules. All functional modules can be executed in parallel without prioritization. A robot task can be understood as an organic composition of functional modules (**Figure 8**).

To a certain extent, all robot actions are responses to stimuli (inputs). This stress

mode avoids the thinking logic in the SPA architecture and facilitates the rapid action response. In order to achieve the task, we can design a control scheme to change the stress level of the action. Therefore, we need a global controller to coordinate the choice of actions in order to achieve our intended purpose. The behaviorbased software design framework has good flexibility, but it increases the difficulty of control. When multiple actions can affect the output, problems are easy to occur. Therefore, combined with the SPA software architecture and behavior-based software architecture, we design a software framework that combines the advantages of both architectures for continuous operation of a dual-arm picking robot. Its

1.Hierarchical modular design: The software architecture absorbs the advantages of the SPA architecture and also adopts a hierarchical design. The layered design is mainly logical, which makes it easier for users to understand the working mode of the robot. At the same time, it also absorbs the advantages of behavior-based architecture, that is, functional modularity. Based on the

basis for the accuracy of subsequent planning and movement.

**3.2 Behavior-based architecture**

*Action-based software system architecture.*

characteristics are as follows:

*DOI: http://dx.doi.org/10.5772/intechopen.92254*

**Figure 7.** *SPA layered design.*

**Figure 8.**

**Figure 6.** *SPA work pattern.*

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations DOI: http://dx.doi.org/10.5772/intechopen.92254*

**Figure 7.** *SPA layered design.*

*Service Robotics*

**2.3 Communication architecture of the dual-arm picking robot**

computer. It is connected to it through an Ethernet cable.

**3. Software frame design**

The communication architecture of the dual-arm picking robot is shown in **Figure 5**. The motor driver is a Gold series motor driver produced by Elmo and is equipped with the same series of multi-axis controllers. The manufacturer has provided a complete motor driver to multi-axis controller communication protocol and communication protocol implementation and does not require customers to conduct secondary development. The multi-axis controller uses Modbus TCP communication as the communication between the lower computer and the industrial

In the process of programming, we generally manually divide the tasks of the robot. Once the tasks are effectively divided, we can stack the tasks with the smallest functional components, so the difficulty of task division is how to use human prior knowledge to divide the minimum granularity of robot skills, which is a learning process. At the same time, how to easily and effectively combine the divided

The system architecture describes the functional structure of the subdivision and the topological relationship between them and a series of specifications that need to be set for subsequent development. The basic requirements of software engineering include modularity, code reuse, and function sharing. Using a common framework is helpful for decomposing development tasks and code migration. Robot software also follows the general rules of software engineering. Architecture is how you break up the robot's functions and organize your code. A clear architecture that matches the project directly determines your development efficiency and even the success or failure of the final function. There are two main approaches to robot system architecture: SPA architecture and behavior-based architecture.

The software system architecture is "Sense Plan Act"(SPA), as shown in **Figure 6**. The robot maps the external environment space through sensors and uses a certain modeling method to structure and model the perception information and then analyzes the model to plan the robot's actions. Finally, the action instructions

The typical software architecture in a SPA robot system is a three-layer architecture: the perception layer, the planning layer, and the motion control layer, as shown in **Figure 7**. The perception layer receives and processes the sensor data, the planning layer plans the motion trajectory, and the motion control layer ensures the accurate execution of the movement. The SPA robot software system architecture

are executed in the environment to achieve a complete interactive process.

components is also an important part of completing the task simulation.

**114**

**Figure 6.** *SPA work pattern.*

**3.1 SPA**

#### **Figure 8.**

*Action-based software system architecture.*

pays more attention to the perception and modeling of the world because this is the basis for the accuracy of subsequent planning and movement.

#### **3.2 Behavior-based architecture**

The behavior-based software system architecture is a top-down software design. The small functions of each robot are packaged into individual small modules. All functional modules can be executed in parallel without prioritization. A robot task can be understood as an organic composition of functional modules (**Figure 8**).

To a certain extent, all robot actions are responses to stimuli (inputs). This stress mode avoids the thinking logic in the SPA architecture and facilitates the rapid action response. In order to achieve the task, we can design a control scheme to change the stress level of the action. Therefore, we need a global controller to coordinate the choice of actions in order to achieve our intended purpose. The behaviorbased software design framework has good flexibility, but it increases the difficulty of control. When multiple actions can affect the output, problems are easy to occur.

Therefore, combined with the SPA software architecture and behavior-based software architecture, we design a software framework that combines the advantages of both architectures for continuous operation of a dual-arm picking robot. Its characteristics are as follows:

1.Hierarchical modular design: The software architecture absorbs the advantages of the SPA architecture and also adopts a hierarchical design. The layered design is mainly logical, which makes it easier for users to understand the working mode of the robot. At the same time, it also absorbs the advantages of behavior-based architecture, that is, functional modularity. Based on the

analysis and understanding of the robot's internal architecture, we divide the functions of the robot into seven modules, each of which is functionally independent of each other. Combining the above two is a hierarchical modular design, which divides the functional modules into a specific layer according to the attributes of the functions, thus strengthening the logic of the system.

2.Finite state machine control: In order to solve the shortcomings of behaviorbased architecture, we have designed a task planning module based on finite state machines, which is used to schedule and control the execution order of each functional module to complete a specific task.
