**1. Introduction**

With the development of technologies such as industrial robots and computer image processing, a series of research and experiments on intelligent picking robots have been carried out in Japan and other related countries, such as tomato, apple, and grape picking robots [1]. Research on picking robots has focused on two parts: the first is a hardware device that can achieve rapid picking, that is, how to design a stable, efficient, and adaptable mechanical and visual sensing system, and the second is to design intelligent software for continuous operations, that is, to accurately identify, distinguish, and locate targets and to solve the problem of task planning and task scheduling during continuous picking operations.

In the research field of picking robots, Kondo et al. [2] designed a tomato picking robot using a 7-degree-of-freedom manipulator. It used fingers and pneumatic nozzles in conjunction with a color camera to complete the picking operation. The experiment achieved a picking success rate of about 70% [3]. Tanigaki et al. [4]

developed a cherry picking robot using a four-degree-of-freedom manipulator and a specially designed end effector with suction and shear functions. The visual part uses a light emitter, a photodetector, and a scanning device. The fruit was picked in 14 s, and the success rate was about 84%. The CROPS plan completed in 2014 was jointly completed by many European countries and units and aims to develop a modular picking robot system for different mission scenarios. The greenhouse bell pepper picking robot platform completed in the experiment in the Netherlands [5] used a 9-degree-of-freedom manipulator, two color CCD cameras and a depth-measuring camera, and its end effector was also equipped with a small camera to complete the picking with higher accuracy. Taqi et al. [6] have developed small household cherry tomato picking robots that can achieve very accurate picking tasks in specific environments.

The continuous operation of the picking robot can make intelligent decisions on multi-task under multi-objective scenarios and plan the operation according to the picking needs. The research goal is that the picking robot system can intelligently select and pick fruits that meet the picking conditions, thereby greatly improving the degree of automation of the picking process and improving the quality of the harvested fruits. Because the picking robot is still in the laboratory research stage, the research on continuous operation is also very limited, and it is basically in its infancy. Japan's Nagata et al. [7] used the shape to judge and classify strawberry quality. The accuracy in the experiment is acceptable, but the speed is much slower than artificial. Zhao et al. [8] investigated the visual recognition of apple maturity, using multi-spectral laser beams to complete fruit identification and positioning and ripeness judgment. Guo [9] and others judged strawberry maturity based on HIS color space algorithm. In the field test, the accuracy of the goal of picking fruits with maturity of 80% or more was more than 90%. Wang [10] and Ling et al. [11] carried out research on selective harvest information acquisition and path planning of tomato picking robots. Multi-sensor information fusion method was used for tomato quality detection and classification and selective picking decision. The appearance maturity of the fruit is detected by the H-means in the computer image, the fruits are classified in real time according to the agricultural industry standards, and the selective harvesting decision is made through the progressive identification of feature information and the fusion decision. At harvest time, path planning is performed on multiple targets through an optimization algorithm and then boxed by level.

This chapter mainly focuses on developing an intelligent software system for the continuous operation of the dual-arm picking robot in a plant factory. Typically, the semi-structure environment of the greenhouse poses challenges for autonomous operations of the robot, and the complex tasks mainly include identification and positioning under variable light conditions, selective picking in multi-cluster growth environment, and complex multi-task programming. For the difficulty of developing the software system, a hierarchical modular software system framework is designed. Moreover, a scheduling method of functional modules is designed based on the idea of finite state machine for the complex multi-task planning problem of tomato continuous picking process. The task scheduling design based on the Finite State Machine (FSM) reduces the difficulty of development work and improves the efficiency of development.

This chapter is organized as follows: Section 2 briefly describes the hardware structure of a dual-arm robot, and Section 3 presents the software framework with the highlights of deploying SMACH- and ViSP-based nodes in a ROS development environment. The experimental results are also included at the end of Section 3.

**111**

**Figure 1.**

*The structure of the robot body.*

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations*

The semi-structured operating environment in the plant factory is relatively complicated, such as the occlusion of fruit branches and leaves, the challenging grasping shape of the fruit, the changing light, and the variety of tasks, which requires to consider the multi-tasking ability of the robot when designing the hardware of the picking robot. At the same time, the complexity of the picking environment requires that the execution of the robot be robust to the environment in which the target is located, only in this way can it pick fruits in different states. Based on the above two design goals, we designed a dual-arm picking robot to

The mechanical structure of the robot body mimics human arms, and the left and right arms each have three joints: a vertical lift joint, a boom rotation joint, and a forearm rotation joint. The vertical lifting joint is driven by a servo motor to drive the roller screw. The actual effective stroke is about 300 mm. The big and small arm joints are driven by a servo motor connected to a harmonic reducer. There is a waist joint between the body and the base, which can provide 360° rotary motion. The single-arm movement of the robot is similar to that of the SCARA robot. The vertical positioning is achieved by the lifting joint, and the rotation of the large and small arms realizes the positioning in the plane. There are three degrees of freedom in motion. Each of the left and right forearms is designed with a mounting flange surface, and end effectors can be installed as required. The dual-arm robot base is installed on a mobile cart and is transported along the track to different picking

Due to the limitation of the freedom of the robot body, for complex tomato picking environments, it is not enough to rely only on the freedom of the arms, so we can use the design of the end effector to increase the freedom of our robot and enable the robot to flexibly complete the picking operation. We designed a shearing end effector on the cutting hand of the dual-arm robot and designed a suction-type end sleeve on the auxiliary hand to fix the target tomato and assist the hand in

*DOI: http://dx.doi.org/10.5772/intechopen.92254*

simulate human picking operations.

**2.1 Design of dual-arm robot body**

points for picking operations.

picking (**Figure 1**).

**2. Hardware design of a dual-arm robot**

*Manipulating Complex Robot Behavior for Autonomous and Continuous Operations DOI: http://dx.doi.org/10.5772/intechopen.92254*
