Vertices of Graph 100 1000 2000 Avg. Path Length 4.62 5.98 7.34 Hold-up Time [%] 63.96% 27.34% 38.96%

The presented results concerning risk quantification and minimisation demonstrate the effectiveness of guaranteeing safety for human agents in the realm of close human-robot

In order to evaluate the situation and activity recognition module of the MARCOCO framework, different courses of action were executed. On the one hand, efficient analysis of different scenarios requires automated means of feature value setting. Thus, value pre-sets were incorporated into the framework which allows for usage of pre-defined feature vectors. Such pre-sets enable investigation of interesting use cases without capturing sensor data. Also, recognition results can be directly related to defined feature changes through pre-sets. Nevertheless, recognition based on actual sensor data is compulsory in order to

Based on these pre-sets and on actual sensor data all experiments were conducted. Natural movements and transitions between actions have been tested and special use cases have

Table 4 summarises the results of 2140 recognition cycles. The average processing time amounts to approximately 550 ms. The lower bound is less than half of this. There are also casual outliers which take up to 10 seconds. Further investigations based on feature value pre-setting have shown that long duration times might not be directly linkable to the recognition module itself (Graf et al., 2010c). Though, further research needs to be

presented in Table 3. Further details are explained in (Graf et al., 2009).

evaluate recognition results over time and prolonged actions (Fig. 8).

Fig. 8. Usage of feature value pre-sets and actual sensor data.

conducted on optimising runtime behaviour of the recognition module.

Table 3. Results of path planner run-time analysis.

collaboration.

been investigated.

**5.3 Situation awareness** 

Compared to simple distance measures and Gaussian mixture models, the two-threaded fuzzy system allows for precise modelling of situations and according risk assignments. For examination purposes the same sensor input sequence was evaluated by the above mentioned methods and risk assignments were compared. The results confirm our assumption about flexibility and effectiveness of the here presented fuzzy method (Fig. 7). Further details can be found in (Graf et al., 2010a).

The conducted experiments also demonstrate that training a Support Vector Regression resulted in unreliable and noisy risk estimation compared to the implemented two-threaded fuzzy system. Thus, the fuzzy system outperforms the Support Vector Regression and is used as preferred risk estimation method. Grounded on the results of the here described fuzzy logic implementation, safety and efficiency for human-robot cooperation is achievable in real-time.

Fig. 7. Selected situations of human posture and corresponding configuration subspace for first three robot axes. Size and colour of each node of the subspace is determined by risk assignment.

For the experimental analysis of the path re-planning technique different scenarios were tested in simulation. Especially, the size of the configuration space graph was subject of evaluation in order to capture scalability of the algorithm. For testing, a sequence of human motion was recorded and played back during simulation (Fig. 7). Thus, arbitrary movements were recorded and thereby the simulation was related to real-world setups. The tested scenarios do not consider human-robot interaction or cooperation, but instead, the robot has a given repetitive task and has to avoid human co-worker in its working area. The

Compared to simple distance measures and Gaussian mixture models, the two-threaded fuzzy system allows for precise modelling of situations and according risk assignments. For examination purposes the same sensor input sequence was evaluated by the above mentioned methods and risk assignments were compared. The results confirm our assumption about flexibility and effectiveness of the here presented fuzzy method (Fig. 7).

The conducted experiments also demonstrate that training a Support Vector Regression resulted in unreliable and noisy risk estimation compared to the implemented two-threaded fuzzy system. Thus, the fuzzy system outperforms the Support Vector Regression and is used as preferred risk estimation method. Grounded on the results of the here described fuzzy logic implementation, safety and efficiency for human-robot cooperation is achievable

 Fig. 7. Selected situations of human posture and corresponding configuration subspace for first three robot axes. Size and colour of each node of the subspace is determined by risk

For the experimental analysis of the path re-planning technique different scenarios were tested in simulation. Especially, the size of the configuration space graph was subject of evaluation in order to capture scalability of the algorithm. For testing, a sequence of human motion was recorded and played back during simulation (Fig. 7). Thus, arbitrary movements were recorded and thereby the simulation was related to real-world setups. The tested scenarios do not consider human-robot interaction or cooperation, but instead, the robot has a given repetitive task and has to avoid human co-worker in its working area. The

Further details can be found in (Graf et al., 2010a).

in real-time.

assignment.

overall hold-up time of the robot reaches about 27% during evaluation. The results are presented in Table 3. Further details are explained in (Graf et al., 2009).


Table 3. Results of path planner run-time analysis.

The presented results concerning risk quantification and minimisation demonstrate the effectiveness of guaranteeing safety for human agents in the realm of close human-robot collaboration.

## **5.3 Situation awareness**

In order to evaluate the situation and activity recognition module of the MARCOCO framework, different courses of action were executed. On the one hand, efficient analysis of different scenarios requires automated means of feature value setting. Thus, value pre-sets were incorporated into the framework which allows for usage of pre-defined feature vectors. Such pre-sets enable investigation of interesting use cases without capturing sensor data. Also, recognition results can be directly related to defined feature changes through pre-sets. Nevertheless, recognition based on actual sensor data is compulsory in order to evaluate recognition results over time and prolonged actions (Fig. 8).

Fig. 8. Usage of feature value pre-sets and actual sensor data.

Based on these pre-sets and on actual sensor data all experiments were conducted. Natural movements and transitions between actions have been tested and special use cases have been investigated.

Table 4 summarises the results of 2140 recognition cycles. The average processing time amounts to approximately 550 ms. The lower bound is less than half of this. There are also casual outliers which take up to 10 seconds. Further investigations based on feature value pre-setting have shown that long duration times might not be directly linkable to the recognition module itself (Graf et al., 2010c). Though, further research needs to be conducted on optimising runtime behaviour of the recognition module.

Cognitive Robotics in Industrial Environments 231

implemented for risk evaluation of situations according to human pose features and relation between human and robots. The system is flexible and effective. In comparison to Support Vector Classification and other means of risk estimation the two-threaded fuzzy system is

Results of risk estimation are used for adapting robotic behaviour. Adaption is realised by path re-planning if a look-a-head functionality determines impending collisions of human and robot before they occur. That allows for safe and efficient path traversal and, thus,

In order to achieve true human-robot cooperation situation awareness and action recognition is necessary. A module for realising this task was implemented using Description Logics for defining appropriate ontologies and for reasoning. The presented system is capable of recognising subsequent, parallel and dependent actions and can generate expectation towards robotic behaviour. Thus, the system reaches beyond sole

Future work will carry on development towards a system that achieves close human-robot collaboration. There are still many open challenges that need to be tackled before this goal is reached. The usage of more than one camera can either widen the supervised work area or

Currently, only one human agent can be detected and its kinematics can be reconstructed. Extension of the presented algorithms is needed for multi-human pose estimation. Moreover, the algorithms need to be adapted in order to cope with more arbitrary movements of human co-workers. Some movements are not covered by the current human

Object recognition and semantic mapping of the work area are also important means for modelling interactions of human agents and robots with the surrounding environment. Particularly object recognition will enable more diverse and differentiated analysis of situations. Semantic mapping of objects and places in the robots' work area will allow for recognition of human action plans and, thus, a better understanding of intentions behind

As pointed out above, implemented virtual features need to be realized for the demonstrator. Moreover, runtime optimisations of the current situation and activity recognition module need to be investigated and implemented. This will allow for evaluation of real-world scenarios of interaction and cooperation. Also, realisation of industrial applications with the MAROCO system will enable evaluation of capabilities and user acceptance. This experimental evaluation can be realised stepwise beginning with simple risk minimisation and collision avoidance, advancing on to telepresence-like systems and

The MAROCO system emphasises on real-time computation and safety for human coworkers. Nevertheless, the implemented system is a research base and does not permit safety certification. Hopefully, achievements of the human-robot cooperation research community will migrate into applicable industrial systems. Safety regulations and engineers

situation recognition and enables understanding human activities.

the most reliable and accurate one.

reduced time of robot hold-up times.

enable multi-view capturing of the scene.

human actions.

pose reconstruction process, e.g., stooping down.

concluding in fully autonomous human-robot cooperation.

have to adapt to this young field of research.


Table 4. Results from evaluation of the recognition module.

In Fig. 9, results of the recognition module depending on the human pose are depicted. It demonstrates the capabilities of analysing solely kinematical features of the human agent and its relations to a robot.

Fig. 9. Left: Human agent is watching the robot. Recognized situation: Monitoring. The robot is expected to carry on with its task of following a planned path. Right: Human agent is communicating. The complex action to signal a left turning movement is recognized. The robot is expected to comply with user instructions.

By adapting the virtual features according to the generated expectations the interaction between reasoner results and robotic behaviour can be demonstrated. Thus, the capabilities of the presented approach reach beyond sole activity and situation recognition. By generating expectations towards robot behaviour, an understanding of the situation can be achieved. This induction of relations between concepts can hardly be realized by purely probabilistic methods. The achieved processing cycle time of approximately 550 ms does not allow for safe cooperation based only on the recognition module. Thus, the MAROCO framework uses its implemented techniques and algorithms to enforce safety and real-time capabilities during robot motion.

## **6. Conclusion**

The presented framework MAROCO and the incorporated approaches are based on the identification of different modules that have to be taken into account when designing a system for close human-robot collaboration based on a depth imaging sensor. Experimental results give confidence in continuing to strive for true contact based cooperation between robot and human. Thus, our work is a stepping stone for future development.

Thus far, a system was implemented which analysis depth images taken from a 3D camera system mounted beneath the ceiling. Robust features like motion, head and body orientation, position and arm poses are robustly end efficiently estimated. Evaluation has shown that high accuracy is achieved.

All these features are used to reconstruct the human kinematics which is the foundation for risk quantification. A two-threaded fuzzy system with a novel hyperinference operator is

In Fig. 9, results of the recognition module depending on the human pose are depicted. It demonstrates the capabilities of analysing solely kinematical features of the human agent

 Fig. 9. Left: Human agent is watching the robot. Recognized situation: Monitoring. The robot is expected to carry on with its task of following a planned path. Right: Human agent is communicating. The complex action to signal a left turning movement is recognized. The

By adapting the virtual features according to the generated expectations the interaction between reasoner results and robotic behaviour can be demonstrated. Thus, the capabilities of the presented approach reach beyond sole activity and situation recognition. By generating expectations towards robot behaviour, an understanding of the situation can be achieved. This induction of relations between concepts can hardly be realized by purely probabilistic methods. The achieved processing cycle time of approximately 550 ms does not allow for safe cooperation based only on the recognition module. Thus, the MAROCO framework uses its implemented techniques and algorithms to enforce safety and real-time

The presented framework MAROCO and the incorporated approaches are based on the identification of different modules that have to be taken into account when designing a system for close human-robot collaboration based on a depth imaging sensor. Experimental results give confidence in continuing to strive for true contact based cooperation between

Thus far, a system was implemented which analysis depth images taken from a 3D camera system mounted beneath the ceiling. Robust features like motion, head and body orientation, position and arm poses are robustly end efficiently estimated. Evaluation has

All these features are used to reconstruct the human kinematics which is the foundation for risk quantification. A two-threaded fuzzy system with a novel hyperinference operator is

robot and human. Thus, our work is a stepping stone for future development.
