Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot Competitions

*Matthew Studley, Sarah Carter, Francisco J. Perez-Grau, Antidio Viguria Jiménez, Gabriele Ferri, Fausto Ferreira, Deebul Nair, Sven Schneider, Paul G. Plöger, Pedro U. Lima, Meysam Basiri, Gerhard K. Kraetzschmar, Daniele Nardi, Lun Wang, Emanuele Antonioni, Vincenzo Suriani and Luca Iocchi*

## **Abstract**

The SciRoc project, started in 2018, is an EU-H2020 funded project supporting the European Robotics League (ERL) and builds on the success of the EU-FP7/H2020 projects RoCKIn, euRathlon, EuRoC and ROCKEU2. The ERL is a framework for robot competitions currently consisting of three challenges: ERL Consumer, ERL Professional and ERL Emergency. These three challenge scenarios are set up in urban environments and converge every two years under one major tournament: the ERL Smart Cities Challenge. Smart cities are a new urban innovation paradigm promoting the use of advanced technologies to improve citizens' quality of life. A key novelty of the SciRoc project is the ERL Smart Cities Challenge, which aims to show how robots will integrate into the cities of the future as physical agents. The SciRoc Project ran two such ERL Smart Cities Challenges, the first in Milton Keynes, UK (2019) and the second in Bologna, Italy (2021). In this chapter we evaluate the three challenges of the ERL, explain why the SciRoc project introduced a fourth challenge to bring robot benchmarking to Smart Cities and outline the process in conducting a Smart City event under the ERL umbrella. These innovations may pave the way for easier robotic benchmarking in the future.

**Keywords:** robots, benchmarking, smart cities

## **1. Introduction**

Technology has an increasing impact on all our lives, and as the population of the world becomes more predominantly urban, it is in cities that most of us will feel this

impact. Two technologies which seem destined to shape our experience are Smart Cities and Robotics. Of course, for robots to be useful we need to trust that they will do their job properly and reliably, and this requires that we should be able to test them against standards, and see their performance with our own eyes.

Resulting from a series of initiatives funded by the European Union's Framework 7 and Horizon 2020 programmes, the European Robotics League (ERL) has been organising robot competitions for robots throughout Europe. Competitions are based around benchmarking, and aim to give robot developers data to enable the comparison of different systems, algorithms and approaches to solving complex tasks [1]. The ERL originally consisted of three challenges, which were suitable for Industrial, Service, and Emergency Robots (the latter challenge required that robots would work together in air, land and sea to solve a simulated disaster similar to that experienced in Fukushima). We have renamed the Industrial challenge as 'Professional' and the Service challenge as 'Consumer' to recognise the fact these sectors are converging around human and robot co-working with a focus on human-robot interaction.

One factor which defines the ERL's competition events is that they are most often in locations that are hidden away from the public such as robotics labs, test facilities, or disused ports. This distancing from the public is not deliberate and indeed not desired. Most often the competition events take place in locations which are determined by the need for accessible and safe indoor and outdoor environments that meet the needs of the benchmarking tasks at hand.

Surveys such as the Eurobarometer [2] have shown us that public attitudes towards robots are mixed, viewing them often with suspicion. It is unclear whether these attitudes are based on robotic fact or fiction; most people may not have had a direct experience of a robot. Since we wish to engage the public to raise issues around acceptance and desirability, and we wish to empower the people of Europe to direct robotic development, it seems clear that we should move our demonstrations of robots into venues that are easily accessible and where future use cases might involve people and robots working alongside each other.

It was for these reasons that the SciRoc project extended the European Robotics League to include the *Robotics in Smart Cities Challenge*. This allows teams to benchmark their robots and test their skills on a variety of tasks that one might find in a Smart City environment. Furthermore, teams need to interact with the Smart City data infrastructure while performing these tasks in order to demonstrate the value of the city host's investment and its readiness for future technology.

## **2. ERL processes for benchmarking and competition**

We follow the process of designing competitions that has been outlined in the RoCKIn project and visualised in **Figure 1**. One of the driving forces behind a competition is the formulation of a challenge. In the SciRoc context those challenges build the foundation of the three individual leagues:

**ERL Professional Service Robots (ERL Professional)**: How can mobile manipulation robots support low-volume, high-variety manufacturing?

**ERL Consumer Service Robots (ERL Consumer):** How can mobile manipulation robots support (elderly) citizens in their domestic environment?

**ERL Emergency Robots (ERL Emergency):** How can multi-domain robotic systems support rescue teams in emergency situations?

In all of these leagues, we follow the same methodology as outlined in the RoCKIn project;

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

• Functionality benchmarks (FBMs), which evaluate the performance of robot modules dedicated to specific functionalities, in the context of experiments focused on such functionalities.

• Task benchmarks (TBMs), which assess the performance of integrated robot systems facing complex tasks that require the interaction of different functionalities.

SciRoc integrates all three leagues in a common scenario of a Smart City. This particular challenge revolves around the question of how autonomous robots can support citizens in various smart shopping tasks.

While the challenges frame the overall benchmarks and competitions they are too complex to be solved within a reasonable time frame like a single EU-funded project and too generic to be solved by individual researchers. Thus, as a next step, interesting scenarios are derived from the challenge. A scenario consists of the task to be solved, the environment in which this task will be executed and the robot that will solve the task. Each of those three categories is defined in terms of their features, that is, a specification of the possible or allowed variability.

From the perspective of a scientific experiment the features are free or fixed parameters. The definition of a scenario also gives rise to a list of (abstract) functionalities that a robot is required to have. Examples of such functionalities may include task planning, manipulation, grasping, visual object recognition, speech understanding or speech generation. It should be noted that the scenario definition purely describes an application domain, but, by design, is not meant to provide insights into a robot's performance in solving a particular task in that domain. This latter aspect is addressed in the benchmark (competition) definition phase where the goal is to define tests that evaluate a robot's performance. The first step is the definition of questions or hypotheses that should be addressed by the benchmark. Only the right questions allow constraint of the overall design space of a scenario to a competition scenario where few, explicitly chosen features can be varied. The feature variation characterises each benchmark test by a choice or instantiation of all relevant features. Three major aspects govern those choices:


Given different feature variations their impact on the robot's performance can be measured and compared via well-defined metrics. Similarly, benchmarks for the functionalities are defined. The steps to conduct the benchmark are described in the benchmark procedures. One artefact of particular interest that is compiled during

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

this phase is the rulebook which communicates and clarifies the taken decisions and procedures to various stakeholders.

The next phase is the benchmark experiment execution where the tests are instantiated and robots solve the tasks in the real-world environment. The execution consists of the three activities (i) scenario setup, (ii) experiment control; and (iii) data collection. In the final phase, the resulting experimental data, also from multiple instantiations of benchmark experiments, is analysed to provide answers to the questions underlying the benchmark definition.

We now examine how the three European Robotics Leagues implement the same underlying methodology, and the ways in which this was expanded upon to create the Smart City Challenge.

## **3. ERL consumer robots**

The European Robotics League Consumer Service Robots (ERL Consumer) is a research competition that aims at bringing together the benefits of scientific benchmarking with the attraction of scientific competitions in the realm of consumer service robotics. The objectives are to foster research in consumer service robotics for home applications and to raise public awareness of the current and future capabilities of such robot systems to meet societal challenges like healthy ageing and longer independent living.

Currently, ERL Consumer raises challenges in domestic environments that resemble similar challenges to be posed in smart city environments (e.g., shopping malls), such as interacting with humans, recognising and picking objects from shelves, bringing the objects to the human who requested it, and/or moving outside the home to populated areas, so as to prepare teams for the Smart City Tournaments. But ERL Consumer objectives can be adapted to other challenges, still within domestic environments or in other environments, e.g., hospitals.

All the ERL Consumer TBMs and FBMs stem from a User's Story developed during the pioneer EU FP7 RoCKIn project [3]:

*Granny Annie is an elderly person, who lives in an ordinary apartment. Granny Annie is suffering from typical problems of ageing people: she has some mobility constraints. She tires fast. She needs to have some physical exercise, though. She needs to take her medicine regularly. She must drink enough. She must obey her diet. She needs to observe her blood pressure and blood sugar regularly. She needs to take care of her pets. She wants to have a vivid social life and welcome friends in her apartment occasionally, but regularly. Sometimes she has days not feeling so well and needs to stay in bed. She still enjoys intellectual challenges and reads books, solves puzzles, and socialises a lot with friends.*

For all these activities, ERL Consumer is looking into ways to support Granny Annie to live a full and independent life.

#### **3.1 Environment description and testbeds**

The ERL Consumer environment reflects an ordinary European apartment with all its environmental aspects, like walls, windows, doors, or blinds, as well as common household items, furniture and decoration. The apartment depicted in **Figure 2** serves as a guideline. It has

#### **Figure 2.**

*ERL consumer testbed: On the left a real testbed during a RoCKIn@home tournament in 2015; on the right: The reference testbed as a simulated environment.*


*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

The furniture and available objects (e.g., glasses, forks, knifes, pillows, cups) were chosen to set up a long-term research program with challenges for robot navigation (e.g., mirror in the wardrobe, tables with metallic reflective legs) and perception (e.g., glass-top table, natural backgrounds).

In addition to furniture and decoration, the apartment is equipped with a computer network infrastructure:


Home automation embedded devices may be installed and are accessible within the apartment's WLAN, e.g.,


Eight testbeds, certified to follow the standard specifications of this environment, are currently part of the ERL testbed network:


ERL Consumer tournaments take place in these testbeds. Some of the testbeds include a Motion Capture (MoCap) System, which enables tracking with high accuracy robots, people and object locations. MoCaps are mostly used in the FBMs.

## **3.2 Task benchmarks**

Currently, ERL Consumer includes 4 Task Benchmarks:


## **3.3 Functionality benchmarks**

Currently, ERL Consumer includes 6 Functionality Benchmarks [4, 5]:

• **Object Perception**: Evaluates the ability of a robot to recognise and localise a wide range of objects. A set of objects, selected from the list of ERL Consumer items, is positioned, one at the time, on a table located directly in front of the robot. For each object presented, the robot must perform the following activities: *Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

i) Object detection: perception of the presence of an object on the table and association between the perceived object and one of the object classes. ii) Object recognition: association between the perceived object and one of the object instances belonging to the selected class. iii) Object localization: estimation of the 3D pose of the perceived object with respect to the surface of the table (ground-truth provided by the MoCap).


## **4. ERL professional service robots**

ERL Professional Service Robots (ERL-PSR) League is focused on the major challenges addressed by H2020: industrial robots addressing the flexible factories of the future and modern automation issues.

Greater automation in broader application domains than today is essential to ensure European industry remains competitive, production processes are flexible to custom demands and factories can operate safely in harsh or dangerous environments. In the ERL-PSR competition, robots will assist in filling stocks in a department store. The task includes locating, picking, transporting and placing them in the proper shelfs. The combination of human versatility and reliability of mobile robots will optimise the entire process. The ERL-PSR competition is looking to make these innovative and flexible manufacturing systems, such as that required by the smart factory, a reality. This is the inspiration behind the challenge and the following scenario description.

In this version of the ERL-PSR we focus on the problem of picking products from a shelf and placing them in the shopping basket. The domain is a common department store, where the robot helps to arrange the inventory of the department store on the shelves. The main functionality tested in this episode is mobile manipulation using an autonomous robot. In recent years, mobile manipulation has become a problem of interest among researchers due to the variety and complexity of challenges and robot capabilities that are involved. For instance, in a grocery store setting, there are so many complexities that a robotic hand or gripper should take into account, such as handling objects of different weight, sizes, shapes, texture and compliance. Picking up a bag of pet food is very different from picking up bananas or cucumbers. Different robotic competitions [6–11] have picking and packing problem, aiming at 1) measuring the performance of these complex systems, 2) defining metrics and assessing benchmarking criteria, 3) fostering research and development of new approaches and technologies, and 4) creating awareness and drawing more attention from the general public to robotics.

## **4.1 Environment description and testbeds**

The ERL-PSR testbed consists of different elements which include the arena, networked devices, department store shelves and the products in the store. The robot can communicate with the department store inventory management system and to other networked devices. The robot receives tasks to perform from the inventory management system.

The following set of scenario specifications must be met by the ERL-PSR environment.

The environment can consists of various numbers of spatial areas:

1. rows of shelves

2. rows of workstations

**Figure 3**-a shows an example of these areas in the ERL-PSR environment. The spatial areas extend beyond the space occupied by the respective workstations or objects and include the surrounding area as well.

### **Figure 3.**

*ERL PSR different testbeds. a) ERL PSR testbed, Sankt Augustin, Germany. b) Testbed with objects and shelves.*

## *Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

All spatial areas are located on the same level, except where specified otherwise. There are no stairs in the environment. The environment is a replica of department store with aisles of shelves containing different objects. The environment has a boundary. The precise dimensions and the arrangement of the spatial areas are not predefined, but estimated sizes are given. The estimated sizes of the spatial areas are as follows: workstations 2 m × 2 m and shelves 5 m × 0, 5 m. The bounding box of the environment has a minimum area of 16m2 and a maximum area of 100m2 . More space is used, when areas and workstations are doubled for teams working in parallel. Workstations are used as storage areas for objects. They may be accessible from different locations, i.e. it might be possible to reach a workstation from two or more sides. Additionally, there are workstations of different heights present in the environment, ranging from 0 cm up to 15 cm. If a workstation has a height of 0 cm, a tape will mark the area (see **Figure 3**). The tape will be taped on the floor and is blue/white striped. This tape may be crossed by the robot and does not count as a collision.

## *4.1.1 Objects in the environment*

The objects to be manipulated will be selected by the organiser for each tournament. The following lists describe the different categories of objects that can be used by the organiser during a tournament depending upon the scenario being chosen. The different categories of objects are (**Figure 4**):


### *4.1.2 Robots and teams*

A competing team can use single or multiple robots. The robots are not required to be certified for industrial usage. The robot should have at least the following capabilities:


The different subsystems of the robot should work in the environment and manipulate the objects specified in the rulebook. The teams can use any sensor available in the market provided they are safe to use with humans and have the corresponding certification. The teams are not allowed to modify the arena which is finalised by the organiser. The teams are allowed to use any internal communication protocol. The robot shall pass through a safety test before it is admitted in the arena.

## **4.2 Task benchmarks**

## *4.2.1 Fill a box with parts*

This task is one of the primary task in a department store where customers fill their basket with products from shelves. The robots have to deal with flexible task specifications, especially concerning information about object constellations in source and target locations, and task constraints such as limits on the number of objects allowed to be carried simultaneously, etc. The robot has to pick up several parts from different source locations and deliver them to several destination locations.

## **4.3 Functionality benchmarks**

The task benchmark was subdivided into multiple Functionality Benchmarks. The list of benchmarks executed are:


## **5. ERL emergency robots**

ERL Emergency tournaments challenge multi-domain teams of marine, land and aerial robots with search and rescue tasks, inspired by a disaster response scenario [6].

## *Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

Following the experience of the euRathlon 2015 Grand Challenge [7], the ERL Emergency competition started in 2016 and culminated in the ERL 2017 Major Tournament [8] held at the Tor del Sale power plant site of Piombino, on the coast of Tuscany (Italy). The inspiration came from the 2011 Fukushima accident [9]. Land, marine and aerial robots cooperated in a disaster scenario organised at a real power plant, and were required to survey the accident area, to identify and help missing workers (mannequins) and to intervene to stem a leak by closing valves both inside the building containing the machine room of the plant and underwater.

Similar concepts were followed in the SciRoc project, and in 2018 and 2019, ERL Emergency moved to a model of local tournaments with two-domain competitions: land + underwater and land + aerial robots competitions. The Centre for Maritime Research and Experimentation (CMRE) organised two editions of the land + underwater robot competition at its premises in La Spezia (Italy) in 2018 and 2019. The land + aerial competition was hosted by the Advanced Centre for Aerospace Technologies (CATEC) at Seville (Spain) in 2019. These events were a preparation for the Smart City events, where only the tasks of the ERL Emergency involving the aerial robots were present.

Over the years, ERL Emergency has proposed team tasks which required advanced perception skills (to locate numbers positioned underwater, underwater pipes, or persons for first-aid kit deliveries using aerial robots), intertwined with autonomy and cooperation capabilities in realistic scenarios. Multi-domain cooperation has been specifically searched and pushed. Robots are required to be able to accomplish adaptive missions, for instance executing different actions on the basis of different sensed conditions. The increased attention to autonomy and cooperation has also been accompanied by an increasingly metrological attention to benchmarking the robot performance. This has been achieved by separating the Task Benchmarks from the Functionality Benchmarks, following the general trend of the ERL evaluation framework [10].

## **5.1 Land+underwater robots events held at CMRE sea water basin**

In the SciRoc project, CMRE conducted two ERL Emergency local tournaments, in 2018 and 2019, challenging multi-domain teams composed of an underwater and a land robot with a scenario of emergency response to a simulated explosion in a harbour. The areas of operations included a building with the surrounding outdoor space for land robots, and the CMRE sea water basin for underwater robots (see **Figure 5**).

#### **Figure 5.**

*(left) the Feelhippo AUV from the UNIFI robotics team in action. (Centre) the yellow pipe structure with the manipulation console used in the ERL emergency competitions. (right) a multi-beam mosaic by team Tomkyle of the competition area. The different objects of potential interest (buoys, pipeline assembly structures, the gate are visible).*

The marine robots were challenged with the realistic conditions typical of at-sea operations (water salinity, changing light conditions, waves and tides).

The scenario represented an accident of a yacht in a harbour. The vessel clashed on the dock causing an explosion. The accident affected the area around the harbour, both outdoor and a building. The land robots (unmanned ground vehicles – UGVs) were tasked to survey the outdoor area, localise pipes and find a missing worker (represented by a mannequin). The robots were requested to deliver a first-aid kit in proximity to the mannequin. Successively, inside the building, the robots were required to localise and close a valve and to find a canister. The specific valve to be closed was communicated to the UGV by the underwater robot of the same team. UGVs needed to transport the canister from inside the building to a simulated outdoor fire location. Autonomous underwater vehicles (AUVs) had to pass through a validation gate composed of two submerged buoys, survey an area to detect a missing person underwater (a realistic mannequin) and localise an emitting pinger (see **Figure 5**). They also were required to inspect a pipeline structure and localise a damage (represented by a marker), at the same time reporting its shape and size. Underwater robots had to be autonomous, which means that all navigation and perception tasks were needed to be accomplished autonomously as well. Buoys of different colours were to be identified, localised and their colour recognised. The robots had to perform a different action, depending on the buoy colour: as an example, turning in a clockwise circle around the buoy or stopping for 30 seconds increasing the depth. The objective was to push teams to integrate perception with adaptive and reactive mission planning in a realistic scenario such as presented in the CMRE water basin. Here the changing and real conditions, such as the limited visibility underwater, created severe difficulties for object recognition by robots, even in the case the buoys were bright orange or red in colour.

Our experience suggests that one of the important aspects in real-world competitions is the possibility to guarantee teams sufficient time for practicing and tuning their systems to the changing and complex environmental conditions. This is especially true for marine robots, since teams may have difficulties to have access to sea waters for testing before the event. To allow teams to test, a practice arena was prepared.

As a way to increase the level of the challenge, the size of the objects to be detected was reduced with respect to previous competitions. Some of the tasks and functionalities were common to the ERL Emergency Local Tournament in Seville (land + aerial robots), such as the object recognition or map building functionalities. This allowed for a similar comparison of results in both competitions for a given Functionality Benchmark (FBM) (e.g. object detection). The same kind of objects of potential interest were used so the Object Detection FBM could be evaluated in both competitions. Some of the tasks were required to be completed with robots operating in a fully autonomous way, while others could be accomplished remotely controlled (e.g. with the assistance of an operator). However, no manual perception tasks were allowed. All object detections had to be either autonomous (real-time by the robot) or automatic (offline by a computer).

In both events, a mix of well-experienced teams and new teams attended the competitions. In 2018, five teams participated (three for marine domain, one for land and one deploying robots for both domains), while in 2019 [11], the number of teams increased with the participation of seven teams (four for marine domain, two for land domain and one with both segments).

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

Results highlight the improvement of team performance over the years, especially for teams which succeeded in participating in multiple editions of our events. Our policy to increase the task difficulty year after year led in fact the teams to increase their technical and, above all, management skills, thanks to the gained experience from their unavoidable errors and from the interactions with other entries. For these reasons, it is important to guarantee a continuity in the organisation of the competitions, especially when robots have to handle real-world scenarios in non-controllable conditions, which dramatically increase the task difficulties.

#### **5.2 Land+aerial robots events held at CATEC**

CATEC organised an ERL Emergency local tournament in February 2019, targeting aerial and land robots working in an outdoor/indoor environment [12]. The scenario was an earthquake in an industrial area near a factory building, where a robotic team composed of land (UGV) and air robots (UAV) had to intervene. The priorities are to discover and assist missing people, and determine if the building has suffered any serious damage. The robots have to search for missing workers (one outdoors and one indoors), find them as soon as possible and deploy an emergency kit to both of them. Besides, the robots must check the state of the building after the earthquake, for which a detailed map is required to assess the safety of the area (see **Figure 6**).

As mentioned before, some of the functionalities were shared with the land + underwater robot competitions. In this case, the same kind of markers were used as objects to be recognised, so the Object Detection FBM could be properly evaluated in both competitions. From the starting points, land and aerial robots must inspect the area, autonomously detecting and avoiding obstacles while traversing the competition arena. Then, they needed to find a suitable entrance that could be used to enter the building, using similar markers as the ones used for object recognition. While accessing the building, robot navigation must deal with transitioning from an outdoor to an indoor environment. Once inside, robots had to build a detailed map of the scenario, either 2D or 3D according to their sensor suite. During the whole mission execution, as soon as a missing worker was found (represented by a mannequin), the robot provided a first-aid kit as soon as possible.

The setup for the competition was an area of approximately 200 m x 30 m free of obstacles, in an area where operating UAVs in Visual Line Of Sight (VLOS) is not forbidden according to regulations in Spain. The building was represented with a 20 m x 20 m marquee where the aerial and ground robots need to access. Static obstacles (e.g.

#### **Figure 6.**

*(left) the UAV and UGV from team LARICS in action, reaching the outdoor missing worker. (Centre) the rover UGV and the UAV from team raptors looking for the indoor missing worker. (right) top view of a 3D map of the competition area by team LARICS. The trajectory followed by the robot is shown in blue.*

stones, holes, vegetation...) and dynamic obstacles (e.g. birds...) were expected in the outdoor area, as well as loss of Wi-Fi signal. As with any outdoor competition, there was also the possibility of rain, wind, dust and muddy areas, but weather was generally fine during that week. A practice area was also set up to provide teams enough space for testing their systems before deploying them in the competition arena.

## **5.3 Task benchmarks**

TBMs in ERL Emergency require the cooperation of robots in different domains. This often requires teams to work together.

## *5.3.1 Yacht accident in the harbour (land+sea)*

This two-domain task benchmark is focused on acquiring knowledge about the environment and its explicit representation; and to cooperate between domains to search for the missing workers and give them assistance. The ground and underwater robots are required to understand the changes in the environment and interact with it either through cooperation between them (autonomous robot-robot) or their operators (human-robot interaction) or with a mixed approach. A minimum of one land robot and one underwater robot is required to participate in this task.

The motivating scenario is as follows;

*An accident occurs in the harbour when a yacht arriving at the pier damages a gas pipeline which leaks and causes an explosion. This also affects the building containing the pipeline section on land and people that were in the docks area are dispersed. The emergency response team arrives soon but members must maintain a safe distance from the fire. For this reason, the use of robotic vehicles is essential. A robotics team composed of land (UGV) and underwater robots (AUV) is ready to intervene.*

Three missions can be undertaken.


## *5.3.2 Emergency in a building (land+air)*

This two-domain task benchmark is focused on acquiring knowledge about the environment and its explicit representation; and to cooperate between domains to search for the missing workers and give them assistance. The ground and aerial robots are required to understand the changes in the environment and interact with it either through cooperation between them (autonomous robot-robot) or their operators (human-robot interaction) or with a mixed approach. A minimum of one land robot and one aerial robot is required to participate in this task.

The motivating scenario is as follows…

*An earthquake has occurred in an industrial area near a factory building, and two workers are missing (one inside, and one outside). The emergency response team arrives soon but members of the response team must maintain a safe distance from the building. For this reason, the use of robotic vehicles is essential. A robotic team composed of land (UGV) and air robots (UAV) must discover any missing people and whether the building has suffered any serious damage. Robots have to find wrokers as soon as possible and deploy an emergency kit to each. Robots must check the state of the building after the earthquake and create a detailed map for the emergency team.*

This task benchmark comprises two missions' goals:


## **5.4 Functionality benchmarks**


disaster response environment. The benchmark requires that robots detect Objects of Potential Interest (OPIs) and identify the type of each object found, and is scored according to the precision and accuracy of identifying and locating the objects.

## **6. Smart cities competitions**

The label 'Smart Cities' is used with increasing frequency with a main focus on the use of engineering approaches within the built environment to improve citizens' quality of life. These visions present the city as an entity, which gathers and processes data to decision makers and infrastructure, and which offers data services to enable and improve services offered by other stakeholders such as private companies. Since the city senses its environment, processes this data, and takes action through effectors like traffic signals, the Smart City resembles a Robot. We might expect to find that Smart Cities and Robots gain mutual benefits from their integration, and also find some common problems which both must overcome.

The world's population is becoming more urban, so it is likely that most humanrobot interactions will take place in Smart Cities. Therefore, it seems that any ethical challenges that arise from the mutualistic interactions of Robots and Smart Cities will have a significant impact for these future citizens. How can robots act best to sustainably promote our happiness and prosperity?

When we first stated our intention to bring a scientific robot competition to the heart of a major public space, our ambition was to address this question by stimulating public discussions based on robotic facts about the roles which robots might play in the future in their smart city. Our motivation was to demonstrate that robots and smart cities are natural partners which add mutual value, and in doing so to showcase the state of the art in European Robotics to benefit the teams, sponsors, and wider community. We were able to achieve our aims through the successful delivery of two ERL Smart Cities Challenges to date. The first in Milton Keynes, UK in September 2019 and the second in Bologna, Italy in September 2021.

Our intention was to provide for the sustainable future of the ERL beyond the H2020 funded SciRoc project, and as part of this our first competition was led by a project partner working with the city, and with some funding from the project budget, while the second was led by a third party and entirely self-funded. This demonstrates that this mode of public engagement with benchmarking through competition can be repeated using this model in the future.

In the following sections we explain how and why the TBMs and FBMs of the ERl were adapted and extended to create the Smart City Episodes. We also explain how the critical elements which need to be considered for a city host looking to implement a future Smart City event, or similar urban robotic benchmarking challenge.

#### **6.1 Milton Keynes, 2019**

The first Smart City event took place at the Milton Keynes Centre: MK shopping mall over five days, providing teams the opportunity to compete in five different episode scenarios. This public space attracts on average over half a million people each week, and many of these people paused to watch the robots as they competed. The event's sponsors included PAL Robotics and Ocado, and attracted 11 international robotics teams from different Universities throughout Europe.

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

Allied events included a Symposium to debate the risks and opportunities associated with the emergence of a "hybrid society", and a Workshop on the Evaluation and Benchmarking of Human-Centered AI Systems. This public engagement helped foster public debate and equip citizens to be engaged active stakeholders in their city's future.

#### **6.2 Bologna, 2021**

The second Smart City event took place at Palazzo Re Enzo, in the heart of Bologna. In contrast to the first Smart City event in Milton Keynes, this competition was organised by the host city, which provided all the resources to run the event, assisted by technical and scientific support from the SciRoc Consortium to set up and run the five episodes.

The implementation of the competition had to face significant uncertainty caused by the Covid-19 pandemic. Challenges arising through unforeseen travel restrictions were addressed with innovative technical solutions in terms of the arrangements of the competition, to allow teams and robots to compete both locally and remotely.

Such measures included the introduction of a simulation environment for a service robots scenario; remote participation on a physical robot made available on site; introduction of a novel challenge related to sign language that had outstanding success within the deaf community; synergy with the H2020 Eurobench project aiming at implementing benchmarks for robots; design of a new challenge for emergency robots; and finally distributed execution of the competition with both on site and remote participants performing the tests at their own labs.

#### **6.3 Hosting a Smart City event**

In this section we briefly describe the process of hosting a SciRoc Smart City competition in the hope that this assists readers interested in running a similar benchmarking event, or hosting a future Smart City Event.

#### *6.3.1 The selection process*

Smart City hosts are determined through an open call process. Interested candidates can register their interest in response to the call and are required to submit a proposal for the event, and complete an interview with the selection committee. The selection committee, comprising members selected to remove conflicts of interest, determined the host through a final vote.

#### *6.3.2 Competition design*

The competition is divided into a series of robotic challenges, referred to as episodes. This represents an innovation of the structure of the competition, based on Functionality Benchmarks (FBMs) and Task Benchmarks (TBMs). Episodes are intended to be an intermediate challenge between the local tournaments' FBMs and the TBMs. Even though the methodology and the approach to the competition are de facto unchanged, the proposed format based on episodes is designed to make a step forward towards their application in a realistic scenario, to allow for a better communication towards the general public and to lower the entry level for the competing teams.

An episode places a given functionality, tested during a specific FBM in one of the other ERL challenges, into a more social and operational context, while limiting the amount of effort needed for their development.

Episodes are organised into categories depending on the task to achieve and the type of robots involved. We group them into three categories:


## *6.3.3 Funding*

The event is primarily funded through sponsorship funding, although it is expected that the host city will make their own contribution. For example, at the 2021 event in Bologna, the Municipality allowed the use of the competition venue Palazzo Re Enzo at no cost. This contribution was worth 65,000 EURO in kind. The University of Bologna also contributed 36,000 EURO in cash sponsorship towards the event.

## *6.3.4 Logistics*

The host city is responsible for organising the venue for the event. When selecting the venue, organisers should consider that the location must be able to accommodate a number of factors, including: infrastructure for a minimum of five episodes, accessibility for teams to move robots around the venue, the potential use of drones within the venue, secure storage areas for robots and equipment, to name a few.

In addition to the competition venue, the host city should consider the following requirements for the functioning of the event: robot transportation, accommodation, financial support for teams, co-located events, the organising team and roles required within and the data hub.

### *6.3.5 The data hub*

Central to emerging Smart Cities are online platforms for data sharing and reuse, which are normally called Data Hubs. Such Data Hubs have two purposes. They provide static datasets, such as demographic data which changes slowly, and dynamic live data gathered from sensors deployed within the city. The latter could include data about traffic flows, the environment, and the movement of people and robots, etc.

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

The MK Data Hub is a state of the art computational infrastructure, which supports the acquisition and management of city data. Specifically, it provides both a catalogue of several hundred data sources, as well as a development environment to facilitate the creation of data-intensive applications. The MK Data Hub played a key role in the 2019 SciRoc event in Milton Keynes by allowing us to simulate the diversity of systems with which robots were asked to interact throughout the competition.

Going forward, this approach could contribute a valuable additional output of robot competitions. First, by enabling the dynamic simulation of the environment in which the robots operate, the MK Data Hub introduced a contextual element in the benchmarks.

Second, a Data Hub environment provides the opportunity to record a large amount of heterogeneous information about the robots' behaviour. Such information can be reused (a) to enable a deeper analysis of the shortcomings of robots, (b) to compare the performance of the same robot in different trials, and in different competitions among the years, as well as (c) for archival purposes. Ultimately, recording robot messages and analysing their behaviour may constitute the first step towards benchmarking elements such as self-awareness and deliberation capabilities.

#### *6.3.6 Teams*

Teams are recruited through an open call process, which is disseminated via the SciRoc website and social platforms, mailing lists, and by the SciRoc Consortium within academic institutions and the Robotics community. All previous year's teams are also contacted directly to notify them of the open call.

#### **6.4 Scientific contributions**

SciRoc had the ambition to provide a testbed for experimental evaluation of Human-Robot Interaction (HRI) approaches. In fact, competitions offer a unique opportunity to create interaction scenarios for evaluating the performances of different approaches to human-robot interaction. In this respect, the work carried out in SciRoc had an impact on the HRI research community: the results of the experimental evaluation carried out in the SciRoc episodes, specifically designed for this purpose, have been published in the top venues for HRI research. Specifically, the Elevator episode of SciRoc 1, where the robot had to interact with users in taking an elevator, was accompanied by the creation of an ad hoc questionnaire [13] and the data collected during the competition have been used to demonstrate the potential for carrying out experimental studies within robotic competitions [14, 15]. Moreover, the Sign Language episode of SciRoc 2, where the robot was supposed to interpret and produce phrases of the Italian sign language, was accomplished with an unprecedented collaboration and impact with the deaf people community [16].

#### **6.5 The continuation of Smart City events in the future**

Results from teams are encouraging, and successful attempts at challenges of annually increasing difficulty suggest that teams have grown in capability through their repeated engagement with the benchmarking process. We have underlined the importance of continuing the organisation of such events for supporting the growth of teams and research groups, providing them with an annual real-world ground where to test their systems. This is especially true after the lockdown caused by the

COVID19 pandemic, which increased the difficulties for teams to access real-world training areas (in particular marine sites).

Although the SciRoc project will finish in 2022, we are working to continue the ERL and the Smart City Events after the project's close. The euRobotics international non-profit association will adopt the Smart City Events and continue to run them through an open call process, as with our previous Bologna event. It is intended that the design of the SciRoc events are responsive to the needs of the city stakeholders and, to some extent, the events are designed in order to support the communications and ambitions of the host city. We are therefore confident that city partners will continue to respond to our open call, to host Smart City events in the future.

The SciRoc Smart City Events saw collaboration with the H2020 funded EUROBENCH project to extend its work on benchmarking humanoids, and the H2020 METRICS project has been adopted within the ERL brand, following ERL benchmarking methodology. We hope that this will be repeated in the future, and anticipate new ERLs. Future Smart City competitions will provide a venue for more European Robotics projects to showcase their outputs to the public.

## **7. Conclusions**

In this chapter we presented the SciRoc project which continued the work of the ERL and extended it to Smart City Events. We showed how the common TBM & FBM approach of the ERL was adapted and extended to enable these events to increase the value of the benchmarking endeavour through a massive increase in public engagement. The Smart City events delivered value to multiple stakeholder groups;


Bringing Robotics Benchmarking into the public arena in this way is important, we feel. For robots to be useful, they must be trusted, and for this trust to grow, real robots must be seen and their capabilities assessed.

*Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

Due to the restrictions placed upon us by the Covid-19 pandemic and the associated restrictions in travel and public assembly, we had to find new ways of working which would allow the teams flexibility in the ways they interacted with the Smart City Episodes. These methods may provide a basis for future development of benchmarking, and point the way towards a mixture of physical and virtual assessment.

As the project comes to an end, we are happy that we have surpassed our objectives, overcome unforeseen challenges, and that we have prepared the way for the continuation of the ERL through the support of the European Scientific Community and other stakeholders in the years to come. We hope that the trend to harmonise benchmarking under the ERL umbrella continues, and that the Smart City Events provide an arena for more European Projects to meet the public and therefore advance the successful and safe use of robotics within Europe and beyond.

## **Author details**

Matthew Studley1 \*, Sarah Carter1 , Francisco J. Perez-Grau<sup>2</sup> , Antidio Viguria Jiménez2 , Gabriele Ferri3 , Fausto Ferreira4 , Deebul Nair5 , Sven Schneider<sup>5</sup> , Paul G. Plöger<sup>5</sup> , Pedro U. Lima6 , Meysam Basiri6 , Gerhard K. Kraetzschmar<sup>5</sup> , Daniele Nardi7 , Lun Wang7 , Emanuele Antonioni7 , Vincenzo Suriani<sup>7</sup> and Luca Iocchi7

1 University of the West of England, Bristol, UK

2 Advanced Center for Aerospace Technologies (CATEC), Seville, Spain

3 Centre for Maritime Research and Experimentation (CMRE), La Spezia, Italy

4 Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia

5 Hochschule Bonn-Rhein-Sieg, Germany

6 Instituto Superior Técnico, University of Lisbon, Portugal

7 Sapienza University of Rome, Italy

\*Address all correspondence to: matthew2.studley@uwe.ac.uk

© 2023 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## **References**

[1] Lima PU, Nardi D, Kraetzschmar GK, Bischoff R, Matteucci M. RoCKIn and the European robotics league: Building on RoboCup best practices to promote robot competitions in Europe. In: RoboCup 2016: Robot World Cup. Vol. XX. Springer International Publishing; 2017. pp. 181-192

[2] Eurobarometer, Special. Public Attitudes towards Robots. European Commission. 2012 http://www.ab.gov.tr/ files/ardb/evt/Public\_attitudes\_toward\_ robots\_2012.pdf

[3] Lima PU, editor. RoCKIn-Benchmarking through Robot Competitions. London, UK, London, UK: IntechOpen; 2017

[4] Basiri M, Piazza E, Matteucci M, Lima P. Benchmarking functionalities of domestic service robots through scientific competitions. KI-Künstliche Intelligenz. 2019;**33**:357-367

[5] Basiri M, Pereira J, Bettencourt R, Piazza E, Fernandes E, Azevedo C, et al. Functionalities, benchmarking system and performance evaluation for a domestic service robot: People perception, people following, and pick and placing. Applied Sciences. 2022;**12**(10):4819

[6] Ferreira F, Ferri G. Marine robotics competitions: A survey. Current Robotics Reports. 2020;**1**:169-178. DOI: 10.1007/ s43154-020-00022-5

[7] Ferri G, Ferreira F, Djapic V, Petillot Y, Palau M, Winfield A. The euRathlon 2015 grand challenge: The first outdoor multi-domain search and rescue robotics competition - a marine perspective. Marine Technology Society Journal. 2016;**50**(4):81-97

[8] Ferri G, Ferreira F, Djapic V. Multi-domain robotics competitions: The CMRE experience from SAUC-E to the European robotics league emergency robots. In: OCEANS 2017 - Aberdeen. 2017. pp. 1-7. DOI: 10.1109/ OCEANSE.2017.8084767

[9] Nagatani K, Kiribayashi S, Okada Y, Otake K, Yoshida K, Tadokoro S, et al. Emergency response to the nuclear accident at the Fukushima Daiichi nuclear power plants using mobile rescue robots. Journal of Field Robotics. 2013;**30**(1):44-63

[10] Ferreira F, Ferri G, Petillot Y, Liu X, Franco MP, Matteucci M, et al. Scoring robotic competitions: Balancing judging promptness and meaningful performance evaluation. In: In 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). IEEE; 2018. pp. 179-185

[11] Ferri G, Ferreira F, Faggiani A, Fabbri T. From ERL to RAMI: Expanding marine robotics competitions through virtual events. In: OCEANS 2021: San Diego – Porto. 2021. pp. 1-8. DOI: 10.23919/ OCEANS44145.2021.9706098

[12] Ferri G, Ferreira F, Viguria A, Perez-Grau F. Aerial and underwater robots in competitions. IEEE Robotics & Automation Magazine. 2018;**25**(4):119-120

[13] Wang L, Iocchi L, Marrella A, Nardi D. Developing a questionnaire to evaluate customers' perception in the Smart City robotic challenge. In: 2019 28th IEEE International Conference on Robot and Human Interactive Communication (ROMAN). *Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot... DOI: http://dx.doi.org/10.5772/intechopen.108583*

IEEE; 2019. pp. 1-6. DOI: 10.1109/ RO-MAN46459.2019.8956394

[14] Wang L, Iocchi L, Marrella A, Nardi D. HRI users' studies in the context of the SciRoc challenge: Some insights on gender-based differences. In: Proceedings of the 8th International Conference on Human-Agent Interaction (HAI '20). New York, NY, USA: Association for Computing Machinery; 2020. pp. 287-289. DOI: 10.1145/3406499.3418763

[15] Marella A, Wang L, Iocchi L, Nardi D. A methodology to design and evaluate HRI teaming tasks in robotic competition. Journal ACM Transaction on Human-Robot Interaction. 2022. DOI: 10.1145/3528415

[16] Antonioni E et al. Nothing about us without us: A participatory design for an inclusive signing Tiago robot. In: 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). 2022. pp. 1614-1619. DOI: 10.1109/ RO-MAN53752.2022.9900538

## *Edited by Ramana Vinjamuri*

The book *Human-Robot Interaction - Perspectives and Applications* highlights the latest developments and obstacles in the field of human-machine interaction, including collaborative and humanoid robots, symbiosis between humans and robots, humanhuman collaboration, and robotics. Human-robot interaction has immense potential in areas like healthcare, education, manufacturing, military, and space exploration. This volume consists of several chapters that explore various topics such as the use of robotic wheelchairs, deep neural networks for robot grasp recognition, materials and sensors required for human-robot interaction, the use of drone technology in agriculture, healthcare robots in smart hospitals, and more.

Published in London, UK © 2023 IntechOpen © Artystarty / iStock

Human-Robot Interaction - Perspectives and Applications

Human-Robot Interaction

Perspectives and Applications

*Edited by Ramana Vinjamuri*