**2. Unmanned decision making loop**

In this section, we describe the decision making process currently used by UUV systems and we introduce the unmanned decision loop, where observations, orientations, decisions and actions (OODA) occur in a loop enabling adaptive mission planning.

In order to describe the unmanned decision loop, we need to start by modelling the mission environment. A mission environment is defined by the tuple Π = (Σ, Ω), where:

2 Underwater Vehicles

by the deliberative agents while performing a mission. These agents, providing different capabilities and working in collaboration, might even be distributed among the different

In this chapter, we first provide a review to the different approaches solving the decision making process for UUV missions. Then, we propose a semantic framework that provides a solution for hierarchical distributed representation of knowledge for multidisciplinary agent interaction. This framework uses a pool of hierarchical ontologies for representation of the knowledge extracted from the expert and the processed sensor data. It provides a common machine understanding between embedded agents that is generic and extendable. It also includes a reasoning interface for inferring new knowledge from the observed data and guarantee knowledge stability by checking for inconsistencies. This framework improves local (machine level) and global (system level) situation awareness at all levels of service capabilities, from adaptive mission planning and autonomous target recognition to deliberative collision avoidance and escape. It acts as as an enabler for on-board decision

Based on their capabilities, service-oriented agents can then gain access to the different levels of information and contribute to the enrichment of the knowledge. If the required information is unavailable, the framework provides the facility to request other agents with the necessary capabilities to generate the required information, i.e. an target classification algorithm could query the correspondent agent to provide the required object detection

Secondly, we present an algorithm for autonomous mission adaptation. Using the knowledge made available by the semantic framework, our approach releases the operator from decision making tasks. We show how adaptation plays an important role in providing long term autonomy as it allows the platforms to react to events from the environment while at the same time requires less communication with the operator. The aim is to be effective and efficient as a plan costs time to prepare. Once the initial time has been invested preparing the initial plan, when changes occur, it might be more efficient to try to reuse previous efforts by repairing it. Also, commitments might have been made to the current plan: trajectory reported to other intelligent agents, assignment of mission plan sections to executors or assignment of resources, etc. Adapting an existing plan ensures that as few commitments as possible are invalidated. Using plan proximity metrics, we prove how similar plans are more likely to be accepted and trusted by the operator than one that is potentially completely different. Finally, we show during a series of in-water trials how these two elements combined, a decision making algorithm and shared knowledge representation, provide the required interoperability between embedded service-oriented agents to achieve high-level mission goals, detach the operator from the routinary mission decision making and, ultimately, enable

In this section, we describe the decision making process currently used by UUV systems and we introduce the unmanned decision loop, where observations, orientations, decisions and

In order to describe the unmanned decision loop, we need to start by modelling the mission

environment. A mission environment is defined by the tuple Π = (Σ, Ω), where:

platforms or sharing some limited resources.

analysis before proceeding with its classification task.

the permanent presence of dynamic sensing networks underwater.

actions (OODA) occur in a loop enabling adaptive mission planning.

**2. Unmanned decision making loop**

**1.1 Contribution**

making.


The set of all possible mission environments for a given domain is defined as the *domain space* (e.g., the domain space of the underwater domain). It is denoted by Θ. A mission environment Π is an element of one and only one Θ.

From this model, a mission plan *π* that tries to accomplish the mission objectives can be produced. However, this mission environment evolves over time *t* as new observations of the domain model Σ*<sup>t</sup>* and the problem model Ω*<sup>t</sup>* continuously modify it:

$$
\Pi\_t \leftarrow \Pi\_{t-1} \cup \Sigma\_t \cup \Omega\_t \tag{1}
$$

The decision making process to calculate a mission plan *π<sup>t</sup>* for a given mission environment Π*<sup>t</sup>* occurs in a cycle of observe-orient-decide-act. This process was termed by Boyd (1992) as the OODA-loop, and it was modelled on human behaviour. Inside this loop, the *Orientation* phase contains the previously acquired knowledge and initial understanding of the situation of the mission environment (Π*t*−1). The *Observation* phase corresponds to new perceptions of the mission domain model (Σ*t*) and the mission problem model (Ω*t*) that modify the mission environment. The *Decision* component represents the level of comprehension and projection, the central mechanism enabling adaptation before closing the loop with the *Action* stage.

Note that it is possible to make decisions by looking only at orientation inputs without making any use of observations. In this case, Eq. 1 becomes Π*<sup>t</sup>* ← Π*t*−1. In the same way, it is also possible to make decisions by looking only at the observation inputs without making use of available prior knowledge. In this case, Eq. 1 becomes Π*<sup>t</sup>* ← Σ*<sup>t</sup>* ∪ Ω*t*.

In current UUVs implementations, the human operator constitutes the decision phase. See Figure 1 for a schematic representation of the control loop. When high bandwidth communication links exist, the operator remains in the OODA-loop during the mission execution taking the decisions. For each update of the mission environment Π*<sup>t</sup>* received, the operator decides on the correspondent mission plan *π<sup>t</sup>* to be performed. From the list of actions in this mission plan, the mission executive issues the correspondent commands to the platform. Examples of the implementation of this architecture are existing Remotely Operated Vehicles (ROVs).

However, when communication is unreliable or unavailable, the operator must attempt to include all possible if-then-else cases to cope with execution alternatives before the mission starts. This is the case of current UUVs implementations that follow an orientation-only model. Figure 2 shows this model, where the OODA-loop is broken because observations are not reported to the human operator.

We will now discuss a few recent UUV implementations which show where the state-of-the-art is currently positioned. Most implementations rely on pre-scripted mission

Fig. 1. Observation, Orientation, Decision and Action (OODA) loop for unmanned vehicle systems with decision making provided by the human operator.

guidance control (kinematics) uncoupled (Caccia & Veruggio, 2000). Although it contained a

<sup>203</sup> Embedded Knowledge and Autonomous Planning:

A detailed survey of other behaviour-based approaches applied to mission control systems

More recently, Benjamin et al. (2009) has applied multiple objective decision theory to provide a suitable framework for formulating behaviour-based controllers that generate Pareto-optimal and satisfying behaviours. This approach was motivated by the infeasibility of optimal behaviour selection for real-world applications. This approach has been implemented and deployed as part of the IvP Helm extension to MOOS. This method seems to be a more suitable for behaviour selection, although more computationally expensive. Also, the approach is limited to the control of only the direction and velocity parameters of the host

After reviewing this related work, two problems affecting the effectiveness of the decision loop become evident. Firstly, orientation and observation should be linked together because it is desirable to place the new observations in context. Secondly, decision and action should be iterating continuously. These two problems have not been addressed together by previous approaches. These are the two of the goals that we address in this chapter. In order to achieve them autonomously, two additional components are required: a status monitor and a mission plan adapter. The status monitor reports any changes detected in the mission environment during the execution of a mission. When the mission executive is unable to handle the changes detected by the status monitor, the mission planner is called to generate a new modified mission plan that agrees with the updated mission environment. Figure 3 shows the OODA-loop for autonomous decision making. Comparing it to the previous Figure 2, the addition of status monitor and mission planner removes the need for human decisions in the loop. Note that the original mission plan *π*<sup>0</sup> could also be autonomously generated as long as

declarative mission representation, missions were programmed manually.

the high-level goals are provided by the human operator in Π0.

Mission

Fig. 3. Required OODA-loop for autonomous decision making in UUVs. Decision stage for adaptation takes place on-board based on initial orientation provided by the operator and

Adaptive mission planning enables a true unmanned OODA-loop. This autonomous decision making loop copes with condition changes in the mission environment during the mission execution. As a consequence, it releases the operator from decision making tasks in stressful

The potential benefits of adaptive mission planning capabilities for autonomous decision making in UUVs were promoted by Turner (2005), Bellingham et al. (2006) and Patrón & Petillot (2008). Possibly the most advanced autonomous decision making framework for UUVs has been developed at the Monterey Bay Aquarium Research Institute. This architecture, known as *T-REX*, has been deployed successfully inside the Dorado AUV (Rajan

Mission Adapter

πt

Ofine Online

π0

environments containing high levels of uncertainty and dynamism.

Commands

Πt

Executive Platform Environment

Status Monitor

Events

Observations

Mission Generator

Π<sup>0</sup>

observations provided by the status monitor.

for UUVs can be found in Carreras et al. (2006).

The Path Towards Permanent Presence of Underwater Networks

platform.

Fig. 2. Broken OODA-loop. Decision stage on the human operator based only on initial pre-mission orientation.

plan managers that are procedural and static and might not even consider conditional executions (Hagen, 2001). At this level, the mission executive follows a sequence of basic command primitives and issues them to the functional control layer of the platform. Description about how these approaches maintain control of underwater vehicles can be found in Fossen (1994), Ridao et al. (1999) and Yuh (2000). In this situation, decisions taken by the operator are made using only orientation inputs related to some previous experience and *a-priori* knowledge. This has unpredictable consequences, in which unexpected situations can cause the mission to abort and might even cause the loss of the vehicle (Griffiths, 2005; von Alt, 2010).

More modern approaches are able to mitigate this lack of adaptability by introducing sets of behaviours that are activated based on observations (Arkin, 1998). Behaviours divide the control system into a parallel set of competence-levels. They can be seen as manually scripted plans generated *a-priori* to encapsulate the decision loop for an individual task. Under this approach, the key factor is to find the right method for coordinating these competing behaviours.

The subsumption model, attributed to Brooks (1986), arbitrates behaviour priorities through the use of inhibition (one signal inhibits another) and suppression (one signal replaces other) networks. Most recent UUV control systems are a variant of the subsumption architecture.

This model was first applied to the control of UUVs by Turner (1995) during the development of the ORCA system. This system used a set of schemas in a case-based framework. However, its scalability remains unclear as trials for its validation were not conducted.

Later, Oliveira et al. (1998) developed and deployed the CORAL system based on Petri nets. The system was in charge of activating the vehicle primitives needed to carry out the mission. These primitives were chained by preconditions and effects.

The scaling problem was addressed by Bennet & Leonard (2000) using a layered control architecture. Layered control is a variant of the subsumption model that restricts of interaction between layers in order to keep it simple (Bellingham et al., 1990). The system was deployed for the application of adaptive feature mapping.

Another approach for coordinating behaviours is vector summation that averages the action between multiple behaviours. Following this principle, the DAMN system developed by Rosenblatt et al. (2002) used a voting-based coordination mechanism for arbitration implementing utility fusion with fuzzy logic.

The MOOS architecture developed by Newman (2002) was also able to guide UUVs by using a mission control system called Helm. Helm's mission plan was described by a set of prioritised primitive tasks. The most suitable action was selected using a set of prioritised mission goals. It used a state-machine for execution, a simplified version of a Petri net.

The *O*2*CA*<sup>2</sup> system (Carreras et al., 2007) also used a Petri net representation of the mission plan (Palomeras et al., 2009). The system maintains the low level control (dynamics) from the 4 Underwater Vehicles

Commands

Observations

Executive Platform Environment

Events

Mission

Fig. 2. Broken OODA-loop. Decision stage on the human operator based only on initial

plan managers that are procedural and static and might not even consider conditional executions (Hagen, 2001). At this level, the mission executive follows a sequence of basic command primitives and issues them to the functional control layer of the platform. Description about how these approaches maintain control of underwater vehicles can be found in Fossen (1994), Ridao et al. (1999) and Yuh (2000). In this situation, decisions taken by the operator are made using only orientation inputs related to some previous experience and *a-priori* knowledge. This has unpredictable consequences, in which unexpected situations can cause the mission to abort and might even cause the loss of the vehicle (Griffiths, 2005; von

More modern approaches are able to mitigate this lack of adaptability by introducing sets of behaviours that are activated based on observations (Arkin, 1998). Behaviours divide the control system into a parallel set of competence-levels. They can be seen as manually scripted plans generated *a-priori* to encapsulate the decision loop for an individual task. Under this approach, the key factor is to find the right method for coordinating these competing

The subsumption model, attributed to Brooks (1986), arbitrates behaviour priorities through the use of inhibition (one signal inhibits another) and suppression (one signal replaces other) networks. Most recent UUV control systems are a variant of the subsumption architecture. This model was first applied to the control of UUVs by Turner (1995) during the development of the ORCA system. This system used a set of schemas in a case-based framework. However,

Later, Oliveira et al. (1998) developed and deployed the CORAL system based on Petri nets. The system was in charge of activating the vehicle primitives needed to carry out the mission.

The scaling problem was addressed by Bennet & Leonard (2000) using a layered control architecture. Layered control is a variant of the subsumption model that restricts of interaction between layers in order to keep it simple (Bellingham et al., 1990). The system was deployed

Another approach for coordinating behaviours is vector summation that averages the action between multiple behaviours. Following this principle, the DAMN system developed by Rosenblatt et al. (2002) used a voting-based coordination mechanism for arbitration

The MOOS architecture developed by Newman (2002) was also able to guide UUVs by using a mission control system called Helm. Helm's mission plan was described by a set of prioritised primitive tasks. The most suitable action was selected using a set of prioritised mission goals.

The *O*2*CA*<sup>2</sup> system (Carreras et al., 2007) also used a Petri net representation of the mission plan (Palomeras et al., 2009). The system maintains the low level control (dynamics) from the

its scalability remains unclear as trials for its validation were not conducted.

It used a state-machine for execution, a simplified version of a Petri net.

These primitives were chained by preconditions and effects.

for the application of adaptive feature mapping.

implementing utility fusion with fuzzy logic.

Ofine Online

π0

Πt

Human Operator

Π<sup>0</sup>

pre-mission orientation.

Alt, 2010).

behaviours.

guidance control (kinematics) uncoupled (Caccia & Veruggio, 2000). Although it contained a declarative mission representation, missions were programmed manually.

A detailed survey of other behaviour-based approaches applied to mission control systems for UUVs can be found in Carreras et al. (2006).

More recently, Benjamin et al. (2009) has applied multiple objective decision theory to provide a suitable framework for formulating behaviour-based controllers that generate Pareto-optimal and satisfying behaviours. This approach was motivated by the infeasibility of optimal behaviour selection for real-world applications. This approach has been implemented and deployed as part of the IvP Helm extension to MOOS. This method seems to be a more suitable for behaviour selection, although more computationally expensive. Also, the approach is limited to the control of only the direction and velocity parameters of the host platform.

After reviewing this related work, two problems affecting the effectiveness of the decision loop become evident. Firstly, orientation and observation should be linked together because it is desirable to place the new observations in context. Secondly, decision and action should be iterating continuously. These two problems have not been addressed together by previous approaches. These are the two of the goals that we address in this chapter. In order to achieve them autonomously, two additional components are required: a status monitor and a mission plan adapter. The status monitor reports any changes detected in the mission environment during the execution of a mission. When the mission executive is unable to handle the changes detected by the status monitor, the mission planner is called to generate a new modified mission plan that agrees with the updated mission environment. Figure 3 shows the OODA-loop for autonomous decision making. Comparing it to the previous Figure 2, the addition of status monitor and mission planner removes the need for human decisions in the loop. Note that the original mission plan *π*<sup>0</sup> could also be autonomously generated as long as the high-level goals are provided by the human operator in Π0.

Fig. 3. Required OODA-loop for autonomous decision making in UUVs. Decision stage for adaptation takes place on-board based on initial orientation provided by the operator and observations provided by the status monitor.

Adaptive mission planning enables a true unmanned OODA-loop. This autonomous decision making loop copes with condition changes in the mission environment during the mission execution. As a consequence, it releases the operator from decision making tasks in stressful environments containing high levels of uncertainty and dynamism.

The potential benefits of adaptive mission planning capabilities for autonomous decision making in UUVs were promoted by Turner (2005), Bellingham et al. (2006) and Patrón & Petillot (2008). Possibly the most advanced autonomous decision making framework for UUVs has been developed at the Monterey Bay Aquarium Research Institute. This architecture, known as *T-REX*, has been deployed successfully inside the Dorado AUV (Rajan

Fig. 4. Knowledge Base representation system including the *TBox*, *ABox*, the description language and the reasoning components. Its interface is made of orientation rules and agent

Instances are the individual entities represented by a concept of the ontology (e.g. a *remus* is an instance of the concept *UUV*). Relations are used to describe the interactions between individuals (e.g. the relation *isComponentOf* might link the individual *SensorX* to the individual *PlatformY*). This finite set of instances and relations about individuals is called the *Assertion Box ABox*. The combination of *TBox* and *ABox* is what is known as a *Knowledge Base*. *TBox* aligns naturally to the orientation component of *SAV* while *ABox* aligns to the

<sup>205</sup> Embedded Knowledge and Autonomous Planning:

The Path Towards Permanent Presence of Underwater Networks

In the past, authors such as Matheus et al. (2003) and Kokar et al. (2009) have used ontologies for situation awareness in order to assist humans during information fusion and situation analysis processes. Our work extends these previous works by using ontologies for providing unmanned situation awareness in order to assist autonomous decision making algorithms in underwater vehicles. One of the main advantages of using a knowledge base over a classical data base schema to represent *SAV* is the extended querying that it provides, even across heterogeneous data systems. The meta-knowledge within an ontology can assist an intelligent agent (e.g., status monitor, mission planner, etc.) with processing a query. Part of this intelligent processing is due to the capability of reasoning. This enables the publication of machine understandable meta-data, opening opportunities for automated information

For instance, a status monitor agent using meta-data about sensor location could automatically infer the location of an event based on observations from nearby sensors (Miguelanez et al., 2008). Inferences over the ontology are made by reasoners. A reasoner enables the domain's logic to be specified with respect to the context model and applied to the corresponding knowledge i.e., the instances of the model (see Fig. 4). A detailed description of how a reasoner works is outside of the scope of this article. For the implementation of our approach, we use the open source reasoner called Pellet (Sirin et al.,

A library of knowledge bases comprise the overall knowledge framework used in our approach for building *SAV* (Miguelanez et al., 2010; Patrón, Miguelanez, Cartwright & Petillot, 2008). Reasoning capabilities allow concept consistency providing reassurance that *SAV* remains stable through the evolution of the mission. Also, inference of concepts and relationships allows new knowledge to be extracted or derived from the observed data. In order to provide with a design that supported maximum reusability (Gruber, 1995; van Heijst et al., 1996), we adopt a three-level segmentation structure that includes the (1) Foundation,

queries.

observation component.

processing and analysis.

**3.1 Semantic knowledge-based framework**

(2) Core and (3) Application ontology levels (see Fig. 5).

2007).

et al., 2009). This is now providing adaptive planning capabilities to oceanographers for maximising the science return of their UUV missions (McGann et al., 2007; Rajan et al., 2007). Using deliberative reactors for the concurrent integration of execution and planning (McGann, Py, Rajan & Henthorn, 2008), live sensor data can be analysed during mission to adapt the control of the platform in order to measure dynamic and episodic phenomenon, such as chemical plumes (McGann, Py, Rajan, Henthorn & McEwen, 2008; McGann et al., 2009). Alternative approaches to adaptive plume tracing can also be found in the works of Farrell et al. (2005) and Jakuba (2007). Their research goals of all these approaches have been motivated by scientific applications and do not consider the needs of the human operators or the maritime industry.

However, autonomy cannot be achieved without humans, as it is necessary for this autonomy to be ultimately accepted by an operator. Our research is geared towards improving human access to UUVs in order to solve the maritime industry's primary requirement of improving platform operability (Patrón et al., 2007). We propose a goal-based approach to solving adaptive mission planning. The advantage of this approach is that it provides high levels of mission abstraction. This makes the human interface simple, powerful and platform independent, which greatly eases the operator's task of designing and deploying missions (Patrón, 2009). Ultimately, operators will not need any specialist training for an UUV specific platform, and instead missions will be described purely in terms of their goals. Apart from ease of use, we have also demonstrated using a novel metric (Patrón & Birch, 2009) that adaptive mission planners can produce solutions which are close to what a human planner would produce (Patrón et al., 2009a). This means that our solutions can be trusted by an operator.

Another advantage of our research over other state-of-the-art UUV implementations, is that we are industry focussed. Our service-oriented approach provides goal-based mission planning with discoverable capabilities, which meets industry's need for platform independence (Patrón et al., 2009b). Finally, our plan repair approach optimises the resources required for adaptability and maximises consistency with the original plan, which improves human acceptance of autonomy. Resource optimisation and consistency are very important properties for real world implementations, as we demonstrate in our sea trials (Patrón, Miguelanez, Petillot & Lane, 2008).

Section 3 describes how do we link together orientation and observation. Section 4 presents an approach to the continuous iteration of decision and action.
