**Consideration of Human Operators in Designing Manufacturing Systems**

Namhun Kim1 and Richard A. Wysk2 *1Ulsan National Institute of Science and Technology 2North Carolina State University 1Korea 2USA* 

### **1. Introduction**

118 Manufacturing System

Ueda, K., Markus, a, Monostori, L., Kals, H. & Arai, T. (2001). Emergent Synthesis

Vichare, P., Nassehi, A., Kumar, S. & Newman, S.T. (2009). A Unified Manufacturing

Wiendahl, H.P. & Heger, C.L. (2004). Justifying Changeability. A Methodical Approach to

*Integrated Manufacturing*, Vol. 25, No. 6, pp. 999-1007, ISSN 0736-5845. Wiendahl, H. et al. (2007). Changeable Manufacturing - Classification, Design and

No. 2, pp.535-551, ISSN 0007-8506.

6, No. 1-2, pp. 33-40, ISSN 0793-6648.

ISSN 0007-8506.

Methodologies for Manufacturing. *CIRP Annals - Manufacturing Technology*, Vol. 50,

Resource Model for representing CNC machining systems. *Robotics and Computer-*

Operation. *CIRP Annals - Manufacturing Technology*, Vol. 56, No. 2, pp. 783-809,

Achieving Cost Effectiveness. *Journal for Manufacturing Science and Production*, Vol.

A manufacturing system normally includes various types of automated/computer controlled system resources such as material processors (e.g., CNC machines), material handlers (e.g., robots), and material transporters (e.g., AGVs) (Joshi et al., 1995). However, in most cases, to implement fully automated systems where the human is not involved is impractical (Brann et al., 1996), because of both economic and technical reasons. Furthermore, in human-involved automated manufacturing systems, a human can act as one of the most flexible and intelligent system resources in that he or she can perform a large variety of physical tasks ranging from simple material handling to complex tasks such as inspection, assembly, or packaging (Altuntas et al., 2004). From this argument, integrating a human into the system operation is a critical aspect in the design of practical manufacturing systems.

To represent the logical flows of systems' behavior, finite state automaton (FSA), formalism for discrete event-based systems, is widely used in modeling and building a control algorithm of automated manufacturing systems. While FSA-based models can be partially well suited to represent routine human activities, the vast majority of research on control models of human-involved manufacturing systems using FSA tends to consider a human as a system component that can perform tasks without considering dynamic and perceptual conditions of system constraints on human capabilities. (Shin et al., 2006b; Shin et al., 2006c). It is desirable, therefore, to include flexible and dynamic human decision making/tasks in the control of manufacturing systems with consideration of human capabilities and the corresponding system's physical conditions in human-machine co-existing environments.

Under ideal conditions, human operators should be allowed to access all physical components capable of being manipulated in the system (Altuntas et al., 2004). In this sense, a human operator can be considered a distinctive component of the system that is capable of affecting both the logical and physical states of the system. In reality, however, the human can be restricted in affecting the system components given what is afforded (e.g., offered) (Gibson, 1979) by the task environment (e.g., a part on a conveyor may be moving too fast for a human operator to grasp it). To incorporate human capabilities into the system representation, one must consider the control opportunities offered to humans by the system environment as well as the judgment demands placed on human operators.

In this chapter, a framework to develop formalisms for human-machine co-existing manufacturing systems is introduced and illustrative examples are provided in the last section.

### **2. Modelling of manufacturing systems**

The discrete event-based modeling formalism of FSA is introduced in section 2.1. Section 2.2 presents modeling of manufacturing system control using message-based part state graph and its extended version of including human tasks into manufacturing system operations.

#### **2.1 Finite state automata representation of DES**

The fundamental physical properties of nature are considered to be continuous in that they can be expressed using real values as time changes. As systems have become more complex, event-driven approaches have become commonplace for a variety of models. Several computer technologies employ discrete methods to control complex systems such as communication networks, air traffic control, automated manufacturing systems, and computer application programs (Cassandra and Lafortune, 1999; Zeigler, 1976). Discrete event-based system modeling is a common tool to represent physical behaviors of systems, including continuous systems that are broken into discrete models which are suitable for DES-based software applications.

One of popular formalisms used to represent the logical behavior of discrete systems is based on the theories of languages and automata. This approach is based on the notion that any discrete event system can be modeled with discrete states and an underlying event set associated with it. An automaton, formalism for discrete systems, is an atomic mathematical model for finite state automata (FSA). It consists of a finite number of states and transitions that enable the model to jump between states via predetermined rules. These jumps are incurred by transition functions. These transition functions determine which state to go to next, given the current state and a current input symbol. An FSA is an effective technique capable of representing a language according to well-defined rules, which means it is rulebased and the state of the system is tractable (Sipser, 2006).

A commonly used FSA in practice is a *Deterministic Finite Automaton* (DFA), which can be defined as a 5-tuple (Hopcroft, 2001);

$$M^{\rm DFA} = \lhd \mathbf{Z} \llcorner \mathbf{Q} \llcorner \mathbf{Q} \llcorner \mathbf{P} \succcorner \iota$$

where;

*Σ* is a set of input alphabets (a finite non-empty set of symbols),

*Q* is a finite and non-empty set of states,

*q0* is an initial state such that *q0* ∈ *Q*,

*δ* is a state transition function such that *δ*: *Q* × *Σ* →*Q*, and

*F* is a set of final states such that *F* ⊆ *Q.*

For example, a representation of the 5-tuple FSA for the person-climbing-stairs system is shown in Figure 1. A transition from a lower level to an upper level occurs immediately

representation, one must consider the control opportunities offered to humans by the

In this chapter, a framework to develop formalisms for human-machine co-existing manufacturing systems is introduced and illustrative examples are provided in the last section.

The discrete event-based modeling formalism of FSA is introduced in section 2.1. Section 2.2 presents modeling of manufacturing system control using message-based part state graph and its extended version of including human tasks into manufacturing system operations.

The fundamental physical properties of nature are considered to be continuous in that they can be expressed using real values as time changes. As systems have become more complex, event-driven approaches have become commonplace for a variety of models. Several computer technologies employ discrete methods to control complex systems such as communication networks, air traffic control, automated manufacturing systems, and computer application programs (Cassandra and Lafortune, 1999; Zeigler, 1976). Discrete event-based system modeling is a common tool to represent physical behaviors of systems, including continuous systems that are broken into discrete models which are suitable for

One of popular formalisms used to represent the logical behavior of discrete systems is based on the theories of languages and automata. This approach is based on the notion that any discrete event system can be modeled with discrete states and an underlying event set associated with it. An automaton, formalism for discrete systems, is an atomic mathematical model for finite state automata (FSA). It consists of a finite number of states and transitions that enable the model to jump between states via predetermined rules. These jumps are incurred by transition functions. These transition functions determine which state to go to next, given the current state and a current input symbol. An FSA is an effective technique capable of representing a language according to well-defined rules, which means it is rule-

A commonly used FSA in practice is a *Deterministic Finite Automaton* (DFA), which can be

For example, a representation of the 5-tuple FSA for the person-climbing-stairs system is shown in Figure 1. A transition from a lower level to an upper level occurs immediately

δ, *F*>,

Σ, *Q*, *q0*,

system environment as well as the judgment demands placed on human operators.

**2. Modelling of manufacturing systems** 

**2.1 Finite state automata representation of DES** 

based and the state of the system is tractable (Sipser, 2006).

*Σ* is a set of input alphabets (a finite non-empty set of symbols),

*δ* is a state transition function such that *δ*: *Q* × *Σ* →*Q*, and

DES-based software applications.

defined as a 5-tuple (Hopcroft, 2001);

where;

 *MDFA* = <

*Q* is a finite and non-empty set of states, *q0* is an initial state such that *q0* ∈ *Q*,

*F* is a set of final states such that *F* ⊆ *Q.*

following the action of 'climb stairs' which is an input symbol to a current state "lower level."

This model only represents the physical aspects of systems behavior without considering the resource availability, and a person's attention and capability to accomplish a specific action (e.g., climb stairs). To better model human participation, it is essential to take into account the conditions required for human actions which consist of affordances (walk-onability) and effectivities (capability to walk) in the systems as will be explained in Section 3.

Fig. 1. An FSA representation for the person-climbing-stairs system.

### **2.2 Control model of manufacturing systems**

### **2.2.1 Message-based part state graph (MPSG)**

In the 1990's, a formal model for control of discrete manufacturing systems was developed based on FSA, and called MPSG which is an acronym for Message-based Part State Graph (Smith et al., 2003). It is a modified deterministic finite automaton (DFA) similar to a Mealy machine. The MPSG model consists of sets of vertices (nodes) and edges (transitions) which correspond to the part states and the command messages, respectively. The trace of a part advancing trough an automated manufacturing system is described by its part flow diagram, which shows the sequence of part processing states in the system. As shown in Figure 2, the part state graph of a part is represented with a set of vertices and a set of edges. A vertex represents a part position in the part state graph and an edge corresponds to an operation associated with the part.

Fig. 2. An example of a part state graph for a MP class.

A MPSG describes the behavior of a controller from the parts' point of view, and each part within the domain of the controller is in a particular 'state' as described by the MPSG for that controller. The MPSG model provides no information about the system states; it determines which controller events are 'legal' with respect to that part and how to make a transition when one of these legal events occurs.

In the MPSG, all equipment level manufacturing resources are partitioned into material processors (MP; such as numerical control (NC) machines), material handler (MH; such as robots), material transporters (MT; such as automated guided vehicles (AGVs)), automated storage devices (AS), and buffer storage (BS), based on the types of their functionalities. We can create simplified physical connectivity graphs based on the MPSG controller. Figure 3 depicts a physical connectivity graph of a system, which consists of two MPs, MH, and BS, that represents physical interactions and accessibilities among the pieces of equipment. From a system's point of view, the connectivity graph is quite similar to the automaton that consists of states and transitions. However, for the individual resources, more detailed and sophisticated state transition mechanisms need to be considered, and the MPSG enables to describe the states of the entities (parts) in the system by means of the physical connectivity graph.

(a) Physical layout of the system. (b) Representation of connectivity graph.

Fig. 3. Connectivity graph with two MPs, MH, BS, and port.

The MPSG *M* is defined formally as an eight-tuple, *M=<QM, q0, F, ΣM, A, PM, δM, γ>*, where definitions of the components are as follows:


#### **2.2.2 Extended MPSG for human-involvement in manufacturing systems**

The MPSG is a formal representation of a shop floor controller and assumes that all the resources are run in an automated way without any human involvement. To incorporate human characteristics into an automated manufacturing systems, Shin et al. investigated human-involved manufacturing systems and developed a novel formal representation by adding the tuples associated with a human element to the MPSG model (Shin et al., 2006b).

In the MPSG, all equipment level manufacturing resources are partitioned into material processors (MP; such as numerical control (NC) machines), material handler (MH; such as robots), material transporters (MT; such as automated guided vehicles (AGVs)), automated storage devices (AS), and buffer storage (BS), based on the types of their functionalities. We can create simplified physical connectivity graphs based on the MPSG controller. Figure 3 depicts a physical connectivity graph of a system, which consists of two MPs, MH, and BS, that represents physical interactions and accessibilities among the pieces of equipment. From a system's point of view, the connectivity graph is quite similar to the automaton that consists of states and transitions. However, for the individual resources, more detailed and sophisticated state transition mechanisms need to be considered, and the MPSG enables to describe the states of the entities (parts) in the

(a) Physical layout of the system. (b) Representation of connectivity graph.

The MPSG *M* is defined formally as an eight-tuple, *M=<QM, q0, F, ΣM, A, PM, δM, γ>*, where

The MPSG is a formal representation of a shop floor controller and assumes that all the resources are run in an automated way without any human involvement. To incorporate human characteristics into an automated manufacturing systems, Shin et al. investigated human-involved manufacturing systems and developed a novel formal representation by adding the tuples associated with a human element to the MPSG model (Shin et al., 2006b).

system by means of the physical connectivity graph.

Fig. 3. Connectivity graph with two MPs, MH, BS, and port.

definitions of the components are as follows:

*F* ⊆ *QM* : Set of final or accepting states, *ΣM* : Finite set of controller events, *A* : Finite set of controller actions, *PM* : Physical preconditions, *δM : QM*×*Σ<sup>M</sup> QM*: State transition function, and

*γ : QM*×*Σ<sup>M</sup> A* : Controller action transition function.

**2.2.2 Extended MPSG for human-involvement in manufacturing systems** 

*QM* : Finite set of states, *q0* ∈ *QM* : Initial or start state, The extended MPSG model enables a human operator to cooperate with the automated pieces of equipment.

In Figure 4, solid arcs represent connections between two pieces of equipment made by automated MH equipment, whereas dotted arcs are newly created ones made by a human operator who plays as a material handler. In general, when a human operator who performs material handling tasks in a system that consists of *n* pieces of equipment is considered, 2×*n* of arcs for human transitions are created (Altuntas et al.*,* 2004). It should be noted that the complexity of the connectivity graph increases in a linear manner.

(a) Physical layout of the system. (b) Representation of connectivity graph.

Fig. 4. Change of connectivity graphs with consideration of a human MH.

In order to express the newly created transitions by incorporating a human operator, the representation of part states is extended by incorporating information about a part location within a system and a part handling subject such that it becomes *Q* = *Q<sup>M</sup>* × *L* × *I(p)*, where *L* represents a set of physical locations in the system and *I(p)* is an interaction status with a human. In this way, an extended MPSG with a human operator, denoted by *ME,* is constructed. It is defined formally as also the eight-tuple, *ME =<Q, qE0, FE, ΣE, AE, PE, δE, γE>*, where the definitions of the components are as follows:

*Q = QM*×*L*×*I(p)* : Finite set of states, where the set of state *QM* is the state of the original MPSG controller,

*qE0* ∈ *Q* : Initial or start state,

*FE* ⊆ *Q* : Set of final or accepting states,

*ΣE = ΣM*∪*ΣH* : Finite set of controller events, where *ΣM* is a set of messages for a machine operation and *ΣH* is a set of messages associated with human actions,

*AE= A*∪*{actions caused by human activities}* : Finite set of controller actions and human actions,

*δE : Q*×*ΣEQ* : State transition function,

*γE : Q*×*ΣEAE* : Controller action transition function,

*L* : Set of all physical locations in the system, and

*I(p)* : Indicator function of interaction status with a human. If a human is dealing with a part *p*, *I(p)=1*,. Otherwise, *I(p) =0*.

In the concept of the human-involved semi-automated system, its control depends on the complexity of a system, since a controller should recognize current status of the system and provide a proper set of commands for possible tasks based on the logical and physical preconditions. Hence, when a human material handler (human MH) performs tasks during system operation, assessment of the part flow complexity of the system needs to be conducted in developing an effective and efficient control mechanism for the system. The part flow complexity represents the possible number of tasks with a part and the possible outcomes of the tasks in terms of part states (Shin et al., 2006a).

Using this point of view, the major difference of the control schemes between the automated system and the human-involved semi-automated system is whether a human act as a passive resource of the system or a supervisory controller. The human MH can play a role as a self-regulating component which does not subordinate to the computer controller whereas other automated components perform operations in response to a given command for the controller. As such, the human MH can be considered to act as a supervisory controller, and he or she shares the system information via interfaces and sensors as shown in the Figure 5. This perspective will be further developed to expand the human's participation in complex systems.

### **3. Modelling and control of human-machine cooperative manufacturing systems**

In section 3.1, a representation of human-involvement considering prospective human action opportunities (affordance) is introduced. The modeling basis and formal control model for affordance-based human-machine cooperative manufacturing system are presented in section 3.2 and 3.3, respectively. The example of affordance-based MPSG system control with a simple and typical manufacturing cell is illustrated in section 3.4.

### **3.1 Human-involvement in system representation**

In dynamic situations, the interactions between humans and environs play a key role in achieving an ecosystem's goal. Identifying opportunities for interactions between them is important to the modeling and operation of human-involved systems in an effective way. In this section, we introduce a formal modeling methodology that combines human actions into the system control scheme in formal mathematical FSA.

The concept of affordances implies that human-involved systems are composed of two or more related objects including at least one human and one environmental component, (an affordance complementary property consisting of the dual relationship between animals (humans) and their environs). The terms of affordance and effectivity are treated as an environmental reference and the animal's capability to take actions in the environment. In the sense of a formal representation of affordances, the environmental and animal components are combined together so that they incur a different property to be activated (Turvey, 1992).

Turvey presents a formal definition of affordances mathematically using a juxtaposition function as follows;

Let *Wpq=j(Xp,Zq)* be a function that is composed of two different objects *X* and *Z*, and further *p* and *q* be properties of *X* and *Z*, respectively. Then, *p* refers to an affordance of *X* and *q* is the effectivity of *Z*, if and only if there exists a third property *r* such that:

i. *Wpq=j(Xp,Zq)* possesses *r*,

124 Manufacturing System

In the concept of the human-involved semi-automated system, its control depends on the complexity of a system, since a controller should recognize current status of the system and provide a proper set of commands for possible tasks based on the logical and physical preconditions. Hence, when a human material handler (human MH) performs tasks during system operation, assessment of the part flow complexity of the system needs to be conducted in developing an effective and efficient control mechanism for the system. The part flow complexity represents the possible number of tasks with a part and the possible

Using this point of view, the major difference of the control schemes between the automated system and the human-involved semi-automated system is whether a human act as a passive resource of the system or a supervisory controller. The human MH can play a role as a self-regulating component which does not subordinate to the computer controller whereas other automated components perform operations in response to a given command for the controller. As such, the human MH can be considered to act as a supervisory controller, and he or she shares the system information via interfaces and sensors as shown in the Figure 5. This perspective will be further developed to expand the

 (a) MPSG controller. (b) Human-involved extended MPSG. Fig. 5. Control scheme of the MPSG and extended MPSG controllers (Shin et al.*,* 2006a).

**3. Modelling and control of human-machine cooperative manufacturing** 

In section 3.1, a representation of human-involvement considering prospective human action opportunities (affordance) is introduced. The modeling basis and formal control model for affordance-based human-machine cooperative manufacturing system are presented in section 3.2 and 3.3, respectively. The example of affordance-based MPSG system control with a simple and typical manufacturing cell is illustrated in section 3.4.

In dynamic situations, the interactions between humans and environs play a key role in achieving an ecosystem's goal. Identifying opportunities for interactions between them is important to the modeling and operation of human-involved systems in an effective way. In

outcomes of the tasks in terms of part states (Shin et al., 2006a).

human's participation in complex systems.

**3.1 Human-involvement in system representation** 

**systems** 


For example, a person (*Z*) can walk (*q*), stairs (*X*) that can support something (*p*), and they together yield a climbing property (*r*) as shown in Figure 6**.** This formal definition corresponds to a mathematical formalism of an FSA in that it describes properties as discrete states, and the juxtaposition function can be mapped to the state transition function in the FSA. The existence of a formal definition of an affordance provides a foundation that the concept of an affordance can be combined with software engineering and systems theory.

Fig. 6. An example of a 'person-climbing-stairs' system.

If we regard the states of the environmental system as discrete ones and consider the transitions among the states which are triggered by possible actions of animals or other system resources, an ecosystem of an environment and humans can be represented by an FSA (Kim et al., 2010). The theory of automata corresponds to the ecological sense of affordances for at least the following two reasons: 1) an environmental system can be defined as a set of nodes and arcs which describe discrete states of the system and the transitions between states, respectively, and 2) a set of transitions between states represents a set of potential properties (affordances) of the environmental system which can be triggered by certain human activities and lead to the next states. Therefore, affordanceeffectivity combinations can be considered conditions for identifying possible human actions using FSA representations.

There is a set of physically connected transitions from one state to another, which corresponds to a set of dispositional properties of affordances in the system. The set of feasible transitions is triggered if and only if the input symbol is taken as a parameter of a transition function in the environmental system. This input symbol is considered an effectivity. Next, the circumstances need to be specified in order for a human transition to occur in terms of the general representation of the FSA, *MDFA* = <Σ, *Q*, *q0*, δ, *F*>. The conditions that allow humans to make transitions within a system can be represented by a four-tuple, <*Xp, Zq, J, Wpq*>, which comes directly from Turvey's definition of affordance. By merging these two sets of tuples, an extended automaton for incorporating affordances of a system and effectivities of humans within the system can be constructed. The new representation for the formal model of affordance and effectivity in FSA is *MDFA'*=<Σ, *Q*, *q0*, δ, *F, Xp, Zq, J, Wpq*>,where;

*J* is a Juxtaposition function such that *J*: *Xp* × *Zq* → *Wpq ,* 

*Xp* is a set of affordances in the system,

*Zq* is a set of effectivities of human in the system,

*Wpq* is a set of possible human actions in the system, and

all other definitions of tuples are the same as those of *MDFA*.

The graphical representation of the affordance-based FSA, *MDFA'*, for the person-climbingstairs system is shown in Figure 7. Transition from a lower level to an upper level occurs, *if and only if a human is able to 'walk (Xp)' and the stairs are 'walk-on-able (Zq)' for human, which means 'Climb Stairs (system input of human action)* ∈ *Wpq*.'

Fig. 7. Affordance-based FSA for the person-climbing-stairs system(Kim et al., 2010).

From an ecosystem's perspective, if the set of all transitions among the system states can be considered Σ*.* A state transition occurs only when the transition for some input alphabet is included in the set of transitions in the system, *a*∈Σ, where *a* represents an input alphabet. From the human's point of view, he or she has a set of effectivities (capabilities or available actions) regardless of the transitions included in Σ. Thus, transitions occur if and only if the set of possible human actions, *Wpq*, are executed (the results of juxtaposition between specific affordance and effectivity). Component, *h*∈*Wpq* ⊆ Σ, represents the possible set of actions for a human to actualize on the environmental system, causing state transitions. In this sense, the dispositional properties that come from joining the properties of affordance and effectivity are

triggered by certain human activities and lead to the next states. Therefore, affordanceeffectivity combinations can be considered conditions for identifying possible human

There is a set of physically connected transitions from one state to another, which corresponds to a set of dispositional properties of affordances in the system. The set of feasible transitions is triggered if and only if the input symbol is taken as a parameter of a transition function in the environmental system. This input symbol is considered an effectivity. Next, the circumstances need to be specified in order for a human transition to

conditions that allow humans to make transitions within a system can be represented by a four-tuple, <*Xp, Zq, J, Wpq*>, which comes directly from Turvey's definition of affordance. By merging these two sets of tuples, an extended automaton for incorporating affordances of a system and effectivities of humans within the system can be constructed. The new

The graphical representation of the affordance-based FSA, *MDFA'*, for the person-climbingstairs system is shown in Figure 7. Transition from a lower level to an upper level occurs, *if and only if a human is able to 'walk (Xp)' and the stairs are 'walk-on-able (Zq)' for human, which* 

Fig. 7. Affordance-based FSA for the person-climbing-stairs system(Kim et al., 2010).

From an ecosystem's perspective, if the set of all transitions among the system states can be

From the human's point of view, he or she has a set of effectivities (capabilities or available

set of possible human actions, *Wpq*, are executed (the results of juxtaposition between specific

human to actualize on the environmental system, causing state transitions. In this sense, the dispositional properties that come from joining the properties of affordance and effectivity are

*.* A state transition occurs only when the transition for some input alphabet is

Σ

Σ

Σ

representation for the formal model of affordance and effectivity in FSA is *MDFA'*=<

Σ

, where *a* represents an input alphabet.

. Thus, transitions occur if and only if the

, represents the possible set of actions for a

, *Q*, *q0*,

δ

Σ, *Q*, *q0*,

, *F*>. The

occur in terms of the general representation of the FSA, *MDFA* = <

*J* is a Juxtaposition function such that *J*: *Xp* × *Zq* → *Wpq ,* 

*means 'Climb Stairs (system input of human action)* ∈ *Wpq*.'

included in the set of transitions in the system, *a*∈

actions) regardless of the transitions included in

affordance and effectivity). Component, *h*∈*Wpq* ⊆

*Zq* is a set of effectivities of human in the system, *Wpq* is a set of possible human actions in the system, and all other definitions of tuples are the same as those of *MDFA*.

actions using FSA representations.

, *F, Xp, Zq, J, Wpq*>,where;

*Xp* is a set of affordances in the system,

δ

considered

Σ

considered possible human actions on the human-environmental system. In many unstructured instances, the set of actions can be an infinite set, but for well-structured environs the set of actions can be a very small set. The relationship among affordances of a system, effectivities, and actions of human can be depicted as shown in Figure 8.

Fig. 8. System affordances, human effectivities, and actions in the ecological point of view (Kim et al., 2010).

The FSA-based modeling formalism for manufacturing systems control can take human activities into account, where source and sink state nodes are defined within the system state behaviors. However, the existing control model of human-involved manufacturing systems lacks prospective control perspectives. It only considers human operators as flexible system components acting like robots, rather than animals which have nondeterministic natures of recognitions and physical limitations. To develop the formal model of humanmachine cooperative systems, the ecological sense of system affordances and human effectivities should be included in the model for the seamless control of the systems.

Special care needs to be taken for human operators since their actions are those of a nondeterministic autonomous agent that perceives, measures, and makes a judgment in the system in consideration of other resources and environmental aspects. For this reason, affordances for a human operator in the system need to be considered carefully for humanmachine cooperative systems. It can then contribute to assess the human effects on the system in a more effective way.

### **3.2 Representation of affordances in human-involved manufacturing systems**

For a formal control model of human-involved manufacturing systems, it is necessary to incorporate affordances within a system that accounts for possible human actions with regard to at least the material handling processes with consideration of physical limitations for the actions, such as size, weight, and temperature. This corresponds to distinguishing possible human actions from human capable actions (effectivities). We remark that the set of possible human actions are a part of the collection of human *potentially* capable actions considering that human may or may not take actions due to his or her cognitive recognition of actions or physical limitations imposed by an environment. From the viewpoint of the manufacturing system with a human material handler, the affordance can be described as;

Define *Wpq* as a set of *possible human actions in manufacturing system*. Let *Xp* be a physical state of a part or a piece of equipment in a system where *p* is a human accessibility (affordance) to it, and *Zq* be a human material handler where *q* is a human action if and only if there is the third property *r* such that:


In order to incorporate system affordances into the manufacturing system controller, a formal representation of occurrences of the third properties, called dispositions, needs to be established. For some typical possible human actions for material handling (e.g., *access*, *pick*, *move*, and *put*), the corresponding circumstance can be specified as follows (Kim et al., 2010);

The specific classes of human activities to be addressed include the following:


As described above, there are some obvious state transitions for a human material handler in the system. Based on the examples, we decompose human material handling tasks into four types of actions (*access*, *pick*, *move*, and *put*), and also define system affordances and human capable actions corresponding to them as follows;

#### **<Affordances in the manufacturing system with human material handlers>**

The specific classes of human activities to be addressed include the following:


#### **<Human MH's capable actions (effectivities) in manufacturing systems>**


i. *Wpq=j(Xp,Zq)* possesses *r* : human material handing action (location change),

iii. Neither *X* nor *Z* possesses *r*, where *r* is a joining or juxtaposition function.

The specific classes of human activities to be addressed include the following:

In order to incorporate system affordances into the manufacturing system controller, a formal representation of occurrences of the third properties, called dispositions, needs to be established. For some typical possible human actions for material handling (e.g., *access*, *pick*, *move*, and *put*), the corresponding circumstance can be specified as follows (Kim et al., 2010);

1. For a human material handler to be able to **access** a machine (resource), the machine should be stopped before the human starts to work and other MHs (robots) should not run on it. Also, the human needs to perceive that he or she can work on the machine. 2. For a human material handler to be able to **pick** a part, at least one degree of freedom (DOF) of the part needs to be released and the human can separate the part from the

3. For a human material handler to be able to **move** a part, he or she should be holding the part, and the central position of the part can be changed in the global Cartesian

4. For a human material handler to be able to **put** a part on a machine, the machine should be stopped and the number of parts on the machine should not exceed the capacity of

As described above, there are some obvious state transitions for a human material handler in the system. Based on the examples, we decompose human material handling tasks into four types of actions (*access*, *pick*, *move*, and *put*), and also define system affordances and

1. A machine is **accessible**; the machine is stopped and waits to process a part, and no other MH is working on it. The machine volume should be within the human's access

2. A part is **pickable**; the chuck or fixture holding the part is open, and at least one DOF of the part is available. The part should weigh less than maximum lifting force, and

4. A part is **putable**; the machine stops working, and it can support the part upright

should be less than maximum grapping width for a human material handler. 3. A part is **movable**; the part is held by a human, and the location of the part can be changed by human actions. There are no *substantial* obstacles from a starting point to an

**<Human MH's capable actions (effectivities) in manufacturing systems>** 

2. A human material handler can **pick** a part from a piece of equipment.

1. A human material handler can **access** a piece of equipment.

the machine. Also, the machine needs to support the part (fixture without slip).

**<Affordances in the manufacturing system with human material handlers>**  The specific classes of human activities to be addressed include the following:

only if there is the third property *r* such that:

ii. *Wpq=j(Xp,Zq)* possesses neither *p* nor *q*, and

coordinate system as a result of the *move* action.

human capable actions corresponding to them as follows;

position.

ranges.

without slip.

ending position of the human.

(affordance) to it, and *Zq* be a human material handler where *q* is a human action if and


The third property in Turvey's affordance formalism is mapped on a subset of possible human actions. By doing this, the juxtaposition function can be formulated based on its definition. In the definition of the set of possible human actions, denoted by *Wpq=j(Xp,Zq)*, *j* is the joining or juxtaposition function. If *Xp* and *Zq* have multiple dispositions, the juxtaposition function *j* needs to filter *p* and *q* from the dispositions possessed by *Xp* and *Zq* to realize the possible actions of *Wpq*. To construct a juxtaposition function to address this, *Xp* and *Zq* are expressed as row matrices that consist of '0' and '1', which represent a certain property *exists* ('1') or *not* ('0') in the system. Thus, in a manufacturing system with human, the sets of *P* and *Q* can be expressed as following equation (1) and (2), respectively;

$$\mathcal{P} = \{ \text{Accessible}, \text{Pickable}, \text{Mourable}, \text{Pultable} \}; \text{Propereties of the system} \tag{1}$$

$$Q = \{ \text{Can Acess, Can Pick, Can Move, Can Put} \} \text{: Proprities of a human} \tag{2}$$

By multiplying each component in the matrices, the juxtaposition function of this problem is defined as in equation (3) to obtain the third properties and possible state transitions;

*j* : *Xp × Zq* → *Wpq* and *π* : *P × Q × C* → *PA,* where *P* is a set of affordance status for a part state, *Q* is a set of action capability (effectivity) status to the human operator, *PA* is a set of possible human actions in the system, and *C* is a set of physical action conditions (preconditions for human actions).

Suppose, *P* = {*pi* : *i*=1,2,3,4}, *Q* = {*qj* : *j*=1,2,3,4} where *pi* and *qj* are binary numbers, then

$$PA = \begin{cases} \phi & \text{if } \ p\_i q\_i = 0. \\ \langle (p\_2 q\_2 \times \text{'} \text{pick'}), (p\_3 q\_3 \times \text{'} \text{move'}), (p\_4 q\_4 \times \text{'} \text{put'}) \rangle - \langle 0 \rangle & \text{if } \ p\_i q\_i = 1 \text{ &C is true.} \end{cases} \tag{3}$$

Note that the empty set refers to a situation that a human operator cannot access resource.

#### **3.3 Formalism for human-machine cooperative systems: Affordance-based MPSG**

As mentioned in the previous section, some human actions become available depending on the environmental affordances, and transitions made by human actions can be realized by satisfying both system affordances and corresponding human effectivities (capable actions). Affordances should have ontological assumptions related to space and time as in Gibson's ecological definition (Gibson, 1979). In the sense of system controller such as the MPSG, it is one of the key factors to build formal representation of the affordance concepts that imposing quantifiable metrics on affordances.

From the MPSG point of view, the supervisory controller, called a Big-E, has a module to generate possible transitions based on the logical validation of preconditions as shown in the Figure 9. The existing MPSG controller generates process plans based on the fully automated systems that are assumed to properly operate as planned beforehand. In this sense, as long as the system is working without *critical* failures, the human action is not necessary and a human is allowed to intervene between machine operations whenever he or she decides to do so. However, when an unanticipated incident occurs (e.g., machine down, oversized part), which is usually beyond the controller's resolution capability, human involvement is required. In this case, the Big-E controller notifies a human of the case so that a human operator can step in the process for preceding the system to the next available proper transition (Kim et al., 2010).

From the viewpoint of a human operator, he or she could make a transition in the system to move a part forward toward completion (one of the feasible ways to proceed when the system requires some human action). The set of feasible transitions are mapped into the Big-E controller, which can generate possible alternative action commands based on the logical validation modules. It is worth note that this exactly corresponds to the set of system affordances for this case. It should also be noted that not all feasible transitions are available for the human operator because the system affordances for the human operator have ontological assumptions of physical and time domains, as mentioned above.

Fig. 9. Control flow of human-involved automated system with consideration of affordances (Kim et al., 2010).

In order to realize a human cooperative system in the ecological sense, generation modules for two distinctive logical sets and the Boolean operator for juxtaposing these two logical sets need to be constructed for a human operator to cooperate with the controller with consideration of affordances as shown in Figure 9.

Considering the formal representation of affordances, the extended MPSG for humaninvolved system control can be improved in such a way that it can consider more realistic transitions by human possible actions. In this chapter, the affordance-based MPSG, denoted by *MA*, is defined as a 12-tuple, which comprises eight-tuples from the initial extended MPSG model, *ME* , and four-tuple from the affordance representation. It is defined formally as *MA =<Q, qE0, FE, ΣE, AE, P*α*, δE, γE, X, Z, J, W >*, where the definitions of the components are as follows:

*J* is a juxtaposition function such that *J*: *X* × *Z* → *W*, *X* is a set of affordances, *Z* is a set of effectivities (human capable actions), *W* is a set of possible human actions,

where;

130 Manufacturing System

case, the Big-E controller notifies a human of the case so that a human operator can step in the process for preceding the system to the next available proper transition (Kim et al., 2010).

From the viewpoint of a human operator, he or she could make a transition in the system to move a part forward toward completion (one of the feasible ways to proceed when the system requires some human action). The set of feasible transitions are mapped into the Big-E controller, which can generate possible alternative action commands based on the logical validation modules. It is worth note that this exactly corresponds to the set of system affordances for this case. It should also be noted that not all feasible transitions are available for the human operator because the system affordances for the human operator have

Fig. 9. Control flow of human-involved automated system with consideration of affordances

In order to realize a human cooperative system in the ecological sense, generation modules for two distinctive logical sets and the Boolean operator for juxtaposing these two logical sets need to be constructed for a human operator to cooperate with the controller with

Considering the formal representation of affordances, the extended MPSG for humaninvolved system control can be improved in such a way that it can consider more realistic transitions by human possible actions. In this chapter, the affordance-based MPSG, denoted by *MA*, is defined as a 12-tuple, which comprises eight-tuples from the initial extended MPSG model, *ME* , and four-tuple from the affordance representation. It is defined formally

*, δE, γE, X, Z, J, W >*, where the definitions of the components

(Kim et al., 2010).

as *MA =<Q, qE0, FE, ΣE, AE, P*

are as follows:

consideration of affordances as shown in Figure 9.

α

ontological assumptions of physical and time domains, as mentioned above.

$$\mathbf{J}(\mathbf{x}(\mathbf{p}, \mathbf{l}), \mathbf{z}(\mathbf{p}, \mathbf{l})) = \mathbf{W}$$

$$= \begin{cases} \begin{array}{c} \bigotimes & \text{if } \ \mathbf{x}\_{1}\mathbf{z}\_{1} = \mathbf{0} \\\\ j(X, Z) = \begin{cases} \mathbf{x}\_{2}\mathbf{z}\_{2} \times pick\\_p\\_from\\_l\_{l\_1}, \\ \mathbf{x}\_{3}\mathbf{z}\_{3} \times move\\_p\\_from\\_l\_{l\_1} \to \underline{l}\_2, \\ \mathbf{x}\_{4}\mathbf{z}\_{4} \times punt\\_p\\_on\\_l\_{l\_2} \end{cases} \end{cases} \end{cases}$$

where;

**l** ⊆**L**, x(p, **l**) ∈ X, z(p, **l**) ∈Z,

*x*(*p, l*)*=x*(*p,* {*l1,l2*})*=* (a location set {*l1,l2*}is accessible, part *'p'* is 'pickable' at *l1* , part *'p'* is movable from *l1* to *l2* , part *'p'* is 'putable' on *l2*), and

*z*(*p, l*)*=z*(*p,* {*l1,l2*})= (access to a location set {*l1,l2*}, pick the part *'p'* at *l1* ,move the part *'p'* from *l1* to *l2* ,put the part *'p'* on *l2*)'

*δ<sup>E</sup>* is a state transition function such that *δE* : *Q* × *Σ<sup>E</sup>* → *Q.*,

where;

$$\begin{aligned} \delta\_{\mathbb{E}}\left( (\mathbf{v}, l, I(p)), a, W \right) &= (\delta\_{M}(\mathbf{v}, a), l, 0) \end{aligned} \text{  $\begin{aligned} \delta\_{M}(\mathbf{v}, a), l, 0$  \text{  $} & \text{ $ \begin{aligned} \mathbf{j} \text{  $a \in \Sigma\_{M}$ } $} \text{ (by a MH)} \\ \delta\_{\mathbb{E}}\left( (\mathbf{v}, l, I(p)), a, W \right) &= \delta\_{H}\left( (\mathbf{v}, l, I(p)), a, W \right) \end{aligned} \text{ $ \begin{aligned} \mathbf{j} \text{  $a \in W$ } & \text{  $a \in W$   $} \subset \Sigma\_{H}$  \text{ (by a human)} \\ \mathbf{j} \text{  $a \in W$   $} & \text{ $ a \notin W $ $ } \end{aligned} \text{ (by a human)} $$

where *δ<sup>M</sup>* is a state transition function by automated MHs (robots) and *δ<sup>H</sup>* is a state transition function by a human material handler, and all other definitions of tuples are the same as those of *ME*.

Based on the above definition, the juxtaposition function can generate a set of possible human actions under a particular circumstance when system affordances are defined as environmental situations, time limitation, physical layout of a system, and part properties, e.g., size, volume, and speed. The human transition set of the affordance-based MPSG is a subset of that of the extended MPSG as shown in Figure 10.

Thus, the complexity of the MPSG controller of human cooperative systems can be reduced when the concept of affordances are taken into account. In the extended MPSG controller, *ME*, physical preconditions, *P*α, are evaluated so that some impossible transitions can be prevented. However, the physical preconditions may account for only a small part of system affordances that can be measured by pre-installed sensors, while most of possible human transitions are determined by human cognitions. It is noteworthy that system affordances for humans have much greater impact on the operations and control of the human-machine cooperative systems.

Fig. 10. Transition action sets in the Affordance-based MPSG controller (Kim et al., 2010).

### **3.4 Illustrative example: Affordance-based MPSG model**

This section presents an application example to illustrate the proposed manufacturing control model with affordances. As shown in Figure 11, two types of graphs are constructed to represent the system's physical configuration and the logical control logic. The first graph shows the relationship among the resources in a system and possible path for parts. Based on this connectivity graph in Figure 11(a), the affordance-based FSA representation in Figure 11(b) can be created to develop a control scheme for the system. This is then used to generate an affordance-based MPSG controller that incorporates operations of each piece of equipment and possible human actions (Kim et al., 2010).

Fig. 11. FSA representation; MP2 has no affordance of put-ability for human operators (Kim et al., 2010).

Specifically, Figure 11 depicts a case in which a human operator can move a part from 'MP1' to anywhere when the part is not 'putable' on 'MP2', i.e., the MP2 is located so far from the operator that he or she cannot see if the MP2 is empty. The affordance and effectivity matrices between 'node 1' and 'node 2' are expressed with the proposed model as follows,

$$\text{ax}(part, \{\text{MP1}, \text{MP2}\}) = (\mathbf{x}\_1, \mathbf{x}\_2, \mathbf{x}\_3, \mathbf{x}\_4) = (1, 1, 0, 1) \quad \text{and} \quad \mathbf{x}\_i$$

<sup>1234</sup> *z part* ( ,{MP1,MP2}) ( , , , ) (1,1,1,1) *zz zz* .

So, the juxtaposition function can be,

132 Manufacturing System

Fig. 10. Transition action sets in the Affordance-based MPSG controller (Kim et al., 2010).

This section presents an application example to illustrate the proposed manufacturing control model with affordances. As shown in Figure 11, two types of graphs are constructed to represent the system's physical configuration and the logical control logic. The first graph shows the relationship among the resources in a system and possible path for parts. Based on this connectivity graph in Figure 11(a), the affordance-based FSA representation in Figure 11(b) can be created to develop a control scheme for the system. This is then used to generate an affordance-based MPSG controller that incorporates operations of each piece of

 (a) Connectivity graph (b) Affordance-based MPSG representation Fig. 11. FSA representation; MP2 has no affordance of put-ability for human operators (Kim

Specifically, Figure 11 depicts a case in which a human operator can move a part from 'MP1' to anywhere when the part is not 'putable' on 'MP2', i.e., the MP2 is located so far from the operator that he or she cannot see if the MP2 is empty. The affordance and effectivity matrices between 'node 1' and 'node 2' are expressed with the proposed

<sup>1234</sup> *x part* ( ,{MP1,MP2}) ( , , , ) (1,1,0,1) *xxxx* and

**3.4 Illustrative example: Affordance-based MPSG model** 

equipment and possible human actions (Kim et al., 2010).

et al., 2010).

model as follows,

$$W = \begin{bmatrix} 1 \times \text{pick\\_part\\_from\\_M1} \\ 1 \times \text{move\\_part\\_from\\_M1\\_to\\_MP2} \\ 0 \times \text{put\\_part\\_on\\_MP2} \end{bmatrix} - \text{(0)}$$
 
$$\therefore \text{ } W = \text{(pick\\_part\\_from\\_M1, move\\_part\\_from\\_M1\\_to\\_MP2)}$$

If the human material handler wants to make a transition between 'MP1' and 'MP2', he or she needs to take three actions (pick, move, and put) between the nodes. However, the action, 'put', is not available in the system. It means that by taking the affordances in the system, the complexity of the graph in terms of the number of possible human actions in the FSA representation can be reduced.

The eligible affordances and effectivities of the example can be expressed as follows;

The affordance chart at time *t*:

$$\begin{aligned} \mathfrak{x}(part, \{\text{MPl}, \text{MP2}\}) &= (1, 1, 0, 1) \\\\ \mathfrak{x}(part, \{\text{MPl}, \text{BS}\}) &= (1, 1, 1, 1) \\\\ \mathfrak{x}(part, \{\text{MPl}, \text{PORT}\}) &= (1, 1, 1, 1) \end{aligned}$$

The effectivity chart at this point:

*z part* ( ,{MP1,MP2}) (1,1,1,1) *z part* ( ,{MP1,BS}) (1,1,1,1) *z part* ( ,{MP1,PORT}) (1,1,1,1) 

From the above affordances and effectivities relationships, we obtain;

{ \_ \_ \_MP1, \_ \_ \_MP1\_ \_MP2, \_ \_ \_ MP1\_ \_BS, \_ \_ \_MP1\_ \_PORT, \_ \_ \_BS, \_ \_ \_PORT} *W Pick part from Move part from to Move part from to Move part from to Put part on Put part on* 

If the controller is to allow a part transition between MP1 and MP2 by a human material handler, *W* should contain a complete set of actions which is composed of *pick*, *move*, and *put* between MP1 and MP2. However, *W* does not have '*put*' on MP2 actions in itself in this example. Thus, the human operator cannot make the part transit between MP1 to MP2 as a material handler.

#### **4. Function allocation between human and machine**

Dynamic task allocation control scheme for realization of human-machine cooperative systems is introduced in section 4.1. Classification of errors and their recoveries in humanmachine cooperative systems using affordance-based MPSG are presented in section 4.2.

### **4.1 Work allocations in human-machine cooperative systems**

Sheridan (2000) discusses a list to assert "what men are better at" and "what machines are better at" (MABA-MABA) as follows;

#### **<Humans are usually conceived to be better at>:**


#### **<Machines are better at>:**


The gaps between 'what machines are better at and what humans are better at' are getting narrower as machines are replacing human more and more with the development of artificial intelligence technologies. However, the complete replacing humans with the automated machines are almost impossible and impractical partly because of both economic and technical reasons (Brann et al., 1996).

In this sense, the function allocations between machines and humans in the human-involved automated system are one of the vital factors to control the system in effective and flexible ways. As pointed out in the previous section 3.1, human actions are available depending on the environmental affordances, and transitions by human can be realized by satisfying both system affordances and corresponding human effectivities Thus, from the system point of view, the controller needs to differentiate the set of actions that humans are better at from actions that machine are better at with consideration of availability of human actions identified by the model.

Suppose that the material handling time (e.g., time for picking up, moving, and putting a part) and material lifting capability (e.g., part weight, volume, size, and temperature) can be critical factors to allocate work between human material handlers and robots in a manufacturing cell. If the controller is able to evaluate the availability of a resource (either human or machine) which can reduce a processing time for a material handling job at a certain point of time and space, the whole system works faster and more intelligent to increase its productivity. For example, if we consider a simple human-machine cooperative manufacturing cell shown in Figure 4 with following characteristics;

*Time for a robot to move a part from a resource to a resource = 10 ± 0.5 sec., Time for a human to move a part between adjacent resources = 5 ± 2 sec., Time for a human to move a part between facing resources = 10 ± 5 sec., and Human capable part size and weight ≤ 3ft × 3ft × 3ft and 20 LB.* 

The controller is able to evaluate expected average processing time for each task and allocate the task between a human operator and a robot based on information of the dynamic location of a part and the human operator, and system working status. For instance, if the human operator is waiting for a message from Big-E within three seconds walking distance from MP1, and a part, whose size and weight are 1 ft × 1 ft × 1 ft and 10 LB, needs to be moved from MP1 to BS, the expected average time of the human task to move the part is eight seconds and that of the robot is 10 seconds. Thus, the human operator is supposed to be faster than the robot to accomplish this specific task, and the controller will allocate the task to the human operator as shown in Figure 12. In this case, the external transition function in affordance-based MPSG needs to be revised as follows,

$$
\delta\_{\mathbf{E}} \colon \mathbf{Q} \times \Sigma\_{\mathbf{E}} \to \mathbf{Q}
$$

*δE*((*v*, *l*, *I*(part)), a) = *δ<sup>H</sup>* ((*v*, *l*, *I*(part)), a) if a ∈ *PA* ⊆ *ΣH* and the human is expected to perform a task faster than the automated MH, and

*δE*(*v, l, I*(part)), a) = ( *δ<sup>M</sup>* (*v, a*), *l*, 0), otherwise.

134 Manufacturing System

Sheridan (2000) discusses a list to assert "what men are better at" and "what machines are

The gaps between 'what machines are better at and what humans are better at' are getting narrower as machines are replacing human more and more with the development of artificial intelligence technologies. However, the complete replacing humans with the automated machines are almost impossible and impractical partly because of both economic

In this sense, the function allocations between machines and humans in the human-involved automated system are one of the vital factors to control the system in effective and flexible ways. As pointed out in the previous section 3.1, human actions are available depending on the environmental affordances, and transitions by human can be realized by satisfying both system affordances and corresponding human effectivities Thus, from the system point of view, the controller needs to differentiate the set of actions that humans are better at from actions that machine are better at with consideration of availability of human actions

Suppose that the material handling time (e.g., time for picking up, moving, and putting a part) and material lifting capability (e.g., part weight, volume, size, and temperature) can be critical factors to allocate work between human material handlers and robots in a manufacturing cell. If the controller is able to evaluate the availability of a resource (either human or machine) which can reduce a processing time for a material handling job at a certain point of time and space, the whole system works faster and more intelligent to increase its productivity. For example, if we consider a simple human-machine cooperative

The controller is able to evaluate expected average processing time for each task and allocate the task between a human operator and a robot based on information of the dynamic

manufacturing cell shown in Figure 4 with following characteristics;

*Human capable part size and weight ≤ 3ft × 3ft × 3ft and 20 LB.* 

*Time for a robot to move a part from a resource to a resource = 10 ± 0.5 sec., Time for a human to move a part between adjacent resources = 5 ± 2 sec., Time for a human to move a part between facing resources = 10 ± 5 sec., and* 

**4.1 Work allocations in human-machine cooperative systems** 

1. Detecting small amount of visual, auditory, or chemical energy.

4. Storing information for long periods of time, and recalling appropriate parts.

better at" (MABA-MABA) as follows;

2. Perceiving patterns of light or sound. 3. Improvising and using flexible procedures.

1. Responding quickly to control signals. 2. Applying great force smoothly and precisely. 3. Storing information briefly, erasing it completely.

and technical reasons (Brann et al., 1996).

5. Reasoning inductively. 6. Exercising judgment. **<Machines are better at>:** 

4. Reasoning deductively.

identified by the model.

**<Humans are usually conceived to be better at>:** 

Fig. 12. Task allocation between human and machine in affordance-based MPSG.

#### **4.2 Classification of errors**

In the perspectives of systems theory and controls, a human agent is neither completely controllable nor perfectly predictable because of his or her nondeterministic and complex behaviors. For this reason, human-machine interactive system models need to harness dynamic human decision making processes into discrete system contexts. The level of modeling grains is defined with respect to the modeling purposes and modelers' perspectives on the systems. Representation of systems using finite numbers of states and transitions poses a lot of challenges to make a model complete by itself. Thus, comprehensive definition and classification of errors and error states in discrete system models can increase modeling easiness, simplicity and completeness.

For instance, the human-involved automata model of 'a semi-automated manufacturing system' illustrated in section 3.4 should contain an additional system state of the absorbing (error) state. In this modeling representation, human actions and system transitions that may not lead to the desired states, which come from the goal of the human-involved system, directly go to the absorbing state. Only valid interactions between a human and a system can be parts of a human-involved or human-machine cooperative process that change system states from a current to a next state which is placed within the process to the desired goal states.

It may not be critical to investigate errors in descriptive system representation as mentioned above. However, in the perspectives of system control models, system recoveries from errors are important to accomplish the seamless and complete modeling of human-involved systems. Thus, investigation of errors and their proper classification in systems are one of keys to develop control models for human-machine cooperative systems.

### **4.2.1 Error classification in extended MPSG systems**

In human-involved systems, human errors are considered important factors from a control point of view because sometimes system status is significantly changed by the human errors which are not within traceable states. It is well known that there are a number of topics to be addressed in terms of human errors. Shin et al. (2006c) investigated human operational errors concerning the human material handling in extended MPSG controls. In the authors' research, only human operational errors that are directly related with physical material handling tasks are considered, and human operational errors are classified into two separate categories; location errors and orientation errors.

A location error means that a human material handler made a mistake to pick or put a part on a wrong resource location. A human may commit a location error during his or her material handling task by loading or unloading a specific part on some equipment (resources) which are not in the proper process plans for the part. An orientation error is the case of not properly placing a part on equipment. For example, a human may commit an orientation error when he or she places and fixes the part on the controllable vice. Even if the human operator places the part on the right equipment (location), he or she may make an orientation error because of placement of the part in wrong directions and fixation of the part improperly.

Location and orientation errors may hinder the system from starting a proper operation in processes, and this failure causes the system to stop and wait for a recovery action. Every part in system operations is represented by its own unique state that is specified in a partstate graph with electronic sensors that can check the physical precondition of a system operation *α*, *ρα*∈*PE*. Therefore, location and orientation errors are checked by sensors installed on equipment before the system starts a process.

### **4.2.2 Error classification in affordance-based MPSG systems**

The location and orientation errors stated in the previous section 4.2.1 are taxonomies under physical preconditions regarding coordination states of a part in systems. However, in the ecological definition of affordances, properties of affordances, effectivities, and possible human actions in systems should have ontological assumptions related with space and time (Turvey, 1992), and the failure to satisfy these assumptions can lead a system state to undesirable states or make an improper transition. The cases of failing to satisfy assumptions in space dimension fall into the category of location and orientation errors. The cases of failing to satisfy assumptions in time domains, however, were not investigated.

In the perspectives of control models, actual system status and behaviors should coincide with representation of states and events within the same time and space domains. The detection of location and orientation errors can be easily performed by using sensors installed on resources (equipment) in control systems, while the detection of failing to

It may not be critical to investigate errors in descriptive system representation as mentioned above. However, in the perspectives of system control models, system recoveries from errors are important to accomplish the seamless and complete modeling of human-involved systems. Thus, investigation of errors and their proper classification in systems are one of

In human-involved systems, human errors are considered important factors from a control point of view because sometimes system status is significantly changed by the human errors which are not within traceable states. It is well known that there are a number of topics to be addressed in terms of human errors. Shin et al. (2006c) investigated human operational errors concerning the human material handling in extended MPSG controls. In the authors' research, only human operational errors that are directly related with physical material handling tasks are considered, and human operational errors are classified into two separate

A location error means that a human material handler made a mistake to pick or put a part on a wrong resource location. A human may commit a location error during his or her material handling task by loading or unloading a specific part on some equipment (resources) which are not in the proper process plans for the part. An orientation error is the case of not properly placing a part on equipment. For example, a human may commit an orientation error when he or she places and fixes the part on the controllable vice. Even if the human operator places the part on the right equipment (location), he or she may make an orientation error because of placement of the part in wrong directions and

Location and orientation errors may hinder the system from starting a proper operation in processes, and this failure causes the system to stop and wait for a recovery action. Every part in system operations is represented by its own unique state that is specified in a partstate graph with electronic sensors that can check the physical precondition of a system operation *α*, *ρα*∈*PE*. Therefore, location and orientation errors are checked by sensors

The location and orientation errors stated in the previous section 4.2.1 are taxonomies under physical preconditions regarding coordination states of a part in systems. However, in the ecological definition of affordances, properties of affordances, effectivities, and possible human actions in systems should have ontological assumptions related with space and time (Turvey, 1992), and the failure to satisfy these assumptions can lead a system state to undesirable states or make an improper transition. The cases of failing to satisfy assumptions in space dimension fall into the category of location and orientation errors. The cases of failing to satisfy assumptions in time domains, however, were not investigated.

In the perspectives of control models, actual system status and behaviors should coincide with representation of states and events within the same time and space domains. The detection of location and orientation errors can be easily performed by using sensors installed on resources (equipment) in control systems, while the detection of failing to

keys to develop control models for human-machine cooperative systems.

**4.2.1 Error classification in extended MPSG systems** 

categories; location errors and orientation errors.

installed on equipment before the system starts a process.

**4.2.2 Error classification in affordance-based MPSG systems** 

fixation of the part improperly.

satisfy time constraints cannot be considered in the existing extended MPSG control systems. Specifically, the automated equipment in systems run based on the logical preconditions within systems, but a human in the system tends to take an action relying on his or her perception-based actions which are available within a specific space dimension and time duration containing the affordance-effectivity duals for those actions. For this reason, one additional error classification for a human needs to be considered; a set of transition errors with respect to time and space constraints between a human and a part. A human may commit transition errors if he or she missed to perform a desired task within a specific time range.

An example of transition errors can be expressed in affordance-based MPSGs as shown in Figure 13. The errors can be detected and checked when a specific human action is not taking within the time and space conditions described in a set of action conditions, *C*. The action conditions can be estimated based on the information of the relative properties between a human material handler and a part, such as size and weight of the part, lifting and moving capabilities of the human material handler, relative distance between the part and human. The size and weight of a part can be detected by sensors installed on equipment, the human capabilities are pre-programmed based on the personal information, and the location, viewing, and moving direction of the human material handler can be detected by a vision sensor installed in the shop floor system. The representation of affordance-based MPSG systems contains time-related tuples which can measure and check the time constraints for existence of possible human actions. The time advances are checked within control programs for equipment and the system allows a human material handler to perform human tasks only within a specific time range.

Fig. 13. Examples of location and orientation errors in affordance-based MPSG systems.

#### **4.2.3 Error recovery**

The detection and classification of human errors in human-machine cooperative systems are crucial to validate the control processes of human-involved systems. The analysis of error status in systems can guarantee the prompt and proper recovery of the systems from undesirable system states.

When location and orientation errors are occurred, the system will stop and wait for recovery action by either incurring automatic recovery module or calling human material handlers. In case of a transition error, the system can simply recover it by re-allocating the human task to a machine without stopping and recalling an error recovery module as shown in Figure 14, if the desired task can be performed by either a human or a machine. If a desired human task is failed to be performed within an eligible time range, machine can take an action instead of a human. However, if the required task for a specific system transition can be done only by a human, it should be recovered by human operators. The transition error recovery process is described as shown in Figure 15.

Fig. 14. Recovery of a transition error by re-allocating a human task to machine.

Fig. 15. Human transition error recovery procedure.

#### **5. Summary**

This chapter presents the modeling concept and formal representation of human-involved manufacturing control systems called affordance-based MPSG. With consideration of affordances in manufacturing systems, a human can participate in system operations and dynamic task allocation between a human and a machine is available.

Investigation of errors and their classification are also discussed. In regard to human transition errors, the automatic task reallocation to machine is a solution to solve the errors in easy ways. However, if the original task for a specific transition was only available for a human operator, an error recovery task by a human should be incurred to solve it.

#### **6. References**

138 Manufacturing System

When location and orientation errors are occurred, the system will stop and wait for recovery action by either incurring automatic recovery module or calling human material handlers. In case of a transition error, the system can simply recover it by re-allocating the human task to a machine without stopping and recalling an error recovery module as shown in Figure 14, if the desired task can be performed by either a human or a machine. If a desired human task is failed to be performed within an eligible time range, machine can take an action instead of a human. However, if the required task for a specific system transition can be done only by a human, it should be recovered by human operators. The

transition error recovery process is described as shown in Figure 15.

Fig. 14. Recovery of a transition error by re-allocating a human task to machine.

This chapter presents the modeling concept and formal representation of human-involved manufacturing control systems called affordance-based MPSG. With consideration of affordances in manufacturing systems, a human can participate in system operations and

Fig. 15. Human transition error recovery procedure.

dynamic task allocation between a human and a machine is available.

**5. Summary** 


Zeigler, B. P. (1976). *Theory of Modeling and Simulation*, John Wiley & Sons, ISBN 0-12-778455- 1, New York, NY
