**2.1 What are temporal postural synergies?**

The synergies presented in this chapter are purely kinematic synergies. These are the synergies derived from angular velocities of the finger joints of the human hand collected during grasping tasks. For example, in the Fig. 1 two synergies (**s**1, **s**2) combine using a weighted linear combination (*w*1**s**<sup>1</sup> + *w*2**s**2) to achieve a grasping hand movement. *w*<sup>1</sup> and *w*<sup>2</sup> represent weights of control signals. Each row of a synergy corresponds to the angular velocity profile of a finger joint; For example, the first synergy represents the synchronous large movement of first joint and medium movement of the second joint followed by a small movement of the third joint. In this example, **s**<sup>1</sup> (blue) and **s**<sup>2</sup> (brown) form a weighted (*w*<sup>1</sup> = *w*<sup>2</sup> = 0.5) combination to result in the aggregate movement(black) on the right hand side. For illustration purposes, only 3 of 10 joints of the hand are shown in the synergies and the reconstructed movement. Also shown in the figure are the hand postures of the reconstructed movement across time. As these synergies preserve both the temporal structure and the postural information these are termed as *temporal posturalsynergies* ( Vinjamuri et al. (2010a;b)).

#### **2.2 Applications of synergies**

In our attempt to apply linear and nonlinear dimensionality reduction methods to solve the problem of extraction of synergies, let us first know how these synergies are being used in the real world applications in the areas of prosthesis and rehabilitation.

(i) Prosthetics: Apart from neuro-physiological significance, synergies are viewed to be crucial design elements in future generation prosthetic hands. Biologically inspired synergies have already taken prime place in artificial hands ( Popovic & Popovic (2001)). Synergies based 2 Will-be-set-by-IN-TECH

and the term has been generalized to indicate the shared patterns observed in the behaviors of muscles, joints, forces, actions, etc. Synergies in hand movements especially present a complex optimization problem as to how the central nervous system (CNS) controls the hand with over 25 degrees of freedom(DoF) ( Mackenzie & Iberall (1994)). Yet, the CNS handles all the movements effortlessly and at the same time dexterously. Endeavoring to solve the DoF problem, many researchers have proposed several concepts of synergies such as the following: (i) Postural synergies: In Jerde et al. (2003); Mason et al. (2001); Santello et al. (1998; 2002); Thakur et al. (2008); Todorov & Ghahramani (2004), researchers found that the entire act of grasp can be described by a small number of dominant postures, which were defined as

(ii) Kinematic synergies: Studies in Grinyagin et al. (2005); Vinjamuri et al. (2007) expressed the angular velocities of finger joints as linear combinations of a small number of kinematic synergies, which were also angular velocities of finger joints but were extracted from a large set of natural movements. Kinematic synergies are not limited to hand movements. In ( d'Avella et al. (2006)), d'Avella et al. reported that kinematic synergies were found in tracking

(iii) Dynamic synergies: Dynamic synergies were defined as stable correlations between joint

The above classification was already presented in Vinjamuri et al. (2010b). In addition to synergies proposed in postures, and kinematics which are of relevance to the current study,

The synergies presented in this chapter are purely kinematic synergies. These are the synergies derived from angular velocities of the finger joints of the human hand collected during grasping tasks. For example, in the Fig. 1 two synergies (**s**1, **s**2) combine using a weighted linear combination (*w*1**s**<sup>1</sup> + *w*2**s**2) to achieve a grasping hand movement. *w*<sup>1</sup> and *w*<sup>2</sup> represent weights of control signals. Each row of a synergy corresponds to the angular velocity profile of a finger joint; For example, the first synergy represents the synchronous large movement of first joint and medium movement of the second joint followed by a small movement of the third joint. In this example, **s**<sup>1</sup> (blue) and **s**<sup>2</sup> (brown) form a weighted (*w*<sup>1</sup> = *w*<sup>2</sup> = 0.5) combination to result in the aggregate movement(black) on the right hand side. For illustration purposes, only 3 of 10 joints of the hand are shown in the synergies and the reconstructed movement. Also shown in the figure are the hand postures of the reconstructed movement across time. As these synergies preserve both the temporal structure and the postural information these are termed as *temporal posturalsynergies* ( Vinjamuri et al.

In our attempt to apply linear and nonlinear dimensionality reduction methods to solve the problem of extraction of synergies, let us first know how these synergies are being used in the

(i) Prosthetics: Apart from neuro-physiological significance, synergies are viewed to be crucial design elements in future generation prosthetic hands. Biologically inspired synergies have already taken prime place in artificial hands ( Popovic & Popovic (2001)). Synergies based

real world applications in the areas of prosthesis and rehabilitation.

torques that were found during precision grip movements in Grinyagin et al. (2005).

synergies were also proposed in muscle activities d'Avella et al. (2006).

postural synergies.

7 DoF arm movements.

(2010a;b)).

**2.2 Applications of synergies**

**2.1 What are temporal postural synergies?**

Fig. 1. Two distinct synergies (**s**1, **s**2) use a weighted linear combination (*w*1**s**<sup>1</sup> + *w*2**s**2) to achieve a grasping hand movement. *w*<sup>1</sup> and *w*<sup>2</sup> represent weights of control signals. Each row of a synergy corresponds to the angular velocity profile of a finger joint; For example, the first synergy represents the synchronous large movement of first joint and medium movement of the second joint followed by a small movement of the third joint. In this example, **s**<sup>1</sup> (blue) and **s**<sup>2</sup> (brown) form a weighted (*w*<sup>1</sup> = *w*<sup>2</sup> = 0.5) combination to result in the aggregate movement(black) on the right hand side. For illustration purposes, only 3 of 10 joints of the hand are shown in the synergies and the reconstructed movement. Also shown in the figure are the hand postures of the reconstructed movement across time. Adapted from Vinjamuri et al. (2011)

on the principles of data reduction and dimensionality reduction, are soon to find place in tele-surgery and tele-robotics ( Vinjamuri et al. (2007)). Synergies are projected to be miniature windows to provide immense help in next generation rehabilitation. Recently our group has demonstrated a synergy based brain machine interface where two control signals calculated from the spectral powers of the brain signals controlled two synergies, that commanded a 10 DoF virtual hand ( Vinjamuri et al. (2011)). This showed promising results for controlling a synergy-based neural prosthesis.

(ii) Diagnostics: Applying similar concepts of synergies on the hand movements of the individuals with movement disorders, the sources that contain the tremor were isolated. Using blind source separation and dimensionality reduction methods, the possible neural sources that contained tremor were extracted from the hand movements of individuals with Essential Tremor ( Vinjamuri et al. (2009)). This led to an efficient quantification of tremor.

(iii) Robotics: Biologically inspired synergies are being used in balance control of humanoid robots ( Hauser et al. (2007)). Based on the principle that biological organisms recruit kinematic synergies that manage several joints, a control strategy for balance of humanoid robots was developed. This control strategy reduced computational complexity following a biological framework that central nervous system reduces the computational complexity of managing numerous degrees of freedom by effectively utilizing the synergies. Biologically inspired neural network controller models ( Bernabucci et al. (2007)) that can manage ballistic arm movements have been developed. The models simulated the kinematic aspects, with bell-shaped wrist velocity profiles, and generated movement specific muscular synergies for the execution of movements.

(iv) Rehabilitation: Bimanual coordination is damaged in brain lesions and brain disorders Vinjamuri et al. (2008). Using a small set of modifiable and adjustable synergies

Another concept of synergies was proposed by d'Avella et al. (2003). Although their work was not directly related to the hand movements, they investigated the muscle synergies of frogs during a variety of motor behaviors such as kicking. Using a gradient descent method, they decomposed the muscle activities into linear combinations of three task-independent time-varying synergies. They also observed that these synergies were very much related to movement kinematics and that similarities existed between synergies in different tasks.

Application of Linear and Nonlinear Dimensionality Reduction Methods 111

In this section, we first recorded the joint angles when ten subjects participated in an experiment of reaching and grasping tasks while wearing a dataglove. Then we transformed the recorded joint angles into joint angular velocities and further preprocessed it to prepare

> -0.02 0 0.02

> -0.01 0 0.01

**(radians/sample)**








**Angular**

Fig. 2. (a) Objects grasped by the subjects. (b) An example of a rapid grasp of a wooden toy nut. (c) Sensors of the CyberGlove used for analysis (dark circles.) (d) A sample of rapid movement profile (finger-joint-angular-velocity profile). Onset and end of movements are marked in the figure. Abbreviations: T, thumb; I, index finger; M, middle finger; R, ring finger; P, pinky finger; CMC, carpometacarpal joint; MCP, metacarpophalangeal joint; IP, interphalangeal joint; PIP, proximal interphalangeal joint; DIP, distal interphalangeal joint.

The experimental setup consisted of a right-handed CyberGlove (CyberGlove Systems LLC, San Jose, CA, USA) equipped with 22 sensors which can measure angles at all the finger joints. For the purpose of reducing computational burden, in this study we only considered 10 of the sensors which correspond to the metacarpophalangeal (MCP) and interphalangeal (IP) joints of the thumb and the MCP and proximal interphalangeal (PIP) joints of the other four fingers as shown in Fig. 2(c). These ten joints can capture most characteristics of the hand in grasping

**velocities of ten joints**

> -0.1 0

<sup>0</sup> <sup>50</sup> <sup>100</sup> <sup>150</sup> -0.2

**Samples**

**T\_MCP TͲIP IͲMCP IͲPIP MͲMCP MͲPIP RͲMCP RͲPIP PͲMCP PͲPIP**

(d)

**3. Preparing the hand kinematics for dimensionality reduction**

datasets to be used as inputs to the dimensionality reduction methods.

(b) (c)

The dotted lines showed the onset and end of movement.

(a)

**3.1 Experiment**

tasks.

tremendously simplifies the task of learning new skills or adapting to new environments. Constructing internal neural representations from a linear combination of a reduced set of basis functions might be crucial for generalizing to novel tasks and new environmental conditions ( Flash & Hochner (2005); Poggio & Bizzi (2004)).

#### **2.3 Extraction of synergies**

Synergies or movement primitives are viewed as small building blocks of movement that are present inherently within the movements and are shared across several movements. In other words, for example, in a set of hundred grasping movements, there might be a five or six synergies that are shared and common across all the movements. So it is to say that these hundred hand movements are composed of synergies. How do we decompose these hundred hand movements to a few building blocks of movement? This is the problem we are trying to solve.

In order to extract these primitives, several methods have been used. Several researchers view this as a problem of extracting basis functions. In fact, PCA can be viewed as extracting basis functions that are orthogonal to each other. Radial basis functions were also used as synergy approximations. Gradient descend method and non-negative matrix factorization methods ( d'Avella et al. (2003)), multivariate statistical techniques ( Santello et al. (2002)) were used in extracting the synergies. Different from the above interpretations of synergies, Todorov & Ghahramani (2004) suggested that synergistic control may not mean dimensionality reduction or simplification, but might imply task optimization using optimal feedback control.

In the coming sections we will use linear and nonlinear dimensionality reduction methods in extracting the synergies.

#### **2.4 Dimensionality reduction methods for extracting synergies**

In the previous section, we listed different methods used to extract the synergies. In this section these methods were limited to dimensionality reduction methods as these are of relevance to this chapter.

Based on the principal component analysis, Jerde et al. ( Jerde et al. (2003)) found support for the existence of postural synergies of angular configuration. The shape of human hand can be predicted using a reduced set of variables and postural synergies. Similarly, Santello et al. (1998) showed that a small number of postural synergies were sufficient to describe how human subjects grasped a large set of different objects. Moreover, Mason et al. (2001) used singular value decomposition (SVD) to demonstrate that a large number of hand postures during reach-to-grasp can be constructed by a small number of principal components or eigen postures.

With PCA, Braido & Zhang (2004) examined the temporal co-variation between finger-joint angles. Their results supported the view that the multi-joint acts of the hand are subject to stereotypical motion patterns controlled via simple kinematic synergies. In the above mentioned study of eigen postures, Mason et al. (2001) also investigated the temporal evolutions of the eigen postures and observed similar kinematic synergies across subjects and grasps. In addition, kinematic synergies have been observed in the spatiotemporal coordination between thumb and index finger movements and co-ordination of tip-to-tip finger movements ( Cole & Abbs (1986)).

Another concept of synergies was proposed by d'Avella et al. (2003). Although their work was not directly related to the hand movements, they investigated the muscle synergies of frogs during a variety of motor behaviors such as kicking. Using a gradient descent method, they decomposed the muscle activities into linear combinations of three task-independent time-varying synergies. They also observed that these synergies were very much related to movement kinematics and that similarities existed between synergies in different tasks.
