**6. Alignment based frameworks**

Since 2DPCA can be viewed as the row-based PCA, that means the information contains only in row direction. Although, combining it with the column-based 2DPCA can consider the information in both row and column directions. But there still be other directions which should be considered.

## **6.1 Diagonal-based 2DPCA (DiaPCA)**

The motivation for developing the DiaPCA method originates from an essential observation on the recently proposed 2DPCA (Yang et al., 2004). In contrast to 2DPCA, DiaPCA seeks the optimal projective vectors from diagonal face images and therefore the correlations between variations of rows and those of columns of images can be kept. Therefore, this problem can solve by transforming the original face images into corresponding diagonal face images, as shown in Fig. 2 and Fig. 3. Because the rows (columns) in the transformed diagonal face images simultaneously integrate the information of rows and columns in original images, it can reflect both information between rows and those between columns. Through the entanglement of row and column information, it is expected that DiaPCA may find some useful block or structure information for recognition in original images. The sample diagonal face images on Yale database are displayed in Fig. 4.

Experimental results on a subset of FERET database (Zhang et al., 2006) show that DiaPCA is more accurate than both PCA and 2DPCA. Furthermore, it is shown that the accuracy can be further improved by combining DiaPCA and 2DPCA together.

#### **6.2 Image cross-covariance analysis**

12 Will-be-set-by-IN-TECH

For illustration, we assume that there are 4 classes, as shown in Fig. 1. The input image must be normalized with the averaging images of all 4 classes. And then project to 2DPCA subspaces of each class. After that the image is reconstructed by the projection matrices (**X**) in each class. The DFCSS is used now to measure the similarity between the reconstructed image and the normalized original image on each CSS. From Fig. 1, the DFCSS of the first class is minimum,

Since 2DPCA can be viewed as the row-based PCA, that means the information contains only in row direction. Although, combining it with the column-based 2DPCA can consider the information in both row and column directions. But there still be other directions which

The motivation for developing the DiaPCA method originates from an essential observation on the recently proposed 2DPCA (Yang et al., 2004). In contrast to 2DPCA, DiaPCA seeks the optimal projective vectors from diagonal face images and therefore the correlations between variations of rows and those of columns of images can be kept. Therefore, this problem can solve by transforming the original face images into corresponding diagonal face images, as shown in Fig. 2 and Fig. 3. Because the rows (columns) in the transformed diagonal face images simultaneously integrate the information of rows and columns in original images, it can reflect both information between rows and those between columns. Through the entanglement of row and column information, it is expected that DiaPCA may find some

Fig. 1. CSS-based 2DPCA diagram.

**6. Alignment based frameworks**

**6.1 Diagonal-based 2DPCA (DiaPCA)**

should be considered.

thus we decide this input image is belong to the first class.

In PCA, the covariance matrix provides a measure of the strength of the correlation of all pixel pairs. Because of the limit of the number of training samples, thus this covariance cannot be well estimated. While the performance of 2DPCA is better than PCA, although all of the correlation information of pixel pairs are not employed for estimating the image covariance matrix. Nevertheless, the disregard information may possibly include the useful information. Sanguansat et al. (2007a) proposed a framework for investigating the information which was neglected by original 2DPCA technique, so called Image Cross-Covariance Analysis (ICCA). To achieve this point, the *image cross-covariance matrix* is defined by two variables, the first variable is the original image and the second one is the shifted version of the former. By our shifting algorithm, many image cross-covariance matrices are formulated to cover all of the information. The Singular Value Decomposition (SVD) is applied to the image cross-covariance matrix for obtaining the optimal projection matrices. And we will show that these matrices can be considered as the orthogonally rotated projection matrices of traditional 2DPCA. ICCA is different from the original 2DPCA on the fact that the transformations of our method are generalized transformation of the original 2DPCA.

First of all, the relationship between 2DPCA's image covariance matrix **G**, in Eq. (5), and PCA's covariance matrix **C** can be considered as

$$\mathbf{C}\left(i,j\right) = \sum\_{k=1}^{m} \mathbf{C}\left(m\left(i-1\right) + k, m\left(j-1\right) + k\right) \tag{43}$$

where **G**(*i*, *j*) and **C**(*i*, *j*) are the *i th* row, *j th* column element of matrix **G** and matrix **C**, respectively. And *m* is the height of the image.

For illustration, let the dimension of all training images are 3 by 3. Thus, the covariance matrix of these images will be a 9 by 9 matrix and the dimension of image covariance matrix is only 3 by 3, as shown in Fig. 5.

From Eq. (43), each elements of **G** is the sum of all the same label elements in **C**, for example:

$$\begin{aligned} \mathbf{G}(\mathbf{1},\mathbf{1}) &= \mathbf{C}(\mathbf{1},\mathbf{1}) + \mathbf{C}(\mathbf{2},\mathbf{2}) + \mathbf{C}(\mathbf{3},\mathbf{3}),\\ \mathbf{G}(\mathbf{1},\mathbf{2}) &= \mathbf{C}(\mathbf{1},\mathbf{4}) + \mathbf{C}(\mathbf{2},\mathbf{5}) + \mathbf{C}(\mathbf{3},\mathbf{6}),\\ \mathbf{G}(\mathbf{1},\mathbf{3}) &= \mathbf{C}(\mathbf{1},\mathbf{7}) + \mathbf{C}(\mathbf{2},\mathbf{8}) + \mathbf{C}(\mathbf{3},\mathbf{9}). \end{aligned} \tag{44}$$

It should be note that the total power of image covariance matrix equals and traditional covariance matrix **C** are identical,

$$\operatorname{tr}(\mathbf{G}) = \operatorname{tr}(\mathbf{C}).\tag{45}$$

From this point of view in Eq. (43), we can see that image covariance matrix is collecting the classification information only 1/*m* of all information collected in traditional covariance matrix. However, there are the other (*m* − 1)/*m* elements of the covariance matrix still be not

Fig. 5. The relationship of covariance and image covariance matrix.

*m* ∑ *k*=1

**G***L*(*i*, *j*)=

*<sup>f</sup>*(*x*) =

The samples of shifted images **B***<sup>L</sup>* are presented in Fig. 6.

the PCA's covariance matrix as

the *image cross-covariance matrix* or

where 1 ≤ *L* ≤ *mn*.

cross-covariance matrix.

considered. By the experimental results in Sanguansat et al. (2007a). For investigating how the retaining information in 2D subspace is rich for classification, the new **G** is derived from

Two-Dimensional Principal Component Analysis and Its Extensions 15

*x* + *L* − 1, 1 ≤ *x* ≤ *mn*−*L*+1

The **G***<sup>L</sup>* can also be determined by applying the shifting to each images instead of averaging certain elements of covariance matrix. Therefore, the **G***<sup>L</sup>* can alternatively be interpreted as

where **B***<sup>L</sup>* is the *Lth* shifted version of image **A** that can be created via algorithm in Table 3.

In 2DPCA, the columns of the projection matrix, **X**, are obtained by selection the eigenvectors which corresponding to the *d* largest eigenvalues of image covariance matrix, in Eq. (5). While in ICCA, the eigenvalues of image cross-covariance matrix, **G***L*, are complex number with non-zero imaginary part. The Singular Value Decomposition (SVD) is applied to this matrix instead of Eigenvalue decomposition. Thus, the ICCA projection matrix contains a set of orthogonal basis vectors which corresponding to the *d* largest singular values of image

For understanding the relationship between the ICCA projection matrix and the 2DPCA projection matrix, we will investigate in the simplest case, i.e. there are only one training

**C** (*f* (*m* (*i* − 1) + *k*), *m* (*j* − 1) + *k*), (46)

*<sup>x</sup>* <sup>−</sup> *mn* <sup>+</sup> *<sup>L</sup>* <sup>−</sup> 1, *mn*−*L*+<sup>2</sup> <sup>≤</sup> *<sup>x</sup>* <sup>≤</sup> *mn* (47)

**<sup>G</sup>***<sup>L</sup>* <sup>=</sup> *<sup>E</sup>*[(**B***<sup>L</sup>* <sup>−</sup> *<sup>E</sup>*[**B***L*])*T*(**<sup>A</sup>** <sup>−</sup> *<sup>E</sup>*[**A**])] (48)

Fig. 2. Illustration of the ways for deriving the diagonal face images: If the number of columns is more than the number of rows

Fig. 3. Illustration of the ways for deriving the diagonal face images: If the number of columns is less than the number of rows

Fig. 4. The sample diagonal face images on Yale database.

Fig. 5. The relationship of covariance and image covariance matrix.

considered. By the experimental results in Sanguansat et al. (2007a). For investigating how the retaining information in 2D subspace is rich for classification, the new **G** is derived from the PCA's covariance matrix as

$$\mathbf{G}\_{L}(i,j) = \sum\_{k=1}^{m} \mathbf{C}\left(f\left(m\left(i-1\right)+k\right), m\left(j-1\right)+k\right),\tag{46}$$

$$f(\mathbf{x}) = \begin{cases} \mathbf{x} + \mathbf{L} - \mathbf{1}, & \mathbf{1} \le \mathbf{x} \le mn - \mathbf{L} + \mathbf{1} \\ \mathbf{x} - mn + \mathbf{L} - \mathbf{1}, \ mn - \mathbf{L} + \mathbf{2} \le \mathbf{x} \le mn \end{cases} \tag{47}$$

where 1 ≤ *L* ≤ *mn*.

14 Will-be-set-by-IN-TECH

Fig. 2. Illustration of the ways for deriving the diagonal face images: If the number of

Fig. 3. Illustration of the ways for deriving the diagonal face images: If the number of

columns is more than the number of rows

columns is less than the number of rows

Fig. 4. The sample diagonal face images on Yale database.

The **G***<sup>L</sup>* can also be determined by applying the shifting to each images instead of averaging certain elements of covariance matrix. Therefore, the **G***<sup>L</sup>* can alternatively be interpreted as the *image cross-covariance matrix* or

$$\mathbf{G}\_L = E[(\mathbf{B}\_L - E[\mathbf{B}\_L])^T(\mathbf{A} - E[\mathbf{A}])] \tag{48}$$

where **B***<sup>L</sup>* is the *Lth* shifted version of image **A** that can be created via algorithm in Table 3. The samples of shifted images **B***<sup>L</sup>* are presented in Fig. 6.

In 2DPCA, the columns of the projection matrix, **X**, are obtained by selection the eigenvectors which corresponding to the *d* largest eigenvalues of image covariance matrix, in Eq. (5). While in ICCA, the eigenvalues of image cross-covariance matrix, **G***L*, are complex number with non-zero imaginary part. The Singular Value Decomposition (SVD) is applied to this matrix instead of Eigenvalue decomposition. Thus, the ICCA projection matrix contains a set of orthogonal basis vectors which corresponding to the *d* largest singular values of image cross-covariance matrix.

For understanding the relationship between the ICCA projection matrix and the 2DPCA projection matrix, we will investigate in the simplest case, i.e. there are only one training


Table 3. The Image Shifting Algorithm for ICCA

image. Therefore, the image covariance matrix and image cross-covariance matrix are simplified to **A***T***A** and **B***<sup>T</sup> <sup>L</sup>***A**, respectively.

The image **A** and **B***<sup>L</sup>* can be decomposed by using Singular Value Decomposition (SVD) as

$$\mathbf{A} = \mathbf{U}\_{\mathbf{A}} \mathbf{D}\_{\mathbf{A}} \mathbf{V}\_{\mathbf{A}'}^T \tag{49}$$

$$\mathbf{B}\_L = \mathbf{U}\_\mathbf{B} \mathbf{D}\_\mathbf{B} \mathbf{V}\_\mathbf{B}^T. \tag{50}$$

Where **VA** and **VB***<sup>L</sup>* contain a set of the eigenvectors of **A***T***A** and **B***<sup>T</sup> <sup>L</sup>***B***L*, respectively. And **UA** and **UB***<sup>L</sup>* contain a set of the eigenvectors of **AA***<sup>T</sup>* and **B***L***B***<sup>T</sup> <sup>L</sup>*, respectively. And **DA** and **DB***<sup>L</sup>* contain the singular values of **A** and **B***L*, respectively. If all eigenvectors of **A***T***A** are selected then the **VA** is the 2DPCA projection matrix, i.e. **X** = **VA**.

Let **Y** = **AVA** and **Z** = **B***L***VB***<sup>L</sup>* are the projected matrices of **A** and **B**, respectively. Thus,

$$\mathbf{B}\_L^T \mathbf{A} = \mathbf{V}\_{\mathbf{B}\_L} \mathbf{Z}^T \mathbf{Y} \mathbf{V}\_{\mathbf{A}}^T. \tag{51}$$

Denoting the SVD of **Z***T***Y** by

$$\mathbf{Z}^T \mathbf{Y} = \mathbf{P} \mathbf{D} \mathbf{Q}^T,\tag{52}$$

and substituting into Eq. (51) gives

$$\begin{split} \mathbf{B}\_{L}^{T}\mathbf{A} &= \mathbf{V}\_{\mathbf{B}\_{L}}\mathbf{P}\mathbf{D}\mathbf{Q}^{T}\mathbf{V}\_{\mathbf{A}}^{T} \\ &= \mathbf{R}\mathbf{D}\mathbf{S}^{T} \end{split} \tag{53}$$

where **RDS***<sup>T</sup>* is the singular value decomposition of **B***<sup>T</sup> <sup>L</sup>***A** because of the unique properties of the SVD operation. It should be note that **B***<sup>T</sup> <sup>L</sup>***<sup>A</sup>** and **<sup>Z</sup>***T***<sup>Y</sup>** have the same singular values. Therefore,

$$\mathbf{R} = \mathbf{V}\_{\mathbf{B}\_L} \mathbf{P}\_{\prime} \tag{54}$$

$$\mathbf{S} = \mathbf{V}\_{\mathbf{A}} \mathbf{Q} = \mathbf{X} \mathbf{Q} \tag{55}$$

Fig. 6. The samples of shifted images on the ORL database.

Two-Dimensional Principal Component Analysis and Its Extensions 17

can be thought of as orthogonally rotated of projection matrices **VA** and **VB***<sup>L</sup>* , respectively.

As a result in Eq. (55), the ICCA projection matrix is the orthogonally rotated of original 2DPCA projection matrix.

16 Will-be-set-by-IN-TECH

and the number of shifting *L* (2 ≤ *L* ≤ *mn*). *S*2: Initialize the row index, *irow* = [2, . . . , *n*, 1], and output image **B** = *m* × *n* zero matrix.

*S*4: Sort the first row of **A** by the row index, *irow*. *S*5: Set the last row of **B** = the first row of **A**.

image. Therefore, the image covariance matrix and image cross-covariance matrix are

The image **A** and **B***<sup>L</sup>* can be decomposed by using Singular Value Decomposition (SVD) as

**A** = **UADAV***<sup>T</sup>*

**<sup>B</sup>***<sup>L</sup>* = **UB***L***DB***L***V***<sup>T</sup>*

contain the singular values of **A** and **B***L*, respectively. If all eigenvectors of **A***T***A** are selected

*<sup>L</sup>***<sup>A</sup>** <sup>=</sup> **VB***L***Z***T***YV***<sup>T</sup>*

*<sup>L</sup>***<sup>A</sup>** <sup>=</sup> **VB***L***PDQ***T***V***<sup>T</sup>*

= **RDS***T*,

can be thought of as orthogonally rotated of projection matrices **VA** and **VB***<sup>L</sup>* , respectively. As a result in Eq. (55), the ICCA projection matrix is the orthogonally rotated of original

Let **Y** = **AVA** and **Z** = **B***L***VB***<sup>L</sup>* are the projected matrices of **A** and **B**, respectively. Thus,

**B***T*

**B***T*

*th* row of **B** = the (*j* + 1)*th* row of **A**.

**<sup>A</sup>**, (49)

**<sup>B</sup>***<sup>L</sup>* . (50)

**<sup>A</sup>**. (51)

**<sup>A</sup>** (53)

*<sup>L</sup>***A** because of the unique properties

*<sup>L</sup>***<sup>A</sup>** and **<sup>Z</sup>***T***<sup>Y</sup>** have the same singular values.

**Z***T***Y** = **PDQ***T*, (52)

**R** = **VB***L***P**, (54) **S** = **VAQ** = **XQ** (55)

*<sup>L</sup>***B***L*, respectively. And **UA**

*<sup>L</sup>*, respectively. And **DA** and **DB***<sup>L</sup>*

*S*1: Input *m* × *n* original image **A**

*S*3: For *i* = 1, 2, . . . , *L* − 1

*S*6: For *j* = 1, 2, . . . , *m* − 1

*<sup>L</sup>***A**, respectively.

Where **VA** and **VB***<sup>L</sup>* contain a set of the eigenvectors of **A***T***A** and **B***<sup>T</sup>*

and **UB***<sup>L</sup>* contain a set of the eigenvectors of **AA***<sup>T</sup>* and **B***L***B***<sup>T</sup>*

then the **VA** is the 2DPCA projection matrix, i.e. **X** = **VA**.

where **RDS***<sup>T</sup>* is the singular value decomposition of **B***<sup>T</sup>*

of the SVD operation. It should be note that **B***<sup>T</sup>*

*S*7: Set the *j*

*S*8: End For *S*9: Set **A** = **B** *S*10: End For

Table 3. The Image Shifting Algorithm for ICCA

simplified to **A***T***A** and **B***<sup>T</sup>*

Denoting the SVD of **Z***T***Y** by

2DPCA projection matrix.

Therefore,

and substituting into Eq. (51) gives

Fig. 6. The samples of shifted images on the ORL database.

*S*1: Project image, **A**, by Eq. (10).

from **Y** (*r* < *m*).

*S*5: End For

prediction.

*S*2: For *i* = 1 to the number of classifiers

Table 4. Two-Dimensional Random Subspace Analysis Algorithm

*S*4: Construct the nearest neighbor classifier, **C***<sup>r</sup>*

2DPCA, because the column direction does not depend on the eigenvalue.

**7.2 Two-dimensional diagonal random subspace analysis (2D**2**RSA)**

Similar to 2DRSA, the 2D2RSA algorithm is listed in Table 5.

*S*3: Randomly select a *r* dimensional random subspace, **Z***<sup>r</sup>*

*S*6: Combine the output of each classifiers by using majority voting.

Two-Dimensional Principal Component Analysis and Its Extensions 19

Different from PCA, the 2DPCA feature is a matrix form. Thus, RSM is more suitable for

A framework of Two-Dimensional Random Subspace Analysis (2DRSA) (Sanguansat et al., n.d.) is proposed to extend the original 2DPCA. The RSM is applied to feature space of 2DPCA for generating the vast number of feature subspaces, which be constructed by an autonomous, pseudorandom procedure to select a small number of dimensions from a original feature space. For a *m* by *n* feature matrix, there are 2*<sup>m</sup>* such selections that can be made, and with each selection a feature subspace can be constructed. And then individual classifiers are created only based on those attributes in the chosen feature subspace. The outputs from different individual classifiers are combined by the uniform majority voting to give the final

The Two-Dimensional Random Subspace Analysis consists of two parts, 2DPCA and RSM. After data samples was projected to 2D feature space via 2DPCA, the RSM are applied here by taking advantage of high dimensionality in these space to obtain the lower dimensional multiple subspaces. A classifier is then constructed on each of those subspaces, and a combination rule is applied in the end for prediction on the test sample. The 2DRSA algorithm is listed in Table 4, the image matrix, **A**, is projected to feature space by 2DPCA projection in Eq. (10). In this feature space, it contains the data samples in matrix form, the *m* × *d* feature matrix, **Y** in Eq. (10). The dimensions of feature matrix **Y** depend on the height of image (*m*) and the number of selected eigenvectors of the image covariance matrix **G** (*d*). Therefore, only the information which embedded in each element on the row direction was sorted by the eigenvalue but not on the column direction. It means this method should randomly pick up some rows of feature matrix **Y** to construct the new feature matrix **Z**. The dimension of **Z** is *r* × *d*, normally *r* should be less than *m*. The results in Ho (1998b) have shown that for a variety of data sets adopting half of the feature components usually yields good performance.

The extension of 2DRSA was proposed in Sanguansat et al. (2007b), namely the Two-Dimensional Diagonal Random Subspace Analysis. It consists of two parts i.e. DiaPCA and RSM. Firstly, all images are transformed into the diagonal face images as in Section 6.1. After that the transformed image samples was projected to 2D feature space via DiaPCA, the RSM are applied here by taking advantage of high dimensionality in these space to obtain the lower dimensional multiple subspaces. A classifier is then constructed on each of those subspaces, and a combination rule is applied in the end for prediction on the test sample.

*i* ,

*i* .
