3.3. The relationship between consistent sign set (CSS) and cosine similarity of coefficient vector (COS)

Since proposed proximity based on both CSS and COS are trying to exploit more information from the solution coefficient of sparse representation, the relationship between each other is following:

	- Transform the coefficients matrix А to DA: DA ið Þ¼ ; j 1 A ið Þ ; j > 0 <sup>0</sup> else � ;
	- The weight between x<sup>i</sup> and x<sup>j</sup> is:

$$\bullet \quad w\_{\vec{\eta}} = \begin{cases} \frac{DA^i \cdot DA^j}{n} & i \neq j \\ 0 & i = j \end{cases}, \\ \text{where } DA^i \text{ denotes the } i \text{-th column vector of DA.} $$

Obviously, the inner product (DA<sup>i</sup> DAj ) between DA<sup>i</sup> and DA<sup>j</sup> is equal to CSS (xi, xj)'s cardinal |CSS(xi, xj)|.

To illustrate the differences between our approaches for weight construction and others also using sparse representation, an example is given as follows. Assume that the coefficient matrix А of a data set with five objects obtained from solution coefficients of Eq. (6) is as the following 5 � 5 matrix:

$$\mathbf{A} = \boldsymbol{\alpha}\_{i,j}^{'} = \begin{bmatrix} 0 & 0.3 & 0.6 & 0.6 & \text{-0.7} \\ 0.4 & 0 & 0.5 & 0.6 & \text{-0.6} \\ 0.4 & 0.4 & 0 & \text{-0.1} & \text{-0.2} \\ \text{-0.6} & \text{-0.3} & 0.2 & 0 & \text{0.7} \\ \text{-0.5} & 0.3 & 0.2 & 0.4 & 0 \end{bmatrix}$$

According to the above introduction of different weight constructions:


#### 3.4. Algorithm description

Αð Þ¼ i; j α

is greater than 0, that is:

162 Recent Applications in Data Clustering

where α<sup>i</sup> ' 0

α 0 <sup>i</sup> � α 0 j

α0 i � � � � <sup>2</sup> � α<sup>0</sup> j � � � � � � 2

The weight between object x<sup>i</sup> and x<sup>j</sup> is defined as the cosine similarity of α<sup>i</sup>

8 >>><

>>>:

wij <sup>¼</sup> max 0; <sup>α</sup>

0 B@

3.3. The relationship between consistent sign set (CSS) and cosine similarity of

denotes the i-th row vector of А.

use the whole solution coefficients.

coefficient vector (COS)

experimental.

that based on CSS.

following:

i,j <sup>¼</sup> <sup>α</sup>i,j <sup>i</sup> 6¼ <sup>j</sup> <sup>0</sup> <sup>i</sup> <sup>¼</sup> <sup>j</sup> :

(13)

(15)

> 0, (14)

' and α<sup>j</sup> ' :

�

A directed edge is placed from object x<sup>i</sup> and x<sup>j</sup> if angle cosine of the two corresponding vectors

0 <sup>i</sup> � α 0 j

0 i ¼ j

1

CA <sup>i</sup> 6¼ <sup>j</sup>

:

1 A ið Þ ; j > 0 0 else

;

�

α0 i � � � � <sup>2</sup> � α<sup>0</sup> j � � � � � � 2

From the above similarity calculation formula, two objects have large similarity in condition that the corresponding solution coefficients of Eq. (6) are much similar, which is expected to

Since proposed proximity based on both CSS and COS are trying to exploit more information from the solution coefficient of sparse representation, the relationship between each other is

1. Both of them asses the weight between two objects according to the similarity between the corresponding coefficient vectors of the two objects. However, the difference is that proximity based on CSS uses the column vectors of the coefficient matrix А while proximity based on COS calculates the similarity between row vectors, which means two understandings of the coefficient matrix. The reason for defining these two approaches like this is just

2. Proximity based on COS is to calculate the similarity of the original coefficient vector, while CSS can be considered as the discretization of the original coefficient vector with threshold zero. Therefore, proximity based on COS can be seen as the generalization of

3. Specifically, another equivalent way to understand proximity based on CSS is as follows:

• Transform the coefficients matrix А to DA: DA ið Þ¼ ; j

• The weight between x<sup>i</sup> and x<sup>j</sup> is:

Algorithm 1 describes the general procedure for spectral clustering of high-dimensional data, using sparse representation. The basic idea is to extract coefficients of sparse representation (Lines 1–4); construct a weight matrix using the coefficients (Line 5); and feed the weight matrix into a spectral clustering algorithm (Line 6) to find the best partitioning efficiently.

Algorithm 1. General procedure for spectral clustering of high-dimensional data.

Input: high-dimensional training data set X = [x1, x2, …, xn] ∈ R<sup>m</sup> � <sup>n</sup> , where x<sup>i</sup> = [xi1, xi2,…, xim]T ∈ R<sup>m</sup> represents the i-th data object; the number of clusters K. Parameter: penalty coefficient λ for Lasso optimization Output: cluster labels corresponding to each data object: c = [c1, c2,…, cn] //standardize the input data for Lasso optimization

```
1 for each data object xi ∈ X do
   // Solve Eq. (6) with Lasso optimization
2 Set Xi = X\xi = [x1, …,xi-1, xi + 1,…, xn];
3 αi
       * arg min λ||αi||1 + || xi - Xαi ||2;
4 end
5 W ConstructWeightMatrix(α);
6 c SpectralClustering(W);
7 return c.
```
The construct weight matrix () sub-routine can exploit any weight matrix construction method, such as those mentioned in Section 4. In particular, we describe the algorithm for computing the two newly proposed weight matrices, one based on the CSS (see Section 4.1) and the other based on COS of sparse coefficient vectors (see Section 4.2).

COS of two vectors of length n is O(n), and there are O(n<sup>2</sup>

Input: Coefficients for sparse representation α

6 if cosine > 0 then wij cosine;

0 i �α 0 j <sup>α</sup><sup>0</sup> k k<sup>i</sup> <sup>2</sup> � <sup>α</sup><sup>0</sup> j 2

whole object set to calculate one element in the weight matrix.

Output: Weight matrix W

3 if j = i then wij 0;

<sup>5</sup> cosine <sup>α</sup>

7 else wij 0;

4. Experimental results

1 for i 1 to n do 2 for j 1 ton do

4 else

8 end 9 end 10 end

data.

needs to be computed. Thus the complexity for COS-based weight matrix construction is O(n<sup>3</sup>

As shown in line 6 of Algorithm 1, after constructing the weight matrix W, we can use the classical spectral clustering algorithm [10] to discover the cluster structure of high-dimensional

The main characteristics of our proposed algorithm include the following: (1) compared to traditional graph construction induced from the Euclidean distance or other measures in the original high-dimensional space, the weight matrix is constructed after transforming the high dimensional data space into another space via sparse representation, which is expected to have better performance resulting from the superiority of compressed sensing [58] for highdimensional data; (2) our graph construction based on consistent sign set or similarity of coefficient vector can simultaneously complete both the graph adjacency and weight matrix, while traditional graph constructions (such as ε-ball neighborhood or k-nearest neighbors) complete the two tasks separately, which are interrelated and should not be separated [19]; (3) rather than existing graph constructions via sparse representation directly and independently applying the solution of l<sup>1</sup> optimization for each object in Eq. (6) to determine a row of the weight matrix, our approach considers the global information from the coefficients of the

In this section, we use experimental results to demonstrate the performance of our proposed

approaches on real-world data sets using several effectiveness evaluations.

Algorithm 3. Construct weight matrix based on similarity of coefficient vector.

) pairs of data objects whose COS

Robust Spectral Clustering via Sparse Representation http://dx.doi.org/10.5772/intechopen.76586

).

165

Algorithm 2 describes the procedure to construct the weight matrix according to the concept of CSS. To find the CSS of each pair of data objects (the two outermost loops), there is the need of checking the sparse coefficients of each remaining object to these two objects, so the time complexity of weight matrix constructions based on CSS is O (n<sup>3</sup> ).

Algorithm 2. Construct weight matrix based on consistent sign set.

```
Input: Coefficients for sparse representation α
Output: Weight matrix W
1 for i 1 to n do
2 for j 1 ton do
3 if j = i then wij 0;
4 else
5 ncss 0;
6 for k 1 to n do
7 if k 6¼ i and k 6¼ j and αk,i > 0 andαk,j > 0 then
8 ncss ncss + 1;
9 end
10 end
11 wij ncss/n;
12 end
13 end
14 end
```
Algorithm 3 describes the procedure to construct the weight matrix according to the COS of the sparse coefficients between each pair of items. The computation complexity for calculating the COS of two vectors of length n is O(n), and there are O(n<sup>2</sup> ) pairs of data objects whose COS needs to be computed. Thus the complexity for COS-based weight matrix construction is O(n<sup>3</sup> ).

#### Algorithm 3. Construct weight matrix based on similarity of coefficient vector.

```
Input: Coefficients for sparse representation α
Output: Weight matrix W
1 for i 1 to n do
2 for j 1 ton do
3 if j = i then wij 0;
4 else
5 cosine α
                       0
                       i �α
                        0
                        j
                   α0 k ki 2
                       � α0
                         j

                           2
6 if cosine > 0 then wij cosine;
7 else wij 0;
8 end
9 end
10 end
```
1 for each data object x<sup>i</sup> ∈ X do

164 Recent Applications in Data Clustering

5 W ConstructWeightMatrix(α); 6 c SpectralClustering(W);

3 α<sup>i</sup>

7 return c.

4 end

// Solve Eq. (6) with Lasso optimization 2 Set X<sup>i</sup> = X\x<sup>i</sup> = [x1, …,xi-1, x<sup>i</sup> + 1,…, xn];

Input: Coefficients for sparse representation α

Output: Weight matrix W

3 if j = i then wij 0;

5 ncss 0;

9 end 10 end

11 wij ncss/n;

6 for k 1 to n do

8 ncss ncss + 1;

1 for i 1 to n do 2 for j 1 ton do

4 else

12 end 13 end 14 end

\* arg min <sup>λ</sup>||αi||1 + || <sup>x</sup><sup>i</sup> - Xα<sup>i</sup> ||2;

based on COS of sparse coefficient vectors (see Section 4.2).

complexity of weight matrix constructions based on CSS is O (n<sup>3</sup>

Algorithm 2. Construct weight matrix based on consistent sign set.

7 if k 6¼ i and k 6¼ j and αk,i > 0 andαk,j > 0 then

The construct weight matrix () sub-routine can exploit any weight matrix construction method, such as those mentioned in Section 4. In particular, we describe the algorithm for computing the two newly proposed weight matrices, one based on the CSS (see Section 4.1) and the other

Algorithm 2 describes the procedure to construct the weight matrix according to the concept of CSS. To find the CSS of each pair of data objects (the two outermost loops), there is the need of checking the sparse coefficients of each remaining object to these two objects, so the time

Algorithm 3 describes the procedure to construct the weight matrix according to the COS of the sparse coefficients between each pair of items. The computation complexity for calculating the

).

As shown in line 6 of Algorithm 1, after constructing the weight matrix W, we can use the classical spectral clustering algorithm [10] to discover the cluster structure of high-dimensional data.

The main characteristics of our proposed algorithm include the following: (1) compared to traditional graph construction induced from the Euclidean distance or other measures in the original high-dimensional space, the weight matrix is constructed after transforming the high dimensional data space into another space via sparse representation, which is expected to have better performance resulting from the superiority of compressed sensing [58] for highdimensional data; (2) our graph construction based on consistent sign set or similarity of coefficient vector can simultaneously complete both the graph adjacency and weight matrix, while traditional graph constructions (such as ε-ball neighborhood or k-nearest neighbors) complete the two tasks separately, which are interrelated and should not be separated [19]; (3) rather than existing graph constructions via sparse representation directly and independently applying the solution of l<sup>1</sup> optimization for each object in Eq. (6) to determine a row of the weight matrix, our approach considers the global information from the coefficients of the whole object set to calculate one element in the weight matrix.
