4. Fusion of local manifold learning methods

So far we have discussed four basic local objects: ∥∇tanfðxÞ∥<sup>2</sup> , <sup>j</sup>σfðxÞj<sup>2</sup> , ∥Htan <sup>f</sup> <sup>ð</sup>xiÞ∥<sup>2</sup> <sup>F</sup>, and <sup>j</sup>κfðxiÞj<sup>2</sup> . From different perspectives, they depict the geometric information of the manifold. We look forward to collect these geometric information together to better reflect the geometric structure of the underlying manifold. Notice that we can estimate these local objects under the local tangent coordinate system according to Eqs. (9), (14), (21), and (28), respectively. Taking stock of the structure of these equations, it is not hard to discover that we can fuse these local objects together under our proposed framework. Assume that there are M different local manifold learning algorithms, we can define the fused local object as follows:

fðxij

should respect the local geometry determined by the ui

coordinate fðxij

140 Manifolds - Current Research Areas

local object:

where f

for a fixed f

and therefore,

Let <sup>W</sup><sup>i</sup> ¼ ðI−ðU<sup>i</sup>

where fðxiÞ is the mean of fðxij

<sup>i</sup> ¼ ½ <sup>f</sup>ðxi<sup>1</sup> <sup>Þ</sup>;…; <sup>f</sup>ðxik Þ�<sup>T</sup>, <sup>U</sup><sup>i</sup> ¼ ½ui

<sup>i</sup> is given by:

Þ † Ui Þ <sup>T</sup>ðI<sup>−</sup> <sup>1</sup> <sup>k</sup> ee<sup>T</sup><sup>Þ</sup>

<sup>j</sup>κfðxiÞj<sup>2</sup> ¼ jð<sup>f</sup>

tion Li to minimize the following global functional:

Þ ¼ <sup>f</sup>ðxiÞþð∇tanfðxiÞÞ<sup>T</sup>

From the above equation, we can discover that there are some relations between the global

<sup>Þ</sup>≈fðxiÞ þ Liui

should align the local coordinate with the global coordinate, and we can define the following

ones. Naturally, we should seek to find the optimal mapping f and a local affine transforma-

Obviously, the optimal affine transformation Li that minimizes the local reconstruction error

<sup>j</sup>κfðxÞj<sup>2</sup>

<sup>T</sup>, the local object <sup>κ</sup>fðxi<sup>Þ</sup> can be estimated as:

From different perspectives, they depict the geometric information of the manifold. We look forward to collect these geometric information together to better reflect the geometric structure

, <sup>j</sup>σfðxÞj<sup>2</sup>

, ∥Htan

<sup>f</sup> <sup>ð</sup>xiÞ∥<sup>2</sup>

ð M

i Þ T− 1 k ðf i Þ T

represents the local geometry. The LTSA algorithm requires the global coordinates fðxij

fðxij

<sup>j</sup>κfðxiÞj<sup>2</sup> ¼ jð<sup>f</sup>

<sup>1</sup>; ui <sup>2</sup>;…; ui

KðfÞ ¼

Li ¼ ðf i Þ T � I− 1 k ee<sup>T</sup> � ðUi Þ

> i Þ T � I− 1 k eeT � ðI−ðU<sup>i</sup> Þ † Ui Þj2

<sup>j</sup>κfðxiÞj<sup>2</sup> ¼ jð<sup>f</sup>

i Þ T � I− 1 k eeT � ðI−ðU<sup>i</sup> Þ † Ui Þj<sup>2</sup> ¼ ð<sup>f</sup> i Þ <sup>T</sup>ðW<sup>i</sup> Þ <sup>T</sup>ðW<sup>i</sup> Þðf i

4. Fusion of local manifold learning methods

So far we have discussed four basic local objects: ∥∇tanfðxÞ∥<sup>2</sup>

<sup>Þ</sup> in the low-dimensional feature space and the local coordinate ui

ui

j :

j

ee<sup>T</sup>−LiU<sup>i</sup> j 2

Þ, j ¼ 1;…;k. Inspired by LTSA, the affine transformation Li

<sup>j</sup> <sup>þ</sup> <sup>O</sup>ð∥u<sup>i</sup> j ∥2

Þ (22)

; (23)

<sup>k</sup>�, and e is a k-dimensional column vector of all

dx (25)

† (26)

; (27)

Þ (28)

<sup>F</sup>, and <sup>j</sup>κfðxiÞj<sup>2</sup>

.

; (24)

<sup>j</sup> which

Þ that

$$LO\_f(\mathbf{x}) = \sum\_{j=1}^{M} c\_j LO\_j(\mathbf{x}) \tag{29}$$

where {cj} M <sup>j</sup>¼<sup>1</sup> are the nonnegative balance parameters, {LOjðxÞ} M <sup>j</sup>¼<sup>1</sup> are the local objects, such as ∥∇tanfðxÞ∥<sup>2</sup> , <sup>j</sup>σfðxÞj<sup>2</sup> , ∥Htan <sup>f</sup> <sup>ð</sup>xiÞ∥<sup>2</sup> <sup>F</sup>, and <sup>j</sup>κfðxiÞj<sup>2</sup> , from different algorithms. It is worth to note that the other local manifold learning algorithms can also be reformulated to incorporate into our unified framework.

We employ the truncation function from differential manifold to connect the local objects with their corresponding global functional such that we can obtain a consistent alignment of the local objects to discover a single global coordinate system of lower dimensionality. The truncation function is a crucial tool in differential geometry to build relationships between global and local properties of the manifold. Assume that U and V are two nonempty subsets of a smooth manifold M, where V is compact and V∈U ( V is the closure of V). Accordingly, the truncation function [15] can be defined as a smooth function s : M ! ℝ such that:

$$s(p) = \begin{cases} 1, & p \in V \\ 0, & p \notin U. \end{cases} \tag{30}$$

The truncation function s can be discretely approximated by the 0-1 selection matrix S<sup>i</sup> ∈ℝ<sup>N</sup> · <sup>k</sup> . An entry of Si is defined as:

$$(S^i)\_{p\eta} = \begin{cases} 1, & p = N\_i \{q\} \\ 0, & p \not\sim N\_i \{q\}. \end{cases} \tag{31}$$

where Ni ¼ {i1;…;ik} denotes the set of indices for the k-nearest neighborhoods of data point xi. Let <sup>f</sup> ¼ ½fðx1Þ;…;fðxNÞ�∈ℝ<sup>N</sup> be a function defined on the whole data set sampled from the global manifold. Thus, the local mapping f <sup>i</sup> ¼ ½fðxi 1Þ;…;fðx<sup>i</sup> <sup>k</sup>Þ�∈ℝ<sup>k</sup> can be expressible by f <sup>i</sup> ¼ ðSi Þ Tf . With the help of the selection matrix, we can discretely approximate the global functional GðfÞ as follows:

$$\begin{split} \mathcal{G}(f) &= \int\_{\mathcal{M}} LO\_f(\mathbf{x}) \, d\mathbf{x} = \frac{1}{N} \sum\_{i=1}^{N} LO\_f(\mathbf{x}\_i) \\ &= \frac{1}{N} \sum\_{i=1}^{N} \left( f^i \right)^T \left( \sum\_{j=1}^{M} c\_j L\_j^i \right) f^i = f^T \left( \sum\_{j=1}^{M} c\_j P^j \right) f \end{split} \tag{32}$$

where {Li j } M <sup>j</sup>¼<sup>1</sup> are the local matrices such as <sup>ð</sup>Gi Þ TGi , <sup>ð</sup>W<sup>i</sup> Þ TWi , <sup>ð</sup>H<sup>i</sup> Þ THi , and <sup>ð</sup>W<sup>i</sup> Þ TWi which are defined in Eqs. (9), (14), (21), and (28). <sup>P</sup><sup>j</sup> <sup>¼</sup> <sup>1</sup> <sup>N</sup> <sup>∑</sup><sup>N</sup> <sup>i</sup>¼<sup>1</sup>Si Li j ðSi Þ <sup>T</sup> is the alignment matrix of the j-th local manifold learning method. The global embedding coordinates <sup>Y</sup> ¼ ½y1;y2;…;yN�∈ℝ<sup>d</sup> · <sup>N</sup> can be obtained by minimizing the functional GðfÞ. Let y ¼ f ¼ ½fðx1Þ;…;fðxNÞ� be a row vector of Y. It is not hard to show that the global embedding coordinates and the nonnegative weights c ¼ ½c1;…;cM� can be obtained by minimizing the following objective function:

$$\underset{\mathbf{c}\_{\mathbf{y}\_c}}{\operatorname{argmin}} \sum\_{j=1}^{M} c\_j^r Tr(\mathbf{Y} P^j \mathbf{Y}^T) \text{ s.t.}\\\mathbf{Y} \mathbf{Y}^T = I; \sum\_{j=1}^{M} c\_j = 1, \,\mathbf{c}\_{\mathbf{y}} \succeq \mathbf{0}. \tag{33}$$

where the power parameter r > 1 is set to avoid the phenomenon that the solution to c is cj ¼ 1 corresponding to the minimum TrðYPj <sup>Y</sup><sup>T</sup><sup>Þ</sup> over different local methods and ck <sup>¼</sup> <sup>0</sup>ðk≠j<sup>Þ</sup> otherwise, since our aim is to utilize the complementary geometric information from different manifold learning methods.

We propose to solve the objective function [Eq. (33)] by employing the alternating optimization [16] method, which iteratively updates Y and c in an alternating fashion. First, we fix c to update Y. The optimization problem in Eq. (33) is equivalent to:

$$\underset{Y}{\text{argmin}} \, Tr(\mathbf{Y}PY^T) \text{ s.t. } \mathbf{Y}Y^T = I \tag{34}$$

where <sup>P</sup> <sup>¼</sup> <sup>∑</sup><sup>M</sup> <sup>j</sup>¼<sup>1</sup>cr jPj . When c is fixed, we can solve the optimization problem [Eq. (34)] and obtain the global optimal solution Y as the second to ðd þ 1Þ st smallest eigenvectors of the matrix P. Second, we fix Y to update c. While Y is fixed, we can minimize the objective function [Eq. (33)] analytically through utilizing a Lagrange multiplier to enforce the constraint that ∑<sup>M</sup> <sup>j</sup>¼<sup>1</sup>cj <sup>¼</sup> 1. And the global optimal <sup>c</sup> can be obtained as:

$$\mathcal{L}\_{j} = \frac{\left(1/Tr(\mathbf{Y}\mathbf{P}^{j}\mathbf{Y}^{T})\right)^{1/(r-1)}}{\sum\_{j=1}^{M} \left(1/Tr(\mathbf{Y}\mathbf{P}^{j}\mathbf{Y}^{T})\right)^{1/(r-1)}}, j = \{1,...,M\} \tag{35}$$
