**6. Basis exchange algorithm**

The collinearity criterion function Φ(**w**) (38), like other convex and piecewise linear (*CPL*) criterion functions, can be minimized using the basis exchange algorithm [8]. The basis exchange algorithm aimed at minimization of the collinearity criterion function Φ(**w**) (38) is described below.

According to the basis exchange algorithm, the optimal vertex **w**k\*, which constitutes the minimum value Φk(**w**k\*) (39) of the collinearity function Φk(**w**) (38), is achieved after a finite number *L* of the steps *l* as a result of guided movement between selected vertices **w**<sup>k</sup> (50) [8]:

$$\mathbf{w}\_0 \to \mathbf{w}\_1 \to \dots \to \mathbf{w}\_L \tag{59}$$

The sequence of vertices **w**<sup>k</sup> (59) is related by (47) to the following sequence of the inverse matrices *B*<sup>k</sup> �<sup>1</sup> (49):

$$\mathbf{B}\_0^{-1} \to \mathbf{B}\_1^{-1} \to \dots \to \mathbf{B}\_L^{-1} \tag{60}$$

The sequence of vertices **w**k(*l*) (59) typically starts at the vertex **w**<sup>0</sup> = [0,..., 0]T related to the identity matrix *B*<sup>0</sup> = *I*<sup>n</sup> = [**e**1,..., **e**n] <sup>T</sup> of the dimension *n* x *n* [7]. The final vertex **w***<sup>L</sup>* (59) should assure the minimum value of the collinearity criterion function Φ(**w**) (38):

$$(\forall \mathbf{w}) \,\,\Phi(\mathbf{w}) \ge \Phi(\mathbf{w}\_L) \ge \mathbf{0} \tag{61}$$

If the criterion function Φ(**w**) (38) is defined on *m* (*m* ≤ *n*) linearly independent vectors **x**<sup>j</sup> (**x**<sup>j</sup> ∈ *C* (1)) then the value Φ(**w***L*) of the collinearity criterion function Φ(**w**) (38) at the final vertex **w***<sup>L</sup>* (59) becomes zero (Φ(**w***L*) = 0) [8]. The rank *rL* (*Def.* 4) of the final vertex **w**<sup>L</sup> (59) can be equal to the number *m* of feature vectors **x**<sup>j</sup> (*rL* = *m*) or it can be less than *m* (*rL* < *m*). The rank *rL* of the final vertex **w***<sup>L</sup>* (59) is less than *m* (*rL* < *m*) if the final vertex **w***<sup>L</sup>* is degenerate [7].

Consider the reversible matrix *B*<sup>k</sup> = [**x**1,..., **x**k, **e**i(k + 1),..., **e**i(*<sup>n</sup>*)] <sup>T</sup> (48), which determines the vertex **w**<sup>k</sup> (50) and the value Φk(**w**k) of the criterion function Φk(**w**) (38) in the *k*-th step. In the step (*l* + 1), one of the unit vectors **e**<sup>i</sup> in the matrix **B**<sup>k</sup> (48) is replaced by the feature vector **x**k+1 and the matrix **B**k+1 = [**x**1,..., **x**k, **x**k+1, **e**i(k + 2),..., **e**i(*<sup>n</sup>*)] <sup>T</sup> appears. The unit vector **e**i(k + 1) leaving matrix *B*<sup>k</sup> (48) is indicated

by an *exit criterion* based on the gradient of the collinearity criterion function Φ(**w**) (38) [7]. The exit criterion allows to determine the exit edge **r**k+1 (49) of the greatest descent of the collinearity criterion function Φ(**w**) (38). As a result of replacing the unit vector **e**i(k + 1) with the feature vector **x**k + 1, the value Φ(**w**k) of the collinearity function Φ(**w**) (38) decreases (41):

$$\Phi(\mathbf{w}\_{k+1}) \le \Phi(\mathbf{w}\_k) - \varrho\_{k+1}(\mathbf{w}\_k) \tag{62}$$

After a finite number *L* (*L* ≤ *m*) of the steps *k*, the collinearity function Φ(**w**) (38) reaches its minimum (61) at the final vertex **w***<sup>L</sup>* (59).

The sequence (60) of the inverse matrices *B*<sup>k</sup> �<sup>1</sup> is obtained in a multi-step process of minimizing the function Φ(**w**) (38). During the *k*-th step, the matrix *Bk*-1 = [**x**1, … , **x**k-1, **e**i(k), … ., **e**i(n)] <sup>T</sup> (12) is transformed into the matrix *B*<sup>k</sup> by replacing the unit vector **e**i(k) with the feature vector **x**k:

$$(\forall k \in \{1, \ldots, L\}) \; \mathbf{B\_{k-1} \to \mathbf{B\_k}} \tag{63}$$

According to the vector Gauss-Jordan transformation, replacing the unit vector **e**<sup>i</sup> (k) with the feature vector **x**<sup>k</sup> during the *k* - th stage results in the following modifications of the co + lumns **r**i(*k*) of the inverse matrix *B<sup>l</sup>* �<sup>1</sup> = [**x**1, … , **x***l*, **e***i*(*<sup>l</sup>*+1), … , **e***i*(*<sup>n</sup>*)] T (49) [6]:

$$\mathbf{r}\_{\mathbf{i}(l+1)}(l+1) = \left(\mathbf{1}/\mathbf{r}\_{\mathbf{i}(l+1)}(l)^{\mathbf{T}}\mathbf{x}\_{l+1}\right)\mathbf{r}\_{\mathbf{i}(l+1)}(l) \tag{64}$$

*and*

$$\begin{aligned} \mathbf{r}(l\mathbf{\forall}\neq\mathbf{i}(l+1))\ \mathbf{r}\_{\mathbf{i}}(l+1) &= \mathbf{r}\_{\mathbf{i}}(l) - \left(\mathbf{r}\_{\mathbf{i}}(l)^{\mathbf{T}}\mathbf{x}\_{l+1}\right)\ \mathbf{r}\_{\mathbf{i}(l)}(l+1) = \\ &= \mathbf{r}\_{\mathbf{i}}(l) - \left(\mathbf{r}\_{\mathbf{i}}(l)^{\mathbf{T}}\mathbf{x}\_{\mathbf{j}(l+1)} / \mathbf{r}\_{\mathbf{i}(l)}(l)^{\mathbf{T}}\mathbf{x}\_{l+1}\right)\ \mathbf{r}\_{\mathbf{i}(l)}(l) \end{aligned}$$

where *i*(*l*+1) is the index of the unit vector **e**i(*l*+1) leaving the basis *B<sup>l</sup>* = [**x**1,..., **x***l*, **e**i(*<sup>l</sup>*+1),..., **e**i(*<sup>n</sup>*)] <sup>T</sup> during the *l*-th stage.

*Remark* 2: The vector Gauss-Jordan transformation (64) resulting from the replacing of the unit vector **e**i(k) with the feature vector **x**<sup>k</sup> in the basis *B*k-1 = [**x**1,..., **x**k-1, **e**i(k),..., **e**i(*<sup>n</sup>*)] <sup>T</sup> cannot be executed when the below *collinearity condition* is met [7]:

$$\left(\mathbf{r}\_{i(k)}(k)\right)^{\mathrm{T}}\mathbf{x}\_{k} = \mathbf{0} \tag{65}$$

The collinearity condition (65) causes a division by zero in Eq. (64).

Let the symbol **r***l*[*k*] denote the *l*-th column **r***l*(*k*) = [r*l*,1(*k*),..., r*l*,n(*k*)]<sup>T</sup> of the inverse matrix *B*<sup>k</sup> �<sup>1</sup> = [**r**1(*k*), … , **r**k-1(*k*), **r**k(*k*), … ., **r***n*(*k*)] (49) after the reduction of the last *n-k* components r*l*,i(*k*):

$$\mathbf{r}\_l[k] = \begin{bmatrix} \mathbf{r}\_{l,1}(k), \dots, \mathbf{r}\_{l,k}(k) \end{bmatrix}^T \tag{66}$$

Similarly, the symbol **x**<sup>j</sup> [*k*] = [xj,1,...,xj,k] <sup>T</sup> means the reduced vector obtained from the feature vector **x**<sup>j</sup> = [xj,1,...,xj,n] <sup>T</sup> after he reducing of the last *n* - *k* components xj,i:

$$\mathbf{x}\_{\mathbf{i}}(\forall j \in \{1, \ldots, m\}, \mathbf{x}\_{\mathbf{j}}[k] = \begin{bmatrix} \mathbf{x}\_{\mathbf{j},1}, \ldots, \mathbf{x}\_{\mathbf{j},k} \end{bmatrix}^{\mathrm{T}} \tag{67}$$

*Lemma* 3: The collinearity condition (65) appears during the *k*-th step when the reduced vector **x**k[*k*] (66) is a linear combination of the basis reduced vectors **x**j[*k*] (67) with *j* < *k*:

$$\mathbf{x}\_{\mathbf{k}}[k] = \alpha\_1 \mathbf{x}\_1[k] + \dots + \alpha\_{\mathbf{k}-1} \mathbf{x}\_{\mathbf{k}-1}[k] \tag{68}$$

where (∀*i* ∈ {1, … , *k* - 1}) α<sup>i</sup> ∈ *R*<sup>1</sup> .

The proof of this lemma results directly from the collinearity condition (65) [7].
