**4. Collinearity criterion function**

Minimizing the collinearity criterion function is used to extract collinear patterns from large, multidimensional data sets *C* (1) [7]. Linear models of multivariate interactions can be formulated on the basis of representative collinear patterns [9].

The collinearity penalty functions φj(**w**) are determined by individual feature vectors **x**<sup>j</sup> = [xj,1,...,xj,n] <sup>T</sup> in the following manner [9]:

$$\begin{aligned} \left(\forall \mathbf{x}\_{\circ} \in \mathbf{C} \ (1)\right) \\ \mathbf{q}\_{\circ}(\mathbf{w}) = \left|\mathbf{1} - \mathbf{x}\_{\circ}^{\mathrm{T}} \mathbf{w}\right| = \begin{array}{c} \mathbf{1} - \mathbf{x}\_{\circ}^{\mathrm{T}} \mathbf{w} \quad \text{if} \quad \mathbf{x}\_{\circ}^{\mathrm{T}} \mathbf{w} \le \mathbf{1} \\ \mathbf{x}\_{\circ}^{\mathrm{T}} \mathbf{w} - \mathbf{1} \quad \text{if} \quad \mathbf{x}\_{\circ}^{\mathrm{T}} \mathbf{w} > \mathbf{1} \end{array} \end{aligned} \tag{36}$$

The penalty functions φj(**w**) (36) can be related to the following dual hyperplanes *h*j <sup>1</sup> in the parameter (weight) space *R*<sup>n</sup> (**w** ∈ *R*<sup>n</sup> ):

$$\left(\forall j=1,\ldots,m\right)h\_{\rangle}^{-1} = \left\{\mathbf{w}:\mathbf{x}\_{\rangle}^{\mathrm{T}}\mathbf{w} = \mathbf{1}\right\}\tag{37}$$

The *CPL* penalty φ<sup>j</sup> (**w**) (36) is equal to zero (φ<sup>j</sup> c (**w**) = 0) in the point **w** = [w1,..., wn] <sup>T</sup> if and only if the point **w** is located on the dual hyperplane *h*<sup>j</sup> <sup>1</sup> (37).

The collinearity criterion function Φk(**w**) is defined as the weighted sum of the penalty functions φj(**w**) (36) determined by feature vectors **x**<sup>j</sup> forming the data subset *C*<sup>k</sup> (*C*<sup>k</sup> ⊂ *C* (1)):

$$\Phi\_{\mathbf{k}}(\mathbf{w}) = \Sigma\_{\mathbf{j}} \mathfrak{k}\_{\mathbf{j}} \mathfrak{q}\_{\mathbf{j}}(\mathbf{w}) \tag{38}$$

where the sum takes into account only the indices *J* of the set *J*<sup>k</sup> = {*j*: **x**<sup>j</sup> ∈ *C*k}, and the positive parameters β<sup>j</sup> (β<sup>j</sup> > 0) in the function Φk(**w**) (38) can be treated as the *prices* of particular feature vectors **x**<sup>j</sup> . The standard choice of the parameters β<sup>j</sup> values is one ((∀*j* ∈ *J*k) β<sup>j</sup> = 1.0).

The collinearity criterion function Φk(**w**) (38) is convex and piecewise-linear (*CPL*) as the sum of this type of penalty functions φj(**w**) (36) [9]. The vector **w**k\* determines the minimum value Φk(**w**k\*) of the criterion function Φk(**w**) (38):

$$(\exists \mathbf{w}\_{\mathbf{k}} \, ^\*) \, (\forall \mathbf{w}) \, \Phi\_{\mathbf{k}}(\mathbf{w}) \ge \Phi\_{\mathbf{k}}(\mathbf{w}\_{\mathbf{k}} \, ^\*) \ge \mathbf{0} \tag{39}$$

*Definition* 3: The data subset *C*<sup>k</sup> (*C*<sup>k</sup> ⊂ *C* (1)) is *collinear* when all feature vectors **x**<sup>j</sup> from this subset are located on some hyperplane *H*(**w***,* θ)={**x**: **w**<sup>T</sup> **x** = θ} with θ ¼6 0.

*Theorem* 3: The minimum value Φ<sup>k</sup> p (**v**k\*) (39) of the collinearity criterion function Φk(**w**) (38) defined on the feature vectors **x**<sup>j</sup> constituting a data subset *C*<sup>k</sup> (*C*<sup>k</sup> ⊂ *C* (1)) is equal to zero (Φ<sup>k</sup> p (**v**k\*) = 0) when this subset *C*<sup>k</sup> is collinear (*Def.* 3) [9].

*Computing on Vertices in Data Mining DOI: http://dx.doi.org/10.5772/intechopen.99315*

Different collinear subsets *C*<sup>k</sup> can be extracted from data set *C* (1) with a large number *m* of elements **x**<sup>j</sup> by minimizing the collinearity criterion function Φ<sup>k</sup> p (**w**) (38) [9].

The minimum value Φ<sup>k</sup> p (**v**k\*) (39) of the collinearity criterion function Φk(**w**) (38) can be reduced to zero by omitting some feature vectors **x**<sup>j</sup> from the data subset *C*<sup>k</sup> (*C*<sup>k</sup> ⊂ *C* (1)). If the minimum value Φk(**w**k\*) (39) is greater than zero (Φk(**w**k\*) > 0) then we can select feature vectors **x**<sup>j</sup> (*j* ∈ *J*k(**w**k\*)) with the penalty φj(**w**k\*) (36) greater than zero:

$$(\forall j \in f\_{\mathbf{k}}(\mathbf{w\_k}^\* \ ^\*)) \ \boldsymbol{\varrho}\_{\mathbf{j}}(\mathbf{w\_k} \ ^\*) = |\mathbf{1} - \mathbf{x\_j}^T \ \mathbf{w\_k} \ ^\*| > 0 \tag{40}$$

Omitting one feature vector **x**<sup>j</sup> <sup>0</sup> (*j* 0 ∈ *J*k(**w**k\*)) with the above property results in the following reduction of the minimum value Φ<sup>k</sup> p (**v**k\*) (39);

$$\Phi\_{\mathbf{k}'} (\mathbf{w}\_{\mathbf{k}'} \, ^\*) \le \Phi\_{\mathbf{k}} (\mathbf{w}\_{\mathbf{k}} \, ^\*) - \Phi\_{\mathbf{j}'} (\mathbf{w}\_{\mathbf{k}} \, ^\*) \tag{41}$$

where Φk0(**w**k0\*) is the minimum value (39) of the collinearity criterion function Φk0(**w**) (38) defined on feature vectors **x**<sup>j</sup> constituting the data subset *C*<sup>k</sup> reduced by the vector **x**<sup>j</sup> 0.

The regularized criterion function Ψk(**w**) is defined as the sum of the collinearity criterion function Φk(**w**) (38) and some additional *CPL* penalty functions φ<sup>j</sup> 0 (**w**) [7]:

$$\Psi\_{\mathbf{k}}(\mathbf{w}) = \Phi\_{\mathbf{k}}(\mathbf{w}) + \lambda \,\Sigma\_{\mathbf{i}} \,\chi\_{\mathbf{i}}(\mathbf{w}) = \Sigma\_{\mathbf{j}} \,\mathfrak{k}\_{\mathbf{j}} \,\mathfrak{q}\_{\mathbf{j}}(\mathbf{w}) + \lambda \,\Sigma\_{\mathbf{i}} \,\chi\_{\mathbf{i}} \,\mathfrak{q}\_{\mathbf{i}}^{0}(\mathbf{w}) \tag{42}$$

where λ ≥ 0 is the *cost level*. The standard values of the cost parameters γ<sup>i</sup> are equal to one ((∀*i* ∈ {1, … ,*n*}) γ<sup>i</sup> = 1). The additional *CPL* penalty functions φ<sup>j</sup> 0 (**w**) are defined below [7]:

$$(\forall i = 1, \ldots, n) \tag{43}$$

$$|\chi\_{\mathbf{i}}(\mathbf{w}) = |\ e\_{\mathbf{i}}^{\mathrm{T}} \mathbf{w}| = \begin{array}{c} -\mathbf{w}\_{\mathbf{j}} & \circlearrowright \\ \mathbf{w}\_{\mathbf{j}} & \circlearrowright \end{array} \le \mathbf{0}$$

The functions φ<sup>j</sup> 0 (**w**) (43) are related to the following dual hyperplanes *h*<sup>j</sup> <sup>0</sup> in the parameter (*weight*) space *R*<sup>n</sup> (**w** ∈ *R*<sup>n</sup> ):

$$\left(\forall i=1,\ldots,n\right)h\_{\rangle}^{0} = \left\{\mathbf{w}:\mathbf{e}\_{\rangle}^{\mathrm{T}}\mathbf{w} = \mathbf{0}\right\} = \left\{\mathbf{w}:\mathbf{w}\_{\rangle} = \mathbf{0}\right\}\tag{44}$$

The *CPL* penalty function φ<sup>j</sup> 0 (**w**) (43) is equal to zero (φ<sup>j</sup> 0 (**w**) = 0) in the point **w** = [w1,..., wn] <sup>T</sup> if and only if this point is located on the dual hyperplane *h*j <sup>0</sup> (44).
