4. Related fuzzy clustering algorithms

#### 4.1. Conventional FCM

The fuzzy C-means (FCM) algorithm seeks to minimize the following objective function [7].

$$J\_{\rm FCM} = \sum\_{i=1}^{C} \sum\_{n=1}^{N} \mu\_{in}^{m} d\_{in} \tag{4}$$

from the original one by <sup>X</sup> <sup>¼</sup> <sup>w</sup>ð Þ<sup>x</sup> ∗∗X, where \*\* means two-dimensional convolution. The weights wð Þ<sup>x</sup> can be equal or not provided that its centerweight is zero and are summed to unity. From (8), it is clear that the LDFCM aims at minimizing the standard FCM objective function plus another weighted modified FCM function acting as a regularization function. In this regularization FCM function, the distances are generated from the locally-smoothed image data instead of the original image data. Therefore, this correlates the clustering pixel xn with its immediate spatial neighboring pixels which biases the algorithm to provide clustered images with piecewise homogenous regions. The membership uin and the cluster-center vi functions of

Incorporating Local Data and KL Membership Divergence into Hard C-Means Clustering for Fuzzy and Noise-Robust…

uin <sup>¼</sup> <sup>1</sup> P<sup>C</sup> j¼1

P<sup>N</sup>

bership uin and the cluster-center vi functions which indeed can handle additive noise.

vi ¼

presented in [13]. The objective function of the SFCM algorithm is given by

f in ¼

min P

JSFCM <sup>¼</sup> <sup>X</sup> C

dinþα din djnþ<sup>α</sup> djn � � <sup>1</sup>

<sup>n</sup>¼<sup>1</sup> uinð Þ xn <sup>þ</sup> <sup>α</sup>xn

<sup>n</sup>¼<sup>1</sup> uin

ð Þ <sup>1</sup> <sup>þ</sup> <sup>α</sup> <sup>P</sup><sup>N</sup>

It is obvious from (9) and (10) that when α ¼ 0, the membership uin and the cluster-center vi become the ones provided by the standard FCM algorithm in (6) and (7). The advantage of the LDFCM method arises from involving the locally-smoothed data αxn in computing the mem-

An approach to incorporating local spatial data information into the standard FCM has been

X N

inDin, (11)

http://dx.doi.org/10.5772/intechopen.74514

Din ¼ ð Þ 1 � λ dinf in þ λdin (12)

n o (13)

n¼1 um

i¼1

where Din is a modified or weighted distance between the nth pixel and the ith cluster-center. This modified distance is computed from the original or standard distance din ¼ k k xn � vi

where λ ∈½ � 0; 1 is an experimentally selected weight, and f in is a spatial or local data function

P

It is obvious from (12) that with λ ¼ 1, the SFCM clustering method reduces to the standard FCM method. The spatial data function f in is dependent on the original distances of the set of pixels Nn in the immediate neighborhood of the nth pixel. If all pixels in the neighbor set do

<sup>k</sup><sup>∈</sup> Nn dik

<sup>k</sup><sup>∈</sup> Nn dik; i ¼ 1; 2; ::; C

ð Þ m�1

(9)

39

(10)

2 as

the LDFCM method are given by [10–12].

4.3. Spatial-based fuzzy C-means (SFCM)

follows

given by [13].

It is obvious that the difference between the FCM algorithm and HCM one is the incorporation of the exponent parameter m, called the fuzzification parameter, and if it is settled to 1, the FCM algorithm reduces to the HCM one. Thus, due to this exponent m, the membership of the nth pixel to the ith cluster, uin, can take on an infinite set of values from 0 to 1. Thus each nth pixel may belong to all clusters with equal membership values of 1=C which in this case we obtain too fuzzy membership function. Then the exponent parameter 1 < m is incorporated to control the degrees of fuzzification; the bigger the m, the more the fuzzification. Finally, the fuzzy membership uin should satisfy [7].

$$\mu\_{\dot{m}} \in \mathcal{U} = \left\{ \mu\_{\dot{m}} \in [0, 1]; \sum\_{i=1}^{C} \mu\_{\dot{m}} = 1 \forall n; 0 < \sum\_{n=1}^{N} \mu\_{\dot{m}} < N \forall \dot{i} \right\}, \tag{5}$$

The membership uin and the cluster-center vi that minimize the FCM function in (4), subject to P<sup>C</sup> <sup>i</sup>¼<sup>1</sup> uin <sup>¼</sup> <sup>1</sup>∀<sup>n</sup> are given by [7].

$$\mu\_{\rm int} = \frac{1}{\sum\_{j=1}^{C} \binom{d\_{\rm in}}{d\_{\rm in}}^{\frac{1}{(m-1)}}} \tag{6}$$

$$\upsilon\_{i} = \frac{\sum\_{n=1}^{N} \mu\_{in} \mathbf{x}\_{n}}{\sum\_{n=1}^{N} \mu\_{in}} \tag{7}$$

#### 4.2. Local spatial data-based FCM (LDFCM)

The neighboring pixels of an image are highly correlated and are thus highly expected to belong to the same cluster or object. To get benefit from this spatial data information, the standard FCM objective function in (4) has been modified by adding a weighted regularization function dependent on local image data information [10–12]. That is, the LDFCM objective function is given by

$$J\_{\rm LDFCM} = J\_{\rm FCM} + \alpha \sum\_{i=1}^{\mathcal{C}} \sum\_{n=1}^{N} u\_{in}^{m} \overline{d\_{in}} \tag{8}$$

where α is a weight to be experimentally selected by the user, m is a fuzzification parameter, din ¼ k k xn � vi 2 , xn ∈ X is the nth pixel of the locally-smoothed image, X, obtained in advance from the original one by <sup>X</sup> <sup>¼</sup> <sup>w</sup>ð Þ<sup>x</sup> ∗∗X, where \*\* means two-dimensional convolution. The weights wð Þ<sup>x</sup> can be equal or not provided that its centerweight is zero and are summed to unity. From (8), it is clear that the LDFCM aims at minimizing the standard FCM objective function plus another weighted modified FCM function acting as a regularization function. In this regularization FCM function, the distances are generated from the locally-smoothed image data instead of the original image data. Therefore, this correlates the clustering pixel xn with its immediate spatial neighboring pixels which biases the algorithm to provide clustered images with piecewise homogenous regions. The membership uin and the cluster-center vi functions of the LDFCM method are given by [10–12].

$$u\_{in} = \frac{1}{\sum\_{j=1}^{C} \left(\frac{d\_{in} + \alpha \,\overline{d}\_{in}}{d\_{jn} + \alpha \,d\_{jn}}\right)^{\frac{1}{(m-1)}}} \tag{9}$$

$$w\_i = \frac{\sum\_{n=1}^{N} u\_{in} (\mathbf{x}\_n + \alpha \overline{\mathbf{x}}\_n)}{(1 + \alpha) \sum\_{n=1}^{N} u\_{in}} \tag{10}$$

It is obvious from (9) and (10) that when α ¼ 0, the membership uin and the cluster-center vi become the ones provided by the standard FCM algorithm in (6) and (7). The advantage of the LDFCM method arises from involving the locally-smoothed data αxn in computing the membership uin and the cluster-center vi functions which indeed can handle additive noise.

#### 4.3. Spatial-based fuzzy C-means (SFCM)

such as hybridization or mixing of data which is important in data clustering and decision making. The algorithm is implemented by an iterative procedure as summarized in Table 1.

The fuzzy C-means (FCM) algorithm seeks to minimize the following objective function [7].

i¼1 X<sup>N</sup> <sup>n</sup>¼<sup>1</sup> <sup>u</sup><sup>m</sup>

It is obvious that the difference between the FCM algorithm and HCM one is the incorporation of the exponent parameter m, called the fuzzification parameter, and if it is settled to 1, the FCM algorithm reduces to the HCM one. Thus, due to this exponent m, the membership of the nth pixel to the ith cluster, uin, can take on an infinite set of values from 0 to 1. Thus each nth pixel may belong to all clusters with equal membership values of 1=C which in this case we obtain too fuzzy membership function. Then the exponent parameter 1 < m is incorporated to control the degrees of fuzzification; the bigger the m, the more the fuzzification. Finally, the

<sup>i</sup>¼<sup>1</sup> uin <sup>¼</sup> <sup>1</sup>∀n; <sup>0</sup> <sup>&</sup>lt; <sup>X</sup><sup>N</sup>

n o

The membership uin and the cluster-center vi that minimize the FCM function in (4), subject to

uin <sup>¼</sup> <sup>1</sup> P<sup>C</sup> j¼1 din djn � � <sup>1</sup> ð Þ m�1

> P<sup>N</sup> <sup>n</sup>¼<sup>1</sup> uinxn <sup>P</sup><sup>N</sup> <sup>n</sup>¼<sup>1</sup> uin

The neighboring pixels of an image are highly correlated and are thus highly expected to belong to the same cluster or object. To get benefit from this spatial data information, the standard FCM objective function in (4) has been modified by adding a weighted regularization function dependent on local image data information [10–12]. That is, the LDFCM objective

where α is a weight to be experimentally selected by the user, m is a fuzzification parameter,

X C

X N

n¼1 um

, xn ∈ X is the nth pixel of the locally-smoothed image, X, obtained in advance

i¼1

vi ¼

JLDFCM ¼ JFCM þ α

indin (4)

<sup>n</sup>¼<sup>1</sup> uin <sup>&</sup>lt; <sup>N</sup>∀<sup>i</sup>

indin (8)

, (5)

(6)

(7)

JFCM <sup>¼</sup> <sup>X</sup><sup>C</sup>

X<sup>C</sup>

4. Related fuzzy clustering algorithms

fuzzy membership uin should satisfy [7].

<sup>i</sup>¼<sup>1</sup> uin <sup>¼</sup> <sup>1</sup>∀<sup>n</sup> are given by [7].

function is given by

din ¼ k k xn � vi

2

4.2. Local spatial data-based FCM (LDFCM)

uin ∈ U ¼ uin ∈½ � 0; 1 ;

4.1. Conventional FCM

38 Recent Applications in Data Clustering

P<sup>C</sup>

An approach to incorporating local spatial data information into the standard FCM has been presented in [13]. The objective function of the SFCM algorithm is given by

$$J\_{\text{SFCM}} = \sum\_{i=1}^{\mathcal{C}} \sum\_{n=1}^{N} u\_{in}^{m} D\_{in\prime} \tag{11}$$

where Din is a modified or weighted distance between the nth pixel and the ith cluster-center. This modified distance is computed from the original or standard distance din ¼ k k xn � vi 2 as follows

$$D\_{\rm in} = (1 - \lambda)d\_{\rm in}f\_{\rm in} + \lambda d\_{\rm in} \tag{12}$$

where λ ∈½ � 0; 1 is an experimentally selected weight, and f in is a spatial or local data function given by [13].

$$f\_{\rm in} = \frac{\sum\_{k \in N\_n} d\_{ik}}{\min \left\{ \sum\_{k \in N\_n} d\_{ik}; i = 1, 2, \dots, \mathbb{C} \right\}}\tag{13}$$

It is obvious from (12) that with λ ¼ 1, the SFCM clustering method reduces to the standard FCM method. The spatial data function f in is dependent on the original distances of the set of pixels Nn in the immediate neighborhood of the nth pixel. If all pixels in the neighbor set do not belong to the ith cluster f in is maximum since the denominator is minimum while the numerator is maximum. This implies that f in causes Din to increase when the pixels of the immediate neighborhood of the nth pixel do not belong to the ith cluster. This increase of Din contributes to decreasing the membership uin for achieving and preserving the minim of the SFCM function in (11).

The membership uin and the cluster-center vi associated with the SFCM method are given by [13].

$$\mu\_{\rm in} = \frac{1}{\sum\_{j=1}^{C} \left(\frac{D\_{\rm in}}{D\_{\rm in}}\right)^{\frac{1}{(m-1)}}} \tag{14}$$

5. HCM incorporating local membership KL divergence

i¼1 X<sup>N</sup>

<sup>π</sup>in <sup>¼</sup> <sup>1</sup> NK

uin <sup>¼</sup> <sup>1</sup> P C j¼1

based FCM (LMKLFCM), is given by [18–22].

JLMKLFCM <sup>¼</sup> JHCM <sup>þ</sup> <sup>γ</sup> <sup>X</sup><sup>C</sup>

averages are computed by [18–22].

respectively, by [18].

In [18], an approach to incorporating local spatial membership information into HCM algorithm has been presented. By adding Kullback-Leibler (KL) divergence between the membership function of an entity and the locally-smoothed membership in the immediate spatial neighborhood, the modified objective function, called the local membership KL divergence-

Incorporating Local Data and KL Membership Divergence into Hard C-Means Clustering for Fuzzy and Noise-Robust…

<sup>n</sup>¼<sup>1</sup> uin log uin

<sup>π</sup>in <sup>¼</sup> <sup>1</sup> NK

> X <sup>k</sup><sup>∈</sup> Nn; <sup>k</sup>6¼<sup>n</sup>

cardinality. It is obvious that all entities in the window are weighted equally by wð Þ <sup>u</sup>

where Nn is a set of entities/pixels falling in a square window around the nth pixel and NK is its

Other windows can be used such as Gaussian one provided that the weight of the windowcenter is 0 and the rest weights are summed to unity. The first term in (19) provides hardcluster labeling. It pushes the membership function toward 0 or 1. The KL membership and membership-complement divergences, in addition to providing fuzzification approach to HCM clustering, measure the proximity between the membership of a pixel in a certain cluster and the local average of the membership over the immediate spatial neighborhood pixels in this cluster. So, they push the membership function to the locally smoothed membership function πin. Therefore, this can smooth out additive noise and bias the solution to piecewise homogenous labels which leads to a segmented image with piecewise homogenous regions.

The minimization of the objective function JLMKLFCM in (19) yields uin and vi to be given,

<sup>π</sup>jnð Þ ð Þ <sup>1</sup>�πin expð Þþ din=<sup>γ</sup> <sup>π</sup>in

P<sup>N</sup> <sup>n</sup>¼<sup>1</sup> uinxn <sup>P</sup><sup>N</sup> <sup>n</sup>¼<sup>1</sup> uin

vi ¼

where γ is a weighting parameter experimentally selected to control the fuzziness induced by the second term in (19), uin ¼ 1 � uin is the complement of the membership function uin, πin and πin are the spatial local or moving averages of membership uin and the complement membership uin, functions respectively. These local membership and membership complement

> X <sup>k</sup><sup>∈</sup> Nn; <sup>k</sup>6¼<sup>n</sup>

<sup>π</sup>in � � <sup>þ</sup>X<sup>C</sup>

i¼1 X<sup>N</sup>

<sup>π</sup>in � � � � (19)

<sup>n</sup>¼<sup>1</sup> uin log uin

http://dx.doi.org/10.5772/intechopen.74514

41

uik (20)

pq ¼ 1=NK.

(23)

ð Þ¼ 1 � uik 1 � πin (21)

ð Þ <sup>1</sup>�πjn expð Þ djn=<sup>γ</sup> <sup>þ</sup>πjn � � <sup>π</sup>in <sup>¼</sup> <sup>δ</sup>inπin (22)

$$\upsilon\_i = \frac{\sum\_{n=1}^{N} \mu\_{in} \mathbf{x}\_n}{\sum\_{n=1}^{N} \mu\_{in}} \tag{15}$$

It is obvious from (14) that similar to the standard FCM, the membership uin is inversely proportional to the weighted distance Din, which again means that, increasing Din when the immediate neighboring pixels to the nth pixel do not belong to the ith cluster, decreases the membership function uin. From (15), however, it is clear that the SFCM algorithm computes the cluster-center vi in a similar way as the standard FCM method does. Hence, additive noise can still reduce the accuracy of cluster center vi obtained by the SFCM algorithm.

#### 4.4. HCM incorporating membership entropy

The membership entropy has been incorporated into the HCM for fuzzification. The membership entropy-based FCM (MEFCM) algorithm has the following objective function [17].

$$J\_{\rm MEFCM} = J\_{\rm HCM} + \beta \sum\_{i=1}^{C} \sum\_{n=1}^{N} \left( u\_{in} \log \left( u\_{in} \right) + (1 - u\_{in}) \log \left( 1 - u\_{in} \right) \right) \tag{16}$$

where β > 0 is a weight experimentally selected to control the fuzziness of the entropy term. We still need U to be constrained to satisfy (5). It can be shown that the membership and the cluster-center that minimize (16) are respectively given by [17]

$$\mu\_{\rm in} = \frac{1}{\sum\_{j=1}^{C} \frac{\exp\left(d\_{\rm in}/\beta\right) + 1}{\exp\left(d\_{\rm in}/\beta\right) + 1}} \tag{17}$$

$$w\_i = \frac{\sum\_{n=1}^{N} u\_{in} \mathbf{x}\_n}{\sum\_{n=1}^{N} u\_{in}} \tag{18}$$

It is obvious so far that the membership function of the nth entities provided by FCM, HCM and MEFCM algorithms depends upon the inverse of the square of the Euclidean distance din ¼ k k xn � vi <sup>2</sup> which is a function of only xn with no data or membership information of the clustering entity's neighbors are involved. Hence, the FCM, HCM and MEFCM algorithms miss important spatial local data and membership information. Thus additive noise can degrade xn, vi and din, thereby biasing the membership of a degraded entity to a false cluster.
