**2.1 Self-organizing map (SOM)**

The SOM is a nonlinear algorithm used for data clustering and classification. This algorithm is characterized by the processing of static data, i.e., not considering the data timelines, with the output of this neural network dependent only on present input data (Kohonen, 1990, 2001). The SOM is a single-layer neural network, in which it is recorded the learning by algorithm. This layer usually has low dimension structure (1D or 2D).

The training of the SOM is based on unsupervised learning, by adjusting of prototypes, according to the distribution of the input data, performed as follows:

The weight vector of each unit in the map space *VO* is compared to an input vector. A metric-based criterion is chosen to determine the unit that has the minimum distance (Best Matching Unit), i.e., the neuron with the most similar prototype is selected as winner or winning unit, according to equation 1:

$$b(t) = \arg\min\_{i \neq Vol} \left\{ \left\| \mathbf{x}(t) - \mathbf{w}\_i(t) \right\| \right\} \tag{1}$$

Where:

152 Recurrent Neural Networks and Soft Computing

productivity and information losses in the industrial production processes, which contribute negatively to the composition of the electric power quality indices (Rakov & Uman, 2005). This study sought to recognize severe weather indices patterns, starting from an atmospheric sounding database. It is known that the atmospheric instability may be inferred from available radiosondes atmospheric profiling data. The stability indices drawn from observed atmospheric conditions have been used to warn people of potential losses (Peppier, 1988). Thus, this work analyzed the capacity of the Self-Organizing Map (SOM) and two of its temporal extensions: Temporal Kohonen Map and Recurrent Self-Organizing Map (Chappell & Taylor, 1993; Koskela et al., 1998a, 1998b; Varsta et al., 2000; Varsta et al., 2001) for clustering and classification of atmospheric sounding patterns in order to contribute with the weather studies over the Brazilian Amazon. The option of using this type of neural network was due to the fact that it uses only the input parameters, making it ideal for problems where the patterns are unknown. Although there are other temporal extensions of SOM, such as recursive SOM - RecSOM (Hammer et al., 2004; Voegtlin, 2002), Self-Organizing Map for Structured Data - SOMSD (Hagenbuchner et al., 2003) and Merge Self-Organizing Map – MSOM (Strickert & Hammer, 2005), all these of global context, the option in this work was to apply local context algorithms, leaving to future studies the application of global context algorithms in this knowledge area. It is also important to refer the existence of the recent studies on the TKM

and RSOM networks (Cherif et al., 2011; Huang & Wu, 2010; Ni & Yin, 2009).

these recurrent neural networks for the severe weather patterns recognition.

**2. SOM and temporal extensions (TKM and RSOM)** 

**2.1 Self-organizing map (SOM)** 

winning unit, according to equation 1:

Kohonen Map (TKM) and Recurrent Self-Organizing Map (RSOM).

algorithm. This layer usually has low dimension structure (1D or 2D).

according to the distribution of the input data, performed as follows:

In summary, with the original SOM algorithm and its extensions TKM and RSOM; stability indices data (Peppier, 1988); multivariate statistical techniques (principal component analysis and k-means); confusion matrix (Han & Kamber, 2006) and receiver operating characteristics (ROC) analysis (Fawcett, 2006); it was possible to evaluate the usefulness of

This section discusses the fundamental concepts of Self-Organizing Map (SOM) and the proposed changes to introduce temporal sequence processing by the SOM: Temporal

The SOM is a nonlinear algorithm used for data clustering and classification. This algorithm is characterized by the processing of static data, i.e., not considering the data timelines, with the output of this neural network dependent only on present input data (Kohonen, 1990, 2001). The SOM is a single-layer neural network, in which it is recorded the learning by

The training of the SOM is based on unsupervised learning, by adjusting of prototypes,

The weight vector of each unit in the map space *VO* is compared to an input vector. A metric-based criterion is chosen to determine the unit that has the minimum distance (Best Matching Unit), i.e., the neuron with the most similar prototype is selected as winner or


The neurons of the SOM cooperate to receive future incoming stimuli in an organized manner around the winner neuron. The winner neuron will be the center of a topological neighbourhood where neurons help each other to receive input signals along the iterations of network training. Thus, after obtaining the winning neuron, its weights are adjusted to increase the similarity with the input vector, the same being done for the weights of its neighbours, by an update rule, according to equation 2:

$$\mathbf{w}\_{i}(t+1) = \mathbf{w}\_{i}(t) + \boldsymbol{\chi}(t)h\_{ib}(t)(\mathbf{x}(t) - \mathbf{w}\_{i}(t))\tag{2}$$

Where:


Usually, the learning rate (*t*) is defined by equation 3:

$$\chi(t) = \chi\_0 \exp\left(-\frac{t}{\tau\_1}\right) \tag{3}$$

Where:


The neural network decreases its ability to learn, gradually over time, in order to prevent the drastic change by new data, in the sedimented knowledge through several iterations. The time constant influences the network learning as follows: high *<sup>1</sup>*value generates long period of intensive learning.

The neighbourhood function in a SOM is a similar way to reproduce the interactions of biological neurons, which stimulate their neighbours, in decreasing order, by increasing the lateral distance between them. So, for the SOM, this feature is reproduced by the parameter *hib*(*t*) that determines how each neuron will receive readjustment to gain the future input stimuli. The largest adjustments are applied to the winner neuron and its neighbours, and minors to the neurons further from the winner neuron, because this parameter decreases with increasing lateral distance. Normally it is used the Gaussian function to represent the rate of cooperation between the neurons, by equation 4:

$$h\_{ib}(t) = \exp\left(-\frac{l\_{ib}^2}{2\sigma(t)^2}\right) \tag{4}$$

Recurrent Self-Organizing Map for Severe Weather Patterns Recognition 155

In the original SOM, for each new iteration, the value *1* is applied to the winner neuron and the value *0* to the other neurons. This creates an abrupt change in the neuron activation. In the TKM occurs a smooth change in the activation value (leaky integrator), because it uses the previous activation value, as shown in equation 6. In the TKM algorithm, the neuron that has the highest activation will be considered the winner neuron, according to the

> ( ) argmax ( ) *<sup>i</sup> i Vo bt V t*

After choosing the winner neuron, the TKM network performs operations identical to the

 For the determination of the winner neurons in TKM is necessary to calculate and record the activation *Vi(t)*, while in SOM is necessary to calculate the quantization error

The winner neuron in TKM is one with greater activation *Vi(t)*, while in SOM is one

Interestingly, for *d*=0 the TKM network becomes equivalent to SOM network used for static

Another algorithm that introduced the temporal processing to the SOM was the Recurrent Self-Organizing Map - RSOM using a new form of selection of the winner neuron and weights update rule (Koskela et al., 1998a, 1998b; Varsta et al., 2000; Varsta et al., 2001). This algorithm moved the leaky integrator from the unit outputs into the inputs. The RSOM allows storing information in the map units (difference vectors), considering the past input

( ) (1 ) ( 1) ( ( ) ( )) *ii i* **y y xw** *t t tt*

Considering the term **x***(t)* - **w**i*(t)* with the quantization error, the winner neuron will be one that has the least recursive difference, i.e., the smallest sum of the present and past

( ) arg min ( ) *<sup>i</sup> i Vo*

*b t <sup>t</sup>* **<sup>y</sup>** (9)

( 1) ( ) ( ) ( ) ( ) *<sup>i</sup> <sup>i</sup> ib i* **ww y** *t t γ th t t* (10)

 

 **y***i(t)* is called recursive difference of the neuron *i*, at time *t*; is the leaking coefficient (value between 0 and 1).

In the RSOM the weights update occur according to the equation 10:

quantization errors, according to the equation 9:

The basic differences between TKM and SOM networks are:

with smallest quantization error **x**(*t*)-**w***i*(*t*).

**2.3 Recurrent self-organizing map (RSOM)** 

(7)

(8)

equation 7:

original SOM.

**x**(*t*)-**w***i*(*t*);

data (Salhi et al., 2009).

vectors, by equation 8:

Where:

Where:


Considering that the effective width of the topological neighbourhood will diminish with time increasingly specialized network regions will be built for certain input patterns. Over the course of iterations the radius of a neighbourhood should be smaller, which implies lower *hib*(*t*) values, over time, thereby resulting in a restricted and specialized neighbourhood. For this, the exponential function is usually used, according to equation 5:

$$
\sigma(t) = \sigma\_0 \exp\left(-\frac{t}{\tau\_1}\right) \tag{5}
$$

Where:


#### **2.2 Temporal Kohonen Map (TKM)**

The SOM was originally designed for the static data processing, but for the dynamic data patterns recognition, it becomes necessary to include the temporal dimension in this algorithm. A pioneer algorithm in this adaptation was the Temporal Kohonen Map - TKM (Chappell & Taylor, 1993). It introduces the temporal processing using the same update rule of the original SOM, just changing the way of choosing the winner neuron. It uses the neurons activation history, by equation 6:

$$dV\_i(t) = dV\_i(t-1) \cdot \frac{1}{2} \left\| \mathbf{x}(t) - \mathbf{w}\_i(t) \right\|^2 \tag{6}$$

Where:


A TKM algorithm flow diagram is displayed in Figure 1. The current activation of the neuron is dependent on previous activation.

Fig. 1. TKM algorithm flow diagram

In the original SOM, for each new iteration, the value *1* is applied to the winner neuron and the value *0* to the other neurons. This creates an abrupt change in the neuron activation. In the TKM occurs a smooth change in the activation value (leaky integrator), because it uses the previous activation value, as shown in equation 6. In the TKM algorithm, the neuron that has the highest activation will be considered the winner neuron, according to the equation 7:

$$b(t) = \arg\max\_{i \in Vol} \{V\_i(t)\}\tag{7}$$

After choosing the winner neuron, the TKM network performs operations identical to the original SOM.

The basic differences between TKM and SOM networks are:


Interestingly, for *d*=0 the TKM network becomes equivalent to SOM network used for static data (Salhi et al., 2009).

#### **2.3 Recurrent self-organizing map (RSOM)**

Another algorithm that introduced the temporal processing to the SOM was the Recurrent Self-Organizing Map - RSOM using a new form of selection of the winner neuron and weights update rule (Koskela et al., 1998a, 1998b; Varsta et al., 2000; Varsta et al., 2001). This algorithm moved the leaky integrator from the unit outputs into the inputs. The RSOM allows storing information in the map units (difference vectors), considering the past input vectors, by equation 8:

$$\mathbf{y}\_i(t) = (1 - \alpha)\mathbf{y}\_i(t-1) + \alpha(\mathbf{x}(t) - \mathbf{w}\_i(t))\tag{8}$$

Where:

154 Recurrent Neural Networks and Soft Computing

Considering that the effective width of the topological neighbourhood will diminish with time increasingly specialized network regions will be built for certain input patterns. Over the course of iterations the radius of a neighbourhood should be smaller, which implies lower *hib*(*t*) values, over time, thereby resulting in a restricted and specialized neighbourhood. For this, the exponential function is usually used, according to equation 5:

0

*σ(t) σ*

exp

The SOM was originally designed for the static data processing, but for the dynamic data patterns recognition, it becomes necessary to include the temporal dimension in this algorithm. A pioneer algorithm in this adaptation was the Temporal Kohonen Map - TKM (Chappell & Taylor, 1993). It introduces the temporal processing using the same update rule of the original SOM, just changing the way of choosing the winner neuron. It uses the

<sup>1</sup> <sup>2</sup> ( ) ( -1)- ( ) ( ) <sup>2</sup>

A TKM algorithm flow diagram is displayed in Figure 1. The current activation of the

*V t dV t t t i i <sup>i</sup>* **x w** (6)

1

(5)

*τ*

*t*

*lib* is the lateral distance between neurons *i* and *b*;

*<sup>0</sup>* is the initial value of effective width;

*<sup>1</sup>* is a time constant.

**2.2 Temporal Kohonen Map (TKM)** 

neurons activation history, by equation 6:

 *Vi(t)* is the neuron activation, at time t; *d* is a time constant (value between 0 and 1);

neuron is dependent on previous activation.

 **x**(*t*) is a input vector, at time *t;*  **w***i*(*t*) is a prototype, at time *t.* 

Fig. 1. TKM algorithm flow diagram

(*t*) is the effective width of the topological neighbourhood.

Where:

Where:

Where:

 


Considering the term **x***(t)* - **w**i*(t)* with the quantization error, the winner neuron will be one that has the least recursive difference, i.e., the smallest sum of the present and past quantization errors, according to the equation 9:

$$b(t) = \arg\min\_{i \neq Vol} \left\{ \left\| \mathbf{y}\_i(t) \right\| \right\} \tag{9}$$

In the RSOM the weights update occur according to the equation 10:

$$\mathbf{w}\_{i}(t+1) = \mathbf{w}\_{i}(t) + \boldsymbol{\chi}(t)h\_{i\boldsymbol{\vartheta}}(t)\mathbf{y}\_{i}(t) \tag{10}$$

Recurrent Self-Organizing Map for Severe Weather Patterns Recognition 157

temperature, dewpoint temperature, relative humidity, among others, in various atmospheric levels. These parameters are used to calculate sounding indices that seek to analyze the state of the atmosphere at a given time. Figure 4 shows an example of sounding

For the evaluation of the atmospheric static stability, used for thunderstorms forecasting, several indicators have been developed (Peppier, 1988). Some indicators admit as instability factors the temperature difference and humidity difference between two pressure levels; while others, besides these factors, add the characteristics of the wind (speed and direction) at the same pressure levels. There are also indices based on the energy requirements for the occurrence of convective phenomena. Some indices and parameters used for the thunderstorms forecasting are: Showalter Index, K Index, Lifted Index, Cross Totals Index, Vertical Totals Index, Total Totals Index, SWEAT Index, Convective Inhibition, Convective Available Potential Energy, Level of Free Convection, Precipitable Water, among others.

indices collected from a radiosonde launched on January 1, 2010 at 12 h UTC.

Fig. 3. SBBE station localization (Belem airport).

Where:


Thus, the RSOM takes into account the past inputs and also starts to remember explicitly the space-time patterns.

A RSOM algorithm flow diagram is exhibited in Figure 2.

Fig. 2. RSOM algorithm flow diagram

The basic differences between RSOM and SOM networks are:


To note that if =1 the RSOM network becomes identical to a SOM network (Salhi et al., 2009).

Angelovič (2005) discribes several advantages of the use RSOM for prediction systems. First, the small computing complexity, opposite to the global models. Then, the unsupervised learning. It allows building models from the data with only a little a priori knowledge.
