**2.1.2 Subsets of FRNN: Recurrent neural networks (RNN)**

Additional restrictions can be imposed on the FRNN architecture described in the previous subsection to create other (restricted) Recurrent Neural Network (RNN) architectures. This subsection will describe some of these restricted architectures. Because the FRNN can be written as a state-space model, all .subsets. of FRNN are in many cases most conveniently written as state-space models.

The following categories of restrictions can be used (individually or in a combination):


These restrictions will be looked at in this subsection. Note that the three restrictions listed are fairly general and can be applied to other neural networks architecture than the FRNN, for example to the standard feedforward network.

All three restrictions have a property in common: the number of free parameters of the network is reduced when compared to a non-modified FRNN. Reasons for doing so will be given now. More reasons for applying restrictions will be given in the category descriptions. The training of a neural network is in fact a procedure that tries to estimate the parameters (weights) of the network such that an error measure is minimized. Reducing the number of parameters to be estimated may simplify training. Another good reason for reducing the number of free parameters is to reduce training algorithm overhead, which often grows quickly for an increasing number of weights NW.(Cruse, 2006).

Recurrent Neural Network with Human Simulator Based Virtual Reality 95

Some neural network architectures can be best described as modular architectures. The definition of a modular architecture as used in this report is: an architecture that consists of several static neural networks, that are interconnected in a specific way. There is, in most cases, not a clear boundary between a modular network and a single neural network because the total modular architecture can be looked at as a single neural network, and some existing single networks can also be described as modular networks. It is rather a convenient way of describing complex neural networks.( Paine W. Rainer & Tani Jun,

In this section the category of modular *recurrent* neural network architectures is looked at, modular architectures that all have one or more internal feedback connections. The modular recurrent neural network architectures were not introduced in previous sections, because they do not fit very well in the state-space system description or the NARX

Formally they can be described as a state-space system (like any dynamic system) but this could result in a very complicated and unnecessarily large state-space system description. In this section three classes of modular recurrent neural network architectures are

The first model (RMLP) is a rather specific one and it is included as an example of a modular architecture. Undoubtedly, many more such architectures are proposed in literature and they cannot all be listed here. Another example is the Pipelined Recurrent Neural Network found in [Haykin, 1998] and applied to speech prediction in [Baltersee e.a., 1998]. The second model is far more general and was meant to provide a structured way to describe a large class of recurrent neural networks and their training algorithms. The third model attempts to do the same and it turns out that this model is the most general one: it incorporates the first two as special cases, so in this section the attention will be mainly focussed on the third model, the general modular network framework. An extension of the regular MLP has been proposed by Puskorias e.a. (see [Haykin, 1998]) which adds selffeedback connections for each layer of the standard MLP. The resulting Recurrent

Each layer is a standard MLP layer. The layer outputs are fed forward to the inputs of the next layer and the delayed layer outputs are fed back into the layer itself. So the layer output of time *n*-1 for a certain layer acts as the state variable at time *n* for this layer. The global state of the network consists of all layer states [i(*n*) together. Effectively, this type of network can have both a very large total state vector and a relatively small number of parameters because the neurons in the network are not fully interconnected. There are no recurrent interconnections across layers. All recurrent connections are local (1-layer-to-itself).(Sit,

Multilayer Perceptron (RMLP) structure with N layers is shown in Fig 5.

**2.2 Types of RNN according to NN structure** 


**2.2.1 Recurrent multi-layer perceptron (RMLP)** 

2004).

description.

presented:

2005).

Fig. 3. One example RNN presented as (a). a two-layer neural network with delays and recurrent connections; and (b) as a FRNN with removed connections.

#### **2.1.3 Partially recurrent networks (PRN)**

The output vector (*n*) of the FRNN consists of the first L elements of the state vector [(*n*), as was shown in Fig. 2. So the output signals are a .subset. of state signals. In a general state space description this is certainly not the case, the output is determined by a separate calculation (the output equation) which is some function of the external input and the state. To obtain a network that effectively has separate state and output units (analogous to a state space system that has separate process and output equations), the feedback connections from all L output neurons yi(*n*) with *i*=1.L are removed. An example of the *partially recurrent neural network* (PRN) [Robinson e.a., 1991], also named the *simplified recurrent neural network*  [Janssen, 1998], that results is shown in Fig. 4. The name partially recurrent neural network. will be used in this report to avoid confusion in the terms simple/simplified recurrent networks in the next subsection.

Fig. 4. Example of a partially recurrent neural network
