**2.1.1 Fully recurrent neural networks (FRNN)**

The Fully Recurrent Neural Network (FRNN) is first described here in terms of individual neurons and their connections, as was done in [Williams e.a., 1989]. Then the FRNN is considered as a special case of the general state-space model and a convenient matrix notation of the network is given.

Recurrent Neural Network with Human Simulator Based Virtual Reality 93

Fig. 2. Example of a fully recurrent neural network (of type 1) (Dijk, 1999).

Additional restrictions can be imposed on the FRNN architecture described in the previous subsection to create other (restricted) Recurrent Neural Network (RNN) architectures. This subsection will describe some of these restricted architectures. Because the FRNN can be written as a state-space model, all .subsets. of FRNN are in many cases most conveniently

The following categories of restrictions can be used (individually or in a combination):

2. forcing weights to non-zero value (called *fixing* the weight or making the weight non-

3. forcing weights to be equal to other weights (called *sharing* of weights) or

These restrictions will be looked at in this subsection. Note that the three restrictions listed are fairly general and can be applied to other neural networks architecture than the FRNN,

All three restrictions have a property in common: the number of free parameters of the network is reduced when compared to a non-modified FRNN. Reasons for doing so will be given now. More reasons for applying restrictions will be given in the category descriptions. The training of a neural network is in fact a procedure that tries to estimate the parameters (weights) of the network such that an error measure is minimized. Reducing the number of parameters to be estimated may simplify training. Another good reason for reducing the number of free parameters is to reduce training algorithm overhead, which often grows

1. forcing certain weights to zero (called removing or *pruning* the weight)

approximately equal to other weights (called *soft sharing* of weights)

**2.1.2 Subsets of FRNN: Recurrent neural networks (RNN)** 

for example to the standard feedforward network.

quickly for an increasing number of weights NW.(Cruse, 2006).

written as state-space models.

learnable)

The name Fully Recurrent Neural Network for this network type is proposed by [Kasper e.a., 1999]. Another name for this type of network is the .Real-Time Recurrent Network.. This name will not be used further, because the name strongly implies that training is accomplished using the Real-Time Recurrent Learning (RTRL) algorithm proposed for this network in [Williams e.a., 1989] which is not necessarily the case because other algorithms can be used. In general a FRNN has N neurons, M external inputs and L external outputs. In Fig.2 an example of a FRNN is given which has N=4 neurons, M=2 external inputs u 1(*n*), u2(*n*) and L=2 external outputs y1(*n*), y2(*n*).( Dijk, 1999 ).

Fig. 1. Recurrent neural network architectures hierarchy (numbers indicate sections)

The network is called Fully Recurrent because the output of all neurons is recurrently connected (through N delay elements and N2 weighted feedback connections) to all neurons in the network. The external network inputs are connected to the neurons by N\*M feed forward connections *without* delay element. A bias (also called threshold) can be introduced for every neuron by applying a constant external input u1(*n*) = 1 to the network.

For static neural networks, the number of layers in the neural network can be clearly defined as the number of neurons an input signal passes through before reaching the output. For the FRNN however the same definition is ambiguous, because signals applied at time *n* are fed back and reach the output at times *n*, *n*+1, and so on. The term layer therefore appears to be never used in literature in FRNN descriptions. By redefining the concept of layer to: the *minimum* number of neurons an input signal passes through before reaching the output, a workable definition is obtained for the FRNN.

The name Fully Recurrent Neural Network for this network type is proposed by [Kasper e.a., 1999]. Another name for this type of network is the .Real-Time Recurrent Network.. This name will not be used further, because the name strongly implies that training is accomplished using the Real-Time Recurrent Learning (RTRL) algorithm proposed for this network in [Williams e.a., 1989] which is not necessarily the case because other algorithms can be used. In general a FRNN has N neurons, M external inputs and L external outputs. In Fig.2 an example of a FRNN is given which has N=4 neurons, M=2 external inputs u 1(*n*),

Fig. 1. Recurrent neural network architectures hierarchy (numbers indicate sections)

for every neuron by applying a constant external input u1(*n*) = 1 to the network.

workable definition is obtained for the FRNN.

The network is called Fully Recurrent because the output of all neurons is recurrently connected (through N delay elements and N2 weighted feedback connections) to all neurons in the network. The external network inputs are connected to the neurons by N\*M feed forward connections *without* delay element. A bias (also called threshold) can be introduced

For static neural networks, the number of layers in the neural network can be clearly defined as the number of neurons an input signal passes through before reaching the output. For the FRNN however the same definition is ambiguous, because signals applied at time *n* are fed back and reach the output at times *n*, *n*+1, and so on. The term layer therefore appears to be never used in literature in FRNN descriptions. By redefining the concept of layer to: the *minimum* number of neurons an input signal passes through before reaching the output, a

u2(*n*) and L=2 external outputs y1(*n*), y2(*n*).( Dijk, 1999 ).

Fig. 2. Example of a fully recurrent neural network (of type 1) (Dijk, 1999).
