4.5 Types of activation functions

In the mathematical form, a neuron k may be described by the following

uk ¼ ∑ n i¼1

Let v<sup>k</sup> be the induced local field or activation potential, which is given as:

v<sup>k</sup> ¼ ∑ m i¼0

In Eq. (5), a new synapse with input x0 = +1 is added and its weight is wk0 = bk to

Determination of appropriate neural network architecture is one of the most important tasks in model-building process. Various types of neural networks are analyzed to find the most appropriate architecture of a particular problem. Multilayer feed forward networks are found to outperform all the others. Although multilayer feed forward networks are one of the most fundamental models, they are

There is no fixed rule for selection of hidden layers of a network. Therefore, trial

The ability of the ANN to generalize data not included in training depends on selection of sufficient number of hidden neurons to provide a means for storing higher order relationships necessary for adequately abstracting the process. There is no direct and precise way of determining the most appropriate number of neurons to include in hidden layer and this problem becomes more complicated as number of hidden layer increases. Some studies indicated that more number of neurons in hidden layer provide a solution surface that closely fit to training patterns. But in practice, more number of hidden neurons results the solution surface that deviate significantly from the trend of the surface at intermediate points or provide too literal interpretation of the training points which is called 'over fitting'. Further, large number of hidden neurons reduces the speed of operation of network during

and error method was used for selection of number of hidden layers. Even one hidden layer of neuron (operating sigmoid activation function) can also be suffi-

the most popular type of ANN structure suited for practical applications.

cient to model any solution surface of practical interest [36].

vation function; yk = output signal of the neuron k.

Advanced Evapotranspiration Methods and Applications

Now, Eqs. (1), (2) and (3) can be written as:

4.2 Neural network architecture parameters

consider the effect of the bias.

4.3 Number of hidden layers

4.4 Number of hidden neurons

28

where x1, x2, x3, ……….. xn = input signals; wk1,wk2, …….wkn = synaptic weights of neuron k; uk = linear combiner output due to the input signal; bk = bias; φ(.) = acti-

wkixi (1)

yk ¼ ϕð Þ uk þ bk (2)

v<sup>k</sup> ¼ uk þ bk (3)

yk ¼ ϕðvkÞ (5)

wknxn (4)

equations:

The activation function or transfer function, denoted by φ(v), defines the output of a neuron in terms of the induced local field v. It is valuable in ANN applications as it introduces a degree of nonlinearity between inputs and outputs. Logistic sigmoid, hyperbolic tangent and linear functions are some widely used transfer function in ANN modeling.

Logistic sigmoid function: This function is a continuous function that reduces the output into the range of 0–1 and is defined as [32]:

$$\varphi(v) = \frac{\mathbf{1}}{\mathbf{1} + \exp\left(-v\right)}\tag{6}$$

Hyperbolic tangent function: It is used when the desired range of output of a neuron is between �1 and 1 and is expressed as [32]:

$$\rho(v) = \tanh(v) = \frac{\mathbf{1} - e^{-2v}}{\mathbf{1} + e^{-2v}} \tag{7}$$

Linear function: It calculates the neuron's output by simply returning the value passed to it. It can be expressed as:

$$
\mathfrak{q}(\upsilon) = \upsilon
\tag{8}
$$
