**5. Simulation results**

282 Recurrent Neural Networks and Soft Computing

Fig. 4. Daily Temperature Data of Batu Pahat Region (after normalization)

Fig. 5. Normal Distribution of Temperature Data (after normalization)

Table 2. Summary of Temperature Dataset Segregation

with 2 nodes, and increased by one until a maximum of 5 nodes.

For simulation purposes, the data was segregated into time order and was divided into three sets; 50% for training, 25% for testing and 25% for validation, as shown in Table 2.

> **Training Validation Testing**  914 456 456

For comparison purposes, the JPSN performances on temperature prdiction will be benchmarked againts that of the ordinary PSNN and the widely known MLP. As there is no rule of thumb for identifying the number of input, a trial-and-error procedure was determined. All networks were built considering 5 different number of input nodes ranging from 4 to 8. A single neuron was considered for the output layer. The number of hidden nodes (for MLP), and the higher order terms (for PSNN and JPSN) were initially started The temperature dataset collected from MMD was used to demonstrate the performance of JPSN by considering a few different network parameters. Generally, the factors affecting the network performance include the learning factors, the higher order terms, and the number of neurons in the input layer. Extensive experiments have been conducted for training, testing and validation sets, and average results of 10 simulations/runs have been collected. Two stopping criteria were used during the learning process; the maximum epoch and the minimum error, which were set to 3000 and 0.0001 respectively. In order to assess the performance of all network models, four measurement criteria, namely the number of epoch, Mean Squared Error, Normalized Mean Squared Error, and Signal to Noise Ratio are used. Convergence is achieved when the output of the network meets the earlier mentioned stopping criteria. By considering all in-sample dataset that have been trained, the best value for the momentum term 0.2 and the learning rate 0.1 , were chosen based on extensive simulations done by trial-and-error procedure.

The above discussions have shown that some network parameters may affect the network performances. In conjunction with that, it is necessary to illustrate the robustness of JPSN by comparing its performance with the ordinary PSNN and the MLP. Table 3 to Table 5 show the average results from 10 simulations for the JPSN, the ordinary PSNN and the MLP, respectively.


Table 3. Average Result of JPSN for One-Step-Ahead Prediction.

An Application of Jordan Pi-Sigma

and number of epochs (refer to Table 5).

two benchmarked models.

**NMSE on Testing Dataset**

**Network Models** 

Neural Network for the Prediction of Temperature Time Series Signal 285

As it can be noticed, Table 3 which shows the results for temperature prediction using JPSN reveals that the 2nd order network, with 4 inputs demonstrates the best results using all measuring criteria except for the number of epochs. Meanwhile, Table 4 shows the results that were produced by PSNN on temperature prediction. It demonstrates that the combination of 8 input nodes and PSNN of Order 2 shows the best performance for all measuring criteria. Same thing goes to the MLP, which signifies that MLP with 2 hidden nodes and 8 input nodes attained the best results for all measuring criteria except for SNR

**MAE on** 

**Testing Dataset SNR Number** 

**of Epoch** 

**MSE on Testing Dataset** 

**JPSN** 0.771034 0.006462 0.063458 18.7557 1460.9 **PSNN** 0.779118 0.006529 0.063471 18.71039 1211.8 **MLP** 0.781514 0.006549 0.063646 18.69706 2849.9 Table 6. Comparison on the Best Single Simulation Results for JPSN, PSNN and MLP.

In order to compare the predictive performance of the three models, Table 6 presents the best simulation results for JPSN, PSNN and MLP. Over all the training process, JPSN obtained the lowest MAE, which is 0.063458; while the MAE for PSNN and MLP were 0.063471 and 0.063646, respectively. By considering the MAE, it shows that JPSN is able to make a very close forecasts to the actual output in analysing the temperature. In this respect, JPSN outperformed PSNN by a ratio of <sup>4</sup> 1.95 10 , and <sup>3</sup> 2.9 10 for the MLP. Moreover, it can be seen that JPSN reached higher value of SNR. Therefore, it can be said that the network can track the signal better than PSNN and MLP. Apart from the MAE and SNR, it is verified that JPSN exhibited lower prediction errors, in terms of NMSE and MSE on the out-of-sample dataset. This indicates that the network is capable of representing nonlinear function better than the two benchmarked models. In the case of learning speed, particularly on the number of epoch utilized, PSNN converged much faster than the JPSN and MLP. However, JPSN reached a smaller number of epoch when compared to the MLP. On the whole, the performance of JPSN gives a gigantic comparison when compared to the

For demonstration purpose, the models' performance on their NMSE is depicted in Figure 6. It shows that JPSN steadily gives lower NMSE when compared to both PSNN and MLP. This by means shows that the predicted and the actual values which were obtained by the JPSN are better than both comparable network models in terms of bias and scatter. Consequently, it can be inferred that the JPSN yield more accurate results, providing the choice of network parameters are determined properly. The parsimonious representation of

The plots depicted in Figures 7 to 9 present the temperature forecast on the out-of-sample dataset for all network models. As shown in the plots, the blue line represents the trend of the actual values, while the red line represents the predicted values. The predicted values of daily temperature measurement made by all network models almost fit the actual values with minimum error forecast. On the whole, JPSN practically beat out PSNN and MLP by 1.038% and 1.341%, respectively. It is verified that JPSN has the ability to perform an input-output mapping of temperature data as well as better performance when compared to

higher order terms in JPSN assists the network to model successfully.


Table 4. Average Result of PSNN for One-Step-Ahead Prediction.


Table 5. Average Result of MLP for One-Step-Ahead Prediction.

**MSE on Testing** 

 0.7791 0.0065 18.7104 1211.8 0.7792 0.0065 18.7097 1302.9 0.7797 0.0065 18.7071 1315.3 0.7822 0.0066 18.6935 1201.0

 0.7768 0.0065 18.7234 1222.5 0.7769 0.0065 18.7226 1221.6 0.7775 0.0065 18.7193 1149.9 0.7806 0.0065 18.7023 916.9

 0.7758 0.0065 18.7289 961.3 0.7770 0.0065 18.7222 852.5 0.7775 0.0065 18.7192 1155.6 0.7832 0.0066 18.6880 948.0

 0.7726 0.0065 18.7470 1064.0 0.7733 0.0065 18.7432 911.6 0.7760 0.0065 18.7277 922.1 0.7766 0.0065 18.7244 1072.2

 0.7674 0.0064 18.7688 719.3 0.7677 0.0064 18.7671 877.0 0.7693 0.0065 18.7579 861.4 0.7694 0.0065 18.7574 970.9

**MSE on Testing** 

 0.7815 0.0065 18.6971 2849.9 0.7831 0.0066 18.6881 2468.2 0.7825 0.0066 18.6918 2794.2 0.7827 0.0066 18.6903 2760.9

 0.7803 0.0065 18.7037 2028.6 0.7792 0.0065 18.7097 2451.8 0.7789 0.0065 18.7114 2678.1 0.7792 0.0065 18.7097 2565.8

 0.7750 0.0065 18.7335 2915.1 0.7763 0.0065 18.7261 2837.2 0.7776 0.0065 18.7188 2652.4 0.7783 0.0065 18.7152 2590.8

 0.7742 0.0065 18.7378 2951.2 0.7780 0.0065 18.7164 2566.6 0.7771 0.0065 18.7217 2796.4 0.7786 0.0065 18.7131 2770.0

 0.7734 0.0065 18.7350 2684.8 0.7749 0.0065 18.7268 2647.2 0.7747 0.0065 18.7278 2557.1 0.7753 0.0065 18.7242 2774.6

**Dataset SNR Number of** 

**Dataset SNR Number of** 

**Epoch** 

**Epoch** 

**NMSE on Testing Dataset**

Table 4. Average Result of PSNN for One-Step-Ahead Prediction.

Table 5. Average Result of MLP for One-Step-Ahead Prediction.

**NMSE on Testing Dataset**

**Input Nodes** 

**4** 

**5** 

**6** 

**7** 

**8** 

**Input Nodes** 

**4** 

**5** 

**6** 

**7** 

**8** 

**Hidden Nodes**

**Network Order**

As it can be noticed, Table 3 which shows the results for temperature prediction using JPSN reveals that the 2nd order network, with 4 inputs demonstrates the best results using all measuring criteria except for the number of epochs. Meanwhile, Table 4 shows the results that were produced by PSNN on temperature prediction. It demonstrates that the combination of 8 input nodes and PSNN of Order 2 shows the best performance for all measuring criteria. Same thing goes to the MLP, which signifies that MLP with 2 hidden nodes and 8 input nodes attained the best results for all measuring criteria except for SNR and number of epochs (refer to Table 5).


Table 6. Comparison on the Best Single Simulation Results for JPSN, PSNN and MLP.

In order to compare the predictive performance of the three models, Table 6 presents the best simulation results for JPSN, PSNN and MLP. Over all the training process, JPSN obtained the lowest MAE, which is 0.063458; while the MAE for PSNN and MLP were 0.063471 and 0.063646, respectively. By considering the MAE, it shows that JPSN is able to make a very close forecasts to the actual output in analysing the temperature. In this respect, JPSN outperformed PSNN by a ratio of <sup>4</sup> 1.95 10 , and <sup>3</sup> 2.9 10 for the MLP. Moreover, it can be seen that JPSN reached higher value of SNR. Therefore, it can be said that the network can track the signal better than PSNN and MLP. Apart from the MAE and SNR, it is verified that JPSN exhibited lower prediction errors, in terms of NMSE and MSE on the out-of-sample dataset. This indicates that the network is capable of representing nonlinear function better than the two benchmarked models. In the case of learning speed, particularly on the number of epoch utilized, PSNN converged much faster than the JPSN and MLP. However, JPSN reached a smaller number of epoch when compared to the MLP. On the whole, the performance of JPSN gives a gigantic comparison when compared to the two benchmarked models.

For demonstration purpose, the models' performance on their NMSE is depicted in Figure 6. It shows that JPSN steadily gives lower NMSE when compared to both PSNN and MLP. This by means shows that the predicted and the actual values which were obtained by the JPSN are better than both comparable network models in terms of bias and scatter. Consequently, it can be inferred that the JPSN yield more accurate results, providing the choice of network parameters are determined properly. The parsimonious representation of higher order terms in JPSN assists the network to model successfully.

The plots depicted in Figures 7 to 9 present the temperature forecast on the out-of-sample dataset for all network models. As shown in the plots, the blue line represents the trend of the actual values, while the red line represents the predicted values. The predicted values of daily temperature measurement made by all network models almost fit the actual values with minimum error forecast. On the whole, JPSN practically beat out PSNN and MLP by 1.038% and 1.341%, respectively. It is verified that JPSN has the ability to perform an input-output mapping of temperature data as well as better performance when compared to

An Application of Jordan Pi-Sigma

Neural Network for the Prediction of Temperature Time Series Signal 287

Fig. 7. Temperature Forecast made by JPSN on Out-of sample Dataset.

Fig. 8. Temperature Forecast made by PSNN on Out-of sample Dataset.

both network models. Besides, the evaluations on MAE, NMSE, MSE, and SNR over the temperature data demonstrated that JPSN were merely improved the performance level compared to the two benchmarked network models, PSNN and MLP. The better performance of temperature forecasting is allocated based on the vigour properties it contains. Hence, it can be seen that the thrifty representation of higher order terms in JPSN assists the network to model effectively.

both network models. Besides, the evaluations on MAE, NMSE, MSE, and SNR over the temperature data demonstrated that JPSN were merely improved the performance level compared to the two benchmarked network models, PSNN and MLP. The better performance of temperature forecasting is allocated based on the vigour properties it contains. Hence, it can be seen that the thrifty representation of higher order terms in JPSN

**0.77911793**

Network Models

**0.781514237**

JPSN PSNN MLP

assists the network to model effectively.

0.764

Fig. 6. The NMSE for JPSN, PSNN, and MLP.

0.766

0.768

0.77

0.772

0.774

0.776

0.778

0.78

0.782

0.784

**NMSE**

**0.771034086**

Fig. 7. Temperature Forecast made by JPSN on Out-of sample Dataset.

Fig. 8. Temperature Forecast made by PSNN on Out-of sample Dataset.

An Application of Jordan Pi-Sigma

*Systems, 2 (303)*, pp. 14.

pp. 913-920, IEEE, 2006.

*Neurocomputing, 55*, pp. 363-382.

Science Society, New Jersey, USA.

Science, Engineering and Technology.

Propagating Errors. *Nature, 323 (9)*, pp. 533-536.

*Neural Networks, Dagli*, (pp. 205-210). ASME Press.

pp. 55-58.

Neural Network for the Prediction of Temperature Time Series Signal 289

Chang, F.-J.: Chang, L.-C., Kao, H.-S. & Wu, G.-R. (2010). Assessing the effort of

Cybenko, G. (1989). Approximation by Superpositions of a Sigmoidal Function. *Signals* 

Ghazali, R. & al-Jumeily, D. (2009). Application of Pi-Sigma Neural Networks and Ridge

Ghazali, R.: Hussain, A. & El-Deredy, W. (2006). Application of Ridge Polynomial Neural

Ghazali, R.: Hussain, A. J. & Liatsis, P. (2011). Dynamic Ridge Polynomial Neural Network:

Hussain, A. J. & Liatsis, P. (2002). Recurrent Pi-Sigma Networks for DPCM Image Coding.

Ibrahim, D. (2002). Temperature and its Measurement. In *Microcontroller Based Temperature* 

Jordan, M. I. (1986). *Attractor Dynamics and Parallelism in a Connectionist Sequential Machine.*

Lorenc, A. C. (1986). Analysis Methods for Numerical Weather Prediction. *Quarterly Journal* 

Mielke, A. (2008). Possibilities and Limitations of Neural Networks. Retrieved August 24,

Paras, Mathur, S., Kumar, A. & Chandra, M. (2007). *A Feature Based Neural Network Model for* 

Radhika, Y. & Shashi, M. (2009). Atmospheric Temperature Prediction using Support

Rumelhart, D. E.: Hinton, G. E. & Williams, R. J. (1986). Learning Representations by Back-

Shin, Y. & Ghosh, J. (1991-a). The Pi-Sigma Networks: An Efficient Higher-order Neural

Shin, Y. & Ghosh, J. (1991-b). Realization of Boolean Functions using Binary Pi-Sigma

*International Joint Conference on Neural Networks, Vol.1*, pp.13-18.

meteorological variables for evaporation estimation by self-organizing map neural

Polynomial Neural Networks to Financial Time Series Prediction. In *Artificial Higher Order Neural Networks for Economics and Business* (pp. 271-293). Hershey,

Networks to Financial Time Series Prediction. *In Proceedings of the International Joint Conference on Neural Networks, IJCNN 2006,* part of the IEEE World Congress on Computational Intelligence, WCCI 2006, Vancouver, BC, Canada, 16-21 July 2006*.* 

Forecasting the univariate non-stationary and stationary trading signals. *Elsevier:* 

Paper presented at the Proceedings of the Eighth Conference of the Cognitive

*Weather Forecasting.* Paper presented at the Proceedings of World Academy of

Vector Machines. *International Journal of Computer Theory and Engineering, 1*(1),

Network for Pattern Classification and Function Approximation. *Proceedings of* 

Networks. In Kumara & Shin, (Ed.), *Intelligent Engineering Systems Through Artificial* 

Barry, R. & Chorley, R. (1982). *Atmosphere, Weather, and Climate*: Methuen.

network. *Journal of Hydrology, 384*(1-2), 118-129.

New York: Information Science Reference.

*Expert Systems with Applications.* 38, pp. 3765-3776.

*Monitoring & Control* (pp. 55-61). Oxford: Newnes.

*of the Royal Meteorological Society, 112*(474), pp. 1177-1194.

2009, from http://www.andreas-mielke.de/nn-en-1.html

Fig. 9. Temperature Forecast made by MLP on Out-of sample Dataset.
