4. Discussions

The experiment results showed the DBN composed by multiple RBMs and MLP is the state-of-the-art predictor comparing to all conventional methods in the case of CATS data. Furthermore, the training method for DBN may be more efficient by the RL method SGA for real time series data than using the conventional BP algorithm. Here let us glance back at the development of this useful deep learning method.

• Why the DBN composed by multiple RBMs and MLP [11, 13] is better than the DBN with multiple RBMs only [9]?

The output of the last RBM of DBN, a hidden unit of the last RBM in DBN, has a binary value during pretraining process. So the weights of connections between the unit and units of the visible layer of the last RBM are affected and with lower complexity than using multiple units with continuous values, i.e., MLP, or so-called full connections in deep learning architecture.

• How are RL methods active at ANN training?

In 1992, Williams proposed to adopt a RL method named REINFORCE to modify artificial neural networks [8]. In 2008, Kuremoto et al. showed the RL method SGA is more efficient than the conventional BP method in the case of time series forecasting [6]. Recently, researchers in DeepMind Ltd. adopted RL into deep neural networks and resulted a famous game software AlphaGo [20–23].

• Why SGA is more efficient than BP?

Generally, the training process for ANN by BP uses mean square error as loss function. So every sample data affects the learning process and results including noise data. Meanwhile, SGA uses reward which may be an error zone to modify the

• CO2: Atmospheric CO2 from continuous air samples weekly averages

Change of the learning error during fine-tuning (CATS data [1–980]).

The number of RBMs 3 1 Learning rate of RBM 0.048-0.055-0.026 0.042 Structure of DBN (the number of units and layers) 14-14-18-19-18-2 5-11-2-1 Learning rate of SGA or BP 0.090 0.090 Discount factor γ 0.082 — Coefficient β 1.320 —

The long-term forecasting error comparison of different methods using CATS data.

Method E<sup>1</sup> DBN(SGA) [18] 170 DBN(BP) + ARIMA [14] 244 DBN [11] (BP) 257 Kalman Smoother (the best of IJCNN '04) [4] 408 DBN [9] (2 RBMs) 1215 MLP [9] 1245 A hierarchical Bayesian learning (the worst of IJCNN '04) [4] 1247 ARIMA [1] 1715 ARIMA+MLP(BP) [12] 2153 ARIMA+DBN(BP) [14] 2266

2225 data

Figure 8.

48

Table 2.

Table 1.

A.D. 1882–1998, 1300 data

Meta-parameters of DBN used for the CATS data (block 1).

Time Series Analysis - Data, Methods, and Applications

atmospheric CO2 concentration derived from continuous air samples, Hawaii,

DBN with SGA DBN with BP

• Sea level pressures: Monthly values of the Darwin sea level pressure series,

#### Figure 9.

Prediction results by DBN with BP and SGA. (a) Prediction result of CO2 data. (b) Prediction result of Sea level pressure data. (c) Prediction result of Sun spot number data.

parameters of model. So it has higher robustness for the noisy data and unknown

DBN with BP (the number of units)

1400 400 16-18-18-1 16-20-8-7-2

3078 578 20-20-17-18-1 19-19-20-10-2

CO2 2225 225 15-17-17-1 20-18-7-2

DBN with SGA (the number of units)

Data DBN with BP DBN with SGA CO2 0.2671 0.2047 Sea level pressure 0.9902 0.9003 Sun spot number 733.51 364.05

Training Deep Neural Networks with Reinforcement Learning for Time Series Forecasting

A deep belief net (DBN) composed by multiple restricted Boltzmann machines (RBMs) and multilayer perceptron (MLP) for time series forecasting were introduced in this chapter. The training method of DBN is also discussed as well as a reinforcement learning (RL) method; stochastic gradient ascent (SGA) showed its priority to the conventional error back-propagation (BP) learning method. The robustness of SGA comes from the utilization of relaxed prediction error during the

data for real problems.

Data series Total

data

5. Conclusions

Table 3.

Figure 10.

Sea level pressure

Sun spot number

Table 4.

51

Prediction MSE of real time series data [17].

DOI: http://dx.doi.org/10.5772/intechopen.85457

Changes of learning error by random search for DBN with SGA.

Meta-parameters of DBN used for real time series forecasting.

Testing data

Training Deep Neural Networks with Reinforcement Learning for Time Series Forecasting DOI: http://dx.doi.org/10.5772/intechopen.85457


Table 3.

Prediction MSE of real time series data [17].

Figure 10. Changes of learning error by random search for DBN with SGA.


#### Table 4.

Meta-parameters of DBN used for real time series forecasting.

parameters of model. So it has higher robustness for the noisy data and unknown data for real problems.
