**1. Introduction**

Optimal control problems are existing in engineering and natural sciences for so long, and the applications of the optimal control have been well defined in the literature [1–4]. With the rapid evolution of computer technology, the development of the optimal control techniques is reached a mature level, from classical control to modern control, from proportional-integral-derivative (PID) control to feedback control, and from adaptive control to intelligent control [5–8]. The studies in the optimal control field are still progressing, and attract the interest of, not only engineers and applied mathematicians but also biologists and financialists, to investigate and contribute to the optimal control theory.

In particular, the optimal control algorithm, which integrates system optimization and parameter estimation, gives a new insight into the control community. This algorithm is known as the integrated system optimization and parameter estimation (ISOPE), and its dynamic version is called the dynamic ISOPE (DISOPE). Both of

these algorithms were introduced by Robert [9–11], and Robert and Becerra [12–14], respectively. The basic idea of DISOPE is applying the model-based optimal control, which has different structures and parameters compared to the original optimal control problem, to obtain the correct optimal solution of the original optimal control problem, in spite of model-reality differences. Recently, this algorithm has been extended to cover both deterministic and stochastic versions, and it is known as an integrated optimal control and parameter estimation (IOCPE) algorithm [15, 16]. On the other hand, the application of the optimization techniques, particularly, using the conjugate gradient method for solving the optimal control problem [17–19] has also been studied, where the open-loop control strategy is concerned [3, 8].

where *u k*ð Þ<sup>∈</sup> *<sup>ℜ</sup>m*, *<sup>k</sup>* <sup>¼</sup> 0, 1, <sup>⋯</sup>, *<sup>N</sup>* � 1, and *x k*ð Þ<sup>∈</sup> *<sup>ℜ</sup>n*, *<sup>k</sup>* <sup>¼</sup> 0, 1, <sup>⋯</sup>, *<sup>N</sup>*, are the control sequences and the state sequences, respectively, while *<sup>f</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup><sup>m</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup><sup>n</sup>* represents the real plant, *<sup>L</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup><sup>m</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup>* is the cost under summation, and *<sup>φ</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup>* is the terminal cost. Here, *<sup>J</sup>*<sup>0</sup> is the scalar cost function, and *<sup>x</sup>*<sup>0</sup> is the known initial state vector. It is assumed that all functions in (1) are continu-

*Conjugate Gradient Approach for Discrete Time Optimal Control Problems with Model-Reality…*

This problem, which is referred to as Problem (P), is regarded as the real optimal control problem. Due on the complex and nonlinear structure, solving Problem (P) actually requires the efficient computation techniques. For this reason, the simplified model of Problem (P) is identified to be solved such that the true optimal solution of Problem (P) could be approximated. Hence, this simplified model-based

*N*�1

1 2 *x k*ð ÞT*Qx k*ð Þþ *u k*ð ÞT*Ru k*ð Þ � � <sup>þ</sup> *<sup>γ</sup>*ð Þ*<sup>k</sup>*

*x k*ð Þ<sup>T</sup>*Qx k*ð Þþ *u k*ð Þ<sup>T</sup>*Ru k*ð Þ � � <sup>þ</sup> *<sup>γ</sup>*ð Þ*<sup>k</sup>*

(2)

(3)

*k*¼0

where *γ*ð Þ*k* , *k* ¼ 0, 1, ⋯, *N*, and *α*ð Þ*k* , *k* ¼ 0, 1, ⋯, *N* � 1, are introduced as the adjusted parameters, whereas *A* is an *n* � *n* transition matrix, and *B* is an *n* � *m* control coefficient matrix. Besides, *S N*ð Þ and *Q* are *n* � *n* positive semi-definite matrices, and *R* is a *m* � *m* positive definite matrix. Here, *J*<sup>1</sup> is the scalar cost function. Let this problem is referred to as Problem (M). It can be seen that, because of the different structures and parameters, only solving Problem (M) would not obtain the

Now, an expanded optimal control problem, which combines the real plant and the cost function in Problem (P) into Problem (M) and is referred to as Problem

*N*�1

1 2

*<sup>r</sup>*2k k *x k*ð Þ� *z k*ð Þ <sup>2</sup>

*k*¼0

1 2

<sup>2</sup> *z k*ð Þ<sup>T</sup>*Qz k*ð Þþ *v k*ð Þ<sup>T</sup>*Rv k*ð Þ � � <sup>þ</sup> *<sup>γ</sup>*ð Þ¼ *<sup>k</sup> Lzk* ð Þ ð Þ, *v k*ð Þ, *<sup>k</sup>*

optimal solution of Problem (P) for not taking the adjusted parameters into account. Notice, adding the adjusted parameters into Problem (M) could let us calculate the differences between the real plant and the model used. On this basis, Problem (M) would be solved iteratively to give the correct optimal solution of

ously differentiable with respect to their respective arguments.

*x N*ð ÞT*S N*ð Þ*x N*ð Þþ *<sup>γ</sup>*ð Þþ *<sup>N</sup>* <sup>X</sup>

optimal control problem is defined as follows:

*DOI: http://dx.doi.org/10.5772/intechopen.89711*

*x k*ð Þ¼ þ 1 *Ax k*ð Þþ *Bu k*ð Þþ *α*ð Þ*k* , *x*ð Þ¼ 0 *x*<sup>0</sup>

Problem (P), in spite of model-reality differences.

**3. System optimization with parameter estimation**

*x N*ð Þ<sup>T</sup>*S N*ð Þ*x N*ð Þþ *<sup>γ</sup>*ð Þþ *<sup>N</sup>* <sup>X</sup>

*z N*ð Þ<sup>T</sup>*S N*ð Þ*z N*ð Þþ *<sup>γ</sup>*ð Þ¼ *<sup>N</sup> <sup>φ</sup>*ð Þ *z N*ð Þ, *<sup>N</sup>*

*Az k*ð Þþ *Bv k*ð Þþ *α*ð Þ¼ *k f zk* ð Þ ð Þ, *v k*ð Þ, *k*

*<sup>r</sup>*1k k *u k*ð Þ� *v k*ð Þ <sup>2</sup> <sup>þ</sup>

*x k*ð Þ¼ þ 1 *Ax k*ð Þþ *Bu k*ð Þþ *α*ð Þ*k* , *x*ð Þ¼ 0 *x*<sup>0</sup>

1 2

min *u k*ð Þ *<sup>J</sup>*1ð Þ¼ *<sup>u</sup>*

subject to

(E), is introduced by

1 2

> þ 1 2

1 2

1

*v k*ð Þ¼ *u k*ð Þ *z k*ð Þ¼ *x k*ð Þ

min *u k*ð Þ *<sup>J</sup>*2ð Þ¼ *<sup>u</sup>*

**83**

subject to

In this chapter, the conjugate gradient approach [17, 19] is employed to solve the linear model-based optimal control problem for obtaining the optimal solution of the original optimal control problem. In our approach, the simplified model, which is adding the adjusted parameters, is formulated initially. Then, an expanded optimal control problem, which combines the system dynamic and the cost function from the original optimal control problem into the simplified model, is introduced. By defining the Hamiltonian function and the augmented cost function, the corresponding necessary conditions for optimality are derived. Among these necessary conditions, a set of necessary conditions is for the modified model-based optimal control problem, a set of necessary conditions defines the parameter estimation problem, and a set of necessary conditions calculates the multipliers [15].

By virtue of the modified model-based optimal control problem, an equivalence optimization problem is defined, and the related gradient function is determined. With an initial control sequence, the initial gradient and the initial search direction are computed. Then, the control sequences are updated through the line search technique, where the gradient and search direction would satisfy the conjugacy condition [17, 18]. During the iteration, the state and the costate are updated by the control sequence obtained from the conjugate gradient approach. When the convergence is achieved within a tolerance given, the iterative solution approximates to the correct optimal solution of the original optimal control problem, in spite of model-reality differences. For illustration, examples of linear and nonlinear cases, which are damped harmonic oscillator [7] and continuous stirred-tank chemical tank [8], are studied.

The chapter is organized as follows. In Section 2, the problem statement is described in detail, where the original optimal control problem and the simplified model are discussed. In Section 3, the methodology used is further explained. The necessary conditions for optimality are derived, and the use of the conjugate gradient method is delivered in solving the equivalence optimization problem. In Section 4, examples of a damped harmonic oscillator and a continuous stirred-tank chemical reactor are studied. The results show the efficiency of the algorithm proposed. Finally, concluding remarks are made.

### **2. Problem statement**

Consider a general class of the discrete-time nonlinear optimal control problem, given by

$$\min\_{\mathfrak{u}(k)} J\_0(u) = \varrho(\mathfrak{x}(N), N) + \sum\_{k=0}^{N-1} L(\mathfrak{x}(k), \mathfrak{u}(k), k) \tag{1}$$

subject to

$$\varkappa(k+1) = f(\varkappa(k), \varkappa(k), k), \ \varkappa(0) = \varkappa\_0$$

*Conjugate Gradient Approach for Discrete Time Optimal Control Problems with Model-Reality… DOI: http://dx.doi.org/10.5772/intechopen.89711*

where *u k*ð Þ<sup>∈</sup> *<sup>ℜ</sup>m*, *<sup>k</sup>* <sup>¼</sup> 0, 1, <sup>⋯</sup>, *<sup>N</sup>* � 1, and *x k*ð Þ<sup>∈</sup> *<sup>ℜ</sup>n*, *<sup>k</sup>* <sup>¼</sup> 0, 1, <sup>⋯</sup>, *<sup>N</sup>*, are the control sequences and the state sequences, respectively, while *<sup>f</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup><sup>m</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup><sup>n</sup>* represents the real plant, *<sup>L</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup><sup>m</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup>* is the cost under summation, and *<sup>φ</sup>* : *<sup>ℜ</sup><sup>n</sup>* � *<sup>ℜ</sup>* ! *<sup>ℜ</sup>* is the terminal cost. Here, *<sup>J</sup>*<sup>0</sup> is the scalar cost function, and *<sup>x</sup>*<sup>0</sup> is the known initial state vector. It is assumed that all functions in (1) are continuously differentiable with respect to their respective arguments.

This problem, which is referred to as Problem (P), is regarded as the real optimal control problem. Due on the complex and nonlinear structure, solving Problem (P) actually requires the efficient computation techniques. For this reason, the simplified model of Problem (P) is identified to be solved such that the true optimal solution of Problem (P) could be approximated. Hence, this simplified model-based optimal control problem is defined as follows:

$$\begin{aligned} \min\_{\mathbf{x}(k)} J\_1(\mathbf{u}) &= \frac{1}{2} \mathbf{x}(N)^T \mathbf{S}(N) \mathbf{x}(N) + \boldsymbol{\gamma}(N) + \sum\_{k=0}^{N-1} \frac{1}{2} \left( \mathbf{x}(k)^T \mathbf{Q} \mathbf{x}(k) + \boldsymbol{u}(k)^T \mathbf{R} \mathbf{u}(k) \right) + \boldsymbol{\gamma}(k) \\ \text{subject to} \\ \mathbf{x}(k+1) &= A \mathbf{x}(k) + B \mathbf{u}(k) + \boldsymbol{a}(k), \ \mathbf{x}(0) = \mathbf{x}\_0 \end{aligned} \tag{2}$$

where *γ*ð Þ*k* , *k* ¼ 0, 1, ⋯, *N*, and *α*ð Þ*k* , *k* ¼ 0, 1, ⋯, *N* � 1, are introduced as the adjusted parameters, whereas *A* is an *n* � *n* transition matrix, and *B* is an *n* � *m* control coefficient matrix. Besides, *S N*ð Þ and *Q* are *n* � *n* positive semi-definite matrices, and *R* is a *m* � *m* positive definite matrix. Here, *J*<sup>1</sup> is the scalar cost function.

Let this problem is referred to as Problem (M). It can be seen that, because of the different structures and parameters, only solving Problem (M) would not obtain the optimal solution of Problem (P) for not taking the adjusted parameters into account. Notice, adding the adjusted parameters into Problem (M) could let us calculate the differences between the real plant and the model used. On this basis, Problem (M) would be solved iteratively to give the correct optimal solution of Problem (P), in spite of model-reality differences.
