3. Prediction of PU channel state based on hidden Markov and Markov switching model

In this part, the system model for forecasting the near future of PU channel state is divided into three models: (1) the model detecting the PU channel state (i.e., PU signal present or PU signal) which follows the conventional single-user energy detection (i.e., fusion techniques mentioned in Section 2.1 can also be considered here); (2) the model that generates a time series to capture PU channel state based on the detection sequence; and (3) the model for predicting the generated time series used to capture PU channel state based on hidden Markov model (HMM) and Markov switching model (MSM). The block diagram in Figure 5. illustrates these three models.

The PU channel state detection model can be written using Eq. (4); by giving probability of false alarm Pf, the detection threshold for single-user energy detector can be written as:

$$
\lambda = \left(\sqrt{\frac{2}{N}} \mathbf{Q}^{-1}(\mathbf{P}\_{\mathbf{f}}) + 1\right) \sigma\_{\mathbf{u}}^{\;2} \tag{24}
$$

where Q�<sup>1</sup> ð Þ: is the inverse of the Q ð Þ: function.

And the decision of the sensing (i.e., PU detection sequence) over the time can be written as follows:

$$\mathbf{D}\_{\mathbf{t}} = \begin{cases} \begin{array}{c} \text{"0"} \\ \text{ $^\cdot$ T} \end{array} \begin{array}{c} \text{PU absent} \\ \text{ $^\cdot$ T $} \end{array} \begin{array}{c} \text{Y}\_{\mathbf{t}} < \lambda \\ \text{$ ^\cdot $T$ } \end{array} \mathbf{1} \leq \mathbf{t} \leq \mathbf{T} \end{array} \tag{25}$$

#### 3.1. Time series generation model

for computing the single-user threshold). We can notice from both Figure 4. and Table 2 that KNN and decision tree classifier perform better than Naïve Bayes and SVM classifier in terms of

Table 3 shows the accuracy, precession, and the recall for decision tree classifier when used to classify 3000 frames after training it over a set containing 1000 frames for the same cognitive system used to generate Figure 3. The single-user threshold is used for training the classifier. The simulation was run with different number of samples for energy detection process. It is clear from the table that decision tree can classify all of the 3000 frames correctly or achieve 100% detection rate using only 200 samples for the energy detection process. And, due to the fact that the sensing time is proportional to the number of samples taken by energy detector, a less number of samples used for energy detection leads to less sensing time. Thus, when we use machine-learning-based fusion, such as decision tree or KNN, we can reduce the sensing time from 200 to 40 μs for 5 MHz bandwidth channel as an example, while we still achieve

Classifier Accuracy (%) Precession (%) Recall (%) KNN 100 100 100 Decision Tree 100 100 100 Naïve Bayes 98.9 100 91.2 SVM 97.6 83.9 100

Table 2. The accuracy, precession and recall of KNN, SVM, NB, and DT classifiers used in classifying 1000 new frames

Number of samples Accuracy (%) Precession (%) Recall (%) 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

3. Prediction of PU channel state based on hidden Markov and Markov

Table 3. The accuracy, precession, and recall for decision tree classifier used in classifying 3000 frames for different

In this part, the system model for forecasting the near future of PU channel state is divided into three models: (1) the model detecting the PU channel state (i.e., PU signal present or PU signal) which follows the conventional single-user energy detection (i.e., fusion techniques mentioned

the accuracy of classifying the new frames.

128 Machine Learning - Advanced Techniques and Emerging Applications

100% detection rate of the spectrum hole.

after being trained with 1000 frames.

switching model

number of samples.

Given PU channel state detection sequence over the time (i.e., PU absent, PU present), if we denote the period that the PU is inactive as "idle state," and the period that PU is active as "occupied state," our goal now is to predict when the detection sequence Dt will change from one state to another (i.e., "idle" to "occupied "or vice versa) before that happens so that the secondary user can avoid interfering with primary user transmission. For this reason, we generate a time series zt to map each state of the detection sequence Dt (i.e., "PU present" and "PU absent") into another observation space using two different random variable distributions for each state (i.e., zt ∈f g v1; v2…vL represents PU absent or idle state and zt ∈f g vLþ<sup>1</sup>…:vM represents PU occupied or present), the time series zt can be written as

$$\mathbf{z}\_{\mathbf{t}} \in \begin{cases} \{\mathbf{v}\_{\mathrm{1}}, \mathbf{v}\_{\mathrm{2}}, \dots, \mathbf{v}\_{\mathrm{L}}\} & \mathbf{Y}\_{\mathrm{t}} < \lambda \\\ \{\mathbf{v}\_{\mathrm{L}+1}, \dots, \mathbf{v}\_{\mathrm{M}}\} & \mathbf{Y}\_{\mathrm{t}} \ge \lambda \end{cases} \tag{26}$$

Now, supposing that we have given observations value O ¼ O1; O2; Ot f g ;…OT , Ot ∈ f g v1; v2…vM and a PU channel state at time step t, Xt ∈ si, i ¼ 1, 2…:K, si ∈f g 0; 1 (i.e., 0 for

Figure 5. Block diagram of PU channel state prediction model.

PU idle and 1 PU occupied state), and we want to estimate the channel state at one time step ahead of the current state Xtþ1. We can solve this problem using hidden Markov model Viterbi algorithm [15].

#### 3.2. Primary users channel state estimation based on hidden Markov model

The generic HMM model can be illustrated by Figure 6.—in this figure, X ¼ X1;Xt f g ; …XT represents the hidden state sequence, where Xt ∈f g s1; s2; …; sK , K represents the number of hidden states or Markov chain and O ¼ O1; Ot f g ;…; OT represents the observation sequence where Ot ∈f g v1; v2;…; vM and M is the number of the observations in the observation space. A and B represent the transition probabilities matrix and the emission probabilities matrix, respectively, while π denotes the initial state probability vector. HMM can be defined by θ ¼ ð Þ π; A; B (i.e., the initial state probabilities, the transition probabilities, and emission probabilities) [15].

Initial state probabilities for HMM can be written as

$$
\boldsymbol{\pi} = (\pi\_1, \pi\_2, \pi\_i \dots \quad \pi\_\mathbb{K})
$$

$$
\pi\_\mathbf{i} = \begin{array}{c} \mathbf{P}(\mathbf{X}\_1 = \mathbf{s}\_i) \quad \mathbf{i} = \mathbf{1}, \mathbf{2}, \dots, \mathbf{K} \end{array} \tag{27}
$$

<sup>B</sup> <sup>¼</sup> b11 b12 b13 b21 b22 b23

Viterbi algorithm to solve the problem mentioned in subsection (3.1) as follows:

ð Þi .

ψt

δtðÞ¼ i max X1, …, Xt�<sup>1</sup>

ð Þi .

And we let ψ<sup>t</sup>

state qT

1) step 1 initializes δtð Þi and ψ<sup>t</sup>

Figure 6. Hidden Markov model.

<sup>∗</sup> at time T as

2) step 2 iterates to update δtð Þi and ψ<sup>t</sup>

δtðÞ¼j max 1 ≤ i ≤ K

> ðÞ¼j argmax 1 ≤ i ≤ K

ψt

b14 b15 b16

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks

http://dx.doi.org/10.5772/intechopen.74599

131

ð Þi to be a vector that stores the arguments that maximize Eq. (30), we can write

P X1, …, Xt ¼ si; O1, …, Ot f g ð Þ jθ (30)

ðÞ¼ i 0, i ¼ 1, …, K (31)

<sup>δ</sup><sup>t</sup>�<sup>1</sup>ð Þ<sup>i</sup> aij � � , <sup>t</sup> <sup>¼</sup> <sup>2</sup>, …, <sup>T</sup>, <sup>j</sup> <sup>¼</sup> <sup>1</sup>, …, <sup>K</sup> (33)

<sup>1</sup> <sup>≤</sup> <sup>i</sup> <sup>≤</sup> <sup>K</sup> ½ � <sup>δ</sup>Tð Þ<sup>i</sup> (34)

½ � δTð Þi (35)

<sup>δ</sup><sup>t</sup>�<sup>1</sup>ð Þ<sup>i</sup> aij � �bjð Þ Ot , <sup>t</sup> <sup>¼</sup> <sup>2</sup>, …, <sup>T</sup>, <sup>j</sup> <sup>¼</sup> <sup>1</sup>, …, <sup>K</sup> (32)

b24 b25 b26 !

Now, for the problem we describe in subsection (3.1), if we assume that HMM parameters θ ¼ ð Þ π; A; B and the observations value O ¼ O1; O2 Ot f g ;…OT are given. If we assume that the maximum probability of state sequence t that end at state i to be equal to δtð Þi where

δtðÞ¼ i πibið Þ O1

3) step 3 terminates the update and calculates the likelihood probability P<sup>∗</sup> and the estimated

<sup>P</sup><sup>∗</sup> <sup>¼</sup> max

<sup>∗</sup> <sup>¼</sup> argmax 1 ≤ i ≤ K

In the above case, HMM parameters θ ¼ ð Þ π; A; B are unknown and need to be estimated. We

qT

estimate these parameters statistically using Baum-Welch algorithm [16].

For a HMM model with two hidden states i ¼ 2 ,

$$\pi = (\pi\_1 \pi\_2)$$

And the transition probabilities can be written as,

$$\mathbf{A} = \left(\ \mathbf{a}\_{\overline{\mathbf{\boldsymbol{\eta}}}}\right)\_{\mathbf{K} \times \mathbf{K}}$$

$$\mathbf{a}\_{\overline{\mathbf{\boldsymbol{\eta}}}} = \mathbf{P}\left(\mathbf{X}\_{t+1} = \mathbf{s}\_{\overline{\mathbf{\boldsymbol{\eta}}}} \,|\, \mathbf{X}\_{t} = \mathbf{s}\_{\mathbf{i}}\right) \quad \text{i.i.j} = \mathbf{1}, \ldots, \mathbf{K} \tag{28}$$

where aij is the probability that next state equal sj when current state is equal to si. For HMM model with two states, the matrix A can be written as

$$\mathbf{A} = \begin{pmatrix} \mathbf{a}\_{00} & \mathbf{a}\_{01} \\ \mathbf{a}\_{10} & \mathbf{a}\_{11} \end{pmatrix}.$$

The emission probabilities matrix for HMM model is written as

$$\mathbf{B} = \left(\mathbf{b}\_{\text{jm}}\right)\_{\text{K}\times\text{M}}$$

$$\mathbf{b}\_{\text{j}}(\mathbf{m}) = \mathbf{P}(\mathbf{O}\_{\text{t}} = \mathbf{v}\_{\text{m}} | \mathbf{X}\_{\text{t}} = \mathbf{s}\_{\text{j}}) \triangleq \mathbf{b}\_{\text{j}}(\mathbf{O}\_{\text{t}}), \quad \mathbf{j} = 1...\text{K} \text{ , m} = 1,...\text{M} \tag{29}$$

B and bj represent the probability that current observation is vm when current state is sj. For example, in an HMM model with M ¼ 6 and K ¼ 2, B is written as

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks http://dx.doi.org/10.5772/intechopen.74599 131

Figure 6. Hidden Markov model.

PU idle and 1 PU occupied state), and we want to estimate the channel state at one time step ahead of the current state Xtþ1. We can solve this problem using hidden Markov model Viterbi

The generic HMM model can be illustrated by Figure 6.—in this figure, X ¼ X1;Xt f g ; …XT represents the hidden state sequence, where Xt ∈f g s1; s2; …; sK , K represents the number of hidden states or Markov chain and O ¼ O1; Ot f g ;…; OT represents the observation sequence where Ot ∈f g v1; v2;…; vM and M is the number of the observations in the observation space. A and B represent the transition probabilities matrix and the emission probabilities matrix, respectively, while π denotes the initial state probability vector. HMM can be defined by θ ¼ ð Þ π; A; B (i.e., the initial state probabilities, the transition probabilities, and emission prob-

π ¼ ð Þ π1; π2;πi…: π<sup>K</sup>

π ¼ ð Þ π1π<sup>2</sup>

where aij is the probability that next state equal sj when current state is equal to si. For HMM

<sup>A</sup> <sup>¼</sup> a00 a01 a10 a11 

> B ¼ bjm

B and bj represent the probability that current observation is vm when current state is sj. For

K�M

bjð Þ¼ m P Oð <sup>t</sup> ¼ vmjXt ¼ sjÞ≜ bjð Þ Ot , j ¼ 1…K , m ¼ 1, …::M (29)

A ¼ aij K�K

aij ¼ P Xtþ<sup>1</sup> ¼ sjj Xt ¼ si

π<sup>i</sup> ¼ P Xð Þ <sup>1</sup> ¼ si , i ¼ 1, 2, …, K (27)

, <sup>i</sup>, <sup>j</sup> <sup>¼</sup> <sup>1</sup>, …::, <sup>K</sup> (28)

3.2. Primary users channel state estimation based on hidden Markov model

Initial state probabilities for HMM can be written as

130 Machine Learning - Advanced Techniques and Emerging Applications

For a HMM model with two hidden states i ¼ 2 ,

And the transition probabilities can be written as,

model with two states, the matrix A can be written as

The emission probabilities matrix for HMM model is written as

example, in an HMM model with M ¼ 6 and K ¼ 2, B is written as

algorithm [15].

abilities) [15].

$$\mathbf{B} = \begin{pmatrix} \mathbf{b}\_{11} & \mathbf{b}\_{12} \ \mathbf{b}\_{13} & \mathbf{b}\_{14} & \mathbf{b}\_{15} \ \mathbf{b}\_{16} \\\ \mathbf{b}\_{21} & \mathbf{b}\_{22} \ \mathbf{b}\_{23} & \mathbf{b}\_{24} & \mathbf{b}\_{25} \ \mathbf{b}\_{26} \end{pmatrix}$$

Now, for the problem we describe in subsection (3.1), if we assume that HMM parameters θ ¼ ð Þ π; A; B and the observations value O ¼ O1; O2 Ot f g ;…OT are given. If we assume that the maximum probability of state sequence t that end at state i to be equal to δtð Þi where

$$\mathbf{S}\_{\mathbf{t}}(\mathbf{i}) = \max\_{\mathbf{X}\_{\mathbf{t}},...,\mathbf{X}\_{\mathbf{t}-1}} \left\{ \mathbf{P}(\mathbf{X}\_{\mathbf{1}},...,\mathbf{X}\_{\mathbf{t}} = \mathbf{s}\_{\mathbf{i}}; \mathbf{O}\_{\mathbf{1}},..., \mathbf{O}\_{\mathbf{t}} | \boldsymbol{\Theta} \mid) \right\} \tag{30}$$

And we let ψ<sup>t</sup> ð Þi to be a vector that stores the arguments that maximize Eq. (30), we can write Viterbi algorithm to solve the problem mentioned in subsection (3.1) as follows:

1) step 1 initializes δtð Þi and ψ<sup>t</sup> ð Þi .

$$\mathfrak{d}\_{\mathbf{i}}(\mathbf{i}) = \pi\_{\mathbf{i}} \mathfrak{b}\_{\mathbf{i}}(\mathbf{O}\_{\mathbf{l}})$$

$$\psi\_{\mathbf{i}}(\mathbf{i}) = \mathbf{0}, \quad \mathbf{i} = \mathbf{1}, \ldots, \mathbf{K} \tag{31}$$

2) step 2 iterates to update δtð Þi and ψ<sup>t</sup> ð Þi .

$$\mathsf{St}\_{\mathsf{I}}(\mathsf{j}) = \max\_{1 \le i \le \mathsf{K}} \left[ \mathsf{S}\_{\mathsf{I}-1}(\mathsf{i}) \mathsf{a}\_{\overline{\mathsf{i}}} \right] \mathsf{b}\_{\overline{\mathsf{i}}}(\mathsf{O}\_{\mathsf{I}}) \quad , \mathsf{t} = \mathsf{2}, \dots, \mathsf{T}, \quad \mathsf{j} = 1, \dots, \mathsf{K} \tag{32}$$

$$\boldsymbol{\psi}\_{\mathbf{t}}(\mathbf{j}) = \underset{\mathbf{1} \le \mathbf{i} \le \mathbf{K}}{\operatorname{argmax}} \; \left[ \mathbf{\bar{s}}\_{\mathbf{t}-1}(\mathbf{i}) \mathbf{a}\_{\mathbf{i}\hat{\mathbf{j}}} \right] \; \text{ , } \mathbf{t} = \mathbf{2} \; \dots \; \mathbf{T}, \mathbf{j} = \mathbf{1} \; \dots \; \mathbf{K} \tag{33}$$

3) step 3 terminates the update and calculates the likelihood probability P<sup>∗</sup> and the estimated state qT <sup>∗</sup> at time T as

$$\mathbf{P}^\* = \max\_{1 \le i \le \mathbf{K}} \left[ \delta\_{\mathbf{T}}(\mathbf{i}) \right] \tag{34}$$

$$\mathbf{q}\_{\Gamma}^{\*} = \underset{1 \le i \le K}{\text{argmax}} \; [\delta\_{\Gamma}(\mathbf{i})] \tag{35}$$

In the above case, HMM parameters θ ¼ ð Þ π; A; B are unknown and need to be estimated. We estimate these parameters statistically using Baum-Welch algorithm [16].

#### 3.2.1. Hidden Markov model parameters estimation using Baum-Welch algorithm

If we assume that we have given some training observations with length L O1; O2 Ot f g ; …OL and want to approximate HMM parameters θ ¼ ð Þ π; A; B from them, we can use maximum likelihood estimation. In order to do that, we define γ<sup>t</sup> ð Þi to be the probability of being in state si at time t, given t Ot ,t ¼ 1, 2…L . γ<sup>t</sup> ð Þi is written as

$$\gamma\_t(\mathbf{i}) = \mathbf{P}(\mathbf{X}\_t = \mathbf{s}\_{\mathbf{i}} \mid \mathbf{O}\_1, \dots, \mathbf{O}\_L, \boldsymbol{\Theta}) \tag{36}$$

<sup>b</sup>bið Þ¼ <sup>m</sup> Expected number of times in state sj and observing vm

¼

The estimation algorithm can be summarized in the following steps:

ð Þ<sup>i</sup> � � and E <sup>ζ</sup><sup>t</sup> ð Þ ð Þ <sup>i</sup>; <sup>j</sup> from Eqs. (38) and (39)

parameters {π, A, B } with the help of Eqs. (31), (34), and (35) by setting T ¼ 1:

zt � <sup>μ</sup>z0; <sup>σ</sup>z0

(

μz1; σz1

3.3.1. Derivation of Markov switching model for Gaussian regime switching time series

3.3. Primary users channel state estimation based on Markov switching model

5. Compute according to 5 the new estimate of aij, bið Þ k , πi, and call them θ (k + 1)

1. Get your observations O1 O2, …OL,

2. Set a guess of your first θ estimate θ ð Þ1 , k ¼ 1

3. Compute θ ð Þ k based on O1 O2, …OL and

4. Compute E γ<sup>t</sup>

μz0; σz0

6. Go to 3 if not converged.

<sup>2</sup> � � or N � <sup>μ</sup>z1; <sup>σ</sup>z1

series and estimate its parameters.

occupied." We can rewrite Eq. (26) as follows:

γt

P<sup>L</sup> t ¼ 1 Ot ¼ vm

> P<sup>L</sup> <sup>t</sup>¼<sup>1</sup> γ<sup>t</sup>

Update k ¼ k þ 1

ð Þi , ζtð Þ i; j ∀1 ≤ t ≤ L, ∀1 ≤ i ≤ K, ∀1 ≤ j ≤ K

The prediction for a one-step ahead PU channel state can be done based on the trained

An alternative way to estimate PU channel state is to use Markov switching model (MSM). For the time series in Eq. (26), we assume that zt obeys two different Gaussian distributions N �

<sup>2</sup> � � Yt < λ

<sup>2</sup> � � Yt ≥ λ

It is obvious that Eq. (43) represents a two-state Gaussian regime switching time series which can be modeled using MSM [17]. In order to estimate the switching time of one state ahead of the current state for this time series, we need to derive MSM regression model for the time

A simple Markov switching regression model to describe the two-state Gaussian regime switching time series is given in Eq. (43). This model can be written by following Ref [17] as

<sup>2</sup> � � based on the sensed PU channel state "PU channel idle" or "PU

1 ≤ t ≤ T

(43)

Expected number of times in state sj

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks

γt ð Þi

ð Þ<sup>i</sup> (42)

http://dx.doi.org/10.5772/intechopen.74599

133

We also define ζtð Þ i; j to be the probability of being in state si at time t and transiting to state sj at time t þ 1, given Ot ,t ¼ 1, 2…L . ζtð Þ i; j is written as

$$\mathcal{L}\_{\mathbf{t}}(\mathbf{i}, \mathbf{j}) = \mathbf{P}(\{\mathbf{X}\_{\mathbf{t}} = \mathbf{s}\_{\mathbf{i}}; \mathbf{X}\_{\mathbf{t}+1} = \mathbf{s}\_{\mathbf{j}} | \mathbf{O} \,\mathbf{O}\_1, \dots, \mathbf{O}\_{\mathbf{L}}, \boldsymbol{\Theta} \} \tag{37}$$

Given γ<sup>t</sup> ð Þi and ζtð Þ i; j , the anticipated number of transitions from state si during the path is written as

$$\mathbb{E}\left(\boldsymbol{\gamma}\_{\mathrm{t}}(\mathbf{i})\right) = \sum\_{\mathbf{t}=\mathbf{l}}^{\mathrm{L}-1} \boldsymbol{\gamma}\_{\mathrm{t}}(\mathbf{i}) \tag{38}$$

and the anticipated number of transitions from state si to state sj during the path is written as

$$\operatorname{E}(\mathsf{L}\_{\mathsf{t}}(\mathsf{i}, \mathsf{j})\,) = \sum\_{\mathbf{j}=1}^{L-1} \mathsf{zeta}\_{\mathsf{t}}(\mathsf{i}, \mathsf{j}) \tag{39}$$

Given E ζ<sup>t</sup> ð Þ ð Þ i; j and E γ<sup>t</sup> ð Þ<sup>i</sup> � �, we can extract the model parameters <sup>θ</sup> <sup>¼</sup> ð Þ <sup>π</sup>; <sup>A</sup>; <sup>B</sup> from the training sequence as given in [16] using the step listed below

1- for i <sup>¼</sup> <sup>1</sup>, <sup>2</sup>, <sup>3</sup>…K , let <sup>π</sup>b<sup>i</sup> <sup>¼</sup> expected frequency in state si at time tð Þ <sup>¼</sup> <sup>1</sup>

$$
\pi\_{\mathbf{i}} = \gamma\_1(\mathbf{i}) \tag{40}
$$

2- for i ¼ 1, 2, 3…K and j ¼ 1, 2, 3…K, compute

$$
\hat{\mathbf{a}}\_{\mathbf{i}\mathbf{j}} = \frac{\text{Expected number of transitions fromstate s}\_{\mathbf{i}} \text{ to state s}\_{\mathbf{j}}}{\text{Expected number of transitions from state s}\_{\mathbf{i}}}
$$

$$\mathbf{H} = \frac{\mathbf{E}(\zeta\_{\mathbf{t}}(\mathbf{i}, \mathbf{j}))}{\mathbf{E}(\gamma\_{\mathbf{t}}(\mathbf{i}))} = \frac{\sum\_{\mathbf{j}=1}^{\mathcal{L}-1} \zeta\_{\mathbf{t}}(\mathbf{i}, \mathbf{j})}{\sum\_{\mathbf{t}=1}^{\mathcal{L}-1} \gamma\_{\mathbf{t}}(\mathbf{i})} \tag{41}$$

3- for i ¼ 1, 2, 3…K and j ¼ 1, 2, 3…K, compute

<sup>b</sup>bið Þ¼ <sup>m</sup> Expected number of times in state sj and observing vm Expected number of times in state sj

$$\mathbf{h} = \frac{\sum\_{\mathbf{t}=\mathbf{t}\_{m}}^{\mathbf{L}} \gamma\_{\mathbf{t}}(\mathbf{i})}{\sum\_{\mathbf{t}=\mathbf{t}\_{\mathbf{t}}}^{\mathbf{L}} \gamma\_{\mathbf{t}}(\mathbf{i})} \tag{42}$$

The estimation algorithm can be summarized in the following steps:

1. Get your observations O1 O2, …OL,

3.2.1. Hidden Markov model parameters estimation using Baum-Welch algorithm

ζtð Þ¼ i; j P Xt ¼ si; Xtþ<sup>1</sup> ¼ sj

E γ<sup>t</sup>

likelihood estimation. In order to do that, we define γ<sup>t</sup>

132 Machine Learning - Advanced Techniques and Emerging Applications

at time t þ 1, given Ot ,t ¼ 1, 2…L . ζtð Þ i; j is written as

training sequence as given in [16] using the step listed below

2- for i ¼ 1, 2, 3…K and j ¼ 1, 2, 3…K, compute

3- for i ¼ 1, 2, 3…K and j ¼ 1, 2, 3…K, compute

1- for i <sup>¼</sup> <sup>1</sup>, <sup>2</sup>, <sup>3</sup>…K , let <sup>π</sup>b<sup>i</sup> <sup>¼</sup> expected frequency in state si at time tð Þ <sup>¼</sup> <sup>1</sup>

γt

si at time t, given t Ot ,t ¼ 1, 2…L . γ<sup>t</sup>

Given γ<sup>t</sup>

written as

Given E ζ<sup>t</sup> ð Þ ð Þ i; j and E γ<sup>t</sup>

If we assume that we have given some training observations with length L O1; O2 Ot f g ; …OL and want to approximate HMM parameters θ ¼ ð Þ π; A; B from them, we can use maximum

ð Þi is written as

We also define ζtð Þ i; j to be the probability of being in state si at time t and transiting to state sj

ð Þi and ζtð Þ i; j , the anticipated number of transitions from state si during the path is

L�1

t¼1 γt

L�1

j¼1

ð Þ<sup>i</sup> � �, we can extract the model parameters <sup>θ</sup> <sup>¼</sup> ð Þ <sup>π</sup>; <sup>A</sup>; <sup>B</sup> from the

� �

ð Þ<sup>i</sup> � � <sup>¼</sup> <sup>X</sup>

and the anticipated number of transitions from state si to state sj during the path is written as

<sup>b</sup>aij <sup>¼</sup> Expected number of transitions fromstate si to state sj Expected number of transitions from state si

> L P�1 j¼1

> > L P�1 t¼1 γt ð Þi

ζtð Þ i; j

<sup>¼</sup> <sup>E</sup> <sup>ζ</sup><sup>t</sup> ð Þ ð Þ <sup>i</sup>; <sup>j</sup> E γ<sup>t</sup> ð Þ<sup>i</sup> � � <sup>¼</sup>

<sup>E</sup> <sup>ζ</sup><sup>t</sup> ð Þ¼ ð Þ <sup>i</sup>; <sup>j</sup> <sup>X</sup>

ð Þi to be the probability of being in state

ð Þi (38)

ζtð Þ i; j (39)

(41)

π<sup>i</sup> ¼ γ1ð Þi (40)

ðÞ¼ i P Xt ¼ si ð Þ j O1;…; OL; θ (36)

�O O1;…; OL; <sup>θ</sup> � � (37)

2. Set a guess of your first θ estimate θ ð Þ1 , k ¼ 1

$$\mathbf{U} \text{update } \mathbf{k} = \mathbf{k} + 1$$

3. Compute θ ð Þ k based on O1 O2, …OL and

γt ð Þi , ζtð Þ i; j ∀1 ≤ t ≤ L, ∀1 ≤ i ≤ K, ∀1 ≤ j ≤ K


The prediction for a one-step ahead PU channel state can be done based on the trained parameters {π, A, B } with the help of Eqs. (31), (34), and (35) by setting T ¼ 1:

#### 3.3. Primary users channel state estimation based on Markov switching model

An alternative way to estimate PU channel state is to use Markov switching model (MSM). For the time series in Eq. (26), we assume that zt obeys two different Gaussian distributions N � μz0; σz0 <sup>2</sup> � � or N � <sup>μ</sup>z1; <sup>σ</sup>z1 <sup>2</sup> � � based on the sensed PU channel state "PU channel idle" or "PU occupied." We can rewrite Eq. (26) as follows:

$$\mathbf{z}\_t \sim \begin{cases} \left( \left( \mu\_{\mathbf{z}0}, \sigma\_{\mathbf{z}0} \right)^2 \right) & \mathbf{Y}\_t < \lambda \\ \left( \mu\_{\mathbf{z}1}, \sigma\_{\mathbf{z}1} \right)^2 & \mathbf{Y}\_t \gtrsim \lambda \end{cases} \tag{43}$$

It is obvious that Eq. (43) represents a two-state Gaussian regime switching time series which can be modeled using MSM [17]. In order to estimate the switching time of one state ahead of the current state for this time series, we need to derive MSM regression model for the time series and estimate its parameters.

#### 3.3.1. Derivation of Markov switching model for Gaussian regime switching time series

A simple Markov switching regression model to describe the two-state Gaussian regime switching time series is given in Eq. (43). This model can be written by following Ref [17] as

$$\mathbf{z}\_t = \boldsymbol{\mu}\_{s\_t} + \boldsymbol{\epsilon}\_t \qquad \boldsymbol{\epsilon}\_t \sim \left( \mathbf{0}, \sigma\_{s\_t}^{\ \ \ \mathbf{z}} \right) \tag{44}$$

where μst is an array of predetermined variables measured at time t, which may include the lagged values of zt, e<sup>t</sup> is the white noise process, st ¼ f g 0; 1 is a hidden Markov chain which has a mean and standard deviation over the time equal to μst ¼ μ0ð Þþ 1 � st μ<sup>1</sup> st and σst ¼ σ0ð Þþ 1 � st σ<sup>1</sup> st, respectively (the state variable st follows first order Markov chain (i.e., two-state Markov chain as in [18])). Given the past history of st, the probability of st taking a certain value depends only on st�<sup>1</sup> , which takes the following Markov property:

$$P(\mathbf{s}\_{\mathbf{t}}=\mathbf{j}|\mathbf{s}\_{\mathbf{t}-1}=\mathbf{i}) = P\_{\vec{\mathbf{j}}} \tag{45}$$

Normally, the density of zt conditional on <sup>ψ</sup><sup>t</sup>�<sup>1</sup> and st <sup>¼</sup> i, ið Þ <sup>¼</sup> <sup>0</sup>; <sup>1</sup> is written as

<sup>¼</sup> <sup>1</sup> ffiffiffiffiffiffiffiffiffiffiffi 2πσst <sup>p</sup> <sup>e</sup>

<sup>F</sup><sup>ð</sup> zt <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � �

where F represent the probability density function. Given the prediction probabilities P ðst ¼ i

� �

� �

; θ � � �

; θ � � �

where P0i ¼ P ðstþ<sup>1</sup> ¼ i sj <sup>t</sup> ¼ 0Þ and P1i ¼ P ðstþ<sup>1</sup> ¼ i sj <sup>t</sup> ¼ 1Þ are the transition probabilities. By setting the initial values as given in [19] assuming the Markov chain is presumed to be ergodic:

� �

P ðst ¼ i ψ<sup>t</sup>

<sup>F</sup><sup>ð</sup> zt <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

P ðstþ<sup>1</sup> ¼ i ψ<sup>t</sup>

; θ � � �

<sup>F</sup><sup>ð</sup> zt st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � �

> ; θ � � �

þ P1i P ðst ¼ 1 ψ<sup>t</sup>

þ P st ¼ 1; stþ<sup>1</sup> ¼ ijψ<sup>t</sup> ; θ � �

; θ �

� zt�μ<sup>s</sup> ð Þ<sup>t</sup> 2

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks

¼

<sup>F</sup><sup>ð</sup> zt st <sup>¼</sup> <sup>1</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

<sup>F</sup> zt<sup>j</sup> st <sup>¼</sup> <sup>0</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � �

<sup>2</sup>σst<sup>2</sup> (47)

http://dx.doi.org/10.5772/intechopen.74599

135

� � (48)

� � (50)

� � (49)

<sup>F</sup><sup>ð</sup> zt st <sup>¼</sup> <sup>i</sup>;ψ<sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � �

, the density of zt conditional on <sup>ψ</sup><sup>t</sup>�<sup>1</sup> can be obtained from

<sup>¼</sup> <sup>P</sup> <sup>ð</sup>st <sup>¼</sup> <sup>0</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

<sup>¼</sup> <sup>P</sup> <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

¼ P ðst ¼ 0, stþ<sup>1</sup> ¼ i ψ<sup>t</sup>

¼ P0i P ðst ¼ 0 ψ<sup>t</sup>

For i ¼ 0; 1, the filtering probabilities of st are given by:

The prediction probabilities are:

<sup>þ</sup> <sup>P</sup> <sup>ð</sup>st <sup>¼</sup> <sup>1</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

<sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � �

Figure 7. Markov switching model.

where Pijð Þ i; j ¼ 0; 1 denotes the transition probabilities of st ¼ j, given that st�<sup>1</sup> ¼ i. Clearly, the transition probabilities satisfy Pi0 þ Pi1 ¼ 1 . We can gather the transition probabilities Pij into a transition matrix as follows:

$$\mathbf{P} = \begin{pmatrix} \mathbf{P}(\mathbf{s}\_{\mathbf{t}} = \mathbf{0} \mid \mathbf{s}\_{\mathbf{t}-1} = \mathbf{0}) & \mathbf{P}(\mathbf{s}\_{\mathbf{t}} = \mathbf{0} \mid \mathbf{s}\_{\mathbf{t}-1} = \mathbf{1})\\\mathbf{P}(\mathbf{s}\_{\mathbf{t}} = \mathbf{1} \mid \mathbf{s}\_{\mathbf{t}-1} = \mathbf{0}) & \mathbf{P}(\mathbf{s}\_{\mathbf{t}} = \mathbf{1} \mid \mathbf{s}\_{\mathbf{t}-1} = \mathbf{1}) \end{pmatrix}$$

$$= \begin{pmatrix} \mathbf{P}\_{00} & \mathbf{P}\_{01} \\ \mathbf{P}\_{10} & \mathbf{P}\_{11} \end{pmatrix} \tag{46}$$

The transition matrix P is used to govern the behavior of the state variable st , and it holds only two parameters (P00 and P11 ). Assuming that we do not observe st directly, we only deduce its operation from the observed behavior of zt. The parameters that need to be estimated to fully describe the probability law governing zt are the variance of the Gaussian innovation σ0, σ<sup>1</sup> , the expectation of the dependent variable μ0, μ<sup>1</sup><sup>0</sup> and the two-state transition probabilities P00 and P11.

#### 3.3.2. Markov switching model parameters estimation via maximum likelihood estimation

There are many ways to estimate the parameters for the Markov switching model. Among these ways are Quasi-maximum likelihood estimation (QMLE) and Gibbs sampling. In this section, we focus on maximum likelihood estimation (MLE).

If we denote <sup>ψ</sup><sup>t</sup>�<sup>1</sup> = {zt�<sup>1</sup>, zt, ztþ<sup>1</sup>…z1} to be a vector of the training data until time t � 1 and denote θ ={σ0, σ1, μ0, μ<sup>1</sup><sup>0</sup> P00, P11} to be the vector of MSM parameters, then ψ<sup>L</sup> = {zt�<sup>1</sup>, zt, …, zL} to a vector of the available information with the length L sample (see Figure 7.). In order to evaluate the likelihood of the state variable st based on the current trend of zt, we need to assess its conditional expectations st ¼ i, ið Þ ¼ 0; 1 based on ψ and θ . These conditional expectations include prediction probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> , which are based on the information prior to time t, the filtering probabilities P ðst ¼ i∣ψ<sup>t</sup> ; θ) which are based on the past and current information, and finally the smoothing probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup>L; <sup>θ</sup> which are based on the fullsample information L. After getting these probabilities, we can obtain the log-likelihood function as a byproduct, and then we can compute the maximum likelihood estimates.

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks http://dx.doi.org/10.5772/intechopen.74599 135

Figure 7. Markov switching model.

zt ¼ μst þ e<sup>t</sup> e<sup>t</sup> � 0; σst

where μst is an array of predetermined variables measured at time t, which may include the lagged values of zt, e<sup>t</sup> is the white noise process, st ¼ f g 0; 1 is a hidden Markov chain which has a mean and standard deviation over the time equal to μst ¼ μ0ð Þþ 1 � st μ<sup>1</sup> st and σst ¼ σ0ð Þþ 1 � st σ<sup>1</sup> st, respectively (the state variable st follows first order Markov chain (i.e., two-state Markov chain as in [18])). Given the past history of st, the probability of st taking a certain value depends only on st�<sup>1</sup> , which takes the following Markov property:

where Pijð Þ i; j ¼ 0; 1 denotes the transition probabilities of st ¼ j, given that st�<sup>1</sup> ¼ i. Clearly, the transition probabilities satisfy Pi0 þ Pi1 ¼ 1 . We can gather the transition probabilities Pij

> <sup>P</sup> <sup>¼</sup> <sup>P</sup>ðst <sup>¼</sup> 0 s<sup>j</sup> <sup>t</sup>�<sup>1</sup> <sup>¼</sup> <sup>0</sup><sup>Þ</sup> <sup>P</sup><sup>ð</sup> st <sup>¼</sup> 0 s<sup>j</sup> <sup>t</sup>�<sup>1</sup> <sup>¼</sup> <sup>1</sup><sup>Þ</sup> Pð st ¼ 1 s j Þ <sup>t</sup>�<sup>1</sup> ¼ 0 Pð st ¼ 1 sj Þ <sup>t</sup>�<sup>1</sup> ¼ 1

> > <sup>¼</sup> P00 P01 P10 P11

The transition matrix P is used to govern the behavior of the state variable st , and it holds only two parameters (P00 and P11 ). Assuming that we do not observe st directly, we only deduce its operation from the observed behavior of zt. The parameters that need to be estimated to fully describe the probability law governing zt are the variance of the Gaussian innovation σ0, σ<sup>1</sup> , the expectation of the dependent variable μ0, μ<sup>1</sup><sup>0</sup> and the two-state transition probabilities P00

There are many ways to estimate the parameters for the Markov switching model. Among these ways are Quasi-maximum likelihood estimation (QMLE) and Gibbs sampling. In this

If we denote <sup>ψ</sup><sup>t</sup>�<sup>1</sup> = {zt�<sup>1</sup>, zt, ztþ<sup>1</sup>…z1} to be a vector of the training data until time t � 1 and denote θ ={σ0, σ1, μ0, μ<sup>1</sup><sup>0</sup> P00, P11} to be the vector of MSM parameters, then ψ<sup>L</sup> = {zt�<sup>1</sup>, zt, …, zL} to a vector of the available information with the length L sample (see Figure 7.). In order to evaluate the likelihood of the state variable st based on the current trend of zt, we need to assess its conditional expectations st ¼ i, ið Þ ¼ 0; 1 based on ψ and θ . These conditional expec-

sample information L. After getting these probabilities, we can obtain the log-likelihood function

, which are based on the information

which are based on the full-

; θ) which are based on the past and current

3.3.2. Markov switching model parameters estimation via maximum likelihood estimation

section, we focus on maximum likelihood estimation (MLE).

tations include prediction probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup>

information, and finally the smoothing probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup>L; <sup>θ</sup>

as a byproduct, and then we can compute the maximum likelihood estimates.

prior to time t, the filtering probabilities P ðst ¼ i∣ψ<sup>t</sup>

into a transition matrix as follows:

134 Machine Learning - Advanced Techniques and Emerging Applications

and P11.

<sup>2</sup> (44)

(46)

Pð st ¼ j sj <sup>t</sup>�<sup>1</sup> ¼ iÞ ¼ Pij (45)

Normally, the density of zt conditional on <sup>ψ</sup><sup>t</sup>�<sup>1</sup> and st <sup>¼</sup> i, ið Þ <sup>¼</sup> <sup>0</sup>; <sup>1</sup> is written as

$$\mathcal{F}(\left|\mathbf{z}\_{\mathbf{t}}\right|\mathbf{s}\_{\mathbf{t}}=\mathbf{i},\psi\_{\mathbf{t}-1};\boldsymbol{\Theta})=\frac{1}{\sqrt{2\pi\sigma\_{\mathbf{s}\mathbf{t}}}}\mathbf{e}^{-\frac{\left(\frac{\mathbf{z}\_{\mathbf{t}}-\boldsymbol{\mu}\_{\mathbf{s}}}{2\sigma\_{\mathbf{s}\mathbf{t}}}\right)^{2}}{2\pi\sigma\_{\mathbf{s}\mathbf{t}}}\tag{47}$$

where F represent the probability density function. Given the prediction probabilities P ðst ¼ i <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � � , the density of zt conditional on <sup>ψ</sup><sup>t</sup>�<sup>1</sup> can be obtained from

$$\mathcal{F}(\mathbf{z}\_t|\boldsymbol{\psi}\_{t-1};\boldsymbol{\Theta}) = $$

$$= \quad \mathbf{P}\left(\mathbf{s}\_t = 0 \middle| \boldsymbol{\psi}\_{t-1}; \boldsymbol{\Theta}\right) \mathcal{F}\left(\mathbf{z}\_t|\operatorname{ s}\_t = 0 \middle| \boldsymbol{\psi}\_{t-1}; \boldsymbol{\Theta}\right)$$

$$+ \mathbf{P}\left(\mathbf{s}\_t = 1 \middle| \boldsymbol{\psi}\_{t-1}; \boldsymbol{\Theta}\right) \mathcal{F}\left(\mathbf{z}\_t|\operatorname{ s}\_t = 1 \middle| \boldsymbol{\psi}\_{t-1}; \boldsymbol{\Theta}\right) \tag{48}$$

For i ¼ 0; 1, the filtering probabilities of st are given by:

$$\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{t}};\Theta\right)$$

$$=\frac{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{t}-1};\Theta\right)\mathcal{F}(\mathbf{z}\_{\mathbf{t}}\mid\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{t}-1};\Theta\right)}{\mathcal{F}(\mathbf{z}\_{\mathbf{t}}\middle|\psi\_{\mathbf{t}-1};\Theta)}\tag{49}$$

The prediction probabilities are:

$$\mathbf{P}\left(\mathbf{s}\_{t+1} = \mathbf{i} \, \middle| \, \psi\_{\mathbf{t}}; \Theta\right)$$

$$\mathbf{s} = \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}} = 0, \mathbf{s}\_{\mathbf{t}+1} = \mathbf{i} \, \middle| \, \psi\_{\mathbf{t}}; \Theta\right) + \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}} = 1, \mathbf{s}\_{\mathbf{t}+1} = \mathbf{i} \, \middle| \, \psi\_{\mathbf{t}}; \Theta\right)$$

$$= \mathbf{P}\_{\text{0i}} \, \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}} = \mathbf{0} \, \middle| \, \psi\_{\mathbf{t}}; \Theta\right) + \mathbf{P}\_{\text{1i}} \, \middle| \, \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}} = \mathbf{1} \, \middle| \, \psi\_{\mathbf{t}}; \Theta\right) \tag{50}$$

where P0i ¼ P ðstþ<sup>1</sup> ¼ i sj <sup>t</sup> ¼ 0Þ and P1i ¼ P ðstþ<sup>1</sup> ¼ i sj <sup>t</sup> ¼ 1Þ are the transition probabilities. By setting the initial values as given in [19] assuming the Markov chain is presumed to be ergodic:

$$\mathbf{P} \left( \mathbf{s}\_{\mathbf{0}} = \mathbf{i} \middle| \psi\_{\mathbf{0}} \right) = \frac{1 - \mathbf{P}\_{\overline{\mathbf{y}}}}{\mathbf{2} - \mathbf{P}\_{\overline{\mathbf{u}}} - \mathbf{P}\_{\overline{\mathbf{y}}}} $$

we can iterate the Eqs. (49) and (50) to obtain the filtering probabilities P ðst ¼ i ψ<sup>t</sup> ; θ � � � and the conditional densities <sup>F</sup> zt<sup>j</sup> st <sup>¼</sup> <sup>0</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � for t <sup>¼</sup> <sup>1</sup>, <sup>2</sup>, …::T. Then we can compute the logarithmic likelihood function using

$$\log\left(\mathcal{L}\left(\hat{\Theta}\right)\right) = \sum\_{\mathbf{t}=1}^{\mathsf{T}} \sum\_{\mathbf{i}=1}^{2} \log\left(\mathcal{F}\left(\mathbf{Z}\_{\mathbf{t}} \, \middle|\, \mathbf{S}\_{\mathbf{t}} = \mathbf{i}, \psi\_{\mathbf{t}-1}; \Theta\right) \times \mathcal{P}\left(\mathbf{S}\_{\mathbf{t}} = \mathbf{i} \,\middle|\, \psi\_{\mathbf{t}}; \Theta\right)\right) \tag{51}$$

where L θb � � is the maximized value of the likelihood function. The model estimation can finally be obtained by finding the set of parameters θb that maximize the Eq. (51) using numerical-search algorithm. The estimated filtering and prediction probabilities can then be easily calculated by plugging θb into the equation formulae of these probabilities. We adopt the approximation in Ref [20] for computing the smoothing probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup>L; <sup>θ</sup> � � �

$$\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\;\middle|\;\mathbf{s}\_{\mathbf{t}+1}=\mathbf{j},\psi\_{\mathbf{l}};\Theta\right)\approx\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\;\middle|\;\mathbf{s}\_{\mathbf{t}+1},\psi\_{\mathbf{i}};\Theta\right)$$

$$=\frac{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i},\mathbf{s}\_{\mathbf{t}+1}\middle|\;\psi\_{\mathbf{t}-1};\Theta\right)}{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{j}\middle|\;\psi\_{\mathbf{i}};\Theta\right)}$$

$$=\frac{\mathbf{P}\_{\mathbf{0i}}\;\middle|\;\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{t}};\Theta\right)}{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{j}\middle|\psi\_{\mathbf{i}};\Theta\right)}$$

time. We use this training observations to train Baum-Welch algorithm in order to estimate

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks

http://dx.doi.org/10.5772/intechopen.74599

137

π<sup>1</sup> ¼ ð Þ 1 0

A1 <sup>¼</sup> <sup>0</sup>:85 0:<sup>15</sup> <sup>0</sup>:10 0:<sup>90</sup>

B1 <sup>¼</sup> <sup>0</sup>:17 0:16 0:17 0:16 0:17 0:17 0:<sup>17</sup> <sup>0</sup>:60 0:08 0:08 0:08 0:08 0:08 0:<sup>08</sup>

Figure 9a shows the performance of HMM algorithm in estimating the PU channel states (i.e., PU idle or PU occupied) of the time series that capture the detection sequence for a single-user cognitive radio network. Figure 9a contains three plots; the top plot shows the randomly distributed PU channel states over time T ¼ 500 ms . The middle plot shows the generated time series following the distribution zt ∈f g 1; 2; 3 for idle states and zt ∈ f g 4; 5; 6 for occupied states (note: we can construct the observation space from these two distributions as Ot ∈f g 1; 2; 3; 4; 5; 6 , t ¼ 1, 2…500 ms). The bottom plot shows performance of HMM algorithm in forecasting the

Figure 9b shows the performance of MSM algorithm in predicting the switching process between the two PU channel states for the same PU detection sequence given in Figure 9a T ¼ 500 ms . The top graph in Figure 9b shows the generated time series with the following distribution zt � ð Þ 0:1; 0:5 for idle states and zt � ð Þ 0:01; 0:2 for occupied states and the bottom graph shows the prediction performance using MSM. As it is clear from the figure, the

HMM model parameters θ ¼ ð Þ π; A; B , assuming that the first estimate of θ ð Þ1 is:

time series generated to capture PU detection sequence.

Figure 8. The training detection sequence for HMM and MSM.

prediction performance is smoother than HMM approach.

And, for i; j ¼ 0; 1, smoothing probabilities is expressed as:

$$\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{l}};\Theta\right)$$

$$=\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=0\middle|\psi\_{\mathbf{l}};\Theta\right)\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{1}\middle|\mathbf{s}\_{\mathbf{t}+1}=0,\psi\_{\mathbf{l}};\Theta\right)$$

$$+\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{1}\middle|\psi\_{\mathbf{l}};\Theta\right)\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\mathbf{s}\_{\mathbf{t}+1}=\mathbf{1},\psi\_{\mathbf{l}};\Theta\right)$$

$$=\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}}=\mathbf{i}\middle|\psi\_{\mathbf{i}};\Theta\right)\times\left(\frac{\mathbf{P}\_{\mathbf{i}\mathbf{0}}\ \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{0}\middle|\psi\_{\mathbf{l}};\Theta\right)}{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{0}\middle|\psi\_{\mathbf{i}};\Theta\right)}+\frac{\mathbf{P}\_{\mathbf{i}\mathbf{1}}\ \mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{1}\middle|\psi\_{\mathbf{l}};\Theta\right)}{\mathbf{P}\left(\mathbf{s}\_{\mathbf{t}+1}=\mathbf{1}\middle|\psi\_{\mathbf{i}};\Theta\right)}\right)\tag{52}$$

Using P SL <sup>¼</sup> <sup>i</sup>jψL; <sup>θ</sup> � � as the initial value, we can iterate the equations regressively for filtering and prediction probabilities along with the equation above to get the smoothing probabilities for t ¼ L � 1, � �, k þ 1.

#### 3.4. Results and discussions

Figure 8 shows the training detection sequence which we generate as a training observation using randomly distributed PU channel state "idle and occupied" over T = 250 ms simulation

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks http://dx.doi.org/10.5772/intechopen.74599 137

Figure 8. The training detection sequence for HMM and MSM.

P ðs0 ¼ i ψ<sup>0</sup> � � �

we can iterate the Eqs. (49) and (50) to obtain the filtering probabilities P ðst ¼ i ψ<sup>t</sup>

rithmic likelihood function using

where L θb

� �

log L <sup>θ</sup><sup>b</sup> � � � �

<sup>¼</sup> <sup>X</sup> T

136 Machine Learning - Advanced Techniques and Emerging Applications

And, for i; j ¼ 0; 1, smoothing probabilities is expressed as:

¼ P ðst ¼ i ψ<sup>t</sup>

ities for t ¼ L � 1, � �, k þ 1.

3.4. Results and discussions

; θ � � � �

<sup>¼</sup> <sup>P</sup> <sup>ð</sup>stþ<sup>1</sup> <sup>¼</sup> <sup>0</sup> <sup>ψ</sup>L; <sup>θ</sup> �

<sup>þ</sup><sup>P</sup> <sup>ð</sup>stþ<sup>1</sup> <sup>¼</sup> <sup>1</sup> <sup>ψ</sup>L; <sup>θ</sup> �

t¼1

X 2

i¼1

conditional densities <sup>F</sup> zt<sup>j</sup> st <sup>¼</sup> <sup>0</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> � � for t <sup>¼</sup> <sup>1</sup>, <sup>2</sup>, …::T. Then we can compute the loga-

logð<sup>F</sup> Zt St <sup>¼</sup> <sup>i</sup>;ψ<sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

finally be obtained by finding the set of parameters θb that maximize the Eq. (51) using numerical-search algorithm. The estimated filtering and prediction probabilities can then be easily calculated by plugging θb into the equation formulae of these probabilities. We adopt the

<sup>¼</sup> <sup>P</sup> <sup>ð</sup>st <sup>¼</sup> <sup>i</sup>, stþ<sup>1</sup> <sup>ψ</sup><sup>t</sup>�<sup>1</sup>; <sup>θ</sup> �

P ðstþ<sup>1</sup> ¼ j ψ<sup>t</sup>

P ðstþ<sup>1</sup> ¼ j ψ<sup>t</sup>

<sup>P</sup> <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup>L; <sup>θ</sup> � � �

� �

� �

Pi0 <sup>P</sup> <sup>ð</sup>stþ<sup>1</sup> <sup>¼</sup> <sup>0</sup> <sup>ψ</sup>L; <sup>θ</sup> �

Using P SL <sup>¼</sup> <sup>i</sup>jψL; <sup>θ</sup> � � as the initial value, we can iterate the equations regressively for filtering and prediction probabilities along with the equation above to get the smoothing probabil-

Figure 8 shows the training detection sequence which we generate as a training observation using randomly distributed PU channel state "idle and occupied" over T = 250 ms simulation

P ðstþ<sup>1</sup> ¼ 0 ψ<sup>t</sup>

<sup>¼</sup> P0i P st <sup>¼</sup> <sup>i</sup>jψ<sup>t</sup>

approximation in Ref [20] for computing the smoothing probabilities P <sup>ð</sup>st <sup>¼</sup> <sup>i</sup> <sup>ψ</sup>L; <sup>θ</sup> �

<sup>P</sup> <sup>ð</sup>st <sup>¼</sup> i stþ<sup>1</sup> <sup>¼</sup> <sup>j</sup>;ψL; <sup>θ</sup> � � �

� �

is the maximized value of the likelihood function. The model estimation can

≈ P ðst ¼ i stþ<sup>1</sup>;ψ<sup>t</sup>

� �

; θ � � �

; θ � � �

<sup>P</sup> <sup>ð</sup>st <sup>¼</sup> 1 stþ<sup>1</sup> <sup>¼</sup> <sup>0</sup>;ψL; <sup>θ</sup> � � �

<sup>P</sup> <sup>ð</sup>st <sup>¼</sup> i stþ<sup>1</sup> <sup>¼</sup> <sup>1</sup>;ψL; <sup>θ</sup> � � �

!

Pi1 <sup>P</sup> <sup>ð</sup>stþ<sup>1</sup> <sup>¼</sup> <sup>1</sup> <sup>ψ</sup>L; <sup>θ</sup> �

P ðstþ<sup>1</sup> ¼ 1 ψ<sup>t</sup>

� �

(52)

; θ � � �

� �

; θ � � � <sup>þ</sup>

; θ � �

; θ � � �

� P St ¼ ijψ<sup>t</sup> ; θ � � � � (51)

<sup>¼</sup> <sup>1</sup> � Pjj 2 � Pii � Pjj

> ; θ � � �

� �

and the

time. We use this training observations to train Baum-Welch algorithm in order to estimate HMM model parameters θ ¼ ð Þ π; A; B , assuming that the first estimate of θ ð Þ1 is:

$$
\pi\_1 = \begin{pmatrix} 1 & 0 \end{pmatrix}
$$

$$
\mathbf{A}\_1 = \begin{pmatrix} 0.85 & 0.15 \\ 0.10 & 0.90 \end{pmatrix}
$$

$$
\mathbf{B}\_1 = \begin{pmatrix} 0.17 & 0.16 & 0.17 & 0.16 & 0.17 & 0.17 & 0.17 \\ 0.60 & 0.08 & 0.08 & 0.08 & 0.08 & 0.08 \end{pmatrix}
$$

Figure 9a shows the performance of HMM algorithm in estimating the PU channel states (i.e., PU idle or PU occupied) of the time series that capture the detection sequence for a single-user cognitive radio network. Figure 9a contains three plots; the top plot shows the randomly distributed PU channel states over time T ¼ 500 ms . The middle plot shows the generated time series following the distribution zt ∈f g 1; 2; 3 for idle states and zt ∈ f g 4; 5; 6 for occupied states (note: we can construct the observation space from these two distributions as Ot ∈f g 1; 2; 3; 4; 5; 6 , t ¼ 1, 2…500 ms). The bottom plot shows performance of HMM algorithm in forecasting the time series generated to capture PU detection sequence.

Figure 9b shows the performance of MSM algorithm in predicting the switching process between the two PU channel states for the same PU detection sequence given in Figure 9a T ¼ 500 ms . The top graph in Figure 9b shows the generated time series with the following distribution zt � ð Þ 0:1; 0:5 for idle states and zt � ð Þ 0:01; 0:2 for occupied states and the bottom graph shows the prediction performance using MSM. As it is clear from the figure, the prediction performance is smoother than HMM approach.

problem. We finally showed by the means of simulation that both hidden Markov model and Markov switching model perform very well in predicting the time series that capture the

Machine Learning Approaches for Spectrum Management in Cognitive Radio Networks

http://dx.doi.org/10.5772/intechopen.74599

139

I gratefully acknowledge the funding received from Shanghai Jiao Tong University to under-

take my PhD. I also thank Prof. Bin Guo for his encouragement and help on the topic.

Department of Electronic Engineering, School of Electronic Information and Electrical

[1] Teguig D, Scheers B, Le Nir V. Data fusion schemes for cooperative spectrum sensing in cognitive radio networks. Communications and Information Systems Conference (MCC),

[2] Zhai X, Jianguo P. Energy-detection based spectrum sensing for cognitive radio. 2007:944-

[3] Mikaeil AM, Guo B, Bai X, Wang Z. Machine learning to data fusion approach for cooperative spectrum sensing. 2014 International Conference on Cyber Enabled Distributed Computing and Knowledge Discovery(CyberC), Shanghai,13–15 October 2014, 429-

[4] Zhe C, Qiu RC. Prediction of channel state for cognitive radio using higher-order hidden Markov model. IEEE SoutheastCon 2010 (SoutheastCon), Proceedings of the IEEE. 2010

[5] Zhe C et al. Channel state prediction in cognitive radio, part II: Single-user prediction.

[6] Chang-Hyun P et al. HMM based channel status predictor for cognitive radio." Micro-

[7] Mikaeil AM, Guo B, Bai X, Wang Z. Hidden Markov and Markov switching model for primary user channel state prediction in cognitive radio. IEEE 2nd International Confer-

Address all correspondence to: ahmed\_mikaeil@yahoo.co.uk

Engineering, Shanghai Jiao Tong University, Shanghai, China

actual primary user channel state.

Acknowledgements

Author details

References

947

434

Ahmed Mohammed Mikaeil

2012 Military. IEEE; 2012

Proceedings of IEEE Southeast Con. 2011

wave Conference, 2007. APMC 2007. Asia-Pacific. IEEE, 2007

ence on Systems and informatics (ICSAI). 2014:854-859

Figure 9. (a) Shows the performance of HMM algorithm in predicting the generated time series to capture PU channel state detection sequence. (b) Shows the performance of MSM algorithm in predicting the generated time series to capture the same PU detection sequence in Figure 8.
