**6. Conclusions**

14 Will-be-set-by-IN-TECH

blocking probability for primary users while the MDP formulation implies the exploration of the Pareto front, since there is no a priori relationship between *α* and this blocking probability. On the other hand, implementing the policy solving the CMDP problem implies to randomize at least one control (it can be shown that the number of required randomized controls equals the number of constraints). While this is technically feasible, a stationary deterministic policy

For the auction-based access we consider again the three scenarios defined in previous section. Additionally we define three classes of secondary users (SU), characterized by the price that they offer per minute of channel occupation. The bid offers per class are: class 1: 0.01 \$/m, class 2: 0.02 \$/m and class 3: 0.03 \$/m. Additionally, we define the probability of an SU incoming call being of each class. The SU class probability distribution is: class 1 probability: 0.5, class 2 probability: 0.3 and class 3 probability: 0.2. We summarize SU class definition in

> **SU class class 1 class 2 class 3** offered price (\$/m) 0.01 0.02 0.03 probability 0.5 0.3 0.2

Note that both the offered prices and their probability distributions are static, *i.e.* they do not change over time and are independent of the system occupation. It is not completely unrealistic taking into account typical tariff policies of wireless operators. In this environment the class structure and the probability distribution may be seen as types of contracts for secondary users and market penetration of each type of contract respectively. However, for a more dynamical auction process, where bidders are able to change their bid offers adaptively, the model should be revised. One possibility would be to define one probability distribution for each state. More detailed modeling strategies would increase the complexity of the MDP solving algorithm or even make them intractable. This is a classic problem of MDP formulation, known as the *curse of dimensionality* and is typically addressed by means of

<sup>0</sup> <sup>5</sup> <sup>10</sup> <sup>15</sup> <sup>20</sup> <sup>0</sup>

<sup>0</sup> <sup>10</sup> <sup>20</sup> <sup>30</sup> <sup>0</sup>

Income from SU (\$/h) (c) scenario 3

0.02

0.04

LU blocking probability

0.06

0.08

Income from SU (\$/h) (b) scenario 2

Fig. 4. Pareto fronts obtained for the auction-based access in scenario 1 (a), scenario 2 (b) and

Figure 4 shows the Pareto fronts for the auction-based system in the three scenarios. As in previous subsection, the MDP and the CMDP approaches provided similar results. It

Table 2. Classification of SU in terms their bid offers and their probabilities.

the heuristic approach of approximate dynamic programming.

0.02 0.04 0.06 0.08 0.1

LU blocking probability

<sup>0</sup> <sup>2</sup> <sup>4</sup> <sup>6</sup> <sup>8</sup> <sup>10</sup> 0.04

Income from SU (\$/h) (a) scenario 1

0.05 0.06 0.07 0.08 0.09 0.1

scenario 3 (c)

LU blocking probability

is simpler to implement.

**5.2 Auction based access**

Table 2.

This chapter has surveyed the use of MDP formulation within the framework of cognitive radio. We have reviewed the fundamentals of MDP and its generalizations, such as CMDP, POMDP and constrained POMDP. While most previous works focus on decentralized access, we focus on centralized access. The main difference between them is that when the access relies on a central controller or *spectrum broker*, it generally has full knowledge of the spectrum occupation, while in decentralized access decision have to be taken with partial and sometimes unreliable information about channel occupation. Therefore, centralized schemes are more suitable to MDP or CMDP modeling, while decentralized ones generally require POMDP or constrained POMDP which are intractable in many cases and require approximated or heuristic algorithms. We consider two types of access: one where only one type of secondary user tries to access the licensed spectrum and other where users are classified according to the price they are willing to pay for the use of the spectrum. The first one is referred to as priority-based access and the second one as auction-based access. The main issue of the problems addressed is that two contrary objectives coexist. In priority-based access, the controller tries to reduce the blocking probability of both types of users. In auction-based, the objectives are to reduce blocking probability for licensed users and to increase the income received from spectrum leasing. For these problems there does not exists an *optimal* policy, but a set of *Pareto optimal* policies. The performance of these policies lie on the Pareto front, defined as the set of points where one objective cannot be improved without worsening the other one. We have shown how to compute these Pareto fronts for each access scheme by weighting the objectives in an MDP problem and by formulating a CMDP. The first approach requires solving Bellman's equation and the second requires solving a linear program. We have obtained the Pareto fronts for several scenarios, showing the influence of traffic share on system's performance. The Pareto front is a very usual tool to determine the performance threshold for each objective upon which further increments on this objective require excessive degradation of the other one. MDP and CMDP are useful tools for developing centralized access policies for cognitive radio systems. One drawback is the so-called *curse of dimensionality*, that may render computationally intractable the problem as the sizes of the state and action spaces increase. In addition, although policies can be computed off-line, alleviating the computational overhead of the access controller, the system's parameters may be variable, requiring many pre-computed policies and thus imposing large memory requirements.
