**8. Appendix**

#### **8.1. Proof of Theorem 6.1**

*Proof.* With the selected *M* users by the combinatorial user selection approach under the MDist criterion, suppose that we jointly detect *M* users' signals with the *N* × *MP* channel matrix **H**K using the ML detector. The PEP in detecting *M* users' signals has the following upper bound:

$$\Pr\left(\mathbf{s}\_{(1)} \to \mathbf{s}\_{(2)}\right) \le \text{erfc}\left(\sqrt{\frac{\|\mathbf{H}\_{\mathcal{K}}\bar{\mathbf{d}}\|^2}{2N\_0}}\right),\tag{34}$$

10.5772/57128

177

, (38)

http://dx.doi.org/10.5772/57130

. (41)

*<sup>q</sup>*! . (42)

*<sup>M</sup>* ⌋. Let **<sup>H</sup>**K<sup>1</sup> , **<sup>H</sup>**K<sup>2</sup> ,..., **<sup>H</sup>**K⌊ *<sup>K</sup>*

, which are independent. Let

*<sup>M</sup>* ⌋−1+*ǫ*), (43)

, (44)

*M* ⌋

<sup>K</sup>**H**K**d**, (39)

*<sup>h</sup>* �**d**�2**I**. (40)

For the case that the MDist criterion is employed, we have

max K

**<sup>s</sup>**(1) <sup>→</sup> **<sup>s</sup>**(2)

� ≤ erfc

V2(**H**K) = max

Let **w**<sup>K</sup> = **H**K**d**. Note that **w**<sup>K</sup> is a zero-mean CSCG random vector and

*E* � **w**K**w**<sup>H</sup> K � = *σ*<sup>2</sup>

*fX*(*x*K) = <sup>1</sup> (*σ*<sup>2</sup>

*FX*(*x*K) = <sup>1</sup> <sup>−</sup> *<sup>e</sup>x*K/(*σ*<sup>2</sup>

for selecting **H**<sup>K</sup> with the MDist selection is at least ⌊ *<sup>K</sup>*

*<sup>M</sup>* ⌋−<sup>1</sup>

*EV* erfc

<sup>1</sup> <sup>&</sup>gt; 0 is a constant, and *<sup>ǫ</sup>* <sup>&</sup>gt; 0. Thus, we have

**d**∈D,**d**�=0

� �*σ*2 *h***d**�<sup>2</sup> *N*0

*M* ⌋ �

*fV*(*v*) = *KF*⌊ *<sup>K</sup>*

= *c*<sup>1</sup>

The cumulative distribution function (cdf) is

K

We can show that *X*<sup>K</sup> = ||**w**K||<sup>2</sup> is a chi-square random variable with 2*N* degrees of freedom

*<sup>h</sup>* �**d**�2)*N*(*<sup>N</sup>* <sup>−</sup> <sup>1</sup>)!

*<sup>h</sup>* �**d**�<sup>2</sup>)

To obtain an upper bound on the error probability, we note that the number of alternative combinations of the channel matrices, which are statistically independent with each other,

*<sup>M</sup>* ⌋ of **<sup>w</sup>**K, i.e., **<sup>w</sup>**K<sup>1</sup> , **<sup>w</sup>**K<sup>2</sup> ,..., **<sup>w</sup>**K⌊ *<sup>K</sup>*

*<sup>X</sup>* (*v*)*fX*(*v*) = *<sup>c</sup>*′

 �

�−*N*⌊ *<sup>K</sup> M* ⌋ + *o* 

*N*−1 ∑ *q*=0

 �

min **d**∈D,**d**�=0

max<sup>K</sup> V2(**H**K) 2*N*<sup>0</sup>

**d**H**H**H

*<sup>x</sup>N*−<sup>1</sup> K *e*

(*x*K/(*σ*<sup>2</sup>

*<sup>M</sup>* ⌋ independent alternative combinations of the channel vectors. Then,

1*vN*⌊ *<sup>K</sup>*

max<sup>K</sup> **d**H**H**<sup>H</sup>

2*N*<sup>0</sup>

��*σ*2 *h***d**�<sup>2</sup> *N*0

*M* ⌋

, where *Xm* <sup>=</sup> ||**w**K*<sup>m</sup>* ||2. Using order statistics, the pdf of *<sup>V</sup>*

*<sup>M</sup>* ⌋−<sup>1</sup> <sup>+</sup> *<sup>o</sup>*(*vN*⌊ *<sup>K</sup>*

<sup>K</sup>**H**K**d**

 

�−*N*⌊ *<sup>K</sup>*

*<sup>M</sup>* ⌋+<sup>1</sup> 

−*x*K/(*σ*<sup>2</sup>

*<sup>h</sup>* �**d**�<sup>2</sup>)

*<sup>h</sup>* �**d**�2))*<sup>q</sup>*

Lattice Reduction-Based User Selection in Multiuser MIMO Systems

Pr �

Note that

and its pdf is

represent such ⌊ *<sup>K</sup>*

*<sup>V</sup>* = max �

is given by

where *<sup>c</sup>*′

there are at least ⌊ *<sup>K</sup>*

*X*1, *X*2,..., *X*⌊ *<sup>K</sup>*

*P* ml *<sup>e</sup>* <sup>≤</sup> ∑

where

$$
\overline{\mathbf{d}} = \arg\min\_{\mathbf{d} \in \mathcal{D}, \mathbf{d} \neq 0} \|\mathbf{H}\_{\mathcal{K}} \mathbf{d}\|^2,
$$

$$
\mathcal{D} = \left\{ \mathbf{d} = \mathbf{s} - \mathbf{s}' \mid \mathbf{s} \neq \mathbf{s}' \in \mathcal{S}^{\mathrm{MP}} \right\} \subset \mathbb{Z}^{\mathrm{MP}} + j\mathbb{Z}^{\mathrm{MP}},\tag{35}
$$

and erfc(*x*) is the complementary error function of *x*, i.e., erfc(*x*) = <sup>√</sup> 2 *π* � <sup>+</sup><sup>∞</sup> *<sup>x</sup> <sup>e</sup>*−*z*<sup>2</sup> *dz*.

Let V(**H**K) denote the length of the shortest non-zero vector of the lattice generated by **H**K. Then, we have

$$\Pr\left(\mathbf{s}\_{(1)} \to \mathbf{s}\_{(2)}\right) \le \text{erfc}\left(\sqrt{\frac{\mathcal{V}^2(\mathbf{H}\_{\mathcal{K}})}{2\mathcal{N}\_0}}\right),\tag{36}$$

where

$$\mathcal{V}(\mathbf{H}\boldsymbol{\chi}) = \|\mathbf{H}\boldsymbol{\chi}\mathbf{\bar{d}}\|.\tag{37}$$

For the case that the MDist criterion is employed, we have

$$\Pr\left(\mathbf{s}\_{(1)} \to \mathbf{s}\_{(2)}\right) \le \text{erfc}\left(\sqrt{\frac{\max\_{\mathcal{K}} \mathcal{V}^2(\mathbf{H}\_{\mathcal{K}})}{2N\_0}}\right),\tag{38}$$

Note that

18 Recent Trends in Multiuser MIMO Communications

performance with that based on a combinatorial approach.

Pr �

D = �

> Pr �

**<sup>s</sup>**(1) <sup>→</sup> **<sup>s</sup>**(2)

� ≤ erfc

**<sup>d</sup>**¯ <sup>=</sup> arg min **<sup>d</sup>**∈D,**d**�=<sup>0</sup>

**<sup>d</sup>** <sup>=</sup> **<sup>s</sup>** <sup>−</sup> **<sup>s</sup>**′ <sup>|</sup> **<sup>s</sup>** �<sup>=</sup> **<sup>s</sup>**′ ∈ S *MP*�

� ≤ erfc

Let V(**H**K) denote the length of the shortest non-zero vector of the lattice generated by **H**K.

 �

V2(**H**K) 2*N*<sup>0</sup>

V(**H**K) = �**H**K**d**¯ �. (37)

and erfc(*x*) is the complementary error function of *x*, i.e., erfc(*x*) = <sup>√</sup>

**<sup>s</sup>**(1) <sup>→</sup> **<sup>s</sup>**(2)

In this chapter, we studied the user selection based on the error probability of an actually employed MIMO detector in multiuser MIMO systems. As the complexity becomes prohibitively high if the user selection is based on exhaustive search (i.e., the combinatorial user selection), we considered a greedy user selection approach to keep the complexity low. We showed that low-complexity suboptimal detectors (i.e., the LR-based MMSE-SIC detector) with the MD criterion for the user selection can fully exploit both multiuser and receive diversity and provide good performance even though their complexity is low, which has been confirmed by both theoretical analysis and simulation results. Moreover, according to the simulation results, it was also shown that the LR-based detection with our proposed greedy user selection approach can achieve a similar diversity gain and have a comparable

*Proof.* With the selected *M* users by the combinatorial user selection approach under the MDist criterion, suppose that we jointly detect *M* users' signals with the *N* × *MP* channel matrix **H**K using the ML detector. The PEP in detecting *M* users' signals has the following

> �

�**H**K**d**�2,

�**H**K**d**¯ �<sup>2</sup> 2*N*<sup>0</sup>

, (34)

⊂ **Z***MP* + *j***Z***MP*, (35)

*dz*.

, (36)

2 *π* � <sup>+</sup><sup>∞</sup> *<sup>x</sup> <sup>e</sup>*−*z*<sup>2</sup>

**7. Conclusion**

**8. Appendix**

upper bound:

Then, we have

where

where

**8.1. Proof of Theorem 6.1**

$$\max\_{\mathcal{K}} \mathcal{V}^2(\mathbf{H}\_{\mathcal{K}}) = \max\_{\mathcal{K}} \min\_{\mathbf{d} \in \mathcal{D}, \mathbf{d} \neq 0} \mathbf{d}^{\mathbf{H}} \mathbf{H}\_{\mathcal{K}}^{\mathbf{H}} \mathbf{H}\_{\mathcal{K}} \mathbf{d},\tag{39}$$

Let **w**<sup>K</sup> = **H**K**d**. Note that **w**<sup>K</sup> is a zero-mean CSCG random vector and

$$E\left[\mathbf{w}\_{\mathcal{K}}\mathbf{w}\_{\mathcal{K}}^{\mathrm{H}}\right] = \sigma\_{\hbar}^{2} ||\mathbf{d}||^{2}\mathbf{I}.\tag{40}$$

We can show that *X*<sup>K</sup> = ||**w**K||<sup>2</sup> is a chi-square random variable with 2*N* degrees of freedom and its pdf is

$$f\_{\mathbf{X}}(\mathbf{x}\boldsymbol{\chi}) = \frac{1}{(\sigma\_{\hbar}^{2} \|\mathbf{d}\|^{2})^{N} (N-1)!} \mathbf{x}\_{\boldsymbol{\aleph}}^{N-1} e^{-\mathbf{x}\_{\boldsymbol{\aleph}} / (\sigma\_{\hbar}^{2} \|\mathbf{d}\|^{2})}.\tag{41}$$

The cumulative distribution function (cdf) is

$$F\_{\mathbf{X}}(\mathbf{x}\_{\mathcal{K}}) = 1 - e^{\mathbf{x}\_{\mathcal{K}} / (\sigma\_h^2 \|\mathbf{d}\|^2)} \sum\_{q=0}^{N-1} \frac{(\mathbf{x}\_{\mathcal{K}} / (\sigma\_h^2 \|\mathbf{d}\|^2))^q}{q!}. \tag{42}$$

To obtain an upper bound on the error probability, we note that the number of alternative combinations of the channel matrices, which are statistically independent with each other, for selecting **H**<sup>K</sup> with the MDist selection is at least ⌊ *<sup>K</sup> <sup>M</sup>* ⌋. Let **<sup>H</sup>**K<sup>1</sup> , **<sup>H</sup>**K<sup>2</sup> ,..., **<sup>H</sup>**K⌊ *<sup>K</sup> M* ⌋ represent such ⌊ *<sup>K</sup> <sup>M</sup>* ⌋ independent alternative combinations of the channel vectors. Then, there are at least ⌊ *<sup>K</sup> <sup>M</sup>* ⌋ of **<sup>w</sup>**K, i.e., **<sup>w</sup>**K<sup>1</sup> , **<sup>w</sup>**K<sup>2</sup> ,..., **<sup>w</sup>**K⌊ *<sup>K</sup> M* ⌋ , which are independent. Let *<sup>V</sup>* = max � *X*1, *X*2,..., *X*⌊ *<sup>K</sup> M* ⌋ � , where *Xm* <sup>=</sup> ||**w**K*<sup>m</sup>* ||2. Using order statistics, the pdf of *<sup>V</sup>* is given by

$$f\_V(v) = K F\_X^{\lfloor \frac{K}{M} \rfloor - 1}(v) f\_X(v) = c\_1' v^{N \lfloor \frac{K}{M} \rfloor - 1} + o(v^{N \lfloor \frac{K}{M} \rfloor - 1 + \epsilon}),\tag{43}$$

where *<sup>c</sup>*′ <sup>1</sup> <sup>&</sup>gt; 0 is a constant, and *<sup>ǫ</sup>* <sup>&</sup>gt; 0. Thus, we have

$$\begin{split} P\_{\varepsilon}^{\text{ml}} &\leq \sum\_{\mathbf{d}\in\mathcal{D}, \mathbf{d}\neq 0} E\_{V} \left[ \text{erfc} \left( \sqrt{\frac{\max\_{\mathbf{K}} \mathbf{d}^{\text{H}} \mathbf{H}\_{\boldsymbol{\mathcal{K}}}^{\text{H}} \mathbf{H}\_{\boldsymbol{\mathcal{K}}} \mathbf{d}}{2N\_{0}}} \right) \right] \\ &= c\_{1} \left( \frac{\|\boldsymbol{\sigma}\_{h}^{2} \mathbf{d}\|^{2}}{N\_{0}} \right)^{-N\lfloor\frac{\mathbf{K}}{M}\rfloor} + o \left( \left( \frac{\|\boldsymbol{\sigma}\_{h}^{2} \mathbf{d}\|^{2}}{N\_{0}} \right)^{-N\lfloor\frac{\mathbf{K}}{M}\rfloor + 1} \right), \end{split} \tag{44}$$

where *c*<sup>1</sup> > 0 is a constant. This completes the proof.

#### **8.2. Proof of Theorem 6.2**

*Proof.* It can be shown that under the ME criterion, for a given **H**K, an upper bound on the error probability in detecting *M* users' signals is expressed as

$$\begin{split} P\_{\varepsilon}^{\text{vmax}} &\leq \text{erfc}\left(\sqrt{\frac{\max\_{\mathcal{K}} \lambda\_{\text{min}}(\mathbf{H}\_{\mathcal{K}}^{\text{H}}\mathbf{H}\_{\mathcal{K}}) ||\mathbf{d}||^{2}}{2N\_{0}}}\right) \\ &= \text{erfc}\left(\sqrt{\frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2} \max\_{\mathcal{K}} \bar{X}\_{\mathcal{K}}}{2N\_{0}}}\right) \\ &= \text{erfc}\left(\sqrt{\frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2}V}{2N\_{0}}}\right). \end{split} \tag{45}$$

where *X*˜ <sup>K</sup> = *λ*min(**H**<sup>H</sup> <sup>K</sup>**H**K)/*σ*<sup>2</sup> *<sup>h</sup>* and *<sup>V</sup>* <sup>=</sup> max<sup>K</sup> *<sup>X</sup>*˜ <sup>K</sup>.

Using the pdf of *V* (with the same derivation for the ML case in the last subsection), it can be deduced that

$$\begin{split} P\_{\mathcal{E}}^{\text{mms}} &= E\_{\mathbf{H}\_{\mathcal{K}}} [\Pr \left( \mathbf{s}\_{(1)} \rightarrow \mathbf{s}\_{(2)} \right)] \\ &\leq E\_{V} \left[ \text{erfc} \left( \sqrt{\frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2} V}{2N\_{0}}} \right) \right] . \end{split} \tag{46}$$

For independent alternative combinations of the channel matrices **<sup>H</sup>**K<sup>1</sup> , **<sup>H</sup>**K<sup>2</sup> ,..., **<sup>H</sup>**K⌊ *<sup>K</sup> <sup>M</sup>* ⌋ , similar to the proof of Theorem 5.1, we can obtain that

$$\begin{split} &P\_{\varepsilon}^{\text{mms}} \leq E\_{V} \left[ \text{erfc} \left( \sqrt{\frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2} V}{2N\_{0}}} \right) \right] \\ &\leq \int\_{0}^{+\infty} \text{erfc} \left( \sqrt{\frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2} v}{2N\_{0}}} \right) f\_{V}(v) dv \\ &= c\_{2} \left( \frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2}}{N\_{0}} \right)^{-(N-P+1)\lfloor \frac{K}{M} \rfloor} + o \left( \left( \frac{\sigma\_{h}^{2} ||\mathbf{d}||^{2}}{N\_{0}} \right)^{-(N-P+1)\lfloor \frac{K}{M} \rfloor + 1} \right), \end{split} \tag{47}$$

where *c*<sup>2</sup> > 0 is constant. This completes the proof.

**8.3. Proof of Theorem 6.3**

*δ* | *rρ*,*<sup>ρ</sup>* |

<sup>4</sup> )−<sup>1</sup> <sup>&</sup>gt; <sup>4</sup>

Since **<sup>G</sup>** = **QR**, we have | *<sup>r</sup>*1,1 |

<sup>3</sup> , and

min *<sup>ρ</sup>* <sup>|</sup> *<sup>r</sup>ρ*,*<sup>ρ</sup>* <sup>|</sup>

<sup>2</sup>= �**g**1�<sup>2</sup> and

�**g**1�<sup>2</sup> ≥ min

min *<sup>ρ</sup>* <sup>|</sup> *<sup>r</sup>ρ*,*<sup>ρ</sup>* <sup>|</sup>

min *<sup>ρ</sup>* <sup>|</sup> *<sup>r</sup>ρ*,*<sup>ρ</sup>* <sup>|</sup>

where K is the index set of the selected users.

**d**∈D,**d**�=0

In the proposed user selection for selecting *M* users with the LR-based SIC detectors, (52)

Note that the LR-based SIC detection is considered. Let *n<sup>ρ</sup>* denote the *ρ*th element of **n**˜. Then,

<sup>4</sup> for all *<sup>ρ</sup>*. Thus, the error probability of the LR-based SIC detector can be

− min *q*


the LR-based SIC detection does not have error across all the layers if we have <sup>|</sup>*n<sup>ρ</sup>* <sup>|</sup>

Pr(error) <sup>≃</sup> exp

Note that the approximation in above becomes accurate as *<sup>N</sup>*<sup>0</sup> → 0 (or high SNR).

Then, we can obtain the following inequalities:

<sup>2</sup>≤| *<sup>r</sup>ρ*,*ρ*+<sup>1</sup> |

as

where *β* = (*δ* − <sup>1</sup>

Thus, we have

becomes


<sup>2</sup> < <sup>|</sup>*rρ*,*<sup>ρ</sup>* <sup>|</sup>

estimated by

2

*Proof.* In the LR algorithm, we transform the given channel matrix, e.g., **H**, into a new basis, e.g., denoted by **G**. Here, we have L(**G**) = L(**H**) ⇐⇒ **G** = **HT**, where **T** is an integer unimodular matrix and L(**A**) denotes the lattice generated by **A**. Then, **G** is called LLL-reduced with parameter *δ* if **G** is QR factorized as **G** = **QR** where **Q** is unitary, **R** is upper triangular, and the elements of **R** satisfies (29) and (30) with *m* = *M*. We rewrite (30)

<sup>2</sup> + | *<sup>r</sup>ρ*+1,*ρ*+<sup>1</sup> |

<sup>2</sup><sup>≥</sup> *<sup>β</sup>*−<sup>1</sup> <sup>|</sup> *<sup>r</sup>ρ*,*<sup>ρ</sup>* <sup>|</sup>

<sup>2</sup><sup>≥</sup> *<sup>β</sup>*−*MP*+<sup>1</sup> <sup>|</sup> *<sup>r</sup>*1,1 <sup>|</sup>


10.5772/57128

179

http://dx.doi.org/10.5772/57130

2, *ρ* = 1, 2, . . . , *MP* − 1. (48)

Lattice Reduction-Based User Selection in Multiuser MIMO Systems

2, (49)

<sup>2</sup> . (50)

<sup>|</sup>*rρ*,*<sup>ρ</sup>* <sup>|</sup> <sup>&</sup>lt; <sup>1</sup>

. (54)

<sup>2</sup> or

�**Hd**�<sup>2</sup> = V2(**H**). (51)

<sup>2</sup><sup>≥</sup> *<sup>β</sup>*−*MP*+1V2(**H**). (52)

<sup>2</sup><sup>≥</sup> *<sup>β</sup>*−*MP*+1V2(**H**K), (53)

## **8.3. Proof of Theorem 6.3**

20 Recent Trends in Multiuser MIMO Communications

**8.2. Proof of Theorem 6.2**

where *X*˜ <sup>K</sup> = *λ*min(**H**<sup>H</sup>

*P* mmse *<sup>e</sup>* <sup>≤</sup> *EV*

> ≤ � <sup>+</sup><sup>∞</sup> 0

= *c*<sup>2</sup>

be deduced that

where *c*<sup>1</sup> > 0 is a constant. This completes the proof.

*P* mmse *<sup>e</sup>* <sup>≤</sup> erfc

<sup>K</sup>**H**K)/*σ*<sup>2</sup>

*P* mmse

similar to the proof of Theorem 5.1, we can obtain that

erfc � *σ*2 *<sup>h</sup>* ||**d**||2*<sup>v</sup>* 2*N*<sup>0</sup>

where *c*<sup>2</sup> > 0 is constant. This completes the proof.

�*σ*2 *<sup>h</sup>* �**d**�<sup>2</sup> *N*<sup>0</sup>

 � *σ*2 *<sup>h</sup>* ||**d**||2*<sup>V</sup>* 2*N*<sup>0</sup>

 erfc

error probability in detecting *M* users' signals is expressed as

= erfc

= erfc

 �

 � *σ*2

 � *σ*2 *<sup>h</sup>* ||**d**||2*<sup>V</sup>* 2*N*<sup>0</sup>

*<sup>e</sup>* = *E***H**<sup>K</sup> [Pr

≤ *EV*

*<sup>h</sup>* and *<sup>V</sup>* <sup>=</sup> max<sup>K</sup> *<sup>X</sup>*˜ <sup>K</sup>.

 erfc

Using the pdf of *V* (with the same derivation for the ML case in the last subsection), it can

�

For independent alternative combinations of the channel matrices **<sup>H</sup>**K<sup>1</sup> , **<sup>H</sup>**K<sup>2</sup> ,..., **<sup>H</sup>**K⌊ *<sup>K</sup>*

 

*M* ⌋ + *o* �*σ*2 *<sup>h</sup>* �**d**�<sup>2</sup> *N*<sup>0</sup>

�−(*N*−*P*+1)⌊ *<sup>K</sup>*

*fV*(*v*)*dv*

 � *σ*2 *<sup>h</sup>* ||**d**||2*<sup>V</sup>* 2*N*<sup>0</sup>

**<sup>s</sup>**(1) <sup>→</sup> **<sup>s</sup>**(2)

� ]

> 

> > �−(*N*−*P*+1)⌊ *<sup>K</sup>*

*<sup>M</sup>* ⌋+<sup>1</sup> 

, (47)

*Proof.* It can be shown that under the ME criterion, for a given **H**K, an upper bound on the

max<sup>K</sup> *λ*min(**H**<sup>H</sup>

*<sup>h</sup>* ||**d**||<sup>2</sup> max<sup>K</sup> *<sup>X</sup>*˜ <sup>K</sup> 2*N*<sup>0</sup>

2*N*<sup>0</sup>

<sup>K</sup>**H**K)||**d**||<sup>2</sup>

   

, (45)

. (46)

*<sup>M</sup>* ⌋ ,

*Proof.* In the LR algorithm, we transform the given channel matrix, e.g., **H**, into a new basis, e.g., denoted by **G**. Here, we have L(**G**) = L(**H**) ⇐⇒ **G** = **HT**, where **T** is an integer unimodular matrix and L(**A**) denotes the lattice generated by **A**. Then, **G** is called LLL-reduced with parameter *δ* if **G** is QR factorized as **G** = **QR** where **Q** is unitary, **R** is upper triangular, and the elements of **R** satisfies (29) and (30) with *m* = *M*. We rewrite (30) as

$$\delta \mid r\_{\rho,\rho} \mid^2 \le \vert \; r\_{\rho,\rho+1} \vert^2 + \vert \; r\_{\rho+1,\rho+1} \vert^2, \; \rho = 1,2,\dots,MP-1. \tag{48}$$

Then, we can obtain the following inequalities:

$$|\;r\_{\rho+1,\rho+1}\;|^2 \ge \mathcal{J}^{-1} \mid r\_{\rho,\rho}\;|^2 \,. \tag{49}$$

where *β* = (*δ* − <sup>1</sup> <sup>4</sup> )−<sup>1</sup> <sup>&</sup>gt; <sup>4</sup> <sup>3</sup> , and

$$\min\_{\rho} \mid r\_{\rho,\rho} \mid^2 \ge \beta^{-MP+1} \mid r\_{1,1} \mid^2. \tag{50}$$

Since **<sup>G</sup>** = **QR**, we have | *<sup>r</sup>*1,1 | <sup>2</sup>= �**g**1�<sup>2</sup> and

$$\|\|\mathbf{g}\_1\|\|^2 \ge \min\_{\mathbf{d}\in\mathcal{D}, \mathbf{d}\neq 0} \|\|\mathbf{H}\mathbf{d}\|\|^2 = \mathcal{V}^2(\mathbf{H}).\tag{51}$$

Thus, we have

$$\min\_{\rho} |\ r\_{\rho,\rho}|^2 \ge \beta^{-MP+1} \mathcal{V}^2(\mathbf{H}).\tag{52}$$

In the proposed user selection for selecting *M* users with the LR-based SIC detectors, (52) becomes

$$\min\_{\rho} \mid r\_{\rho,\rho} \mid^2 \ge \beta^{-MP+1} \mathcal{V}^2(\mathbf{H}\_{\mathcal{K}})\_{\prime} \tag{53}$$

where K is the index set of the selected users.

Note that the LR-based SIC detection is considered. Let *n<sup>ρ</sup>* denote the *ρ*th element of **n**˜. Then, the LR-based SIC detection does not have error across all the layers if we have <sup>|</sup>*n<sup>ρ</sup>* <sup>|</sup> <sup>|</sup>*rρ*,*<sup>ρ</sup>* <sup>|</sup> <sup>&</sup>lt; <sup>1</sup> <sup>2</sup> or |*nρ*| <sup>2</sup> < <sup>|</sup>*rρ*,*<sup>ρ</sup>* <sup>|</sup> 2 <sup>4</sup> for all *<sup>ρ</sup>*. Thus, the error probability of the LR-based SIC detector can be estimated by

$$\Pr(\text{error}) \simeq \exp\left(-\min\_{\eta} \frac{|r\_{\rho,\rho}|^2}{4N\_0}\right). \tag{54}$$

Note that the approximation in above becomes accurate as *<sup>N</sup>*<sup>0</sup> → 0 (or high SNR).

Substituting (53) into (54), we have

$$\begin{split} \Pr(\text{error}) &\leq \exp\left(-\boldsymbol{\beta}^{-\text{MP}+1} \mathcal{S}^2(\mathbf{H}\_{\mathcal{K}})\right) \\ &\leq \sum\_{\mathbf{d}\in\mathcal{D}, \mathbf{d}\neq 0} \exp\left(-\boldsymbol{\beta}^{-\text{MP}+1} \frac{\max\_{\mathcal{K}} \mathbf{d}^H \mathbf{H}\_{\mathcal{K}}^H \mathbf{H}\_{\mathcal{K}} \mathbf{d}}{2N\_0}\right). \end{split} \tag{55}$$

Then, with the same approach used in the proof of Theorem 5.1, we can show that the upper bound on the average PEP is

$$P\_{\varepsilon}^{\mathbb{V}} \le c\_3 \left( \frac{\|\sigma\_h^2 \mathbf{d}\|^2}{N\_0} \right)^{-N\lfloor \frac{K}{M} \rfloor} + o\left( \left( \frac{\|\sigma\_h^2 \mathbf{d}\|^2}{N\_0} \right)^{-N\lfloor \frac{K}{M} \rfloor + 1} \right),\tag{56}$$

where *<sup>c</sup>*<sup>3</sup> > 0 is constant. This completes the proof.
