**1. Introduction**

In physical layer communications, the transceiver design is a core technology in multiple-input multiple-output (MIMO) systems, as shown in **Figure 1**. Iterative optimization algorithms for the transceiver design have achieved satisfactory system performance, but they generally require a large number of iterations and have the high-complexity computation, which makes it difficult to be deployed in practical systems. Recently, the deep learning (DL) method, as a primary technique in artificial intelligence, has received great attention in wireless communications, especially in physical layer communications. DL methods employ the deep neural networks (DNNs) and treat the algorithm as a "black-box". Compared to conventional optimization algorithms, DL methods can approximate high-complexity operations with

**Figure 1.** *The architecture of transceiver in physical layer communications.*

lower computational complexity. These DNNs are usually data-driven models, which rely on a large number of data for training. However, it is difficult to obtain training samples in practical communication systems and these data-driven DNNs suffer from poor interpretability and generalization ability. In contrast, model-driven methods exploit known physical mechanisms and domain knowledge. Thus, they require less training samples and it makes the DNNs explainable. Some studies unfolded the iterative optimization algorithms into layer-wise networks with introduced trainable parameters, which reduce the iteration numbers and improve the system performance. This chapter discusses the application of DL-based approaches in physical layer communications, which includes parts of channel estimation and feedback, beamforming, detection, channel decoding and end-to-end learning. Each part will be introduced with data-driven and model-driven approaches.

#### **1.1 Channel estimation and feedback**

In massive MIMO systems, the base station (BS) relies on accurate channel state information (CSI) to achieve potential gains from multiple antennas. However, the large number of antennas brings challenges and huge overhead for channel estimation and feedback, where many DL-based methods have been proposed to exploit the features of CSI and reduce the overhead [1–7]. In [2], the authors exploited the channel sparsity in the angle domain and proposed a DNN for channel estimation and direction-of-arrival (DoA) estimation. The proposed DL method can learn the spatial structures of channels and achieve better performance than conventional methods. As for channel feedback, the CsiNet has been developed in ref. [3] for channel compression, feedback, and reconstruction. It employs the structure of an autoencoder, where an encoder and a decoder are designed for channel compression and construction, respectively. Compared to the traditional compressive sensing (CS) algorithm, the CsiNet improves the CSI recovery quality and compression ratio.

Model-driven DL approaches have also been applied for channel estimation and feedback [5–7]. In ref. [5], a learned denoising-based approximate message passing (LDAMP) network has been proposed for beamspace millimeter-wave (mmWave) MIMO channel estimation, where the convolutional denoising NN is merged into the AMP channel estimation algorithm. In addition, the authors of ref. [6] proposed a dynamic deep-unfolding neural networks (NNs) with adaptive depth for channel estimation, where the layers of NN vary from different inputs. As shown in **Figure 2 (a)**, ℱð Þ� denotes each layer of the NN and a function *ϕ* is defined to control the depth of the NN. When the output of the function *τ* >*ε*, the NN stops to output results. To estimate CSI, the sparse Bayesian learning (SBL) algorithm is unfolded into a layered network with introduced trainable parameters in the framework. In particular, some priori parameters which are difficult to determine in the SBL algorithm are set as

*Deep Learning for MIMO Communications DOI: http://dx.doi.org/10.5772/intechopen.112038*

**Figure 2.** *DDPG-driven deep-unfolding framework with adaptive depth.*

trainable parameters. The other trainable parameters are introduced to approximate the operations with high computational complexity. Besides, to avoid gradient explosion, the trainable parameters are updated by deep deterministic policy gradient (DDPG), rather than updated by the stochastic gradient descent (SGD) algorithm directly. As shown in **Figure 2(b)**, the state, action, and state transition of DDPG correspond to the optimization variables, trainable parameters, and architecture of NN, respectively. As for channel feedback, the authors of ref. [7] proposed a modeldriven multiple-measurement-vectors learned approximate message passing (MMV-LAMP) network for channel estimation and feedback in frequency division duplex (FDD) systems, which reduces the pilot feedback overhead.

#### **1.2 Beamforming**

In massive MIMO systems, beamforming has been a key technique to improve the spectrum efficiency and achieve spatial multiplexing gains. Traditional beamforming algorithms require a large number of iterations and high-complexity computations, which impedes their application in practical systems, especially when the number of antennas is large. Thus, many DL-based approaches for beamforming design have been proposed [8–14]. In ref. [8], the authors proposed a DNN-enabled massive MIMO framework for effective hybrid beamforming. Compared to conventional schemes, the proposed framework achieves better performance with lower computational complexity. Besides, a DL-based joint channel feedback and beamforming approach has been designed in ref. [9]. In addition, deep reinforcement learning (DRL) based beamforming algorithms have also been developed in ref. [10, 11]. The authors of ref. [10] proposed a DRL hybrid beamforming scheme to improve the coverage range of THz communications in the reconfigurable intelligent surfaces (RIS) assisted system.

Apart from the aforementioned data-driven NNs, researchers developed the model-driven methods where iterative beamforming algorithms are unfolded into networks [12–14]. In ref. [12], the authors proposed an iterative algorithm-induced deep-unfolding neural network (IAIDNN) for digital beamforming shown in

#### **Figure 3.**

*The architecture of IAIDNN which unfolds the WMMSE algorithm into a layer-wise network.*

**Figure 3**, where the weighted minimum mean-square error (WMMSE) iterative algorithm is unfolded into a layer-wise network. ℱ, G, and J denote the layers of the network for updating different variables. In particular, inspired by the first-order Taylor expansion, the matrix inversion **A**�<sup>1</sup> is approximated by **A**† **X** þ **AY** þ **Z**, where **X**, **Y**, and **Z** are introduced trainable parameters. **A**† represents the proposed nonlinear operation where the diagonal elements of **A** are taken the reciprocal and nondiagonal elements are set as 0. The computational complexity of matrix inversion is <sup>O</sup> *<sup>n</sup>*<sup>3</sup> ð Þ while that of the proposed approximation is <sup>O</sup> *<sup>n</sup>*<sup>2</sup>*:*<sup>37</sup> ð Þ. In the backpropagation, the authors derived the generalized chain rule (GCR) in matrix form and the trainable parameters are updated based on it. Simulations have shown that the proposed IAIDNN achieves the performance of the WMMSE algorithm with much less iterations. Besides, the authors of ref. [13] developed a deep-unfolding framework for the passive and active beamforming joint design in a RIS-assisted MIMO system, which outperforms the conventional iterative algorithms.

#### **1.3 MIMO detection**

In MIMO systems, the detector plays an important role in the receiver. Traditional iterative detection algorithms are designed based on the assumption that the channel model is subject to a specific distribution, thus the performance is unsatisfactory in variable environments. To tackle the issue, DL-based detectors have been proposed [1, 15–20]. In refs. [1, 15, 16], the authors proposed a DNN-based joint channel estimation and signal detection algorithm for the receiver design where the detectors are designed to adapt to different wireless channels.

Several model-driven DL-based methods based on conventional iterative detectors have also been investigated in recent years [17–20]. The authors of ref. [17] unfolded the orthogonal approximate message passing (OAMP) detector into a layer-wise structure named OAMP-Net2. It only introduces a few learnable parameters to improve the stability and speed of convergence, and the parameters are optimized to adapt to different channel environments. Besides, a deep detector named LoRD-Net has been proposed for signal detection [18]. The LoRD-Net incorporates domain knowledge in its architecture design, thus requiring much fewer parameters than data-driven NNs. Furthermore, a joint channel estimation and signal detection modeldriven NN has been proposed in ref. [20] to reduce the effect of channel estimation errors on detection.

#### **1.4 Channel decoding**

With the development of fifth generation (5G), user data and system capacity have rapidly increased and a higher transmission rate means lower decoding latency demand. However, traditional decoders require high-complexity computation and a large number of iterations. To address the issue, DL-based decoding algorithms have been developed [21–26]. In ref. [21], researchers explored that it is easier for DL decoders to learn the structured codes than random codes and verified that NNs can learn a form of decoding algorithm, rather than only a classifier. The article [22] focuses on the issue that the successive interference cancelation (SIC) decoding is imperfect in the nonorthogonal multiple access (NOMA) system and proposes a novel DL-based scheme for decoding in MIMO-NOMA systems. A non-linear precoder and SIC decoder have been constructed by deep feedforward neural networks (FNNs) which help received signals decode accurately in the SIC manner.

The prior parameters play an important role in conventional iterative decoders but are usually set by experience. Thus, DL is a proper method to find the optimal value for the prior parameters and thus model-driven based decoding methods are promising techniques [24–26]. The authors of ref. [24] utilized the DL method to find the proper weights to the passing messages in the Tanner graph and achieved comparable performance with belief propagation (BP) decoders with less iterations. Furthermore, considering many expensive multiplication operations in ref. [24] which make it difficult to implement, Lugosch and Gross [25] proposed a neural offset min-sum decoding algorithm with no multiplications and less parameter computation. The proposed approach speeds up the training process and is friendly for hardware implementation. In addition, a model-driven low-density parity-check (LDPC) decoding network has been developed in ref. [26]. The iterative decoding progress between checking nodes and variable nodes is unfolded into a propagation network, which combines the advantages of deep learning and conventional normalized min-sum LDPC decoding methods.

#### **1.5 End-to-end learning**

The aforementioned DL-based approaches are optimized locally for individual modules, where global optimality cannot be guaranteed. The modules in the transceiver are usually highly correlated with each other, and thus a joint design can achieve better performance than a separate design. To fulfill the global optimization, several DL-based end-to-end communication systems have been proposed [27–32], where all the trainable parameters are updated based on an end-to-end loss. A DNNbased based end-to-end wireless communication system has been proposed in ref. [27], which includes channel encoding, decoding, modulation, and equalization. A conditional generative adversarial net (GAN) has been designed to model the channel distribution and the proposed end-to-end approach is effective on Rayleigh fading channels. In ref. [28], the authors proposed a DNN-based end-to-end joint transceiver design algorithm for FDD mmWave MIMO systems, which consists of the modules of pilot training, channel feedback and reconstruction, and hybrid beamforming. To avoid CSI mismatch caused by the transmission delay and feedback overhead, a twotimescale scheme has been considered. Specifically, a superframe is introduced as the

long-timescale, where the CSI statistics remain constant. Each superframe consists of several frames, each of which contains a number of time slots that defined the short timescale. During each time slot, the instantaneous CSI remains unchanged. Correspondingly, a two-timescale DNN is developed as shown in **Figure 4**, which consists of a long-term DNN and a short-term DNN. The long-term DNN consists of modules of pilot training, high-dimensional CSI estimation and feedback, and hybrid beamforming, while the short-term DNN composes of modules of pilot training, lowdimensional equivalent CSI estimation and feedback, and digital beamforming. At the end of each frame, the long-term DNN is employed to obtain the high-dimensional full CSI to update the long-term analog beamformers. The short-term digital beamformers are updated based on the low-dimensional equivalent CSI acquired by the short-term DNN. The trainable parameters of all the modules are optimized to minimize the bit-error-rate (BER), which is the system's global optimization objective.

Inspired by ref. [28], the authors of ref. [31] designed a model-driven based endto-end framework for joint transceiver design in time division duplexing (TDD) systems, which consists of a channel estimation deep-unfolding NN (CEDUN) and a hybrid beamforming deep-unfolding NN (HBDUN). As shown in **Figure 5**, the CEDUN is comprised of a pilot training NN and a recursive least squares (RLS) algorithm-induced deep-unfolding NN, where a set of trainable parameters are introduced to increase the degrees of freedom. For hybrid beamforming, the stochastic successive convex approximation (SSCA) algorithm is unfolded into a layer-wise structure in the HBDUN, which consists of an analog NN and a digital NN. Specifically, the phase of analog beamformers is set as trainable parameters in the analog NN. In the digital NN, two non-linear operations are introduced to approximate the matrix inversion. In addition, the authors consider the mixed-timescale scheme, where longterm analog beamformers are optimized based on the CSI statistics during a superframe, and short-term digital beamformers are updated in each time slot based on equivalent CSI. According to the mixed-timescale scheme, a novel two-stage training method is investigated to jointly train the framework. **Figure 5(a)** shows the first stage of training, where the trainable parameters of HBDUN are optimized with the loss function of the negative system sum rate. The second training stage is shown in **Figure 5(b)**, where the parameters of analog NN are fixed, and low-dimensional equivalent CSI is obtained. The parameters of CEDUN and digital NN are optimized in

**Figure 4.** *The architecture of the proposed DNN-based end-to-end framework.*

**Figure 5.** *The architecture of the end-to-end deep-unfolding framework.*

**Figure 6.**

*The architecture of the end-to-end DRL and deep-unfolding framework.*

this stage with the same loss function in the first stage. Simulation results have shown that the deep-unfolding NNs perform comparable with the traditional algorithms with reduced complexity and the joint design method achieves better performance than the separate design.

In addition, the authors of ref. [32] proposed an end-to-end DRL and deepunfolding framework for joint beam selection and digital beamforming design, the architecture of which is shown in **Figure 6**. Specifically, the framework consists of a DRL-based NN for beam selection and a model-driven based NN for digital beamforming. A novel training method has been developed to jointly train the DRLbased NN and unfolding NN in an end-to-end way. This work indicates that the model-driven NNs can be trained with other DL methods such as DRL-based NN.
