1. Introduction

The performance of high-capacity optical communication systems can be significantly degraded by fiber attenuation, chromatic dispersion (CD), polarization mode dispersion (PMD), laser phase noise (PN), and Kerr nonlinearities [1–10]. Using coherent detection, the powerful compensation of transmission impairments can be implemented in electrical domain. With the full information of the received signals, the chromatic dispersion, the polarization mode dispersion, the carrier phase noise, and the fiber Kerr nonlinearities can be equalized and mitigated using digital signal processing (DSP) [11–22].

Due to the high sensitivity of the receiver, coherent optical transmission was investigated extensively in the eighties of last century [23, 24]. However, the development of coherent communication has been delayed for nearly 20 years after that period [25, 26]. Coherent optical detection re-attracted the research interests until 2005, since the advanced modulation formats, i.e., m-level phase shift keying (m-PSK) and m-level quadrature amplitude modulation (m-QAM), can be applied [27–30]. In addition, coherent optical detection allows the electrical mitigation of system impairments. With the two main merits, the reborn coherent detections brought us the enormous potential for higher transmission speed and spectral efficiency in current optical fiber communication systems [31, 32].

With an additional local oscillator (LO) source, the sensitivity of coherent receiver reached the limitation of the shot-noise. Furthermore, compared to the traditional intensity modulation direct detection system, the multilevel modulation formats can be applied using the phase modulations, which can include more information bits in one transmitted symbol than before.

Meanwhile, since the coherent demodulation is linear and all information of the received signals can be detected, signal processing approaches, i.e., tight spectral filtering, CD equalization, PMD compensation, laser PN estimation, and fiber nonlinearity compensation, can be implemented in electrical domain [33–40].

The typical block diagram of the coherent optical transmission system is shown in Figure 1. The transmitted optical signal is combined coherently with the continuous wave from the narrow-linewidth LO laser so that the detected optical intensity in the photodiode (PD) ends can be increased and the phase information of the optical signal can be obtained. The use of LO laser is to increase the receiver sensitivity of the detection of optical signals, and the performance of coherent transmission can even behave close to the Shannon limit [3, 12].

The development of the coherent transmission systems has stopped for more than 10 years due to the invention of Erbium-doped fiber amplifiers (EDFAs) [1, 2]. The coherent transmission techniques attracted the interests of investigation again around 2005, when a new stage of the coherent lightwave systems comes out by combining the digital signal processing techniques [41–46]. This type of coherent lightwave system is called as digital coherent communication system. In the digital coherent transmission systems, the electrical signals output from the photodiodes are sampled and transformed into the discrete signals using high-speed analogue-to-digital convertors (ADCs), which can be further processed by the DSP algorithms.

The phase locking and the polarization adjustment were the main obstacles in the traditional coherent lightwave systems, while they can be solved by the carrier phase estimation and the polarization equalization, respectively, in the digital coherent optical transmission systems [47–55]. Besides, the chromatic dispersion and the nonlinear effects can also be mitigated by using the digital signal processing techniques [56–62]. The typical structure of the DSP compensating modules in the digital coherent receiver is shown in Figure 2.

Figure 1. Schematic of coherent optical communication system with digital signal processing.

Figure 2. Block diagram of DSP in digital coherent receiver.

1. Introduction

298 Optical Fiber and Wireless Communications

before.

digital signal processing (DSP) [11–22].

current optical fiber communication systems [31, 32].

implemented in electrical domain [33–40].

The performance of high-capacity optical communication systems can be significantly degraded by fiber attenuation, chromatic dispersion (CD), polarization mode dispersion (PMD), laser phase noise (PN), and Kerr nonlinearities [1–10]. Using coherent detection, the powerful compensation of transmission impairments can be implemented in electrical domain. With the full information of the received signals, the chromatic dispersion, the polarization mode dispersion, the carrier phase noise, and the fiber Kerr nonlinearities can be equalized and mitigated using

Due to the high sensitivity of the receiver, coherent optical transmission was investigated extensively in the eighties of last century [23, 24]. However, the development of coherent communication has been delayed for nearly 20 years after that period [25, 26]. Coherent optical detection re-attracted the research interests until 2005, since the advanced modulation formats, i.e., m-level phase shift keying (m-PSK) and m-level quadrature amplitude modulation (m-QAM), can be applied [27–30]. In addition, coherent optical detection allows the electrical mitigation of system impairments. With the two main merits, the reborn coherent detections brought us the enormous potential for higher transmission speed and spectral efficiency in

With an additional local oscillator (LO) source, the sensitivity of coherent receiver reached the limitation of the shot-noise. Furthermore, compared to the traditional intensity modulation direct detection system, the multilevel modulation formats can be applied using the phase modulations, which can include more information bits in one transmitted symbol than

Meanwhile, since the coherent demodulation is linear and all information of the received signals can be detected, signal processing approaches, i.e., tight spectral filtering, CD equalization, PMD compensation, laser PN estimation, and fiber nonlinearity compensation, can be

The typical block diagram of the coherent optical transmission system is shown in Figure 1. The transmitted optical signal is combined coherently with the continuous wave from the narrow-linewidth LO laser so that the detected optical intensity in the photodiode (PD) ends can be increased and the phase information of the optical signal can be obtained. The use of LO laser is to increase the receiver sensitivity of the detection of optical signals, and the perfor-

The development of the coherent transmission systems has stopped for more than 10 years due to the invention of Erbium-doped fiber amplifiers (EDFAs) [1, 2]. The coherent transmission techniques attracted the interests of investigation again around 2005, when a new stage of the coherent lightwave systems comes out by combining the digital signal processing techniques [41–46]. This type of coherent lightwave system is called as digital coherent communication system. In the digital coherent transmission systems, the electrical

mance of coherent transmission can even behave close to the Shannon limit [3, 12].

#### 2. Digital signal processing for compensating transmission impairments

In this section, the chromatic dispersion compensation, polarization mode dispersion equalization, and carrier phase noise compensation are analyzed and discussed using corresponding DSP algorithms.

#### 2.1. Chromatic dispersion compensation

Digital filters involving the time-domain least-mean-square (TD-LMS) adaptive filter, the static time-domain finite impulse response (STD-FIR) filter, and the frequency-domain equalizers (FDEs) are investigated for CD compensation. The characters of these filters are analyzed based on a 28-Gbaud dual-polarization quadrature phase shift keying (DP-QPSK) coherent transmission system using postcompensation of dispersion. It is noted that the STD-FIR filter and the FDEs can also be used for the dispersion predistorted coherent communication systems.

#### 2.1.1. Time domain least-mean-square equalizer

The TD-LMS filter employs an iterative algorithm that incorporates successive corrections to weights vector in the negative direction of the gradient vector, which eventually leads to a minimum mean square error [34, 38, 63–65]. The transfer function of the TD-LMS digital filter can be described as follows:

$$y\_{out}(n) = \overrightarrow{W}\_{LMS}^H(n)\overrightarrow{x}\_{in}(n) \tag{1}$$

$$
\overrightarrow{\mathcal{W}}\_{\rm LMS}(n+1) = \overrightarrow{\mathcal{W}}\_{\rm LMS}(n) + \mu\_{\rm LMS} \overrightarrow{\mathbf{x}}\_{\rm in}(n) e^\*\_{\rm LMS}(n) \tag{2}
$$

$$
\sigma\_{LMS}(n) = d\_{LMS}(n) - y\_{out}(n) \tag{3}
$$

where x ! inðnÞ is the vector of received signals, yout(n) is the equalized output signal, n is the index of signal, W ! LMSðnÞ is the vector of tap weights, H is the Hermitian transform operator, dLMS(n) is the desired symbol, eLMS(n) is the error between the desired symbol and the output signal, \* is the conjugation operator, and μLMS is the step size. To ensure the convergence of tap weights W ! LMSðnÞ, the step size μLMS has to meet the condition of μLMS < 1=Umax, where Umax is the largest eigenvalue of the correlation matrix R ¼ x ! inðnÞx !<sup>H</sup> inðnÞ [63]. The TD-LMS dispersion compensation filter can be applied in the "decision-directed" or the "sequence-training" mode [63].

The tap weights in TD-LMS adaptive equalizer for 20 km fiber CD compensation are shown in Figure 3. The convergence for 9 tap weights in the TD-LMS filter with step size equal to 0.1 is shown in Figure 3(a), and it is found that the tap weights reach their convergence after ~5000 iterations. The distribution of the magnitudes of the converged tap weights is plotted in Figure 3(b), and it is found that the central tap weights take more dominant roles than the high-order tap weights [34, 66].

#### 2.1.2. Static time-domain finite impulse response filter

Compared with the iteratively updated TD-LMS filter, the tap weights in STD-FIR filter have a relatively simple specification [34, 67–69], the tap weight in STD-FIR filter is given by the following equations:

2.1. Chromatic dispersion compensation

300 Optical Fiber and Wireless Communications

2.1.1. Time domain least-mean-square equalizer

W !

is the largest eigenvalue of the correlation matrix R ¼ x

2.1.2. Static time-domain finite impulse response filter

munication systems.

can be described as follows:

where x !

weights W !

mode [63].

index of signal, W

!

high-order tap weights [34, 66].

following equations:

Digital filters involving the time-domain least-mean-square (TD-LMS) adaptive filter, the static time-domain finite impulse response (STD-FIR) filter, and the frequency-domain equalizers (FDEs) are investigated for CD compensation. The characters of these filters are analyzed based on a 28-Gbaud dual-polarization quadrature phase shift keying (DP-QPSK) coherent transmission system using postcompensation of dispersion. It is noted that the STD-FIR filter and the FDEs can also be used for the dispersion predistorted coherent com-

The TD-LMS filter employs an iterative algorithm that incorporates successive corrections to weights vector in the negative direction of the gradient vector, which eventually leads to a minimum mean square error [34, 38, 63–65]. The transfer function of the TD-LMS digital filter

> ! <sup>H</sup> LMSðnÞx !

LMSðnÞ þ μLMS x

inðnÞ is the vector of received signals, yout(n) is the equalized output signal, n is the

LMSðnÞ, the step size μLMS has to meet the condition of μLMS < 1=Umax, where Umax

dLMS(n) is the desired symbol, eLMS(n) is the error between the desired symbol and the output signal, \* is the conjugation operator, and μLMS is the step size. To ensure the convergence of tap

sion compensation filter can be applied in the "decision-directed" or the "sequence-training"

The tap weights in TD-LMS adaptive equalizer for 20 km fiber CD compensation are shown in Figure 3. The convergence for 9 tap weights in the TD-LMS filter with step size equal to 0.1 is shown in Figure 3(a), and it is found that the tap weights reach their convergence after ~5000 iterations. The distribution of the magnitudes of the converged tap weights is plotted in Figure 3(b), and it is found that the central tap weights take more dominant roles than the

Compared with the iteratively updated TD-LMS filter, the tap weights in STD-FIR filter have a relatively simple specification [34, 67–69], the tap weight in STD-FIR filter is given by the

! inðnÞe �

LMSðnÞ is the vector of tap weights, H is the Hermitian transform operator,

! inðnÞx !<sup>H</sup>

eLMSðnÞ ¼ dLMSðnÞ � youtðnÞ ð3Þ

inðnÞ ð1Þ

LMSðnÞ ð2Þ

inðnÞ [63]. The TD-LMS disper-

youtðnÞ ¼ W

!

LMSðn þ 1Þ ¼ W

Figure 3. Taps weights of TD-LMS filter. (a) Tap weights magnitudes convergence and (b) converged tap weights magnitudes distribution.

$$a\_k = \sqrt{\frac{jcT^2}{D\lambda^2 L}} \exp\left(-j\frac{\pi cT^2}{D\lambda^2 L}k^2\right) - \left\lfloor \frac{N}{2} \right\rfloor \le k \le \left\lfloor \frac{N}{2} \right\rfloor\tag{4}$$

$$N^A = 2 \times \left\lfloor \frac{|D|\lambda^2 L}{2cT^2} \right\rfloor + 1\tag{5}$$

where D is the CD coefficient, λ is the carrier central wavelength, L is the length of fiber, T is the sampling period, NA is the maximum number of taps, and ⌊x⌋ means the nearest integer smaller than x.

Figure 4. Tap weights of STD-FIR chromatic dispersion compensation filter.

For 20 km fiber with CD coefficient of D ¼ 16ps=ðnm � kmÞ, the distribution of the tap weights in the STD-FIR filter is shown in Figure 4.

#### 2.1.3. Frequency domain equalizers

Since the complexity is very low for compensating large CD [34, 70], the most promising and popular chromatic dispersion compensation filters in coherent transmission systems are the frequency domain equalizers. The transfer function of the frequency domain equalizers is given by the following expression:

$$G\_{\rm c}(L,\omega) = \exp\left(\frac{-jD\lambda^2\omega^2L}{4\pi\text{c}}\right) \tag{6}$$

where D is the chromatic dispersion coefficient, λ is the carrier central wavelength, ω is the angular frequency, L is the length of fiber, and c is the light speed in vacuum.

The frequency domain equalizers are generally implemented using the overlap-save (OLS) and the overlap-add (OLA) approaches based on the fast Fourier transform and the inverse fast Fourier transform (iFFT) convolution algorithms [71–73], as described in Figure 5.

#### 2.2. Polarization mode dispersion equalization

Due to the random character of the polarization mode dispersion and the polarization rotation, the compensation of the PMD and the polarization rotation are generally realized by the adaptive algorithms such as the least-mean-square (LMS) and the constant modulus algorithm (CMA) filters.

Figure 5. Schematic of frequency domain equalizer for chromatic dispersion compensation.

#### 2.2.1. LMS adaptive PMD equalization

For 20 km fiber with CD coefficient of D ¼ 16ps=ðnm � kmÞ, the distribution of the tap weights

Since the complexity is very low for compensating large CD [34, 70], the most promising and popular chromatic dispersion compensation filters in coherent transmission systems are the frequency domain equalizers. The transfer function of the frequency domain equalizers is given

> ω<sup>2</sup>L 4πc

ð6Þ

GcðL, <sup>ω</sup>Þ ¼ exp �jDλ<sup>2</sup>

angular frequency, L is the length of fiber, and c is the light speed in vacuum.

Fourier transform (iFFT) convolution algorithms [71–73], as described in Figure 5.

where D is the chromatic dispersion coefficient, λ is the carrier central wavelength, ω is the

The frequency domain equalizers are generally implemented using the overlap-save (OLS) and the overlap-add (OLA) approaches based on the fast Fourier transform and the inverse fast

Due to the random character of the polarization mode dispersion and the polarization rotation, the compensation of the PMD and the polarization rotation are generally realized by the adaptive algorithms such as the least-mean-square (LMS) and the constant modulus algorithm

in the STD-FIR filter is shown in Figure 4.

Figure 4. Tap weights of STD-FIR chromatic dispersion compensation filter.

2.2. Polarization mode dispersion equalization

2.1.3. Frequency domain equalizers

302 Optical Fiber and Wireless Communications

by the following expression:

(CMA) filters.

In the electrical domain, the impact of the PMD and the polarization fluctuation can be adaptively equalized using the decision-directed LMS (DD-LMS) filter [36, 63], of which the transfer function is given by:

$$
\begin{bmatrix} \mathbf{x}\_{out}(n) \\ \mathbf{y}\_{out}(n) \end{bmatrix} = \begin{bmatrix} \overrightarrow{\boldsymbol{w}}\_{xx}^{H}(n) & \overrightarrow{\boldsymbol{w}}\_{xy}^{H}(n) \\ \overrightarrow{\boldsymbol{w}}\_{yx}^{H}(n) & \overrightarrow{\boldsymbol{w}}\_{yy}^{H}(n) \end{bmatrix} \cdot \begin{bmatrix} \overrightarrow{\boldsymbol{x}}\_{in}(n) \\ \overrightarrow{\boldsymbol{y}}\_{in}(n) \end{bmatrix} \tag{7}
$$

$$\begin{cases} \overrightarrow{w}\_{\text{xx}}(n+1) = \overrightarrow{w}\_{\text{xx}}(n) + \mu\_p \cdot \varepsilon\_x(n) \cdot \overrightarrow{x}\_{in}^\*(n) \\\\ \overrightarrow{w}\_{yx}(n+1) = \overrightarrow{w}\_{yx}(n) + \mu\_p \cdot \varepsilon\_y(n) \cdot \overrightarrow{x}\_{in}^\*(n) \\\\ \overrightarrow{w}\_{xy}(n+1) = \overrightarrow{w}\_{xy}(n) + \mu\_p \cdot \varepsilon\_x(n) \cdot \overrightarrow{y}\_{in}^\*(n) \\\\ \overrightarrow{w}\_{yy}(n+1) = \overrightarrow{w}\_{yy}(n) + \mu\_p \cdot \varepsilon\_y(n) \cdot \overrightarrow{y}\_{in}^\*(n) \end{cases} \tag{8}$$
 
$$\begin{cases} \varepsilon\_x(n) = d\_x(n) - x\_{out}(n) \\\\ \varepsilon\_y(n) = d\_y(n) - y\_{out}(n) \end{cases} \tag{9}$$

where x ! inðnÞ and y ! inðnÞ are the vectors of the input signals, xout(n) and yout(n) are the equalized output signals, respectively, w ! xxðnÞ, w ! xyðnÞ, w ! yxðnÞ and w ! yyðnÞ are the complex tap weights vectors, dx(n) and dy(n) are the desired symbols, εx(n) and εy(n) are the estimation errors between the desired symbols and the output signals in the two polarizations, respectively, and μ<sup>p</sup> is the step size in the DD-LMS algorithm.

#### 2.2.2. CMA adaptive PMD equalization

The influence of the PMD and the polarization fluctuation can also be compensated employing the CMA adaptive filter [74, 75], of which the transfer function can be described as:

$$
\begin{bmatrix} \mathbf{x}\_{out}(n) \\ \mathbf{y}\_{out}(n) \end{bmatrix} = \begin{bmatrix} \stackrel{\cdot}{\boldsymbol{\nu}}\_{\text{xx}}^{H}(n) & \stackrel{\cdot}{\boldsymbol{\nu}}\_{\text{xy}}^{H}(n) \\ \stackrel{\cdot}{\boldsymbol{\nu}}\_{\text{yx}}^{H}(n) & \stackrel{\cdot}{\boldsymbol{\nu}}\_{\text{yy}}^{H}(n) \end{bmatrix} \cdot \begin{bmatrix} \stackrel{\cdot}{\boldsymbol{\overline{x}}\_{in}}(n) \\ \stackrel{\cdot}{\boldsymbol{\overline{y}}\_{in}}(n) \end{bmatrix} \tag{10}
$$

$$\begin{cases} \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm xx}(n+1) = \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm xx}(n) + \boldsymbol{\mu}\_{q} \cdot \boldsymbol{\eta}\_{\rm x}(n) \cdot \overrightarrow{\boldsymbol{x}}\_{\rm in}^{\*}(n) \\ \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm yx}(n+1) = \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm yx}(n) + \boldsymbol{\mu}\_{q} \cdot \boldsymbol{\eta}\_{\rm y}(n) \cdot \overrightarrow{\boldsymbol{x}}\_{\rm in}^{\*}(n) \\ \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm xy}(n+1) = \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm xy}(n) + \boldsymbol{\mu}\_{q} \cdot \boldsymbol{\eta}\_{\rm x}(n) \cdot \overrightarrow{\boldsymbol{y}}\_{\rm in}^{\*}(n) \\ \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm yy}(n+1) = \overrightarrow{\boldsymbol{\dot{\nu}}}\_{\rm yy}(n) + \boldsymbol{\mu}\_{q} \cdot \boldsymbol{\eta}\_{\rm y}(n) \cdot \overrightarrow{\boldsymbol{y}}\_{\rm in}^{\*}(n) \end{cases} \tag{11}$$

$$\eta\_{yy}(n+1) = \dot{\nu}\_{yy}(n) + \mu\_q \cdot \eta\_y(n) \cdot \dot{\nu}\_{in}(n)$$

$$\begin{cases} \eta\_x(n) = 1 - |\mathbf{x}\_{out}(n)|^2\\ \eta\_y(n) = 1 - |y\_{out}(n)|^2 \end{cases} \tag{12}$$

where x ! inðnÞ and y ! inðnÞ are the vectors of the input signals, xout(n) and yout(n) are the equalized output signals, respectively, ν ! xxðnÞ, ν ! xyðnÞ, ν ! yxðnÞ, and ν ! yyðnÞ are the complex tap weights vectors, ηx(n) and ηy(n) are the estimation errors between the desired amplitude and the output signals in the two polarizations, respectively, and μ<sup>q</sup> is the step size in the CMA algorithm.

It can be found that the CMA algorithm is based on the principle of minimizing the modulus variation of the output signal to update its weight vector.

#### 2.3. Carrier phase estimation

In this section, the analyses on different carrier phase estimation algorithms, involving the onetap normalized LMS, the differential phase estimation, the block-wise average (BWA), and the Viterbi-Viterbi (VV) methods in the coherent optical transmission systems, will be presented.

#### 2.3.1. The normalized LMS carrier phase estimation

ν !

The one-tap normalized LMS filter can be employed effectively for carrier phase estimation [76–78], of which the tap weight is expressed as:

$$\mathfrak{w}\_{\rm NLMS}(n+1) = \mathfrak{w}\_{\rm NLMS}(n) + \frac{\mu\_{\rm NLMS}}{\left| \mathfrak{x}\_{\rm in}(n) \right|^2} \mathfrak{x}\_{\rm in}^\*(n) \mathfrak{e}\_{\rm NLMS}(n) \tag{13}$$

$$c\_{\rm NLMS}(n) = d\_{\rm PE}(n) - w\_{\rm NLMS}(n) \cdot x\_{\rm in}(n) \tag{14}$$

where wNLMS(n) is the tap weight, xin(n) is the input signal, n is the symbol index, dPE(n) is the desired symbol, and eNLMS(n) is the carrier phase estimation error between the desired symbol and the output signal, and μNLMS is the step size in the one-tap normalized LMS filter.

It has been demonstrated that the one-tap normalized LMS carrier phase estimation behaves similar to the differential phase estimation [28, 53, 55, 76], of which the BER floor in the m-PSK coherent optical transmission systems can be approximately described by the following analytical expression:

$$BER\_{floor}^{NLMS} \approx \frac{1}{\log\_2 m} \text{erfc}\left(\frac{\pi}{m\sqrt{2}\sigma}\right) \tag{15}$$

where σ is the square root of the phase noise variance. The schematic of the one-tap normalized LMS carrier phase estimation is illustrated in Figure 6.

#### 2.3.2. Differential carrier phase estimation

xoutðnÞ youtðnÞ � �

> ν !

8 >>>>><

>>>>>:

where x !

inðnÞ and y

304 Optical Fiber and Wireless Communications

output signals, respectively, ν

2.3. Carrier phase estimation

lytical expression:

!

ν !

ν !

ν !

> ! xxðnÞ, ν ! xyðnÞ, ν !

variation of the output signal to update its weight vector.

2.3.1. The normalized LMS carrier phase estimation

[76–78], of which the tap weight is expressed as:

<sup>¼</sup> <sup>ν</sup> !<sup>H</sup> xxðnÞ ν !H xyðnÞ

xxðn þ 1Þ ¼ ν

yxðn þ 1Þ ¼ ν

xyðn þ 1Þ ¼ ν

yyðn þ 1Þ ¼ ν

(

2 6 4

ν !<sup>H</sup> yxðnÞ ν !H yyðn<sup>Þ</sup>

!

!

!

!

3 7 <sup>5</sup> � <sup>x</sup> ! inðnÞ y ! inðnÞ

xxðnÞ þ μ<sup>q</sup> � ηxðnÞ � x

yxðnÞ þ μ<sup>q</sup> � ηyðnÞ � x

xyðnÞ þ μ<sup>q</sup> � ηxðnÞ � y

yyðnÞ þ μ<sup>q</sup> � ηyðnÞ � y

yxðnÞ, and ν

inðnÞ are the vectors of the input signals, xout(n) and yout(n) are the equalized

!

<sup>η</sup>xðnÞ ¼ <sup>1</sup> � jxoutðnÞj<sup>2</sup> <sup>η</sup>yðnÞ ¼ <sup>1</sup> � jyoutðnÞj<sup>2</sup>

vectors, ηx(n) and ηy(n) are the estimation errors between the desired amplitude and the output signals in the two polarizations, respectively, and μ<sup>q</sup> is the step size in the CMA algorithm.

It can be found that the CMA algorithm is based on the principle of minimizing the modulus

In this section, the analyses on different carrier phase estimation algorithms, involving the onetap normalized LMS, the differential phase estimation, the block-wise average (BWA), and the Viterbi-Viterbi (VV) methods in the coherent optical transmission systems, will be presented.

The one-tap normalized LMS filter can be employed effectively for carrier phase estimation

where wNLMS(n) is the tap weight, xin(n) is the input signal, n is the symbol index, dPE(n) is the desired symbol, and eNLMS(n) is the carrier phase estimation error between the desired symbol

It has been demonstrated that the one-tap normalized LMS carrier phase estimation behaves similar to the differential phase estimation [28, 53, 55, 76], of which the BER floor in the m-PSK coherent optical transmission systems can be approximately described by the following ana-

and the output signal, and μNLMS is the step size in the one-tap normalized LMS filter.

jxinðnÞj<sup>2</sup> <sup>x</sup>�

eNLMSðnÞ ¼ dPEðnÞ � wNLMSðnÞ � xinðnÞ ð14Þ

wNLMSð<sup>n</sup> <sup>þ</sup> <sup>1</sup>Þ ¼ wNLMSðnÞ þ <sup>μ</sup>NLMS

" #

!� inðn<sup>Þ</sup>

!� inðn<sup>Þ</sup>

!� inðn<sup>Þ</sup>

!� inðn<sup>Þ</sup> ð10Þ

ð11Þ

ð12Þ

yyðnÞ are the complex tap weights

inðnÞeNLMSðnÞ ð13Þ

The differential signal demodulation can be also applied for carrier phase estimation in coherent transmission system [28, 53, 55], where the differentially encoded data can be recovered using the "delay and multiply" algorithm. Using differential carrier phase estimation, the encoded information can be recovered according to the phase difference between the two consecutive symbols, i.e., the decision variable <sup>Ψ</sup> <sup>¼</sup> xnx<sup>∗</sup> <sup>n</sup>þ<sup>1</sup> exp <sup>f</sup>iπ=mg, where xn and xn+1 are the consecutive n-th and (n+1)-th received symbols. The BER floor of the differential carrier phase estimation can be evaluated using the principle of conditional probability. For the m-PSK coherent systems, the BER floor in differential phase estimation is expressed as the following equation [28, 53]:

$$BER\_{floor}^{Differential} = \frac{1}{\log\_2 m} \text{erfc}\left(\frac{\pi}{m\sqrt{2}\sigma}\right) \tag{16}$$

where σ is the square root of the phase noise variance. The schematic of the differential carrier phase estimation is described in Figure 7.

Figure 6. Schematic of one-tap normalized LMS carrier phase estimation.

Figure 7. Schematic of differential carrier phase estimation.

#### 2.3.3. The block-wise average carrier phase estimation

The block-wise average approach calculates the m-th power of the received symbols in each processing unit to remove the information of phase modulation, and the computed phase is summed and averaged over the entire process block, where the length of the process block is called block size. Then the averaged phase is divided by m, and the result leads to the phase estimate for the entire data block [79–81]. For the m-PSK coherent communication system, the estimated carrier phase in each process block using the block-wise average approach is given by the following expression:

$$\hat{\Phi}\_{BWA}(n) = \frac{1}{m} \arg \left\{ \sum\_{k=1+(M-1)\cdot N\_b}^{M\cdot N\_b} \mathfrak{x}^m(k) \right\} \tag{17}$$

$$M = \left[\frac{n}{N\_b}\right] \tag{18}$$

where Nb is the block size in the BWA approach, and ⌈x⌉ means the nearest integer larger than x.

The performance of the block-wise average carrier phase estimation method in the m-PSK coherent optical communication system can be derived based on the Taylor expansion of the estimated carrier phase error, and the BER floor in the block-wise average carrier phase estimation can be described using the following expression [52, 53, 55, 79]:

$$BER\_{floor}^{BWA} \approx \frac{1}{N\_b \cdot \log\_2 m} \cdot \sum\_{k=1}^{N\_b} \text{erfc}\left(\frac{\pi}{m\sqrt{2}\sigma\_{BWA,k}}\right) \tag{19}$$

$$
\sigma\_{\text{BWA},k}^2 = \frac{\sigma^2}{6N\_b^2} \cdot \left[ 2(k-1)^3 + 3(k-1)^2 + 2(N\_b - k)^3 + 3(N\_b - k)^2 + N\_b - 1 \right] \tag{20}
$$

where σ<sup>2</sup> represents the total phase noise variance in the coherent transmission system. The schematic of the block-wise average carrier phase estimation is shown in Figure 8.

Figure 8. Schematic of block-wise average carrier phase estimation.

#### 2.3.4. The Viterbi-Viterbi carrier phase estimation

2.3.3. The block-wise average carrier phase estimation

<sup>Φ</sup>^ BWAðnÞ ¼ <sup>1</sup>

estimation can be described using the following expression [52, 53, 55, 79]:

Nb � log2<sup>m</sup> �

<sup>3</sup> <sup>þ</sup> <sup>3</sup>ð<sup>k</sup> � <sup>1</sup><sup>Þ</sup>

schematic of the block-wise average carrier phase estimation is shown in Figure 8.

f loor <sup>≈</sup> <sup>1</sup>

BERBWA

� 2ðk � 1Þ

Figure 8. Schematic of block-wise average carrier phase estimation.

<sup>m</sup> arg

8 < :

<sup>M</sup> <sup>¼</sup> <sup>n</sup> Nb � �

where Nb is the block size in the BWA approach, and ⌈x⌉ means the nearest integer larger than x. The performance of the block-wise average carrier phase estimation method in the m-PSK coherent optical communication system can be derived based on the Taylor expansion of the estimated carrier phase error, and the BER floor in the block-wise average carrier phase

> X Nb

erf c <sup>π</sup> m ffiffiffi 2 <sup>p</sup> <sup>σ</sup>BWA, <sup>k</sup> !

<sup>3</sup> <sup>þ</sup> <sup>3</sup>ðNb � <sup>k</sup><sup>Þ</sup>

<sup>2</sup> <sup>þ</sup> Nb � <sup>1</sup>

k¼1

where σ<sup>2</sup> represents the total phase noise variance in the coherent transmission system. The

<sup>2</sup> <sup>þ</sup> <sup>2</sup>ðNb � <sup>k</sup><sup>Þ</sup>

h i

M X�Nb k¼1þðM�1Þ�Nb

xmðk<sup>Þ</sup>

9 = ;

ð17Þ

ð18Þ

ð19Þ

ð20Þ

by the following expression:

306 Optical Fiber and Wireless Communications

σ2

BWA, <sup>k</sup> <sup>¼</sup> <sup>σ</sup><sup>2</sup>

6N<sup>2</sup> b

The block-wise average approach calculates the m-th power of the received symbols in each processing unit to remove the information of phase modulation, and the computed phase is summed and averaged over the entire process block, where the length of the process block is called block size. Then the averaged phase is divided by m, and the result leads to the phase estimate for the entire data block [79–81]. For the m-PSK coherent communication system, the estimated carrier phase in each process block using the block-wise average approach is given The Viterbi-Viterbi carrier phase estimation approach also operates the symbols in each process block into the m-th power to remove the information of the phase modulation. The computed phase is also summed and averaged over the entire process block, where the length of the process block is also called block size. Then the averaged phase is divided by m as the estimated carrier phase. However, compared to the BWA approach, the estimated phase in the Viterbi-Viterbi carrier phase estimation approach is only applied in the phase recovery of the central symbol in each process block [55, 81–83]. The estimated carrier phase in the Viterbi-Viterbi approach in m-PSK optical communication systems is given by the following expression:

$$\hat{\Phi}\_{VV}(n) = \frac{1}{m} \arg \left\{ \sum\_{k=-(N\_r-1)/2}^{(N\_v-1)/2} \mathbf{x}^m (n+k) \right\}, \ N\_v = 1, 3, 5, 7... \tag{21}$$

where N<sup>ν</sup> is the block size in the Viterbi-Viterbi carrier phase estimation approach.

The performance of the Viterbi-Viterbi carrier phase estimation in the m-PSK coherent optical communication system can also be derived employing the Taylor expansion of the estimated carrier phase. The BER floor in the Viterbi-Viterbi carrier phase estimation for the m-PSK transmission system can be expressed as follows [52, 53, 55]:

$$BER\_{floor}^{VV} \approx \frac{1}{\log\_2 m} \text{erfc}\left(\frac{\pi}{m\sqrt{2}\sigma\_{VV}}\right) \tag{22}$$

$$
\sigma\_{VV}^2 = \sigma^2 \cdot \frac{N\_v^2 - 1}{12N\_v} \tag{23}
$$

where σ<sup>2</sup> represents the total phase noise variance in the coherent transmission system. The schematic of the Viterbi-Viterbi carrier phase estimation is illustrated in Figure 9.

According to Eqs. (20) and (23), it can be found that the phase estimate error in the Viterbi-Viterbi carrier phase estimation corresponds to the phase estimate error of the central symbol (the smallest error) in the block-wise average carrier phase estimation. Therefore, the Viterbi-

Figure 9. Schematic of Viterbi-Viterbi carrier phase estimation.

Viterbi approach will generally perform better than the block-wise average approach, in terms of the phase estimate error. However, it requires more computational complexity to update the process unit for the phase estimation of each symbol.

It is noted that the one-tap normalized LMS algorithm can also be employed for the m-QAM coherent transmission systems, while the block-wise average and the Viterbi-Viterbi methods cannot be easily used for the classical m-QAM coherent systems except the circular constellation m-QAM systems.
