**Meet the editor**

Dr Aymeric Histace was born in 1977 in Lyon, France. He received the Master degree in Signals/Images for Computer Aided Diagnosis in 2001 (University of Angers, France). He received, in 2004, the Ph.D degree in signal and image processing at the Laboratoire d'Ingénierie des Systèmes Automatisés (LISA), University d'Angers, France. He is since 2006, an associate profes-

sor in signal and image processing with application to life sciences (mainly Computer Aided Diagnosis) at the University of Cergy-Pontoise, Cergy, France with ETIS laboratory (UMR CNRS 8051).

Contents

**Preface IX** 

Vitaly Kober

Chapter 3 **Entropic Image Restoration** 

Yuri S. Popkov

**Part 1 Theory and Scientific Advances 1** 

**with Running Sinusoidal Transforms 3** 

Chapter 2 **Statistical-Based Approaches for Noise Removal 19** 

Chapter 4 **Surface Topography and Texture Restoration** 

Chapter 6 **Regularized Image Restoration 119** 

Chapter 8 **Defocused Image Restoration** 

Liyun Su

Chapter 5 **Image Restoration via Topological Derivative 97**  Ignacio Larrabide and Raúl A. Feijóo

Chapter 7 **Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 145** 

Pradeepa D. Samarasinghe and Rodney A. Kennedy

**with Local Polynomial Regression and IWF 171** 

Chapter 9 **Image Restoration Using Two-Dimensional Variations 185**  Olga Milukova, Vitaly Kober and Victor Karnaukhov

Miguel A. Santiago, Guillermo Cisneros and Emiliano Bernués

State Luminiţa, Cătălina-Lucia Cocianu and Vlamos Panayiotis

**as a Dynamic System with Entropy Operator 45** 

**from Sectional Optical Imaging by Focus Analysis 73**  Mathieu Fernandes, Yann Gavet and Jean-Charles Pinoli

Chapter 1 **Space-Variant Image Restoration** 

## Contents

## **Preface XI**

	- **Part 2 Applications 227**
	- **Part 3 Interdisciplinarity 351**

## Preface

This book represents a sample of recent contributions of researchers all around the world in the field of image restoration.

The book consists of 15 chapters organized in three main sections (Theory, Applications, Interdisciplinarity). Topics cover some different aspects of the theory of image restoration. But this book is also an occasion to highlight some new topics of research related to the emergence of some original imaging devices (see "Applications" Section). From this arise some real challenging problems related to image reconstruction/restoration that open the way to some new fundamental scientific questions closely related with the world we interact with.

An effort has been made by each author to give a large access to their work: Image restoration is not only a problem that interests specialists but also other researchers from different areas (like Robotic, AI, etc.) who can find some inspiration in the proposed Chapters (let have a look to Chapter 15 for instance!)

This book is certainly a small sample of the research activity going on around the globe as you read it, but it surely covers a good deal of what has been done in the field recently, and as such it works as a valuable source for researchers interested in the involved subjects.

Special thanks to all authors who have interested a great deal of time to write such interesting chapters and who have accepted to share their work.

> **Aymeric Histace**  University of Cergy-Pontoise, Cergy, France

**Part 1** 

**Theory and Scientific Advances** 

## **Part 1**

**Theory and Scientific Advances** 

**1** 

Vitaly Kober

 *Mexico* 

**Space-Variant Image Restoration** 

*Computer Science Department, CICESE,* 

 **with Running Sinusoidal Transforms** 

Various restoration methods (linear, nonlinear, iterative, noniterative, deterministic, stochastic, etc.) optimized with respect to different criteria have been introduced (Bertero & Boccacci, 1998; Biemond et al., 1990; Kundur, & Hatzinakos, 1996; Banham & Katsaggelos, 1997; Jain, 1989; Bovik, 2005; Gonzalez & Woods, 2008). These techniques may be broadly divided in two classes: (i) fundamental algorithms and (ii) specialized algorithms. One of the most popular fundamental techniques is a linear minimum mean square error (LMMSE) method. It finds the linear estimate of the ideal image for which the mean square error between the estimate and the ideal image is minimum. The linear operator acting on the observed image to determine the estimate is obtained on the basis of a priori second-order statistical information about the image and noise processes. In the case of stationary processes and space-invariant blurs, the LMMSE estimator takes the form of the Wiener filter (Jain, 1989). The Kalman filter determines the causal LMMSE estimate recursively. Specialized algorithms can be viewed as extensions of the fundamental algorithms to specific restoration problems. It is based on a state-space representation of the imaging system, and image data are used to define the state vectors. Specialized algorithms can be viewed as extensions of the fundamental algorithms to specific restoration problems. In this paper we deal with restoration of images degraded by space-variant blurs. Basically, all fundamental algorithms apply to the restoration of images degraded by space-variant blurs. However, because Fourier transforms cannot be utilized when the blur is space-variant, space-domain implementations of these algorithms may be computationally formidable due to large matrix operations. Several specialized methods were developed to attack the spacevariant restoration problem. The first class referred to as sectioning is based on assumption that the blur is approximately space-invariant within local regions of the image. Therefore, the entire image can be restored by applying well-known space-invariant techniques to the local image regions. A drawback of sectioning methods is the generation of artifacts at the region boundaries. The second class is based on a coordinate transformation (Sawchuk, 1974), which is applied to the observed image so that the blur in the transformed coordinates becomes space-invariant. Therefore, the transformed image can be restored by a space-invariant filter and then transformed back to obtain the final restored image. However, the statistical properties of the image and noise processes are affected by the

**1. Introduction** 

## **Space-Variant Image Restoration with Running Sinusoidal Transforms**

Vitaly Kober *Computer Science Department, CICESE, Mexico* 

## **1. Introduction**

Various restoration methods (linear, nonlinear, iterative, noniterative, deterministic, stochastic, etc.) optimized with respect to different criteria have been introduced (Bertero & Boccacci, 1998; Biemond et al., 1990; Kundur, & Hatzinakos, 1996; Banham & Katsaggelos, 1997; Jain, 1989; Bovik, 2005; Gonzalez & Woods, 2008). These techniques may be broadly divided in two classes: (i) fundamental algorithms and (ii) specialized algorithms. One of the most popular fundamental techniques is a linear minimum mean square error (LMMSE) method. It finds the linear estimate of the ideal image for which the mean square error between the estimate and the ideal image is minimum. The linear operator acting on the observed image to determine the estimate is obtained on the basis of a priori second-order statistical information about the image and noise processes. In the case of stationary processes and space-invariant blurs, the LMMSE estimator takes the form of the Wiener filter (Jain, 1989). The Kalman filter determines the causal LMMSE estimate recursively. Specialized algorithms can be viewed as extensions of the fundamental algorithms to specific restoration problems. It is based on a state-space representation of the imaging system, and image data are used to define the state vectors. Specialized algorithms can be viewed as extensions of the fundamental algorithms to specific restoration problems. In this paper we deal with restoration of images degraded by space-variant blurs. Basically, all fundamental algorithms apply to the restoration of images degraded by space-variant blurs. However, because Fourier transforms cannot be utilized when the blur is space-variant, space-domain implementations of these algorithms may be computationally formidable due to large matrix operations. Several specialized methods were developed to attack the spacevariant restoration problem. The first class referred to as sectioning is based on assumption that the blur is approximately space-invariant within local regions of the image. Therefore, the entire image can be restored by applying well-known space-invariant techniques to the local image regions. A drawback of sectioning methods is the generation of artifacts at the region boundaries. The second class is based on a coordinate transformation (Sawchuk, 1974), which is applied to the observed image so that the blur in the transformed coordinates becomes space-invariant. Therefore, the transformed image can be restored by a space-invariant filter and then transformed back to obtain the final restored image. However, the statistical properties of the image and noise processes are affected by the

Space-Variant Image Restoration with Running Sinusoidal Transforms 5

First, we recall the definitions for various discrete sinusoidal transforms. Notation {.} denotes a matrix, the order of which is represented by a subscript. For clarity, the normalization factor 2 *N* for all forward transforms is neglected until the inverse

*ns DCT I k k*

<sup>1</sup> *N sn* cos ,

1 2 0 ,

*N* 

1/2

*N*

1/2

*N*

1/2 1/2

*N*

1/2

*N*

1/2

*N*

1/2 1/2

*N*

*n s*

, (6)

*n s*

(2)

, (3)

, (4)

, (5)

, (7)

, (8)

, (9)

*N*

transforms. The kernel of the orthogonal DCT-I for the order *N*+1 is defined as

1 . *m m n or m s if m or m N <sup>k</sup> otherwise*

> *N s* cos *s n DCT II k*

*N n* cos *n s DCT III k*

<sup>1</sup> *<sup>N</sup>* sin *ns DST I*

*N s* sin *s n DST II k*

*N n* sin *n s DST III k*

*<sup>N</sup>* sin

*DST IV*

where n, s=1, 2,…, N-1. The kernels of the DST-II, DST-III, and DST-IV for the order N are

The kernels of the DCT-II, DCT-III, and DCT-IV for the order *N* are given as

*<sup>N</sup>* cos

where *n, s*=0, 1,…, *N*-1. The kernel of the DST-I for the order *N*-1 is defined as

*DCT IV*

**2.1 Discrete sinusoidal transforms** 

where *n, s*=*0*,…, *N*;

given as follows:

where *n, s*=1, 2,…, *N*.

coordinate transformation. In particular, the stationarity in the original spatial coordinates is not preserved in the transform coordinate system.

In this chapter, we carry out the space-variant restoration using running discrete sinusoidal transform coefficients. The running transform is based on the concept of short-time signal processing (Oppenheim & Shafer 1989). A short-time orthogonal transform of a signal x*k* is defined as

$$X\_s^k = \sum\_{n=-\infty}^{\infty} x\_{k+n} w\_n \wp'(n, s) \, , \tag{1}$$

where w*n* is a window sequence, *(n,s)* represents the basis functions of an orthogonal transform. We use one-dimensional notation for simplicity. Equation (1) can be interpreted as the orthogonal transform of x*k+n* as viewed through the window w*n*. *<sup>k</sup> Xs* displays the orthogonal transform characteristics of the signal around time *k*. Note that while increased window length and resolution are typically beneficial in the spectral analysis of stationary data, for time-varying data it is preferable to keep the window length sufficiently short so that the signal is approximately stationary over the window duration. Assume that the window has finite length around *n*=0, and it is unity for all *n*-*N*1, *N*2. Here *N1* and *N2* are integer values. This leads to signal processing in a running window (Vitkus & Yaroslavsky, 1987; Yaroslavsky & Eden, 1996). In other words, local filters in the domain of an orthogonal transform at each position of a moving window modify the orthogonal transform coefficients of a signal to obtain only an estimate of the pixel x*k* of the window. The choice of orthogonal transform for running signal processing depends on many factors.

We carry out the space-variant restoration using running discrete transform coefficients. The discrete cosine transforms (DCT) and discrete sine transforms (DST) are widely used. This is because the DCT and DST perform close to the optimum Karhunen-Loeve transform (KLT) for the first-order Markov stationary data (Jain, 1989). For signals with the correlation coefficient near to unity, the DCT provides a better approximation of the KLT than the DST. On the other hand, the DST is closer to the KLT, when the correlation coefficient lies in the interval (-0.5, 0.5). Since the KLT is constructed from the eigenvectors of the covariance matrix of data, there are neither single unique transform for all random processes nor fast algorithms. Unlike the KLT, the DCT and DST are not data dependent, and many fast algorithms were proposed. To provide image processing in real time, fast recursive algorithms for computing the running sinusoidal transforms are utilized (Kober, 2004, 2007). We introduce local adaptive restoration of nonuniform degraded images using several running sinusoidal transforms. Computer simulation results using a real image are provided and compared with those of common restoration techniques.

## **2. Fast algorithms of running discrete sinusoidal transforms**

The discrete cosine and sine transforms are widely used in signal processing applications. Recently, forward and inverse algorithms for fast computing of various DCTs ({DCT-I, DCT-II, DCT-III, DCT-IV) and DSTs (DST-I, DST-II, DST-III, DST-IV) were proposed (Kober, 2004).

#### **2.1 Discrete sinusoidal transforms**

4 Image Restoration – Recent Advances and Applications

coordinate transformation. In particular, the stationarity in the original spatial coordinates is

In this chapter, we carry out the space-variant restoration using running discrete sinusoidal transform coefficients. The running transform is based on the concept of short-time signal processing (Oppenheim & Shafer 1989). A short-time orthogonal transform of a signal x*k* is

> , *<sup>k</sup> s kn n n*

 

transform. We use one-dimensional notation for simplicity. Equation (1) can be interpreted as the orthogonal transform of x*k+n* as viewed through the window w*n*. *<sup>k</sup> Xs* displays the orthogonal transform characteristics of the signal around time *k*. Note that while increased window length and resolution are typically beneficial in the spectral analysis of stationary data, for time-varying data it is preferable to keep the window length sufficiently short so that the signal is approximately stationary over the window duration. Assume that the window has finite length around *n*=0, and it is unity for all *n*-*N*1, *N*2. Here *N1* and *N2* are integer values. This leads to signal processing in a running window (Vitkus & Yaroslavsky, 1987; Yaroslavsky & Eden, 1996). In other words, local filters in the domain of an orthogonal transform at each position of a moving window modify the orthogonal transform coefficients of a signal to obtain only an estimate of the pixel x*k* of the window. The choice of orthogonal transform for running

We carry out the space-variant restoration using running discrete transform coefficients. The discrete cosine transforms (DCT) and discrete sine transforms (DST) are widely used. This is because the DCT and DST perform close to the optimum Karhunen-Loeve transform (KLT) for the first-order Markov stationary data (Jain, 1989). For signals with the correlation coefficient near to unity, the DCT provides a better approximation of the KLT than the DST. On the other hand, the DST is closer to the KLT, when the correlation coefficient lies in the interval (-0.5, 0.5). Since the KLT is constructed from the eigenvectors of the covariance matrix of data, there are neither single unique transform for all random processes nor fast algorithms. Unlike the KLT, the DCT and DST are not data dependent, and many fast algorithms were proposed. To provide image processing in real time, fast recursive algorithms for computing the running sinusoidal transforms are utilized (Kober, 2004, 2007). We introduce local adaptive restoration of nonuniform degraded images using several running sinusoidal transforms. Computer simulation results using a real image are

The discrete cosine and sine transforms are widely used in signal processing applications. Recently, forward and inverse algorithms for fast computing of various DCTs ({DCT-I, DCT-II, DCT-III, DCT-IV) and DSTs (DST-I, DST-II, DST-III, DST-IV) were proposed (Kober, 2004).

, (1)

*(n,s)* represents the basis functions of an orthogonal

*X x w ns*

provided and compared with those of common restoration techniques.

**2. Fast algorithms of running discrete sinusoidal transforms** 

not preserved in the transform coordinate system.

where w*n* is a window sequence,

signal processing depends on many factors.

defined as

First, we recall the definitions for various discrete sinusoidal transforms. Notation {.} denotes a matrix, the order of which is represented by a subscript. For clarity, the normalization factor 2 *N* for all forward transforms is neglected until the inverse transforms. The kernel of the orthogonal DCT-I for the order *N*+1 is defined as

$$DCT - I\_{N+1} = \left\{ k\_s k\_n \cos \left( \pi \frac{ns}{N} \right) \right\},\tag{2}$$

where *n, s*=*0*,…, *N*; 1 2 0 , 1 . *m m n or m s if m or m N <sup>k</sup> otherwise* 

The kernels of the DCT-II, DCT-III, and DCT-IV for the order *N* are given as

$$DCT - II\_N = \left\{ k\_s \cos \left( \pi \frac{s(n+1/2)}{N} \right) \right\},\tag{3}$$

$$DCT - III\_N = \left\{ k\_n \cos \left( \pi \frac{n(s+1/2)}{N} \right) \right\},\tag{4}$$

$$DCT - IV\_N = \left\{ \cos \left( \pi \frac{(n+1/2)(s+1/2)}{N} \right) \right\} \,\,\,\tag{5}$$

where *n, s*=0, 1,…, *N*-1. The kernel of the DST-I for the order *N*-1 is defined as

$$DST - I\_{N-1} = \left\{ \sin \left( \pi \frac{ns}{N} \right) \right\},\tag{6}$$

where n, s=1, 2,…, N-1. The kernels of the DST-II, DST-III, and DST-IV for the order N are given as follows:

$$DST - II\_N = \left\{ k\_s \sin\left(\pi \frac{s\left(n - 1/2\right)}{N}\right) \right\},\tag{7}$$

$$DST-III\_N = \left\{ k\_n \sin\left(\pi \frac{n\left(s-1\right/2\right)}{N} \right) \right\},\tag{8}$$

$$DST - IV\_N = \left\{ \sin \left( \pi \frac{(n - 1/2)(s - 1/2)}{N} \right) \right\},\tag{9}$$

where *n, s*=1, 2,…, *N*.

Space-Variant Image Restoration with Running Sinusoidal Transforms 7

The number of arithmetic operations required for computing the running discrete cosine transforms at a given window position is evaluated as follows. The SDCT-I for the order *N*+1 with *N*=*N*1+*N*2 requires *N*-1 multiplication operations and 4(*N*+2) addition operations. The SDCT-II for the order *N* with *N*= *N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N*+5 addition operations. A fast algorithm for the SDCT-III for the order *N* with *N*=*N*1+ *N*2+1 is based on the recursive equation given in line 3 of Table 1. Next it is useful to

<sup>2</sup>

 *Ns Ns N* (here [x/y] is the integer quotient) and Eq. (10), the number of operations required to compute the DSCT-III can be evaluated as [3/2*N*] multiplication operations and 4*N* addition operations. An additional memory buffer of *N* elements is also required. Finally, the SDCT-IV for the order *N* with *N*=*N*1+ *N*2+1 requires

where the array <sup>1</sup> ; 01 . 1 *k k X X x s , ,.. N s s kN* is stored in a memory buffer of *<sup>N</sup>*

The number of arithmetic operations required for computing the running discrete sine transforms at a given window position can be evaluated as follows. The SDST-I for the order *N*-1 with *N*=*N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N* addition operations.

symmetric on the interval [1, *N*-1]; that is, *f*(s)=*f*(*N*-*s*), *s*=1,..*N*/2-1. Therefore, only *N*/2-1 multiplication operations are required to compute this term. The total number of multiplications is reduced to 3*N*/2-2. The SDST-II for the order *N* with *N*=*N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N*+5 addition operations. Taking into account the property of symmetry of the sine and cosine functions, the SDST-III for the order *N* with *N*=*N*1+ *N*2+1 requires *2N* multiplications and *4N* addition operations. However, if *N* is even,

is symmetric on the interval [1, *N*]; that is, *g(s*)=*g(N-s*+1), *s*=1,..*N*/2. Therefore, only *N*/2 addition operations are required to compute the sum. If *N* is odd, the sum

interval [1, *N*]; that is, *p(s)=p(N-s*+1), *s*=1,..[*N*/2]. Hence, [*N*/2] addition operations are required to compute this sum. So, the total number of additions can be reduced to [7*N*/2]. Finally, the SDST-IV for the order *N* with *N*=*N*1+ *N*2+1 requires *3N* multiplication operations and *3N*+2 addition operations. The length of a moving window for the proposed algorithms

**2.3 Fast inverse algorithms for running signal processing with sinusoidal transforms**  The inverse discrete cosine and sine transforms for signal processing in a running window are performed for computing only the pixel xk of the window. The running signal

processing can be performed with the use of the SDCT and SDST algorithms.

elements. From the property of symmetry of the sine function, sin 1 / 2

1

 *k N* in line 7 of Table I

*sNx k N* in line 7 of Table I is symmetric on the

 

*s N* in line 5 of Table I is

*s N*

1/2 1/2

, (10)

*s s*

*N N*

1 1

 

3*N* multiplication operations and 3*N*+2 addition operations.

However, if *N* is even, 1 2 1 1 1 sin *<sup>s</sup> <sup>f</sup> s x k N xk N*

the sum 1 2 <sup>1</sup> ( ) sin 1 / 2 1 cos 1 / 2 *<sup>s</sup> <sup>g</sup> sx s N x s N k N*

1 2 1 1 sin 1 / 2 1 *<sup>s</sup> ps x k N*

may be an arbitrary integer.

*X XX X x*

cos 1 sin *k kk <sup>k</sup> <sup>s</sup> s ss s k N*

represent the equation as

sin 1 / 2 , 0,1,...[ / 2]

## **2.2 Fast forward algorithms for computing running discrete sinusoidal transforms**

The fast forward algorithms are based on recursive relationships between three subsequent local running spectra. These algorithms for running DCTs (SDCTs) and running DSTs (SDSTs) are based on the second–order recursive equations summarized in Table I.


Table 1. Recursive equations for the computation of forward running sinusoidal transforms.

6 Image Restoration – Recent Advances and Applications

11 22

*s*

*s kN kN kN kN*

*N*

1 1

*xx xx*

*kN kN*

2 2

*N*

 

*s s x x N N*

1 1

1 1

*N*

1/2 1/2 sin 1 cos

1/2 1/2 sin 1 cos 2 2

*s s x x N N*

*s s s kN kN*

 

*s k N k N*

1/2 2 cos <sup>x</sup>

2 2

1 sin

*N N*

 

*s*

*s s*

*kN kN*

1/2 2 cos <sup>x</sup>

2 2

1/2 1/2

2

1 1

2

*s*

1

*N*

1

 

*s*

2

1

2

 

1

1

1

  *s*

*N*

1

*N*

1 1

*N*

1

1

cos 1

2 cos

*k k s s s kN k N*

1

1

*k s s kN k N*

*Xx x*

1 1

*X Xx X*

1/2 <sup>2</sup> cos

*k k k s s kN s*

1 1

1

*x x*

cos 1 sin

*s*

2 cos

*x x*

*<sup>s</sup> XX X*

1 2

11 22

1/2 2 cos <sup>1</sup>

*kk k ss s*

*<sup>s</sup> XX X*

1 1

*k k k s s s s k N*

1 2

*s*

*X X Xx x N*

*k N k N*

*x x*

1 1

*s*

*k k k*

Table 1. Recursive equations for the computation of forward running sinusoidal transforms.

*xx xx*

1 1

*s X X X x N*

*kk k ss s*

1 sin

1 1

*s kN kN kN kN*

1 1

*s*

*k k k*

2 cos

1

1 2

*X X Xx x N*

1 1

*s s s kN kN*

*s k N k N*

1/2 1 sin

*N*

 

*kk k ss s*

*<sup>s</sup> XX X*

*<sup>s</sup> X Xx x*

2 1 cos

**2.2 Fast forward algorithms for computing running discrete sinusoidal transforms**  The fast forward algorithms are based on recursive relationships between three subsequent local running spectra. These algorithms for running DCTs (SDCTs) and running DSTs

(SDSTs) are based on the second–order recursive equations summarized in Table I.

1

2

*s*

*N*

N Transforms Recursive equations

1 SDCT-I

2 SDCT-II

3 SDCT-III

4 SDCT-IV

5 SDST-I

6 SDST-II

7 SDST-III

8 SDST-IV

The number of arithmetic operations required for computing the running discrete cosine transforms at a given window position is evaluated as follows. The SDCT-I for the order *N*+1 with *N*=*N*1+*N*2 requires *N*-1 multiplication operations and 4(*N*+2) addition operations. The SDCT-II for the order *N* with *N*= *N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N*+5 addition operations. A fast algorithm for the SDCT-III for the order *N* with *N*=*N*1+ *N*2+1 is based on the recursive equation given in line 3 of Table 1. Next it is useful to represent the equation as

$$X\_s^{k+1} = \left(X\_s^k + \tilde{X}\_s^k\right) \cos\left(\frac{\pi\left(s + 1/2\right)}{N}\right) - \tilde{X}\_s^{k-1} + \left(-1\right)^s x\_{k+N\_2+1} \sin\left(\frac{\pi\left(s + 1/2\right)}{N}\right),\tag{10}$$

where the array <sup>1</sup> ; 01 . 1 *k k X X x s , ,.. N s s kN* is stored in a memory buffer of *<sup>N</sup>* elements. From the property of symmetry of the sine function, sin 1 / 2 *s N* sin 1 / 2 , 0,1,...[ / 2] *Ns Ns N* (here [x/y] is the integer quotient) and Eq. (10), the number of operations required to compute the DSCT-III can be evaluated as [3/2*N*] multiplication operations and 4*N* addition operations. An additional memory buffer of *N* elements is also required. Finally, the SDCT-IV for the order *N* with *N*=*N*1+ *N*2+1 requires 3*N* multiplication operations and 3*N*+2 addition operations.

The number of arithmetic operations required for computing the running discrete sine transforms at a given window position can be evaluated as follows. The SDST-I for the order *N*-1 with *N*=*N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N* addition operations. However, if *N* is even, 1 2 1 1 1 sin *<sup>s</sup> <sup>f</sup> s x k N xk N s N* in line 5 of Table I is symmetric on the interval [1, *N*-1]; that is, *f*(s)=*f*(*N*-*s*), *s*=1,..*N*/2-1. Therefore, only *N*/2-1 multiplication operations are required to compute this term. The total number of multiplications is reduced to 3*N*/2-2. The SDST-II for the order *N* with *N*=*N*1+ *N*2+1 requires 2(*N*-1) multiplication operations and 2*N*+5 addition operations. Taking into account the property of symmetry of the sine and cosine functions, the SDST-III for the order *N* with *N*=*N*1+ *N*2+1 requires *2N* multiplications and *4N* addition operations. However, if *N* is even, the sum 1 2 <sup>1</sup> ( ) sin 1 / 2 1 cos 1 / 2 *<sup>s</sup> <sup>g</sup> sx s N x s N k N k N* in line 7 of Table I is symmetric on the interval [1, *N*]; that is, *g(s*)=*g(N-s*+1), *s*=1,..*N*/2. Therefore, only *N*/2 addition operations are required to compute the sum. If *N* is odd, the sum 1 2 1 1 sin 1 / 2 1 *<sup>s</sup> ps x k N sNx k N* in line 7 of Table I is symmetric on the interval [1, *N*]; that is, *p(s)=p(N-s*+1), *s*=1,..[*N*/2]. Hence, [*N*/2] addition operations are required to compute this sum. So, the total number of additions can be reduced to [7*N*/2]. Finally, the SDST-IV for the order *N* with *N*=*N*1+ *N*2+1 requires *3N* multiplication operations and *3N*+2 addition operations. The length of a moving window for the proposed algorithms may be an arbitrary integer.

#### **2.3 Fast inverse algorithms for running signal processing with sinusoidal transforms**

The inverse discrete cosine and sine transforms for signal processing in a running window are performed for computing only the pixel xk of the window. The running signal processing can be performed with the use of the SDCT and SDST algorithms.

Space-Variant Image Restoration with Running Sinusoidal Transforms 9

2 1/2 1/2

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse

<sup>1</sup>

*<sup>N</sup> <sup>s</sup> kk k*

 <sup>1</sup> 1

2 1 sin

*N N* 

where *N=N1+N2*+2. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse

 1

Therefore, in the computation only the spectral coefficients with odd indices are involved.

1

where *N=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse

<sup>1</sup>

<sup>1</sup> 21 1

*x XX*

*k sN*

In the computation only the spectral coefficients with odd indices are involved. The computational complexity is one multiplication operation and *N1*+1 addition operations.

<sup>2</sup> 1 1/2 sin

*N N* 

2 1

 

*k s N*

<sup>1</sup> 1/2 2 sin <sup>1</sup>

*x X X*

0 <sup>2</sup> <sup>1</sup> *<sup>N</sup> <sup>s</sup> <sup>k</sup> k s s x X N* 

*k*

 

0

The inverse algorithms for the running DSTs are given as follows.

*N*

*k s s*

*x X*

transform is given by

transform is simplified to

transform is given by

computation.

IDST-I:

IDST-II:

IDST-III:

cos

1 <sup>2</sup> <sup>1</sup>

1

*N*

*k s s*

The complexity is one multiplication operation and *N1* addition operations.

*N N* 

1

*N*

*s*

1

*N k*

*k s s*

*x X*

0

1

*N*

*s*

*N*

1

*x X*

*s*

*k s s*

*x XX X N*

One multiplication operation and *N*-1 addition operations are required to perform this

*N N* 

*k*

 <sup>1</sup> 1

*N s*

2 21 0

*N s*

2 1

<sup>1</sup>

*s N k k*

<sup>1</sup>

*N s*

, (21)

1

. (22)

, (23)

*k k N*

*N s*

, (17)

. (18)

, (19)

. (20)

The inverse algorithms for the running DCTs can be written as follows.

IDCT-I:

$$\mathbf{x}\_{k} = \frac{1}{N} \left( 2 \sum\_{s=1}^{N-1} X\_{s}^{k} \cos \left( \pi \frac{N\_{1}s}{N} \right) + X\_{0}^{k} + \left( -1 \right)^{N\_{1}} X\_{N}^{k} \right) \tag{11}$$

where *N=N1+N2*. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse transform is simplified to

$$\chi\_k = \frac{1}{N} \left( 2 \sum\_{s=1}^{N\_1 - 1} \left( -1 \right)^s X\_{2s}^k + X\_0^k + \left( -1 \right)^{N\_1} X\_N^k \right). \tag{12}$$

Therefore, in the computation only the spectral coefficients with even indices are involved. The number of required operations of multiplication and addition becomes one and *N1*+1, respectively.

IDCT-II:

$$\mathbf{x}\_{k} = \frac{\mathbf{1}}{N} \Big( 2 \sum\_{s=1}^{N-1} X\_{s}^{k} \cos \left( \pi \frac{\left( N\_{1} + 1/2 \right) \mathbf{s}}{N} \right) + X\_{0}^{k} \Big), \tag{13}$$

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse transform is given by

$$\mathbf{x}\_k = \frac{1}{N} \left( 2 \sum\_{s=1}^{N\_1} (-1)^s \, X\_{2s}^k + X\_0^k \right). \tag{14}$$

We see that in the computation only the spectral coefficients with even indices are involved. The computation requires one multiplication operation and *N1*+1 addition operations.

IDCT-III:

$$\alpha\_k = \frac{2}{N} \sum\_{s=0}^{N-1} X\_s^k \cos \left( \pi \frac{N\_1 \left(s + 1/2\right)}{N} \right), \tag{15}$$

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse transform is

$$\mathbf{x}\_{k} = \frac{2}{N} \left( \sum\_{s=0}^{N\_1 - 1} \left( \mathbf{X}\_s^k + \left( -1 \right)^{N\_1} \mathbf{X}\_{N - s - 1}^k \right) \cos \left( \pi \frac{N\_1 \left( s + 1/2 \right)}{N} \right) + \left( -1 \right)^{\left[ N\_1/2 \right]} \mathbf{X}\_{N\_1}^k \frac{\left( 1 + \left( -1 \right)^{N\_1} \right) \left( 1 + \left( -1 \right)^{N - 1} \right)}{4} \right) . \tag{16}$$

If *N1* is even, then the computation requires *N1*+1 multiplication operations and 2*N1* addition operations. Otherwise, the complexity is reduced to *N1* multiplication operations and 2*N1* - 1 addition operations.

IDCT-IV:

$$\alpha\_k = \frac{2}{N} \sum\_{s=0}^{N-1} X\_s^k \cos \left( \pi \frac{(N\_1 + 1/2)(s + 1/2)}{N} \right),\tag{17}$$

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse transform is given by

$$\chi\_k = \frac{\sqrt{2}}{N} \left( \sum\_{s=1}^{N\_1} (-1)^s \left( X\_{2s}^k + X\_{2s-1}^k \right) + X\_0^k \right). \tag{18}$$

One multiplication operation and *N*-1 addition operations are required to perform this computation.

The inverse algorithms for the running DSTs are given as follows.

IDST-I:

8 Image Restoration – Recent Advances and Applications

1

<sup>1</sup> 2 cos <sup>1</sup>

*N s x X XX*

*k s N*

where *N=N1+N2*. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse

<sup>1</sup>

<sup>1</sup> 21 1

*x XX X*

*k sN*

Therefore, in the computation only the spectral coefficients with even indices are involved. The number of required operations of multiplication and addition becomes one and *N1*+1,

> <sup>1</sup> 1

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse

*k k*

*N s*

2 0

*<sup>N</sup> <sup>s</sup> k k*

 <sup>1</sup> 1

<sup>1</sup> <sup>1</sup>

*N N <sup>N</sup>*

If *N1* is even, then the computation requires *N1*+1 multiplication operations and 2*N1* addition operations. Otherwise, the complexity is reduced to *N1* multiplication operations

<sup>1</sup> <sup>1</sup> <sup>1</sup> [ /2]

. (16)

*N s*

2 1/2 cos

where *N=N1+N2*+1. If *xk* is the central pixel of the window, that is, *N1=N2* then the inverse

11 11 <sup>2</sup> 1/2 1 cos <sup>1</sup>

*N N* 

 

<sup>1</sup> 1/2 2 cos

1

*k s s x X X*

*N*

0

*k k N N k s N s N*

*k*

1 1

*N*

*k s s*

*x X*

1

*N N*

*N s <sup>k</sup> x XX <sup>X</sup>*

1 <sup>1</sup> 2 1

We see that in the computation only the spectral coefficients with even indices are involved. The computation requires one multiplication operation and *N1*+1 addition operations.

*x X X N N* 

2 0

 

<sup>1</sup>

1

. (12)

0

. (14)

, (15)

1

4

, (13)

, (11)

0

*s N kk k*

*k kk N*

The inverse algorithms for the running DCTs can be written as follows.

1

*N*

*s*

1

*N N*

1

*N*

*s*

1

*N*

*k s s*

*N*

1

IDCT-I:

respectively.

transform is given by

IDCT-II:

IDCT-III:

transform is

IDCT-IV:

0

and 2*N1* - 1 addition operations.

*s*

transform is simplified to

$$\alpha\_k = \frac{2}{N} \sum\_{s=1}^{N-1} X\_s^k \sin \left( \pi \frac{(N\_1 + 1)s}{N} \right),\tag{19}$$

where *N=N1+N2*+2. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse transform is simplified to

$$\mathbf{x}\_k = \frac{2}{N} \sum\_{s=0}^{N\_1} \begin{pmatrix} -1 \end{pmatrix}^s X\_{2s+1}^k \,. \tag{20}$$

Therefore, in the computation only the spectral coefficients with odd indices are involved. The complexity is one multiplication operation and *N1* addition operations.

IDST-II:

$$\mathbf{x}\_{k} = \frac{1}{N} \Big( 2 \sum\_{s=1}^{N-1} X\_{s}^{k} \sin \left( \pi \frac{\left( N\_{1} + 1 \,/\, \text{2} \right) \mathbf{s}}{N} \right) + \left( -1 \right)^{N\_{1}} X\_{N}^{k} \Big) \,\tag{21}$$

where *N=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2* then the inverse transform is given by

$$\mathbf{x}\_{k} = \frac{\mathbf{1}}{N} \Big( 2 \sum\_{s=0}^{N\_1 - 1} \left( -\mathbf{1} \right)^{s} X\_{2s+1}^{k} + \left( -\mathbf{1} \right)^{N\_1} X\_{N}^{k} \Big). \tag{22}$$

In the computation only the spectral coefficients with odd indices are involved. The computational complexity is one multiplication operation and *N1*+1 addition operations.

IDST-III:

$$\mathbf{x}\_k = \frac{2}{N} \sum\_{s=1}^{N} X\_s^k \sin \left( \pi \frac{(N\_1 + 1)(s - 1/2)}{N} \right),\tag{23}$$

Space-Variant Image Restoration with Running Sinusoidal Transforms 11

*k kn n k* , *n*

where *W*=[w*k,n*] is a distortion matrix, *ν*=[v*k*] is additive noise with zero mean, *k,n=1,…N, N*

x=Wa+v , (30)

H K W WK W K *T T aa aa*

The obtained optimal filter is based on an assumption that an input signal within the window is stationary. The result of filtering is the restored window signal. This corresponds to signal processing in nonoverlapping fragments. The computational complexity of the processing is O(*N2*). However, if the matrix of the optimal filter is diagonal, the complexity is reduced to O(*N*). Such filter is referred as a scalar filter. Actually, any linear filtering can be performed with a scalar filter using corresponding unitary transforms. Now suppose that the signal is processed in a moving window in the domain of a running discrete sinusoidal transform. For each position of the window an estimate of the central pixel should be computed. Using the equations for inverse sinusoidal transforms presented in the previous section, the point-wise MSE (PMSE) for reconstruction of the central element of the window

where A *Al HlXl* is a vector of signal estimate in the domain of a sinusoidal

matrix of the size x *N N* of the coefficients of an inverse sinusoidal transform (see Eqs. (12),

<sup>1</sup> H P PI *U x* x ax

*x x* 1 if 0, else 0 , Pax *EAlXk EXlXk* , P*<sup>x</sup>* <sup>x</sup> . Note that matrix

 *l* **α** is sparse; the number of its non-zero entries is approximately twice less than the size of the window signal. Therefore, the computational complexity of the scalar filters in Eq. (33) and signal processing can be significantly reduced comparing to the complexity for the filter in Eq. (31). For the model of signal distortion in Eq. (30) the filter matrix is given as

1

1 , 2 ,..., *N* is a diagonal matrix of the size x *N N* ,

*l*

 

*N*

2

*PMSE k E a k a k E l A l A l*

transform, H*<sup>U</sup> H l* is a diagonal matrix of the scalar filter,

 

(14), (16), (18), (20), (22), (24), and (26)). Minimizing Eq. (32), we obtain

is the size of fragment. The equation can be rewritten as

where K aa , K , a 0 *T TT aa E EE* 

that the input signal and noise are uncorrelated.

and the optimal filter is given by

can be written as follows:

where I

*diag*

 

  *x wa v* , (29)

1

2

, (32)

, (33)

*l* **α** is a diagonal

 , (31)

**νν ν** are the covariance matrices. It is assumed

where *N=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2* then we can rewrite

$$\mathbf{x}\_{k} = \frac{2}{N} \left| \sum\_{s=1}^{N\_{1}} \left( X\_{s}^{k} + \left( -1 \right)^{N\_{1}} X\_{N-s+1}^{k} \right) \sin \left( \pi \frac{\left( N\_{1} + 1 \right) \left( s - 1/2 \right)}{N} \right) + \left( -1 \right)^{\left[ N\_{1}/2 \right]} X\_{N\_{1} + 1}^{k} \frac{\left( 1 + \left( -1 \right)^{N\_{1}} \right) \left( 1 + \left( -1 \right)^{N-1} \right)}{4} \right|. \tag{24}$$

If *N1* is even, then the computation requires *N1*+1 multiplication operations and 2*N1* addition operations. Otherwise, the complexity is reduced to *N1* multiplication operations and 2*N1* - 1 addition operations.

IDST-IV:

$$\alpha\_k = \frac{2}{N} \sum\_{s=1}^{N} X\_s^k \sin \left( \pi \frac{(N\_1 + 1/2)(s - 1/2)}{N} \right), \tag{25}$$

where *=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2*, then the inverse transform is given by

$$\mathbf{x}\_{k} = \frac{\sqrt{2}}{N} \Big| \sum\_{s=1}^{N\_1} (-1)^{s-1} \left( X\_{2s}^k + X\_{2s-1}^k \right) + \left( -1 \right)^{N\_1} X\_N^k \Big| \,. \tag{26}$$

The complexity is one multiplication operation and *N*-1 addition operations.

## **3. Local image restoration with running transforms**

First we define a local criterion of the performance of filters for image processing and then derive optimal local adaptive filters with respect to the criterion. One the most used criterion in signal processing is the minimum mean-square error (MSE). Since the processing is carried out in a moving window, then for each position of a moving window an estimate of the central element of the window is computed. Suppose that the signal to be processed is approximately stationary within the window. The signal may be distorted by sensor's noise.

Let us consider a generalized linear filtering of a fragment of the input one-dimensional signal (for instance for a fixed position of the moving window). Let *a*=[*ak*] be undistorted real signal, *x*=[*xk*] be observed signal, *k=1,…, N, N* be the size of the fragment, U be the matrix of the discrete sinusoidal transform, *E*{.} be the expected value, superscript T denotes the transpose. Let a Hx be a linear estimate of the undistorted signal, which minimizes the MSE averaged over the window

$$\text{MMSE} = E\left\{ \left( \mathbf{a} - \overline{\mathbf{a}} \right)^{T} \left( \mathbf{a} - \overline{\mathbf{a}} \right) \right\} \Big/ \text{N} \,. \tag{27}$$

The optimal filter for this problem is the Wiener filter (Jain, 1989):

$$\mathbf{H} = E\left\{\mathbf{a}\mathbf{x}^T\right\} \left[E\left\{\mathbf{x}\mathbf{x}^T\right\}\right]^{-1}.\tag{28}$$

Let us consider the known model of a linear degradation:

$$\mathbf{x}\_{k} = \sum\_{n} w\_{k,n} \mathbf{a}\_{n} + \mathbf{v}\_{k} \,\prime \,\prime \,\tag{29}$$

where *W*=[w*k,n*] is a distortion matrix, *ν*=[v*k*] is additive noise with zero mean, *k,n=1,…N, N* is the size of fragment. The equation can be rewritten as

$$\mathbf{x} = \mathbf{W}\mathbf{a} + \mathbf{v} \tag{30}$$

and the optimal filter is given by

10 Image Restoration – Recent Advances and Applications

where *N=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2* then we can

11 11 2 1 12

If *N1* is even, then the computation requires *N1*+1 multiplication operations and 2*N1* addition operations. Otherwise, the complexity is reduced to *N1* multiplication operations

<sup>2</sup> 1/2 1/2 sin

where *=N1+N2*+1. If *xk* is the central pixel of the window; that is, *N1=N2*, then the inverse

 <sup>1</sup> <sup>1</sup> 1 2 21

 

<sup>2</sup> 1 1

*x XX X*

The complexity is one multiplication operation and *N*-1 addition operations.

*k s s N*

First we define a local criterion of the performance of filters for image processing and then derive optimal local adaptive filters with respect to the criterion. One the most used criterion in signal processing is the minimum mean-square error (MSE). Since the processing is carried out in a moving window, then for each position of a moving window an estimate of the central element of the window is computed. Suppose that the signal to be processed is approximately stationary within the window. The signal may be distorted by sensor's noise. Let us consider a generalized linear filtering of a fragment of the input one-dimensional signal (for instance for a fixed position of the moving window). Let *a*=[*ak*] be undistorted real signal, *x*=[*xk*] be observed signal, *k=1,…, N, N* be the size of the fragment, U be the matrix of the discrete sinusoidal transform, *E*{.} be the expected value, superscript T denotes the transpose. Let a Hx be a linear estimate of the undistorted signal, which minimizes

> <sup>1</sup> H ax xx *T T E E*

*N N* 

1 1

1 sin 1

*k k N N k s N s N*

*N s <sup>k</sup> x XX <sup>X</sup>*

1

1

*N*

*s*

*N*

**3. Local image restoration with running transforms** 

The optimal filter for this problem is the Wiener filter (Jain, 1989):

Let us consider the known model of a linear degradation:

*N k*

*k s s*

*x X*

*N N*

<sup>1</sup> <sup>1</sup>

. (24)

<sup>1</sup>

*N s*

*s N k k k*

1 2 1 1

*N N <sup>N</sup>*

1

, (25)

. (26)

aa aa *<sup>T</sup> MMSE E <sup>N</sup>* . (27)

. (28)

1

4

rewrite

IDST-IV:

1

and 2*N1* - 1 addition operations.

the MSE averaged over the window

*s*

transform is given by

$$\mathbf{H} = \mathbf{K}\_{aa}\mathbf{W}^T \left[ \mathbf{W}\mathbf{K}\_{aa}\mathbf{W}^T + \mathbf{K}\_{\nu\nu} \right]^{-1},\tag{31}$$

where K aa , K , a 0 *T TT aa E EE* **νν ν** are the covariance matrices. It is assumed that the input signal and noise are uncorrelated.

The obtained optimal filter is based on an assumption that an input signal within the window is stationary. The result of filtering is the restored window signal. This corresponds to signal processing in nonoverlapping fragments. The computational complexity of the processing is O(*N2*). However, if the matrix of the optimal filter is diagonal, the complexity is reduced to O(*N*). Such filter is referred as a scalar filter. Actually, any linear filtering can be performed with a scalar filter using corresponding unitary transforms. Now suppose that the signal is processed in a moving window in the domain of a running discrete sinusoidal transform. For each position of the window an estimate of the central pixel should be computed. Using the equations for inverse sinusoidal transforms presented in the previous section, the point-wise MSE (PMSE) for reconstruction of the central element of the window can be written as follows:

$$PMSE\left(k\right) = E\left\{ \left[ a\left(k\right) - \overline{a}\left(k\right) \right]^2 \right\} = E\left\{ \left[ \sum\_{l=1}^N a\left(l\right) \left( A\left(l\right) - \overline{A}\left(l\right) \right) \right]^2 \right\},\tag{32}$$

where A *Al HlXl* is a vector of signal estimate in the domain of a sinusoidal transform, H*<sup>U</sup> H l* is a diagonal matrix of the scalar filter, *l* **α** is a diagonal matrix of the size x *N N* of the coefficients of an inverse sinusoidal transform (see Eqs. (12), (14), (16), (18), (20), (22), (24), and (26)). Minimizing Eq. (32), we obtain

$$\mathbf{H}\_{\mathrm{II}} = \left[\mathbf{P}\_{\mathrm{xx}}\right]^{-1} \mathbf{P}\_{\mathrm{ax}} \mathbf{I}\_{\alpha \prime} \tag{33}$$

where I *diag* 1 , 2 ,..., *N* is a diagonal matrix of the size x *N N* , *x x* 1 if 0, else 0 , Pax *EAlXk EXlXk* , P*<sup>x</sup>* <sup>x</sup> . Note that matrix *l* **α** is sparse; the number of its non-zero entries is approximately twice less than the size of the window signal. Therefore, the computational complexity of the scalar filters in Eq. (33) and signal processing can be significantly reduced comparing to the complexity for the filter in Eq. (31). For the model of signal distortion in Eq. (30) the filter matrix is given as

Space-Variant Image Restoration with Running Sinusoidal Transforms 13

For the design of local adaptive filters in the domain of running sinusoidal transforms the covariance matrices and power spectra of fragments of a signal are required. Since they are often unknown, in practice, these matrices can be estimated from observed signals

The objective of this section is to develop a technique for local adaptive restoration of images degraded by nonuniform motion blur. Assume that the blur is owing to horizontal relative motion between the camera and the image, and it is approximately space-invariant within local regions of the image. It is known that point spread functions for motion and focus blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings (Biemond et al., 1990). We assume also that the observation noise is a zero-mean, white Gaussian process that is uncorrelated to the image signal. In this case, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local region of the observed image. To guarantee statistically correct results, 30 statistical trials of each experiment for different realizations of the random noise process were performed. The MSE criterion is used for comparing the quality of restoration. Additionally, a subjective visual

criterion is used. In our computer simulation, the MSE is given by

*N*

*i*

where *ai i N* ( ), 1,.. is the original image, and *ai i N* ( ), 1,.. is the restored image. The subjective visual criterion is defined as an enhanced difference between original and restored images. A pixel is displayed as gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white (with intensity values of 0 and 1, respectively). First, with the help of computer simulation we answer to the question: how to choose the best running discrete sinusoidal transform for

The objective of this section is to test the performance of the scalar filter (see Eq. (34)) designed with different running sinusoidal transforms for local image restoration while the statistics of the degraded image are varied. In our experiments we used realizations of a wide-sense colored stationary process, which is completely defined by the second-order statistics. The zero-mean process has the bi-exponential covariance function with varying

The generated synthetic image is degraded by running 1D horizontal averaging of 5 pixels,

images is 1024x1024. The quality of restoration is measured in terms of the MSE. The size of moving window for local image restoration is 15x15. The best running discrete sinusoidal

 

,

*MSE a a*

**4.1 Choice of discrete sinusoidal transform for local image restoration** 

and then a white Gaussian noise with a given standard deviation

<sup>2</sup> () ()

, (36)

*<sup>n</sup>* is added. The size of

*ai ai*

*N*

(Yaroslavsky & Eden, 1996).

local image restoration?

correlation coefficient

.

**4. Computer simulation results** 

$$\mathbf{H}\_{\mathrm{II}} = \left[\mathbf{U}\left(\mathbf{W}\mathbf{K}\_{\mathrm{aa}}\,\mathbf{W}^{T} + \mathbf{K}\_{\nu\nu}\right)\mathbf{U}^{T}\right]^{-1}\mathbf{U}\mathbf{K}\_{\mathrm{aa}}\,\mathbf{W}^{T}\,\mathbf{U}^{T}\,\mathbf{I}\_{\mathrm{a}}\,\mathrm{.}\tag{34}$$

If the matrices U WK W K U *T T aa* and UK W U*T T aa* in Eq. (34) are close to diagonal, the matrix of the scalar filter in (34) is close to diagonal, and the filter can be written as

$$H(l) \approx \frac{P\_1(l)}{P\_2(l) + P\_{\nu\nu}(l)} \, ^\prime \tag{35}$$

where *Pl Pl P l* 1 2 , , *nn* are diagonal elements of the following matrices UK W U I , *T T aa* U WK W U*T T aa* , UK U*<sup>T</sup>* , *l=1,… N* .

For a real symmetric matrix of the covariance function, say K , there exists a unitary matrix U such that UKU*<sup>T</sup>* is a diagonal matrix. Actually, it is the KLT. It was shown (Jain, 1989) that some discrete sinusoidal transforms perform close to the optimum KLT for the firstorder Markov stationary data under specific conditions. In our case, the covariance matrices K WU*T T aa* and WK W*<sup>T</sup> aa* are not symmetric. Therefore, under different conditions of degradation different discrete sinusoidal transforms can better diagonalize these matrices. For instance, if a signal has a high correlation coefficient and a smoothed version of the signal is corrupted by additive, weakly-correlated noise, then the matrix U WK W K U *T T aa* is close to diagonal. Figure 1 shows the covariance matrix of the smoothed and noisy signal having the correlation coefficient of 0.95 as well as the discrete cosine transform of the covariance matrix. The linear convolution between a signal *x* and the matrix K W*<sup>T</sup> aa* in the domain of the running DCT-II can be well approximated by a diagonal matrix UK W U I X *T T aa diag* . Therefore, the matrix of the scalar filter in Eq. (34) will be close to diagonal.

Fig. 1. (a) Covariance matrix of a noisy signal, (b) DCT-II of the covariance matrix.

For the design of local adaptive filters in the domain of running sinusoidal transforms the covariance matrices and power spectra of fragments of a signal are required. Since they are often unknown, in practice, these matrices can be estimated from observed signals (Yaroslavsky & Eden, 1996).

## **4. Computer simulation results**

12 Image Restoration – Recent Advances and Applications

and UK W U*T T*

where *Pl Pl P l* 1 2 , , *nn* are diagonal elements of the following matrices

, *l=1,… N* .

For a real symmetric matrix of the covariance function, say K , there exists a unitary matrix U such that UKU*<sup>T</sup>* is a diagonal matrix. Actually, it is the KLT. It was shown (Jain, 1989) that some discrete sinusoidal transforms perform close to the optimum KLT for the firstorder Markov stationary data under specific conditions. In our case, the covariance matrices

degradation different discrete sinusoidal transforms can better diagonalize these matrices. For instance, if a signal has a high correlation coefficient and a smoothed version of the signal is corrupted by additive, weakly-correlated noise, then the matrix

smoothed and noisy signal having the correlation coefficient of 0.95 as well as the discrete cosine transform of the covariance matrix. The linear convolution between a signal *x* and the

(a) (b)

Fig. 1. (a) Covariance matrix of a noisy signal, (b) DCT-II of the covariance matrix.

*aa* in the domain of the running DCT-II can be well approximated by a

2 *P l H l*

 1

*Pl P l* 

 

*aa* in Eq. (34) are close to diagonal, the

, (35)

*aa*

. (34)

*aa* are not symmetric. Therefore, under different conditions of

is close to diagonal. Figure 1 shows the covariance matrix of the

. Therefore, the matrix of the scalar filter in Eq. (34)

 <sup>1</sup> H U WK W K U UK W U I *T T T T*

matrix of the scalar filter in (34) is close to diagonal, and the filter can be written as

*U aa*

*aa* , UK U*<sup>T</sup>*

*aa*

U WK W U*T T*

If the matrices U WK W K U *T T*

UK W U I , *T T aa*

K WU*T T*

*aa* and WK W*<sup>T</sup>*

diagonal matrix UK W U I X *T T aa diag*

U WK W K U *T T aa*

will be close to diagonal.

matrix K W*<sup>T</sup>*

The objective of this section is to develop a technique for local adaptive restoration of images degraded by nonuniform motion blur. Assume that the blur is owing to horizontal relative motion between the camera and the image, and it is approximately space-invariant within local regions of the image. It is known that point spread functions for motion and focus blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings (Biemond et al., 1990). We assume also that the observation noise is a zero-mean, white Gaussian process that is uncorrelated to the image signal. In this case, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local region of the observed image. To guarantee statistically correct results, 30 statistical trials of each experiment for different realizations of the random noise process were performed. The MSE criterion is used for comparing the quality of restoration. Additionally, a subjective visual criterion is used. In our computer simulation, the MSE is given by

$$MSE\left(a,\overline{a}\right) = \frac{\sum\_{i}^{N} \left[a(i) - \overline{a}(i)\right]^2}{N},\tag{36}$$

where *ai i N* ( ), 1,.. is the original image, and *ai i N* ( ), 1,.. is the restored image. The subjective visual criterion is defined as an enhanced difference between original and restored images. A pixel is displayed as gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white (with intensity values of 0 and 1, respectively). First, with the help of computer simulation we answer to the question: how to choose the best running discrete sinusoidal transform for local image restoration?

### **4.1 Choice of discrete sinusoidal transform for local image restoration**

The objective of this section is to test the performance of the scalar filter (see Eq. (34)) designed with different running sinusoidal transforms for local image restoration while the statistics of the degraded image are varied. In our experiments we used realizations of a wide-sense colored stationary process, which is completely defined by the second-order statistics. The zero-mean process has the bi-exponential covariance function with varying correlation coefficient .

The generated synthetic image is degraded by running 1D horizontal averaging of 5 pixels, and then a white Gaussian noise with a given standard deviation *<sup>n</sup>* is added. The size of images is 1024x1024. The quality of restoration is measured in terms of the MSE. The size of moving window for local image restoration is 15x15. The best running discrete sinusoidal

Space-Variant Image Restoration with Running Sinusoidal Transforms 15

A real test aerial image is shown in Fig. 2(a). The size of image is 512x512, each pixel has 256 levels of quantization. The signal range is [0, 1]. The image quadrants are degraded by running 1D horizontal averaging with the following sizes of the moving window: 5, 6, 4, and 3 pixels (for quadrants from left to right, from top to bottom). The image is also corrupted by a zero-mean additive white Gaussian noise. The degraded image with the

In our tests the window length of 15x15 pixels is used, it is determined by the minimal size of details to be preserved after filtering. Since there exists difference in spectral distributions of the image signal and wide-band noise, the power spectrum of noise can be easily measured from the experimental covariance matrix. We carried out three parallel processing of the degraded image with the use of SDCT-II, SDST-I, and SDST-II transforms. At each position of the moving window the local correlation coefficient is estimated from the restored images. On the base of the correlation value and the standard deviation of noise, the resultant image is formed from the outputs obtained with either SDCT-II or SDST-I, or

(a) (b)

The results of image restoration by the global parametric Wiener filtering (Jain, 1989) and the proposed method are shown in Figs. 3(a) and 3(b), respectively. Figs. 3(c) and 3(d) show differences between the original image and images restored by global Wiener algorithm and

We also performed local image restoration using only the SDCT. As expected, the result of restoration is slightly worse than that of adaptive local restoration. We see that the proposed

Fig. 2. (a) Test image, (b) space-variant degraded test image.

by the proposed algorithm, respectively.

**4.2 Local adaptive restoration of real degraded image** 

noise standard deviation of 0.05 is shown in Fig. 2(b).

SDST-II according to Table 2.

transform a function of the model parameters ( *<sup>n</sup>* and ) is presented in Table 2. So, in similar degradation circumstances and image model, local adaptive restoration yields the best results if the three sinusoidal transforms depending on local statistics of the processed image are used. The decision rule for choosing the best sinusoidal transform at each position of the moving window is given in Table 2. Next, we carry out adaptive local restoration with real images.


Table 2. Best local restoration with running discrete sinusoidal transforms versus the model parameters.

## **4.2 Local adaptive restoration of real degraded image**

14 Image Restoration – Recent Advances and Applications

similar degradation circumstances and image model, local adaptive restoration yields the best results if the three sinusoidal transforms depending on local statistics of the processed image are used. The decision rule for choosing the best sinusoidal transform at each position of the moving window is given in Table 2. Next, we carry out adaptive local restoration with

*<sup>n</sup>* 0.05 0.1 0.2 0.25

0.95 SDCT-II SDCT-II SDCT-II SDCT-II

0.9 SDCT-II SDCT-II SDCT-II SDCT-II

0.8 SDCT-II SDCT-II SDCT-II SDCT-II

0.7 SDCT-II SDCT-II SDCT-II SDCT-II

0.6 SDCT-II SDCT-II SDCT-II SDCT-II

0.5 SDST-I SDCT-II SDCT-II SDCT-II

0.3 SDST-I SDST-I SDST-I SDST-I

0 SDST-I SDST-I SDST-I SDST-I

0.3 SDST-I SDST-I SDST-I SDST-I

0.5 SDST-II SDST-II SDST-II SDST-II

0.7 SDST-II SDST-II SDST-II SDST-II

0.8 SDST-II SDST-II SDST-II SDST-II

0.9 SDST-II SDST-II SDST-II SDST-II

Table 2. Best local restoration with running discrete sinusoidal transforms versus the model

*<sup>n</sup>* and

) is presented in Table 2. So, in

transform a function of the model parameters (

real images.

parameters.

A real test aerial image is shown in Fig. 2(a). The size of image is 512x512, each pixel has 256 levels of quantization. The signal range is [0, 1]. The image quadrants are degraded by running 1D horizontal averaging with the following sizes of the moving window: 5, 6, 4, and 3 pixels (for quadrants from left to right, from top to bottom). The image is also corrupted by a zero-mean additive white Gaussian noise. The degraded image with the noise standard deviation of 0.05 is shown in Fig. 2(b).

In our tests the window length of 15x15 pixels is used, it is determined by the minimal size of details to be preserved after filtering. Since there exists difference in spectral distributions of the image signal and wide-band noise, the power spectrum of noise can be easily measured from the experimental covariance matrix. We carried out three parallel processing of the degraded image with the use of SDCT-II, SDST-I, and SDST-II transforms. At each position of the moving window the local correlation coefficient is estimated from the restored images. On the base of the correlation value and the standard deviation of noise, the resultant image is formed from the outputs obtained with either SDCT-II or SDST-I, or SDST-II according to Table 2.

Fig. 2. (a) Test image, (b) space-variant degraded test image.

The results of image restoration by the global parametric Wiener filtering (Jain, 1989) and the proposed method are shown in Figs. 3(a) and 3(b), respectively. Figs. 3(c) and 3(d) show differences between the original image and images restored by global Wiener algorithm and by the proposed algorithm, respectively.

We also performed local image restoration using only the SDCT. As expected, the result of restoration is slightly worse than that of adaptive local restoration. We see that the proposed

Space-Variant Image Restoration with Running Sinusoidal Transforms 17

Fig. 4. Performance of restoration algorithms in terms of MSE versus standard deviation of

In this chapter we treated the problem of local adaptive technique for space-variant restoring linearly degraded and noisy images. The minimum MSE estimator in the domain of running discrete sinusoidal transforms was derived. To provide image processing at high rate, fast recursive algorithms for computing the running sinusoidal transforms were utilized. Extensive testing using various parameters of degradations (nonuniform motion blurring and corruption by noise) has shown that the original image can be well restored by proper choice of the parameters of the proposed adaptive local

Banham, M. & Katsaggelos, A. (1997). Digital image restoration. *IEEE Signal Processing* 

Bertero, M. & Boccacci, P. (1998). *Introduction to inverse problems in imaging,* Institute of

Biemond, J., Lagendijk, R.L. & Mersereau, R.M. (1990). Iterative methods for image deblurring. *Proc. IEEE. Vol. 78*, No. 5, (May 1990) (856-883), ISSN 0018-9219 Bovik, A. (2005). *Handbook of image and video processing* (2nd ed.), Academic Press, ISBN 0-12-

González, R. & Woods, R. (2008). *Digital image processing* (3rd ed.), Prentice Hall, ISBN 0-13-

*Magazine,* Vol. 14, No.2, (March 1997), pp. 24-41, ISSN 1053-5888

Physics Publishing, ISBN 0-7503-0435-9, Bristol, UK

additive noise.

**4. Conclusion** 

restoration algorithm.

119792-1, NJ, USA

1687288, NJ, USA

**5. References** 

algorithm is capable to perform a good space-variant image restoration and noise suppression. Finally, we investigate the robustness of the tested restoration techniques to additive noise. The performance of the global parametric Wiener filtering and the local adaptive filtering is shown in Fig. 4.

Fig. 3. (a) Global Wiener restoration, (b) local adaptive restoration in domain of running transforms, (c) difference between the original image and restored image by Wiener filtering, (d) difference between the original image and restored image by proposed algorithm.

Fig. 4. Performance of restoration algorithms in terms of MSE versus standard deviation of additive noise.

## **4. Conclusion**

16 Image Restoration – Recent Advances and Applications

algorithm is capable to perform a good space-variant image restoration and noise suppression. Finally, we investigate the robustness of the tested restoration techniques to additive noise. The performance of the global parametric Wiener filtering and the local

(a) (b)

(c) (d)

Fig. 3. (a) Global Wiener restoration, (b) local adaptive restoration in domain of running transforms, (c) difference between the original image and restored image by Wiener filtering, (d) difference between the original image and restored image by proposed algorithm.

adaptive filtering is shown in Fig. 4.

In this chapter we treated the problem of local adaptive technique for space-variant restoring linearly degraded and noisy images. The minimum MSE estimator in the domain of running discrete sinusoidal transforms was derived. To provide image processing at high rate, fast recursive algorithms for computing the running sinusoidal transforms were utilized. Extensive testing using various parameters of degradations (nonuniform motion blurring and corruption by noise) has shown that the original image can be well restored by proper choice of the parameters of the proposed adaptive local restoration algorithm.

## **5. References**


**2** 

**Statistical-Based Approaches** 

State Luminiţa1, Cătălina-Lucia Cocianu1 and Vlamos Panayiotis2

Image restoration methods are used to improve the appearance of an image by the application of a restoration process based on a mathematical model to explain the way the image was distorted by noise. Examples of types of degradation include blurring caused by motion or atmospheric disturbance, geometric distortion caused by imperfect lenses, superimposed interference patterns caused by mechanical systems, and noise induced by

Usually, it is assumed that the degradation model is either known or can be estimated from data. The general idea is to model the degradation process and then apply the inverse process to restore the original image. In cases when the available knowledge does not allow to adopt a reasonable model for the degradation mechanism it becomes necessary to extract information about the noise directed by data and then to use this information for restoration purposes. The knowledge about the particular generation process of the image is application specific. For example, it proves helpful to know how a specific lens distorts an image or how mechanical vibration from a satellite affects an image. This information can be gathered from the analysis of the image acquisition process and by applying image analysis

The restoration can be viewed as a process that attempts to reconstruct or recover a degraded image using some available knowledge about the degradation mechanism. Typically, the noise can be modeled with either a Gaussian, uniform or salt and pepper distribution. The restoration techniques are usually oriented toward modeling the type of degradation in order to infer the inverse process for recovering the given image. This approach usually involves the option for a criterion to numerically evaluate the quality of the resulted image and consequently the restoration process can be expressed in terms of an

The special filtering techniques of mean type prove particularly useful in reducing the normal/uniform noise component when the mean parameter is close to 0. In other words, the effects determined by the application of mean filters are merely the decrease of the local

**1. Introduction** 

electronic sources.

optimization problem.

techniques to samples of degraded images.

 **for Noise Removal** 

*2Academy of Economic Studies* 

*1University of Piteşti* 

*3Ionian University* 

*1,2Romania 3Greece* 


## **Statistical-Based Approaches for Noise Removal**

State Luminiţa1, Cătălina-Lucia Cocianu1 and Vlamos Panayiotis2

*1University of Piteşti 2Academy of Economic Studies 3Ionian University 1,2Romania 3Greece* 

## **1. Introduction**

18 Image Restoration – Recent Advances and Applications

Jain, A.K. (1989). *Fundamentals of digital image processing*, Prentice Hall, ISBN 0-13-332578-4,

Kober, V. (2004). Fast algorithms for the computation of running discrete sinusoidal

Kober, V. (2007). Fast algorithms for the computation of running discrete Hartley

Kober, V. & Ovseevich, I.A. (2008). Image restoration with running sinusoidal transforms.

Kundur, D. & Hatzinakos, D. (1996). Blind image deconvolution. *IEEE Signal Processing* 

Oppenheim, A.V. & Shafer, R.W. (1989). *Discrete-time signal processing*, Prentice Hall, ISBN 0-

Sawchuk, A.A. (1974). Space-variant image restoration by coordinate transformations. *J. Opt. Soc. Am.* Vol. 64, No. 2, (February 1974), pp. 138—144, ISSN 1084-7529 Vitkus, R.Y. & Yaroslavsky, L.P. (1987). Recursive algorithms for local adaptive linear filtration. in: *Mathematical Research*., Academy Verlag, pp. 34-39, Berlin, Germany Yaroslavsky, L. P. & Eden, M. (1996). *Fundamentals of digital optics*, Birkhauser, ISBN 3-7643-

*Magazine,* Vol. 13, No. 3, (May 1996), pp. 73-76, ISSN 1053-5888

transforms. *IEEE Trans. on Signal Process*. Vol. 52, No 6, (June 2004), pp. 1704-1710,

transforms. *IEEE Trans. on Signal Process*. Vol. 55, No 6, (June 2007), pp. 2937-2944,

*Pattern Recognition and Image Analysis* Vol. 18, No. 4, (December 2008), pp. 650-654,

NY, USA

ISSN 1053-587X

ISSN 1053-587X

ISSN 1054-6618

13-216292-X, NJ, USA

3833-9, Boston, USA

Image restoration methods are used to improve the appearance of an image by the application of a restoration process based on a mathematical model to explain the way the image was distorted by noise. Examples of types of degradation include blurring caused by motion or atmospheric disturbance, geometric distortion caused by imperfect lenses, superimposed interference patterns caused by mechanical systems, and noise induced by electronic sources.

Usually, it is assumed that the degradation model is either known or can be estimated from data. The general idea is to model the degradation process and then apply the inverse process to restore the original image. In cases when the available knowledge does not allow to adopt a reasonable model for the degradation mechanism it becomes necessary to extract information about the noise directed by data and then to use this information for restoration purposes. The knowledge about the particular generation process of the image is application specific. For example, it proves helpful to know how a specific lens distorts an image or how mechanical vibration from a satellite affects an image. This information can be gathered from the analysis of the image acquisition process and by applying image analysis techniques to samples of degraded images.

The restoration can be viewed as a process that attempts to reconstruct or recover a degraded image using some available knowledge about the degradation mechanism. Typically, the noise can be modeled with either a Gaussian, uniform or salt and pepper distribution. The restoration techniques are usually oriented toward modeling the type of degradation in order to infer the inverse process for recovering the given image. This approach usually involves the option for a criterion to numerically evaluate the quality of the resulted image and consequently the restoration process can be expressed in terms of an optimization problem.

The special filtering techniques of mean type prove particularly useful in reducing the normal/uniform noise component when the mean parameter is close to 0. In other words, the effects determined by the application of mean filters are merely the decrease of the local

Statistical-Based Approaches for Noise Removal 21

Now, recursively, for any k, 2 *k n* , if we denote by *L* 1 1 ,..., *<sup>k</sup>* the linear subspace orthogonal on the linear subspace generated by the first (k-1) directions, *<sup>k</sup>* Rn is the k-th

Note that the principal directions 1 ,..., *n* of any signal are an orthogonal basis of Rn , and *<sup>T</sup> Y X* is the signal representation in terms of the principal directions, where

Consequently, if 1 ,..., *<sup>n</sup>* are unit eigen vectors of Σ, then , <sup>1</sup> ,..., *<sup>T</sup> Cov Y Y diag*

correlates the components of X. In the particular case of Gaussian signals, X~ *N*0, , the

Theorem. Let X be a n-dimensional real-valued random vector such that *E X* 0 and

A series of approaches are based on the assumption that the signal results as a mixture of a finite number of hidden independent sources and noise. This sort of attempts are usually referred as techniques of Independent Component Analysis type. The simplest model is the linear one, given by X=AS+ *η* , where A is an unknown matrix (mixing matrix), S is the ndimensional random vector whose components are independent and 1 2 , ,..., *<sup>T</sup>*

random vector representing the noise. The problem is to recover the hidden sources being

Gaussianity of *<sup>T</sup> w V* can be maximized on the basis of *<sup>T</sup> w X* if we use an expression that

The kurtosis (the fourth-order cumulant) corresponding to a real-valued random variable Y

The non-Gaussianity can be also measured using the Shannon neg-entropy (mutual information). Being given the computational difficulty of evaluating the exact expression of neg-entropy, usually an approximation of it is used instead, for instance the approximation

Usually the maximization of non-Gaussianity is performed on the pre-processed signal version *X* , applied in order to whiten the original clean signal. In case of the additive noise

, for any *w* Rn , *<sup>T</sup> w*

proposed in (Hyvarinen, Karhunen & Oja, 2001), <sup>2</sup>

*~ N*0,1 .

<sup>2</sup> 4 2 *kurt Y E Y E Y* <sup>3</sup> . In case Y is normally distributed, we get

*~* 0, *<sup>T</sup> Nww*

 

 

*k*

The value *<sup>T</sup> <sup>k</sup> X* is referred as the k-th principal component of the signal X.

The fundamental result is given by the celebrated Karhunen-Loeve theorem:

1 *k n* , the k-th principal direction is an eigen vector of Σ associated to

*<sup>n</sup>* are the eigen values of Σ, that is the linear transform of matrix *<sup>T</sup>* de-

1 var sup var

*L*

1 1 ,...,

*X X* 

.

*k T T*

*<sup>n</sup> I* , *E Y* 0 and , *T T Cov Y Y* .

*<sup>i</sup>* , 1 *i n* .

*<sup>n</sup>* the eigen values of Σ, then, for any k,

*k* .

*η*

*~ N* 0,

E E *T T*

, where *X*<sup>0</sup> is the original clean signal (unknown) and

*GJ wV G wV G <sup>ν</sup>* , where

. Consequently, the non-

, that is 0 *<sup>T</sup> kurt w*

 *<sup>n</sup>* is a

. Then, if we

.

 *<sup>n</sup>* ,

principal direction if 1 *<sup>k</sup>* and

<sup>1</sup> ,..., *<sup>n</sup>* . Obviously, *T T*

, *<sup>T</sup> Cov X X* . If we denote by 1 2 ...

components of Y are also normal distributed, *Yi* ~ *N*0,

given the signal X without knowing the mixing matrix A.

.

is defined as

*~ N* 0,

G is a non-polynomial function and

superposition model, *X X* <sup>0</sup>

For simplicity sake, the noise model is of Gaussian type, that is

denote *V AS* , then, for any vector *w* Rn , *T TT wX wV w*

where 1 ,..., 

vanishes the component *<sup>T</sup> w*

*kurt Y* 0 . Since

variance corresponding to each processed window, and consequently to inhibit the variance component of the noise. The AMVR algorithm (Adaptive Mean Variance Removal) allows the removal of the normal/uniform noise whatever the mean of the noise is (Cocianu, State, & Vlamos, 2002). Similar to MMSE (Minimum Mean Square Error) filtering technique (Umbaugh, 1998) the application of the AMVR algorithm requires that the noise parameters and some additional features are known.

The multiresolution support set is a data structure suitable for developing noise removal algorithms. (Bacchelli & Papi, 2006; Balster et al., 2003). The multiresolution algorithms perform the restoration tasks by combining, at each resolution level, according to a certain rule, the pixels of a binary support image. Some others use a selective wavelet shrinkage algorithm for digital image denoising aiming to improve the performance. For instance Balster (Balster, Zheng & Ewing, 2003) proposes an attempt of this sort together with a computation scheme, the denoising methodology incorporated in this algorithm involving a two-threshold validation process for real time selection of wavelet coefficients.

A new solution of the denoising problem based on the description length of the noiseless data in the subspace of the basis is proposed in (Beheshti & Dahleh, 2003), where the desired description length is estimated for each subspace and the selection of the subspace corresponding to the minimum length is suggested.

In (Bacchelli & Papi, 2006), a method for removing Gaussian noise from digital images based on the combination of the wavelet packet transform and the PCA is proposed. The method leads to tailored filters by applying the Karhunen-Loeve transform in the wavelet packet domain and acts with a suitable shrinkage function on these new coefficients, allowing the noise removal without blurring the edges and other important characteristics of the images.

Wavelet thresholding methods modifying the noisy coefficients were proposed by several authors (Buades, Coll & Morel, 2005; Stark, Murtagh & Bijaoui, 1995). The attempts are based on the idea that images are represented by large wavelet coefficients that have to be preserved whereas the noise is distributed across the set of small coefficients that have to be canceled. Since the edges lead to a considerable amount of wavelet coefficients of lower values than the threshold, the cancellation of these wavelet coefficients may cause small oscillations near the edges resulting spurious wavelets in the restored image.

## **2. Mathematics behind the noise removal and image restoration algorithms**

## **2.1 Principal Component Analysis (PCA) and Independent Component Analysis (ICA)**

We assume that the signal is represented by a n-dimensional real-valued random vector X of 0 mean and covariance matrix Σ. The principal directions of the repartition of X are the directions corresponding to the maximum variability, where the variability is expressed in terms of the variance.

Definition. The vector <sup>1</sup> Rn is the first principal direction if 1 1 and <sup>1</sup> 1 var sup var *n T T R X X* .

The value 1 *<sup>T</sup> X* is referred as the first principal component of X. 20 Image Restoration – Recent Advances and Applications

variance corresponding to each processed window, and consequently to inhibit the variance component of the noise. The AMVR algorithm (Adaptive Mean Variance Removal) allows the removal of the normal/uniform noise whatever the mean of the noise is (Cocianu, State, & Vlamos, 2002). Similar to MMSE (Minimum Mean Square Error) filtering technique (Umbaugh, 1998) the application of the AMVR algorithm requires that the noise parameters

The multiresolution support set is a data structure suitable for developing noise removal algorithms. (Bacchelli & Papi, 2006; Balster et al., 2003). The multiresolution algorithms perform the restoration tasks by combining, at each resolution level, according to a certain rule, the pixels of a binary support image. Some others use a selective wavelet shrinkage algorithm for digital image denoising aiming to improve the performance. For instance Balster (Balster, Zheng & Ewing, 2003) proposes an attempt of this sort together with a computation scheme, the denoising methodology incorporated in this algorithm involving a

A new solution of the denoising problem based on the description length of the noiseless data in the subspace of the basis is proposed in (Beheshti & Dahleh, 2003), where the desired description length is estimated for each subspace and the selection of the subspace

In (Bacchelli & Papi, 2006), a method for removing Gaussian noise from digital images based on the combination of the wavelet packet transform and the PCA is proposed. The method leads to tailored filters by applying the Karhunen-Loeve transform in the wavelet packet domain and acts with a suitable shrinkage function on these new coefficients, allowing the noise removal without blurring the edges and other important characteristics of the images. Wavelet thresholding methods modifying the noisy coefficients were proposed by several authors (Buades, Coll & Morel, 2005; Stark, Murtagh & Bijaoui, 1995). The attempts are based on the idea that images are represented by large wavelet coefficients that have to be preserved whereas the noise is distributed across the set of small coefficients that have to be canceled. Since the edges lead to a considerable amount of wavelet coefficients of lower values than the threshold, the cancellation of these wavelet coefficients may cause small

two-threshold validation process for real time selection of wavelet coefficients.

oscillations near the edges resulting spurious wavelets in the restored image.

*<sup>T</sup> X* is referred as the first principal component of X.

**2. Mathematics behind the noise removal and image restoration algorithms 2.1 Principal Component Analysis (PCA) and Independent Component Analysis (ICA)**  We assume that the signal is represented by a n-dimensional real-valued random vector X of 0 mean and covariance matrix Σ. The principal directions of the repartition of X are the directions corresponding to the maximum variability, where the variability is expressed in

Definition. The vector <sup>1</sup> Rn is the first principal direction if 1 1 and

and some additional features are known.

corresponding to the minimum length is suggested.

terms of the variance.

var sup var

The value 1

 <sup>1</sup> 1

*n T T R X X* . Now, recursively, for any k, 2 *k n* , if we denote by *L* 1 1 ,..., *<sup>k</sup>* the linear subspace orthogonal on the linear subspace generated by the first (k-1) directions, *<sup>k</sup>* Rn is the k-th principal direction if 1 *<sup>k</sup>* and 1 1 ,..., 1 var sup var *k T T k L X X* .

The value *<sup>T</sup> <sup>k</sup> X* is referred as the k-th principal component of the signal X.

Note that the principal directions 1 ,..., *n* of any signal are an orthogonal basis of Rn , and *<sup>T</sup> Y X* is the signal representation in terms of the principal directions, where <sup>1</sup> ,..., *<sup>n</sup>* . Obviously, *T T <sup>n</sup> I* , *E Y* 0 and , *T T Cov Y Y* . Consequently, if 1 ,..., *<sup>n</sup>* are unit eigen vectors of Σ, then , <sup>1</sup> ,..., *<sup>T</sup> Cov Y Y diag <sup>n</sup>* , where 1 ,..., *<sup>n</sup>* are the eigen values of Σ, that is the linear transform of matrix *<sup>T</sup>* decorrelates the components of X. In the particular case of Gaussian signals, X~ *N*0, , the components of Y are also normal distributed, *Yi* ~ *N*0,*<sup>i</sup>* , 1 *i n* .

The fundamental result is given by the celebrated Karhunen-Loeve theorem:

Theorem. Let X be a n-dimensional real-valued random vector such that *E X* 0 and , *<sup>T</sup> Cov X X* . If we denote by 1 2 ... *<sup>n</sup>* the eigen values of Σ, then, for any k, 1 *k n* , the k-th principal direction is an eigen vector of Σ associated to *k* .

A series of approaches are based on the assumption that the signal results as a mixture of a finite number of hidden independent sources and noise. This sort of attempts are usually referred as techniques of Independent Component Analysis type. The simplest model is the linear one, given by X=AS+ *η* , where A is an unknown matrix (mixing matrix), S is the ndimensional random vector whose components are independent and 1 2 , ,..., *<sup>T</sup> η <sup>n</sup>* is a random vector representing the noise. The problem is to recover the hidden sources being given the signal X without knowing the mixing matrix A.

For simplicity sake, the noise model is of Gaussian type, that is *~ N* 0, . Then, if we denote *V AS* , then, for any vector *w* Rn , *T TT wX wV w* . Consequently, the non-Gaussianity of *<sup>T</sup> w V* can be maximized on the basis of *<sup>T</sup> w X* if we use an expression that vanishes the component *<sup>T</sup> w* .

The kurtosis (the fourth-order cumulant) corresponding to a real-valued random variable Y is defined as <sup>2</sup> 4 2 *kurt Y E Y E Y* <sup>3</sup> . In case Y is normally distributed, we get *kurt Y* 0 . Since *~ N* 0, , for any *w* Rn , *<sup>T</sup> w ~* 0, *<sup>T</sup> Nww* , that is 0 *<sup>T</sup> kurt w* .

The non-Gaussianity can be also measured using the Shannon neg-entropy (mutual information). Being given the computational difficulty of evaluating the exact expression of neg-entropy, usually an approximation of it is used instead, for instance the approximation proposed in (Hyvarinen, Karhunen & Oja, 2001), <sup>2</sup> E E *T T GJ wV G wV G <sup>ν</sup>* , where G is a non-polynomial function and *~ N*0,1 .

Usually the maximization of non-Gaussianity is performed on the pre-processed signal version *X* , applied in order to whiten the original clean signal. In case of the additive noise superposition model, *X X* <sup>0</sup> , where *X*<sup>0</sup> is the original clean signal (unknown) and

Statistical-Based Approaches for Noise Removal 23

Note that the computation of *c k <sup>j</sup>* carried out in Step 1 imposes that either the periodicity

images, the values of 0 *c* are computed for each pixel (*x*,*y*) as 0

. A label procedure is applied to each

the noise component from the wavelet coefficients computed for *I*. In case for each pixel (*x*,*y*) of *I*, the distribution of the coefficients is available, the significance level corresponding to

constant at the resolution level *j* in case the amount of noise in *I* at this resolution level can be neglected. Let 0 be the hypothesis <sup>0</sup> : *I* is local constant at the resolution level *j*. In case there is significant amount of noise in *I* at the resolution level *j*, we get that the

<sup>2</sup>

 , 

, where *k* is a selected constant 3 *k* , because

*j jj jj j* , ,,

*jj jj*

 

, (2)

*<sup>j</sup>x y*, can be established using a statistical test. We say that *I* is local

*z*

(1)

*xy z dt* 

*<sup>j</sup> <sup>j</sup> <sup>z</sup>*

1

*c xy c xy xy*

,, ,

, in case of

1

 .

*<sup>j</sup>x y*, in order to remove

*p p j j*

*p p j j*

*ck ck k*

, then the wavelet coefficients also encode

*<sup>j</sup>* . In order to define the critical region *W* of

2

*t*

be the *a priori* selected significance

. Accordingly, the significance level of

*<sup>j</sup> x*,*y* is a significant coefficient if and only if

 

is,

condition *ckN ck j j* or the continuity property *ck N cN j j* holds.

Since the representation of the original sampled signal is 0

*<sup>j</sup> x y*, ~ <sup>2</sup> 0, *N*

<sup>1</sup> 1 Prob , exp <sup>2</sup> <sup>2</sup>

In other words, the probability of rejecting 0 (hence accept <sup>0</sup> ) when 0 is true is

*P xy k P xy k P xy k*

2 , 21 , *P xy k P xy k*

, 21 *<sup>j</sup> k jj z P xy k*

Using the significance level, we set to 1 the statistically significant coefficient and

 1 , , ,, , *p*

*j I xy c xy g xy xy* 

*p j j j*

.

End-for

Output: The set 1,..., , *j p j p k c*

some information about

each component

level and let *z*

*<sup>j</sup>x*,*yW* .

where *g* is defined by

Usually, *z*

If the input image *I* encodes a noise component

alternative hypothesis <sup>0</sup> :

be such that when 0 is true,

*j*

 

 

respectively we set to 0 the non-significant ones. The restored image *I*

the statistical test we proceed as follows. Let 0 1

the wavelet coefficients is given by the rule:

is taken as *<sup>j</sup> k*

and consequently, the critical region is *W zz*

 *~ N* 0, . In case *X*<sup>0</sup> and are independent and , *<sup>T</sup> Cov* is known, we get 0 0 , *<sup>T</sup> Cov X X* , where , *<sup>T</sup> Cov X X* a n d t h e c o v a r i a n c e m a t r i x corresponding to the observed signal X is assumed to be estimated from data. Then

 1 1 11 2 2 22 *XXX* 0 0 *X* , where <sup>1</sup> 2 *X*<sup>0</sup> and are independent and the covariance matrix of <sup>1</sup> 2 *X*<sup>0</sup> is the unit matrix *nI* .If *X*<sup>0</sup> results by the linear transform of matrix A applied to the sources S, *X AS* <sup>0</sup> , then *X BS* , where <sup>1</sup> *B A*<sup>2</sup> . Consequently, the sources S are determined by maximizing the non-Gaussianity of *X BS* . Usually, for simplicity sake, the matrix B is assumed to be orthogonal.

## **2.2 The use of concepts and tools of multiresolution analysis for noise removal and image restoration purposes**

The multiresolution based algorithms perform the restoration tasks by combining, at each resolution level, according to a certain rule, the pixels of a binary support image. The values of the support image pixels are either 1 or 0 depending on their significance degree. At each resolution level, the contiguous areas of the support image corresponding to 1-value pixels are taken as possible objects of the image. The multiresolution support is the set of all support images and it can be computed using the statistically significant wavelet coefficients.

Let *j* be a certain multiresolution level. Then, for each pixel *x*,*y* of the input image *I*, the multiresolution support at the level *j* is *M I jxy* ;,, 1 *I* contains significant information at the level *j* about the pixel (*x*,*y*).

If we denote by be the mother wavelet function, then the generic evaluation of the multiresolution support set results by computing the wavelet transform of the input image using followed by the computation of *MI*;,, *j x y* on the basis of the statistically significant wavelet coefficients for each resolution level *j* and for each pixel (*x*,*y*).

The computation of the wavelet transform of an one dimensional signal can be performed using the algorithm "À Trous" (Stark, Murtagh & Bijaoui, 1995). The algorithm can be extended to perform this computation in case of two-dimensional signals as, for instance, image signals

Using the resolution levels 1,2,..., *p* , where *p* is a selected level, the "À Trous" algorithm computes the wavelet coefficients according to the following scheme (Stark, Murtagh & Bijaoui, 1995).

Input: The sampled signal *c k* <sup>0</sup>

For *j*=0,1,…,*p* do

**Step 1.** *j*=*j*+1; compute, <sup>1</sup> <sup>1</sup> 2*<sup>j</sup> j j l c k hlc k l* .

**Step 2.** Step 2. Compute *jj j k c k ck* <sup>1</sup> .

#### End-for

22 Image Restoration – Recent Advances and Applications

corresponding to the observed signal X is assumed to be estimated from data. Then

 0 0 

*X*<sup>0</sup> results by the linear transform of matrix A applied to the sources S, *X AS* <sup>0</sup> , then

**2.2 The use of concepts and tools of multiresolution analysis for noise removal and** 

The multiresolution based algorithms perform the restoration tasks by combining, at each resolution level, according to a certain rule, the pixels of a binary support image. The values of the support image pixels are either 1 or 0 depending on their significance degree. At each resolution level, the contiguous areas of the support image corresponding to 1-value pixels are taken as possible objects of the image. The multiresolution support is the set of all support

Let *j* be a certain multiresolution level. Then, for each pixel *x*,*y* of the input image *I*, the multiresolution support at the level *j* is *M I jxy* ;,, 1 *I* contains significant information

multiresolution support set results by computing the wavelet transform of the input image

The computation of the wavelet transform of an one dimensional signal can be performed using the algorithm "À Trous" (Stark, Murtagh & Bijaoui, 1995). The algorithm can be extended to perform this computation in case of two-dimensional signals as, for instance,

Using the resolution levels 1,2,..., *p* , where *p* is a selected level, the "À Trous" algorithm computes the wavelet coefficients according to the following scheme (Stark, Murtagh &

<sup>1</sup> 2*<sup>j</sup>*

*c k hlc k l* .

*jj j k c k ck* <sup>1</sup>

followed by the computation of *MI*;,, *j x y* on the basis of the statistically

, where <sup>1</sup>

images and it can be computed using the statistically significant wavelet coefficients.

significant wavelet coefficients for each resolution level *j* and for each pixel (*x*,*y*).

are independent and , *<sup>T</sup> Cov*

*X*

, where , *<sup>T</sup> Cov X X* a n d t h e c o v a r i a n c e m a t r i x

. Consequently, the sources S are determined by

be the mother wavelet function, then the generic evaluation of the

2 *X*<sup>0</sup> 

is the unit matrix *nI* .If

. Usually, for simplicity sake, the matrix B is

 

is known, we

2 *X*<sup>0</sup> and

*~ N* 0,

*X BS*  assumed to be orthogonal.

**image restoration purposes** 

at the level *j* about the pixel (*x*,*y*).

Input: The sampled signal *c k* <sup>0</sup>

**Step 1.** *j*=*j*+1; compute, <sup>1</sup>

*j j l*

If we denote by

image signals

Bijaoui, 1995).

For *j*=0,1,…,*p* do

**Step 2.** Step 2. Compute

using

get 0 0 , *<sup>T</sup> Cov X X*

. In case *X*<sup>0</sup> and

 

, where <sup>1</sup>

maximizing the non-Gaussianity of *X BS*

 1 1 11 2 2 22 *XXX*

> *B A*<sup>2</sup>

are independent and the covariance matrix of <sup>1</sup>

Output: The set 1,..., , *j p j p k c*

Note that the computation of *c k <sup>j</sup>* carried out in Step 1 imposes that either the periodicity condition *ckN ck j j* or the continuity property *ck N cN j j* holds.

Since the representation of the original sampled signal is 0 1 *p p j j ck ck k* , in case of images, the values of 0 *c* are computed for each pixel (*x*,*y*) as 0 1 ,, , *p p j j c xy c xy xy* .

If the input image *I* encodes a noise component , then the wavelet coefficients also encode some information about . A label procedure is applied to each *<sup>j</sup>x y*, in order to remove the noise component from the wavelet coefficients computed for *I*. In case for each pixel (*x*,*y*) of *I*, the distribution of the coefficients is available, the significance level corresponding to each component *<sup>j</sup>x y*, can be established using a statistical test. We say that *I* is local constant at the resolution level *j* in case the amount of noise in *I* at this resolution level can be neglected. Let 0 be the hypothesis <sup>0</sup> : *I* is local constant at the resolution level *j*. In case there is significant amount of noise in *I* at the resolution level *j*, we get that the alternative hypothesis <sup>0</sup> :*<sup>j</sup> x y*, ~ <sup>2</sup> 0, *N <sup>j</sup>* . In order to define the critical region *W* of the statistical test we proceed as follows. Let 0 1 be the *a priori* selected significance level and let *z*be such that when 0 is true,

$$1 - \varepsilon = \text{Prob}\left(\left|\rho\_{\rangle}(\mathbf{x}, y)\right| < z\_{\varepsilon}\right) = \frac{1}{\sqrt{2\pi}\sigma\_{\mid}} \int\_{-z\_{\varepsilon}}^{z\_{\varepsilon}} \exp\left\{-\frac{t^2}{2\sigma\gamma^2}\right\} dt \tag{1}$$

In other words, the probability of rejecting 0 (hence accept <sup>0</sup> ) when 0 is true is and consequently, the critical region is *W zz* , . Accordingly, the significance level of the wavelet coefficients is given by the rule: *<sup>j</sup> x*,*y* is a significant coefficient if and only if *<sup>j</sup>x*,*yW* .

Usually, *z* is taken as *<sup>j</sup> k*, where *k* is a selected constant 3 *k* , because

$$\begin{split} P\left( \left| \left. o\_{\boldsymbol{\upbeta}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) \right| > k \sigma\_{\boldsymbol{\upbeta}} \right) = P\left( \left. o\_{\boldsymbol{\upbeta}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) > k \sigma\_{\boldsymbol{\upbeta}} \right) + P\left( \left. o\_{\boldsymbol{\upbeta}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) < -k \sigma\_{\boldsymbol{\upbeta}} \right) \right) = 1 \\ = 2P\left( \left. o\_{\boldsymbol{\upbeta}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) > k \sigma\_{\boldsymbol{\upbeta}} \right) = 2\left( 1 - P\left( \left. o\_{\boldsymbol{\uprho}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) \le k \sigma\_{\boldsymbol{\uprho}} \right) \right) \right. \\ \left. \left. \left. \left. \left. \mathbf{z} \right) \right|\_{\mathbf{x}} = 2\left( \left. o\_{\boldsymbol{\uprho}} \left( \mathbf{x}, \boldsymbol{\uprho} \right) \right|\_{\mathbf{y}} > k \sigma\_{\boldsymbol{\uprho}} \right) \right) \ge 2\left( 1 - \varepsilon \right) \end{split}$$

Using the significance level, we set to 1 the statistically significant coefficient and respectively we set to 0 the non-significant ones. The restored image *I* is,

$$\tilde{I}\left(\mathbf{x},\boldsymbol{y}\right) = c\_p\left(\mathbf{x},\boldsymbol{y}\right) + \sum\_{j=1}^{p} \mathbf{g}\left(\sigma\_{\boldsymbol{\cdot}}, o\_{\boldsymbol{\cdot}}\left(\mathbf{x},\boldsymbol{y}\right)\right) o\_{\boldsymbol{\cdot}}\left(\mathbf{x},\boldsymbol{y}\right) \tag{2}$$

where *g* is defined by

Statistical-Based Approaches for Noise Removal 25

shows the scatter of samples around their class expected vectors and it is typically given by

The between-class scatter matrix is the scatter of the expected vectors around the mixture

The mixture scatter matrix is the covariance matrix of all samples regardless of their class assignments and it is defined by *SSS m wb* . Note that all these scatter matrices are

In order to formulate criteria for class separability, these matrices should be converted into a number. This number should be larger when the between-class scatter is larger or the

*SS SS SS S S S S* 1 2 , , ,, , , , , *bw bm wm mw* and their values can be taken as measures of overall class separability. Obviously, both criteria are invariant under linear non-singular transforms and they are currently used for feature extraction purposes [8]. When the linear feature extraction problem is solved on the base of either 1 *J* or 2 *J* , their values are taken as numerical indicators of the loss of information implied by the reduction of dimensionality and implicitly deteriorating class separability. Consequently, the best linear feature

stands for the desired number of features , *J mA <sup>k</sup>* , is the value of the criterion , 1,2 *<sup>k</sup> J k* in

respectively, the value of each of the criteria , 1,2 *<sup>k</sup> J k* is a measure of overall class separability as well as well as a measure of the amount of information discriminating between these classes. In other words, , 1,2 *<sup>k</sup> J k* can be taken as measuring the effects of the noise removing filter expressing a measure of the quantity of information lost due to the use

The probability of error is the most effective measure of a classification decision rule usefulness, but its evaluation involves integrations on complicated regions in high

extraction is formulated as the optimization problem arg inf , *n m*\* *k k A R*

the transformed m-dimensional space of *<sup>T</sup> Y AX* , where A is a \* *n m* matrix .

 \* \* 1 2 arg inf , , ,..., , , 0 *n m*

unit eigenvectors corresponding to the m largest eigenvalues of <sup>1</sup> *S S*2 1

If the pattern classes are represented by the noisy image *X*

*k k <sup>m</sup> A R J mA J A A <sup>R</sup>*

1 <sup>ˆ</sup> *<sup>m</sup> i i i* 

 .

  *<sup>i</sup>* is the prototype of *Hi* and *<sup>i</sup>*

and each prototype is computed as the

0 represents the expected vector of the

1 21 *J tr S S* , <sup>1</sup>

*J mA J*

and the filtered image *F X*

*m m*

, where 1 ,..., *m* are

2 21 *J SS* ln , where

where m

(Cocianu, State &

is

 

the expression 1 1

*S XX* 

Very often, the *a priori* probabilities are taken <sup>1</sup> *<sup>i</sup> <sup>m</sup>*

ˆ ˆ *m N <sup>T</sup>*

where

within-class scatter is smaller. Typical criteria are <sup>1</sup>

*i k*

the *a priori* probability of , 1, *Hi m <sup>i</sup>* .

mean as 0 0 1 1

*i k*

invariant under coordinate shifts.

mixture distribution; usually

*S*

of the particular filter.

Vlamos, 2004).

Lemma 3. For any m, 1 *m n* ,

*b ii i*

ˆ ˆ *m N <sup>T</sup> i i w i k ik i*

weighted mean of the patterns belonging to the respective class.

, where ˆ

0 is taken as 0

$$\lg\left(\sigma\_{\boldsymbol{\beta}'}\boldsymbol{o}\_{\boldsymbol{\beta}}\left(\boldsymbol{x},\boldsymbol{y}\right)\right) = \begin{cases} 1, \left|o\_{\boldsymbol{\beta}}\left(\boldsymbol{x},\boldsymbol{y}\right)\right| \geq k\sigma\_{\boldsymbol{\beta}'}\\ 0, \left|o\_{\boldsymbol{\beta}}\left(\boldsymbol{x},\boldsymbol{y}\right)\right| < k\sigma\_{\boldsymbol{\beta}'} \end{cases}.$$

#### **2.3 Information-based approaches in image restoration**

The basics of the informational-based method for image restoration purposes are given by the following theoretical results (State, Cocianu & Vlamos, 2001).

Lemma 1 Let *X* be a continuous *n* -dimensional random vector and *AMR <sup>n</sup>* a nonsingular matrix, *Y AX* . Then, *H X* = *HY A* ln , where

$$H(X) = \mathbb{E} \int\_{\mathbb{R}^\*} f(x) \ln f(x) dx$$

is the differential entropy (Shannon), and *f* is the density function of *X* .

Lemma 2 Let *X* be a continuous *n* -dimensional normally distributed random vector, *X N* 0, and let *q* be a natural number, 1 *q* <n. If 1 2 *X X X* where <sup>1</sup> *X* is *q* -

dimensional, then, for any <sup>2</sup> *<sup>x</sup>* Rn-q, 12 2 *HX X x* <sup>=</sup> 1 12 2 *HX EX X x* , where

12 2 *EX X x*

is the regression function of <sup>1</sup> *X* on 2 2 *X x* , and 12 2 *HX X x* is the conditional entropy of <sup>1</sup> *X* on 2 2 *X x* .

Since 12 2 *HX X x* represents a measure of the amount of incertitude still remaining with respect to <sup>1</sup> *X* when <sup>2</sup> *X* is known, we get that the whole information carried by <sup>2</sup> *X* with respect to <sup>1</sup> *X* is concentrated on 12 2 *EX X x* .

$$\begin{array}{ll}\text{If } \text{ we } \quad \text{denotate } \Sigma\_{11} = \text{cov}\left(X^{(1)}, X^{(1)\top}\right)\_{\prime}, \Sigma\_{22} = \text{cov}\left(X^{(2)}, X^{(2)\top}\right)\_{\prime}, \quad \Sigma\_{12} = \text{cov}\left(X^{(1)}, X^{(2)\top}\right)\_{\prime}, \text{ we get} \\\text{E}\left(X^{(1)} \middle| X^{(2)} = x^{(2)}\right) = \Sigma\_{12} \Sigma\_{22}^{-1} \text{ and } \text{Var}^{(1)} \sim N\left(0, \Sigma\_{112}\right)\_{\prime}, \\\text{where } \quad Y^{(1)} = X^{(1)} - \Sigma\_{12} \Sigma\_{22}^{-1} X^{(2)} = X^{(1)} - E\left(X^{(1)} \middle| X^{(2)} = x^{(2)}\right)\_{\prime} \text{ and } \Sigma\_{112} = \Sigma\_{11} - \Sigma\_{12} \Sigma\_{22}^{-1} \left(\Sigma\_{12}\right)^{\intercal} \\\text{Consequently } H\left(X^{(1)} - E\left(X^{(1)} \middle| X^{(2)} = x^{(2)}\right)\right) = H\left(X^{(1)} \middle| X^{(2)} = x^{(2)}\right) = \frac{q}{2} \ln 2\pi e + \frac{1}{2} \ln \left|\Sigma\_{112}\right|. \end{array}$$

#### **2.4 The image restoration method based on scatter matrices and on bounds on the probability of error**

In statistical discriminant analysis, within-class, between-class and mixture scatter matrices are used to formulate criteria of class separability.

In case we need to discriminate between *m* classes , 1, *Hi m <sup>i</sup>* and <sup>1</sup> ,..., *i i X XN* are examples of patterns coming respectively from these classes, the within -class scatter matrix

24 Image Restoration – Recent Advances and Applications

 

*j j*

*x y k*

.

 1 2 *X*

where <sup>1</sup> *X* is *q* -

<sup>12</sup> cov , *<sup>T</sup> X X* , we get

*T*

*e* .

11.2 11 12 22 12

<sup>1</sup> ln 2 ln 2 2

*q*

*X* 

*X*

<sup>22</sup> cov , *<sup>T</sup> X X* , 1 2

*j j*

*x y k*

*j j*

 

the following theoretical results (State, Cocianu & Vlamos, 2001).

singular matrix, *Y AX* . Then, *H X* = *HY A* ln , where

*X N* 0, and let *q* be a natural number, 1 *q* <n. If

with respect to <sup>1</sup> *X* is concentrated on 12 2 *EX X x* .

where 1 1 <sup>1</sup> 2 1 12 2 *Y X x X EX X x* 12 22

are used to formulate criteria of class separability.

<sup>11</sup> cov , *<sup>T</sup> X X* , 2 2

Consequently 1 12 2 *HX EX X x* <sup>=</sup> 12 2 *HX X x* <sup>=</sup> 11.2

12 22*x* and <sup>1</sup> *Y N*0,11.2 ,

entropy of <sup>1</sup> *X* on 2 2 *X x* .

If we denote 1 1

12 2 *EX X x* = <sup>1</sup> <sup>2</sup>

**probability of error** 

**2.3 Information-based approaches in image restoration** 

*g xy*

is the differential entropy (Shannon), and *f* is the density function of *X* .

1, , , , 0, ,

The basics of the informational-based method for image restoration purposes are given by

Lemma 1 Let *X* be a continuous *n* -dimensional random vector and *AMR <sup>n</sup>* a non-

*H X* =- ln *n R*

Lemma 2 Let *X* be a continuous *n* -dimensional normally distributed random vector,

dimensional, then, for any <sup>2</sup> *<sup>x</sup>* Rn-q, 12 2 *HX X x* <sup>=</sup> 1 12 2 *HX EX X x* , where

12 2 *EX X x*

is the regression function of <sup>1</sup> *X* on 2 2 *X x* , and 12 2 *HX X x* is the conditional

Since 12 2 *HX X x* represents a measure of the amount of incertitude still remaining with respect to <sup>1</sup> *X* when <sup>2</sup> *X* is known, we get that the whole information carried by <sup>2</sup> *X*

, and <sup>1</sup>

**2.4 The image restoration method based on scatter matrices and on bounds on the** 

In statistical discriminant analysis, within-class, between-class and mixture scatter matrices

In case we need to discriminate between *m* classes , 1, *Hi m <sup>i</sup>* and <sup>1</sup> ,..., *i i X XN* are examples of patterns coming respectively from these classes, the within -class scatter matrix

*<sup>f</sup> <sup>x</sup> <sup>f</sup> x dx*

shows the scatter of samples around their class expected vectors and it is typically given by the expression 1 1 ˆ ˆ *m N <sup>T</sup> i i w i k ik i i k S XX* , where ˆ*<sup>i</sup>* is the prototype of *Hi* and *<sup>i</sup>* is the *a priori* probability of , 1, *Hi m <sup>i</sup>* .

Very often, the *a priori* probabilities are taken <sup>1</sup> *<sup>i</sup> <sup>m</sup>* and each prototype is computed as the weighted mean of the patterns belonging to the respective class.

The between-class scatter matrix is the scatter of the expected vectors around the mixture mean as 0 0 1 1 ˆ ˆ *m N <sup>T</sup> b ii i i k S* where 0 represents the expected vector of the

mixture distribution; usually 0 is taken as 0 1 <sup>ˆ</sup> *<sup>m</sup> i i i* .

The mixture scatter matrix is the covariance matrix of all samples regardless of their class assignments and it is defined by *SSS m wb* . Note that all these scatter matrices are invariant under coordinate shifts.

In order to formulate criteria for class separability, these matrices should be converted into a number. This number should be larger when the between-class scatter is larger or the within-class scatter is smaller. Typical criteria are <sup>1</sup> 1 21 *J tr S S* , <sup>1</sup> 2 21 *J SS* ln , where *SS SS SS S S S S* 1 2 , , ,, , , , , *bw bm wm mw* and their values can be taken as measures of overall class separability. Obviously, both criteria are invariant under linear non-singular transforms and they are currently used for feature extraction purposes [8]. When the linear feature extraction problem is solved on the base of either 1 *J* or 2 *J* , their values are taken as numerical indicators of the loss of information implied by the reduction of dimensionality and implicitly deteriorating class separability. Consequently, the best linear feature extraction is formulated as the optimization problem arg inf , *n m*\* *k k A R J mA J* where m stands for the desired number of features , *J mA <sup>k</sup>* , is the value of the criterion , 1,2 *<sup>k</sup> J k* in the transformed m-dimensional space of *<sup>T</sup> Y AX* , where A is a \* *n m* matrix .

If the pattern classes are represented by the noisy image *X* and the filtered image *F X* respectively, the value of each of the criteria , 1,2 *<sup>k</sup> J k* is a measure of overall class separability as well as well as a measure of the amount of information discriminating between these classes. In other words, , 1,2 *<sup>k</sup> J k* can be taken as measuring the effects of the noise removing filter expressing a measure of the quantity of information lost due to the use of the particular filter.

Lemma 3. For any m, 1 *m n* ,

 \* \* 1 2 arg inf , , ,..., , , 0 *n m m m k k <sup>m</sup> A R J mA J A A <sup>R</sup>* , where 1 ,..., *m* are unit eigenvectors corresponding to the m largest eigenvalues of <sup>1</sup> *S S*2 1 (Cocianu, State & Vlamos, 2004).

The probability of error is the most effective measure of a classification decision rule usefulness, but its evaluation involves integrations on complicated regions in high

Statistical-Based Approaches for Noise Removal 27

4 *<sup>n</sup>* <sup>=</sup>

If the linear feature extractor is defined by the matrix *nxm A R* , then the value of the

+ 1 11 *BA m m m A m* 12 2 1 21

*<sup>T</sup> i i m AA* , 1,2 *<sup>i</sup>* and <sup>1</sup> 1 1

 

1 11 22 1 2 12 1 11 11 2 1 21

Obviously the criterion function *J* is invariant with respect to non-singular transforms and, using standard arguments, one can prove that <sup>1</sup> ,..., *<sup>m</sup> m* can be taken as the suboptimal linear feature extractor where , 1, *<sup>i</sup> i m* are unit eigenvectors corresponding to

1

, a series of attempts to find suboptimal feature extractors have been proposed

Since there is no known procedure available to optimize the criterion *J* when 1 2 and

**3.1 Minimum mean-square error filtering (MMSE), and the adaptive mean-variance** 

The minimum mean-square error filter (MMSE) is an adaptive filter in the sense that its basic behavior changes as the image is processed. Therefore an adaptive filter could process

*Am m m Am Am m m Am* 

1 2 .

*<sup>J</sup> m A*, <sup>=</sup> 1 1

The critical points of *<sup>J</sup> m A*, are the solutions of the equation , <sup>0</sup> *JmA*

Suboptimal solutions can be identified as the solutions of the system

such that 1

But, in case of image restoration problem, each of the assumptions

unrealistic, therefore, we are forced to accept the hypothesis that

21 2 1 *A Am m* .

1

4 *n*

21 12


12 21 2 *TT TT B AA AA AA AA Im* .

*TT TT mI AA AA AA AA*

1 1 <sup>2</sup>

*j j j* 

 - ln 2

4

4

that is,

(8)

*A*

=0 (7)

0 0

<sup>111</sup> ... ... *m n*

.

*m n*

 

> 

1 2 

1 2 

, 1 2 is

and 1 2 .

*<sup>n</sup>* (5)

*<sup>m</sup>* . (6)

*J* = 1 1

<sup>1</sup> ln 2

*j n* are the eigenvalues of <sup>1</sup>

1 11 *BA m m m A m* 22 1 2 12

or equivalently, 1 1

 *<sup>m</sup>* of <sup>1</sup> 2 1

the eigenvalues 1, ...,

instead (Fukunaga, 1990)

**3. Noise removal algorithms** 

**removal algorithm (AMVR)** 

1 2 

where , 1, *<sup>j</sup>* 

where

21 12

<sup>4</sup> *nI* - ln 2

Bhattacharyya distance in the transformed space *<sup>T</sup> Y AX* is given by,

dimensional spaces. When a closed-form expression for the error probability can not be obtained, we may seek either for approximate expressions, or upper/lower bounds for the error probability.

Assume that the design of the Bayes classifier is intended to discriminate between two pattern classes and the available information is represented by mean vectors *<sup>i</sup>* , *i* 1,2 and the covariance matrices *<sup>i</sup>* , *i* 1,2 corresponding to the repartitions of the classes respectively. The Chernoff upper bounds of the Bayesian error (Fukunaga, 1990) are given by <sup>1</sup> <sup>1</sup> 12 1 2 *<sup>s</sup> <sup>s</sup> s s s <sup>f</sup> x f x dx* , *s*0,1 , where 1 2 , is the *a priori* distribution and *<sup>i</sup> f* is the density function corresponding to the *i th* class, 1,2 *i* . When both density functions are normal, *<sup>i</sup> f Ni i* , 1,2 *i* , the integration can be carried out to obtain a closed-form expression for *<sup>s</sup>* , that is <sup>1</sup> 1 2 *s s f x f x dx* <sup>=</sup> exp*s* where

$$\mu(s) = \frac{s(1-s)}{2} (\mu\_2 - \mu\_1)^{\mathrm{T}} \left( s\Sigma\_1 + (1-s)\Sigma\_2 \right)^{-1} \left( \mu\_2 - \mu\_1 \right) + \frac{1}{2} \ln \frac{\left| s\Sigma\_1 + (1-s)\Sigma\_2 \right|}{\left| \Sigma\_1 \right|^{\mathrm{s}} \left| \Sigma\_2 \right|^{1-s}} \tag{3}$$

The upper bound <sup>1</sup> 2 <sup>=</sup> 1 1 2 2 1 2 1 1 8 2 *T* + 1 2 1 2 <sup>1</sup> <sup>2</sup> ln 2 is called the

Bhattacharyya distance and it is frequently used as a measure of the separability between two repartitions. Using straightforward computations, the Bhattacharyya distance can be written as,

$$\mu \left( \frac{1}{2} \right) = \frac{1}{8} tr \left\{ \Sigma^{-1} \left( \mu\_2 - \mu\_1 \right) \left( \mu\_2 - \mu\_1 \right)^{\mathrm{T}} \right\} + \frac{1}{4} \ln \left| 2I\_n + \Sigma\_1 \Sigma\_2^{-1} + \Sigma\_2 \Sigma\_1^{-1} \right| - \frac{n}{4} \ln 2 \tag{4}$$

where

$$
\overline{\Sigma} = \frac{\Sigma\_1 + \Sigma\_2}{2}
$$

Note that one of the first two terms in (4) vanishes, when 1 2 , 1 2 respectively, that is the first term expresses the class separability due to the mean-difference while the second one gives the class separability due to the covariance difference.

The Bhattacharyya distance can be used as criterion function as well to express the quality of a linear feature extractor of matrix *nxm A R* .

When 1 2 <sup>=</sup> , <sup>1</sup> <sup>2</sup> *<sup>J</sup>* <sup>=</sup> <sup>1</sup> 2 12 1 1 8 *T tr* therefore *J* is a particular case of the criterion 1 *<sup>J</sup>* for *S*<sup>2</sup> and 1 2 12 1 *<sup>T</sup> S S <sup>b</sup>* . Consequently the whole information about the class separability is contained by an unique feature 1 2 1 1 1 2 1 . When 1 2 and 1 2 ,

$$J = \frac{1}{4} \text{ln} \left| 2I\_n + \Sigma\_2^{-1} \Sigma\_1 + \Sigma\_1^{-1} \Sigma\_2 \right| \cdot \frac{n}{4} \text{ln} \, 2 = \frac{1}{4} \sum\_{\langle \rangle = 1}^{n} \left( 2 + \lambda\_{\rangle} + \frac{1}{\lambda\_{\langle \rangle}} \right) \cdot \frac{n}{4} \text{ln} \, 2 \tag{5}$$

where , 1, *<sup>j</sup> j n* are the eigenvalues of <sup>1</sup> 1 2 .

If the linear feature extractor is defined by the matrix *nxm A R* , then the value of the Bhattacharyya distance in the transformed space *<sup>T</sup> Y AX* is given by,

$$J \text{ (} m\_\prime A \text{)} = \frac{1}{4} \ln \left| 2I\_m + \left( A^T \Sigma\_2 A \right)^{-1} \left( A^T \Sigma\_1 A \right) + \left( A^T \Sigma\_1 A \right)^{-1} \left( A^T \Sigma\_2 A \right) \right| \cdot \frac{m}{4} \ln 2 \,\tag{6}$$

The critical points of *<sup>J</sup> m A*, are the solutions of the equation , <sup>0</sup> *JmA A* that is,

$$B\left\{\Sigma\_2 A \Sigma\_2^{-1} \left(m\right) \Sigma\_1 \left(m\right) \Sigma\_2^{-1} \left(m\right) - \Sigma\_1 A \Sigma\_2^{-1} \left(m\right)\right\}^+ B \left\{\Sigma\_1 A \Sigma\_2^{-1} \left(m\right) \Sigma\_2 \left(m\right) \Sigma\_1^{-1} \left(m\right) - \Sigma\_2 A \Sigma\_1^{-1} \left(m\right)\right\} = 0 \tag{7}$$

where

26 Image Restoration – Recent Advances and Applications

dimensional spaces. When a closed-form expression for the error probability can not be obtained, we may seek either for approximate expressions, or upper/lower bounds for the

Assume that the design of the Bayes classifier is intended to discriminate between two

the covariance matrices *<sup>i</sup>* , *i* 1,2 corresponding to the repartitions of the classes respectively. The Chernoff upper bounds of the Bayesian error (Fukunaga, 1990) are given

and *<sup>i</sup> f* is the density function corresponding to the *i th* class, 1,2 *i* . When both density

*s s f x f x dx* <sup>=</sup> exp

1

1

 

<sup>1</sup> ln 2

 

<sup>+</sup> 1 2

1 2 

*T*

therefore *J* is a particular case of

+

 , that is <sup>1</sup> 1 2

1

<sup>=</sup>

1 2 2 1 2 1

Bhattacharyya distance and it is frequently used as a measure of the separability between two repartitions. Using straightforward computations, the Bhattacharyya distance can be

*T*

+ 1 1

1 2 2 

is the first term expresses the class separability due to the mean-difference while the second

The Bhattacharyya distance can be used as criterion function as well to express the quality of

2 12 1

*S S <sup>b</sup>* 

information about the class separability is contained by an unique feature

<sup>=</sup> <sup>1</sup>

and 1 2 ,

 

8 2 *T*

2 12 1

21 1 2 21

*s s*

 

*i i* , 1,2 *i* , the integration can be carried out to obtain a

1 1 ln

<sup>1</sup> <sup>2</sup> ln 2

12 21

<sup>4</sup> *nI* - ln 2

2 *s s s s*

1 2

1 2

4

, 1 2 respectively, that

. Consequently the whole

*s* where

1 2 , is the *a priori* distribution

1 1 2

 

(3)

is called the

*<sup>n</sup>* (4)

*<sup>i</sup>* , *i* 1,2 and

pattern classes and the available information is represented by mean vectors

*<sup>f</sup> x f x dx* , *s*0,1 , where

1

<sup>=</sup> <sup>1</sup>

Note that one of the first two terms in (4) vanishes, when

one gives the class separability due to the covariance difference.

1 8

the criterion 1 *<sup>J</sup>* for *S*<sup>2</sup> and 1 2 12 1 *<sup>T</sup>*

1 2 

*tr*

*s s <sup>T</sup>* 

> 2

 

error probability.

*s* 

written as,

where

by <sup>1</sup> <sup>1</sup> 12 1 2 *<sup>s</sup> <sup>s</sup> s s*

functions are normal, *<sup>i</sup> f N*

closed-form expression for *<sup>s</sup>*

*<sup>s</sup>* <sup>=</sup>

1

2

The upper bound <sup>1</sup>

1 2 

1 8

*tr*

a linear feature extractor of matrix *nxm A R* .

<sup>2</sup> *<sup>J</sup>* 

. When

When 1 2 <sup>=</sup> , <sup>1</sup>

 

2 1

2 1

1

 

1 1

 

$$\Sigma\_i(m) = A^T \Sigma\_i A \text{ , } i = \mathbf{1}, \mathbf{2} \text{ and } B = \left[ \left( A^T \Sigma\_1 A \right)^{-1} \left( A^T \Sigma\_2 A \right) + \left( A^T \Sigma\_2 A \right)^{-1} \left( A^T \Sigma\_1 A \right) + 2I\_m \right]^{-1}.$$

Suboptimal solutions can be identified as the solutions of the system

$$\begin{cases} \Sigma\_2 A \Sigma\_2^{-1} \left( m \right) \Sigma\_1 \left( m \right) \Sigma\_2^{-1} \left( m \right) - \Sigma\_1 A \Sigma\_2^{-1} \left( m \right) = 0\\ \Sigma\_1 A \Sigma\_1^{-1} \left( m \right) \Sigma\_2 \left( m \right) \Sigma\_1^{-1} \left( m \right) - \Sigma\_2 A \Sigma\_1^{-1} \left( m \right) = 0 \end{cases} \tag{8}$$

or equivalently, 1 1 21 2 1 *A Am m* .

Obviously the criterion function *J* is invariant with respect to non-singular transforms and, using standard arguments, one can prove that <sup>1</sup> ,..., *<sup>m</sup> m* can be taken as the suboptimal linear feature extractor where , 1, *<sup>i</sup> i m* are unit eigenvectors corresponding to

$$\text{If the eigenvalues } \lambda\_1 \dots \lambda\_n \text{ of } \Sigma\_2^{-1} \Sigma\_1 \text{ such that } \lambda\_1 + \frac{1}{\lambda\_1} \ge \dots \ge \lambda\_n + \frac{1}{\lambda\_n} \ge \dots \ge \lambda\_n + \frac{1}{\lambda\_n}.$$

But, in case of image restoration problem, each of the assumptions 1 2 , 1 2 is unrealistic, therefore, we are forced to accept the hypothesis that 1 2 and 1 2 . Since there is no known procedure available to optimize the criterion *J* when 1 2 and 1 2 , a series of attempts to find suboptimal feature extractors have been proposed instead (Fukunaga, 1990)

#### **3. Noise removal algorithms**

#### **3.1 Minimum mean-square error filtering (MMSE), and the adaptive mean-variance removal algorithm (AMVR)**

The minimum mean-square error filter (MMSE) is an adaptive filter in the sense that its basic behavior changes as the image is processed. Therefore an adaptive filter could process

Statistical-Based Approaches for Noise Removal 29

Let us consider the following information transmission/processing system. The signal *X* representing a certain image is transmitted through a channel and its noise-corrupted

*X* , an approximation of the initial signal *X* . In our attempt (Cocianu, State & Vlamos, 2004) we assumed that there is no available information about the initial signal *X* , therefore the

 

 , where

> 

 on *X* ,

11.2 11 12 22 12

, <sup>2</sup> *E X*

covariance matrices. We consider the working assumption that the 2 *r c* -dimensional

 *T* (see § 2.3).

> and *X*

As a particular case , using the conclusions established by the lemmas 1 and 2 (§ 2.3), we can

and *EFX X*

 

is received. Next, a noise-removing binomial filter is applied to the output *X*

where

. Obviously, *X*

is also normally distributed, therefore the conditional distribution of

 

 = 1 2 <sup>1</sup> 12 22 *X* 

 *N*

> >


is submitted to a restoration process producing

 and *F X* 

. We assume that the

<sup>1</sup> ,..., *X XN* the resulted outputs and

<sup>1</sup> ,..., *X XN* as a Bernoullian sample of

and *F X*

and let <sup>11</sup> , 22 be their

(9)

minimizes the variance and

in the class of linear functions of *X*

 and

with respect to *F X*

 

, and 1 1

<sup>1</sup> ,..., *X XN* is a

are normally

.

is

contains the

Output The image *X*ˆ .

version *X*

resulting *F X*

by 1 1

vector *X FX* , 

 *F X* on *X* 

 *X* 

**3.2 Information-based algorithms for noise removal** 

. Finally, the signal *F X*

restoration process should be based exclusively on *X*

<sup>1</sup> ,..., *X XN* their corresponding filtered versions.

the random *r c* -dimensional vector *X X*

sample of the filtered random vector *F X*

distributed. Let us denote <sup>1</sup> *EFX*

 

is the regression function of *F X*

maximizes the correlation between *F X*

 <sup>1</sup>

> 

> > .

conclude that *HFX EFX X HFX X*

and <sup>12</sup> cov , *FX X*

Moreover, *EFX X*

contained by *EFX X*

message *X* is transmitted *N* times and we denote by 2 2

If we denote the given image by *<sup>X</sup>* , then we model 2 2

 is 1 2 <sup>1</sup> 12 22 11.2 *N X* , 

> *EFX X*

It is well known (Anderson, 1958) that *FX EFX X*

 is *X* 

are independent, the whole information carried by *X*

 

differently on different segments of an image. The particular MMSE filter may act as a mean filter on some windows of the image and as a median filter on other windows of the image. The MMSE filter allows the removal of the normal/uniform additive noise and its computation is carried out as

$$\overline{X}\left(l,\mathcal{c}\right) = Y\left(l,\mathcal{c}\right) - \frac{\sigma^2}{\sigma\_{l,\mathcal{c}}^2} \left[Y\left(l,\mathcal{c}\right) - \mu\_{l,\mathcal{c}}\right] \nu$$

for *t l R tt c C t* , , where Y is a *R C* noisy image, Wl,c is the *n n* window centered in (l,c), where *n t t l R tt c C t* 2 1, , , <sup>2</sup> is the noise variance, <sup>2</sup> *l c*. is the local variance (in the window *Wl c*, ), and <sup>2</sup> *l c*. is the local mean (average in the window *Wl c*, ).

Note that since the background region of the image is an area of fairly constant value in the original uncorrupted image, the noise variance is almost equal to the local variance, and consequently the MMSE performs as a mean filter. In image areas where the local variances are much larger than the noise variance, the filter computes a value close to the pixel value corresponding to the unfiltered image data. The magnitude of the original and local means

respectively used to modify the initial image are weighted by 2 2 *l c*, , the ratio of noise to local

variance. As the value of the ratio increases, implying primarily noise in the window, the filter returns primarily the value of the local average. As this ratio decreases, implying high local detail, the filter returns more of the original unfiltered image. Consequently, the MMSE filter adapts itself to the local image properties, preserving image details while removing noise.(Umbaugh,1998).

The special filtering techniques of mean type prove particularly useful in reducing the normal/uniform noise component when the mean parameter is close to 0. In other words, the effects determined by the application of mean filters are merely the decrease of each processed window local variance and consequently the removal of the variance component of the noise.

The AMVR algorithm allows to remove the normal/uniform noise whatever the mean of the noise is. Similar to MMSE filtering technique in application of the AMVR algorithm, the noise parameters and features are known. Basically, the AMVR algorithm works in two stages, namely the removal of the mean component of the noise (Step 1 and Step 2), and the decrease of the variance of the noise using the adaptive filter MMSE. The description of the AMVR algorithm is (Cocianu, State & Vlamos, 2002) is,

Input The image Y of dimensions *R C* , representing a normal/uniform disturbed version of the initial image X, <sup>0</sup> , , , *Y lc X lc l c* , 1 ,1 *lR cC* , where <sup>0</sup> *l c*, is a sample of the random variable *l c*, distributed either <sup>2</sup> , , , *N lc lc* or <sup>2</sup> , , , *U lc lc* .


**Step 3.** Compute the estimate *X*ˆ of X using the adaptive filter MMSE, *X MMSE X* ˆ .

Output The image *X*ˆ .

28 Image Restoration – Recent Advances and Applications

differently on different segments of an image. The particular MMSE filter may act as a mean filter on some windows of the image and as a median filter on other windows of the image. The MMSE filter allows the removal of the normal/uniform additive noise and its

<sup>2</sup>

*X lc Y lc Y lc*

centered in (l,c), where *n t t l R tt c C t* 2 1, , , <sup>2</sup>

respectively used to modify the initial image are weighted by

AMVR algorithm is (Cocianu, State & Vlamos, 2002) is,

, , , *Y lc X lc*

*l c*, distributed either <sup>2</sup>

the processed image Y, where , , , *<sup>i</sup> X lc Y lc <sup>i</sup>*

1 <sup>1</sup> , , *n i i X lc X lc n*

of the initial image X, <sup>0</sup>

sample of the random variable

pixel values,

local variance (in the window *Wl c*, ), and <sup>2</sup>

, ,, , *l c l c*

,

for *t l R tt c C t* , , where Y is a *R C* noisy image, Wl,c is the *n n* window

Note that since the background region of the image is an area of fairly constant value in the original uncorrupted image, the noise variance is almost equal to the local variance, and consequently the MMSE performs as a mean filter. In image areas where the local variances are much larger than the noise variance, the filter computes a value close to the pixel value corresponding to the unfiltered image data. The magnitude of the original and local means

variance. As the value of the ratio increases, implying primarily noise in the window, the filter returns primarily the value of the local average. As this ratio decreases, implying high local detail, the filter returns more of the original unfiltered image. Consequently, the MMSE filter adapts itself to the local image properties, preserving image details while removing

The special filtering techniques of mean type prove particularly useful in reducing the normal/uniform noise component when the mean parameter is close to 0. In other words, the effects determined by the application of mean filters are merely the decrease of each processed window local variance and consequently the removal of the variance component of the noise. The AMVR algorithm allows to remove the normal/uniform noise whatever the mean of the noise is. Similar to MMSE filtering technique in application of the AMVR algorithm, the noise parameters and features are known. Basically, the AMVR algorithm works in two stages, namely the removal of the mean component of the noise (Step 1 and Step 2), and the decrease of the variance of the noise using the adaptive filter MMSE. The description of the

Input The image Y of dimensions *R C* , representing a normal/uniform disturbed version

**Step 2.** Compute *X* , the sample mean estimate of the initial image X, by averaging the

, 1 ,1 *lR cC* .

**Step 3.** Compute the estimate *X*ˆ of X using the adaptive filter MMSE, *X MMSE X* ˆ .

, , , *N* 

*l c* , 1 ,1 *lR cC* , where <sup>0</sup>

 *lc lc* or <sup>2</sup> , , , *U lc lc* . *l c* , 1 ,1 *lL cC* and ,

*l c*, is a sample of the

*i l c* is a

*l c*, from

**Step 1.** Generate the sample of images *XX X* 1 2 , ,..., *<sup>n</sup>* , by subtracting the noise

*l c*, . 2 .

is the noise variance, <sup>2</sup>

*l c*. is the local mean (average in the window *Wl c*, ).

2 2 *l c*, 

*l c*. is the

, the ratio of noise to local

computation is carried out as

noise.(Umbaugh,1998).

random variable

### **3.2 Information-based algorithms for noise removal**

Let us consider the following information transmission/processing system. The signal *X* representing a certain image is transmitted through a channel and its noise-corrupted version *X* is received. Next, a noise-removing binomial filter is applied to the output *X* resulting *F X* . Finally, the signal *F X* is submitted to a restoration process producing *X* , an approximation of the initial signal *X* . In our attempt (Cocianu, State & Vlamos, 2004) we assumed that there is no available information about the initial signal *X* , therefore the restoration process should be based exclusively on *X* and *F X* . We assume that the message *X* is transmitted *N* times and we denote by 2 2 <sup>1</sup> ,..., *X XN* the resulted outputs and by 1 1 <sup>1</sup> ,..., *X XN* their corresponding filtered versions.

If we denote the given image by *<sup>X</sup>* , then we model 2 2 <sup>1</sup> ,..., *X XN* as a Bernoullian sample of the random *r c* -dimensional vector *X X* where *N*, and 1 1 <sup>1</sup> ,..., *X XN* is a sample of the filtered random vector *F X* . Obviously, *X* and *F X* are normally distributed. Let us denote <sup>1</sup> *EFX* , <sup>2</sup> *E X* and let <sup>11</sup> , 22 be their covariance matrices. We consider the working assumption that the 2 *r c* -dimensional vector *X FX* , is also normally distributed, therefore the conditional distribution of *F X* on *X* is 1 2 <sup>1</sup> 12 22 11.2 *N X* , , where

$$E\left(F\left(X^{(\eta)}\right)\middle|X^{(\eta)}\right) = \mu^{(1)} + \Sigma\_{12}\Sigma\_{22}^{-1}\left(X^{(\eta)} - \mu^{(2)}\right) \tag{9}$$

is the regression function of *F X* on *X* ,

$$\mathbf{B} \text{ and } \boldsymbol{\Sigma}\_{12} = \text{cov}\left(\boldsymbol{F}\left(\boldsymbol{X}^{(\boldsymbol{\eta})}\right), \boldsymbol{X}^{(\boldsymbol{\eta})}\right) \\ \boldsymbol{\Sigma}\_{11,2} = \boldsymbol{\Sigma}\_{11} - \boldsymbol{\Sigma}\_{12} \boldsymbol{\Sigma}\_{22}^{-1} \left(\boldsymbol{\Sigma}\_{12}\right)^{\top} \text{ (see § 2.3).} $$

It is well known (Anderson, 1958) that *FX EFX X* minimizes the variance and maximizes the correlation between *F X* and *X* in the class of linear functions of *X* . Moreover, *EFX X* is *X* -measurable and, since *FX EFX X* and *X* are independent, the whole information carried by *X* with respect to *F X* is contained by *EFX X* .

As a particular case , using the conclusions established by the lemmas 1 and 2 (§ 2.3), we can conclude that *HFX EFX X HFX X* and *EFX X* contains the

Statistical-Based Approaches for Noise Removal 31

**Step 2.** For each row 1 *i r* , compute

<sup>1</sup> <sup>ˆ</sup> ˆ ˆ , , 1,2 <sup>1</sup> *<sup>N</sup> <sup>T</sup> ttss*

.

ˆ 2 1 12 22 ˆ ˆ 

*i Xi iXi i t s*

experimental arguments, the recommended range of

, 4 *t* to each component of 2 2

 

**Step 4.** Compute the rows *X i* of the restored image *X* ,

12 1 11 1 1 2 <sup>1</sup> *EFX X* 12 22 *X* 12 22 *X* 12 22

the correction term used at Step 4 is precisely the sample mean estimation of the

The idea of our attempt is to use the most informative features discriminating between

attempt is justified by the argument that besides information about the removed noise, the

appropriate information allowing quality improvement of the image *F X*

 

*N*

 , 1 <sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

*i kk k*

Since <sup>1</sup> *<sup>b</sup> rank S* , we get <sup>1</sup> <sup>1</sup> *w b rank S S* , that is when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* ,the matrix 1 *S S*2 1 has an unique positive eigenvalue, one of its unit eigenvectors being given by

for getting correction terms in restoring the filtered images *F X*

and *SSS m wb* where

*X X*

*<sup>N</sup> <sup>T</sup> iiii*

, 1,2 *<sup>i</sup>*

 *ii i i* ˆ ˆ , where

constant conveniently determined to prevent the restoration of the initial noise. By

(12)

 

<sup>1</sup> ,..., *X XN* be the sample of noise corrupted versions of the

  1

ˆ by applying a threshold filter to <sup>1</sup>

and *F X*

<sup>1</sup> ,..., *X XN* their filtered versions,

0.5 , therefore the scatter matrices become

is 1.5,5.5 .

*<sup>N</sup> p p*

<sup>1</sup> <sup>ˆ</sup> , 1 , 1,2

,

*i X i i rp <sup>N</sup>*

*k k*

<sup>1</sup> ,..., *X XN* ,

ˆ *i* .

. The

would contain

(Cocianu,

is a noise-preventing

 

121 <sup>1</sup> 2 2

121

1

**Step 3.** For each row 1 *i r* , compute <sup>1</sup> *T i*

*k*

*ts k k*

Note that since the regression function can be written as,

most informative features discriminating between *X*

*r c* dimensional image X and 1 1

*k k X*

1 2 , 1 *X FX i N i i* .

12

*M t <sup>t</sup> <sup>t</sup>* 

*N*

 <sup>1</sup> *Xi T i* 

> 

 <sup>1</sup> <sup>1</sup> *E X* 12 22 .

and *F X*

1 2

State & Vlamos, 2004). Let 2 2

ˆ ˆ *Sw* , <sup>1212</sup> ˆˆˆˆ *<sup>T</sup> Sb* 

1 2 , 1 *X FX i N i i* . We assume 1 2

 1 <sup>1</sup> <sup>ˆ</sup> *<sup>N</sup> i i*

*N*

 *X* 

whole information existing in *X* with respect to *F X* a part of it being responsible for the initial existing noise and another component being responsible for the quality degradation.

According to our regression-based algorithm, the rows of the restored image *X* are computed sequentially on the basis of the samples 2 2 <sup>1</sup> ,..., *X XN* and 1 1 <sup>1</sup> ,..., *X XN* representing the available information about *X* and *F X* 

If we denote by *<sup>p</sup> X i <sup>k</sup>* the i-th row of , 1,2, 1 *<sup>p</sup> Xk p k N* , then the mean vectors *p* are estimated by the corresponding sample means ˆˆ ˆ 1 ,..., *pp p r* , where 1 <sup>1</sup> <sup>ˆ</sup> , 1 *<sup>N</sup> p p k k i X i ir N* and the covariance matrices , , 1,2 *ts i ts* are estimated respectively by their sample covariance matrices counterparts, 1 <sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup> *<sup>N</sup> <sup>T</sup> ttss ts k k k i Xi iXi i N* . Frequently enough it happens that the matrices <sup>ˆ</sup> , 1,2 *tt i t* are ill conditioned, therefore in our method the Penrose pseudoinverse ˆ *tt <sup>i</sup>* is used instead of <sup>1</sup> <sup>ˆ</sup> *tt <sup>i</sup>* .

Since the aim is to restore as much as possible the initial image X, we have to find out ways to improve the quality of *F X* in the same time preventing the introduction additional noise.

Obviously,

$$\mu^{(1)} = E\left(F\left(X + \eta\right)\right) = F\left(X + E\left(\eta\right)\right) = F\left(X\right) + F\left(E\left(\eta\right)\right) = F\left(X\right) + E\left(\eta\right) \tag{10}$$

$$
\mu^{(2)} = E(X + \eta) = X + E(\eta) \tag{11}
$$

hence 1 2 *FX X* and <sup>1</sup> *X X FX E* . In other words, 1 2 can be viewed as measuring the effects of the noise as well as the quality degradation while the term <sup>1</sup> *X* retains more information about the quality of image and less information about (Cocianu, State & Vlamos, 2004). This argument entails the heuristic used by our method (Step 4), the restored image being obtained by applying a threshold filter to <sup>1</sup> and adding the correction term <sup>1</sup> 2 1 12 22 , <sup>1</sup> *XT i* ˆ <sup>1</sup> 2 1 12 22 .

The heuristic regression -based algorithm (HRBA) for image restoration (Cocianu, State & Vlamos, 2004)

Input: The sample 2 2 <sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* dimensional image X **Step 1.** Compute the sample 1 1 <sup>1</sup> ,..., *X XN* by applying the binomial filter of mask 30 Image Restoration – Recent Advances and Applications

According to our regression-based algorithm, the rows of the restored image *X* are

and the covariance matrices , , 1,2 *ts i ts* are estimated

respectively by their sample covariance matrices counterparts,

 . Frequently enough it happens that the matrices <sup>ˆ</sup> , 1,2 *tt i t* are ill conditioned, therefore in our method the Penrose

> *tt <sup>i</sup>* .

Since the aim is to restore as much as possible the initial image X, we have to find out ways

<sup>1</sup>

<sup>2</sup>

*EFX FX E FX FE FX E*

 *EX X E* 

heuristic used by our method (Step 4), the restored image being obtained by applying a

The heuristic regression -based algorithm (HRBA) for image restoration (Cocianu, State &

 *FX X* and <sup>1</sup> *X X FX E* 

and *F X*

, 1,2, 1 *<sup>p</sup> Xk p k N* , then the mean vectors *p*

in the same time preventing the introduction additional

 

(Cocianu, State & Vlamos, 2004). This argument entails the

<sup>1</sup> ,..., *X XN* by applying the binomial filter of mask

and adding the correction term <sup>1</sup> 2 1

<sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* dimensional image X

 

retains more information about the quality of image

 

and another component being responsible for the quality degradation.

a part of it being responsible for the

<sup>1</sup> ,..., *X XN* and 1 1

 *r* , where

> 

(11)

. In other words,

as well as the quality

 12 22 ,

(10)

<sup>1</sup> ,..., *X XN*

are

with respect to *F X*

computed sequentially on the basis of the samples 2 2

estimated by the corresponding sample means ˆˆ ˆ 1 ,..., *pp p*

 

whole information existing in *X*

representing the available information about *X*

*i Xi iXi i*

<sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

*tt <sup>i</sup>*

*ts k k*

*<sup>p</sup> X i <sup>k</sup>* the i-th row of

*<sup>N</sup> <sup>T</sup> ttss*

is used instead of <sup>1</sup> <sup>ˆ</sup>

can be viewed as measuring the effects of the noise

 

 12 22 .

initial existing noise

If we denote by

 1 <sup>1</sup> <sup>ˆ</sup> , 1 *<sup>N</sup> p p*

*N*

*N*

pseudoinverse ˆ

noise.

Obviously,

 1 2 

 <sup>1</sup> *XT i* 

Vlamos, 2004)

*k k*

1

to improve the quality of *F X*

hence 1 2 

 

and less information about

threshold filter to <sup>1</sup>

Input: The sample 2 2

degradation while the term <sup>1</sup> *X*

ˆ <sup>1</sup> 2 1 

**Step 1.** Compute the sample 1 1

*k*

*i X i ir*

$$\begin{aligned} M\_i &= \frac{1}{12+t} \begin{bmatrix} 1 & 2 & 1 \\ 2 & t & 2 \\ 1 & 2 & 1 \end{bmatrix} & t \ge 4 \qquad \text{to} \qquad \text{each} \qquad \text{component} \qquad \text{of} \qquad \left\{ X\_1^{(2)}, \ldots, X\_N^{(2)} \right\} \cup \\ X\_i^{(1)} &= F \left( X\_i^{(2)} \right) \,, 1 \le i \le N \ . \end{aligned}$$

**Step 2.** For each row 1 *i r* , compute 1 <sup>1</sup> <sup>ˆ</sup> , 1 , 1,2 *<sup>N</sup> p p k k i X i i rp <sup>N</sup>* , 1 <sup>1</sup> <sup>ˆ</sup> ˆ ˆ , , 1,2 <sup>1</sup> *<sup>N</sup> <sup>T</sup> ttss ts k k k i Xi iXi i t s N* .

**Step 3.** For each row 1 *i r* , compute <sup>1</sup> *T i* ˆ by applying a threshold filter to <sup>1</sup> ˆ *i* .

**Step 4.** Compute the rows *X i* of the restored image *X* , <sup>1</sup> *Xi T i* ˆ 2 1 12 22 ˆ ˆ *ii i i* ˆ ˆ , where is a noise-preventing constant conveniently determined to prevent the restoration of the initial noise. By experimental arguments, the recommended range of is 1.5,5.5 .

Note that since the regression function can be written as,

$$E\left(F\left(X^{(\eta)}\right)\middle|X^{(\eta)}\right) = \mu^{(1)} + \Sigma\_{12}\Sigma\_{22}^{-1}\left(X^{(\eta)} - \mu^{(2)}\right) = \Sigma\_{12}\Sigma\_{22}^{-1}\left(X^{(\eta)} - \mu^{(1)}\right) + \mu^{(1)} - \Sigma\_{12}\Sigma\_{22}^{-1}\left(\mu^{(2)} - \mu^{(1)}\right) \tag{12}$$

the correction term used at Step 4 is precisely the sample mean estimation of the <sup>1</sup> <sup>1</sup> *E X* 12 22 .

The idea of our attempt is to use the most informative features discriminating between *X* and *F X* for getting correction terms in restoring the filtered images *F X* . The attempt is justified by the argument that besides information about the removed noise, the most informative features discriminating between *X* and *F X* would contain appropriate information allowing quality improvement of the image *F X* (Cocianu, State & Vlamos, 2004). Let 2 2 <sup>1</sup> ,..., *X XN* be the sample of noise corrupted versions of the *r c* dimensional image X and 1 1 <sup>1</sup> ,..., *X XN* their filtered versions, 1 2 , 1 *X FX i N i i* . We assume 1 2 0.5 , therefore the scatter matrices become 1 2 ˆ ˆ *Sw* , <sup>1212</sup> ˆˆˆˆ *<sup>T</sup> Sb* and *SSS m wb* where

$$\hat{\boldsymbol{\mu}}^{(i)} = \frac{1}{N} \sum\_{k=1}^{N} \mathbf{X}\_{k}^{(i)} \; , \; \hat{\boldsymbol{\Sigma}}\_{i} = \frac{1}{N-1} \sum\_{k=1}^{N} \left( \mathbf{X}\_{k}^{(i)} - \hat{\boldsymbol{\mu}}^{(i)} \right) \left( \mathbf{X}\_{k}^{(i)} - \hat{\boldsymbol{\mu}}^{(i)} \right)^{T} \; , \; i = \mathbf{1}, \mathbf{2}$$

Since <sup>1</sup> *<sup>b</sup> rank S* , we get <sup>1</sup> <sup>1</sup> *w b rank S S* , that is when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* ,the matrix 1 *S S*2 1 has an unique positive eigenvalue, one of its unit eigenvectors being given by

Statistical-Based Approaches for Noise Removal 33

containing the information about class separability between 2 2

<sup>1</sup> ,..., *Xi Xi <sup>N</sup>* expressed in terms of the criterion function 1 *J* (see §2.4)

**Step 7.** Compute the row *X i* of the restored image *X* by correcting the filtered image

is a noise-preventing constant conveniently determined to prevent the restoration

*<sup>n</sup>* be the eigenvalue matrix of *Sm* and the matrix having as columns

<sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image X

<sup>1</sup> ,..., *X XN* as described in Step 1 of the variant (a) of the

ˆ using the most informative feature, *X i* = <sup>1</sup>

Note that at Step 4, the computation of *S i <sup>w</sup>* is carried out instead of <sup>1</sup> *S i <sup>w</sup>*

modification being needed because the matrix *S i <sup>w</sup>* could happen to be ill-conditioned. b. The variant of the HSBA when *S S* <sup>2</sup> *<sup>m</sup>* and *S S* <sup>1</sup> *<sup>w</sup>* (Cocianu, State & Vlamos, 2004)

the corresponding unit eigenvectors. According to the algorithm of simultaneous diagonalization (Duda & Hart, 1973), the optimal linear feature extractor is given by

where is an orthogonal matrix whose columns are unit eigenvectors of

. The most informative features about the separability of the classes

<sup>1</sup> ,..., *X XN* and 1 1

The number of significant features is either pre-established or dynamically determined by

therefore the restoration can be performed by adding the correction term

.

The variant of HSBA when *S S* <sup>2</sup> *<sup>m</sup>* and *S S* <sup>1</sup> *<sup>w</sup>* can be described as follows.

<sup>1</sup> ,..., *X XN* .

ˆ by applying a threshold filter to <sup>1</sup>

and the Penrose pseudoinverse

*i*

 

<sup>1</sup> ,..., *Xi Xi <sup>N</sup>* and

 

 

, this

ˆ ,

*AY* to

1 1 2

*Si i*

*Si i*

<sup>1</sup> <sup>1</sup> 1 2 *w w*

 

ˆ *i* and the correction term

<sup>1</sup> *T i Yi*

<sup>1</sup> ,..., *X XN* are the entries of*Y A* <sup>1</sup>

 ˆ ,where

**Step 4.** Compute <sup>1212</sup> ˆˆˆˆ *<sup>T</sup>*

*S i <sup>w</sup>* of the matrix 1 2

1 1

of the initial noise, 0<

**Step 6.** Compute <sup>1</sup> *T i*

 <sup>1</sup> *<sup>T</sup> Y i* <sup>1</sup> ˆ *i*

 <sup>1</sup> *T i* 

> <sup>1</sup> ,...,

1 1 2 2 *T*

represented by the samples 2 2

the magnitude of the eigenvalues of <sup>1</sup> *S S*2 1

**Step 1.** Compute the sample 1 1

Input : The sample 2 2

HSBA

ˆ the filtered prototype of 1 1

Let *diag*

*A* <sup>2</sup> 

 <sup>1</sup> *T i* 

1

*K Sw* 

*Si i i i i <sup>b</sup>* 

> ˆ ˆ ˆ *Si i i <sup>w</sup>*

**Step 5.** Compute the optimal linear feature extractor

<1.

$$\Phi\_1 = \frac{S\_w^{-1} \left(\mu^{(1)} - \mu^{(2)}\right)}{\left\| S\_w^{-1} \left(\mu^{(1)} - \mu^{(2)}\right) \right\|} \cdot \mathsf{I}$$

The heuristic scatter matrices-based algorithms (HSBA) for image restoration (Cocianu, State & Vlamos, 2004)

The idea of our attempt is to use the most informative features discriminating between *X* and *F X* for getting correction terms in restoring the filtered images *F X* The attempt is justified by the argument that besides information about the removed noise, the most informative features discriminating between *X* and *F X* would contain appropriate information allowing quality improvement of the image *F X* (Cocianu, State & Vlamos, 2004). Let 2 2 <sup>1</sup> ,..., *X XN* be the sample of noise corrupted versions of the *r c* dimensional image X and 1 1 <sup>1</sup> ,..., *X XN* their filtered versions, 1 2 , 1 *X FX i N i i* . We assume 1 2 0.5 ,therefore the scatter matrices are

$$S\_w = \hat{\Sigma}\_1 + \hat{\Sigma}\_2 \ \ \ \ S\_b = \left(\hat{\mu}^{(1)} - \hat{\mu}^{(2)}\right) \left(\hat{\mu}^{(1)} - \hat{\mu}^{(2)}\right)^T \text{ and } \ \ S\_m = S\_w + S\_b$$

where

$$\hat{\boldsymbol{\mu}}^{(i)} = \frac{1}{N} \sum\_{k=1}^{N} X\_k^{(i)} \; , \; \hat{\Sigma}\_i = \frac{1}{N-1} \sum\_{k=1}^{N} \left( X\_k^{(i)} - \hat{\boldsymbol{\mu}}^{(i)} \right) \left( X\_k^{(i)} - \hat{\boldsymbol{\mu}}^{(i)} \right)^T \; ; \; i = 1, 2 \dots \hat{i}$$

Since <sup>1</sup> *<sup>b</sup> rank S* , we get <sup>1</sup> <sup>1</sup> *w b rank S S* , that is when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* ,the matrix 1 *S S*2 1 has an unique positive eigenvalue, its unit eigenvector being given by

$$\Phi\_1 = \frac{S\_w^{-1} \left(\mu^{(1)} - \mu^{(2)}\right)}{\left\| S\_w^{-1} \left(\mu^{(1)} - \mu^{(2)}\right) \right\|}.$$

a. The variant of the HSBA when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* (Cocianu, State & Vlamos, 2004) Input : The sample 2 2 <sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image X


$$\begin{aligned} \textbf{Step 3.} \quad \textbf{Compute} \quad & \hat{\boldsymbol{\mu}}^{(\boldsymbol{\rho})}(\boldsymbol{i}) = \frac{1}{N} \sum\_{k=1}^{N} X\_{k}^{(\boldsymbol{\rho})}(\boldsymbol{i}) \ \hat{\boldsymbol{\Sigma}}\_{\boldsymbol{\nu}}(\boldsymbol{i}) = \frac{1}{N-1} \sum\_{k=1}^{N} \left( X\_{k}^{(\boldsymbol{\rho})}(\boldsymbol{i}) - \hat{\boldsymbol{\mu}}^{(\boldsymbol{\rho})}(\boldsymbol{i}) \right) \left( X\_{k}^{(\boldsymbol{\rho})}(\boldsymbol{i}) - \hat{\boldsymbol{\mu}}^{(\boldsymbol{\rho})}(\boldsymbol{i}) \right)^{\top}, \\ & p = 1, 2 \end{aligned}$$

32 Image Restoration – Recent Advances and Applications

<sup>1</sup> <sup>1</sup> 1 2 *w w*

The heuristic scatter matrices-based algorithms (HSBA) for image restoration (Cocianu,

The idea of our attempt is to use the most informative features discriminating between

attempt is justified by the argument that besides information about the removed noise, the

*S S*

State & Vlamos, 2004)

& Vlamos, 2004). Let 2 2

1 2 , 1 *X FX i N i i* . We assume 1 2

most informative features discriminating between *X*

*r c* dimensional image X and 1 1

1 2

 1 <sup>1</sup> <sup>ˆ</sup> *<sup>N</sup> i i*

*N*

Input : The sample 2 2

HRBA.

*p* 1,2

**Step 1.** Compute the sample 1 1

**Step 3.** Compute

**Step 2.** For each row 1 *i r* , do Step 3 until Step 7

<sup>1</sup> <sup>ˆ</sup> *<sup>N</sup> p p*

1

*k k i Xi N*

*k k X*

appropriate information allowing quality improvement of the image *F X*

 

ˆ ˆ *Sw* , <sup>1212</sup> ˆˆˆˆ *<sup>T</sup> Sb* 

*N*

has an unique positive eigenvalue, its unit eigenvector being given by

 , 1 <sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

> *S S*

 

a. The variant of the HSBA when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* (Cocianu, State & Vlamos, 2004)

*i kk k*

Since <sup>1</sup> *<sup>b</sup> rank S* , we get <sup>1</sup> <sup>1</sup> *w b rank S S* , that is when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>b</sup>* ,the matrix 1 *S S*2 1

<sup>1</sup> <sup>1</sup> 1 2 *w w*

and *F X*

 *X* 

where

 

 

 

.

<sup>1</sup> ,..., *X XN* be the sample of noise corrupted versions of the

0.5 ,therefore the scatter matrices are

 

*<sup>N</sup> <sup>T</sup> iiii*

, *<sup>i</sup>* 1,2 .

<sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image X

 , 1

*k*

*N*

*p kk*

<sup>1</sup> ,..., *X XN* by applying the binomial filter as in Step 1 of

<sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

*i X i iX i i*

,

*<sup>N</sup> <sup>T</sup> pppp*

 

*X X*

 

 

 

1 1 2

and *F X*

<sup>1</sup> ,..., *X XN* their filtered versions,

and *SSS m wb*

would contain

(Cocianu, State

The

 

for getting correction terms in restoring the filtered images *F X*

1 1 2


Note that at Step 4, the computation of *S i <sup>w</sup>* is carried out instead of <sup>1</sup> *S i <sup>w</sup>* , this modification being needed because the matrix *S i <sup>w</sup>* could happen to be ill-conditioned.

b. The variant of the HSBA when *S S* <sup>2</sup> *<sup>m</sup>* and *S S* <sup>1</sup> *<sup>w</sup>* (Cocianu, State & Vlamos, 2004)

Let *diag*<sup>1</sup> ,...,*<sup>n</sup>* be the eigenvalue matrix of *Sm* and the matrix having as columns the corresponding unit eigenvectors. According to the algorithm of simultaneous diagonalization (Duda & Hart, 1973), the optimal linear feature extractor is given by 1 *A* <sup>2</sup> where is an orthogonal matrix whose columns are unit eigenvectors of 1 1 2 2 *T K Sw* . The most informative features about the separability of the classes represented by the samples 2 2 <sup>1</sup> ,..., *X XN* and 1 1 <sup>1</sup> ,..., *X XN* are the entries of*Y A* <sup>1</sup> ˆ , therefore the restoration can be performed by adding the correction term *AY* to

The number of significant features is either pre-established or dynamically determined by the magnitude of the eigenvalues of <sup>1</sup> *S S*2 1 .

<sup>1</sup> ,..., *X XN* .

The variant of HSBA when *S S* <sup>2</sup> *<sup>m</sup>* and *S S* <sup>1</sup> *<sup>w</sup>* can be described as follows.

ˆ the filtered prototype of 1 1

 <sup>1</sup> *T i* 

Input : The sample 2 2 <sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image X

**Step 1.** Compute the sample 1 1 <sup>1</sup> ,..., *X XN* as described in Step 1 of the variant (a) of the HSBA

Statistical-Based Approaches for Noise Removal 35

 , 1

*k*

<sup>=</sup>

 

2 1 1 2 2 1 ˆ ˆ <sup>1</sup> ˆ ˆ ˆ ˆ 8 2 *<sup>T</sup> i i*

> <sup>1</sup> ,...,

whose columns *i* are eigenvectors of <sup>1</sup>

1 1 *<sup>s</sup>*

*M i <sup>k</sup>* .

**Step 6.** Arrange the eigenvalues such that for any 1 *sjc* ,

*T T s j*

*N*

*p kk*

<sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

1

2 2 2 1 2 1 ˆ ˆ ˆ ˆ 1 1 ln 2 ln 2

*T*

**Step 8.** Compute the row *X i* of the restored image *X* by correcting the filtered image

The multiresolution support provides a suitable framework for noise filtering and image restoration by noise removal. Briefly, the idea is to determine a set of statistically significant wavelet coefficients from which the multiresolution support is extracted, that is the procedure is mainly based on an underlying statistical image model governing the whole

We extend the MNR algorithm to the algorithm GMNR to allow the noise removal in more general cases when the noise mean can be any real number, and compare the performances of the resulted method against the most frequently used restoration algorithms (MMSE and AMVR). Briefly, the MNR algorithm is described as follows (Stark, Murtagh & Bijaoui, 1995). The parameter k used in Step 2 controls the width of the confidence interval, its value

process. The multiresolution support is the basis of subsequent filtering process.

Input: The image *X*<sup>0</sup> , the number of resolution levels p and the heuristic threshold k.

ˆ by applying a threshold filter to <sup>1</sup>

*s j j*

ˆ

ˆ using the information contained by the selected features,

is a noise-preventing constant conveniently determined to prevent the restoration

*i X i iX i i*

*<sup>N</sup> <sup>T</sup> pppp*

 *i i*

*s j*

 *i i*

> ˆ *i*

*<sup>c</sup>* of the matrix <sup>1</sup>

2 1

, *<sup>p</sup>* 1,2

 

*i i*

1 2

ˆ ˆ

1 2 ln 2 ˆ ˆ

ˆ ˆ *i i* and a matrix *Ai*

<sup>2</sup> *AiA i i <sup>T</sup>* <sup>ˆ</sup> ,

 

*i i*

1 2

 

+

ˆ ˆ *i i* such that <sup>1</sup>

2 1

**Step 3.** Compute

**Step 4.** Compute

1 , 2 *<sup>i</sup>*

*i c* 1, .

**Step 7.** Compute <sup>1</sup> *T i*

 <sup>1</sup> *T i* 

**3.3 Wavelet-based denoising** 

being set to a value around 3.

of the initial noise, 0<

where 

 1

**Step 5.** Compute the eigenvalues

 *i i*

and select the feature matrix <sup>1</sup> ,...,

*X i* = <sup>1</sup> *T i MiYi* 

ˆ

<1.

and the correction term *<sup>T</sup> Yi M i* <sup>1</sup> *T i*

 

 *i i*

*k k i Xi N*

<sup>1</sup> <sup>ˆ</sup> *<sup>N</sup> p p*


In case we take *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>m</sup>* we obtain a variant of the HSBA similar to the variant (b). In our approach, for each row 1 *i r* of the processed images, the most informative features used in getting the correction term are determined using the matrix *M i* whose columns are eigenvectors of Sw*Sm* such that *<sup>T</sup> M <sup>w</sup> iM i S i* .

Our image restoration algorithm based on the Bhattacharyya distance can be described as follows.

The HBA for image restoration (Cocianu, State & Vlamos, 2004)

Input : The sample 2 2 <sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image *X* and the number *k* of desired features.


**Step 3.** Compute

34 Image Restoration – Recent Advances and Applications

 

*iii*

**Step 6.** Compute *i* a matrix whose columns are unit eigenvectors of *Ki*. The most

ˆ by applying a threshold filter to <sup>1</sup>

**Step 8.** Compute the row *X i* of the restored image *X* by correcting the filtered image

conveniently determined to prevent the restoration of the initial noise, 0<

In case we take *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>m</sup>* we obtain a variant of the HSBA similar to the variant (b). In our approach, for each row 1 *i r* of the processed images, the most informative features used in getting the correction term are determined using the matrix *M i* whose

 such that *<sup>T</sup> M <sup>w</sup> iM i S i* . Our image restoration algorithm based on the Bhattacharyya distance can be described as

c. The variant of the HSBA when *S S* <sup>2</sup> *<sup>w</sup>* and *S S* <sup>1</sup> *<sup>m</sup>* (Cocianu, State & Vlamos, 2004)

 , 1

*k*

ˆ ˆ *Si i i <sup>w</sup>* and <sup>1212</sup> ˆˆˆˆ *<sup>T</sup>*

*t*

*N*

<sup>1</sup> *i i* ,..., *<sup>n</sup>* of *S i <sup>m</sup>* . Select the largest *t* eigenvalues and let

informative feature vectors responsible for the class separability between

<sup>1</sup> ,..., *Xi Xi <sup>N</sup>* are the columns of

ˆ using the information contained by the selected

<sup>1</sup> ,..., *X XN* of noise corrupted versions of the *r c* -dimensional image

<sup>1</sup> ,..., *X XN* as described in Step 1 of the variant (a) of the

*p kk*

<sup>1</sup> <sup>ˆ</sup> ˆ ˆ <sup>1</sup>

*i X i iX i i*

*Si Si i i i i m w*

<sup>1</sup> *i i* ,..., *<sup>n</sup>* and the corresponding unit eigenvectors

,

*<sup>N</sup> <sup>T</sup> pppp*

ˆ *i* and the correction term

is a noise-preventing constant

<1.

 

**Step 2.** For each row 1 *i r* , do Step 3 until Step 8

<sup>1</sup> <sup>ˆ</sup> *<sup>N</sup> p p*

ˆ

 

1 1

<sup>1</sup> ,..., *Xi Xi <sup>N</sup>* and 1 1

*T t t Ki i i S i i tw t* .

1

*k k i Xi N*

2 2

<sup>1</sup> ,..., , <sup>1</sup> ,..., *<sup>t</sup>*

**Step 3.** Compute

**Step 4.** Compute 1 2

**Step 5.** Compute the eigenvalues

2 2

**Step 7.** Compute <sup>1</sup> *T i*

 <sup>1</sup> *T i* 

columns are eigenvectors of Sw*Sm*

Input : The sample 2 2

HSBA

*X* and the number *k* of desired features.

**Step 1.** Compute the sample 1 1

**Step 2.** For each row 1 *i r* , do Step 3 until Step 8

follows.

 1

ˆ

features, *X i* = <sup>1</sup> *T i AiYi* 

 

The HBA for image restoration (Cocianu, State & Vlamos, 2004)

ˆ , where

<sup>2</sup> *<sup>t</sup> <sup>A</sup> <sup>t</sup> i i ii* .

*Yi A i* <sup>1</sup> *T i*

 *t t i diag i i* 

*p* 1,2

$$\hat{\mu}^{(p)}\left(\mathbf{i}\right) = \frac{1}{N} \sum\_{k=1}^{N} X\_k^{(p)}\left(\mathbf{i}\right) \cdot \hat{\Sigma}\_p\left(\mathbf{i}\right) = \frac{1}{N-1} \sum\_{k=1}^{N} \left(X\_k^{(p)}\left(\mathbf{i}\right) - \hat{\mu}^{(p)}\left(\mathbf{i}\right)\right) \left(X\_k^{(p)}\left(\mathbf{i}\right) - \hat{\mu}^{(p)}\left(\mathbf{i}\right)\right)^T \cdot p = 1, \mathcal{D}$$

**Step 4.** Compute

$$\mu\left(\frac{1}{2},i\right) = \frac{1}{8} \left(\boldsymbol{\mu}^{(2)}\left(i\right) - \boldsymbol{\mu}^{(1)}\left(i\right)\right)^{\mathrm{T}} \left(\frac{\hat{\Sigma}\_{1}\left(i\right) + \hat{\Sigma}\_{2}\left(i\right)}{2}\right)^{-1} \left(\boldsymbol{\mu}^{(2)}\left(i\right) - \boldsymbol{\mu}^{(1)}\left(i\right)\right) \\ + \frac{1}{2} \ln \frac{\left|\frac{\hat{\Sigma}\_{1}\left(i\right) + \hat{\Sigma}\_{2}\left(i\right)}{2}\right|}{\sqrt{\hat{\Sigma}\_{1}\left(i\right) \left\|\hat{\Sigma}\_{2}\left(i\right)\right\|}}$$


$$\frac{\left[\Phi\_{\boldsymbol{s}}^{\mathrm{T}}\left(\boldsymbol{\hat{\mu}}^{\{\boldsymbol{2}\}}\left(\boldsymbol{i}\right)-\boldsymbol{\hat{\mu}}^{\{\boldsymbol{1}\}}\left(\boldsymbol{i}\right)\right)\right]^{2}}{1+\lambda\_{\boldsymbol{s}}}+\ln\left(2+\lambda\_{\boldsymbol{s}}+\frac{1}{\lambda\_{\boldsymbol{.}}}\right) \geq \frac{\left[\Phi\_{\boldsymbol{\rho}}^{\mathrm{T}}\left(\boldsymbol{\hat{\mu}}^{\{\boldsymbol{2}\}}\left(\boldsymbol{i}\right)-\boldsymbol{\hat{\mu}}^{\{\boldsymbol{1}\}}\left(\boldsymbol{i}\right)\right)\right]^{2}}{1+\lambda\_{\boldsymbol{\rho}}}+\ln\left(2+\lambda\_{\boldsymbol{\rho}}+\frac{1}{\lambda\_{\boldsymbol{\rho}}}\right)$$

and select the feature matrix <sup>1</sup> ,..., *T M i <sup>k</sup>* .


where is a noise-preventing constant conveniently determined to prevent the restoration of the initial noise, 0<<1.

#### **3.3 Wavelet-based denoising**

The multiresolution support provides a suitable framework for noise filtering and image restoration by noise removal. Briefly, the idea is to determine a set of statistically significant wavelet coefficients from which the multiresolution support is extracted, that is the procedure is mainly based on an underlying statistical image model governing the whole process. The multiresolution support is the basis of subsequent filtering process.

We extend the MNR algorithm to the algorithm GMNR to allow the noise removal in more general cases when the noise mean can be any real number, and compare the performances of the resulted method against the most frequently used restoration algorithms (MMSE and AMVR). Briefly, the MNR algorithm is described as follows (Stark, Murtagh & Bijaoui, 1995). The parameter k used in Step 2 controls the width of the confidence interval, its value being set to a value around 3.

Input: The image *X*<sup>0</sup> , the number of resolution levels p and the heuristic threshold k.

Statistical-Based Approaches for Noise Removal 37

<sup>0</sup>

*pp j j*

 .

*i c E*

 

1 *<sup>i</sup>*

**A Λ Φ** to Y and get the representation

*i*

*<sup>T</sup>* **A X μ** . Finally, using the equation AAT=Σ-1, we

*<sup>n</sup>* be the eigen values of **Σ** ,

2

*i* 

.

*<sup>n</sup>* **0 I** . Consequently, the

to derive accurate

to **Z** (Hyvarinen,

. If **Y X** <sup>0</sup> **μ η** ,

, ' 0 *m* and

<sup>1</sup> , ,, , , *<sup>i</sup> <sup>n</sup> <sup>p</sup>*

**Step 4.** Compute an approximation of the original image 0*I* using the multiresolution

computed at Step 3 is 0 *II E* ', where E'~ <sup>2</sup> *N m*', '

In the following, the data **X** is a collection of image representations modeled as a sample **X**0 coming from a multivariate wide sense stationary stochastic process of mean **μ** and covariance matrix **Σ** , each instance being affected by additively superimposed random noise. In general, the parameters **μ** and **Σ** can not be assumed as been known and they are estimated from data. The most frequently used model for the noise component **η** is also a wide sense multivariate stationary stochastic process of Gaussian type. Denoting by n the dimensionality of the image representations, the simplest noise model is the "white" model,

The data are preprocessed to get normalized and centered representations. The

**Φ** 1 2 ( , ,..., ) **ΦΦ Φ***n* an orthogonal matrix whose columns are unit eigen vectors of **Σ** ,

*TT T* **Z AY A X μ A η** . Using the assumptions concerning the noise,

In the following, we combine the above described estimation process with a compression/decompression scheme, in order to remove the noise in a feature space of less dimensionality. For given *m mn* , 1 , we denote by 1 2 , ,..., *<sup>m</sup>* **Φ ΦΦ Φ** *<sup>m</sup>* and

 

1 *T T* 2 

sign max 0, 2

the matrix having as columns the first m columns of and the

respectively.

**0 Λ** and consequently, the components of *<sup>T</sup>* **A η** are independent, each

*gy y y*

 

*<sup>n</sup>* the diagonal matrix whose entries are

*I x y c x y E x y x y x y*

1 1

filtering based on the statistically significant wavelet coefficients.

**3.4 A combined noise removal method based on PCA and shrinkage functions** 

that is **η η** *<sup>t</sup>* , 0, *t* , where for any t≥0, **η** t ~ <sup>2</sup> N ,

The aim is to process the data **X** using estimates of **μ** , **Σ** and <sup>2</sup>

preprocessing step is needed to enable the hypotheses that <sup>2</sup> 0 1

 *<sup>n</sup>* **I** . Let 1 2 ... 

By applying the shrinkage function <sup>2</sup>

of **Z**<sup>0</sup> = <sup>0</sup>

mathematical model for the noisy image versions is, **X X** <sup>0</sup> **η** .

We apply the linear transform of matrix

**<sup>X</sup> μ ΣAZ** of X0.

diagonal matrix having as entries the first m entries of

<sup>0</sup>

component being of Gaussian type.

2001), we get the estimate **Z**<sup>0</sup>

get the estimate 0 0 ˆ

**Λ***m m* diag , ,..., 1 2 

*i j*

*n*

Note that *I*

approximations of **X**<sup>0</sup> .

then <sup>2</sup> Cov , *<sup>T</sup>* **Y Y Σ**

and diag 1 2 , ,..., **Λ** 

*<sup>T</sup>* **A η** ~ 2 1 N ,

 2 2 *E* ' .

**Step 1.** Compute the sequence of image variants 1, *<sup>j</sup> j p <sup>X</sup>* using a discrete low-pass filter h and get the wavelet coefficients by applying the "À Trous" algorithm

$$X\_{\slash}(r,\mathcal{c}) = \sum\_{l} \sum\_{k} h(l,k) X\_{\slash -1} \left( r + 2^{l-1} l, \mathcal{c} + 2^{l-1} k \right) \quad \text{or} \; \mathcal{O}\_{\slash}(r,\mathcal{c}) = X\_{\slash -1}(r,\mathcal{c}) - X\_{\slash}(r,\mathcal{c}) \; \mathcal{O}\_{\slash}(r,\mathcal{c})$$

**Step 2.** Select the significant coefficients where *<sup>j</sup> r c*, is taken as being significant if , *j j rc k* , for *j p* 1,...,

**Step 3.** Use the filter g defined by 1, , , , 0, , *j j j j j j rc k g rc rc k* to compute the

$$\text{restored image, } \tilde{X}\left(r,\mathcal{c}\right) = X\_p\left(r,\mathcal{c}\right) + \sum\_{j=1}^p \mathbf{g}\left(\sigma\_{j\prime}\alpha\_j\left(r,\mathcal{c}\right)\right)\alpha\_j\left(r,\mathcal{c}\right) \dots$$

Output The restored image *X* .

In the following, the algorithm GMNR is an extension of the MNR algorithm aiming to get the multiresolution support set in case of arbitrary noise mean, and to use this support set for noise removal purposes (Cocianu, State, Stefanescu, Vlamos, 2004). Let us denote by X the original "clean" image, and ~ <sup>2</sup> *N m*, be the additively superimposed noise, that is the image to be processed is *Y X* . Using the two-dimensional filter , the sampled variants of X, Y and result by convoluting them with respectively,

$$\begin{aligned} \text{tr}\_{\mathbb{O}}\left(\mathbf{x},\mathbf{y}\right) &= \left\langle Y(l,\mathbf{c}), \phi(\mathbf{x}-l,\mathbf{y}-\mathbf{c})\right\rangle, & I\_{\mathbb{O}}\left(\mathbf{x},\mathbf{y}\right) &= \left\langle X(l,\mathbf{c}), \phi(\mathbf{x}-l,\mathbf{y}-\mathbf{c})\right\rangle, \\\\ E\_{\mathbf{o}}\left(\mathbf{x},\mathbf{y}\right) &= \left\langle \eta\left(l,\mathbf{c}\right), \phi\left(\mathbf{x}-l,\mathbf{y}-\mathbf{c}\right)\right\rangle, & \mathbf{c}\_{\mathbf{o}} &= I\_{\mathbf{o}} + E\_{\mathbf{o}}\left.\end{aligned}$$

The wavelet coefficients of 0 *c* computed by the algorithm "À Trous" are,

$$\alpha\_{\rangle}^{c\_0}(\mathbf{x}, y) = \alpha\_{\rangle}^{l\_0}(\mathbf{x}, y) + \alpha\_{\rangle}^{\mathbb{R}\_0}(\mathbf{x}, y) \text{, where } \frac{1}{2}\wp\left(\frac{\mathbf{x}}{2}\right) = \wp(\mathbf{x}) - \frac{1}{2}\wp\left(\frac{\mathbf{x}}{2}\right).$$

For any pixel *x*,*y* , we get *c x <sup>p</sup>* ,*y I x p p* , , *y E x y* , where p stands for the number of the resolution levels, and the image 0 *c* is

$$\log\_o(\mathbf{x}, \mathbf{y}) = I\_p(\mathbf{x}, \mathbf{y}) + E\_p(\mathbf{x}, \mathbf{y}) + \sum\_{j=1}^p \alpha\_j^{l\_0}(\mathbf{x}, \mathbf{y}) + \sum\_{j=1}^p \alpha\_j^{\tilde{c}\_0}(\mathbf{x}, \mathbf{y}) \tag{13}$$

The noise mean can be inhibited by applying the following "white-wall" type technique.


36 Image Restoration – Recent Advances and Applications

*j j*

*p jj j*

 

1 , , ,, , *p*

In the following, the algorithm GMNR is an extension of the MNR algorithm aiming to get the multiresolution support set in case of arbitrary noise mean, and to use this support set for noise removal purposes (Cocianu, State, Stefanescu, Vlamos, 2004). Let us denote by X

*j Xrc X rc g rc rc*

 ,

, , 0, ,

*g rc rc k*

. Using the two-dimensional filter

, *I xy X lc x ly c* <sup>0</sup> , ,, ,

, 00 0 *cIE* .

*xy xy* , where 1 1

,, , ,

For any pixel *x*,*y* , we get *c x <sup>p</sup>* ,*y I x p p* , , *y E x y* , where p stands for the number of

*c x* <sup>0</sup> ,*y* 0 0

The noise mean can be inhibited by applying the following "white-wall" type technique. **Step 1.** Get the set of images *<sup>i</sup> E* , 1 *i n* , by additively superimposing noise <sup>2</sup> *N m*,

*pp j j*

*j j* 

and get the wavelet coefficients by applying the "À Trous" algorithm

*<sup>X</sup>* using a discrete low-pass filter h

*jj j rc X rc X rc* , ,, <sup>1</sup> ,

*j j*

*rc k*

*j j*

be the additively superimposed noise, that is

,

 *x* .

, 1 *i n* , 1 *j p* , by applying the

respectively,

22 22 *x x*

 *x y x y*

(13)

1 1

*j j*

*p p I E* , the sampled

on

1, ,

 

 

*<sup>j</sup> r c*, is taken as being significant if

to compute the

**Step 1.** Compute the sequence of image variants 1, *<sup>j</sup> j p*

*j j l k*

, *j j*

Output The restored image *X* .

the original "clean" image, and

variants of X, Y and

the image to be processed is *Y X*

 <sup>0</sup> , *<sup>c</sup> j* 

the resolution levels, and the image 0 *c* is

a "white wall" image.

"À Trous" algorithm.

**Step 2.** Compute *<sup>j</sup> <sup>c</sup>* , *<sup>i</sup>*

**Step 3.** Get the image *I*

*rc k*

**Step 2.** Select the significant coefficients where

, for *j p* 1,...,

 1 1 <sup>1</sup> , , 2,2 *j j*

**Step 3.** Use the filter g defined by

*c xy Y lc x ly c* <sup>0</sup> , ,, , 

 *x y* 0 0 , , *I E j j*

restored image,

 ~ <sup>2</sup> *N m*,

result by convoluting them with

*E xy lc x ly c* <sup>0</sup> , ,, , 

The wavelet coefficients of 0 *c* computed by the algorithm "À Trous" are,

 

*I x y E x y*

*Ej* , and the coefficients <sup>0</sup> , *<sup>i</sup> <sup>c</sup> <sup>E</sup>*

by averaging the resulted versions,

 

*X rc h lk X r lc k* ,

$$\tilde{I}\left(\mathbf{x},\boldsymbol{y}\right) = \frac{1}{n} \sum\_{i=1}^{n} \left[ \boldsymbol{c}\_{\boldsymbol{\rho}}\left(\mathbf{x},\boldsymbol{y}\right) - \boldsymbol{E}\_{\boldsymbol{\rho}}^{(i)}\left(\mathbf{x},\boldsymbol{y}\right) + \sum\_{j=1}^{p} \left(\boldsymbol{o}\boldsymbol{o}\_{\boldsymbol{\rho}}^{\boldsymbol{\epsilon}\_{0}}\left(\mathbf{x},\boldsymbol{y}\right) - \boldsymbol{o}\boldsymbol{o}\_{\boldsymbol{\rho}}^{\boldsymbol{\epsilon}^{(i)}}\left(\mathbf{x},\boldsymbol{y}\right)\right) \right] \cdot \boldsymbol{I}$$

**Step 4.** Compute an approximation of the original image 0*I* using the multiresolution filtering based on the statistically significant wavelet coefficients.

Note that *I* computed at Step 3 is 0 *II E* ', where E'~ <sup>2</sup> *N m*', ' , ' 0 *m* and 2 2 *E* ' .

#### **3.4 A combined noise removal method based on PCA and shrinkage functions**

In the following, the data **X** is a collection of image representations modeled as a sample **X**0 coming from a multivariate wide sense stationary stochastic process of mean **μ** and covariance matrix **Σ** , each instance being affected by additively superimposed random noise. In general, the parameters **μ** and **Σ** can not be assumed as been known and they are estimated from data. The most frequently used model for the noise component **η** is also a wide sense multivariate stationary stochastic process of Gaussian type. Denoting by n the dimensionality of the image representations, the simplest noise model is the "white" model, that is **η η** *<sup>t</sup>* , 0, *t* , where for any t≥0, **η** t ~ <sup>2</sup> N , *<sup>n</sup>* **0 I** . Consequently, the mathematical model for the noisy image versions is, **X X** <sup>0</sup> **η** .

The aim is to process the data **X** using estimates of **μ** , **Σ** and <sup>2</sup> to derive accurate approximations of **X**<sup>0</sup> .

The data are preprocessed to get normalized and centered representations. The preprocessing step is needed to enable the hypotheses that <sup>2</sup> 0 1 . If **Y X** <sup>0</sup> **μ η** , then <sup>2</sup> Cov , *<sup>T</sup>* **Y Y Σ** *<sup>n</sup>* **I** . Let 1 2 ... *<sup>n</sup>* be the eigen values of **Σ** , **Φ** 1 2 ( , ,..., ) **ΦΦ Φ***n* an orthogonal matrix whose columns are unit eigen vectors of **Σ** , and diag 1 2 , ,..., **Λ** *<sup>n</sup>* the diagonal matrix whose entries are 2 1 *<sup>i</sup> i* .

We apply the linear transform of matrix 1 *T T* 2 **A Λ Φ** to Y and get the representation <sup>0</sup> *TT T* **Z AY A X μ A η** . Using the assumptions concerning the noise, *<sup>T</sup>* **A η** ~ 2 1 N , **0 Λ** and consequently, the components of *<sup>T</sup>* **A η** are independent, each component being of Gaussian type.

By applying the shrinkage function <sup>2</sup> sign max 0, 2 *i gy y y* to **Z** (Hyvarinen,

2001), we get the estimate **Z**<sup>0</sup> of **Z**<sup>0</sup> = <sup>0</sup> *<sup>T</sup>* **A X μ** . Finally, using the equation AAT=Σ-1, we get the estimate 0 0 ˆ **<sup>X</sup> μ ΣAZ** of X0.

In the following, we combine the above described estimation process with a compression/decompression scheme, in order to remove the noise in a feature space of less dimensionality. For given *m mn* , 1 , we denote by 1 2 , ,..., *<sup>m</sup>* **Φ ΦΦ Φ** *<sup>m</sup>* and **Λ***m m* diag , ,..., 1 2 the matrix having as columns the first m columns of and the diagonal matrix having as entries the first m entries of respectively.

Statistical-Based Approaches for Noise Removal 39

The computations carried out in the module M1 involve the stored parameters **μ Σ** *Nk Nk* 0 0 1, 1 0 0 1 1 , , **Λ Φ** *Nk Nk* and the noisy current image **X***N k* <sup>0</sup> to update the new values of the parameters 0 0 **μ Σ** *Nk Nk* , 0 0 , , **Λ Φ** *Nk Nk* . The new values of the parameters are fed into the module M2 and they are used to clean the input. The computation of the new values of

0 00

**ΣΣ Σ** 00 00 1 1 0

00 0 <sup>0</sup>

<sup>0</sup> 0 0 <sup>0</sup>

*<sup>T</sup> N k N k <sup>n</sup> Nk Nk j i N k N k i i Nk Nk j <sup>j</sup> i j j i* 

 

> <sup>0</sup> <sup>1</sup> ,.., *Nk Nk N k <sup>n</sup> diag*

**Λ**

**Y X** *Nk Nk Nk* <sup>000</sup> **μ**

0 0 <sup>0</sup>

 0 00 *T*

0 0 0 00

**X***Nk Nk Nk NkNk* **μ Σ A Z**

**Z AY** *Nk Nk Nk*

*Nk Nk N k*

**A Φ Λ**

 *i i i Nk i* **ψ Σψ**

 **ψ Σψ ψ ψ ψ**

*<sup>T</sup> Nk Nk Nk N k*

**μ μ X**

0 0

*Nk Nk*

1 1 *N k N k N k*

0 0

0 0

0 0

 0 0 <sup>0</sup> <sup>1</sup> ,..., *Nk Nk N k n* **Φψψ**

 

**X***N k* of each input **X***N k* <sup>0</sup> is obtained by applying the previously

1 2

 

0 1 1 1

1 1 1 1 1 1 1

1

1 *<sup>T</sup> Nk Nk Nk Nk N k*

**X μ X μ**

0 1

0 0 1 1 <sup>0</sup> 0

ˆ

Compute **Z***N k* <sup>0</sup> by applying the shrinkage function to **Z***N k* <sup>0</sup> .

ˆ

described shrinkage technique, as follows.

1 <sup>1</sup> *Nk Nk N k N k*

> 

*N k*

1

Compute 0 ˆ

Output 0 ˆ **X***N k*

*k k* 1

Endwhile

**X***N k* (M2)

the parameters is performed as,

The cleaned version 0

The noise removal process in the m-dimensional feature space applied to <sup>1</sup> 2 *<sup>T</sup> <sup>m</sup> <sup>F</sup> <sup>m</sup>* **Λ Φ Y** produces the cleaned version *F*<sup>0</sup> , and consequently the estimate of **X**<sup>0</sup> results by decompressing

$$F\_0 \llcorner \hat{\mathbf{X}}\_0 = \left( \left( \mathbf{A}\_m \right)^{-\frac{1}{2}} \left( \mathbf{O}^{\mu \nu} \right)^T \right)^\* F\_0 \llcorner \bot$$

The model-free version of CSPCA is a learning from data method that computes estimates of the first and second order statistics on the basis of a series of n-dimensional noisy images **X ,X ,..., X** 1 2 *<sup>N</sup>* ,... (State, Cocianu, Sararu, Vlamos, 2009). Also, the estimates of the eigen values and eigen vectors of the sample covariance matrix are obtained using first order approximations derived in terms of perturbation theory. The first and second order statistics are computed in a classical way, that is

$$\mathfrak{p}\_N = \frac{1}{N} \sum\_{i=1}^N \mathbf{X}\_i \quad \text{and} \ \mathbf{Z}\_N = \frac{1}{N-1} \sum\_{i=1}^N (\mathbf{X}\_i - \mathfrak{p}\_N) \left(\mathbf{X}\_i - \mathfrak{p}\_N\right)^T.$$

Using staightforward computation, the following recursive equations can be derived 1 1 1 1 1 *N NN N N N* **μ μ X** , 1 1 *NN N N* **ΣΣ Σ** 1 1 1 1 *T N NN N <sup>N</sup>* **X μ X μ** .

Denoting by **ΣΣ Σ** *NN N* <sup>1</sup> , in case the eigen values of **Σ***N* are pairwise distinct, using arguments of perturbation theory type, the recursive equations for the eigen values and eigen vectors can be also derived (State, Cocianu, Vlamos, Stefanescu, 2006)

$$\mathcal{Z}\_{i}^{N+1} = \mathcal{Z}\_{i}^{N} + \left(\mathbf{u}\boldsymbol{\mathfrak{y}}\_{i}^{N}\right)^{T} \boldsymbol{\Delta}\boldsymbol{\Sigma}\_{N}\boldsymbol{\mathfrak{y}}\boldsymbol{\upmu}\_{i}^{N}, \\ \boldsymbol{\mathfrak{y}}\boldsymbol{\upmu}\_{i}^{N+1} = \boldsymbol{\mathfrak{y}}\boldsymbol{\upmu}\_{i}^{N} + \sum\_{j=1 \atop j\neq i}^{n} \frac{\left(\boldsymbol{\upmu}\_{N}^{j}\right)^{T} \boldsymbol{\Delta}\boldsymbol{\Sigma}\_{N}\boldsymbol{\upmu}\_{i}^{N}}{\boldsymbol{\upmu}\_{i}^{N} - \boldsymbol{\upmu}\_{j}^{N}} \boldsymbol{\upmu}\_{j}^{N}\boldsymbol{\upmu}\_{i}^{N}.$$

Assume that the information is represented by , , , **μΣ Λ Φ** *NN N N* , and a new noisy image XN+1 is presented as input. Then the cleaned version 1 ˆ **X***N* of XN+1 is computed and supplied as output followed by the updating 1, 1 1 1 , , **μΣ Λ Φ** *NN N N* . The updated values of these parameters are fed into the restoration module and they will be used for the next test.

The restoring algorithm can be described as follows. Assuming that 1 2 *<sup>N</sup>*<sup>0</sup> **X ,X ,...,X** is the initial collection of noisy images, we evaluate 00 0 0 , , , **μΣ Λ Φ** *NN N N* . On the basis of these information, a number of M noisy images **X ,...,X** *N NM* 0 0 1 are next processed according to the following scheme.

*k* 1 While *k M* Get **X***N k* <sup>0</sup> Compute 0 0 **μ Σ** *Nk Nk* , 0 0 , , **Λ Φ** *Nk Nk* (M1) Compute 0 ˆ **X***N k* (M2)

Output 0 ˆ **X***N k*

*k k* 1

38 Image Restoration – Recent Advances and Applications

The noise removal process in the m-dimensional feature space applied to

*F*<sup>0</sup> ,

The model-free version of CSPCA is a learning from data method that computes estimates of the first and second order statistics on the basis of a series of n-dimensional noisy images **X ,X ,..., X** 1 2 *<sup>N</sup>* ,... (State, Cocianu, Sararu, Vlamos, 2009). Also, the estimates of the eigen values and eigen vectors of the sample covariance matrix are obtained using first order approximations derived in terms of perturbation theory. The first and second order statistics

**<sup>μ</sup> X** and

Using staightforward computation, the following recursive equations can be derived

Denoting by **ΣΣ Σ** *NN N* <sup>1</sup> , in case the eigen values of **Σ***N* are pairwise distinct, using arguments of perturbation theory type, the recursive equations for the eigen values and

Assume that the information is represented by , , , **μΣ Λ Φ** *NN N N* , and a new noisy image XN+1

output followed by the updating 1, 1 1 1 , , **μΣ Λ Φ** *NN N N* . The updated values of these parameters are fed into the restoration module and they will be used for the next test.

The restoring algorithm can be described as follows. Assuming that 1 2 *<sup>N</sup>*<sup>0</sup> **X ,X ,...,X** is the initial collection of noisy images, we evaluate 00 0 0 , , , **μΣ Λ Φ** *NN N N* . On the basis of these information, a number of M noisy images **X ,...,X** *N NM* 0 0 1 are next processed according to

ˆ

1

*NN N N*

eigen vectors can be also derived (State, Cocianu, Vlamos, Stefanescu, 2006)

**ψ Σψ** , <sup>1</sup>

1 1

1

*<sup>N</sup> iNiN N <sup>i</sup>* **<sup>Σ</sup> <sup>X</sup> <sup>μ</sup> <sup>X</sup> <sup>μ</sup>** .

**ΣΣ Σ** 1 1 1 1

1

 **ψ Σψ ψ ψ ψ** .

 

**Λ Φ Y** produces the cleaned version *F*<sup>0</sup> , and consequently the estimate of

1 <sup>2</sup> 0 0 ˆ *<sup>T</sup> <sup>m</sup>*

 **<sup>X</sup> Λ Φ** .

*<sup>m</sup> F*

*<sup>N</sup> <sup>T</sup>*

*N NN N <sup>N</sup>*

*<sup>T</sup> <sup>j</sup> <sup>N</sup> <sup>n</sup> N N N Ni N i i N N j <sup>j</sup> i j j i* 

**X***N* of XN+1 is computed and supplied as

**X μ X μ** .

*T*

 1 2 *<sup>T</sup> <sup>m</sup> <sup>F</sup> <sup>m</sup>* 

**X**<sup>0</sup> results by decompressing

are computed in a classical way, that is

1

1 1

 

Compute 0 0 **μ Σ** *Nk Nk* , 0 0 , , **Λ Φ** *Nk Nk* (M1)

*N N* **μ μ X** , 1

1 1 *N NN*

*N*

the following scheme.

*k* 1

While *k M*

Get **X***N k* <sup>0</sup>

1 1 *<sup>N</sup> N i N <sup>i</sup>*

<sup>1</sup> *<sup>T</sup> N NN N*

*i i i Ni*

is presented as input. Then the cleaned version 1

Endwhile

The computations carried out in the module M1 involve the stored parameters **μ Σ** *Nk Nk* 0 0 1, 1 0 0 1 1 , , **Λ Φ** *Nk Nk* and the noisy current image **X***N k* <sup>0</sup> to update the new values of the parameters 0 0 **μ Σ** *Nk Nk* , 0 0 , , **Λ Φ** *Nk Nk* . The new values of the parameters are fed into the module M2 and they are used to clean the input. The computation of the new values of the parameters is performed as,

$$\mathbf{p}\_{N\_0+k} = \frac{N\_0+k-1}{N\_0+k}\mathbf{p}\_{N\_0+k-1} + \frac{1}{N\_0+k}\mathbf{X}\_{N\_0+k}$$

$$\boldsymbol{\Sigma}\_{N\_0+k} = \mathbf{E}\_{N\_0+k-1} - \frac{1}{N\_0+k-1}\boldsymbol{\Sigma}\_{N\_0+k-1} + \frac{1}{N\_0+k}(\mathbf{X}\_{N\_0+k} - \mathbf{p}\_{N\_0+k-1})\left(\mathbf{X}\_{N\_0+k} - \mathbf{p}\_{N\_0+k-1}\right)^T$$

$$\boldsymbol{\Lambda}\_i^{N\_0+k} = \boldsymbol{\Lambda}\_i^{N\_0+k-1} + \left(\mathbf{u}\_i^{\mathbf{y}\_i^{N\_0+k-1}}\right)^T \boldsymbol{\Delta}\boldsymbol{\Sigma}\_{N\_0+k-1}\mathbf{u}\_i^{N\_0+k-1}$$

$$\mathbf{w}\_i^{N\_0+k} = \mathbf{u}\_i^{N\_0+k-1} + \sum\_{j=1}^n \frac{\left(\mathbf{w}\_j^{N\_0+k-1}\right)^T \boldsymbol{\Delta}\boldsymbol{\Sigma}\_{N\_0+k-1}\mathbf{u}\_j^{N\_0+k-1}}{\mathbf{z}\_i^{N\_0+k-1} - \mathbf{z}\_j^{N\_0+k-1}}\mathbf{u}\_j^{N\_0+k-1}$$

$$\mathbf{A}\_{N\_0+k} = \text{diag}\left\{\mathbf{z}\_1^{N\_0+k}, \dots, \mathbf{z}\_n^{N\_0+k}\right\}$$

$$\boldsymbol{\Theta}\_{N\_0+k} = \left(\mathbf{u}\_1^{N\_0+k}, \dots, \mathbf{u}\_n^{N\_0+k}\right)$$

The cleaned version 0 ˆ **X***N k* of each input **X***N k* <sup>0</sup> is obtained by applying the previously described shrinkage technique, as follows.

$$\begin{aligned} \mathbf{Y}\_{N\_0+k} &= \mathbf{X}\_{N\_0+k} - \mathbf{p}\_{N\_0+k} \\\\ \mathbf{A}\_{N\_0+k} &= \mathbf{O}\_{N\_0+k} \left(\mathbf{A}\_{N\_0+k}\right)^{-\frac{1}{2}} \\\\ \mathbf{Z}\_{N\_0+k} &= \left(\mathbf{A}\_{N\_0+k}\right)^{\top} \mathbf{Y}\_{N\_0+k} \end{aligned}$$

Compute **Z***N k* <sup>0</sup> by applying the shrinkage function to **Z***N k* <sup>0</sup> .

$$
\hat{\mathbf{X}}\_{\mathcal{N}\_0 \ast k} = \mathbf{p}\_{\mathcal{N}\_0 \ast k} + \mathbf{E}\_{\mathcal{N}\_0 \ast k} \mathbf{A}\_{\mathcal{N}\_0 \ast k} \tilde{\mathbf{Z}}\_{\mathcal{N}\_0 \ast k}
$$

Statistical-Based Approaches for Noise Removal 41

in Table 1, and Table 2. The aims of the comparative analysis were to establish quantitative indicators to express both the quality and efficiency of each algorithm. The values of the variances in modeling the noise in images processed by the NFPCA represent the maximum of the variances per pixel resulted from the decorrelation process. We denote by U(a,b) the

It seems that the AMVR algorithm proves better performances from the point of view of mean error per pixel in case of uniform distributed noise as well as in case of Gaussian type noise. Also, it seems that at least for 0-mean Gaussian distributed noise, the mask 2 *h* provides less

Several tests were performed to investigate the potential of the proposed CSPCA. The tests were performed on data represented by linearized monochrome images decomposed in blocks of size 8x8. The preprocessing step was included in order to get normalized, centered representations. Most of the tests were performed on samples of volume 20, the images of each sample sharing the same statistical properties. The proposed method proved good performance for cleaning noisy images keeping the computational complexity at a reasonable level. An example of noisy image and its cleaned version respectively are

mean error per pixel when the restoration is performed by the MNR algorithm.

*Restoration algorithm Type of noise Mean error/pixel*

N(40,200)

N(50,100)

Table 1. Comparative analysis on the performance of the proposed algorithms

MNR *h2* 9.53

MNR *h2* 11.74

*Restoration algorithm Type of noise Mean error/pixel* 

37.51

46.58

N(0,100)

N(0,200)

The tests performed on new sample of images pointed out good generalization capacities and robustness of CSPCA. The computational complexity of CSPCA method is less than the

11.6

14.16

MMSE U(30,80) 52.08 AMVR 10.94 MMSE U(40,70) 50.58 AMVR 8,07

AMVR 11.54 GMNR 14.65 NFPCA 12.65

AMVR 9.39 GMNR 12.23 NFPCA 10.67

Table 2. Comparative analysis on MNR

complexity of the ICA code shrinkage method.

, the Gaussian distribution of

uniform distribution on the interval [a,b] and by <sup>2</sup> *N*

mean  and variance <sup>2</sup>

presented in Figure 1.

MMSE

MMSE

MNR *h1*

MNR *h1*

.

Let us assume that the L gray levels of the initial image X are affected by noise of Gaussian type ~ *N*0, , and denote by an orthogonal *n n* matrix whose columns are unit eigen vectors of , where n is the dimension of the input space. If is known, the matrix can be computed by classical techniques, respectively in cases when is not known, the columns of can be learned adaptively by PCA networks (Rosca, State, Cocianu, 2008).

We denote by Y the resulted image, *Y X* . The images are represented by RxC matrices, they are processed by blocks of size *R C* <sup>1</sup> , 1 *C pC p C* , 2 . In the preprocessing step, using the matrix , the noise is removed by applying the MNR algorithm to the decorrelated blocks of ' *<sup>T</sup> Y Y* .

The restoration process of the image Y using the learned features is performed as follows (State, Cocianu, 2007)


$$Y\_{\;\;i,j}^{\prime\prime} = \text{MNR}\left(Y\_{\;\;i,j}^{\prime}\right) \equiv \Phi^T X\_{\;\;i,j} \;/\; \forall \; i = 1, \dots, R \;/\; \; j = 1, \dots, C\_1 \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;/\; \;$$

**Step 3.** Compute an approximation *X X* of the initial image X by applying the linear transform of matrix to Y", , , ,, " *<sup>T</sup> X Y XX <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup>* , 1,..., *i R* , 1 *j C* 1,...,

## **4. Conclusions and experimental comparative analysis on the performances of some noise removal and restoration algorithms**

In order to evaluate the performance of the proposed noise removal algorithms, a series of experiments were performed on different 256 gray level images. We compare the performance of our algorithm NFPCA against MMSE, AMVR, and GMNR. The implementation of the GMNR algorithm used the masks

$$h\_1 = \begin{pmatrix} 1 & 1 & 3 & 1 & 1 \\ \frac{1}{256} & \frac{1}{64} & \frac{3}{128} & \frac{1}{64} & \frac{1}{256} \\ \frac{1}{64} & \frac{1}{16} & \frac{3}{32} & \frac{1}{16} & \frac{1}{64} \\ \frac{3}{128} & \frac{3}{32} & \frac{9}{64} & \frac{3}{32} & \frac{3}{128} \\ \frac{1}{64} & \frac{1}{16} & \frac{3}{32} & \frac{1}{16} & \frac{1}{64} \\ \frac{1}{64} & \frac{1}{16} & \frac{3}{32} & \frac{1}{16} & \frac{1}{64} \\ \frac{1}{256} & \frac{1}{64} & \frac{3}{128} & \frac{1}{64} & \frac{1}{256} \end{pmatrix} \text{and } h\_2 = \begin{pmatrix} 1 & 1 & 1 \\ \frac{1}{20} & \frac{1}{10} & \frac{1}{20} \\ \frac{1}{10} & \frac{2}{5} & \frac{1}{10} \\ \frac{1}{20} & \frac{1}{10} & \frac{1}{20} \end{pmatrix}.$$

Some of the conclusions experimentally derived concerning the comparative analysis of the restoration algorithms presented in the paper against some similar techniques are presented 40 Image Restoration – Recent Advances and Applications

Let us assume that the L gray levels of the initial image X are affected by noise of Gaussian

they are processed by blocks of size *R C* <sup>1</sup> , 1 *C pC p C* , 2 . In the preprocessing step, using the matrix , the noise is removed by applying the MNR algorithm to the de-

The restoration process of the image Y using the learned features is performed as follows

**Step 1.** Compute the image Y' on the basis of the initial image by de-correlating the noise

1 2 , ,..., 

" ' , ,, *<sup>T</sup> Y MNR Y X <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup>* , 1,..., *i R* , 1 *j C* 1,..., ,

**Step 3.** Compute an approximation *X X* of the initial image X by applying the linear transform of matrix to Y", , , ,, " *<sup>T</sup> X Y XX <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup>* , 1,..., *i R* , 1 *j C* 1,...,

**4. Conclusions and experimental comparative analysis on the performances** 

In order to evaluate the performance of the proposed noise removal algorithms, a series of experiments were performed on different 256 gray level images. We compare the performance of our algorithm NFPCA against MMSE, AMVR, and GMNR. The

Some of the conclusions experimentally derived concerning the comparative analysis of the restoration algorithms presented in the paper against some similar techniques are presented

and 2

*h*

unit eigen vectors of the noise covariance matrix. Then ' *<sup>T</sup>*

, where

image Y" results by labeling each wavelet coefficient of each pixel.

 ~ *N*0, , and denote by an orthogonal *n n* matrix whose columns are unit eigen vectors of , where n is the dimension of the input space. If is known, the matrix can be computed by classical techniques, respectively in cases when is not known, the columns of can be learned adaptively by PCA networks (Rosca, State, Cocianu, 2008).

. The images are represented by RxC matrices,

, 1 *i R* , 1 1 *j C* , where is a matrix of

*<sup>n</sup>* are the eigen values of .

~ *N*0, because

 

.

 

from the image Y' using its multirezolution support. The

type 

We denote by Y the resulted image, *Y X*

component ,, , ' ' *T T Y YX ij ij ij*

**of some noise removal and restoration algorithms** 

implementation of the GMNR algorithm used the masks

 

1

*h*

 1 2 , ,..., *<sup>T</sup> <sup>n</sup> diag* 

correlated blocks of ' *<sup>T</sup> Y Y* .

**Step 2.** Remove the noise '

(State, Cocianu, 2007)

in Table 1, and Table 2. The aims of the comparative analysis were to establish quantitative indicators to express both the quality and efficiency of each algorithm. The values of the variances in modeling the noise in images processed by the NFPCA represent the maximum of the variances per pixel resulted from the decorrelation process. We denote by U(a,b) the uniform distribution on the interval [a,b] and by <sup>2</sup> *N* , the Gaussian distribution of mean and variance <sup>2</sup> .

It seems that the AMVR algorithm proves better performances from the point of view of mean error per pixel in case of uniform distributed noise as well as in case of Gaussian type noise. Also, it seems that at least for 0-mean Gaussian distributed noise, the mask 2 *h* provides less mean error per pixel when the restoration is performed by the MNR algorithm.

Several tests were performed to investigate the potential of the proposed CSPCA. The tests were performed on data represented by linearized monochrome images decomposed in blocks of size 8x8. The preprocessing step was included in order to get normalized, centered representations. Most of the tests were performed on samples of volume 20, the images of each sample sharing the same statistical properties. The proposed method proved good performance for cleaning noisy images keeping the computational complexity at a reasonable level. An example of noisy image and its cleaned version respectively are presented in Figure 1.


Table 1. Comparative analysis on the performance of the proposed algorithms


Table 2. Comparative analysis on MNR

The tests performed on new sample of images pointed out good generalization capacities and robustness of CSPCA. The computational complexity of CSPCA method is less than the complexity of the ICA code shrinkage method.

Statistical-Based Approaches for Noise Removal 43

The tests performed on new sample of images pointed out good generalization capacities and robustness of CSPCA. The computational complexity of CSPCA method is less than the complexity of the ICA code shrinkage method. The authors aim to extend the work from both, methodological and practical points of view. From methodological point of view, some refinements of the proposed procedures and their performances are going to be evaluated on standard large size image databases are in progress. From practical point of view, the

So far, the tests were performed on monochrome images only. Some efforts that are still in progress aim to adapt and extend the proposed methodology to colored images. Although the extension is not straightforward and some major modifications have to be done, the already obtained results encourage the hopes that efficient variants of these algorithms can

The tests on the proposed algorithms were performed on images of size 256x256 pixels, by splitting the images in blocks of smaller size, depending on the particular algorithm. For instance, in case of the algorithms MNR and GMNR, the images are processed pixel by pixel, and the computation of the wavelet coefficients by the "A Trous" algorithm is carried out using 3x3 and 5x5 masks. The tests performed on NFPCA, CSPCA, and the model free

Anderson, T.W. (1958) An Introduction to Multivariate Statistical Analysis, John Wiley &Sons Bacchelli, S., Papi S. (2006). Image denoising using principal component analysis in the

Balster, E. J., Zheng, Y. F., Ewing, R. L. (2003). Fast, Feature-Based Wavelet Shrinkage

Beheshti, S. and Dahleh, M.A. , (2003 ) *A new information-theoretic approach to signal denoising* 

Buades, A., Coll, B., and Morel, J.-M., (2005) *A non-local algorithm for image denoising, In:*IEEE

Chatterjee, C., Roychowdhury, V.P., Chong, E.K.P. (1998). On Relative Convergence

Chellappa, R. , Jinchi, H. (1985). A nonrecursive Filter for Edge Preserving Image Restoration In: Proc. Intl.Conf.on Acoustic, Speech and Signal Processing, Tampa 1985 Cocianu, C., State, L., Stefanescu, V., Vlamos, P. (2004) On the Efficiency of a Certain Class

Cocianu, C., State, L., Vlamos, P. (2002). On a Certain Class of Algorithms for Noise Removal in

Engineering Held in Cambridge, MA on 30 September-October 4

2005. Volume 2, Issue , 20-25 June 2005 Page(s): 60 - 65 vol. 2

wavelet domain. Journal of Computational and Applied Mathematics, *Volume 189,* 

Algorithm for Image Denoising. In: International Conference on Integration of Knowledge IntensiveMulti-Agent Systems. KIMAS '03: Modeling, Exploration, and

*and best basis selection In:* IEEE Trans. On *Signal Processing*, Volume: 53, Issue:10,Part

Computer Society Conference on Computer Vision and Pattern Recognition. CVPR

Properties of Principal Component Analysis Algorithms. In: IEEE Transaction on

of Noise Removal Algorithms in Solving Image Processing Tasks. In: Proceedings

Image Processing:A Comparative Study, In: Proceedings of the Third IEEE Conference on Information Technology ITCC-2002, Las Vegas, Nevada, USA, April 8-10, 2002

procedures are going to be extended in solving specifics GIS tasks.

be obtained for noise removal in case of colored images too.

version of CSPCA processed blocks of 8x8 pixels.

*Issues 1-2*, 1 May 2006, 606-62

Neural Networks, vol.9,no.2

of the ICINCO 2004, Setubal, Portugal

1, 2003: 3613- 3624

**5. References** 

A synthesis of the comparative analysis on the quality and efficiency corresponding to the restoration algorithms presented in section 3.2 is supplied in Table 3.

So far, the tests were performed on monochrome images only. Some efforts that are still in progress aim to adapt and extend the proposed methodology to colored images. Although the extension is not straightforward and some major modifications have to be done, the already obtained results encourage the hopes that efficient variants of these algorithms can be obtained for noise removal in case of colored images too.

The tests on the proposed algorithms were performed on images of size 256x256 pixels, by splitting the images in blocks of smaller size, depending on the particular algorithm. For instance, in case of algorithms MNR and GMNR, the images are processed pixel by pixel, and the computation of the wavelet coefficients by the "A Trous" algorithm is carried out using 3x3 and 5x5 masks. The tests performed on NFPCA, CSPCA, and the model free version of CSPCA processed blocks of 8x8 pixels.


Table 3. Comparative analysis on the performance of the proposed algorithms

The initial noisy image The cleaned version of the initial image

Fig. 1. The performance of model-free version of CSPCA

The comparison of the proposed algorithm NFPCA and the currently used approaches MMSE and AMVR points out better results of NFPCA in terms of the mean error per pixel. Some of the conclusions are summarized in Table 1 and Table 2, where the noise was modeled using the uniform and normal distributions. As it is shown in Table 1, in case of the AMVR algorithm the mean error per pixel is slightly less than in case of using NFPCA, but the AMVR algorithm induces some blur effect in the image while the use of the NFPCA seems to assure reasonable small errors without inducing any annoying side effects.

The tests performed on new sample of images pointed out good generalization capacities and robustness of CSPCA. The computational complexity of CSPCA method is less than the complexity of the ICA code shrinkage method. The authors aim to extend the work from both, methodological and practical points of view. From methodological point of view, some refinements of the proposed procedures and their performances are going to be evaluated on standard large size image databases are in progress. From practical point of view, the procedures are going to be extended in solving specifics GIS tasks.

So far, the tests were performed on monochrome images only. Some efforts that are still in progress aim to adapt and extend the proposed methodology to colored images. Although the extension is not straightforward and some major modifications have to be done, the already obtained results encourage the hopes that efficient variants of these algorithms can be obtained for noise removal in case of colored images too.

The tests on the proposed algorithms were performed on images of size 256x256 pixels, by splitting the images in blocks of smaller size, depending on the particular algorithm. For instance, in case of the algorithms MNR and GMNR, the images are processed pixel by pixel, and the computation of the wavelet coefficients by the "A Trous" algorithm is carried out using 3x3 and 5x5 masks. The tests performed on NFPCA, CSPCA, and the model free version of CSPCA processed blocks of 8x8 pixels.

## **5. References**

42 Image Restoration – Recent Advances and Applications

A synthesis of the comparative analysis on the quality and efficiency corresponding to the

So far, the tests were performed on monochrome images only. Some efforts that are still in progress aim to adapt and extend the proposed methodology to colored images. Although the extension is not straightforward and some major modifications have to be done, the already obtained results encourage the hopes that efficient variants of these algorithms can

The tests on the proposed algorithms were performed on images of size 256x256 pixels, by splitting the images in blocks of smaller size, depending on the particular algorithm. For instance, in case of algorithms MNR and GMNR, the images are processed pixel by pixel, and the computation of the wavelet coefficients by the "A Trous" algorithm is carried out using 3x3 and 5x5 masks. The tests performed on NFPCA, CSPCA, and the model free

The initial noisy image The cleaned version of the initial image

The comparison of the proposed algorithm NFPCA and the currently used approaches MMSE and AMVR points out better results of NFPCA in terms of the mean error per pixel. Some of the conclusions are summarized in Table 1 and Table 2, where the noise was modeled using the uniform and normal distributions. As it is shown in Table 1, in case of the AMVR algorithm the mean error per pixel is slightly less than in case of using NFPCA, but the AMVR algorithm induces some blur effect in the image while the use of the NFPCA

seems to assure reasonable small errors without inducing any annoying side effects.

*Mean error/pixel* Noise distributed N(50,200)

*Mean error/pixel* Noise distributed N(30,150)

Mean 9.422317 12.346784 HRBA 9.333114 11.747860 HSBA 9.022712 11.500245 HBA 9.370968 11.484837 Table 3. Comparative analysis on the performance of the proposed algorithms

restoration algorithms presented in section 3.2 is supplied in Table 3.

be obtained for noise removal in case of colored images too.

version of CSPCA processed blocks of 8x8 pixels.

Fig. 1. The performance of model-free version of CSPCA

*Restoration algorithm* 

Anderson, T.W. (1958) An Introduction to Multivariate Statistical Analysis, John Wiley &Sons


**0**

**3**

Yuri S. Popkov

*Russia*

*Institute for Systems Analysis*

**with Entropy Operator**

**Entropic Image Restoration as a Dynamic System**

Entropy and the classical variational principle of the statistical physics are the effective tools for modeling and solving a lot of applied problems. There are many definitions of "entropy" functions. The book by Kapur (1989) contains some of them. The classical definition of the *physical* entropy was introduced by L.Boltzmann Boltzmann (1871) and was developed for Fermi- and Einstein-statistics Landau & Livshitz (1964). Notion of entropy was introduced for para-statistics that have position between Fermi- and Einstein-statistics (Ohnuki & Kamefuchi

Variation principle of entropy maximization turned out very useful for information theory, the base of which connected with Shannon (Shannon (1948), Kullback & Leibler (1951)). This direction is developed in the book Popkov (1995), where introduced generalized information entropies by Fermi-Dirac and Bose-Einstein (entropy with parameters). Entropy maximization are applied to image reconstruction from projections Byrne (1993). A large number of applications of the entropy maximization principle is contained in Fang et al.

In these papers the entropy conditional maximization problems with linear constraints equalities were considered only. However there are many problems of entropy maximization with feasible set that is described by a system of inequalities and not only a linear one.

In this paper we design the models of the entropy image reconstruction from projections (EIRP) as the entropy linear (ELP) and quadratic maximization problems (EQP), where the feasible sets are described by the system of the equalities and inequalities of appropriate types

The regular procedure for design of multiplicative algorithms with *p*-active variables with respect to dual variables and to mixed (dual and primal) variables proposed for the problem solving. The choice of the active variables is implemented by feedback control with respect to

The problem of reconstruction of images of the objects distorted by noises and hidden from direct observation arises in the different fields. One of the trends in the solution of the problem is based on the tomographic investigation of an object, i.e., the construction of its layer-by-layer projections. The projections can be formed as external irradiation sources(X-ray, ultrasonic sources) and internal ones (positron emission) as also with the aid of

**1. Introduction**

(1982), Dorofeev et al. (2008)).

(1997), Maslov (2003).

(linear and quadratic one).

the current state of the iterative process.


## **Entropic Image Restoration as a Dynamic System with Entropy Operator**

Yuri S. Popkov *Institute for Systems Analysis Russia*

## **1. Introduction**

44 Image Restoration – Recent Advances and Applications

Cocianu, C., State, L., Vlamos, P. (2007). Principal Axes – Based Classification with

Diamantaras, K.I., Kung, S.Y. (1996). Principal Component Neural Networks: theory and

Duda, R.O., Hart,P.E. Pattern Classification and Scene Analysis,Wiley&Sons, 1973 Fukunaga K., Introduction to Statistical Pattern Recognition, Academic Press,Inc. 1990

Haykin, S. (1999) Neural Networks A Comprehensive Foundation, Prentice Hall, Inc. Hyvarinen, A., Hoyer, P., Oja, P. (1999). Image Denoising by Sparse Code Shrinkage,

Hyvarinen, A., Karhunen, J., Oja,E. (2001) Independent Component Analysis, John Wiley &Sons J. Karhunen, E. Oja. (1982). New Methods for Stochastic Approximations of Truncated

Rioul, O., Vetterli, M. (1991). Wavelets and signal processing. IEEE Signal Processing Mag.

Rosca, I., State, L., Cocianu, C. (2008) Learning Schemes in Using PCA Neural Networks for

Sanger, T.D. (1989). An Optimality Principle for Unsupervised Learning, Advances in Neural Information Systems, ed. D.S. Touretzky, Morgan Kaufmann Sonka, M., Hlavac, V. (1997). Image Processing, Analyses and Machine Vision, Chapman &

Stark, J.L., Murtagh, F., Bijaoui, A. (1995). Multiresolution Support Applied to Image

State L, Cocianu C, Panayiotis V., Attempts in Using Statistical Tools for Image Restoration Purposes, The Proceedings of SCI2001, Orlando, USA, July 22-25, 2001 State, L, Cocianu, C, Vlamos, P. (2008). A New Unsupervized Learning Scheme to Classify

State, L, Cocianu, C, Vlamos, P., Stefanescu, V. (2006) PCA-Based Data Mining Probabilistic

State, L., Cocianu, C., Săraru, C., Vlamos, P. (2009) New Approaches in Image Compression

State, L., Cocianu, C., Vlamos, P. (2007) The Use of Features Extracted from Noisy Samples for Iimage Restoration Purposes, Informatica Economică, Nr. 41/2007 State, L., Cocianu, C., Vlamos, P., Stefanescu, V.(2005) Noise Removal Algorithm Based on

Systemics, Cybernetics and Informatics (WMSCI 2005), Orlando, USA

Umbaugh, S. (1998). Computer Vision and Image Processing, Prentice Hall

Karhunen-Loeve Expansions, In: Proceedings 6th International Conference on

Image Restoration Purposes. WSEAS Transactions on Information Science and

Data of Relative Small Volume. Economic Computation and Economic Cybernetics

and Fuzzy Approaches with Applications in Pattern Recognition.In: Proceedings of

and Noise Removal, Proceedings of the First International Conference on Advances in Satellite and Space Communications, SPACOMM 2009, IEEE Computer Society

Code Shrinkage Technique, Proceedings of the 9th World Multiconference on

Gonzales, R., Woods, R. (2002) Digital Image Processing, Prentice Hall

Jain, A. K., Kasturi, R., Schnuck, B. G. (1995) Machine Vision, McGraw Hill

Pitas, I. (1993). Digital Image Processing Algorithms, Prentice Hall

866-876

8(4), 14-38

Hall Computing

ICSOFT 2006

Press, 2009

applications,. John Wiley &Sons

Pattern Recognition, Springer Verlag

Applications, Vol. 5, July 2008, 1149-1159,

Filtering and Restoration, Technical Report

Studies and Research, 2008, 109-120

*www.cis.hut.fi/projects/ica*,

Application in Gaussian Distributed Sample Recognition. Economic Computation and Economic Cybernetics Studies and Research, Vol. 41, No 1-2/2007, 159-166 Deng, G., Tay, D., Marusic, S. (2007). *A signal denoising algorithm based on overcomplete wavelet* 

*representations and Gaussian models.* Signal Processing *Volume 87, Issue 5 (May 2007),*

Entropy and the classical variational principle of the statistical physics are the effective tools for modeling and solving a lot of applied problems. There are many definitions of "entropy" functions. The book by Kapur (1989) contains some of them. The classical definition of the *physical* entropy was introduced by L.Boltzmann Boltzmann (1871) and was developed for Fermi- and Einstein-statistics Landau & Livshitz (1964). Notion of entropy was introduced for para-statistics that have position between Fermi- and Einstein-statistics (Ohnuki & Kamefuchi (1982), Dorofeev et al. (2008)).

Variation principle of entropy maximization turned out very useful for information theory, the base of which connected with Shannon (Shannon (1948), Kullback & Leibler (1951)). This direction is developed in the book Popkov (1995), where introduced generalized information entropies by Fermi-Dirac and Bose-Einstein (entropy with parameters). Entropy maximization are applied to image reconstruction from projections Byrne (1993). A large number of applications of the entropy maximization principle is contained in Fang et al. (1997), Maslov (2003).

In these papers the entropy conditional maximization problems with linear constraints equalities were considered only. However there are many problems of entropy maximization with feasible set that is described by a system of inequalities and not only a linear one.

In this paper we design the models of the entropy image reconstruction from projections (EIRP) as the entropy linear (ELP) and quadratic maximization problems (EQP), where the feasible sets are described by the system of the equalities and inequalities of appropriate types (linear and quadratic one).

The regular procedure for design of multiplicative algorithms with *p*-active variables with respect to dual variables and to mixed (dual and primal) variables proposed for the problem solving. The choice of the active variables is implemented by feedback control with respect to the current state of the iterative process.

The problem of reconstruction of images of the objects distorted by noises and hidden from direct observation arises in the different fields. One of the trends in the solution of the problem is based on the tomographic investigation of an object, i.e., the construction of its layer-by-layer projections. The projections can be formed as external irradiation sources(X-ray, ultrasonic sources) and internal ones (positron emission) as also with the aid of

On the each step *t* of the dynamic procedure *t-posterior image* is build up as a solution of the ELP or EQP, using the current *t-prior image* and the current projection. The (*t* − *s*),(*t* − *s* + 1),..., *t-posterior images* take part in formation of *(t+1)-prior image*. So the dynamic procedures

Entropic Image Restoration as a Dynamic System with Entropy Operator 47

The dynamic procedures are closed in the sense that at each stage for the current *t-prior image* and the *t-projection*, the entropy-optimal *t-posterior image* is built up, by which the *(t+1)-prior*

We consider a diverse structures of the dynamic procedures with feedback and investigate

It is shown that the proposed dynamic procedures of the EIRP represent the dynamic systems with entropy operator (DSEO). And *our third contribution* is an elements of the *qualitative analysis of the DSEO*. We consider the properties of the entropy operator (boundedness,

Consider a common diagram of monochrome tomographic investigation (fig. 1), where external beams of photons **S** irradiate the flat object in the direction *AB*. The object is monochromatic, and is described by the two-dimensional function of optical density *ψ*(*x*, *y*) in the system of Cartesian coordinates. Positive values of the density function are limited:

The intensity of irradiation (projection) *w* at the point *B* of the detector **D** (fig. 1) is related by

*l*∈*AB*

*<sup>ψ</sup>*(*x*, *<sup>y</sup>*)*dl*

− 

It is common to manipulate the digital representation of the density function *ψ*(*l*,*s*), (*l* = 1, *L*, *s* = 1, *S*). Introduce *i* = *S*(*l* − 1) + *s*, *i* = 1, *m*, *m* = *LS*, and martix Ψ = [*ψ*(*l*,*s*)| *l* =

where **L**(*ψ*¯) is the *h*-vector function, and **g** is the *h*-vector. We consider the quadratic

where *<sup>L</sup>* is the (*<sup>h</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with nonnegative elements *lki* <sup>≥</sup> 0; **<sup>Q</sup>**(*ψ*¯) is the *<sup>h</sup>*-vector of the

*<sup>w</sup>*(*B*) = exp

The tomographic procedure form some feasible sets for the vector *ψ*¯:

where *<sup>Q</sup><sup>k</sup>* is the symmetric (*<sup>m</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with elements *<sup>q</sup><sup>k</sup>*

0 < *a* ≤ *ψ*(*x*, *y*) ≤ *b* < 1. (2.1)

<sup>L</sup> <sup>=</sup> {*ψ*¯ : **<sup>L</sup>**(*ψ*¯) <sup>≤</sup> **<sup>g</sup>**}, (2.3)

**L**(*ψ*¯) = *L ψ*¯ + **Q**(*ψ*¯), (2.4)

**Q**(*ψ*¯) = *ψ*¯� *Q<sup>k</sup> ψ*¯, (2.5)

*ij* ≥ 0.

, (2.2)

their properties. The example of application of these procedures is presented.

**2. Mathematical model of the static EIRP procedure**

where the integration is realized along straight *AB*.

1, *<sup>L</sup>*, *<sup>s</sup>* <sup>=</sup> 1, *<sup>S</sup>*] as a vector *<sup>ψ</sup>*¯ <sup>=</sup> {*ψ*1,..., *<sup>ψ</sup>m*}.

approximation of the function *L*:

quadratic forms:

are procedures with feedback.

*image* is corrected.

Lipschitz constant).

the Radon transformation:

their combination (nuclear magnetic resonance) (Herman (1980), Dhawan (2003)). In Popkov (1997) is shown that a distribution of the absobed photons in slab maximizes the generalized entropy by Fermi-Dirac under the set of projections. A generalization consists in the inclusion of an additional parameters in entropy function, through which it is possible to take into account prior information on the object.

Our contribution to the theory and applications of the EIRP consists in three parts.

*The first contribution* is the general entropy models in the terms of entropy linear programming (ELP) or entropy quadratic programming (EQP) that underlie in the *static* procedures of computer tomography. At the beginning of the static procedure, it is occured an accumulation of a complete set of the projections by means of the external irradiation of the object. Then it is solved the ELP or EQP. As a result, for the prescribed prior image, we obtain an entropy-optimal restored image, which we will be called a *posterior image*.

It calls for a rather high irradiation intensity so as to afford a sufficient noise immunity of a reconstructed image. However, for some classes of tomographic investigations a high irradiation intensity is extremely undesirable.

Multiplicative procedures represent to the ELP and EQP solving. Apparently, the first general approach for synthesis of such procedures was proposed in (Dubov et al. (1983)). Simple multiplicative algorithm was applied to minimization of strictly convex functions on nonnegative orthant. Later, the multiplicative algorithms with respect to dual variables are used for solving conditional minimization and mathematical programming problems (Aliev et al. (1985), Popkov (1988), Popkov (1995a)). Also, the multiplicative algorithms are used for solving nonlinear equations (Popkov (1996)). The multiplicative procedures for finding nonnegative solutions of the minimization problems over nonnegative optant were proposed again in the paper (Iusem et al. (1996)).

Some types of the multiplicative algorithms are derived from approach based on the Bregman function and generalized projections with Shannon's entropy. In this case we obtain so-called *row-action algorithms*, iterations of which have a multiplicative form. The algorithms of this type was developed and modified (Herman (1982), Censor (1981), Censor (1987), Byrne (1996), Censor & Zenios (1997)).

It is necessary to note that in the most cited works the multiplicative algorithms are applied to the problems of entropy maximization with linear constraints equalities. We consider the ELP problem, where a feasible set is described by the system of the linear equalities and inequalities. The regular procedure for design of multiplicative algorithms with *p-active* variables is proposed for solving of this problem. On the basis of the procedure above we sinthesize the algorithms with respect to dual variables and to mixed (dual and primal) variables simultaneously. The choice of the active variables is implemented by feedback control with respect to the current state of the iterative process. Convergence study of the multiplicative algorithms is based on the continuous analogues of the algorithms and equivalence of the iterative sequences generated by the dual and mixed type algorithms. (Popkov (2006)).

*Our second contribution* is connected with a *basically another approach* to the EIRP. It is a *dynamic* procedure consisting in the sequential refinement in time of the image synthesized. The suggested procedures do not require a high irradiation intensity and display a high noise stability.

2 Will-be-set-by-IN-TECH

their combination (nuclear magnetic resonance) (Herman (1980), Dhawan (2003)). In Popkov (1997) is shown that a distribution of the absobed photons in slab maximizes the generalized entropy by Fermi-Dirac under the set of projections. A generalization consists in the inclusion of an additional parameters in entropy function, through which it is possible to take into

*The first contribution* is the general entropy models in the terms of entropy linear programming (ELP) or entropy quadratic programming (EQP) that underlie in the *static* procedures of computer tomography. At the beginning of the static procedure, it is occured an accumulation of a complete set of the projections by means of the external irradiation of the object. Then it is solved the ELP or EQP. As a result, for the prescribed prior image, we obtain an

It calls for a rather high irradiation intensity so as to afford a sufficient noise immunity of a reconstructed image. However, for some classes of tomographic investigations a high

Multiplicative procedures represent to the ELP and EQP solving. Apparently, the first general approach for synthesis of such procedures was proposed in (Dubov et al. (1983)). Simple multiplicative algorithm was applied to minimization of strictly convex functions on nonnegative orthant. Later, the multiplicative algorithms with respect to dual variables are used for solving conditional minimization and mathematical programming problems (Aliev et al. (1985), Popkov (1988), Popkov (1995a)). Also, the multiplicative algorithms are used for solving nonlinear equations (Popkov (1996)). The multiplicative procedures for finding nonnegative solutions of the minimization problems over nonnegative optant were proposed

Some types of the multiplicative algorithms are derived from approach based on the Bregman function and generalized projections with Shannon's entropy. In this case we obtain so-called *row-action algorithms*, iterations of which have a multiplicative form. The algorithms of this type was developed and modified (Herman (1982), Censor (1981), Censor (1987), Byrne (1996),

It is necessary to note that in the most cited works the multiplicative algorithms are applied to the problems of entropy maximization with linear constraints equalities. We consider the ELP problem, where a feasible set is described by the system of the linear equalities and inequalities. The regular procedure for design of multiplicative algorithms with *p-active* variables is proposed for solving of this problem. On the basis of the procedure above we sinthesize the algorithms with respect to dual variables and to mixed (dual and primal) variables simultaneously. The choice of the active variables is implemented by feedback control with respect to the current state of the iterative process. Convergence study of the multiplicative algorithms is based on the continuous analogues of the algorithms and equivalence of the iterative sequences generated by the dual and mixed type algorithms.

*Our second contribution* is connected with a *basically another approach* to the EIRP. It is a *dynamic* procedure consisting in the sequential refinement in time of the image synthesized. The suggested procedures do not require a high irradiation intensity and display a high noise

Our contribution to the theory and applications of the EIRP consists in three parts.

entropy-optimal restored image, which we will be called a *posterior image*.

account prior information on the object.

irradiation intensity is extremely undesirable.

again in the paper (Iusem et al. (1996)).

Censor & Zenios (1997)).

(Popkov (2006)).

stability.

On the each step *t* of the dynamic procedure *t-posterior image* is build up as a solution of the ELP or EQP, using the current *t-prior image* and the current projection. The (*t* − *s*),(*t* − *s* + 1),..., *t-posterior images* take part in formation of *(t+1)-prior image*. So the dynamic procedures are procedures with feedback.

The dynamic procedures are closed in the sense that at each stage for the current *t-prior image* and the *t-projection*, the entropy-optimal *t-posterior image* is built up, by which the *(t+1)-prior image* is corrected.

We consider a diverse structures of the dynamic procedures with feedback and investigate their properties. The example of application of these procedures is presented.

It is shown that the proposed dynamic procedures of the EIRP represent the dynamic systems with entropy operator (DSEO). And *our third contribution* is an elements of the *qualitative analysis of the DSEO*. We consider the properties of the entropy operator (boundedness, Lipschitz constant).

## **2. Mathematical model of the static EIRP procedure**

Consider a common diagram of monochrome tomographic investigation (fig. 1), where external beams of photons **S** irradiate the flat object in the direction *AB*. The object is monochromatic, and is described by the two-dimensional function of optical density *ψ*(*x*, *y*) in the system of Cartesian coordinates. Positive values of the density function are limited:

$$0 < a \le \psi(x, y) \le b < 1. \tag{2.1}$$

The intensity of irradiation (projection) *w* at the point *B* of the detector **D** (fig. 1) is related by the Radon transformation:

$$w(B) = \exp\left(-\int\_{l \in AB} \psi(x, y) dl\right),\tag{2.2}$$

where the integration is realized along straight *AB*.

It is common to manipulate the digital representation of the density function *ψ*(*l*,*s*), (*l* = 1, *L*, *s* = 1, *S*). Introduce *i* = *S*(*l* − 1) + *s*, *i* = 1, *m*, *m* = *LS*, and martix Ψ = [*ψ*(*l*,*s*)| *l* = 1, *<sup>L</sup>*, *<sup>s</sup>* <sup>=</sup> 1, *<sup>S</sup>*] as a vector *<sup>ψ</sup>*¯ <sup>=</sup> {*ψ*1,..., *<sup>ψ</sup>m*}.

The tomographic procedure form some feasible sets for the vector *ψ*¯:

$$\mathcal{L} = \{\bar{\psi} \colon \mathbf{L}(\bar{\psi}) \le \mathbf{g}\},\tag{2.3}$$

where **L**(*ψ*¯) is the *h*-vector function, and **g** is the *h*-vector. We consider the quadratic approximation of the function *L*:

$$\mathbf{L}(\boldsymbol{\uppsi}) = L\,\boldsymbol{\uppsi} + \mathbf{Q}(\boldsymbol{\uppsi}),\tag{2.4}$$

where *<sup>L</sup>* is the (*<sup>h</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with nonnegative elements *lki* <sup>≥</sup> 0; **<sup>Q</sup>**(*ψ*¯) is the *<sup>h</sup>*-vector of the quadratic forms:

$$\mathbf{Q}(\bar{\psi}) = \bar{\psi}' \mathcal{Q}^k \bar{\psi},\tag{2.5}$$

where *<sup>Q</sup><sup>k</sup>* is the symmetric (*<sup>m</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with elements *<sup>q</sup><sup>k</sup> ij* ≥ 0.

The class P of the density vectors is characterized by the following inequalities:

the possible set (*h*) the following inequality exists:

entropy by Fermi-Dirac):

absorption in the object);

where *e* = 2, 73.

designations:

on the feasible set construction.

where:

density vectors *<sup>ψ</sup>*¯ (2.9), that belong to the feasible set <sup>D</sup> (2.7, 2.8).

*<sup>H</sup>*(*ψ*¯ <sup>|</sup> **<sup>a</sup>**, **<sup>b</sup>**, **<sup>E</sup>**) = <sup>−</sup>[*ψ*¯ <sup>−</sup> **<sup>a</sup>**]


function (the generalized information entropy by Boltzmann):

Thus, the problem of the EIRP can be formulated in the next form:

**3. Statements and algorithms for the ELP and the EQP**

*<sup>H</sup>*(*ψ*¯ <sup>|</sup> **<sup>a</sup>**, **b, E**) <sup>⇒</sup> max


It is assumed that among dimensions of the density vectors (*m*), the projection vectors (*n*), and

Entropic Image Restoration as a Dynamic System with Entropy Operator 49

It is assumed that the feasible set is nonempty for the class (2.9), and there exists a set of the

We will use the variation principle of the EIRP Popkov (1997), according to which the realizable density vector (function) *ψ*¯ maximizes the entropy (the generalized information

� ln *<sup>ψ</sup>*¯ <sup>−</sup> **<sup>a</sup>**


If there is information about more or less "grey" object then we can use the next entropy

*<sup>H</sup>*(*ψ*¯ <sup>|</sup> **<sup>E</sup>**) = <sup>−</sup>*ψ*¯� ln *<sup>ψ</sup>*¯

where the feasible set <sup>D</sup>(*ψ*¯) is described by the expressions (2.7 - 2.8) and the class <sup>P</sup> is described by the inequalities (2.9). This problem is related to the EQP or the ELP depend

Transform the problem (2.13) to the general form, for that introduce the following

**<sup>u</sup>**˜ <sup>=</sup> **<sup>u</sup>** − {**a**� *<sup>F</sup><sup>r</sup>* **<sup>a</sup>**, *<sup>r</sup>* <sup>=</sup> 1, *<sup>n</sup>*} − *<sup>T</sup>***a**.

**<sup>x</sup>** <sup>=</sup> *<sup>ψ</sup>*¯ <sup>−</sup> **<sup>a</sup>**, **<sup>b</sup>**˜ <sup>=</sup> **<sup>b</sup>** <sup>−</sup> **<sup>a</sup>**,

*<sup>ψ</sup>*¯ , *<sup>ψ</sup>*¯ ∈ D(*ψ*¯)

**<sup>g</sup>**˜ <sup>=</sup> **<sup>g</sup>** − {**a**� *<sup>Q</sup><sup>k</sup>* **<sup>a</sup>**, *<sup>k</sup>* <sup>=</sup> 1, *<sup>h</sup>*} − *<sup>L</sup>***a**, (3.1)

**<sup>E</sup>** <sup>−</sup> [**<sup>b</sup>** <sup>−</sup> *<sup>ψ</sup>*¯]

**<sup>0</sup>** <sup>&</sup>lt; **<sup>a</sup>** <sup>≤</sup> *<sup>ψ</sup>*¯ <sup>≤</sup> **<sup>b</sup>** <sup>&</sup>lt; **<sup>1</sup>**. (2.9)

*m* > *n* + *h*. (2.10)

� ln[**<sup>b</sup>** <sup>−</sup> *<sup>ψ</sup>*¯], (2.11)

*<sup>e</sup>***E**, (2.12)

<sup>P</sup>, (2.13)

Fig. 1. Tomography scheme

Now it is returned to the projection function (2.2), and we use its quadratic approximation:

$$\mathbf{w}(B) \simeq T\,\boldsymbol{\uppsi} + \mathbf{F}(\boldsymbol{\uppsi}) = \mathbf{u},\tag{2.6}$$

where: **<sup>w</sup>**(*B*) = {*w*(*B*1),..., *<sup>w</sup>*(*Bn*)}, *<sup>T</sup>* is the (*<sup>n</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with elements *tki* <sup>≥</sup> 0; **<sup>F</sup>**(*ψ*¯) is the *<sup>n</sup>*-vector function with components *<sup>F</sup>rψ*¯) = *<sup>ψ</sup>*¯� *<sup>F</sup><sup>r</sup> <sup>ψ</sup>*¯, where *<sup>F</sup><sup>r</sup>* is the symmetric (*<sup>m</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with elements *f<sup>r</sup> ij* ≥ 0.

Any tomographic investigation occurs in the presence of noises. So the *n*-projections vector **<sup>u</sup>** is a random vector with independent components *un*, *<sup>n</sup>* <sup>=</sup> 1, *<sup>m</sup>*, <sup>M</sup>**<sup>u</sup>** <sup>=</sup> **<sup>u</sup>**<sup>0</sup> <sup>≥</sup> 0,M(**<sup>u</sup>** <sup>−</sup> **u**0)<sup>2</sup> = diag [*σ*2], where **u**<sup>0</sup> is the ideal projections vector (without noise), and *σ*<sup>2</sup> is the dispersion of the noise. It is assumed that the dispersions of the noise components are equal.

Thus, the feasible set <sup>D</sup>(*ψ*¯) is described the following expressions:


$$T\ddot{\psi} + \mathbf{F}(\ddot{\psi}) = \mathbf{u},\tag{2.7}$$


$$L\,\boldsymbol{\psi} + \mathbf{Q}(\boldsymbol{\psi}) \le \mathbf{g}.\tag{2.8}$$

The class P of the density vectors is characterized by the following inequalities:

$$\mathbf{0} < \mathbf{a} \le \overline{\psi} \le \mathbf{b} < \mathbf{1}.\tag{2.9}$$

It is assumed that among dimensions of the density vectors (*m*), the projection vectors (*n*), and the possible set (*h*) the following inequality exists:

$$m > n + h.\tag{2.10}$$

It is assumed that the feasible set is nonempty for the class (2.9), and there exists a set of the density vectors *<sup>ψ</sup>*¯ (2.9), that belong to the feasible set <sup>D</sup> (2.7, 2.8).

We will use the variation principle of the EIRP Popkov (1997), according to which the realizable density vector (function) *ψ*¯ maximizes the entropy (the generalized information entropy by Fermi-Dirac):

$$H(\bar{\psi} \mid \mathbf{a}, \mathbf{b}, \mathbf{E}) = -[\bar{\psi} - \mathbf{a}]' \ln \frac{\bar{\psi} - \mathbf{a}}{\mathbf{E}} - [\mathbf{b} - \bar{\psi}]' \ln[\mathbf{b} - \bar{\psi}],\tag{2.11}$$

where:

4 Will-be-set-by-IN-TECH

Now it is returned to the projection function (2.2), and we use its quadratic approximation:

where: **<sup>w</sup>**(*B*) = {*w*(*B*1),..., *<sup>w</sup>*(*Bn*)}, *<sup>T</sup>* is the (*<sup>n</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix with elements *tki* <sup>≥</sup> 0; **<sup>F</sup>**(*ψ*¯) is the *<sup>n</sup>*-vector function with components *<sup>F</sup>rψ*¯) = *<sup>ψ</sup>*¯� *<sup>F</sup><sup>r</sup> <sup>ψ</sup>*¯, where *<sup>F</sup><sup>r</sup>* is the symmetric (*<sup>m</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix

Any tomographic investigation occurs in the presence of noises. So the *n*-projections vector **<sup>u</sup>** is a random vector with independent components *un*, *<sup>n</sup>* <sup>=</sup> 1, *<sup>m</sup>*, <sup>M</sup>**<sup>u</sup>** <sup>=</sup> **<sup>u</sup>**<sup>0</sup> <sup>≥</sup> 0,M(**<sup>u</sup>** <sup>−</sup> **u**0)<sup>2</sup> = diag [*σ*2], where **u**<sup>0</sup> is the ideal projections vector (without noise), and *σ*<sup>2</sup> is the dispersion of the noise. It is assumed that the dispersions of the noise components are equal.

Thus, the feasible set <sup>D</sup>(*ψ*¯) is described the following expressions:

**<sup>w</sup>**(*B*) � *<sup>T</sup> <sup>ψ</sup>*¯ <sup>+</sup> **<sup>F</sup>**(*ψ*¯) = **<sup>u</sup>**, (2.6)

*Tψ*¯ + **F**(*ψ*¯) = **u**, (2.7)

*<sup>L</sup> <sup>ψ</sup>*¯ <sup>+</sup> **<sup>Q</sup>**(*ψ*¯) <sup>≤</sup> **<sup>g</sup>**. (2.8)

Fig. 1. Tomography scheme

*ij* ≥ 0.


with elements *f<sup>r</sup>*





If there is information about more or less "grey" object then we can use the next entropy function (the generalized information entropy by Boltzmann):

$$H(\bar{\psi} \mid \mathbf{E}) = -\bar{\psi}' \ln \frac{\bar{\psi}}{e\mathbf{E}'} \tag{2.12}$$

where *e* = 2, 73.

Thus, the problem of the EIRP can be formulated in the next form:

$$H(\tilde{\psi} \mid \mathbf{a}, \mathbf{b}, \mathbf{E}) \Rightarrow \max\_{\tilde{\psi}}, \qquad \tilde{\psi} \in \mathcal{D}(\tilde{\psi}) \bigcap \mathcal{P}\_{\prime} \tag{2.13}$$

where the feasible set <sup>D</sup>(*ψ*¯) is described by the expressions (2.7 - 2.8) and the class <sup>P</sup> is described by the inequalities (2.9). This problem is related to the EQP or the ELP depend on the feasible set construction.

#### **3. Statements and algorithms for the ELP and the EQP**

Transform the problem (2.13) to the general form, for that introduce the following designations:

$$\begin{aligned} \mathbf{x} &= \bar{\psi} - \mathbf{a}, \quad \tilde{\mathbf{b}} = \mathbf{b} - \mathbf{a}, \\ \tilde{\mathbf{g}} &= \mathbf{g} - \{ \mathbf{a}' \boldsymbol{Q}^k \mathbf{a}, k = \overline{1, h} \} - L\mathbf{a}, \\ \tilde{\mathbf{u}} &= \mathbf{u} - \{ \mathbf{a}' F^r \mathbf{a}, r = \overline{1, n} \} - T\mathbf{a}. \end{aligned} \tag{3.1}$$

From the optimality conditions (3.9) we have:

*xi*(**z**, *μ*¯) = *yi*(**z**) *di*(*μ*¯),

*bi* <sup>=</sup> �*bi*, *<sup>i</sup>* <sup>=</sup> 1, *<sup>m</sup>*.

*m* ∑ *i*=1

> *m* ∑ *i*=1

*μk*Γ*k*(**z**, *μ*¯) = 0, *μ<sup>k</sup>* ≥ 0, *k* = 1, *h*.

**2. Multiplicative algorithms with** (*p+q*)**-active variables** An active variables are vary at the *s*th iteration, and the remaining variables are not vary. We will consider multiplicative algorithms with respect to dual variables (**z**, *μ*¯) for solution of the system (3.16). At the each step of iteration it will be used *p* components of the vector **z**, and *q* components of the vector

The multiplicative algorithms with *p+q*-active variables can be represented in the following

**<sup>z</sup>**<sup>0</sup> <sup>≥</sup> 0, *<sup>μ</sup>*¯<sup>0</sup> <sup>≥</sup> 0;

*<sup>j</sup>* , *j* = 1, *n*, *j* �= *j*1(*s*),..., *jp*(*s*);

*<sup>t</sup>*, *t* = 1, *h*, *t* �= *t*1(*s*),..., *tq*(*s*);

, *μ*¯*<sup>s</sup>* ),

, *μ*¯*<sup>s</sup>* ),

*<sup>j</sup>*1(*s*)Θ*<sup>γ</sup> j*1(*s*) (**z***s* , *μ*¯*<sup>s</sup>* ),

*jp* (*s*)Θ*<sup>γ</sup> jp* (*s*) (**z***s* , *μ*¯*<sup>s</sup>* ),

*<sup>t</sup>*1(*s*)[<sup>1</sup> <sup>−</sup> *<sup>α</sup>*Γ*t*1(*s*)(**z***<sup>s</sup>*

*tq* (*s*)[<sup>1</sup> <sup>−</sup> *<sup>α</sup>*Γ*tq* (*s*)(**z***<sup>s</sup>*

*uj*

*n* ∏ *j*=1 *z tji j*

Entropic Image Restoration as a Dynamic System with Entropy Operator 51

The Lagrange multipliers *μ*¯ and the exponential Lagrange multipliers **z** = exp(*λ*¯) are defined

exp � *<sup>h</sup>* ∑ *k*=1

⎤ ⎦ −1

*tjiyi*(**z**)*di*(*μ*¯) = 1, *j* = 1, *n*,

*μklki*��−<sup>1</sup>

,

*lkiyi*(**z**)*di*(*μ*¯) ≥ 0, (3.16)

*p* + *q* ≤ *n* + *h*. (3.17)

··· ··· ··· ··· , (3.18)

··· ··· ··· ··· , (3.19)

, (3.15)

*yi*(**z**) = ˜ *bi* ⎡ ⎣1 + 1 *Ei*

*di*(*μ*¯) = ˜ *bi* � 1 + 1 *Ei*

˜

<sup>Θ</sup>*j*(**z**, *<sup>μ</sup>*¯) = <sup>1</sup>

Γ*k*(**z**, *μ*¯) = *gk* −

*μ*¯. The number of the active variables is valid to the next relation:

*zs*+<sup>1</sup> *<sup>j</sup>*1(*s*) <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*zs*+<sup>1</sup> *jp* (*s*) <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*zs*+<sup>1</sup> *<sup>j</sup>* <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*μs*+<sup>1</sup> *<sup>t</sup>*1(*s*) <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

*μs*+<sup>1</sup> *tq* (*s*) <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

*μs*+<sup>1</sup> *<sup>t</sup>* <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

by the next equations and inequalities:

form:

(a)*initial step*

(b)*iterative step*

Then the problem (2.13) takes a form:

$$H(\mathbf{x}, \tilde{\mathbf{b}}, \mathbf{E}) = -\mathbf{x}' \ln \frac{\mathbf{x}}{\mathbf{E}} - [\tilde{\mathbf{b}} - \mathbf{x}]' \ln[\tilde{\mathbf{b}} - \mathbf{x}] \Rightarrow \max,\tag{3.2}$$

under the following constraints:


$$T\mathbf{\tilde{x}} + \{\mathbf{x}'F'\mathbf{x}, r=\overline{1,n}\} = \mathbf{\tilde{u}},\tag{3.3}$$


$$L\mathbf{\tilde{x}} + \{\mathbf{x'}\,Q^k\mathbf{x}, k=\overline{1,h}\} \le \mathbf{\tilde{g}}\tag{3.4}$$

where

$$\begin{aligned} \tilde{T} &= T + A\_{F\prime\prime} & A\_F &= 2[\mathbf{a}^\prime F^r, r = \overline{1, n}], \\ \tilde{L} &= L + A\_{Q\prime} & A\_Q &= 2[\mathbf{a}^\prime Q^k, k = \overline{1, h}] \end{aligned} \tag{3.5}$$

Remark that the constraints (2.9) are absent in the problem (3.2, 3.4), as they are included to the goal function.

#### **3.1 The ELP problem**

**1. Optimality conditions.** The feasible set in the ELP problem is described by the next expressions:

$$T\mathbf{x} = \hat{\mathbf{u}}\_{\prime} \qquad L\mathbf{x} \le \hat{\mathbf{g}}\_{\prime} \tag{3.6}$$

where

$$
\hat{\mathbf{u}} = \mathbf{u} - T\mathbf{a}, \qquad \hat{\mathbf{g}} = \mathbf{g} - L\mathbf{a}.\tag{3.7}
$$

Consider the Lagrange function for the ELP (3.2, 3.6):

$$\mathbf{L}(\mathbf{x}, \vec{\lambda}, \vec{\mu}) = H(\mathbf{x}, \vec{\mathbf{b}}, \mathbf{E}) + [\mathbf{\hat{u}} - T\mathbf{x}]'\,\bar{\lambda} + [\mathbf{\hat{g}} - L\mathbf{x}]'\,\bar{\mu},\tag{3.8}$$

where *λ*¯ , *μ*¯ are the Lagrange multipliers for constraints-equalities and -inequalities (3.6) correspondingly. Assume that the Slater conditions are valid, i.e., there exists a vector **x**<sup>0</sup> such that *L***x**<sup>0</sup> < **g**ˆ, *T***x**<sup>0</sup> = **u**ˆ.

According to Polyak (1987) the following expressions give the necessary and sufficient conditions optimality of the triple (**x**, *λ*¯ , *μ*¯) for the problem (3.2 - 3.5):

$$
\nabla\_{\mathbf{x}} \mathbf{L} = \mathbf{0}, \quad \nabla\_{\mathbf{J}} \mathbf{L} = \mathbf{0}, \quad \nabla\_{\mathbf{J}} \mathbf{L} \ge \mathbf{0}, \tag{3.9}
$$

$$
\overline{\mu} \otimes \nabla\_{\mu} \mathcal{L} = 0, \qquad \overline{\mu} \ge 0,\tag{3.10}
$$

where ⊗ designates a coordinate-wise multiplication.

The following designations are used in these expressions:

$$\nabla\_{\mathbf{x}} \mathbf{L} = \frac{\partial H}{\partial \mathbf{x}} - T' \bar{\lambda} - L' \bar{\mu} \tag{3.11}$$

$$\nabla\_{\bar{\lambda}} \mathbf{L} = \mathbf{\hat{u}} - T \mathbf{x} \tag{3.12}$$

	- (3.14)

From the optimality conditions (3.9) we have:

6 Will-be-set-by-IN-TECH

**<sup>E</sup>** <sup>−</sup> [**b**˜ <sup>−</sup> **<sup>x</sup>**]

� ln[**b**˜ <sup>−</sup> **<sup>x</sup>**] <sup>⇒</sup> max, (3.2)

*<sup>T</sup>*˜**<sup>x</sup>** <sup>+</sup> {**x**� *<sup>F</sup><sup>r</sup>* **<sup>x</sup>**, *<sup>r</sup>* <sup>=</sup> 1, *<sup>n</sup>*} <sup>=</sup> **<sup>u</sup>**˜, (3.3)

**<sup>x</sup>** <sup>+</sup> {**x**� *<sup>Q</sup><sup>k</sup>* **<sup>x</sup>**, *<sup>k</sup>* <sup>=</sup> 1, *<sup>h</sup>*} ≤ **<sup>g</sup>**˜, (3.4)

, *r* = 1, *n*], *L*˜ = *L* + *AQ*, *AQ* = 2[ **a**� *Qk*, *k* = 1, *h*] (3.5)

*T***x** = **u**ˆ, *L***x** ≤ **g**ˆ, (3.6)

� *μ*¯, (3.8)

(3.14)

**u**ˆ = **u** − *T***a**, **g**ˆ = **g** − *L***a**. (3.7)

� *<sup>λ</sup>*¯ + [**g**<sup>ˆ</sup> <sup>−</sup> *<sup>L</sup>***x**]

∇**x**L = 0, ∇*λ*¯ L = 0, ∇*μ*¯L ≥ 0, (3.9)

*μ*¯ ⊗ ∇*μ*L = 0, *μ*¯ ≥ 0, (3.10)

∇*λ*¯ L = **u**ˆ − *T***x**, (3.12) ∇*μ*¯L = **g**ˆ − *L***x**, (3.13)

*<sup>∂</sup>***<sup>x</sup>** <sup>−</sup> *<sup>T</sup>*� *<sup>λ</sup>*¯ <sup>−</sup> *<sup>L</sup>*� *<sup>μ</sup>*¯, (3.11)

Then the problem (2.13) takes a form:

under the following constraints:



the goal function.

expressions:

where

**3.1 The ELP problem**

such that *L***x**<sup>0</sup> < **g**ˆ, *T***x**<sup>0</sup> = **u**ˆ.

where

*<sup>H</sup>*(**x**, **<sup>b</sup>**˜, **<sup>E</sup>**) = <sup>−</sup>**x**� ln **<sup>x</sup>**

*L*˜

Consider the Lagrange function for the ELP (3.2, 3.6):

where ⊗ designates a coordinate-wise multiplication. The following designations are used in these expressions:

*T*˜ = *T* + *AF*, , *AF* = 2[ **a**� *F<sup>r</sup>*

<sup>L</sup>(**x**, *<sup>λ</sup>*¯ , *<sup>μ</sup>*¯) = *<sup>H</sup>*(**x**, **<sup>b</sup>**˜, **<sup>E</sup>**)+[**u**<sup>ˆ</sup> <sup>−</sup> *<sup>T</sup>***x**]

<sup>∇</sup>**x**<sup>L</sup> <sup>=</sup> *<sup>∂</sup><sup>H</sup>*

conditions optimality of the triple (**x**, *λ*¯ , *μ*¯) for the problem (3.2 - 3.5):

Remark that the constraints (2.9) are absent in the problem (3.2, 3.4), as they are included to

**1. Optimality conditions.** The feasible set in the ELP problem is described by the next

where *λ*¯ , *μ*¯ are the Lagrange multipliers for constraints-equalities and -inequalities (3.6) correspondingly. Assume that the Slater conditions are valid, i.e., there exists a vector **x**<sup>0</sup>

According to Polyak (1987) the following expressions give the necessary and sufficient

$$\begin{aligned} \boldsymbol{x}\_{i}(\mathbf{z},\boldsymbol{\bar{\mu}}) &= y\_{i}(\mathbf{z})\,d\_{i}(\boldsymbol{\bar{\mu}}),\\ \boldsymbol{y}\_{i}(\mathbf{z}) &= \boldsymbol{\bar{b}}\_{i} \left[ 1 + \frac{1}{E\_{i}} \prod\_{j=1}^{n} \boldsymbol{z}\_{j}^{l\_{ji}} \right]^{-1},\\ \boldsymbol{d}\_{i}(\boldsymbol{\bar{\mu}}) &= \boldsymbol{\bar{b}}\_{i} \left[ 1 + \frac{1}{E\_{i}} \exp\left( \sum\_{k=1}^{h} \mu\_{k} l\_{ki} \right) \right]^{-1},\\ \boldsymbol{\bar{b}}\_{i} &= \sqrt{\boldsymbol{b}\_{i}}, \qquad i = \overline{1,m}. \end{aligned} \tag{3.15}$$

The Lagrange multipliers *μ*¯ and the exponential Lagrange multipliers **z** = exp(*λ*¯) are defined by the next equations and inequalities:

$$\begin{aligned} \Theta\_{j}(\mathbf{z},\bar{\boldsymbol{\mu}}) &= \frac{1}{\boldsymbol{\mu}\_{j}} \sum\_{i=1}^{m} t\_{ji} y\_{i}(\mathbf{z}) d\_{i}(\bar{\boldsymbol{\mu}}) = 1, \qquad j = \overline{1,n}, \\ \Gamma\_{k}(\mathbf{z},\bar{\boldsymbol{\mu}}) &= \mathcal{g}\_{k} - \sum\_{i=1}^{m} l\_{ki} y\_{i}(\mathbf{z}) d\_{i}(\bar{\boldsymbol{\mu}}) \ge 0, \\ \mu\_{k} \Gamma\_{k}(\mathbf{z},\bar{\boldsymbol{\mu}}) &= 0, \quad \mu\_{k} \ge 0, \qquad k = \overline{1,h}. \end{aligned} \tag{3.16}$$

**2. Multiplicative algorithms with** (*p+q*)**-active variables** An active variables are vary at the *s*th iteration, and the remaining variables are not vary. We will consider multiplicative algorithms with respect to dual variables (**z**, *μ*¯) for solution of the system (3.16). At the each step of iteration it will be used *p* components of the vector **z**, and *q* components of the vector *μ*¯. The number of the active variables is valid to the next relation:

$$p + q \le n + h.\tag{3.17}$$

The multiplicative algorithms with *p+q*-active variables can be represented in the following form:

(a)*initial step*

$$\mathbf{z}^0 \ge 0, \qquad \bar{\mu}^0 \ge 0;$$

(b)*iterative step*

$$\begin{aligned} z\_{j\_1(s)}^{s+1} &= z\_{j\_1(s)}^s \Theta\_{j\_1(s)}^\gamma(\mathbf{z}^s, \overline{\mu}^s), \\ \dots & \dots & \dots & \dots \\ z\_{j\_p(s)}^{s+1} &= z\_{j\_p(s)}^s \Theta\_{j\_p(s)}^\gamma(\mathbf{z}^s, \overline{\mu}^s), \\ z\_j^{s+1} &= z\_j^s, \quad j = \overline{1, n\_\prime}, \; j \neq j\_1(s), \dots, j\_p(s); \\ \mu\_{t\_1(s)}^{s+1} &= \mu\_{t\_1(s)}^s [1 - a \Gamma\_{t\_1(s)}(\mathbf{z}^s, \overline{\mu}^s), \\ &\dots & \dots & \dots \\ \mu\_{t\_q(s)}^{s+1} &= \mu\_{t\_q(s)}^s [1 - a \Gamma\_{t\_q(s)}(\mathbf{z}^s, \overline{\mu}^s), \\ \mu\_t^{s+1} &= \mu\_t^s, \quad t = \overline{1, h}, \; t \neq t\_1(s), \dots, t\_q(s); \end{aligned} \tag{3.19}$$

**2. Multiplicative algorithms of the mixed type with** (*p+q+w*)**-active variables.** We use *p* active prime **x** variables, *q* active dual variables **z** for the constraints-equalities, and *w* active

Entropic Image Restoration as a Dynamic System with Entropy Operator 53

**<sup>x</sup>**<sup>0</sup> <sup>≥</sup> 0, **<sup>z</sup>**<sup>0</sup> <sup>≥</sup> 0, *<sup>μ</sup>*¯<sup>0</sup> <sup>≥</sup> 0;

*<sup>j</sup>* , *j* = 1, *m*, *j* � *j*1(*s*),..., *jp*(*s*);

*<sup>t</sup>*, *t* = 1, *n*, *t* � *t*1(*s*),..., *tq*(*s*);

)],

)],

*<sup>k</sup>*, *k* = 1, *h*, *k* � *k*1(*s*),..., *kw*(*s*);

, *μ*¯*<sup>s</sup>*

, *μ*¯*<sup>s</sup>*

One of the possible rules is a choice with respect the maximum residual. In this case it is necessary to select *p* maximum residual *ϑi*<sup>1</sup> ,..., *ϑip* and *q* maximum residual *εk*<sup>1</sup> ,...,*εkq* for the each iterative step *s*. The numbers *i*1,..., *ip* and *k*1,..., *kq* belong to the intervals [1, *n*] and

Consider the step *s* and find the maximal residual *ϑi*<sup>1</sup> (**z***s*, *μ*¯*s*) among *ϑ*1(**z***s*, *μ*¯*s*),..., *ϑn*(**z***s*, *μ*¯*s*).

*<sup>ϑ</sup>i*1−1(**z***s*, *<sup>μ</sup>*¯*s*), *<sup>ϑ</sup>i*1+1(**z***s*, *<sup>μ</sup>*¯*s*)*ϑn*(**z***s*, *<sup>μ</sup>*¯*s*), and etc., until all *<sup>p</sup>* maximal residuals will be found.

Selection of the maximal residuals *<sup>ε</sup>k*<sup>1</sup> (**z***s*, *<sup>μ</sup>*¯*s*),...,*εkq* (**z***s*, *<sup>μ</sup>*¯*s*) is implemented similary.

··· ··· ··· ··· , (3.24)

··· ··· ··· ··· , (3.25)

··· ··· ··· ··· , (3.26)

)|, *i* = 1, *n*;

), *k* = 1, *h*. (3.27)

dual *μ*¯ variables for the the constraints-inequalities. The algorithm takes a form:

*<sup>j</sup>*1(*s*)A*<sup>β</sup> j*1(*s*) (**x***s* , **z***s* , *μ*¯*<sup>s</sup>* ),

*jp* (*s*)A*<sup>β</sup> jp* (*s*) (**x***s* , **z***s* , *μ*¯*<sup>s</sup>* ),

*<sup>t</sup>*1(*s*)B*<sup>γ</sup> t*1 (**x***s* ),

*tq* (*s*)B*<sup>γ</sup> tq* (**x**)*<sup>s</sup>* ,

*<sup>k</sup>*1(*s*)[<sup>1</sup> <sup>−</sup> *<sup>α</sup>*C*k*<sup>1</sup> (**x***<sup>s</sup>*

*kw*(*s*)[<sup>1</sup> <sup>−</sup> *<sup>α</sup>*C*kw* (**x***<sup>s</sup>*

) = <sup>|</sup><sup>1</sup> <sup>−</sup> <sup>Θ</sup>*i*(**z***<sup>s</sup>*

) = *μk*Γ*k*(**z***<sup>s</sup>*

..., *ϑn*(**z***s*, *μ*¯*s*), and find the maximal residual *ϑi*<sup>2</sup> (**z***s*, *μ*¯*s*) among *ϑ*1(**z***s*, *μ*¯*s*),...,

**3. Active variables.** To choice active variables we use feedback control with respect to the residuals on the each step of iteration. Consider the choosing rule of the active variables for

*xs*+<sup>1</sup> *<sup>j</sup>*1(*s*) <sup>=</sup> *<sup>x</sup><sup>s</sup>*

*xs*+<sup>1</sup> *jp* (*s*) <sup>=</sup> *<sup>x</sup><sup>s</sup>*

*xs*+<sup>1</sup> *<sup>j</sup>* <sup>=</sup> *<sup>x</sup><sup>s</sup>*

*zs*+<sup>1</sup> *<sup>t</sup>*1(*s*) <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*zs*+<sup>1</sup> *tq* (*s*) <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*zs*+<sup>1</sup> *<sup>t</sup>* <sup>=</sup> *<sup>z</sup><sup>s</sup>*

*μs*+<sup>1</sup> *<sup>k</sup>*1(*s*) <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

*μs*+<sup>1</sup> *kw*(*s*) <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

The parameters *β*, *γ*, *α* are the step coefficients.

[1, *h*] respectively.

*μs*+<sup>1</sup> *<sup>k</sup>* <sup>=</sup> *<sup>μ</sup><sup>s</sup>*

the ELP problem (3.2, 3.3, 3.4). Introduce the residuals

*ϑi*(**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>*

*εk*(**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>*

Exclude the residual *ϑi*<sup>1</sup> (**z***s*, *μ*¯*s*) from the set *ϑ*1(**z***s*, *μ*¯*s*),

(a)*initial step*

(b)*iterative step*

The parameters *γ*, *α* are the step coefficients. In Popkov (2006) the multiplicative algorithms in respect to the mixed type (prime and dual variables) are introduced, and the method of the convergence of these algorithms are proposed.

#### **3.2 The EQL problem**

**1. Optimality condition.** Consider the EQL problem (3.2 - 3.5), and introduce the Lagrange function:

$$\begin{split} \mathbf{L}(\mathbf{x}, \tilde{\lambda}, \tilde{\mu}) &= H(\mathbf{x}, \tilde{\mathbf{b}}, \mathbf{E}) + \tilde{\lambda}^{\prime} [\tilde{\mathbf{u}} - \tilde{\mathbf{T}} \mathbf{x} - \{\mathbf{x}^{\prime} F^{r} \mathbf{x}, r = \overline{1, n}\}] + \\ &+ \tilde{\mu}^{\prime} [\tilde{\mathbf{g}} - \tilde{\mathbf{L}} \mathbf{x} - \{\mathbf{x}^{\prime} Q^{k} \mathbf{x}, k = \overline{1, h}\}]. \end{split} \tag{3.20}$$

According to the optimality conditions (3.9, 3.10) we have:

$$\begin{aligned} \nabla\_{\mathbf{X}} \mathbf{L} &= \frac{\partial H}{\partial \mathbf{x}} - [T + \Phi(\mathbf{x})]' \bar{\lambda} - [L + \Pi(\mathbf{x})]' \bar{\mu} = 0, \\ \nabla\_{\bar{\lambda}} \mathbf{L} &= \bar{\mathbf{u}} - T \mathbf{x} - \{\mathbf{x}' F^{r} \mathbf{x}, r = \overline{1, n}\}, \\ \nabla\_{\bar{\mu}} \mathbf{L} &= \bar{\mathbf{g}} - L \mathbf{x} - \{\mathbf{x}' Q^{k} \mathbf{x}, k = \overline{1, n}\} \ge 0, \\ \bar{\mu} \otimes \nabla\_{\bar{\mu}} \mathbf{L} &= 0, \qquad \mathbf{x} \ge 0, \ \bar{\mu} \ge 0, \end{aligned} \tag{3.21}$$

where

$$\Phi(\mathbf{x}) = [\![\![\rho\_{ri}(\mathbf{x})\,]\,\!]\,\!] = \overline{\mathbf{1}\!\!/\!m}\,\!\!/ \, i = \overline{\mathbf{1}\!\!/\,m}\!\!/\!]\,\!\qquad \eta\_{ri}(\mathbf{x}) = 2\sum\_{j=1}^{m} x\_{j} f\_{ij}^{r}{}\_{j}$$

$$\Pi(\mathbf{x}) = [\![\pi\_{ki}(\mathbf{x})\,]\,\!\!/\:k = \overline{\mathbf{1}\!\!/\,h}\,\!\!/\,i = \overline{\mathbf{1}\!\!/\,m}\!\!\!/\,)\,\!\qquad \pi\_{kj}(\mathbf{x}) = 2\sum\_{j=1}^{m} x\_{j} q\_{ij}^{k}{}\_{j}$$

Transform these equations and inequalities to the conventional form in which all variables are nonnegative one:

$$\begin{aligned} \frac{A\_j(\mathbf{x}, \mathbf{z}, \bar{\mu}) \, E\_j}{x\_j[1 + A\_j(\mathbf{x}, \mathbf{z}, \bar{\mu}) \, E\_j]} &= \mathcal{A}\_j(\mathbf{x}, \mathbf{z}, \bar{\mu}) = 1, \qquad j = \overline{1, m}\_{\prime} \\ \frac{1}{i\hbar} \left( \sum\_{i=1}^m \tilde{\mathfrak{t}}\_{i\prime} \mathbf{x}\_i + \sum\_{i,l=1}^m \mathbf{x}\_i \mathbf{x}\_l \, f\_{\bar{\mu}l}^r \right) &= \mathcal{B}\_r(\mathbf{x}) = 1, \qquad r = \overline{1, n}\_{\prime} \\ \frac{\mu\_k}{\tilde{g}\_k} \left( \sum\_{i=1}^m \tilde{\mathfrak{l}}\_{k\bar{i}} \mathbf{x}\_i + \sum\_{i,l=1}^m \mathbf{x}\_i \mathbf{x}\_l \, q\_{\bar{i}l}^k \right) &= \mathcal{C}\_k(\mathbf{x}) = 0, \qquad k = \overline{1, h\_\prime} \\ \mathbf{x} \ge 0, \quad \mathbf{z} = \exp(-\bar{\lambda}) \ge 0, \quad \bar{\mu} \ge 0, \end{aligned}$$

where

$$\begin{split} A\_{\boldsymbol{f}}(\mathbf{x}, \mathbf{z}, \boldsymbol{\mu}) &= \prod\_{r=1}^{n} z\_{r}^{\tilde{t}\_{\boldsymbol{f}}} \prod\_{p=1}^{n} z\_{r}^{q\_{\boldsymbol{f}}\left(\mathbf{x}\right)} \times \\ &\times \exp\left(-\sum\_{k=1}^{h} \mu\_{k} \tilde{l}\_{k\boldsymbol{f}}\right) \exp\left(-\sum\_{k=1}^{h} \mu\_{k} \sum\_{l=1}^{m} \mathbf{x}\_{l} q\_{\boldsymbol{f}\boldsymbol{l}}^{k}\right). \end{split} \tag{3.23}$$

**2. Multiplicative algorithms of the mixed type with** (*p+q+w*)**-active variables.** We use *p* active prime **x** variables, *q* active dual variables **z** for the constraints-equalities, and *w* active dual *μ*¯ variables for the the constraints-inequalities. The algorithm takes a form:

(a)*initial step*

8 Will-be-set-by-IN-TECH

The parameters *γ*, *α* are the step coefficients. In Popkov (2006) the multiplicative algorithms in respect to the mixed type (prime and dual variables) are introduced, and the method of the

**1. Optimality condition.** Consider the EQL problem (3.2 - 3.5), and introduce the Lagrange

[ **<sup>g</sup>**˜ <sup>−</sup> *<sup>L</sup>*˜ **<sup>x</sup>** − {**x**� *<sup>Q</sup><sup>k</sup>* **<sup>x</sup>**, *<sup>k</sup>* <sup>=</sup> 1, *<sup>h</sup>*} ].

*<sup>λ</sup>*¯ <sup>−</sup> [ *<sup>L</sup>* <sup>+</sup> <sup>Π</sup>(**x**) ]�

<sup>∇</sup>*λ*¯ <sup>L</sup> <sup>=</sup> **<sup>u</sup>**˜ <sup>−</sup> *<sup>T</sup>***<sup>x</sup>** − {**x**� *<sup>F</sup><sup>r</sup>* **<sup>x</sup>**, *<sup>r</sup>* <sup>=</sup> 1, *<sup>n</sup>*}, (3.21)

*<sup>∂</sup>***<sup>x</sup>** <sup>−</sup> [ *<sup>T</sup>* <sup>+</sup> <sup>Φ</sup>(**x**) ]�

*μ*¯ ⊗ ∇*μ*¯L = 0, **x** ≥ 0, *μ*¯ ≥ 0,

<sup>∇</sup>*μ*¯L <sup>=</sup> **<sup>g</sup>**˜ <sup>−</sup> *<sup>L</sup>***<sup>x</sup>** − {**x**� *<sup>Q</sup><sup>k</sup>* **<sup>x</sup>**, *<sup>k</sup>* <sup>=</sup> 1, *<sup>n</sup>*} ≥ 0,

Φ(**x**)=[*ϕri*(**x**)|*r* = 1, *n*, *i* = 1, *m*], *ϕri*(**x**) = 2

Π(**x**)=[*πki*(**x**)| *k* = 1, *h*, *i* = 1, *m*], *πkj*(**x**) = 2

Transform these equations and inequalities to the conventional form in which all variables are

*xj*[<sup>1</sup> <sup>+</sup> *Aj*(**x**, **<sup>z</sup>**, *<sup>μ</sup>*¯) *Ej*] <sup>=</sup> <sup>A</sup>*j*(**x**, **<sup>z</sup>**, *<sup>μ</sup>*¯) = 1, *<sup>j</sup>* <sup>=</sup> 1, *<sup>m</sup>*,

*xi xl f<sup>r</sup> il* 

*xi xl q<sup>k</sup> il* 

[ **<sup>u</sup>**˜ <sup>−</sup> *<sup>T</sup>*˜**<sup>x</sup>** − {**x**� *<sup>F</sup><sup>r</sup>* **<sup>x</sup>**, *<sup>r</sup>* <sup>=</sup> 1, *<sup>n</sup>*}] + (3.20)

*μ*¯ = 0,

*m* ∑ *j*=1 *xj f<sup>r</sup> ij*,

*m* ∑ *j*=1 *xjq<sup>k</sup> ij*.

= B*r*(**x**) = 1, *r* = 1, *n*, (3.22)

. (3.23)

= C*k*(**x**) = 0, *k* = 1, *h*,

convergence of these algorithms are proposed.

L(**x**, *λ*¯ , *μ*¯) = *H*(**x**, **b**˜, **E**) + *λ*¯ �

+ *μ*¯�

According to the optimality conditions (3.9, 3.10) we have:

<sup>∇</sup>**x**<sup>L</sup> <sup>=</sup> *<sup>∂</sup><sup>H</sup>*

*Aj*(**x**, **z**, *μ*¯) *Ej*

˜*trixi* +

*n* ∏*r*=1 *z* ˜*trj r*

× exp

*m* ∑ *i*,*l*=1

*m* ∑ *i*,*l*=1

**<sup>x</sup>** <sup>≥</sup> 0, **<sup>z</sup>** <sup>=</sup> exp(−*λ*¯) <sup>≥</sup> 0, *<sup>μ</sup>*¯ <sup>≥</sup> 0,

 − *h* ∑ *k*=1 *μk* ˜ *lkj* exp − *h* ∑ *k*=1 *μk m* ∑ *l*=1 *xlq<sup>k</sup> jl* 

*n* ∏ *p*=1 *z ϕrj*(**x**) *r* ×

1 *u*˜*r*

*μk g*˜*k*

 *m* ∑ *i*=1

 *m* ∑ *i*=1 ˜ *lkixi* +

*Aj*(**x**, **z**, *μ*¯) =

**3.2 The EQL problem** 

function:

where

where

nonnegative one:

$$\mathbf{x}^0 \ge 0, \quad \mathbf{z}^0 \ge 0, \qquad \bar{\mu}^0 \ge 0;$$

(b)*iterative step*

$$\begin{aligned} &x\_{j\_1}^{s+1}(s) = x\_{j\_1(s)}^s \mathcal{A}\_{j\_1(s)}^{\beta}(\mathbf{x}^s, \mathbf{z}^s, \mathbf{\bar{\mu}}^s), \\ &\dots \quad \dots \dots \dots \dots \tag{3.24} \\ &x\_{j\_p(s)}^{s+1} = x\_{j\_p(s)}^s \mathcal{A}\_{j\_p(s)}^{\beta}(\mathbf{x}^s, \mathbf{z}^s, \mathbf{\bar{\mu}}^s), \\ &x\_j^{s+1} = x\_j^s, \quad j = \overline{1, m}, \; j \neq j\_1(s), \dots, j\_p(s); \\ &z^{s+1} = z\_{l(s)} \mathcal{B}\_{l}^{\gamma}(\mathbf{x}^s), \\ &\dots \\ &z^{s+1} = z\_{l\_0(s)} \mathcal{B}\_{l\_1}^{\gamma}(\mathbf{x}^s), \\ &z\_t^{s+1} = z\_{l\_t}^s, \; t = \overline{1, n}, \; t \neq t\_1(s), \dots, t\_q(s); \\ &\omega\_t^{s+1} = \mu\_{k\_1(s)}^s [1 - a \mathcal{C}\_{k\_1}(\mathbf{x}^s)], \\ &\mu\_{k\_0(s)}^{s+1} = \mu\_{k\_0(s)}^s [1 - a \mathcal{C}\_{k\_0}(\mathbf{x}^s)], \\ &\mu\_{k\_0(s)}^{s+1} = \mu\_{k\_0}^s, \; k = \overline{1, h}, \; k \neq k\_1(s), \dots, k\_w(s); \end{aligned} \tag{3.26}$$

The parameters *β*, *γ*, *α* are the step coefficients.

**3. Active variables.** To choice active variables we use feedback control with respect to the residuals on the each step of iteration. Consider the choosing rule of the active variables for the ELP problem (3.2, 3.3, 3.4). Introduce the residuals

$$\begin{aligned} \theta\_i(\mathbf{z}^s, \overline{\mu}^s) &= |1 - \Theta\_i(\mathbf{z}^s, \overline{\mu}^s)|, \qquad i = \overline{1, n};\\ \varepsilon\_k(\mathbf{z}^s, \overline{\mu}^s) &= \mu\_k \Gamma\_k(\mathbf{z}^s, \overline{\mu}^s), \qquad k = \overline{1, h}. \end{aligned} \tag{3.27}$$

One of the possible rules is a choice with respect the maximum residual. In this case it is necessary to select *p* maximum residual *ϑi*<sup>1</sup> ,..., *ϑip* and *q* maximum residual *εk*<sup>1</sup> ,...,*εkq* for the each iterative step *s*. The numbers *i*1,..., *ip* and *k*1,..., *kq* belong to the intervals [1, *n*] and [1, *h*] respectively.

Consider the step *s* and find the maximal residual *ϑi*<sup>1</sup> (**z***s*, *μ*¯*s*) among *ϑ*1(**z***s*, *μ*¯*s*),..., *ϑn*(**z***s*, *μ*¯*s*). Exclude the residual *ϑi*<sup>1</sup> (**z***s*, *μ*¯*s*) from the set *ϑ*1(**z***s*, *μ*¯*s*),

..., *ϑn*(**z***s*, *μ*¯*s*), and find the maximal residual *ϑi*<sup>2</sup> (**z***s*, *μ*¯*s*) among *ϑ*1(**z***s*, *μ*¯*s*),...,

*<sup>ϑ</sup>i*1−1(**z***s*, *<sup>μ</sup>*¯*s*), *<sup>ϑ</sup>i*1+1(**z***s*, *<sup>μ</sup>*¯*s*)*ϑn*(**z***s*, *<sup>μ</sup>*¯*s*), and etc., until all *<sup>p</sup>* maximal residuals will be found. Selection of the maximal residuals *<sup>ε</sup>k*<sup>1</sup> (**z***s*, *<sup>μ</sup>*¯*s*),...,*εkq* (**z***s*, *<sup>μ</sup>*¯*s*) is implemented similary.

**4. Dynamic EIRP procedure with feedback**

depends on the measured projections (2.8). On the stage *t* we have the *t-prior* image **E***<sup>t</sup>*

In the general case the procedure holds

<sup>L</sup>˜(**E***<sup>t</sup>*

where *�* is a small positive real number.

Represent the operator <sup>L</sup>˜ in the following form:

, *ψ*¯(*t*,∗)

**4.1 Structures of the dynamic EIRP procedures**

by the rule (4.1).

**E**(*t*+1):

*ψ*¯*t* ∗(**E***<sup>t</sup>* |**u***t*

The problem (2.7 - 2.9, 2.13) is solved and we have *t-posteriori image*

) = arg max *<sup>ψ</sup>*¯

**<sup>E</sup>**(*t*+1) <sup>=</sup> <sup>L</sup>˜(**E***<sup>t</sup>*

and *t,*(*t* − 1)*,...,* (*t* − *s*)*-posterior* images to the (*t* + 1)*-prior* image.

,..., *ψ*¯((*t*−*s*),∗)

Then the dynamic EIRP procedure (the discrete procedure) takes a form:

**<sup>E</sup>**(*t*+1) <sup>=</sup> **<sup>E</sup>***<sup>t</sup>* <sup>+</sup> *�* <sup>L</sup>(**E***<sup>t</sup>*

*dt* <sup>=</sup> <sup>L</sup>(**E***<sup>t</sup>*

dynamic EIRP that is described be the differential equation:

*d***E**(*t*)

The basic idea of the dynamic procedure lies in the sequential (stage by stage) obtaining of the projections and the solution of the sequence of the appropriate ELP or EQP. The feasible sets in these problems consist on two subsets. One of them describes the class of the possible density functions (2.7), and it is not depends from irradiation of the object. The other subset

Entropic Image Restoration as a Dynamic System with Entropy Operator 55

 *H*(*ψ*¯, **E***<sup>t</sup>*

On the next (*<sup>t</sup>* <sup>+</sup> <sup>1</sup>)*-stage* the prior image **<sup>E</sup>**(*t*+1) is formed on the basis of *<sup>t</sup>*,(*<sup>t</sup>* <sup>−</sup> <sup>1</sup>),...,(*<sup>t</sup>* <sup>−</sup> *s*)-posterior images *ψ*¯(*t*,∗), *ψ*¯((*t*−1),∗),..., *ψ*¯((*t*−*s*),∗). Each of posterior images are reconstructed

, *ψ*¯(*t*,∗)

where L is the feedback operator, which characterizes the transformation of the *t-prior* image,

) = **<sup>E</sup>***<sup>t</sup>* <sup>+</sup> *�* <sup>L</sup>(**E***<sup>t</sup>*

, *ψ*¯(*t*,∗)

Now let the variable *t* is continuous one. Then under *�* → 0 we will have the continuous

, *ψ*¯(*t*,∗)

The *t,*(*t* − 1)*,...,* (*t* − *s*)*-posterior* images are defined by the ELP or EQP problems, which represent the appropriate *entropy operators* (4.1). So the dynamic EIRP procedure (4.1, 4.2, (4.4)) represents the discrete dynamic system with entropy operator (the discrete DSEO) and its the continuous analog (4.5) represents the continuous dynamic system with entropy operator (the continuous DSEO). Some general properties of the DSEO will be described in the next section.

Let us consider some partial cases. One of them relates to the examination of a Markov version of the procedure when the information only on the *t*-posterior image is used to shape up

**<sup>E</sup>**(*t*+1) <sup>=</sup> <sup>L</sup>(**E***<sup>t</sup>*

, the projection's vector **u***<sup>t</sup>*

,..., *ψ*¯((*t*−*s*),∗)

, *ψ*¯(*t*,∗)

,..., *ψ*¯((*t*−*s*),∗)

,..., *ψ*¯((*t*−*s*),∗)

,..., *ψ*¯((*t*−*s*),∗)

)|*ψ*¯*<sup>t</sup>* ∈ D(**u***<sup>t</sup>*

)  , measured with noise ¯

. (4.1)

), (4.2)

), (4.3)

). (4.4)

). (4.5)

, *ψ*¯*t*,∗). (4.6)

*ξ*.

Now we represent the formalized procedure of selection. Introduce the following designations:

$$\frac{n}{p} = I + \delta, \qquad I = \left[\frac{n}{p}\right], \quad 0 \le \delta \le p - 1;$$

$$\frac{h}{q} = I + \omega, \qquad J = \left[\frac{h}{q}\right], \quad 0 \le \omega \le q - 1. \tag{3.28}$$

$$\varrho = s \left( \text{mod}\,(I+1) \right), \qquad \kappa = s \left( \text{mod}\,(J+1) \right). \tag{3.29}$$

Consider the index sets:

$$N = \{1, \ldots, n\} \qquad N\_{\mathbf{r}}(\mathbf{s}) = \{i\_1(\mathbf{s}), \ldots, i\_{\mathbf{r}}(\mathbf{s})\};$$

$$K = \{1, \ldots, h\} \qquad K\_{\overline{\mathbf{v}}}(\mathbf{s}) = \{k\_1(\mathbf{s}), \ldots, k\_{\overline{\mathbf{v}}}(\mathbf{s})\},\tag{3.30}$$

where

$$r = \begin{cases} [1, p]\_{\prime} & \text{if } \varrho < I; \\ [1, \delta]\_{\prime} & \text{if } \varrho = I; \\ 0, & \text{if } \delta = 0 \end{cases} \qquad v = \begin{cases} [1, q]\_{\prime} & \text{if } \kappa < f; \\ [1, \omega] & \text{if } \kappa = f; \\ 0, & \text{if } \omega = 0. \end{cases} \tag{3.31}$$

Introduce the following sets:

$$P\_{r-1}(s) = \left[\bigcup\_{l=1}^{q} N\_p(s-l)\right] \bigcup N\_{r-1}(s), \qquad G\_{r-1} = N \backslash P\_{r-1}(s).$$

$$Q\_{v-1}(s) = \left[\bigcup\_{l=1}^{k} K\_{\overline{v}}(s-l)\right] \bigcup K\_{v-1}(s), \qquad R\_{v-1} = K \backslash Q\_{v-1}(s). \tag{3.32}$$

The numbers *r* and *v* are determined by the equalities (3.31), and

$$N\_0(s) = K\_0(s) = P\_0(s) = G\_0(s) = Q\_0(s) = R\_0(s) = \bigcirc \quad \text{for all } s.$$

Now we define the rule of the (*p+q*)-maximal residual in the following form:

$$i\_{\vec{l}}(s) = \arg\max\_{\left[i \in \mathcal{G}\_{l-1}(s)\right]} \theta\_{\vec{l}}(\mathbf{z}^s, \vec{\mu}^s),$$

$$k\_l(s) = \arg\max\_{\left[k \in \mathcal{R}\_{l-1}(s)\right]} \varepsilon\_k(\mathbf{z}^s, \vec{\mu}^s). \tag{3.33}$$

According to this rule we have the chain of inequalities:

$$
\theta\_{\dot{i}\_{\mathcal{P}}}(\mathbf{z}^{s}, \bar{\mu}^{s}) < \theta\_{\dot{i}\_{\mathcal{P}-1}}(\mathbf{z}^{s}, \bar{\mu}^{s}) < \dots < \theta\_{\dot{i}\_{1}}(\mathbf{z}^{s}, \bar{\mu}^{s}),
$$

$$
\varepsilon\_{k\_{\mathcal{q}}}(\mathbf{z}^{s}, \bar{\mu}^{s}) < \varepsilon\_{k\_{\mathcal{q}-1}}(\mathbf{z}^{s}, \bar{\mu}^{s}) < \dots < \varepsilon\_{k\_{1}}(\mathbf{z}^{s}, \bar{\mu}^{s}). \tag{3.34}
$$

We can see that all dual variables are sequentially transformed to active ones during *I* + *J* + 2 iterations It is repeated with a period of *I* + *J* + 2.

#### **4. Dynamic EIRP procedure with feedback**

10 Will-be-set-by-IN-TECH

Now we represent the formalized procedure of selection. Introduce the following

� *n p* �

> � *h q* �

*N* = {1, . . . , *n*} *Nr*(*s*) = {*i*1(*s*),..., *ir*(*s*)};

, *v* =

, 0 ≤ *δ* ≤ *p* − 1;

*�* = *s*(mod (*I* + 1)), *κ* = *s*(mod (*J* + 1)). (3.29)

*K* = {1, . . . , *h*} *Kv*(*s*) = {*k*1(*s*),..., *kv*(*s*)}, (3.30)

[1, *q*], if *κ* < *J*;

[1, *ω*] if *κ* = *J*;

0, if *ω* = 0.

� *Nr*−1(*s*), *Gr*−<sup>1</sup> <sup>=</sup> *<sup>N</sup>* \ *Pr*−1(*s*).

� *Kv*−1(*s*), *Rv*−<sup>1</sup> <sup>=</sup> *<sup>K</sup>* \ *Qv*−1(*s*).

⎧ ⎪⎪⎪⎨

⎪⎪⎪⎩

*ϑi*(**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>* ),

*εk*(**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>*

) <sup>&</sup>lt; ··· <sup>&</sup>lt; *<sup>ϑ</sup>i*<sup>1</sup> (**z***<sup>s</sup>*

) <sup>&</sup>lt; ··· <sup>&</sup>lt; *<sup>ε</sup>k*<sup>1</sup> (**z***<sup>s</sup>*

, *μ*¯*<sup>s</sup>* ),

, *μ*¯*<sup>s</sup>*

, 0 ≤ *ω* ≤ *q* − 1. (3.28)

(3.31)

(3.32)

). (3.33)

). (3.34)

designations:

where

Consider the index sets:

Introduce the following sets:

*n*

*h*

*r* =

*Pr*−1(*s*) =

*Qv*−1(*s*) =

⎧ ⎪⎪⎪⎨

⎪⎪⎪⎩

� � *�*

� �*κ l*=1

According to this rule we have the chain of inequalities:

*<sup>ϑ</sup>ip* (**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>*

*<sup>ε</sup>kq* (**z***<sup>s</sup>* , *μ*¯*<sup>s</sup>*

iterations It is repeated with a period of *I* + *J* + 2.

*l*=1

*<sup>p</sup>* <sup>=</sup> *<sup>I</sup>* <sup>+</sup> *<sup>δ</sup>*, *<sup>I</sup>* <sup>=</sup>

*<sup>q</sup>* <sup>=</sup> *<sup>J</sup>* <sup>+</sup> *<sup>ω</sup>*, *<sup>J</sup>* <sup>=</sup>

[1, *p*], if *�* < *I*;

[1, *δ*], if *�* = *I*;

0, if *δ* = 0

*Np*(*s* − *l*)

*Kv*(*s* − *l*)

Now we define the rule of the (*p+q*)-maximal residual in the following form:

*ij*(*s*) = arg max [*i*∈*Gj*−<sup>1</sup>(*s*)]

*kl*(*s*) = arg max [*k*∈*Rl*−<sup>1</sup>(*s*)]

) <sup>&</sup>lt; *<sup>ϑ</sup>ip*−<sup>1</sup> (**z***<sup>s</sup>*

) <sup>&</sup>lt; *<sup>ε</sup>kq*−<sup>1</sup> (**z***<sup>s</sup>*

The numbers *r* and *v* are determined by the equalities (3.31), and

�

�

*N*0(*s*) = *K*0(*s*) = *P*0(*s*) = *G*0(*s*) = *Q*0(*s*) = *R*0(*s*) = ∅, for all *s*.

, *μ*¯*<sup>s</sup>*

, *μ*¯*<sup>s</sup>*

We can see that all dual variables are sequentially transformed to active ones during *I* + *J* + 2

The basic idea of the dynamic procedure lies in the sequential (stage by stage) obtaining of the projections and the solution of the sequence of the appropriate ELP or EQP. The feasible sets in these problems consist on two subsets. One of them describes the class of the possible density functions (2.7), and it is not depends from irradiation of the object. The other subset depends on the measured projections (2.8).

On the stage *t* we have the *t-prior* image **E***<sup>t</sup>* , the projection's vector **u***<sup>t</sup>* , measured with noise ¯ *ξ*. The problem (2.7 - 2.9, 2.13) is solved and we have *t-posteriori image*

$$\psi\_\*^t(\mathbf{E}^t|\mathbf{u}^t) = \arg\max\_{\tilde{\psi}} \left\{ H(\tilde{\psi}, \mathbf{E}^t) | \tilde{\psi}^t \in \mathcal{D}(\mathbf{u}^t) \right\}. \tag{4.1}$$

On the next (*<sup>t</sup>* <sup>+</sup> <sup>1</sup>)*-stage* the prior image **<sup>E</sup>**(*t*+1) is formed on the basis of *<sup>t</sup>*,(*<sup>t</sup>* <sup>−</sup> <sup>1</sup>),...,(*<sup>t</sup>* <sup>−</sup> *s*)-posterior images *ψ*¯(*t*,∗), *ψ*¯((*t*−1),∗),..., *ψ*¯((*t*−*s*),∗). Each of posterior images are reconstructed by the rule (4.1).

In the general case the procedure holds

$$\mathbf{E}^{(t+1)} = \tilde{\mathcal{L}}(\mathbf{E}^t, \bar{\boldsymbol{\psi}}^{(t, \ast)}, \dots, \bar{\boldsymbol{\psi}}^{((t-s), \ast)}),\tag{4.2}$$

where L is the feedback operator, which characterizes the transformation of the *t-prior* image, and *t,*(*t* − 1)*,...,* (*t* − *s*)*-posterior* images to the (*t* + 1)*-prior* image.

Represent the operator <sup>L</sup>˜ in the following form:

$$\vec{\mathcal{L}}(\mathbf{E}^t, \vec{\Psi}^{(t, \*)}, \dots, \vec{\Psi}^{((t-s), \*)}) = \mathbf{E}^t + \varepsilon \, \mathcal{L}(\mathbf{E}^t, \vec{\Psi}^{(t, \*)}, \dots, \vec{\Psi}^{((t-s), \*)}), \tag{4.3}$$

where *�* is a small positive real number.

Then the dynamic EIRP procedure (the discrete procedure) takes a form:

$$\mathbf{E}^{(t+1)} = \mathbf{E}^t + \varepsilon \mathcal{L}(\mathbf{E}^t, \bar{\boldsymbol{\psi}}^{(t, \ast)}, \dots, \bar{\boldsymbol{\psi}}^{((t-s), \ast)}).\tag{4.4}$$

Now let the variable *t* is continuous one. Then under *�* → 0 we will have the continuous dynamic EIRP that is described be the differential equation:

$$\frac{d\mathbf{E}(t)}{dt} = \mathcal{L}(\mathbf{E}^t, \bar{\boldsymbol{\Psi}}^{(t, \ast)}, \dots, \bar{\boldsymbol{\Psi}}^{((t-s), \ast)}).\tag{4.5}$$

The *t,*(*t* − 1)*,...,* (*t* − *s*)*-posterior* images are defined by the ELP or EQP problems, which represent the appropriate *entropy operators* (4.1). So the dynamic EIRP procedure (4.1, 4.2, (4.4)) represents the discrete dynamic system with entropy operator (the discrete DSEO) and its the continuous analog (4.5) represents the continuous dynamic system with entropy operator (the continuous DSEO). Some general properties of the DSEO will be described in the next section.

#### **4.1 Structures of the dynamic EIRP procedures**

Let us consider some partial cases. One of them relates to the examination of a Markov version of the procedure when the information only on the *t*-posterior image is used to shape up **E**(*t*+1):

$$\mathbf{E}^{(t+1)} = \mathcal{L}(\mathbf{E}^t, \vec{\Psi}'^ \*). \tag{4.6}$$

In the second case, information collections at *t*, *t* −1, . . . , *t* − *s* stages are used for the estimation of the current mean *ψ*¯¯*<sup>t</sup>* of the posterior image:

$$E^{(t+1)} = \mathcal{L}(E^t, \bar{\psi}^t). \tag{4.7}$$

Finally, in the third case, information collections at *t*, *t* − 1, . . . , *t* − *s* stages are used for the estimation of the current mean *ψ*¯¯*<sup>t</sup>* and dispersion **d***<sup>t</sup>* of the posterior image:

$$\mathbf{E}^{(t+1)} = \mathcal{L}(\mathbf{E}^t, \tilde{\boldsymbol{\psi}}^t, \mathbf{d}^t). \tag{4.8}$$

We will introduce the following types of the dynamic procedures of the EIRP:

• the identical feedback (*I* − *f eedback*)

$$\mathbf{E}^{(t+1)} = \arg\max\_{\boldsymbol{\upupup}} H(\bar{\boldsymbol{\upup}}^t) |\mathbf{E}^t\rangle \mid \bar{\boldsymbol{\upup}}^t \in \mathcal{D}(\mathbf{u}^t);\tag{4.9}$$

• the feedback with respect to the current mean of image (*CM* − *f eedback*)

$$\begin{aligned} \mathbf{E}^{(t+1)} &= \mathbf{E}^t + \mathfrak{a} (\mathbf{E}^t - \bar{\mathfrak{y}}^t), \\ \tilde{\boldsymbol{\psi}}^{(t+1)} &= \tilde{\boldsymbol{\psi}}^t + \frac{1}{t+1} \left( \boldsymbol{\psi}^{(t, \ast)} - \tilde{\boldsymbol{\psi}}^t \right); \\ \tilde{\boldsymbol{\psi}}^{(t, \ast)} &= \arg\max\_{\boldsymbol{\Psi}} \{ H(\bar{\boldsymbol{\psi}}^t, \mathbf{E}^t) | \bar{\boldsymbol{\psi}}^t \in \mathcal{D}(\mathbf{u}^t) \}; \end{aligned} \tag{4.11}$$

• the feedback with respect to the current mean and dispersion of image (*CMD* − *f eedback*)

$$\mathbf{E}^{(t+1)} = \mathbf{E}^t + \mathfrak{a}(\mathbf{d}^t)(\mathbf{E}^t - \bar{\psi}^t),\tag{4.12}$$

$$
\bar{\psi}^{(t+1)} = \bar{\psi}^t + \frac{1}{t+1} \left( \bar{\psi}^{(t,\*)} - \bar{\psi}^t \right) . \tag{4.13}
$$

$$\begin{aligned} \mathbf{d}^{(t+1)} &= \mathbf{d}^t + \frac{1}{t+1} \left( \mathbf{d}^t + [\bar{\boldsymbol{\psi}}^{(t,\ast)} - \bar{\boldsymbol{\psi}}^t]^2 \right), \\ \bar{\boldsymbol{\psi}}^{(t,\ast)} &= \arg \max\_{\boldsymbol{\psi}} \{ H(\bar{\boldsymbol{\psi}}, \mathbf{E}^t) \mid \bar{\boldsymbol{\psi}}^t \in \mathcal{D}(\boldsymbol{u}^t) \}. \end{aligned}$$

(4.14)

According to the definition of the *I*-feedback procedure we have:

*ϕi*[**z***<sup>t</sup>* (**E***<sup>t</sup>* )] =

*Ei* <sup>=</sup> *Ei*

*<sup>x</sup>* <sup>=</sup> <sup>Ψ</sup>(*x*) = *<sup>x</sup>*

*x* + *a*

(0) = 1/*a* ≥ 1, (*a* ≤ 1), then the auxiliary equation has the unique solution, and the method of the

Now it is necessary to find a conditions when *ϕi*[**z**(**E**)] ≤ 1. The sufficient conditions for it is

**Theorem 2.** *Let the matrix T in (2.7) has the complete rank n, and the following conditions be valid*:

− *umax* > 0, *umax* = max

− *umin* < 0, *umin* = min

*<sup>i</sup>* <sup>+</sup> *<sup>ϕ</sup>i*(**z***t*)]<sup>2</sup> <sup>≤</sup> 0, (*j*, *<sup>k</sup>*) = 1, *<sup>n</sup>*.

). Its elements take a form:

, *x* ≥ 0.

) = *<sup>E</sup><sup>t</sup>*

*Et*

*i*

)] , *<sup>i</sup>* <sup>=</sup> 1, *<sup>m</sup>*, (4.17)

*tji* <sup>≥</sup> 0. (4.18)

(*x*) > 0 for all *x* > 0 and Ψ(∞) = 0),

*uj*; (4.20)

*uj*. (4.21)

*<sup>j</sup>* ∈ [0, 1], *j* = 1, *n*, and

*Ei* <sup>+</sup> *<sup>ϕ</sup>i*[**z**(**E**)] , *<sup>i</sup>* <sup>=</sup> 1, *<sup>m</sup>*. (4.19)

*j*∈[1,*n*]

*j*∈[1,*n*]

*<sup>i</sup>* <sup>+</sup> *<sup>ϕ</sup>i*[**z***t*(**E***<sup>t</sup>*

Entropic Image Restoration as a Dynamic System with Entropy Operator 57

*n* ∏ *j*=1 [*zt j*]

The iterative process (4.17) can be considered as the method of the simple iteration applying

*<sup>i</sup>* <sup>=</sup> <sup>Ψ</sup>*i*(**E***<sup>t</sup>*

**Theorem 1.***Let ϕi*[**z**(**E**)] ≤ 1 *for all i* = 1, *m*, **z** ≥ **0**, **0** ≤ **E** ≤ **1**. *Then the system of equations (4.19) has the unique solution E*∗.

We can see that the function Ψ(*x*) is strictly monotone increasing (Ψ�

 *m* ∑ *i*=1

*tji <sup>E</sup><sup>t</sup> i Et <sup>i</sup>* + 1

> <sup>=</sup> <sup>−</sup> <sup>1</sup> *zk m* ∑ *i*=1

Therefore, under the theorem's conditions, the solution of the equations (4.16) *z*∗

*tji*

*tji tki E<sup>t</sup>*

[*E<sup>t</sup>*

*<sup>i</sup> <sup>ϕ</sup>i*(**z***<sup>t</sup>* )

The equality to zero is reached when **z** → ∞. So, the functions Φ1,..., Φ*<sup>n</sup>* are strictly monotone

and is strictly convex (Ψ� �(*x*) < 0, *x* > 0 and Ψ� �(∞) = 0).

max *j*∈[1,*n*]

 *m* ∑ *i*=1

**Proof.** Consider the Jacobian of the vector-function Φ¯ (**z***<sup>t</sup>*

*∂*Φ*j*(**z***<sup>t</sup>* ) *∂zk*

*Et*+<sup>1</sup>

where

If Ψ�

decreasing.

to the eguations:

**Proof.** Consider the auxiliary equation:

simple iteration is converged to this solution.

min *j*∈[1,*n*]

*Then ϕi*[*z*(*E*)] ≤ 1 *for all i* = 1, *m.*

the functions 0 < *φj*(**z**(**E**)) ≤ 1.

formed by the following theorem.

#### **4.2 Investigation of the dynamic EIRP procedure with** *I***-feedback**

Consider the problem (2.13) in which the feasible set is the polyhedron, **a** = **0**, **b** = **1**, and the constraints to the possible density functions (2.7) are absent. In this case *t*-posterior density function hold:

$$\psi\_{i}^{t,\*} = \frac{E\_i^t}{E\_i^t + \prod\_{j=1}^n [z\_j^t]^{t\_{ji}}}, \qquad i = \overline{1,m}. \tag{4.15}$$

The exponential Lagrange multipliers *z*1,..., *zn* are defined from the following equations:

$$\Phi\_{\dot{j}}(\mathbf{z}^{t}) = \sum\_{i=1}^{m} \frac{t\_{\dot{j}i} E\_{i}^{t}}{E\_{i}^{t} + \prod\_{j=1}^{n} [z\_{j}^{t}]^{t\_{\dot{j}}}} = u\_{\dot{j}\prime} \qquad \dot{j} = \overline{1\,\mu}. \tag{4.16}$$

According to the definition of the *I*-feedback procedure we have:

$$E\_i^{t+1} = \Psi\_i(\mathbf{E}^t) = \frac{E\_i^t}{E\_i^t + \varphi\_i[\mathbf{z}^t(\mathbf{E}^t)]}, \qquad i = \overline{1, m}, \tag{4.17}$$

where

12 Will-be-set-by-IN-TECH

In the second case, information collections at *t*, *t* −1, . . . , *t* − *s* stages are used for the estimation

Finally, in the third case, information collections at *t*, *t* − 1, . . . , *t* − *s* stages are used for the

, *ψ*¯¯*<sup>t</sup>*

, *ψ*¯¯*<sup>t</sup>* , **d***<sup>t</sup>*

)|**E***<sup>t</sup>*

*<sup>ψ</sup>*(*t*,∗) <sup>−</sup> *<sup>ψ</sup>*¯¯*<sup>t</sup>*

, **E***<sup>t</sup>*

)<sup>|</sup> *<sup>ψ</sup>*¯*<sup>t</sup>* ∈ D(**u***<sup>t</sup>*

 ;

)|*ψ*¯*<sup>t</sup>* ∈ D(**u***<sup>t</sup>*

)<sup>|</sup> *<sup>ψ</sup>*¯*<sup>t</sup>* ∈ D(*u<sup>t</sup>*

] 2 ,

)}.

*tji* , *<sup>i</sup>* = 1, *<sup>m</sup>*. (4.15)

*tji* = *uj*, *<sup>j</sup>* = 1, *<sup>n</sup>*. (4.16)

). (4.7)

). (4.8)

), (4.10)

), (4.12)

, (4.13)

)};

); (4.9)

(4.11)

(4.14)

*<sup>E</sup>*(*t*+1) <sup>=</sup> <sup>L</sup>(*E<sup>t</sup>*

**<sup>E</sup>**(*t*+1) <sup>=</sup> <sup>L</sup>(**E***<sup>t</sup>*

estimation of the current mean *ψ*¯¯*<sup>t</sup>* and dispersion **d***<sup>t</sup>* of the posterior image:

We will introduce the following types of the dynamic procedures of the EIRP:

**<sup>E</sup>**(*t*+1) <sup>=</sup> arg max *<sup>ψ</sup><sup>t</sup> <sup>H</sup>*(*ψ*¯*<sup>t</sup>*

• the feedback with respect to the current mean of image (*CM* − *f eedback*)

*ψ*¯¯(*t*+1) = *ψ*¯¯*<sup>t</sup>* +

**<sup>E</sup>**(*t*+1) <sup>=</sup> **<sup>E</sup>***<sup>t</sup>* <sup>+</sup> *<sup>α</sup>*(**E***<sup>t</sup>* <sup>−</sup> *<sup>ψ</sup>*¯¯*<sup>t</sup>*

*<sup>ψ</sup>*¯(*t*,∗) <sup>=</sup> arg max *<sup>ψ</sup>* {*H*(*ψ*¯*<sup>t</sup>*

**E**(*t*+1) = **E***<sup>t</sup>* + *α*(**d***<sup>t</sup>*

*ψ*¯¯(*t*+1) = *ψ*¯¯*<sup>t</sup>* +

**d**(*t*+1) = **d***<sup>t</sup>* +

**4.2 Investigation of the dynamic EIRP procedure with** *I***-feedback**

*ψt*,<sup>∗</sup>

Φ*j*(**z***<sup>t</sup>* ) = *m* ∑ *i*=1

*<sup>i</sup>* <sup>=</sup> *<sup>E</sup><sup>t</sup>*

*Et <sup>i</sup>* <sup>+</sup> <sup>∏</sup>*<sup>n</sup>*

*Et <sup>i</sup>* <sup>+</sup> <sup>∏</sup>*<sup>n</sup>*

function hold:

1 *t* + 1 

• the feedback with respect to the current mean and dispersion of image (*CMD* − *f eedback*)

1 *t* + 1 

1 *t* + 1 

Consider the problem (2.13) in which the feasible set is the polyhedron, **a** = **0**, **b** = **1**, and the constraints to the possible density functions (2.7) are absent. In this case *t*-posterior density

*i*

*j*=1[*z<sup>t</sup> j* ]

The exponential Lagrange multipliers *z*1,..., *zn* are defined from the following equations:

*j*=1[*z<sup>t</sup> j* ]

*tjiE<sup>t</sup> i*

*<sup>ψ</sup>*¯(*t*,∗) <sup>=</sup> arg max *<sup>ψ</sup>* {*H*(*ψ*¯, **<sup>E</sup>***<sup>t</sup>*

)(**E***<sup>t</sup>* <sup>−</sup> *<sup>ψ</sup>*¯¯*<sup>t</sup>*

*<sup>ψ</sup>*¯(*t*,∗) <sup>−</sup> *<sup>ψ</sup>*¯¯*<sup>t</sup>*

**<sup>d</sup>***<sup>t</sup>* + [*ψ*¯(*t*,∗) <sup>−</sup> *<sup>ψ</sup>*¯¯*<sup>t</sup>*

of the current mean *ψ*¯¯*<sup>t</sup>* of the posterior image:

• the identical feedback (*I* − *f eedback*)

$$\varphi\_i[\mathbf{z}^t(\mathbf{E}^t)] = \prod\_{j=1}^n [z\_j^t]^{t\_{ji}} \ge 0. \tag{4.18}$$

The iterative process (4.17) can be considered as the method of the simple iteration applying to the eguations:

$$E\_i = \frac{E\_i}{E\_i + \varrho\_i[\mathbf{z}(\mathbf{E})]}, \qquad i = \overline{1, m}. \tag{4.19}$$

**Theorem 1.***Let ϕi*[**z**(**E**)] ≤ 1 *for all i* = 1, *m*, **z** ≥ **0**, **0** ≤ **E** ≤ **1**.

*Then the system of equations (4.19) has the unique solution E*∗.

**Proof.** Consider the auxiliary equation:

$$\mathfrak{x} = \Psi(\mathfrak{x}) = \frac{\mathfrak{x}}{\mathfrak{x} + a'}, \quad \mathfrak{x} \ge 0.$$

We can see that the function Ψ(*x*) is strictly monotone increasing (Ψ� (*x*) > 0 for all *x* > 0 and Ψ(∞) = 0), and is strictly convex (Ψ� �(*x*) < 0, *x* > 0 and Ψ� �(∞) = 0).

If Ψ� (0) = 1/*a* ≥ 1, (*a* ≤ 1), then the auxiliary equation has the unique solution, and the method of the simple iteration is converged to this solution.

Now it is necessary to find a conditions when *ϕi*[**z**(**E**)] ≤ 1. The sufficient conditions for it is formed by the following theorem.

**Theorem 2.** *Let the matrix T in (2.7) has the complete rank n, and the following conditions be valid*:

$$\max\_{j \in [1, n]} \left( \sum\_{i=1}^{m} t\_{ji} \right) - u\_{\max} > 0, \qquad u\_{\max} = \max\_{j \in [1, n]} u\_j. \tag{4.20}$$

$$\min\_{\substack{j \in [1, n]}} \left( \sum\_{i=1}^{m} t\_{ji} \left( \frac{E\_i^t}{E\_i^t + 1} \right) \right) - u\_{\min} < 0, \qquad u\_{\min} = \min\_{j \in [1, n]} u\_j. \tag{4.21}$$

*Then ϕi*[*z*(*E*)] ≤ 1 *for all i* = 1, *m.*

**Proof.** Consider the Jacobian of the vector-function Φ¯ (**z***<sup>t</sup>* ). Its elements take a form:

$$\frac{\partial \Phi\_{j}(\mathbf{z}^{t})}{\partial z\_{k}} = -\frac{1}{z\_{k}} \sum\_{i=1}^{m} \frac{t\_{ji} \, t\_{ki} \, \mathbf{E}\_{i}^{t} \, \dot{\varphi}\_{i}(\mathbf{z}^{t})}{[\mathbf{E}\_{i}^{t} + \varphi\_{i}(\mathbf{z}^{t})]^{2}} \le 0, \qquad (j,k) = \overline{1,n}.$$

The equality to zero is reached when **z** → ∞. So, the functions Φ1,..., Φ*<sup>n</sup>* are strictly monotone decreasing.

Therefore, under the theorem's conditions, the solution of the equations (4.16) *z*∗ *<sup>j</sup>* ∈ [0, 1], *j* = 1, *n*, and the functions 0 < *φj*(**z**(**E**)) ≤ 1.

Fig. 3. Test image 2

stage *t* we shaped up the auxiliary vector

*<sup>ψ</sup>*˜(*t*,∗) *<sup>i</sup>* =

> ¯ *<sup>ψ</sup>*˜(*t*+1) *<sup>i</sup>* <sup>=</sup> ¯ *ψ*˜*t <sup>i</sup>* +

> > *Et*+<sup>1</sup> *<sup>i</sup>* <sup>=</sup> ¯

The modified dynamic procedure takes the form:

**5. Dynamic systems with entropy operator (DSEO)**

The quality of the right image is obvious. The test image 2 is shown in the fig. 3.

equality (3.4).

 *<sup>ψ</sup>*(*t*,∗) *<sup>i</sup>* , if <sup>|</sup>*E<sup>t</sup>*

> *Et i*

In parallel, the current mean of the components of the vector *ψ*˜*t*,<sup>∗</sup> are calculated:

The modification of the dynamic procedure with *I- feedback* involved the following. At each

Entropic Image Restoration as a Dynamic System with Entropy Operator 59

, if <sup>|</sup>*E<sup>t</sup>*

*<sup>ψ</sup>*˜(*t*+1)

In fig. 2, in the right upper window, the result of the EIRP with the static procedure is shown. In the middle lower window, fig. 2 shows results of the EIRP by the dynamic procedure with *I -feedback*, and in the right lower window results of the modified dynamic procedure is shown.

We can see that dynamic procedures of the image restoration from projections represent a dynamic discrete system with the particular type of the entropy operator with the generalized entropy Fermi-Dirac (3.2), and the feasible set that is described by the inequality and the

1 *t* + 1

*<sup>i</sup>* <sup>−</sup> *<sup>ψ</sup>*(*t*,∗)

*<sup>i</sup>* <sup>−</sup> *<sup>ψ</sup>*(*t*,∗)

(*ψ*˜*t*,<sup>∗</sup> *<sup>i</sup>* <sup>−</sup> ¯ *ψ*˜*t*

*<sup>i</sup>* | ≥ *δ*,

*i* = 1, *m*. (4.24)

*<sup>i</sup>* , *i* = 1, *m*. (4.26)

*<sup>i</sup>* <sup>|</sup> <sup>&</sup>lt; *<sup>δ</sup>* , (4.23)

). (4.25)

Fig. 2. Test image 1

## **4.3 Computer experiment**

Consider the dynamic EIRP when the tomographic device gives the orthogonal linear projections with the matrix

$$T = \begin{pmatrix} 1 & \cdots & 1 & \cdots & 0 & \cdots & 0 \\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots \\ 0 & \cdots & 0 & \cdots & 1 & \cdots & 1 \\ 1 & \cdots & 0 & \cdots & 1 & \cdots & \cdots \\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots & \cdots \\ 0 & \cdots & 1 & \cdots & 0 & \cdots & 1 \end{pmatrix} \tag{4.22}$$

As a test image, use is made of the LENA test (IEEE Image Processing), on which the spot is placed (fig. 2, the left upper window).

To the right and below window, the projections with noise are shown. In the example, the noise/signal ratio amounted to 0.3.

It is necessary to restored *the LENA with the spot* having the noisy projections. We use "the pure LENA", which is shown in the second upper window, as the 0*-prior image E*<sup>0</sup> <sup>=</sup> {*E*<sup>0</sup> <sup>1</sup>,..., *<sup>E</sup>*<sup>0</sup> *<sup>m</sup>*}.

This problem of the EIRP is described by the ELP with the constrains-equalities. The multiplicative algorithms with *1*-active dual variable (3.18) is used for solution of the problem.

Fig. 3. Test image 2

14 Will-be-set-by-IN-TECH

Consider the dynamic EIRP when the tomographic device gives the orthogonal linear

As a test image, use is made of the LENA test (IEEE Image Processing), on which the spot is

To the right and below window, the projections with noise are shown. In the example, the

It is necessary to restored *the LENA with the spot* having the noisy projections. We use "the pure

This problem of the EIRP is described by the ELP with the constrains-equalities. The multiplicative algorithms with *1*-active dual variable (3.18) is used for solution of the problem.

LENA", which is shown in the second upper window, as the 0*-prior image E*<sup>0</sup> <sup>=</sup> {*E*<sup>0</sup>

 ··· 1 ··· 0 ··· 0 ··· ··· ··· ··· ··· ··· ··· ··· 0 ··· 1 ··· 1 ··· 0 ··· 1 ··· ··· ··· ··· ··· ··· ··· ··· ··· ··· 1 ··· 0 ··· 1

⎞

⎟⎟⎟⎟⎟⎟⎠

(4.22)

<sup>1</sup>,..., *<sup>E</sup>*<sup>0</sup>

*<sup>m</sup>*}.

*T* =

⎛

⎜⎜⎜⎜⎜⎜⎝

Fig. 2. Test image 1

**4.3 Computer experiment**

projections with the matrix

placed (fig. 2, the left upper window).

noise/signal ratio amounted to 0.3.

The modification of the dynamic procedure with *I- feedback* involved the following. At each stage *t* we shaped up the auxiliary vector

$$\tilde{\psi}\_{i}^{(t,\ast)} = \begin{cases} \psi\_{i}^{(t,\ast)} & \text{if } |E\_{i}^{t} - \psi\_{i}^{(t,\ast)}| \ge \delta, \\ E\_{i'}^{t} & \text{if } |E\_{i}^{t} - \psi\_{i}^{(t,\ast)}| < \delta \end{cases} \tag{4.23}$$

$$i = \overline{1, m}.\tag{4.24}$$

In parallel, the current mean of the components of the vector *ψ*˜*t*,<sup>∗</sup> are calculated:

$$
\bar{\psi}\_i^{(t+1)} = \bar{\psi}\_i^t + \frac{1}{t+1} (\tilde{\psi}\_i^{t,\*} - \bar{\psi}^t). \tag{4.25}
$$

The modified dynamic procedure takes the form:

$$E\_i^{t+1} = \overline{\psi}\_i^{(t+1)}, \qquad i = \overline{1, m}. \tag{4.26}$$

In fig. 2, in the right upper window, the result of the EIRP with the static procedure is shown. In the middle lower window, fig. 2 shows results of the EIRP by the dynamic procedure with *I -feedback*, and in the right lower window results of the modified dynamic procedure is shown. The quality of the right image is obvious.

The test image 2 is shown in the fig. 3.

#### **5. Dynamic systems with entropy operator (DSEO)**

We can see that dynamic procedures of the image restoration from projections represent a dynamic discrete system with the particular type of the entropy operator with the generalized entropy Fermi-Dirac (3.2), and the feasible set that is described by the inequality and the equality (3.4).

Fig. 4. Classificatory graph

+(**v**−, **<sup>v</sup>**+) into the set Y ⊂ *<sup>R</sup><sup>m</sup>*

�**y**[**u**(1)

by two local Lipschitz-constants - *LU* and *LV*, i.e.

] <sup>−</sup> **<sup>y</sup>**[**u**(2)

We will use the upper estimations of local Lipschitz-constant:

Evaluate *the normalized entropy operator* in the following form:

*LU* ≤ max *U<sup>m</sup>* +

*ti* =

*m* ∑ *i*=1 ⎛ ⎝*t* 2 *ki* −

, **v**(1)

*V<sup>n</sup>*

where

quadratic matrix *T T*�

**5.2 Estimation of the local Lipschitz-constants for the** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.**

Entropic Image Restoration as a Dynamic System with Entropy Operator 61

]� ≤ *LU* �**u**(1) <sup>−</sup> **<sup>u</sup>**(2)

�*YU*�, *LV* = max

� **<sup>x</sup>**� ln **<sup>x</sup>** *e***u** �

*ti*

˜*tki*, *tki* <sup>=</sup> ˜*tki*

all *i* = 1, *m*. Also it is assumed that the condition of the dominating diagonal is valid for the

, i.e. the following inequalities take a form:

⎞

*tkitji*

where *YU* and *YV* are the *U*-Jacobian and the *V*-Jacobian of the operator **y**[**u**, **v**] respectively.

*H*(**x** | **u**) = −

The matrix *T* in (5.11) has a full rank *n* and the normalized elements, i.e. ∑*<sup>n</sup>*

*n* ∑ *k*=1

*n* ∑ *j*�=*k* *V<sup>n</sup>* +

**x**(**u**, **v**) = arg max (*H*[**x**, **u**] | *T***x** = **v**, **x** ≥ **0**), (5.11)

<sup>+</sup> of the operator's values. We will characterize this mapping

� <sup>+</sup> *LV* �**v**(1) <sup>−</sup> **<sup>v</sup>**(2)

�*YV*�, (5.10)

. (5.12)

, *i* = 1, *m*. (5.13)

⎠ ≥ > 0, *k* = 1, *n*. (5.14)

+(**u**−, **<sup>u</sup>**+) and

�. (5.9)

*<sup>k</sup>*=<sup>1</sup> *tki* = 1 for

The �*HD*, *<sup>B</sup>*, *Eq*, *Plh*�-entropy operator describes the mapping of the sets *<sup>U</sup><sup>m</sup>*

, **v**(2)

In general case the class of the continuous DSEO is described by the following differential equations:

$$\frac{d\mathbf{u}(t)}{dt} = \mathcal{U}\left(\mathbf{u}(t), \mathbf{v}(t), \mathbf{y}[t, \mathbf{u}(t), \mathbf{v}(t)]\right), \qquad \mathbf{u}(0) = \mathbf{u}^0. \tag{5.1}$$

$$\frac{d\mathbf{v}(t)}{dt} = \mathcal{V}\left(\mathbf{u}(t), \mathbf{v}(t), \mathbf{y}[t, \mathbf{u}(t), \mathbf{v}(t)]\right), \qquad \mathbf{v}(0) = \mathbf{v}^0. \tag{5.2}$$

with the entropy operator:

$$\mathbf{y}[t,\mathbf{u}(t),\mathbf{v}(t)] = \arg\max\_{\mathbf{y}} \left\{ H(t,\mathbf{y},\mathbf{u}(t)) \,|\, \mathbf{y} \in \mathcal{D}[t,\mathbf{v}(t)] \right\},\tag{5.3}$$

where: *H*(*t*, **y** | **u**) is an entropy function with the *H*-parameters **u**; the vectors (**y**, **u**) ∈ *<sup>R</sup>n*, **<sup>v</sup>** <sup>∈</sup> *<sup>R</sup><sup>m</sup>* , and <sup>D</sup>(*t*, **<sup>v</sup>**) is a feasible set depended from the <sup>D</sup>-parameters **<sup>v</sup>**.

In these equations U is the *n*-vector-function, and V is the *m*-vector-function.

#### **5.1 Classification of the DSEO**

Some physical analogues we will use for construction of the classificatory graph. In particular, from the equations (5.1, 5.2) it is seen the rates of parameters is proportional to the flows. The entropy function is a probability characteristics of a stochastic process. So the *H*-parameters are the parameters of this process.

We will use the following classificatory indicators:


The classificatory graph is shown in the fig. 4.

At the beginning we consider some properties of the entropy operator, notably, the �*HD*, *B*, *Eq*, *Plh*�-entropy operator that is included to the �*HD*�-DSEO:

$$\mathbf{y}[\mathbf{u}, \mathbf{v}] = \arg\max \left\{ H\_{\mathcal{B}}[\mathbf{y}, \mathbf{u}] \,|\, \mathbf{y} \in \mathcal{D}[\mathbf{v}] \right\},\tag{5.4}$$

where Boltzmann-entropy function is

$$H(\mathbf{y} \mid \mathbf{u}) = -\left(\mathbf{y}' \ln \frac{\mathbf{y}}{e\mathbf{u}}\right), \quad \mathbf{y} \in \mathbb{R}\_{+}^{m} \tag{5.5}$$

and the feasible set is

$$\tilde{\mathcal{D}}[\mathbf{v}] = \{ \mathbf{y} : \mathcal{T}\mathbf{y} = \mathbf{v}, \ \mathbf{y} \ge \mathbf{0} \}. \tag{5.6}$$

In these expressions the vector ln **<sup>y</sup>** *<sup>e</sup>***<sup>u</sup>** <sup>=</sup> {ln *<sup>y</sup>*<sup>1</sup> *eu*<sup>1</sup> , . . . , ln *ym eum* }, and the vectors

$$\mathbf{u} \in \mathcal{U}\_{+}^{m}(\mathbf{u}^{-}, \mathbf{u}^{+}) \subset \mathcal{R}\_{+}^{m}, \quad \mathbf{v} \in V\_{+}^{n}(\mathbf{v}^{-}, \mathbf{v}^{+}) \subset \mathcal{R}\_{+}^{n}, \qquad n < m. \tag{5.7}$$

and

$$\begin{aligned} \mathcal{U}\_+^m(\mathbf{u}^-, \mathbf{u}^+) &= \{ \mathbf{u} : \mathbf{0} < \mathbf{u}^- \le \mathbf{u} \le \mathbf{u}^+ \le \mathbf{1} \}, \\\mathcal{V}\_+^{\mathbb{R}}(\mathbf{v}^-, \mathbf{v}^+) &= \{ \mathbf{v} : \mathbf{0} < \mathbf{v}^- \le \mathbf{v} \le \mathbf{v}^+ \}. \end{aligned} \tag{5.8}$$

The (*<sup>n</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix *<sup>T</sup>*˜ = [˜*tki* <sup>≥</sup> <sup>0</sup>] has a full rank equal *<sup>n</sup>*.

Fig. 4. Classificatory graph

## **5.2 Estimation of the local Lipschitz-constants for the** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.**

The �*HD*, *<sup>B</sup>*, *Eq*, *Plh*�-entropy operator describes the mapping of the sets *<sup>U</sup><sup>m</sup>* +(**u**−, **<sup>u</sup>**+) and *V<sup>n</sup>* +(**v**−, **<sup>v</sup>**+) into the set Y ⊂ *<sup>R</sup><sup>m</sup>* <sup>+</sup> of the operator's values. We will characterize this mapping by two local Lipschitz-constants - *LU* and *LV*, i.e.

$$\|\mathbf{y}[\mathbf{u}^{(1)}, \mathbf{v}^{(1)}] - \mathbf{y}[\mathbf{u}^{(2)}, \mathbf{v}^{(2)}]\| \le L\_{U} \|\mathbf{u}^{(1)} - \mathbf{u}^{(2)}\| + L\_{V} \|\mathbf{v}^{(1)} - \mathbf{v}^{(2)}\|.\tag{5.9}$$

We will use the upper estimations of local Lipschitz-constant:

$$L\_{\mathcal{U}} \le \max\_{\mathcal{U}\_+^w} \|\mathcal{Y}\_{\mathcal{U}}\|\_{\mathcal{\prime}} \qquad L\_{\mathcal{V}} = \max\_{\mathcal{V}\_+^w} \|\mathcal{Y}\_{\mathcal{V}}\|\_{\mathcal{\prime}} \tag{5.10}$$

where *YU* and *YV* are the *U*-Jacobian and the *V*-Jacobian of the operator **y**[**u**, **v**] respectively.

Evaluate *the normalized entropy operator* in the following form:

$$\mathbf{x}(\mathbf{u}, \mathbf{v}) = \arg\max \left( H[\mathbf{x}, \mathbf{u}] \mid T\mathbf{x} = \mathbf{v}, \; \mathbf{x} \ge \mathbf{0} \right),\tag{5.11}$$

where

16 Will-be-set-by-IN-TECH

In general case the class of the continuous DSEO is described by the following differential

where: *H*(*t*, **y** | **u**) is an entropy function with the *H*-parameters **u**; the vectors (**y**, **u**) ∈

Some physical analogues we will use for construction of the classificatory graph. In particular, from the equations (5.1, 5.2) it is seen the rates of parameters is proportional to the flows. The entropy function is a probability characteristics of a stochastic process. So the *H*-parameters

• (**A**), types of the state coordinates (�*H*�-coordinates **u**, �*D*�-coordinates **v**,

At the beginning we consider some properties of the entropy operator, notably, the

 **<sup>y</sup>**� ln **<sup>y</sup>** *e***u** 

<sup>+</sup>, **<sup>v</sup>** <sup>∈</sup> *<sup>V</sup><sup>n</sup>*

+(**u**−, **<sup>u</sup>**+) = {**<sup>u</sup>** : **<sup>0</sup>** <sup>&</sup>lt; **<sup>u</sup>**<sup>−</sup> <sup>≤</sup> **<sup>u</sup>** <sup>≤</sup> **<sup>u</sup>**<sup>+</sup> <sup>≤</sup> **<sup>1</sup>**},

*<sup>e</sup>***<sup>u</sup>** <sup>=</sup> {ln *<sup>y</sup>*<sup>1</sup>

*HB*[**y**, **<sup>u</sup>**] <sup>|</sup> **<sup>y</sup>** <sup>∈</sup> <sup>D</sup>˜ [**v**]

*eu*<sup>1</sup> , . . . , ln *ym*

, **<sup>y</sup>** <sup>∈</sup> *<sup>R</sup><sup>m</sup>*

+(**v**−, **<sup>v</sup>**+) <sup>⊂</sup> *<sup>R</sup><sup>n</sup>*

+(**v**−, **<sup>v</sup>**+) = {**<sup>v</sup>** : **<sup>0</sup>** <sup>&</sup>lt; **<sup>v</sup>**<sup>−</sup> <sup>≤</sup> **<sup>v</sup>** <sup>≤</sup> **<sup>v</sup>**+}. (5.8)

<sup>D</sup>˜ [**v**] = {**<sup>y</sup>** : *<sup>T</sup>*˜ **<sup>y</sup>** <sup>=</sup> **<sup>v</sup>**, **<sup>y</sup>** <sup>≥</sup> **<sup>0</sup>**}. (5.6)

*eum* }, and the vectors

, (5.4)

<sup>+</sup>, (5.5)

<sup>+</sup>, *n* < *m*, (5.7)

• (**C**), entropy functions (�*F*�-Fermi-, �*E*�-Einstein-, �*B*�-Boltzmann-entropy functions); • (**D**), models of the feasible sets (�*Eq*�-equalities, �*Ieq*�-inequalities, �*Mx*�-mixed); • (**F**), types of the feasible sets (�*Plh*�- polyhedron, �*Cnv*�-convex, �*nCnv*�-non-convex).

*<sup>R</sup>n*, **<sup>v</sup>** <sup>∈</sup> *<sup>R</sup><sup>m</sup>* , and <sup>D</sup>(*t*, **<sup>v</sup>**) is a feasible set depended from the <sup>D</sup>-parameters **<sup>v</sup>**. In these equations U is the *n*-vector-function, and V is the *m*-vector-function.

• (**B**), flows (�*Add*� - an additive flow, �*Mlt*� - a multiplicative flow);

�*HD*, *B*, *Eq*, *Plh*�-entropy operator that is included to the �*HD*�-DSEO: **y**[**u**, **v**] = arg max

*H*(**y** | **u**) = −

+(**u**−, **<sup>u</sup>**+) <sup>⊂</sup> *<sup>R</sup><sup>m</sup>*

*U<sup>m</sup>*

*Vn*

The (*<sup>n</sup>* <sup>×</sup> *<sup>m</sup>*)-matrix *<sup>T</sup>*˜ = [˜*tki* <sup>≥</sup> <sup>0</sup>] has a full rank equal *<sup>n</sup>*.

*dt* <sup>=</sup> <sup>U</sup> (**u**(*t*), **<sup>v</sup>**(*t*), **<sup>y</sup>**[*t*, **<sup>u</sup>**(*t*), **<sup>v</sup>**(*t*)]), **<sup>u</sup>**(0) = **<sup>u</sup>**0. (5.1)

*dt* <sup>=</sup> <sup>V</sup> (**u**(*t*), **<sup>v</sup>**(*t*), **<sup>y</sup>**[*t*, **<sup>u</sup>**(*t*), **<sup>v</sup>**(*t*)]), **<sup>v</sup>**(0) = **<sup>v</sup>**0. (5.2)

**<sup>y</sup>**[*t*, **<sup>u</sup>**(*t*), **<sup>v</sup>**(*t*)] = arg max **<sup>y</sup>** {*H*(*t*, **<sup>y</sup>**, **<sup>u</sup>**(*t*))<sup>|</sup> **<sup>y</sup>** ∈ D[*t*, **<sup>v</sup>**(*t*)]} , (5.3)

equations:

*d***u**(*t*)

*d***v**(*t*)

with the entropy operator:

**5.1 Classification of the DSEO**

are the parameters of this process.

�*HD*�-coordinates **u**, **v**);

We will use the following classificatory indicators:

The classificatory graph is shown in the fig. 4.

where Boltzmann-entropy function is

In these expressions the vector ln **<sup>y</sup>**

**<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>*

and the feasible set is

and

$$H(\mathbf{x} \mid \mathbf{u}) = -\left(\mathbf{x}' \ln \frac{\mathbf{x}}{e\mathbf{u}}\right). \tag{5.12}$$

$$t\_i = \sum\_{k=1}^{n} \tilde{t}\_{ki} \, \, \, t\_{ki} = \frac{\tilde{t}\_{ki}}{t\_i} \, \, \, i = \overline{1, m}. \tag{5.13}$$

The matrix *T* in (5.11) has a full rank *n* and the normalized elements, i.e. ∑*<sup>n</sup> <sup>k</sup>*=<sup>1</sup> *tki* = 1 for all *i* = 1, *m*. Also it is assumed that the condition of the dominating diagonal is valid for the quadratic matrix *T T*� , i.e. the following inequalities take a form:

$$\sum\_{i=1}^{m} \left( t\_{ki}^2 - \sum\_{j \neq k}^{n} t\_{ki} t\_{ji} \right) \ge \varrho > 0, \qquad k = \overline{1, n}. \tag{5.14}$$


where the (*m* × *m*)-matrix

the (*m* × *n*)-matrix

and the *n* × *m*-matrix

rows of the matrix.

From this implies that

have:

where

It is known that

�*A*�*<sup>E</sup>* =

It is assumed that �*XU*� = �*XU*�2. We have from (5.19) the following equality:

*X***<sup>u</sup>** = diag [

� *∂λ<sup>j</sup> ∂ui*

Λ*<sup>U</sup>* =

*xmax* = max (*i*,**u**,**v**)

by **u**. We obtain the following matrix equation:

Here the (*n* × *n*)-matrix Φ*<sup>λ</sup>* has elements

*φλ ks* = −

Λ*<sup>U</sup>* = � *∂λ<sup>k</sup> ∂ui*

*m* ∑ *i*=1

*uitkitjs* exp

⎛ ⎝− � ∑ *i*,*j* |*aij*|2.

Entropic Image Restoration as a Dynamic System with Entropy Operator 63

�*A*�<sup>2</sup> ≤ �*A*�*<sup>E</sup>*

*xi ui*

*X<sup>λ</sup>* = −**x** ⊗ *T*�

In these expressions ⊗ is coordinate-wise multiplication of the vector's components to the

According to (5.21) and the relation between the spectral norm and the Euclidean norm, we

*xi*(**u**, **v**), *u*<sup>−</sup>

Now consider the equations (5.20), and differentiate the left and right sides of these equations


*n* ∑ *j*=1

�*X***u**�*<sup>E</sup>* <sup>≤</sup> <sup>√</sup>*<sup>m</sup>*

�*Xλ*�*<sup>E</sup>* <sup>≤</sup> *<sup>x</sup>max*�*T*�*<sup>E</sup>* <sup>=</sup> *<sup>x</sup>max*

, *j* = 1, *n*, *i* = 1, *m*

*xmax u*− *min*

*min* = min *i u*−

�

⎞

*λj*(**u**, **v**)*tji*

<sup>=</sup> <sup>−</sup>Φ−<sup>1</sup>

���� *<sup>m</sup>*,*<sup>n</sup>* ∑ *i*=1,*j*=1 *t* 2

Φ*<sup>λ</sup>* Λ*<sup>U</sup>* = −Φ**u**, (5.29)

*XU* = *X***<sup>u</sup>** + *X<sup>λ</sup>* Λ*U*, (5.21)

�

�*XU*�<sup>2</sup> ≤ �*X***u**�*<sup>E</sup>* + �*Xλ*�*<sup>E</sup>* �Λ*U*�*E*, (5.25)


; (5.23)

, (5.26)

*<sup>i</sup>* . (5.27)

*ij*. (5.28)

*<sup>λ</sup>* Φ**u**. (5.30)

⎠ , (*k*,*s*) = 1, *n*; (5.31)

(5.24)

The feasible set D = {**x** : *T* **x** = **v**, **x** ≥ **0**} is not empty, notable, there exists some subset of the nonnegative vectors **x** ∈ D.

Designate the Lipschitz-constant for the normalized operator (5.11) as *L*˜ *<sup>U</sup>* and *L*˜ *<sup>V</sup>* respectively, i.e.

$$\|\mathbf{x}[\mathbf{u}^{(1)}, \mathbf{v}^{(1)}] - \mathbf{x}[\mathbf{u}^{(2)}, \mathbf{v}^{(2)}]\| \le \tilde{L}\_{ll} \|\mathbf{u}^{(1)} - \mathbf{u}^{(2)}\| + \tilde{L}\_{V} \|\mathbf{v}^{(1)} - \mathbf{v}^{(2)}\|.\tag{5.15}$$

We will use the upper estimations of local Lipschitz-constant:

$$L\_{ll} \le \max\_{\mathcal{U}\_+^{\text{uv}}} \|X\_{ll}\|\_{\prime} \qquad L\_{V} = \max\_{V\_+^{\text{u}}} \|X\_V\|\_{\prime} \tag{5.16}$$

where *XU* and *XV* are the *U*-Jacobian and the *V*-Jacobian of the operator **x**[**u**, **v**] respectively.

According to (5.13), the following relation between the operators **y**(**u**, **v**) (5.4) and **x**(**u**, **v**) (5.11) exists:

$$\mathbf{y}(\mathbf{u}, \mathbf{v}) = \mathbf{t}^{-1} \otimes \mathbf{x}(\mathbf{u}, \mathbf{v}),\tag{5.17}$$

where the vector **t** <sup>−</sup><sup>1</sup> <sup>=</sup> {*<sup>t</sup>* −1 <sup>1</sup> ,..., *t* <sup>−</sup><sup>1</sup> *<sup>m</sup>* }, where the components *ti* are defined by the equalities (5.13), and ⊗ implies the coordinate-wise multiplication of the vectors.

Thus we have the following equalities:

$$L\_{\rm II} = \|\mathbf{t}^{-1}\| \tilde{L}\_{\rm II} \qquad L\_{\rm V} = \|\mathbf{t}^{-1}\| \tilde{L}\_{\rm V} \tag{5.18}$$

Thus, we will calculate the local Lipschitz-constants estimations for the normalized entropy operator (5.11, 5.12) and then apply the formulas (5.17, 5.18).

The normalized entropy operator **x**(**u**, **v**) can be represented by the form:

$$x\_i(\mathbf{u}, \mathbf{v}) = u\_i \exp\left(-\sum\_{j=1}^n \lambda\_j(\mathbf{u}, \mathbf{v}) \, t\_{ji}\right), \qquad i = \overline{1, m}, \tag{5.19}$$

where the Lagrange multipliers *λj*(**u**, **v**), (*j* = 1, *n*) as the implicit functions from **u**, **v** define by the equations:

$$\left[\Phi\_{\mathbf{k}}\left[\mathbf{u},\lambda\left(\mathbf{u},\mathbf{v}\right)\right] = \sum\_{i=1}^{m} u\_{i}t\_{ki}\exp\left(-\sum\_{j=1}^{n} \lambda\_{j}\left(\mathbf{u},\mathbf{v}\right)t\_{ji}\right) = v\_{k\prime} \quad k=\overline{1,n}.\tag{5.20}$$

**1. Estimations of the norm's matrix** *XU***.** The (*m* × *m*)-matrix *XU* takes a form:

$$\mathbf{X}\_{U} = \left[\frac{\partial \mathbf{x}\_{i}}{\partial \boldsymbol{\mu}\_{j}} \boldsymbol{\mu}\_{i}(\boldsymbol{i}, j) = \overline{\mathbf{1}\_{i} \boldsymbol{m}}\right] \boldsymbol{\mu}\_{i}$$

We will use Euclidean vector norm (�**y**�2), with which two matrix norm are consisted (see Voevodin (1984)):


$$\|A\|\_2 = \sqrt{\sigma\_{\text{max}}}$$

where *σmax* is the maximal eigenvalue of the matrix *A*;


$$\|A\|\_{E} = \sqrt{\sum\_{i,j} |a\_{ij}|^2}.$$

$$\|A\|\_2 \le \|A\|\_E$$

It is known that

18 Will-be-set-by-IN-TECH

The feasible set D = {**x** : *T* **x** = **v**, **x** ≥ **0**} is not empty, notable, there exists some subset of

Designate the Lipschitz-constant for the normalized operator (5.11) as *L*˜ *<sup>U</sup>* and *L*˜ *<sup>V</sup>* respectively,

]� ≤ *<sup>L</sup>*˜ *<sup>U</sup>* �**u**(1) <sup>−</sup> **<sup>u</sup>**(2)

�*XU*�, *<sup>L</sup>*˜ *<sup>V</sup>* <sup>=</sup> max

where *XU* and *XV* are the *U*-Jacobian and the *V*-Jacobian of the operator **x**[**u**, **v**] respectively. According to (5.13), the following relation between the operators **y**(**u**, **v**) (5.4) and **x**(**u**, **v**) (5.11)

<sup>−</sup>1�*L*˜ *<sup>U</sup>*, *LV* <sup>=</sup> �**<sup>t</sup>**

Thus, we will calculate the local Lipschitz-constants estimations for the normalized entropy

where the Lagrange multipliers *λj*(**u**, **v**), (*j* = 1, *n*) as the implicit functions from **u**, **v** define

*n* ∑ *j*=1

, (*i*, *j*) = 1, *m*

We will use Euclidean vector norm (�**y**�2), with which two matrix norm are consisted (see Voevodin

�*A*�<sup>2</sup> <sup>=</sup> <sup>√</sup>*σmax*,

*λj*(**u**, **v**)*tji*

⎞

*λj*(**u**, **v**)*tji*

� , ⎞

**y**(**u**, **v**) = **t**

(5.13), and ⊗ implies the coordinate-wise multiplication of the vectors.

The normalized entropy operator **x**(**u**, **v**) can be represented by the form:

⎛ ⎝−

*uitki* exp

**1. Estimations of the norm's matrix** *XU***.** The (*m* × *m*)-matrix *XU* takes a form:

� *∂xi ∂uj*

*XU* =

*n* ∑ *j*=1

⎛ ⎝−

*LU* = �**t**

operator (5.11, 5.12) and then apply the formulas (5.17, 5.18).

*xi*(**u**, **v**) = *ui* exp

*m* ∑ *i*=1 *V<sup>n</sup>* + � <sup>+</sup> *<sup>L</sup>*˜ *<sup>V</sup>* �**v**(1) <sup>−</sup> **<sup>v</sup>**(2)

<sup>−</sup><sup>1</sup> <sup>⊗</sup> **<sup>x</sup>**(**u**, **<sup>v</sup>**), (5.17)

<sup>−</sup><sup>1</sup> *<sup>m</sup>* }, where the components *ti* are defined by the equalities

�*XV*�, (5.16)

<sup>−</sup>1�*L*˜ *<sup>V</sup>*, (5.18)

⎠ , *i* = 1, *m*, (5.19)

⎠ = *vk*, *k* = 1, *n*. (5.20)

�. (5.15)

the nonnegative vectors **x** ∈ D.

�**x**[**u**(1)

, **v**(1)

<sup>−</sup><sup>1</sup> <sup>=</sup> {*<sup>t</sup>*

Φ*k*[**u**, *λ*(**u**, **v**)] =

where *σmax* is the maximal eigenvalue of the matrix *A*;

Thus we have the following equalities:

−1 <sup>1</sup> ,..., *t*

] <sup>−</sup> **<sup>x</sup>**[**u**(2)

We will use the upper estimations of local Lipschitz-constant:

*<sup>L</sup>*˜ *<sup>U</sup>* <sup>≤</sup> max *U<sup>m</sup>* +

, **v**(2)

i.e.

exists:

where the vector **t**

by the equations:

(1984)):


It is assumed that �*XU*� = �*XU*�2. We have from (5.19) the following equality:

$$X\_{\rm U} = X\_{\rm u} + X\_{\lambda} \,\Lambda\_{\rm U} \,\tag{5.21}$$

where the (*m* × *m*)-matrix

$$X\_{\mathbf{u}} = \text{diag}\left[\frac{\varkappa\_{\mathbf{i}}}{\varkappa\_{\mathbf{i}}} \,|\,\mathbf{i} = \overline{1,m}\right];\tag{5.22}$$

the (*m* × *n*)-matrix

$$X\_{\lambda} = -\mathbf{x} \otimes T';\tag{5.23}$$

and the *n* × *m*-matrix

$$
\Lambda\_{ll} = \left[\frac{\partial \lambda\_j}{\partial u\_i}, j = \overline{1, n}, \, i = \overline{1, m}\right] \tag{5.24}
$$

In these expressions ⊗ is coordinate-wise multiplication of the vector's components to the rows of the matrix.

According to (5.21) and the relation between the spectral norm and the Euclidean norm, we have:

$$\|\|X\_{\mathcal{U}}\|\|\_{2} \le \|\|X\_{\mathbf{u}}\|\|\_{E} + \|\|X\_{\lambda}\|\|\_{E} \|\|\Lambda\_{\mathcal{U}}\|\|\_{E} \tag{5.25}$$

where

$$\|X\_{\mathbf{u}}\|\_{E} \le \sqrt{m} \frac{\mathbf{x}^{\max}}{u\_{\min}^{-}} \text{.} \tag{5.26}$$

$$\mathbf{x}^{\text{max}} = \max\_{(i,\mathbf{u},\mathbf{v})} \mathbf{x}\_i(\mathbf{u}, \mathbf{v}), \ u\_{\text{min}}^{-} = \min\_i \ u\_i^{-}. \tag{5.27}$$

$$\|\|X\_{\lambda}\|\|\_{E} \le \mathfrak{x}^{\max} \|T\|\|\_{E} = \mathfrak{x}^{\max} \sqrt{\sum\_{i=1, j=1}^{m,n} t\_{ij}^{2}}.\tag{5.28}$$

Now consider the equations (5.20), and differentiate the left and right sides of these equations by **u**. We obtain the following matrix equation:

$$
\Phi\_{\lambda} \Lambda\_{U} = -\Phi\_{\mathbf{u}\nu} \tag{5.29}
$$

From this implies that

$$
\Lambda\_{\rm II} = \left[ \frac{\partial \lambda\_k}{\partial u\_i} \mid k = \overline{1, n}, \ i = \overline{1, m} \right] = -\Phi\_{\lambda}^{-1} \Phi\_{\mathbf{u}}.\tag{5.30}
$$

Here the (*n* × *n*)-matrix Φ*<sup>λ</sup>* has elements

$$\phi\_{\rm ks}^{\lambda} = -\sum\_{i=1}^{m} u\_i t\_{\rm ki} t\_{\rm js} \exp\left(-\sum\_{j=1}^{n} \lambda\_j(\mathbf{u}, \mathbf{v}) t\_{\rm ji}\right), \qquad (k, s) = \overline{1, n};\tag{5.31}$$

Thus the norm's estimation of the matrix *XV* takes a form:

well as for the norm's estimation of the matrix *XU* . **3. Estimation of the spectral norm of the matrix** Φ−<sup>1</sup>

maximum eigenvalue *μmax* of the initial matrix, i.e.

<sup>−</sup> *<sup>g</sup>*<sup>+</sup>

where

where according to (5.19)

(see Wilkinson (1970)). We will order them in the following way:

one of the intervals with center −*ck*(*λ*) and the width 2 *ρk*(*λ*):

*g*+ *<sup>k</sup>* (*λ*) =

*g*− *<sup>k</sup>* (*λ*) =

From the conditions (5.46) it follows that

Thus, in a view of (5.28), we have

*<sup>k</sup>* (*λ*) = −*ck*(*λ*) − *ρk*(*λ*) ≤ *μ* ≤ −*ck*(*λ*) + *ρk*(*λ*) = −*g*<sup>−</sup>


min *<sup>k</sup>*,*<sup>λ</sup> <sup>g</sup>*<sup>−</sup>

We can apply the lower estimation for the left side of this interval using (5.14):

�Φ−<sup>1</sup>

�*XV*�<sup>2</sup> ≤ �Φ−<sup>1</sup>

*<sup>λ</sup>* �<sup>2</sup> *<sup>x</sup>max*

Entropic Image Restoration as a Dynamic System with Entropy Operator 65

So, we can see that it is necessary to construct the norm's estimation for the matrix Φ−<sup>1</sup>

strictly negative defined for all *λ*. Therefore, it has *n* real, various, and negative eigenvalues

The spectral norm of an inverse matrix is equal to the inverse value of the *modulus* of the

To definite the value *M* we resort to the Gershgorin theorem (see Wilkinson (1970)). According to the theorem any eigenvalue of a symmetric strictly negative definite matrix lies at least in

�Φ−<sup>1</sup>

*m* ∑ *i*=1 *xi* ⎛ ⎝*t* 2 *ki* + *n* ∑ *j*�=*k*

*m* ∑ *i*=1 *xi* ⎛ ⎝*t* 2 *ki* −

*<sup>k</sup>*,*<sup>λ</sup> <sup>g</sup>*<sup>−</sup>

*xmin* = min (*i*,**u**,**v**)

*<sup>k</sup>* (*λ*), max

���� *<sup>m</sup>*,*<sup>n</sup>* ∑ *i*=1,*j*=1 *t* 2

*μ*<sup>1</sup> = *μmin* < *μ*<sup>2</sup> < ··· < *μ<sup>n</sup>* = *μmax* < 0, |*μmax*| > *M*. (5.44)

*tkitji*

*tkitji*

*n* ∑ *j*�=*k*

*<sup>k</sup>*,*<sup>λ</sup> <sup>g</sup>*<sup>+</sup>

⎞ ⎠ ,

⎞ ⎠ ,

*<sup>k</sup>* (*λ*) <sup>≥</sup> *<sup>M</sup>* <sup>=</sup> *� <sup>x</sup>min*, (5.49)

*<sup>λ</sup>* �<sup>2</sup> <sup>≤</sup> (*� xmin*)<sup>−</sup>1. (5.51)

*xi*(**u**, **v**). (5.50)

*ij*. (5.43)

*<sup>k</sup>* (*λ*), *k* = 1, *n*, (5.46)

*<sup>k</sup>* (*λ*)]. (5.48)

*<sup>λ</sup>* . The matrix Φ*<sup>λ</sup>* (5.31) is symmetric and

*<sup>λ</sup>* � ≤ *<sup>M</sup>*<sup>−</sup>1. (5.45)

*<sup>λ</sup>* , as

(5.47)

and the (*n* × *m*)-matrix Φ**<sup>u</sup>** has elements

$$\boldsymbol{\phi}\_{ki}^{\mathbf{u}} = t\_{ki} \exp\left(-\sum\_{j=1}^{n} \lambda\_j(\mathbf{u}, \mathbf{v}) t\_{ji}\right), \qquad k = \overline{1, n}, \ i = \overline{1, m}. \tag{5.32}$$

According to (5.30) we have

$$\|\Lambda\_{ll}\|\_{2} \le \|\Phi\_{\lambda}^{-1}\|\_{2} \frac{\varkappa^{max}}{\varkappa\_{min}^{-}} \sqrt{\sum\_{j=1, i=1}^{n,m} t\_{ji}^{2}}.\tag{5.33}$$

Thus the norm's estimation of the matrix *XU* takes a form:

$$\|\|\mathbf{X}\_{II}\|\|\_{2} \le \frac{\mathbf{x}^{\max}}{\mathfrak{u}\_{min}^{-}} \left(\sqrt{m} + \mathbf{x}^{\max} \, \|\|\boldsymbol{\Phi}\_{\boldsymbol{\lambda}}^{-1}\|\|\_{2} \sum\_{i=1, j=1}^{m,n} t\_{ij}^{2}\right). \tag{5.34}$$

**2. Estimations of the norm's matrix** *XV***.**The (*m* × *n*)-matrix *XV* takes a form

$$X\_V = \left[\frac{\partial x\_i}{\partial v\_k} \; ; \; i = \overline{1, m} \; k = \overline{1, n}\right] . .$$

It is assumed that �*XV*� = �*XV*�2. We have from (5.19) the following equality:

$$X\_{\rm II} = X\_{\rm A} \,\, \Lambda\_{V'} \tag{5.35}$$

where the (*m* × *n*)-matrix

$$X\_{\lambda} = -\mathbf{x} \otimes T';\tag{5.36}$$

and the *n* × *n*-matrix

$$
\Lambda\_V = \left[\frac{\partial \lambda\_k}{\partial v\_j}, (k, j) = \overline{1, n}\right]. \tag{5.37}
$$

According to (5.36) and the relation between the spectral norm and the Euclidean norm, we have:

$$\|\|X\_V\|\|\_2 \le \|\|X\_\lambda\|\|\_E \|\|\Lambda\_V\|\|\_{E'} \tag{5.38}$$

where

$$\|\|X\_{\lambda}\|\|\_{E} \le \mathfrak{x}^{\max} \|\|T\|\|\_{E} = \mathfrak{x}^{\max} \sqrt{\sum\_{i=1, j=1}^{m,n} t\_{ij}^{2}}.\tag{5.39}$$

Now consider the equations (5.20), and differentiate the left and right sides of these equations by **v**. We obtain the following matrix equation:

$$
\Phi\_{\lambda} \, \Lambda\_V = I. \tag{5.40}
$$

From this implies that

$$
\Lambda\_{\rm II} = \Phi\_{\lambda}^{-1}.\tag{5.41}
$$

Here the (*n* × *n*)-matrix Φ*<sup>λ</sup>* is defined by (5.31). According to (5.41) we have

$$\|\Lambda\_V\|\_2 \le \|\Phi\_\lambda^{-1}\|\_2. \tag{5.42}$$

Thus the norm's estimation of the matrix *XV* takes a form:

$$\|\|X\_V\|\|\_2 \le \|\Phi\_\lambda^{-1}\|\|\_2 \ge^{\max} \sqrt{\sum\_{i=1, j=1}^{m,n} t\_{ij}^2}. \tag{5.43}$$

So, we can see that it is necessary to construct the norm's estimation for the matrix Φ−<sup>1</sup> *<sup>λ</sup>* , as well as for the norm's estimation of the matrix *XU* .

**3. Estimation of the spectral norm of the matrix** Φ−<sup>1</sup> *<sup>λ</sup>* . The matrix Φ*<sup>λ</sup>* (5.31) is symmetric and strictly negative defined for all *λ*. Therefore, it has *n* real, various, and negative eigenvalues (see Wilkinson (1970)). We will order them in the following way:

$$
\mu\_1 = \mu\_{\min} < \mu\_2 < \dots < \mu\_n = \mu\_{\max} < 0,\qquad |\mu\_{\max}| > M. \tag{5.44}
$$

The spectral norm of an inverse matrix is equal to the inverse value of the *modulus* of the maximum eigenvalue *μmax* of the initial matrix, i.e.

$$\|\Phi\_{\lambda}^{-1}\| \le M^{-1}.\tag{5.45}$$

To definite the value *M* we resort to the Gershgorin theorem (see Wilkinson (1970)). According to the theorem any eigenvalue of a symmetric strictly negative definite matrix lies at least in one of the intervals with center −*ck*(*λ*) and the width 2 *ρk*(*λ*):

$$-\mathsf{g}\_{k}^{+}(\lambda) = -\mathsf{c}\_{k}(\lambda) - \rho\_{k}(\lambda) \le \mu \le -\mathsf{c}\_{k}(\lambda) + \rho\_{k}(\lambda) = -\mathsf{g}\_{k}^{-}(\lambda), \qquad k = \overline{1, n}, \tag{5.46}$$

where according to (5.19)

$$\begin{aligned} \mathbf{g}\_k^+ (\lambda) &= \sum\_{i=1}^m \mathbf{x}\_i \left( t\_{ki}^2 + \sum\_{j \neq k}^n t\_{ki} t\_{ji} \right), \\ \mathbf{g}\_k^- (\lambda) &= \sum\_{i=1}^m \mathbf{x}\_i \left( t\_{ki}^2 - \sum\_{j \neq k}^n t\_{ki} t\_{ji} \right), \end{aligned} \tag{5.47}$$

From the conditions (5.46) it follows that

$$|\mu\_{\max}| \in [\min\_{k,\lambda} g\_k^{-}(\lambda) \text{ } \max\_{k,\lambda} g\_k^{+}(\lambda)]. \tag{5.48}$$

We can apply the lower estimation for the left side of this interval using (5.14):

$$\min\_{k,\lambda} g\_k^{-}(\lambda) \ge M = \varrho \ge \mathbf{x}^{\min},\tag{5.49}$$

where

20 Will-be-set-by-IN-TECH

⎞

*xmax u*− *min* ���� *<sup>n</sup>*,*<sup>m</sup>* ∑ *j*=1,*i*=1 *t* 2

*λ* �2

*m*,*n* ∑ *i*=1,*j*=1 *t* 2 *ij* ⎞

� .

*XU* = *X<sup>λ</sup>* Λ*V*, (5.35)

; (5.36)

. (5.37)

*ij*. (5.39)

⎠ , *k* = 1, *n*, *i* = 1, *m*. (5.32)

*ji*. (5.33)

⎠ . (5.34)

*λj*(**u**, **v**)*tji*

*λ* �2

<sup>⎝</sup>√*<sup>m</sup>* <sup>+</sup> *<sup>x</sup>max* �Φ−<sup>1</sup>

*X<sup>λ</sup>* = −**x** ⊗ *T*�

According to (5.36) and the relation between the spectral norm and the Euclidean norm, we

Now consider the equations (5.20), and differentiate the left and right sides of these equations

Λ*<sup>U</sup>* = Φ−<sup>1</sup>

�Λ*V*�<sup>2</sup> ≤ �Φ−<sup>1</sup>

, *i* = 1, *m*, *k* = 1, *n*

, (*k*, *j*) = 1, *n*

�

���� *<sup>m</sup>*,*<sup>n</sup>* ∑ *i*=1,*j*=1 *t* 2

�*XV*�<sup>2</sup> ≤ �*Xλ*�*<sup>E</sup>* �Λ*V*�*E*, (5.38)

Φ*<sup>λ</sup>* Λ*<sup>V</sup>* = *I*. (5.40)

*<sup>λ</sup>* . (5.41)

*<sup>λ</sup>* �2. (5.42)

and the (*n* × *m*)-matrix Φ**<sup>u</sup>** has elements

*φ***u**

According to (5.30) we have

where the (*m* × *n*)-matrix

and the *n* × *n*-matrix

From this implies that

have:

where

*ki* = *tki* exp

Thus the norm's estimation of the matrix *XU* takes a form:

�*XU*�<sup>2</sup> ≤

by **v**. We obtain the following matrix equation:

⎛ ⎝−

*n* ∑ *j*=1

�Λ*U*�<sup>2</sup> ≤ �Φ−<sup>1</sup>

⎛

**2. Estimations of the norm's matrix** *XV***.**The (*m* × *n*)-matrix *XV* takes a form

� *∂xi ∂vk*

It is assumed that �*XV*� = �*XV*�2. We have from (5.19) the following equality:

� *∂λ<sup>k</sup> ∂vj*

�*Xλ*�*<sup>E</sup>* <sup>≤</sup> *<sup>x</sup>max* �*T*�*<sup>E</sup>* <sup>=</sup> *<sup>x</sup>max*

Here the (*n* × *n*)-matrix Φ*<sup>λ</sup>* is defined by (5.31). According to (5.41) we have

Λ*<sup>V</sup>* =

*xmax u*− *min*

*XV* =

$$\mathbf{x}^{\min} = \min\_{(i,\mathbf{u},\mathbf{v})} \boldsymbol{\pi}\_i(\mathbf{u}, \mathbf{v}). \tag{5.50}$$

Thus, in a view of (5.28), we have

$$\|\boldsymbol{\Phi}\_{\boldsymbol{\lambda}}^{-1}\|\_{2} \leq \left(\boldsymbol{\varrho}\,\boldsymbol{\chi}\_{\text{min}}\right)^{-1}.\tag{5.51}$$

We can see that the elements of these Jacobians are nonnegative for all **z** ≥ **0** and **u** ∈

Entropic Image Restoration as a Dynamic System with Entropy Operator 67

The elements of the matrix *P***<sup>z</sup>** and *P***<sup>u</sup>** are nonnegative. Thus, the function Ψ(**z**, **u**) increase in

According to (5.55) the function **x**(**z**, **u**) is analytical one. The system of the equations define

**1. Estimation of the minimum value of the normalized** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.**

Ψ(**z**, **u**−) = **v**−,

*Proof.* According to (5.55) **<sup>x</sup>**(**z**, **<sup>0</sup>**) = 0 and **<sup>x</sup>**(*z*1,..., *zi*−1, 0, *zi*+1,..., *zn*; **<sup>u</sup>**) = 0. As the function (5.55) increases in a monotone way and analytical one, then *xmin* = min*<sup>i</sup> xi*(**u**−, *zmin*) for

**u**) = **0**. As the function Ψ(**z**, **u**) increase in a monotone way and analytical one, then the

**2. Estimation of the maximum value of the normalized** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.** This problem is more complicated then the previous one. So, at the beginning we describe the

On the first step we reduce the equations (5.56) to the equations with a monotone operator,

+(**u**−, **<sup>u</sup>**+).

*z*˜*j*, *j* = 1, *n*,

+(**u**−, **<sup>u</sup>**+), where **<sup>u</sup>**<sup>−</sup> <sup>&</sup>gt; **<sup>0</sup>**, **<sup>u</sup>**<sup>+</sup> <sup>≤</sup> **<sup>1</sup>**. Thus the functions **<sup>x</sup>**(**z**, **<sup>u</sup>**) increase in a monotone

*<sup>j</sup>* ≥ 0, (*k*, *l*) = 1, *n*; (5.59)

*<sup>j</sup>* ≥ 0, *k* = 1, *n*, *s* = 1, *m*. (5.60)

+(**u**−, **<sup>u</sup>**+).

+(**u**−, **<sup>u</sup>**+) and *<sup>V</sup><sup>n</sup>*

+(**v**−, **<sup>v</sup>**+) (see

*U<sup>m</sup>*

and

way on these sets.



Now consider the function Ψ(**z**, **u**) and its the Jacobians:

*m* ∑ *i*=1

the unique differentiable implicit function **z**(**u**, **v**) on the sets *U<sup>m</sup>*

theorem 5, pp. 91-92; theorems 1, 2, pp. 95-96, Popkov (1995)).

**Theorem 2.***Let the matrix T (5.11) has a full rank and* **<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>*

*and z*˜1,..., *z*˜*<sup>n</sup> are the components of the solution of the equation*

*and the vectors* **u**−, **v**− *have enough small components.*

*uitki* 1 *zl*

*n* ∏ *j*=1 *z tjs*

The solution of the problem can be represented by the following theorem.

*zmin* = min *j*

Consider the equations (5.56). We have <sup>Ψ</sup>(**z**, **<sup>0</sup>**), = <sup>Ψ</sup>(*z*1,..., *zi*−1, 0, *zi*+1,..., *zn*;

*n* ∏ *j*=1 *z tji*

*p***z** *kl* =

> *p***u** *ks* = *tks*

a monotone way on the sets **<sup>z</sup>** <sup>≥</sup> **<sup>0</sup>** and **<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>*

*Then xmin* = min*<sup>i</sup> xi*(**u**−, *zmin*), *where:*

enough small components **u**−.

proposition of the theorem is valid.

general procedure of the estimation forming.

which also depends on the variable **z** and parameters **u**, **v**.

**4. Estimation of the local Lipschitz-constants.** According to (5.34) and (5.51), the estimation of the local *U*-Lipshitz-constant for the normalized entropy operator (5.19) takes a form:

$$\tilde{L}\_{II} \le \frac{\mathfrak{x}^{\max}}{\mathfrak{u}\_{min}^{-}} \left( \sqrt{m} + \frac{\mathfrak{x}^{\max}}{\mathfrak{q}\mathfrak{x}^{\min}} \sum\_{i=1, j=1}^{m,n} t\_{ij}^{2} \right) . \tag{5.52}$$

The estimation (5.43) of the local *V*-Lipschitz-constant for the normalized entropy operator (5.19) takes a form:

$$\tilde{L}\_V \le \frac{\mathfrak{x}^{\max}}{\mathfrak{x}^{\min}} \sqrt{\sum\_{i=1, j=1}^{m, n} t\_{ij}^2}. \tag{5.53}$$

Using the links (5.18) between the normalized entropy operator (5.19) and the entropy operator (5.4) we will have:

$$L\_{II} \le \sqrt{\sum\_{i=1}^{m} \left(\sum\_{k=1}^{n} t\_{ki}\right)^2} \frac{\mathbf{x}^{max}}{\mathbf{u}\_{min}^{-}} \left(\sqrt{m} + \frac{\mathbf{x}^{max}}{\varrho \mathbf{x}^{min}} \sum\_{i=1, j=1}^{m,n} t\_{ij}^2\right),$$

$$L\_V \le \sqrt{\sum\_{i=1}^{m} \left(\sum\_{k=1}^{n} t\_{ki}\right)^2 \frac{\mathbf{x}^{max}}{(\mathbf{x}^{min} \varrho)}} \sqrt{\sum\_{i=1, j=1}^{m,n} t\_{ij}^2}. \tag{5.54}$$

#### **5.3 Boundedness of the normalized entropy operator**

Let us consider the normalized entropy operator (5.11, 5.19, 5.20), the parameters of which **<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>* +(**u**−, **<sup>u</sup>**+) and , **<sup>v</sup>** <sup>∈</sup> *<sup>V</sup><sup>n</sup>* +(**v**−, **<sup>v</sup>**+).

Rewrite the equations (5.19, 5.20) in respect to the exponential Lagrange multipliers *zj* = exp(−*λj*):

$$\mathbf{x}\_{i}(\mathbf{z}, \mathbf{u}) = u\_{i} \prod\_{j=1}^{n} z\_{j}^{t\_{ji}} \qquad \overline{1, m} \tag{5.55}$$

$$\Psi\_k[\mathbf{z}, \mathbf{u}] = \sum\_{i=1}^m t\_{ki} u\_i \prod\_{j=1}^n z\_j^{t\_{ji}} = v\_{k\prime} \qquad \mathbf{z} \ge 0, \quad k = \overline{1\,\!/\, n}. \tag{5.56}$$

It is known some properties of the operator (5.55, 5.56) are defined by the Jacobians of the functions **x**(**z**, **u**) and Φ(**z**, **u**) in respect to the variables **z**, **u**.

Consider the function **x**(**z**, **u**). We have the Jacobians:


$$\mathbf{g\_{ik}^{x}} = \boldsymbol{u}\_{i}\mathbf{t}\_{ki}\frac{1}{z\_{k}}\prod\_{j=1}^{n}\boldsymbol{z}\_{j}^{t\_{ji}} \ge 0, \qquad i = \overline{1,m}, \; k = \overline{1,n};\tag{5.57}$$

and


$$\mathbf{g}\_{\rm is}^{\rm u} = \prod\_{j=1}^{n} z\_j^{t\_{j\parallel}} \ge 0, \qquad (i, \mathbf{s}) = \overline{1, m} \tag{5.58}$$

We can see that the elements of these Jacobians are nonnegative for all **z** ≥ **0** and **u** ∈ *U<sup>m</sup>* +(**u**−, **<sup>u</sup>**+), where **<sup>u</sup>**<sup>−</sup> <sup>&</sup>gt; **<sup>0</sup>**, **<sup>u</sup>**<sup>+</sup> <sup>≤</sup> **<sup>1</sup>**. Thus the functions **<sup>x</sup>**(**z**, **<sup>u</sup>**) increase in a monotone way on these sets.

Now consider the function Ψ(**z**, **u**) and its the Jacobians:


$$p\_{kl}^{\mathbf{z}} = \sum\_{l=1}^{m} u\_l t\_{kl} \frac{1}{z\_l} \prod\_{j=1}^{n} z\_j^{t\_{jl}} \ge 0, \qquad (k,l) = \overline{1,n};\tag{5.59}$$

and

22 Will-be-set-by-IN-TECH

**4. Estimation of the local Lipschitz-constants.** According to (5.34) and (5.51), the estimation of the local *U*-Lipshitz-constant for the normalized entropy operator (5.19) takes a form:

The estimation (5.43) of the local *V*-Lipschitz-constant for the normalized entropy operator

Using the links (5.18) between the normalized entropy operator (5.19) and the entropy

⎛ <sup>⎝</sup>√*<sup>m</sup>* <sup>+</sup>

*xmax* (*xmin �*)

Let us consider the normalized entropy operator (5.11, 5.19, 5.20), the parameters of which

Rewrite the equations (5.19, 5.20) in respect to the exponential Lagrange multipliers *zj* =

*n* ∏ *j*=1 *z tji*

It is known some properties of the operator (5.55, 5.56) are defined by the Jacobians of the

*xmax xmin �*

*xmax �xmin*

���� *<sup>m</sup>*,*<sup>n</sup>* ∑ *i*=1,*j*=1 *t* 2

*m*,*n* ∑ *i*=1,*j*=1 *t* 2 *ij* ⎞

> *xmax �xmin*

���� *<sup>m</sup>*,*<sup>n</sup>* ∑ *i*=1,*j*=1 *t* 2

*m*,*n* ∑ *i*=1,*j*=1 *t* 2 *ij* ⎞ ⎠ ,

⎠ . (5.52)

*ij*. (5.53)

*ij*. (5.54)

*<sup>j</sup>* , 1, *m*, (5.55)

*<sup>j</sup>* = *vk*, **z** ≥ 0, *k* = 1, *n*. (5.56)

*<sup>j</sup>* ≥ 0, *i* = 1, *m*, *k* = 1, *n*; (5.57)

*<sup>j</sup>* ≥ 0, (*i*,*s*) = 1, *m* (5.58)

*<sup>L</sup>*˜ *<sup>U</sup>* <sup>≤</sup>

(5.19) takes a form:

**<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>*

exp(−*λj*):

and



operator (5.4) we will have:

*LU* ≤

*LV* ≤

+(**u**−, **<sup>u</sup>**+) and , **<sup>v</sup>** <sup>∈</sup> *<sup>V</sup><sup>n</sup>*

���� *m* ∑ *i*=1

���� *m* ∑ *i*=1

**5.3 Boundedness of the normalized entropy operator**

Ψ*k*[**z**, **u**] =

� *n* ∑ *k*=1 *tki* �<sup>2</sup> *xmax u*− *min*

� *n* ∑ *k*=1 *tki* �<sup>2</sup>

+(**v**−, **<sup>v</sup>**+).

*m* ∑ *i*=1 *tki ui*

> 1 *zk*

*g***u** *is* =

*n* ∏ *j*=1 *z tji*

*n* ∏ *j*=1 *z tji*

functions **x**(**z**, **u**) and Φ(**z**, **u**) in respect to the variables **z**, **u**.

Consider the function **x**(**z**, **u**). We have the Jacobians:

*g***z** *ik* = *uitki*

*xi*(**z**, **u**) = *ui*

*n* ∏ *j*=1 *z tji*

*xmax u*− *min*

*<sup>L</sup>*˜ *<sup>V</sup>* <sup>≤</sup>

⎛ <sup>⎝</sup>√*<sup>m</sup>* <sup>+</sup>


$$p\_{\rm ks}^{\rm u} = t\_{\rm ks} \prod\_{j=1}^{n} z\_j^{t\_{j\rm s}} \ge 0, \qquad k = \overline{1, n}, \ s = \overline{1, m}. \tag{5.60}$$

The elements of the matrix *P***<sup>z</sup>** and *P***<sup>u</sup>** are nonnegative. Thus, the function Ψ(**z**, **u**) increase in a monotone way on the sets **<sup>z</sup>** <sup>≥</sup> **<sup>0</sup>** and **<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>* +(**u**−, **<sup>u</sup>**+).

According to (5.55) the function **x**(**z**, **u**) is analytical one. The system of the equations define the unique differentiable implicit function **z**(**u**, **v**) on the sets *U<sup>m</sup>* +(**u**−, **<sup>u</sup>**+) and *<sup>V</sup><sup>n</sup>* +(**v**−, **<sup>v</sup>**+) (see theorem 5, pp. 91-92; theorems 1, 2, pp. 95-96, Popkov (1995)).

**1. Estimation of the minimum value of the normalized** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.** The solution of the problem can be represented by the following theorem.

**Theorem 2.***Let the matrix T (5.11) has a full rank and* **<sup>u</sup>** <sup>∈</sup> *<sup>U</sup><sup>m</sup>* +(**u**−, **<sup>u</sup>**+).

*Then xmin* = min*<sup>i</sup> xi*(**u**−, *zmin*), *where:*

$$z^{\min} = \min\_{j} \tilde{z}\_{j\prime} \qquad j = \overline{1\prime n}\_{\prime\prime}$$

*and z*˜1,..., *z*˜*<sup>n</sup> are the components of the solution of the equation*

$$\Psi(\mathbf{z}, \mathbf{u}^-) = \mathbf{v}^- \,,$$

*and the vectors* **u**−, **v**− *have enough small components.*

*Proof.* According to (5.55) **<sup>x</sup>**(**z**, **<sup>0</sup>**) = 0 and **<sup>x</sup>**(*z*1,..., *zi*−1, 0, *zi*+1,..., *zn*; **<sup>u</sup>**) = 0. As the function (5.55) increases in a monotone way and analytical one, then *xmin* = min*<sup>i</sup> xi*(**u**−, *zmin*) for enough small components **u**−.

Consider the equations (5.56). We have <sup>Ψ</sup>(**z**, **<sup>0</sup>**), = <sup>Ψ</sup>(*z*1,..., *zi*−1, 0, *zi*+1,..., *zn*;

**u**) = **0**. As the function Ψ(**z**, **u**) increase in a monotone way and analytical one, then the proposition of the theorem is valid.

**2. Estimation of the maximum value of the normalized** �*HD*, *B*, *Eq*, *Plh*�**-entropy operator.** This problem is more complicated then the previous one. So, at the beginning we describe the general procedure of the estimation forming.

On the first step we reduce the equations (5.56) to the equations with a monotone operator, which also depends on the variable **z** and parameters **u**, **v**.

where the vector **z***max* has the components *zmax*.

where the matrix C has the elements

as the matrix *T* has the full rank, and

and *Mkj* is the (*k*, *j*)-minor of the matrix *C*. Introduce the following polyhedral sets:

where

where

5.62). We use the following inequality Bellman (1961): *n* ∏ *j*=1 *h αj j* ≤

Then for the operator (5.62) the following estimate is valid:

*ckj* <sup>=</sup> max **<sup>u</sup>**

It is follows from (5.68) that the matrix *C* takes a form:

*n* ∑ *j*=1

> *m* ∑ *i*=1

*uitki tji* =

Thus, we can consider in the capacity of ˆ**z** the nonnegative solution of the equation:

The general solution of the equation (5.70) can be represented in the following form:

*n* ∑ *j*=1

*<sup>z</sup>*ˆ*k*(**v**) = det *<sup>C</sup><sup>k</sup>*

det *C<sup>k</sup>* =

⎧ ⎨ ⎩**v** :

⎧ ⎨ ⎩**v** :

*n* ∑ *j*=1

*n* ∑ *j*=1 *akjvj* ≥ 0, *k* = 1, *n*

*akjvj* < 0, *k* = 1, *n*

*W*+ =

*W*<sup>−</sup> =

For realization of the this way it is necessary to construct the *majorant* of the operator (5.61,

Entropic Image Restoration as a Dynamic System with Entropy Operator 69

*αjhj*, (*αj*, *hj*) ≥ 0,

*m* ∑ *i*=1

*C* = *T T*�

*n* ∑ *j*=1

*<sup>A</sup>*(**z**, **<sup>u</sup>**, **<sup>v</sup>**) <sup>≤</sup> **<sup>v</sup>**−<sup>1</sup> <sup>⊗</sup> *<sup>C</sup>* **<sup>z</sup>**, (5.67)

*C***z** = **v**, **z** ≥ **0**. (5.70)

det *<sup>C</sup>* <sup>≥</sup> 0, *<sup>k</sup>* <sup>=</sup> 1, *<sup>n</sup>*, (5.71)

det *C* �= 0, (5.72)

*akjvj*, *k* = 1, *n*, (5.73)

*akj* = (−1)(*k*+*j*)*Mkj*, (5.74)

⎫ ⎬ ⎭ ,

⎫ ⎬

<sup>⎭</sup> . (5.75)

*tki tji*, (*k*, *j*) = 1, *n*. (5.68)

, (5.69)

*α<sup>j</sup>* = 1. (5.66)

On the second step we define the variable **z**<sup>0</sup> < **z***min*, where the vector **z***min* has the components *<sup>z</sup>min* (theorem 2). The vector **<sup>z</sup>**<sup>0</sup> <sup>=</sup> {*z*0,..., *<sup>z</sup>*0} such that the values of the monotone operator at the point **z**<sup>0</sup> is more or equal to **z**0.

On the third, we define the vector **<sup>z</sup>***max* <sup>=</sup> {*zmax*,..., *<sup>z</sup>max*} such that the monotone operator is less then **z***max*. For determination of the **z***max* we use the majorant of the monotone operator.

Consider each of the steps in detail.

*2.1. Transformation of the equations (5.56).* Introduce the monotone increasing operator *A*(**z**, **u**, **v**) with the components:

$$A\_k(\mathbf{z}, \mathbf{u}, \mathbf{v}) = \frac{z\_k}{v\_k} \Psi\_k[\mathbf{z}, \mathbf{u}], \qquad k = \overline{1, n}. \tag{5.61}$$

Represent the equations (5.56) in the form:

$$A(\mathbf{z}, \mathbf{u}, \mathbf{v}) = \mathbf{z}.\tag{5.62}$$

This equation has the unique zero-solution **z**∗[**u**, **v**] ≡ **0** and the unique nonnegative solution **z**∗[**u**, **v**] ≥ **0**. Also recall that the elements of the matrix *T* and of the vector **v** in (5.39) are nonnegative.

*2.2. Choice z*0*.* According to the theorem 2 ˜**z** is the solution of the equation (5.62) for **u**−, **v**−. So,

$$\frac{\partial}{\partial z\_j} A\_k(\mathbf{z}, \mathbf{u}^-, \mathbf{v}^-) \mid\_{\mathbf{z}} < 1,$$

It is follows that there exists the vector

$$\mathbf{z}^0 = \mathbf{\tilde{z}} - \mathbf{\varepsilon},\tag{5.63}$$

where *ε* is a vector with small components *ε<sup>k</sup>* > 0, such that in the *ε*-neighborhood ˜**z** is valid the following inequality:

$$A(\mathbf{z}^0, \mathbf{u}^-, \mathbf{v}^- - \varepsilon) > \mathbf{z}^0. \tag{5.64}$$

*2.3. Choice zmax*. Exact value of *zmax* is defined by the solution of the global optimization problem Strongin & Sergeev (2000):

$$z^{\max} = \arg \max\_{\mathbf{u} \in U^{\mathbf{u}}\_+, \mathbf{v} \in V^{\mathbf{u}}\_+, j \in [1, n]} z^\*\_j(\mathbf{u}, \mathbf{v})\_{\forall}$$

where **z**∗(**u**, **v**) is a solution of the equation:

$$
\Psi(\mathbf{z}, \mathbf{u}) = \mathbf{v}.
$$

However this problem is very complicated. So we will calculate an upper estimation of the value *zmax*.

Let us assume that we can find the vector ˆ**z** such that

$$A(\mathbf{\hat{z}}, \mathbf{u}, \mathbf{v}) \le \mathbf{\hat{z}}.$$

Choice *zmax* is equal to max*<sup>j</sup> z*ˆ*j*. Then the nonzero-solution **z**<sup>∗</sup> of the equation (5.62) will belong to the following vector interval (see Krasnoselskii et al. (1969)):

$$\mathbf{z}\_{\rm min} \le \mathbf{z}^\* \le \mathbf{z}\_{\rm max} \tag{5.65}$$

where the vector **z***max* has the components *zmax*.

For realization of the this way it is necessary to construct the *majorant* of the operator (5.61, 5.62). We use the following inequality Bellman (1961):

$$\prod\_{j=1}^{n} h\_j^{a\_j} \le \sum\_{j=1}^{n} a\_j h\_j, \qquad (a\_j, h\_j) \ge 0, \quad \sum\_{j=1}^{n} a\_j = 1. \tag{5.66}$$

Then for the operator (5.62) the following estimate is valid:

$$A(\mathbf{z}, \mathbf{u}, \mathbf{v}) \le \mathbf{v}^{-1} \otimes \mathbb{C}\mathbf{z}\_{\prime} \tag{5.67}$$

where the matrix C has the elements

$$\mathcal{L}\_{k\bar{j}} = \max\_{\mathbf{u}} \sum\_{i=1}^{m} u\_i t\_{ki} \, t\_{\bar{j}i} = \sum\_{i=1}^{m} t\_{ki} \, t\_{\bar{j}i} \quad (k, j) = \overline{1, n}. \tag{5.68}$$

It is follows from (5.68) that the matrix *C* takes a form:

$$\mathbf{C} = \mathbf{T} \,\mathbf{T}',\tag{5.69}$$

Thus, we can consider in the capacity of ˆ**z** the nonnegative solution of the equation:

$$\mathbf{Cz} = \mathbf{v}, \qquad \mathbf{z} \ge \mathbf{0}.\tag{5.70}$$

The general solution of the equation (5.70) can be represented in the following form:

$$\sharp\_k(\mathbf{v}) = \frac{\det \mathbb{C}^k}{\det \mathbb{C}} \ge 0, \qquad k = \overline{1, n}. \tag{5.71}$$

where

24 Will-be-set-by-IN-TECH

On the second step we define the variable **z**<sup>0</sup> < **z***min*, where the vector **z***min* has the components *<sup>z</sup>min* (theorem 2). The vector **<sup>z</sup>**<sup>0</sup> <sup>=</sup> {*z*0,..., *<sup>z</sup>*0} such that the values of the monotone operator

On the third, we define the vector **<sup>z</sup>***max* <sup>=</sup> {*zmax*,..., *<sup>z</sup>max*} such that the monotone operator is less then **z***max*. For determination of the **z***max* we use the majorant of the monotone operator.

*2.1. Transformation of the equations (5.56).* Introduce the monotone increasing operator

This equation has the unique zero-solution **z**∗[**u**, **v**] ≡ **0** and the unique nonnegative solution **z**∗[**u**, **v**] ≥ **0**. Also recall that the elements of the matrix *T* and of the vector **v** in (5.39) are

*2.2. Choice z*0*.* According to the theorem 2 ˜**z** is the solution of the equation (5.62) for **u**−, **v**−.

where *ε* is a vector with small components *ε<sup>k</sup>* > 0, such that in the *ε*-neighborhood ˜**z** is valid

*2.3. Choice zmax*. Exact value of *zmax* is defined by the solution of the global optimization

<sup>+</sup>, **<sup>v</sup>**∈*V<sup>n</sup>*

Ψ(**z**, **u**) = **v**. However this problem is very complicated. So we will calculate an upper estimation of the

*A*(**z**ˆ, **u**, **v**) ≤ **z**ˆ. Choice *zmax* is equal to max*<sup>j</sup> z*ˆ*j*. Then the nonzero-solution **z**<sup>∗</sup> of the equation (5.62) will belong

<sup>+</sup>,*j*∈[1,*n*]

*z*∗ *<sup>j</sup>* (**u**, **v**),

*<sup>z</sup>max* <sup>=</sup> arg max **<sup>u</sup>**∈*U<sup>m</sup>*

*Ak*(**z**, **u**−, **v**−)|**z**˜ < 1,

Ψ*k*[**z**, **u**], *k* = 1, *n*. (5.61)

*A*(**z**, **u**, **v**) = **z**. (5.62)

**<sup>z</sup>**<sup>0</sup> <sup>=</sup> **<sup>z</sup>**˜ <sup>−</sup> *<sup>ε</sup>*, (5.63)

*<sup>A</sup>*(**z**0, **<sup>u</sup>**−, **<sup>v</sup>**<sup>−</sup> <sup>−</sup> *<sup>ε</sup>*) <sup>&</sup>gt; **<sup>z</sup>**0. (5.64)

**z***min* ≤ **z**<sup>∗</sup> ≤ **z***max*, (5.65)

*vk*

*Ak*(**z**, **<sup>u</sup>**, **<sup>v</sup>**) = *zk*

*∂ ∂zj*

at the point **z**<sup>0</sup> is more or equal to **z**0.

Consider each of the steps in detail.

Represent the equations (5.56) in the form:

It is follows that there exists the vector

problem Strongin & Sergeev (2000):

where **z**∗(**u**, **v**) is a solution of the equation:

Let us assume that we can find the vector ˆ**z** such that

to the following vector interval (see Krasnoselskii et al. (1969)):

the following inequality:

*A*(**z**, **u**, **v**) with the components:

nonnegative.

value *zmax*.

So,

$$\text{det}\,\mathbb{C} \neq 0,\tag{5.72}$$

as the matrix *T* has the full rank, and

$$\det \mathbf{C}^k = \sum\_{j=1}^n a\_{kj} v\_{j\prime} \qquad k = \overline{1, n}\_{\prime} \tag{5.73}$$

where

$$a\_{k\circ} = (-1)^{(k+j)} M\_{kj\prime} \tag{5.74}$$

and *Mkj* is the (*k*, *j*)-minor of the matrix *C*.

Introduce the following polyhedral sets:

$$\begin{aligned} \mathcal{W}\_{+} &= \left\{ \mathbf{v} : \sum\_{j=1}^{n} a\_{kj} v\_{j} \ge 0, \quad k = \overline{1, n} \right\}, \\ \mathcal{W}\_{-} &= \left\{ \mathbf{v} : \sum\_{j=1}^{n} a\_{kj} v\_{j} < 0, \quad k = \overline{1, n} \right\}. \end{aligned} \tag{5.75}$$

Censor, Y. & Segman, J. (1987). On block-iterative entropy maximization. *J. Inform. Optim. Sci.*,

Entropic Image Restoration as a Dynamic System with Entropy Operator 71

Censor, Y. & Zenios, S.A. (1997). *Parallel Optimization: Theory, Algorithms and Applications*,

Dorofeev, D.G.; Zon, B.A.; Popkov, Yu.S. (2008). Probabilistic Characteristics of Paramacrosystems. *Automation and Remote Control*, Vol. 69, No. 2, pp. 223-232.

Dubov, Yu.A.; Imelbaev, Sh.S.; Popkov, Yu.S. (1983). Multiplicative schemes for iterative optimization algorithms, *Soviet Math. Dokl.*, Vol. 22, No. 2, pp. 524-526. Fang, S-C.; Rajasekera, J.R.; Tsao, H.S.J. (1997). *Entropy Optimization and Mathematical*

Herman, G.T. (1980). *Image reconstruction from projections: the fundamentals of computerized*

Herman, G.T. (1982). Mathematical optimization versus practical performance: a case study

Iusem, A.N.; Svaiter, B.F.; Teboulle, M. (1996). Multiplicative Interior Gradient Methods for

Kapur, J.N. (1989). *Maximum Entropy Models in Science and Engneering*, John Willey and Sons,

Krasnosel'skii, M.A.; Rutitcki, Ja.B.; Stecenko, V.Ja.; Vainikko, G.M.; Zabreiko, P.P. (1969).

Kullback, S. & Leibler, R.A. (1951). On Information and Sufficiency. *Annals of Mathematical*

Maslov, V.P. & Cherny, A.S. (2003). About Entropy Minimization and Maximization in the

Ohnuki, Y. & Kamefuchi, S. (1982). *Quantum Field and Parastatistics*, University of Tokyo Press,

Popkov, Yu.S. (1988). A Multiplicative Algorithm for Fermi — Models of Macrosystems.

Popkov, Y.S. (1995). *Macrosystems Theory and Applications. Lecture Notes in Control and*

Popkov, Yu.S. (1995). Multiplicative algorithms for finding nonnegative solutions of convex programming problems. *Control and Cybernetics*, Vol. 24, No. 2, pp. 155-169. Popkov, Yu.S. (1996). Multiplicative algorithms for determining positive solutions of nonlinear

Popkov, Yu.S. (1997). Variation Principle of Image Reconstruction from Projections. *Automation*

Popkov, Y.S. (2006). New class of multiplicative algorithms for solving of entropy-linear programs. *European Journal of Operation Research*, Vol. 174, pp. 1368-1379. Shannon, C.E. (1948). Mathematical Theory of Communication. *Bell System Technical Journal*,

Polyak, B.T. (1987). *Introduction to Optimization*, Optimization Software, New York.

*Automation and Remote Control*, Vol. 49, No. 7, Part 1, pp. 320-927.

equations. *J. of Computational and Applied Math.*, Vol. 69, pp. 27-37.

Different disciplines. *Theory of Probabilities and thier Applications*, Vol. 48, No. 3, pp.

*Approximate Solutions of Operator Equations*, Nauka, Moscow.

Landau, L.D. & Livshitz, E.M. (1964). *Statistical Physiscs*, Nauka, Moscow.

based on the maximum entropy criterion in image reconstruction. *Mathematical*

Minimization over the Nonnegative Orthant. *SIAM J. Control and Optimization*, Vol.

Dhawan, A.P. (2003). *Medical Image Analysis*. John Wiley and Sons, New York.

*Programming*, Kluwer Academic Publisher, Dordrecht.

Vol. 8, pp. 275-291.

Oxford University Press, New York.

*tomography*, Academic Press, New York.

*Programming Study*, Vol. 20, pp. 96-112.

34, No. 1, pp. 319-446.

*Statistics*, Vol. 22(1), pp. 79-86.

Berlin, Heidelberg, New York.

*Information Sciences 203*, Springer, London.

*and Remote control*, No. 12, pp. 131-139.

Vol. 27(1):379-423, pp. 623-656.

New York.

466-486.

and the set

$$Q = \begin{cases} V\_+^n \cap W\_+, & \text{if } \det \mathcal{C} \ge 0, \\ V\_+^n \cap W\_-, & \text{if } \det \mathcal{C} < 0. \end{cases} \tag{5.76}$$

From this definition it follows that the set *Q* is the set of the vectors **v** for which the equation (5.70) has the nonnegative solutions. Therefore

$$z^{\max} = \max\_{j} \max\_{\mathbf{v} \in Q} \mathbb{E}\_{j}(\mathbf{v}).\tag{5.77}$$

Thus we proved the following **theorem 3**: *Let the matrix T (5.11) has a full rank and* **v** ∈ *Q (5.76). Then xmax* = max*<sup>i</sup> xi*(**1**, *zmax*), *where:*

$$z^{\max} = \max\_j \max\_{\mathbf{v} \in Q} \mathcal{E}\_j(\mathbf{v})\_\prime$$

*and z*ˆ*j*(**v**) *are the solution of the linear equation*

$$(T^\prime T)\mathbf{z} = \mathbf{v}.$$

#### **6. Conclusions**

Many applied problems can be formulated as the ELP or EQP, models of which it is proposed in the paper. The multiplicative algorithms with *p*-active variables and feedback are the effective methods of their solution. The dynamic procedure of the image restoration from projections (IRP) increase appreciably the quality of the restored image in the presence of noise in the measurements. It is represented a classification of the dynamic procedures and it is investigated a stability of the procedure with *I*-feedback. Also it is shown that in general case the dynamic procedure of the IRP is the dynamic system with entropy operator (EO). The analytical- numerical methods investigation the problem of the EO-boundedness and calculation of the Lipschitz constant for EO are proposed.

#### **7. References**


26 Will-be-set-by-IN-TECH

From this definition it follows that the set *Q* is the set of the vectors **v** for which the equation

Thus we proved the following **theorem 3**: *Let the matrix T (5.11) has a full rank and* **v** ∈ *Q (5.76).*

(*T*� *T*)**z** = **v**.

Many applied problems can be formulated as the ELP or EQP, models of which it is proposed in the paper. The multiplicative algorithms with *p*-active variables and feedback are the effective methods of their solution. The dynamic procedure of the image restoration from projections (IRP) increase appreciably the quality of the restored image in the presence of noise in the measurements. It is represented a classification of the dynamic procedures and it is investigated a stability of the procedure with *I*-feedback. Also it is shown that in general case the dynamic procedure of the IRP is the dynamic system with entropy operator (EO). The analytical- numerical methods investigation the problem of the EO-boundedness and

Aliev, A.S.; Dubov, Yu.A.; Izmailov, R.N.; Popkov, Yu.S. (1985). Convergence multiplicative

Boltzmann, L. (1871). *On the link between the second beginning of mechanical calorie theory and*

Byrne, C.L. (1993). Iterative Image Reconstruction Algorithm Based on Cross-Entropy

Byrne, C. (1996). Block-iterative methods for image reconstruction from projections. *IEEE*

Censor Y. (1981). Row-Action Methods for Huge and Sparse Systems and Their Application.

Beckenbach, E.F. & Bellman, R. (1961). *Inequlities*, Springer-Verlag, Berlin.

*Trans. Image Processing*, Vol. 8, pp. 275-291.

*SIAM Review*, Vol. 23, No. 4, pp. 444-464.

Minimization. *IEEE Trans. Img. Proc.*, Vol. 2, No. 1, pp. 96-103.

algorithms for solving of convex programming problem, in: *Dynamics of Nonhomogeneous Systems, Proceedings of VNIISI*, pp. 59-67, Nauka, Moscow (in

*probability theory in theorems of thermal equilibrium*, in: *Classics of Science*, Nauka,

max **v**∈*Q*

max **v**∈*Q*

*z*ˆ*j*(**v**),

*<sup>W</sup>*+, if det *<sup>C</sup>* ≥ 0,

*<sup>W</sup>*−, if det *<sup>C</sup>* <sup>&</sup>lt; 0. (5.76)

*z*ˆ*j*(**v**). (5.77)

*Q* = *V<sup>n</sup>* +

(5.70) has the nonnegative solutions. Therefore

*Then xmax* = max*<sup>i</sup> xi*(**1**, *zmax*), *where:*

*and z*ˆ*j*(**v**) *are the solution of the linear equation*

calculation of the Lipschitz constant for EO are proposed.

*V<sup>n</sup>* +

*zmax* = max *j*

*zmax* = max *j*

and the set

**6. Conclusions**

**7. References**

Russian).

Moscow.


Dhawan, A.P. (2003). *Medical Image Analysis*. John Wiley and Sons, New York.


**0**

**4**

*France*

**Surface Topography and Texture Restoration**

Mathieu Fernandes, Yann Gavet and Jean-Charles Pinoli

NA2 , (1)

NA = *ni* sin(*α*) , (2)

*École Nationale Supérieure des Mines de Saint-Étienne, CIS/LPMG-CNRS*

**from Sectional Optical Imaging by Focus Analysis**

Observing through any optical imaging device with traditional lens system is often "stained" by restricted depth-of-field. Such a simplified optical imaging system consisting of a convex lens (objective), a spherical diaphragm and a sensor plane (image plane) is depicted in Fig. 1. Let (*O*, �x, y�, �z) denote a Cartesian coordinate system: *O* is the optical center and the *z*-axis is along the optical axis. Imaging with this optical system effectively presents a common

*<sup>δ</sup><sup>z</sup>* <sup>=</sup> *ni<sup>λ</sup>*

where *λ* is the wavelength of illumination, *ni* is the refractive index of the medium in front of the objective and *α* is the angular semi-aperture of the diaphragm (Born & Wolf, 1991; Horn, 2001). Consider a scene surface, either opaque and observed in reflected light or sufficiently transparent and observed in transmitted light, whose profile covers more than this attainable depth-of-field (then described as "thick"). Thus, only portions of the observed surface that lie within the depth-of-field appear in-focus and sharp on the acquired image, whereas the remaining out-of-focus parts are blurred<sup>2</sup> by the point spread function (PSF) of the system (Born & Wolf, 1991; Horn, 2001). The PSF results from the contribution of many blur factors, such as the defocusing, the optical diffraction and aberrations and the sampling, principally. Many theoretical models of PSF have been proposed, with

<sup>1</sup> A Gaussian convex lens of focal length *f* theoretically focuses on a fixed image plane at *zi* only the light rays arising from a single object plane at *zo*, the so-called object focal plane, obeying the Snell's formula: 1/*zi* <sup>−</sup> 1/*zo* <sup>=</sup> 1/ *<sup>f</sup>* for the same medium refractive indexes in both front and back of the lens. <sup>2</sup> By regarding the illumination as incoherent, blurring can be modelled by a 2-D shift-variant linear convolution of the "ideal" sharp image of the object with the point spread function (*i.e.* with the response of the system to a purely impulsive point object that notably varies with the distance of

characteristic: the limited depth-of-field *δz* around its so-called object focal plane1:

moreover when the numerical aperture NA becomes larger:

**1. Introduction**

defocus for *x*, *y*-shifting).


Wilkinson, J.H. (1970). *The Algebraic Eigenvalue Problem*, p. 564, Clarendon Press, Oxford.

## **Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis**

Mathieu Fernandes, Yann Gavet and Jean-Charles Pinoli *École Nationale Supérieure des Mines de Saint-Étienne, CIS/LPMG-CNRS France*

#### **1. Introduction**

28 Will-be-set-by-IN-TECH

72 Image Restoration – Recent Advances and Applications

Strongin, R.G. & Sergeev, Y.D. (2000). *Global Optimization with Non-convex Constraints*, Kluwer

Voevodin, V.V. & Kuznetsov, Yu.A. (1984). *Matrix and calculations*, p. 318, Nauka, Moscow (in

Wilkinson, J.H. (1970). *The Algebraic Eigenvalue Problem*, p. 564, Clarendon Press, Oxford.

Acad. Press.

Russian).

Observing through any optical imaging device with traditional lens system is often "stained" by restricted depth-of-field. Such a simplified optical imaging system consisting of a convex lens (objective), a spherical diaphragm and a sensor plane (image plane) is depicted in Fig. 1. Let (*O*, �x, y�, �z) denote a Cartesian coordinate system: *O* is the optical center and the *z*-axis is along the optical axis. Imaging with this optical system effectively presents a common characteristic: the limited depth-of-field *δz* around its so-called object focal plane1:

$$
\delta z = \frac{n\_i \lambda}{\text{NA}^2} \,\text{\,\,\,}\tag{1}
$$

moreover when the numerical aperture NA becomes larger:

$$\mathbf{NA} = n\_l \sin(a) \,\,,\tag{2}$$

where *λ* is the wavelength of illumination, *ni* is the refractive index of the medium in front of the objective and *α* is the angular semi-aperture of the diaphragm (Born & Wolf, 1991; Horn, 2001). Consider a scene surface, either opaque and observed in reflected light or sufficiently transparent and observed in transmitted light, whose profile covers more than this attainable depth-of-field (then described as "thick"). Thus, only portions of the observed surface that lie within the depth-of-field appear in-focus and sharp on the acquired image, whereas the remaining out-of-focus parts are blurred<sup>2</sup> by the point spread function (PSF) of the system (Born & Wolf, 1991; Horn, 2001). The PSF results from the contribution of many blur factors, such as the defocusing, the optical diffraction and aberrations and the sampling, principally. Many theoretical models of PSF have been proposed, with

<sup>1</sup> A Gaussian convex lens of focal length *f* theoretically focuses on a fixed image plane at *zi* only the light rays arising from a single object plane at *zo*, the so-called object focal plane, obeying the Snell's formula:

<sup>1/</sup>*zi* <sup>−</sup> 1/*zo* <sup>=</sup> 1/ *<sup>f</sup>* for the same medium refractive indexes in both front and back of the lens. <sup>2</sup> By regarding the illumination as incoherent, blurring can be modelled by a 2-D shift-variant linear convolution of the "ideal" sharp image of the object with the point spread function (*i.e.* with the response of the system to a purely impulsive point object that notably varies with the distance of defocus for *x*, *y*-shifting).

concepts require an original sequence with image sections spatially registered, principally by considering magnification variations due to changes in focus setting through the perspective projection of most optical imaging system<sup>4</sup> (as in Fig. 1) (Willson & Shafer, 1991). These magnification changes can be corrected using optical approaches, such as zoom adjustments based on system calibration (Willson, 1994), or computational techniques, commonly referred to as image warping (Darrell & Wohn, 1988). Notice that acquiring the image sequence by displacing either the scene or the imaging system along the *z*-direction with respect to a fixed focus setting ensures at least a constant magnification *γ* for all successive object focal planes, but not for the out-of-focus object planes that always suffer different magnifications than the focal ones (Nayar & Nakagawa, 1994). Otherwise, an all-over constant magnification can be reached through orthographic projection of telecentric optics (Watanabe & Nayar, 1997).

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 75

After briefly describing both Shape-From-Focus (SFF) and Extended Depth-of-Field (EDF) concepts in section 2, their linchpin step consisting in a focus measurement will be particularly studied, reviewed and finally "morphed" in section 3. Indeed, this work especially strives to make changes to classical state-of-the-art focus measurements through different strategies into new evolved approaches that are custom-made to cope with frequently encountered issues, such as ill-illuminated/poor textured or noisy/disturbed acquisitions. An ill-illuminated/poor textured observed surface effectively exhibits few focus cues (high-spatial frequency components) on which the restoration process is based. On the contrary, noisy/disturbed data introducing during the acquisitions produce "false focus cues" that misleads the restoration process. Such issues thus require rather opposite focus measurement behaviours: a high sensitivity to focus cues and a strong robustness to noise, respectively. Thereafter, several tests will be conducted, illustrated and discussed in section 4 on both simulated data and real acquisitions from different application fields (metallography, granulometry, ophthalmology) in conventional optical microscopy. Through such optical imaging system, the inherent use of large magnifications *γ* ∼ NA significantly limits the offered depth-of-field and the performed projection tends towards an orthographic behaviour (and therefore an all-over constant magnification) since the working distance WD = |*zo*| is much larger than the profile thickness of the observed surface (Horn, 2001). Finally, the new introduced approaches (2-D LIP-based focus measurements and 3-D statistical focus measurements) will be compared to classical state-of-the-art ones and will clearly show their

The Shape-From-Focus (SFF) concept exploits the limited depth-of-field to infer the topography of the observed surface by maximizing a focus measurement throughout the *z*-direction of the image sequence. Likewise, the Extended depth-of-field (EDF) concept conversely tries to overcome the depth-of-field limitation by joining through a focus measurement the most in-focus information from the image sequence into a single image: the so-called "texture image". Both complementary approaches work similarly and foremost rely upon an essential preliminary focus measurement that mainly interests this work and will be more closely studied in the next section 3. They are graphically summarized in Fig. 2 and

<sup>4</sup> Since the intersections of the so-called principal rays (the ones passing undeflected through the center of the lens *O*) with the sensor plane vary with the position of this latter, the image magnification changes

efficiency in presence of aforementioned acquisition issues.

**2. Surface topography and texture restoration**

with defocus.

Fig. 1. Illustration of the basic image formation geometry. The green light rays radiated by an in-focus point of the observed surface are well refracted by the convex lens onto the sensor plane contrary to red light rays arising from an out-of-focus point, which converge forward and whose energies are distributed over the "blur circle" patch.

accuracies that depend on considered factors and used approximations3 (Mahajan, 1998; 2001). Introduced by Pentland, a 2-D Gaussian function is often suggested as a PSF model with a widening standard deviation as the distance of defocus increases (Pentland, 1987). Ultimately, the PSF always behaves as a low-pass filter, whose cut-off spatial frequency falls when the degree of defocus raises. In order to fully observe such a "thick" scene surface, a common way then consists in scanning it with the object focal plane of the optical system, more formally by acquiring a large sequence of 2-D images by optical sectioning (Agard, 1984). The final sequence of 2-D images is thus collected by gradually moving the object focal plane along the *z*-direction throughout the surface. Each 2-D optical section joins out-of-focus blurred and in-focus sharp portions, respectively related to parts of the object surface outside and inside the depth-of-field. Less damaged by the low-pass PSF, the latter exhibit much more of high-spatial frequency components corresponding to surface textural details. From such an image sequence, this chapter then focuses on image restoration of both topographical and textural information of the observed surface through the common concepts of Shape-From-Focus (or Depth-From-focus) and Extended Depth-of-Field. Importantly, both

<sup>3</sup> According to geometrical optics, a first-order approximation of the defocusing PSF consists in a homogeneous patch, the so-called blur circle in the case of a spherical diaphragm whose radius increases with the distance of defocus.

2 Will-be-set-by-IN-TECH

Blur circle

*zi*

*f*

*z O*

*y x*

Out-of-focus object plane

Optical axis

Fig. 1. Illustration of the basic image formation geometry. The green light rays radiated by an in-focus point of the observed surface are well refracted by the convex lens onto the sensor plane contrary to red light rays arising from an out-of-focus point, which converge forward

accuracies that depend on considered factors and used approximations3 (Mahajan, 1998; 2001). Introduced by Pentland, a 2-D Gaussian function is often suggested as a PSF model with a widening standard deviation as the distance of defocus increases (Pentland, 1987). Ultimately, the PSF always behaves as a low-pass filter, whose cut-off spatial frequency falls when the degree of defocus raises. In order to fully observe such a "thick" scene surface, a common way then consists in scanning it with the object focal plane of the optical system, more formally by acquiring a large sequence of 2-D images by optical sectioning (Agard, 1984). The final sequence of 2-D images is thus collected by gradually moving the object focal plane along the *z*-direction throughout the surface. Each 2-D optical section joins out-of-focus blurred and in-focus sharp portions, respectively related to parts of the object surface outside and inside the depth-of-field. Less damaged by the low-pass PSF, the latter exhibit much more of high-spatial frequency components corresponding to surface textural details. From such an image sequence, this chapter then focuses on image restoration of both topographical and textural information of the observed surface through the common concepts of Shape-From-Focus (or Depth-From-focus) and Extended Depth-of-Field. Importantly, both

<sup>3</sup> According to geometrical optics, a first-order approximation of the defocusing PSF consists in a homogeneous patch, the so-called blur circle in the case of a spherical diaphragm whose radius

*zo*

and whose energies are distributed over the "blur circle" patch.

increases with the distance of defocus.

*α*

Depth-of-field *δz*

Convex lens

Image plane

Object focal plane

Observed surface

Spherical diaphragm

concepts require an original sequence with image sections spatially registered, principally by considering magnification variations due to changes in focus setting through the perspective projection of most optical imaging system<sup>4</sup> (as in Fig. 1) (Willson & Shafer, 1991). These magnification changes can be corrected using optical approaches, such as zoom adjustments based on system calibration (Willson, 1994), or computational techniques, commonly referred to as image warping (Darrell & Wohn, 1988). Notice that acquiring the image sequence by displacing either the scene or the imaging system along the *z*-direction with respect to a fixed focus setting ensures at least a constant magnification *γ* for all successive object focal planes, but not for the out-of-focus object planes that always suffer different magnifications than the focal ones (Nayar & Nakagawa, 1994). Otherwise, an all-over constant magnification can be reached through orthographic projection of telecentric optics (Watanabe & Nayar, 1997).

After briefly describing both Shape-From-Focus (SFF) and Extended Depth-of-Field (EDF) concepts in section 2, their linchpin step consisting in a focus measurement will be particularly studied, reviewed and finally "morphed" in section 3. Indeed, this work especially strives to make changes to classical state-of-the-art focus measurements through different strategies into new evolved approaches that are custom-made to cope with frequently encountered issues, such as ill-illuminated/poor textured or noisy/disturbed acquisitions. An ill-illuminated/poor textured observed surface effectively exhibits few focus cues (high-spatial frequency components) on which the restoration process is based. On the contrary, noisy/disturbed data introducing during the acquisitions produce "false focus cues" that misleads the restoration process. Such issues thus require rather opposite focus measurement behaviours: a high sensitivity to focus cues and a strong robustness to noise, respectively. Thereafter, several tests will be conducted, illustrated and discussed in section 4 on both simulated data and real acquisitions from different application fields (metallography, granulometry, ophthalmology) in conventional optical microscopy. Through such optical imaging system, the inherent use of large magnifications *γ* ∼ NA significantly limits the offered depth-of-field and the performed projection tends towards an orthographic behaviour (and therefore an all-over constant magnification) since the working distance WD = |*zo*| is much larger than the profile thickness of the observed surface (Horn, 2001). Finally, the new introduced approaches (2-D LIP-based focus measurements and 3-D statistical focus measurements) will be compared to classical state-of-the-art ones and will clearly show their efficiency in presence of aforementioned acquisition issues.

#### **2. Surface topography and texture restoration**

The Shape-From-Focus (SFF) concept exploits the limited depth-of-field to infer the topography of the observed surface by maximizing a focus measurement throughout the *z*-direction of the image sequence. Likewise, the Extended depth-of-field (EDF) concept conversely tries to overcome the depth-of-field limitation by joining through a focus measurement the most in-focus information from the image sequence into a single image: the so-called "texture image". Both complementary approaches work similarly and foremost rely upon an essential preliminary focus measurement that mainly interests this work and will be more closely studied in the next section 3. They are graphically summarized in Fig. 2 and

<sup>4</sup> Since the intersections of the so-called principal rays (the ones passing undeflected through the center of the lens *O*) with the sensor plane vary with the position of this latter, the image magnification changes with defocus.

Sequence of 2-D images

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 77

I

3-D focus degree measure

F

*z*

Largest degrees of focus along the *z*-direction

*z*-coordinates Intensities in I

Depth map Texture

D T

Interpolation Deconvolution

3-D surface reconstruction

Fig. 2. Basic illustrated diagram representing both complementary Shape-From-Focus (left)

When the optical sectioning step is larger than the depth-of-field *δz*, some regions of the observed surface may never appear in-focus throughout the image sequence and therefore on the restored texture image. Pradeed and Ragajolan then proposed to perform a non-stationary Wiener filter to locally deconvolve the texture image *T* (Pradeep & Rajagopalan, 2007). Note

Let us now focus on the essential step of focus measurement, firstly through a literature review that will yield the retention of some classical and recent methods making a representative

and Extended Depth-of-Field (right) concepts.

that no deconvolution process will be used herein.

**3. Focus measurements**

Mapping of the texture image onto the depth map

will be further described below. Notice that a 3-D reconstruction of the surface can finally be obtained by mapping the texture image onto the topography, as illustrated in Fig. 3. Before going on, let us introduce some notations. Let *I*(*x*, *y*, *z*) denotes the sequence of images acquired by optical sectioning, defined on the spatial support **<sup>D</sup>** <sup>=</sup> **<sup>D</sup>***<sup>x</sup>* <sup>×</sup> **<sup>D</sup>***<sup>y</sup>* <sup>×</sup> **<sup>D</sup>***<sup>z</sup>* <sup>⊂</sup> **<sup>R</sup>**<sup>3</sup> and valued into a positive real range [0, *M*) of intensity values. Applying a focus measurement function (FM) on *I*(*x*, *y*, *z*) yields a 3-D focus degree measure *F*(*x*, *y*, *z*) as follows:

$$\begin{aligned} \mathcal{F}: \qquad \mathbb{D} \to \mathbb{R}^+ \\ (\mathfrak{x}, y, z) \mapsto \mathrm{FM}(\mathfrak{L}(\mathfrak{x}, y, z)) \,, \end{aligned} \tag{3}$$

wherein the profile at location (*x*, *<sup>y</sup>*) along the *<sup>z</sup>*-direction is designated as *<sup>F</sup>*|*x*,*<sup>y</sup>* : **<sup>D</sup>***<sup>z</sup>* <sup>→</sup> **<sup>R</sup>**+.

#### **2.1 Topographical information: Shape-From-Focus (SFF)**

The *z*-coordinates (referred to as depths) of the voxels that exhibit the largest degrees of focus infer the topography (or the so-called depth map) *D* of the observed surface from its image sequence *I*(*x*, *y*, *z*) as follows:

$$\begin{aligned} \mathcal{D}: \mathbb{D}\_{\mathbb{X}} \times \mathbb{D}\_{\mathbb{Y}} &\to \mathbb{D}\_{z} \\ (x, y) &\mapsto \operatorname\*{argmax}\_{z \in \mathbb{D}\_{z}} \mathcal{F}|\_{\mathbb{X}, y}(z) \,. \end{aligned} \tag{4}$$

Because of the significant thickness *δz* of the depth-of-field, the recovered topography *D* shows inherent "staircase" effects and an interpolation approach must then be embedded in this basic process of reconstruction. Introduced by Nayar and Nakagawa, the traditional one consists in fitting a Gaussian distribution, whose mean finally constitutes the interpolated depth value, to the three degrees of focus lying on the largest mode (Nayar & Nakagawa, 1994). Similarly, a quadratic (or even more) polynomial model can be fitted, sometimes regarding more than three degrees of focus (Niederöst et al., 2003; Subbarao & Choi, 1995). A subsequent approach (referred to as Focused Image Surface) locally tries to refine the initial recovered topography *D* by optimizing both position and orientation of 2-D planar (then curved) windows throughout the 3-D measure *F* so as to maximize the covered degrees of focus (Ahmad & Choi, 2005; Asif & Choi, 2001; Subbarao & Choi, 1995; Yun & Choi, 1999). Finally, the topography is often smoothed through average, median or recently bilateral filters (Helmli & Scherer, 2001; Khan et al., 2010; Mahmood et al., 2008; Niederöst et al., 2003). Interpolation techniques lying beyond the scope of this paper, only the traditional one will be used herein, sometimes finalised by a median filter.

#### **2.2 Textural information: Extended Depth-of-Field (EDF)**

Throughout the image sequence *I*(*x*, *y*, *z*), the texture image *T* of the observed surface is restored by joining the intensity voxels with the largest degrees of focus:

$$\begin{aligned} \mathcal{T}: \mathbb{D}\_{\mathbf{x}} \times \mathbb{D}\_{\mathcal{Y}} &\to [0, M) \\ (\mathbf{x}, y) &\mapsto \mathcal{Z}(\mathbf{x}, y, \operatorname\*{argmax}\_{z \in \mathbb{D}\_{\mathbb{L}}} \mathcal{F}|\_{\mathbf{x}, y}(z)) \,. \end{aligned} \tag{5}$$

When the optical sectioning step is larger than the depth-of-field *δz*, some regions of the observed surface may never appear in-focus throughout the image sequence and therefore on the restored texture image. Pradeed and Ragajolan then proposed to perform a non-stationary Wiener filter to locally deconvolve the texture image *T* (Pradeep & Rajagopalan, 2007). Note that no deconvolution process will be used herein.

## **3. Focus measurements**

4 Will-be-set-by-IN-TECH

will be further described below. Notice that a 3-D reconstruction of the surface can finally be obtained by mapping the texture image onto the topography, as illustrated in Fig. 3. Before going on, let us introduce some notations. Let *I*(*x*, *y*, *z*) denotes the sequence of images acquired by optical sectioning, defined on the spatial support **<sup>D</sup>** <sup>=</sup> **<sup>D</sup>***<sup>x</sup>* <sup>×</sup> **<sup>D</sup>***<sup>y</sup>* <sup>×</sup> **<sup>D</sup>***<sup>z</sup>* <sup>⊂</sup> **<sup>R</sup>**<sup>3</sup> and valued into a positive real range [0, *M*) of intensity values. Applying a focus measurement

wherein the profile at location (*x*, *<sup>y</sup>*) along the *<sup>z</sup>*-direction is designated as *<sup>F</sup>*|*x*,*<sup>y</sup>* : **<sup>D</sup>***<sup>z</sup>* <sup>→</sup> **<sup>R</sup>**+.

The *z*-coordinates (referred to as depths) of the voxels that exhibit the largest degrees of focus infer the topography (or the so-called depth map) *D* of the observed surface from its image

(*x*, *y*) �→ argmax

Because of the significant thickness *δz* of the depth-of-field, the recovered topography *D* shows inherent "staircase" effects and an interpolation approach must then be embedded in this basic process of reconstruction. Introduced by Nayar and Nakagawa, the traditional one consists in fitting a Gaussian distribution, whose mean finally constitutes the interpolated depth value, to the three degrees of focus lying on the largest mode (Nayar & Nakagawa, 1994). Similarly, a quadratic (or even more) polynomial model can be fitted, sometimes regarding more than three degrees of focus (Niederöst et al., 2003; Subbarao & Choi, 1995). A subsequent approach (referred to as Focused Image Surface) locally tries to refine the initial recovered topography *D* by optimizing both position and orientation of 2-D planar (then curved) windows throughout the 3-D measure *F* so as to maximize the covered degrees of focus (Ahmad & Choi, 2005; Asif & Choi, 2001; Subbarao & Choi, 1995; Yun & Choi, 1999). Finally, the topography is often smoothed through average, median or recently bilateral filters (Helmli & Scherer, 2001; Khan et al., 2010; Mahmood et al., 2008; Niederöst et al., 2003). Interpolation techniques lying beyond the scope of this paper, only the traditional one will be

Throughout the image sequence *I*(*x*, *y*, *z*), the texture image *T* of the observed surface is

(*x*, *<sup>y</sup>*) �→ *<sup>I</sup>*(*x*, *<sup>y</sup>*, argmax *<sup>z</sup>* <sup>∈</sup> **<sup>D</sup>***<sup>z</sup>*

*z* ∈ **D***<sup>z</sup>*

(*x*, *y*, *z*) �→ FM(*I*(*x*, *y*, *z*)), (3)

*F*|*x*,*y*(*z*). (4)

*F*|*x*,*y*(*z*)). (5)

function (FM) on *I*(*x*, *y*, *z*) yields a 3-D focus degree measure *F*(*x*, *y*, *z*) as follows:

*<sup>F</sup>* : **<sup>D</sup>** <sup>→</sup> **<sup>R</sup>**<sup>+</sup>

*D* : **D***<sup>x</sup>* × **D***<sup>y</sup>* → **D***<sup>z</sup>*

**2.1 Topographical information: Shape-From-Focus (SFF)**

used herein, sometimes finalised by a median filter.

**2.2 Textural information: Extended Depth-of-Field (EDF)**

restored by joining the intensity voxels with the largest degrees of focus:

*T* : **D***<sup>x</sup>* × **D***<sup>y</sup>* → [0, *M*)

sequence *I*(*x*, *y*, *z*) as follows:

Let us now focus on the essential step of focus measurement, firstly through a literature review that will yield the retention of some classical and recent methods making a representative

(2*r* + 1) pixels. Given this local behaviour, a certain depth regularity of the observed surface is implicitly assumed. On the one hand, the considered neighborhood has to be as small as possible to guarantee an approximately constant depth within itself and therefore to avoid too much "smoothing" the restoration process around sharp depth slopes and even depth discontinuities (Malik & Choi, 2007). On the other hand, it has to be as large as possible both to always capture focus cues (*i.e.* high-spatial frequency components) within wide homogeneous textural contents of the surface and to average out noise. Consequently, the selection of the optimal window size *r* appears as a trade-off. These approaches classically include two successive steps aiming to emphasize and quantify focus cues, respectively. The second one is simply an energy measurement that is commonly the sum over the considered neighborhoods of the absolute values resulting from the first one, therefore improving the robustness to noise and/or to wide textural contents of the measurement. The first step differs in the specialized literature. Most are based on high-pass filtering (norms of derivatives), such as Laplacian energy (Subbarao et al., 1993), sum-modified-Laplacian (Nayar & Nakagawa, 1994), Brenner (Brenner et al., 1976) or Tenenbaum (Krotkov, 1987) gradients, among others... Others, usually more robust to noise, use statistical tools in the considered neighborhoods, such as (normalized) variance (Groen et al., 1985; Sugimoto & Ichioka, 1985), autocorrelation (Vollath, 1987), sum of eigenvalues (Wee & Paramesran, 2007) or various moments (Yap & Raveendran, 2004; Zhang et al., 2000). Remark that some of them directly combine the two aforementioned steps, *e.g.* the variance in the neighborhoods. The last ones work in different frequency domains through discrete cosine (Kristan et al., 2006) or Fourier (Boddeke et al., 1994; Malik & Choi, 2008) transforms. The latter exploit more robust band-pass filters but lack sensitivity in return. At first, note that neighborhood-based focus measurements was often

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 79

Throughout these state-of-the-art section, some fundamental and recent methods will be

**2-D TEN** Sum over a 2-D window of the squared *L*2-norms of the first derivatives approximated by the horizontal and vertical Sobel operators (TENengrad) (Krotkov, 1987). **2-D SML** Sum over a 2-D window of the *L*1-norms of the second derivatives approximated by the Laplacian operator (Sum-Modified-Laplacian) (Nayar & Nakagawa, 1994). **2-D OPT** Sum over a 2-D window of the absolute values of the real part responses in the spatial domain to an "OPTical" band-pass filter applied in the Fourier domain and based

Other 2-D approaches rely on some form of multiresolution analysis: *e.g.* Laplacian (Burt & Adelson, 1983), ratio-of-low-pass (Toet, 1989), gradient (Burt & Kolczynski, 1993) and steerable pyramids (Liu et al., 2001), and wavelet (Forster et al., 2004; Pajares & de la Cruz, 2004; Valdecasas et al., 2001), shapelet (Meneses et al., 2008) and curvelet (Minhas et al., 2011) transforms, in order to perform high-pass filtering at different resolution level. Contrary to afore-described neighborhood-based methods, these ones thus avoid the choice of a fixed-size filter. They are regularly introduced in the practical context of image fusion that consists in combining information from some (generally between 2 and 5) multi-focus or

retained; their designations, details and references will be summarized as follows:

**2-D VAR** VARiance in a 2-D window (Groen et al., 1985; Sugimoto & Ichioka, 1985).

employed to computationally autofocus imaging system.

on bipolar incoherent image processing (Malik & Choi, 2008).

3.1.2.2 Multiresolution-based methods

Fig. 3. 3-D reconstruction of a human corneal graft by the **2-D SML**� LIP-based focus measurement from a sequence of 32 image sections acquired in conventional optical microscopy by steps of 9.33 *μ*m through a × 4 / 0.1 NA objective in air immersion. Each image section composed of 1932 × 2029 pixels representing 10.62 × 11.11 mm is an undersampled version of a registered mosaic of 5 × 7 image acquisitions.

sample group from a strategic as well as chronological point of view. Some of them will then be developed into novel evolved approaches designated as 2-D LIP-based focus measurements and 3-D statistical focus measurements.

## **3.1 State-of-the-art focus measurements**

In view of the fact that the PSF of defocus acts as a low-pass filter, focus measurements thus try to locally emphasize and quantify high-spatial frequency components of the original image sequence *I*. They can be classified according to the dimensionality of the adopted strategy to do that.

## **3.1.1 One-dimensional (point-based) approaches**

From the early 1980s, some methods using maximum or minimum selection rules throughout single-voxel stacks along the *z*-direction of the image sequence are first proposed *I* (Pieper & Korpel, 1983; Sugimoto & Ichioka, 1985), therefore not offering a large robustness.

## **3.1.2 Two-dimensional approaches**

For the last 40 years, a lot of more reliable focus measurements independently acting (in 2-D) on each image section of the sequence *I* then arose, categorized below as either neighborhood-based or multiresolution-based methods.

## 3.1.2.1 Neighborhood-based methods

Neighborhood-based focus measurements work over local sectional fixed-size windows, described herein by the size value *r* corresponding to an operating window of (2*r* + 1) ×

(2*r* + 1) pixels. Given this local behaviour, a certain depth regularity of the observed surface is implicitly assumed. On the one hand, the considered neighborhood has to be as small as possible to guarantee an approximately constant depth within itself and therefore to avoid too much "smoothing" the restoration process around sharp depth slopes and even depth discontinuities (Malik & Choi, 2007). On the other hand, it has to be as large as possible both to always capture focus cues (*i.e.* high-spatial frequency components) within wide homogeneous textural contents of the surface and to average out noise. Consequently, the selection of the optimal window size *r* appears as a trade-off. These approaches classically include two successive steps aiming to emphasize and quantify focus cues, respectively. The second one is simply an energy measurement that is commonly the sum over the considered neighborhoods of the absolute values resulting from the first one, therefore improving the robustness to noise and/or to wide textural contents of the measurement. The first step differs in the specialized literature. Most are based on high-pass filtering (norms of derivatives), such as Laplacian energy (Subbarao et al., 1993), sum-modified-Laplacian (Nayar & Nakagawa, 1994), Brenner (Brenner et al., 1976) or Tenenbaum (Krotkov, 1987) gradients, among others... Others, usually more robust to noise, use statistical tools in the considered neighborhoods, such as (normalized) variance (Groen et al., 1985; Sugimoto & Ichioka, 1985), autocorrelation (Vollath, 1987), sum of eigenvalues (Wee & Paramesran, 2007) or various moments (Yap & Raveendran, 2004; Zhang et al., 2000). Remark that some of them directly combine the two aforementioned steps, *e.g.* the variance in the neighborhoods. The last ones work in different frequency domains through discrete cosine (Kristan et al., 2006) or Fourier (Boddeke et al., 1994; Malik & Choi, 2008) transforms. The latter exploit more robust band-pass filters but lack sensitivity in return. At first, note that neighborhood-based focus measurements was often employed to computationally autofocus imaging system.

Throughout these state-of-the-art section, some fundamental and recent methods will be retained; their designations, details and references will be summarized as follows:

**2-D VAR** VARiance in a 2-D window (Groen et al., 1985; Sugimoto & Ichioka, 1985).


### 3.1.2.2 Multiresolution-based methods

6 Will-be-set-by-IN-TECH

Fig. 3. 3-D reconstruction of a human corneal graft by the **2-D SML**� LIP-based focus measurement from a sequence of 32 image sections acquired in conventional optical microscopy by steps of 9.33 *μ*m through a × 4 / 0.1 NA objective in air immersion. Each image section composed of 1932 × 2029 pixels representing 10.62 × 11.11 mm is an

sample group from a strategic as well as chronological point of view. Some of them will then be developed into novel evolved approaches designated as 2-D LIP-based focus

In view of the fact that the PSF of defocus acts as a low-pass filter, focus measurements thus try to locally emphasize and quantify high-spatial frequency components of the original image sequence *I*. They can be classified according to the dimensionality of the adopted strategy to

From the early 1980s, some methods using maximum or minimum selection rules throughout single-voxel stacks along the *z*-direction of the image sequence are first proposed *I* (Pieper & Korpel, 1983; Sugimoto & Ichioka, 1985), therefore not offering a large robustness.

For the last 40 years, a lot of more reliable focus measurements independently acting (in 2-D) on each image section of the sequence *I* then arose, categorized below as either

Neighborhood-based focus measurements work over local sectional fixed-size windows, described herein by the size value *r* corresponding to an operating window of (2*r* + 1) ×

undersampled version of a registered mosaic of 5 × 7 image acquisitions.

measurements and 3-D statistical focus measurements.

**3.1.1 One-dimensional (point-based) approaches**

neighborhood-based or multiresolution-based methods.

**3.1 State-of-the-art focus measurements**

**3.1.2 Two-dimensional approaches**

3.1.2.1 Neighborhood-based methods

do that.

Other 2-D approaches rely on some form of multiresolution analysis: *e.g.* Laplacian (Burt & Adelson, 1983), ratio-of-low-pass (Toet, 1989), gradient (Burt & Kolczynski, 1993) and steerable pyramids (Liu et al., 2001), and wavelet (Forster et al., 2004; Pajares & de la Cruz, 2004; Valdecasas et al., 2001), shapelet (Meneses et al., 2008) and curvelet (Minhas et al., 2011) transforms, in order to perform high-pass filtering at different resolution level. Contrary to afore-described neighborhood-based methods, these ones thus avoid the choice of a fixed-size filter. They are regularly introduced in the practical context of image fusion that consists in combining information from some (generally between 2 and 5) multi-focus or

**3.2 Two-dimensional LIP-based focus measurements**

**3.2.1 Logarithmic image processing (LIP) framework**

3.2.1.1 Mathematical fundamentals

real vector space denoted *S*:

*ϕ*−<sup>1</sup> is then defined as:

3.2.1.2 Physical connections

(LIP) framework.

This first work aims at improving sensitivity to focus cues of usual measurements in order to well operate in difficult regions of the observed surface, such as its ill-illuminated/poor textured parts. Let us start with a brief introduction of the Logarithmic Image Processing

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 81

An original mathematical framework, the LIP model, has been introduced in the middle of the 1980s for the processing of intensity images valued in a bounded range (Jourlin & Pinoli, 1987; 1988; 2001). This model is mathematically well defined as well as physically consistent. The reader can refer to Pinoli (1997a;b) for a complete mathematical theory and many physical

In the LIP model, the intensity of an image is completely represented by its associated gray tone function *f* . Such a function is defined on the spatial suppport **D** and valued in the real number range interval [0, *M*), called the gray tone range. Thereafter, this class of gray tone functions, extended to the real number interval (−∞, *M*) and structured with the after-specified vector addition �<sup>+</sup> , scalar multiplication �× and scalar subtraction �− defines a

<sup>∀</sup> *<sup>f</sup>* , *<sup>g</sup>* <sup>∈</sup> *S f* �<sup>+</sup> *<sup>g</sup>* <sup>=</sup> *<sup>f</sup>* <sup>+</sup> *<sup>g</sup>* <sup>−</sup> *f g*

<sup>∀</sup> *<sup>f</sup>* , *<sup>g</sup>* <sup>∈</sup> *S f* �− *<sup>g</sup>* <sup>=</sup> *<sup>M</sup> <sup>f</sup>*−*<sup>g</sup>*

This gray tone vector space *S* is algebraically and topologically isomorphic to the classical vector space defined on the spatial support **D** with values in the real number set **R** through

which is the isomorphic transform of the gray tone *f* . The inverse isomorphic transformation

In addition to abstract linear algebra, this class of (extended) gray tone functions is an ordered

The LIP framework has been proved to be consistent with the transmittance image formation model (Jourlin & Pinoli, 1988), the multiplicative reflectance and transmittance

<sup>1</sup> <sup>−</sup> exp

∀ *<sup>f</sup>* ∈ *<sup>S</sup>*, ∀*<sup>a</sup>* ∈ **<sup>R</sup>** *<sup>a</sup>*�× *<sup>f</sup>* = *<sup>M</sup>* − *<sup>M</sup>*

<sup>∀</sup> *<sup>f</sup>* <sup>∈</sup> *<sup>S</sup> <sup>ϕ</sup>*(*f*) = <sup>−</sup>*<sup>M</sup>* ln

the mapping *ϕ* (called the isomorphic transformation) defined as:

*f* = *ϕ*−1(*ϕ*(*f*)) = *M*

real vector space with the classical order relation ≥ (Pinoli, 1997a).

*<sup>M</sup>* ,

(6)

, (7)

. (8)

 <sup>1</sup> <sup>−</sup> *<sup>f</sup> M a* ,

*<sup>M</sup>*−*<sup>g</sup>* .

<sup>1</sup> <sup>−</sup> *<sup>f</sup> M* 

> <sup>−</sup> *<sup>ϕ</sup>*(*f*) *M*

and/or psychophysical connections and justifications about the LIP framework.

multimodal images of the same scene into a single composite representation. An overview of multiresolution-based schemes for image fusion can be found in (Zhang & Blum, 1999). First, each image section of the original sequence is decomposed into a collection of sub-images at different scales, called a pyramid structure, through alternate combination of convolution and sub-sampling. Different types of details (focus cues) are thus put forward at different levels in the associated pyramid structure. Note that the original image section can be reconstructed by the reverse procedure. A (pixel-based, window-based or region-based (Piella, 2003)) salience measurement (absolute value, sum or variance of absolute values) then tries to quantify focus cues throughout every pyramid structures. The depth map is thus inferred from the largest salience measures. Besides, a composite pyramid structure is constructed by combining coefficients of the original pyramid structures in function of their exhibited salience measures (choose-max or weighted average). Next, a (window-based or region-based) consistency verification is performed on the composite pyramid structure (and on the recovered depth map) so as to check that best salience measures come from the same original image sections, which is equivalent to a smoothing post-processing step. Once the composite pyramid structure is fused, the final texture image is lastly restored by reverse decomposition.

**2-D DWT** Use of the Discrete Wavelet Transform (DWT) based on complex Daubechies wavelets as multiresolution analysis, of the largest absolute value of the wavelet coefficients in the subbands (up to 10 levels) as (pixel-based) salience measurement and of both spatial (window of size *r* = 1) and typical subband consistency checks on the wavelet coefficients. (Forster et al., 2004).

By independently working on each individual image section of the sequence *I*, these 2-D methods are inevitably misled by a rather isolated sectional noisy/disturbance data that appears sharpest, in theory contrary to the following 3-D approaches.

#### **3.1.3 Three-dimensional approaches**

Recently, a 3-D focus measurement has been introduced by Mahmood & Choi (2008) takes fully advantage of the three spatial dimensions of the original image sequence *I*. It is locally based on a Principal Component Analysis (PCA) within a stack of collected sectional neighborhoods along the *z*-direction. Consequently, it simultaneously exploits all focus cues along the axial (or cross-sectional) *z*-direction in order to estimate sectional degrees of focus. Contrary to 1-D/2-D ones, this novel 3-D strategy would allow to improve the robustness. However, it actually appears ineffective due to a severe loss of sensitivity. Indeed, it finally uses the largest principal component to discriminate in-focus information, which represents the global content of the data. Hence, the authors combine it with various previous transforms, such as discrete wavelet (Mahmood, Shim & Choi, 2009) or cosine (Mahmood et al., 2008) transforms, and lately kernel function (Khan et al., 2010). Alternatively, they perform pre- or post-processings through bilateral filtering (Mahmood, Khan & Choi, 2009) or kernel regression (Mahmood & Choi, 2010), respectively.

**3-D DCT-PCA** Discrete cosine transformation (DCT) over sectional 2-D/3-D windows and discrimination of all axially-collected sectional AC<sup>5</sup> data by the first feature of a Principal Component Analysis (PCA) (Mahmood et al., 2008).

<sup>5</sup> By analogy with an electrical signal, the alternating components of the discrete cosine transform.

#### **3.2 Two-dimensional LIP-based focus measurements**

This first work aims at improving sensitivity to focus cues of usual measurements in order to well operate in difficult regions of the observed surface, such as its ill-illuminated/poor textured parts. Let us start with a brief introduction of the Logarithmic Image Processing (LIP) framework.

#### **3.2.1 Logarithmic image processing (LIP) framework**

An original mathematical framework, the LIP model, has been introduced in the middle of the 1980s for the processing of intensity images valued in a bounded range (Jourlin & Pinoli, 1987; 1988; 2001). This model is mathematically well defined as well as physically consistent. The reader can refer to Pinoli (1997a;b) for a complete mathematical theory and many physical and/or psychophysical connections and justifications about the LIP framework.

#### 3.2.1.1 Mathematical fundamentals

8 Will-be-set-by-IN-TECH

multimodal images of the same scene into a single composite representation. An overview of multiresolution-based schemes for image fusion can be found in (Zhang & Blum, 1999). First, each image section of the original sequence is decomposed into a collection of sub-images at different scales, called a pyramid structure, through alternate combination of convolution and sub-sampling. Different types of details (focus cues) are thus put forward at different levels in the associated pyramid structure. Note that the original image section can be reconstructed by the reverse procedure. A (pixel-based, window-based or region-based (Piella, 2003)) salience measurement (absolute value, sum or variance of absolute values) then tries to quantify focus cues throughout every pyramid structures. The depth map is thus inferred from the largest salience measures. Besides, a composite pyramid structure is constructed by combining coefficients of the original pyramid structures in function of their exhibited salience measures (choose-max or weighted average). Next, a (window-based or region-based) consistency verification is performed on the composite pyramid structure (and on the recovered depth map) so as to check that best salience measures come from the same original image sections, which is equivalent to a smoothing post-processing step. Once the composite pyramid

structure is fused, the final texture image is lastly restored by reverse decomposition.

wavelet coefficients. (Forster et al., 2004).

**3.1.3 Three-dimensional approaches**

**2-D DWT** Use of the Discrete Wavelet Transform (DWT) based on complex Daubechies wavelets as multiresolution analysis, of the largest absolute value of the wavelet coefficients in the subbands (up to 10 levels) as (pixel-based) salience measurement and of both spatial (window of size *r* = 1) and typical subband consistency checks on the

By independently working on each individual image section of the sequence *I*, these 2-D methods are inevitably misled by a rather isolated sectional noisy/disturbance data that

Recently, a 3-D focus measurement has been introduced by Mahmood & Choi (2008) takes fully advantage of the three spatial dimensions of the original image sequence *I*. It is locally based on a Principal Component Analysis (PCA) within a stack of collected sectional neighborhoods along the *z*-direction. Consequently, it simultaneously exploits all focus cues along the axial (or cross-sectional) *z*-direction in order to estimate sectional degrees of focus. Contrary to 1-D/2-D ones, this novel 3-D strategy would allow to improve the robustness. However, it actually appears ineffective due to a severe loss of sensitivity. Indeed, it finally uses the largest principal component to discriminate in-focus information, which represents the global content of the data. Hence, the authors combine it with various previous transforms, such as discrete wavelet (Mahmood, Shim & Choi, 2009) or cosine (Mahmood et al., 2008) transforms, and lately kernel function (Khan et al., 2010). Alternatively, they perform pre- or post-processings through bilateral filtering (Mahmood, Khan & Choi, 2009) or kernel regression (Mahmood & Choi, 2010), respectively.

**3-D DCT-PCA** Discrete cosine transformation (DCT) over sectional 2-D/3-D windows and discrimination of all axially-collected sectional AC<sup>5</sup> data by the first feature of a Principal

<sup>5</sup> By analogy with an electrical signal, the alternating components of the discrete cosine transform.

appears sharpest, in theory contrary to the following 3-D approaches.

Component Analysis (PCA) (Mahmood et al., 2008).

In the LIP model, the intensity of an image is completely represented by its associated gray tone function *f* . Such a function is defined on the spatial suppport **D** and valued in the real number range interval [0, *M*), called the gray tone range. Thereafter, this class of gray tone functions, extended to the real number interval (−∞, *M*) and structured with the after-specified vector addition �<sup>+</sup> , scalar multiplication �× and scalar subtraction �− defines a real vector space denoted *S*:

$$\forall f, g \in \mathcal{S} \quad f \triangle g = f + g - \frac{f\mathcal{S}}{M},$$

$$\forall f \in \mathcal{S}, \forall a \in \mathbb{R} \quad a \triangle f = M - M\left(1 - \frac{f}{M}\right)^a,\tag{6}$$

$$\forall f, g \in \mathcal{S} \quad f \triangle g = M \frac{f - \mathcal{S}}{M - \mathcal{S}}.$$

This gray tone vector space *S* is algebraically and topologically isomorphic to the classical vector space defined on the spatial support **D** with values in the real number set **R** through the mapping *ϕ* (called the isomorphic transformation) defined as:

$$\forall f \in S \quad \varphi(f) = -M \ln \left( 1 - \frac{f}{M} \right),\tag{7}$$

which is the isomorphic transform of the gray tone *f* . The inverse isomorphic transformation *ϕ*−<sup>1</sup> is then defined as:

$$f = \varrho^{-1}(\varrho(f)) = M\left(1 - \exp\left(-\frac{\varrho(f)}{M}\right)\right). \tag{8}$$

In addition to abstract linear algebra, this class of (extended) gray tone functions is an ordered real vector space with the classical order relation ≥ (Pinoli, 1997a).

#### 3.2.1.2 Physical connections

The LIP framework has been proved to be consistent with the transmittance image formation model (Jourlin & Pinoli, 1988), the multiplicative reflectance and transmittance

1 2 3 4 5 6 <sup>7</sup> <sup>8</sup> <sup>9</sup> <sup>1</sup>

(a)

**3.3 Three-dimensional statistical focus measurements**

Original sequence of images

Fernandes et al. (2011a).

transformations or processings.

z z z z z

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 83

5

5

Fig. 5. Illustrations for the **3-D EIG** and **3-D N-EIG** statistical focus measurements: (a)

where *F*max is the saturating light intensity level ("glare limit") (Pinoli, 1997b). First, Weber described the human visual detection between two light intensity values *F* and *G* with a "just noticeable difference". The LIP subtraction *<sup>f</sup>* �− *<sup>g</sup>* is consistent with Weber's law (Pinoli, 1997b). In fact, the LIP model defines specific operations acting directly on the physical light intensity function (stimulus) through the gray tone function notion. A few years after Weber, Fechner established logarithmic relationship between the light intensity *F* (stimulus) and the subjectively perceived brightness *B* (light intensity sensation). It has been shown in Pinoli (1997b) that *B* is an affine map of the isomorphic transform *ϕ*(*f*) of the gray tone *f* . Consequently, the fundamental isomorphism *ϕ* (Eq. 7) of the LIP model should enable to deal with brightness (via the usual operations). About human brightness perception, the aforegiven practical limitation accordingly results in revisited measurements attempting to estimate degree of focus in terms of brightness (intensity sensation from physical light stimuli). Further details about these 2-D LIP-based focus measurements can be found in

This second work conversely aims at creating novel 3-D focus measurements offering a large robustness to noise, while preserving a sufficient sensitivity to focus cues (contrary to the **3-D DCT-PCA** method), in order to well operate through noisy/disturbed acquisitions. In spite of a similar basic tool, the after-described multivariate statistical analyses are totally different than the state-of-the-art **3-D DCT-PCA** method. Moreover, they do not require any previous

From a stack of single-voxels along the *z*-direction of the original sequence *I*(*x*, *y*, *z*) of *n* image sections, 2-D sectional windows of *m* pixels are considered and a multivariate *m*-by-*n* data matrix *X* is formed as shown in Fig. 5(a). The rows of this data matrix *X* referred to as the cross-sectional responses are constituted by the same components of all considered sectional windows. Let (*ei*)*i*∈[1,*n*] denotes the canonical basis of these cross-sectional responses, whose each canonical vector *ei* thus abstracts a different depth *zi* throughout the image sequence. Alternatively, each of the columns referred to as the sectional observations fully corresponds to a different original window at depth *z*. Note that the variability in variance of these sectional observations along the *z*-direction matches with the degree of focus, which is

creation of the multivariate data matrix *X*, (b) canonical basis vs. eigenbasis.

Multivariate data matrix

z z z z z

Crosssectional responses

5

5

5

Sectional observations

Crosssectional responses First eigenvector g

(b)

z

z z

Canonical basis <sup>z</sup> <sup>z</sup>

Fig. 4. Basic diagram representing both theoritical and practical (in red) computation of the LIP-based focus measurements.

image formation models (Pinoli, 1997a) and with several laws and characteristics of human brightness perception (Pinoli, 1997b). In the LIP approach, the gray tone range is inverted contrary to the classical grayscale convention. The relationship between a gray tone function *f*(*x*, *y*) and its corresponding classical grayscale function, denoted ¯ *f*(*x*, *y*), is given by:

$$f(\mathbf{x}, y) = \mathbf{M} - \mathbf{f}(\mathbf{x}, y) \,. \tag{9}$$

Indeed, the limits of the gray tone range [0, *M*) are anticlassically defined: 0 designates the total whiteness, while the real number *M* represents the absolute blackness. This scale inversion has been justified on mathematical reasons (Pinoli, 1997a), and physical (in the setting of transmitted light imaging processes) (Jourlin & Pinoli, 1988; 2001) and psychophysical grounds (Pinoli, 1997b).

#### **3.2.2 Two-dimensional LIP-based focus measurements**

LIP-based focus measurements simply consist in reinterpretations of classical ones using the LIP fremawork (*i.e.* by popularizing, from usual operations +, ×, − to respective LIP ones �<sup>+</sup> , �× , �− (Eq. 6)). For the sake of convenience, we only consider the three more widely used 2-D focus measurements: **2-D VAR**, **2-D TEN** and **2-D SML**. Among all retained methods, other 2-D ones work through various frequency transforms that make their reinterpretations less obvious and the selected 3-D strategy strongly damages the sensitivity. These reinterpretations, denoted **2-D VAR**�, **2-D TEN**� and **2-D SML**�, can be clearly simplified through the use of the LIP fundamental isomorphic *ϕ* (see Fig. 4). Nevertheless, they involve a practical subtlety to succeed from a computional point of view. Indeed, LIP-based focus measurements imply some costly operations (typically such as raising to the square) that are not enough distinguishable in the digitized case. The machine precision does not enable to well discriminate such arithmetics, notably in terms of the classical order relation ≥ for maximizing the resulted degrees of focus. In view of the strictly increasing behaviour of the inverse isomorphic transformation *ϕ*−<sup>1</sup> (Eq. 8), the LIP-based focus measurements can thus be computationally reduced to the computation of the respective classical ones (with usual operations +, ×, −) on isomorphic transform *ϕ* of the gray tone function (see Fig. 4).

In the context of human brightness perception, a gray tone function *f*(*x*, *y*) corresponds to an incident light intensity function *F*(*x*, *y*) by the following relationship:

$$f(\mathbf{x}, y) = M \left( 1 - \frac{F(\mathbf{x}, y)}{F\_{\text{max}}} \right) \tag{10}$$

Fig. 5. Illustrations for the **3-D EIG** and **3-D N-EIG** statistical focus measurements: (a) creation of the multivariate data matrix *X*, (b) canonical basis vs. eigenbasis.

where *F*max is the saturating light intensity level ("glare limit") (Pinoli, 1997b). First, Weber described the human visual detection between two light intensity values *F* and *G* with a "just noticeable difference". The LIP subtraction *<sup>f</sup>* �− *<sup>g</sup>* is consistent with Weber's law (Pinoli, 1997b). In fact, the LIP model defines specific operations acting directly on the physical light intensity function (stimulus) through the gray tone function notion. A few years after Weber, Fechner established logarithmic relationship between the light intensity *F* (stimulus) and the subjectively perceived brightness *B* (light intensity sensation). It has been shown in Pinoli (1997b) that *B* is an affine map of the isomorphic transform *ϕ*(*f*) of the gray tone *f* . Consequently, the fundamental isomorphism *ϕ* (Eq. 7) of the LIP model should enable to deal with brightness (via the usual operations). About human brightness perception, the aforegiven practical limitation accordingly results in revisited measurements attempting to estimate degree of focus in terms of brightness (intensity sensation from physical light stimuli). Further details about these 2-D LIP-based focus measurements can be found in Fernandes et al. (2011a).

#### **3.3 Three-dimensional statistical focus measurements**

10 Will-be-set-by-IN-TECH

(�<sup>+</sup> , �× , �− ) Grayscales Gray tones Degree of focus

Fig. 4. Basic diagram representing both theoritical and practical (in red) computation of the

image formation models (Pinoli, 1997a) and with several laws and characteristics of human brightness perception (Pinoli, 1997b). In the LIP approach, the gray tone range is inverted contrary to the classical grayscale convention. The relationship between a gray tone function

Indeed, the limits of the gray tone range [0, *M*) are anticlassically defined: 0 designates the total whiteness, while the real number *M* represents the absolute blackness. This scale inversion has been justified on mathematical reasons (Pinoli, 1997a), and physical (in the setting of transmitted light imaging processes) (Jourlin & Pinoli, 1988; 2001) and

LIP-based focus measurements simply consist in reinterpretations of classical ones using the LIP fremawork (*i.e.* by popularizing, from usual operations +, ×, − to respective LIP ones �<sup>+</sup> , �× , �− (Eq. 6)). For the sake of convenience, we only consider the three more widely used 2-D focus measurements: **2-D VAR**, **2-D TEN** and **2-D SML**. Among all retained methods, other 2-D ones work through various frequency transforms that make their reinterpretations less obvious and the selected 3-D strategy strongly damages the sensitivity. These reinterpretations, denoted **2-D VAR**�, **2-D TEN**� and **2-D SML**�, can be clearly simplified through the use of the LIP fundamental isomorphic *ϕ* (see Fig. 4). Nevertheless, they involve a practical subtlety to succeed from a computional point of view. Indeed, LIP-based focus measurements imply some costly operations (typically such as raising to the square) that are not enough distinguishable in the digitized case. The machine precision does not enable to well discriminate such arithmetics, notably in terms of the classical order relation ≥ for maximizing the resulted degrees of focus. In view of the strictly increasing behaviour of the inverse isomorphic transformation *ϕ*−<sup>1</sup> (Eq. 8), the LIP-based focus measurements can thus be computationally reduced to the computation of the respective classical ones (with usual operations +, ×, −) on isomorphic transform *ϕ* of the gray tone function (see Fig. 4). In the context of human brightness perception, a gray tone function *f*(*x*, *y*) corresponds to an

*<sup>f</sup>*(*x*, *<sup>y</sup>*) = *<sup>M</sup>* <sup>−</sup> ¯

Isomorphic transformation *ϕ*

*f*(*x*, *y*) and its corresponding classical grayscale function, denoted ¯

incident light intensity function *F*(*x*, *y*) by the following relationship:

*f*(*x*, *y*) = *M*

<sup>1</sup> <sup>−</sup> *<sup>F</sup>*(*x*, *<sup>y</sup>*) *F*max

LIP-based focus measurements.

psychophysical grounds (Pinoli, 1997b).

**3.2.2 Two-dimensional LIP-based focus measurements**

LIP framework

Usual operations (+,×, −)

Inverse isomorphic transformation *ϕ*−1

*f*(*x*, *y*), is given by:

, (10)

*f*(*x*, *y*). (9)

This second work conversely aims at creating novel 3-D focus measurements offering a large robustness to noise, while preserving a sufficient sensitivity to focus cues (contrary to the **3-D DCT-PCA** method), in order to well operate through noisy/disturbed acquisitions. In spite of a similar basic tool, the after-described multivariate statistical analyses are totally different than the state-of-the-art **3-D DCT-PCA** method. Moreover, they do not require any previous transformations or processings.

From a stack of single-voxels along the *z*-direction of the original sequence *I*(*x*, *y*, *z*) of *n* image sections, 2-D sectional windows of *m* pixels are considered and a multivariate *m*-by-*n* data matrix *X* is formed as shown in Fig. 5(a). The rows of this data matrix *X* referred to as the cross-sectional responses are constituted by the same components of all considered sectional windows. Let (*ei*)*i*∈[1,*n*] denotes the canonical basis of these cross-sectional responses, whose each canonical vector *ei* thus abstracts a different depth *zi* throughout the image sequence. Alternatively, each of the columns referred to as the sectional observations fully corresponds to a different original window at depth *z*. Note that the variability in variance of these sectional observations along the *z*-direction matches with the degree of focus, which is the concept of the traditional **2-D VAR** focus measurement. Each sectional observation is centered, and normalized or not by their means (that will finally yield a couple of different focus measurements denoted **3-D EIG** and **3-D NEIG**, respectively). The normalization enables to locally compensate for differences in intensity means between the image sections of the sequence. The covariance matrix *CX* of the sectional observations of *X* is then calculated as follows:

$$\mathbb{C}\_{\mathbf{X}} = \frac{1}{m-1} \, ^t \mathbf{X} \mathbf{X} \, , \tag{11}$$

where *<sup>t</sup>* denotes the transpose operation. Afterwards, *CX* is diagonalized such as:

$$
\mathbb{C}\_X G = \Lambda G \, \, \, \, \, \tag{12}
$$

(a)

ones as well as the other state-of-the-art 2-D approaches.

metric with respect to the ground truth (Gonzalez & Woods, 2008).

**4.1.2 Results & discussion**

<sup>1</sup> <sup>2</sup> <sup>3</sup> <sup>4</sup> <sup>0</sup>

(b)

Neighborhood size *r* (in pixel)

2

Fig. 6. (a) Generation of a simulated sequence of images: Brodatz texture D111 (Brodatz, 1966) and artificial depth map (upper left), 3-D synthetic surface (upper right) and four individual image sections (sections 1, 11, 20 and 30 respectively) of the simulated sequence (lower). (b) Performances (RMSE) of the studied 2-D focus measurements for the simulated data in (a) as a function of the size *r* of the used neighborhood. Graph key: **2-D VAR 2-D VAR 2-D TEN 2-D TEN 2-D SML 2-D SML 2-D OPT 2-D DWT**. Note that the multiresolution-based **2-D DWT** method is put into the *r* = 1 bin, as the size of the window used for the spatial consistency check. The 2-D psychophysical LIP-based focus measurements undoubtedly make fewer errors of restoration than their respective traditional

*z*-direction by constant steps as successive locations of the object focal plane. Afterwards, a sequence is collected by making an image for each of these locations through the 2-D shift-variant linear convolution of the "ideal" image of the surface (*i.e.* the texture image) with a modelled PSF function of the distance of defocus (*i.e.* the distance between the considered location and the depth map). The 2-D PSF is approximated by a 2-D Gaussian function (Pentland, 1987) normalized to account for an uniform illumination (*e.g.* a Köhler illumination) (Forster et al., 2004), whose standard deviation is proportional to the distance of defocus. Two different simulated image sequences are generated with various textural and topographical properties: a first exhibiting some discontinuities to assess accuracy and sensitivity of the studied focus measurements (Fig. 6(a)), and a second one imaging a smoother surface but with additive Gaussian or impulse noises to theoretically test their robustness (Fig. 7(a)). Finally, performances are measured in terms of the root-mean-square-error (RMSE)

The first simulated experiment in Fig. 6 puts most sensitive studied 2-D focus measurements to the test, as a function of the used neighborhood size *r*. It notably aims at evaluating the 2-D psychophysical LIP-based focus measurements (**2-D VAR**, **2-D TEN** and **2-D SML**) versus their respective traditional ones (**2-D VAR**, **2-D TEN** and **2-D SML**). The LIP-based reinterpretations clearly outperform their traditional ones (for any of the intances of *r*). They are more sensivite, that is to say they offer a better capacity to distinguish focus cues of

4

6

RMSE

8

10

12

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 85

in order to obtain both its eigenvalues (*λi*)*i*∈[1,*n*] in increasing order and its eigenvectors (*gi*)*i*∈[1,*n*], diagonal components and columns of the matrixes <sup>Λ</sup> and *<sup>G</sup>* respectively. The eigenvectors form a novel orthornormal basis (EIGenbasis) for the cross-sectional responses of *X*. Each of them is associated with a particular eigenvalue that reveals its captured amount of variance among the total one <sup>∑</sup>*i*∈[1,*n*] *<sup>λ</sup><sup>i</sup>* exhibited by the sectional observations of *X*. During the decomposition process of the covariance matrix *CX*, the first eigenvector *g*<sup>1</sup> accounts for as much of this total variance as possible and the next ones then maximize the remaining total variance, in order and subject to the orthogonality condition. Furthermore, less influential noisy information is, to the greatest extent possible, pushed into least dominant (last) eigenvectors, whereas one of interest remains within the first eigenvectors. Finally, the degree of focus at the depth *zi* (with *i* ∈ [1, *n*]) is the norm of the orthogonal projection of the first eigenvector *g*<sup>1</sup> onto the corresponding canonical vector *ei*, that is simply equal to the absolute value of the *i th* component of *g*1. In the simple schematic example of Fig. 5(b), the largest degree of focus is clearly assigned to the depth *z* of index 3 that maximizes the orthogonal projection norm of the first eigenvector *g*1. Obviously, several first eigenvectors can be considered, *e.g.* the first *K* eigenvectors, hence the sum of their orthogonal projection norms respectively weighted by their eigenvalues is regarded. The **3-D EIG** and **3-D NEIG** focus analyses then become less robust to noise but relatively gain sensitivity to focus cues. Further details about these 3-D statistical focus measurements can be found in Fernandes et al. (2011b; n.d.).

### **4. Results**

Both retained state-of-the-art and novel developed focus measurements will now be illustrated, tested and compared through various simulation and real experiments.

#### **4.1 Performance comparison in simulation**

A first serie of experiments using simulated data is conducted in order to dispose of ground truths for carrying out quantitative assessments of the results produced by all aforementioned methods.

#### **4.1.1 Simulation process & performance assessment**

By first mapping an arbitrary texture onto a simulated depth map (that constitutes the ground truth), an artificial 3-D surface is constructed. This virtual surface is then discretized along the

Fig. 6. (a) Generation of a simulated sequence of images: Brodatz texture D111 (Brodatz, 1966) and artificial depth map (upper left), 3-D synthetic surface (upper right) and four individual image sections (sections 1, 11, 20 and 30 respectively) of the simulated sequence (lower). (b) Performances (RMSE) of the studied 2-D focus measurements for the simulated data in (a) as a function of the size *r* of the used neighborhood. Graph key: **2-D VAR 2-D VAR 2-D TEN 2-D TEN 2-D SML 2-D SML 2-D OPT 2-D DWT**. Note that the multiresolution-based **2-D DWT** method is put into the *r* = 1 bin, as the size of the window used for the spatial consistency check. The 2-D psychophysical LIP-based focus measurements undoubtedly make fewer errors of restoration than their respective traditional ones as well as the other state-of-the-art 2-D approaches.

*z*-direction by constant steps as successive locations of the object focal plane. Afterwards, a sequence is collected by making an image for each of these locations through the 2-D shift-variant linear convolution of the "ideal" image of the surface (*i.e.* the texture image) with a modelled PSF function of the distance of defocus (*i.e.* the distance between the considered location and the depth map). The 2-D PSF is approximated by a 2-D Gaussian function (Pentland, 1987) normalized to account for an uniform illumination (*e.g.* a Köhler illumination) (Forster et al., 2004), whose standard deviation is proportional to the distance of defocus. Two different simulated image sequences are generated with various textural and topographical properties: a first exhibiting some discontinuities to assess accuracy and sensitivity of the studied focus measurements (Fig. 6(a)), and a second one imaging a smoother surface but with additive Gaussian or impulse noises to theoretically test their robustness (Fig. 7(a)). Finally, performances are measured in terms of the root-mean-square-error (RMSE) metric with respect to the ground truth (Gonzalez & Woods, 2008).

#### **4.1.2 Results & discussion**

12 Will-be-set-by-IN-TECH

the concept of the traditional **2-D VAR** focus measurement. Each sectional observation is centered, and normalized or not by their means (that will finally yield a couple of different focus measurements denoted **3-D EIG** and **3-D NEIG**, respectively). The normalization enables to locally compensate for differences in intensity means between the image sections of the sequence. The covariance matrix *CX* of the sectional observations of *X* is then calculated

*m* − 1

in order to obtain both its eigenvalues (*λi*)*i*∈[1,*n*] in increasing order and its eigenvectors

eigenvectors form a novel orthornormal basis (EIGenbasis) for the cross-sectional responses of *X*. Each of them is associated with a particular eigenvalue that reveals its captured amount of variance among the total one <sup>∑</sup>*i*∈[1,*n*] *<sup>λ</sup><sup>i</sup>* exhibited by the sectional observations of *X*. During the decomposition process of the covariance matrix *CX*, the first eigenvector *g*<sup>1</sup> accounts for as much of this total variance as possible and the next ones then maximize the remaining total variance, in order and subject to the orthogonality condition. Furthermore, less influential noisy information is, to the greatest extent possible, pushed into least dominant (last) eigenvectors, whereas one of interest remains within the first eigenvectors. Finally, the degree of focus at the depth *zi* (with *i* ∈ [1, *n*]) is the norm of the orthogonal projection of the first eigenvector *g*<sup>1</sup> onto the corresponding canonical vector *ei*, that is simply equal to

the largest degree of focus is clearly assigned to the depth *z* of index 3 that maximizes the orthogonal projection norm of the first eigenvector *g*1. Obviously, several first eigenvectors can be considered, *e.g.* the first *K* eigenvectors, hence the sum of their orthogonal projection norms respectively weighted by their eigenvalues is regarded. The **3-D EIG** and **3-D NEIG** focus analyses then become less robust to noise but relatively gain sensitivity to focus cues. Further details about these 3-D statistical focus measurements can be found in Fernandes et al.

Both retained state-of-the-art and novel developed focus measurements will now be

A first serie of experiments using simulated data is conducted in order to dispose of ground truths for carrying out quantitative assessments of the results produced by all aforementioned

By first mapping an arbitrary texture onto a simulated depth map (that constitutes the ground truth), an artificial 3-D surface is constructed. This virtual surface is then discretized along the

illustrated, tested and compared through various simulation and real experiments.

*t*

, diagonal components and columns of the matrixes Λ and *G* respectively. The

*XX* , (11)

*CXG* = Λ*G* , (12)

*th* component of *g*1. In the simple schematic example of Fig. 5(b),

*CX* <sup>=</sup> <sup>1</sup>

where *<sup>t</sup>* denotes the transpose operation. Afterwards, *CX* is diagonalized such as:

as follows:

(*gi*)*i*∈[1,*n*]

the absolute value of the *i*

**4.1 Performance comparison in simulation**

**4.1.1 Simulation process & performance assessment**

(2011b; n.d.).

**4. Results**

methods.

The first simulated experiment in Fig. 6 puts most sensitive studied 2-D focus measurements to the test, as a function of the used neighborhood size *r*. It notably aims at evaluating the 2-D psychophysical LIP-based focus measurements (**2-D VAR**, **2-D TEN** and **2-D SML**) versus their respective traditional ones (**2-D VAR**, **2-D TEN** and **2-D SML**). The LIP-based reinterpretations clearly outperform their traditional ones (for any of the intances of *r*). They are more sensivite, that is to say they offer a better capacity to distinguish focus cues of

(a) Section 1 (b) Section 14 (c) Section 28 (d) Section 42

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 87

Fig. 8. Some individual 2-D image sections among the 42 constituting the image sequence of

Fig. 9. Some individual 2-D image sections among the 50 constituting the image sequence of the Vickers hardness test. This sequence was imaged in steps of 9 *μ*m through a reflected white-light microscope equipped with a × 10 / 0.3 NA objective in air immersion. Each image section is 766 × 573 pixels, representating 1262 × 944 *μ*m. The marked regions A and B

A

B

A

C

B

Fig. 10. Some individual 2-D image sections among the 40 constituting the image sequence of

the human *ex-vivo* corneal endothelium. This sequence was imaged in steps of 4.5 *μ*m through a transmitted white-light microscope equipped with a × 10 / 0.25 NA objective in air immersion. Each image section is 1040 × 772 pixels, representating 718 × 533 *μ*m. Note that both bottom left corner and right edge regions never appear in-focus throughout the sequence. The marked regions A, B and C will be used as sites for comparing the different restored textures. Some cell fragments present in the immersion biochemical solution are clearly visible on (a) and (b) as small dark spots, *e.g.* throughout the region B. Futhermore, some contrast reversals emerge: the endothelial cell borders, which are normally darker than

(c) Section 34

(c) Section 20

(d) Section 50

(d) Section 30

A

B

A

C

B

the grain of sand. This sequence was imaged in steps of 3.2 *μ*m through a reflected white-light microscope equipped with a × 20 / 0.46 NA objective in air immersion. Each

image section is 766 × 573 pixels, representating 635 × 475 *μ*m.

A

B

(b) Section 17

will be used as sites for comparing the different restored textures.

C

A

B

(b) Section 10

the cell bodies, look brighter for a specific range of distances of defocus.

(a) Section 1

(a) Section 1

A

B

A

C

B

Fig. 7. (a) Generation of a simulated sequence of images: Brodatz texture D5 (Brodatz, 1966) and artificial depth map (upper left), 3-D synthetic surface (upper right) and four individual image sections (sections 1, 11, 20 and 30 respectively) of the simulated sequence (lower). (b-c) Performances (RMSE) of the most robust studied focus measurements for the simulated data in (a) under various noisy conditions (*r* = 8 pixels). The proposed 3-D statistical analyses **3-D EIG** and **3-D NEIG** with *K* set to 1 make fewer errors of restoration in presence of artificial impulsive or Gaussian noises.

poor contrasted/textured or ill-illuminated regions, but at the expense of a sligh loss of robustess. Notice that the improvements are even more obvious for smaller neighborhood sizes. This enables to employ smaller operating windows that smooth less the restoration process, most notably around sharp depth slopes or even discontinuities of the observed surface. Incidentally, LIP-based focus measurements also make fewer restoration errors than the other 2-D retained methods. On account of its multiresolution analysis, the **2-D DWT** approach avoids operating over fixed-size windows, but does not guarantee stability in return. As for the **2-D OPT** focus measurement, its band-pass filter designed for offering robustess inevitably damages the sensitivity, a bit like 3-D approaches that favour robustness to sensitivity.

In Fig. 7, the second simulated test studies most robust aforementioned focus measurements under various artificial noisy conditions. In view of the fact that the synthetic depth map

14 Will-be-set-by-IN-TECH

0

2

4

6

RMSE

Fig. 7. (a) Generation of a simulated sequence of images: Brodatz texture D5 (Brodatz, 1966) and artificial depth map (upper left), 3-D synthetic surface (upper right) and four individual image sections (sections 1, 11, 20 and 30 respectively) of the simulated sequence (lower). (b-c) Performances (RMSE) of the most robust studied focus measurements for the simulated data in (a) under various noisy conditions (*r* = 8 pixels). The proposed 3-D statistical analyses **3-D EIG** and **3-D NEIG** with *K* set to 1 make fewer errors of restoration in presence of

poor contrasted/textured or ill-illuminated regions, but at the expense of a sligh loss of robustess. Notice that the improvements are even more obvious for smaller neighborhood sizes. This enables to employ smaller operating windows that smooth less the restoration process, most notably around sharp depth slopes or even discontinuities of the observed surface. Incidentally, LIP-based focus measurements also make fewer restoration errors than the other 2-D retained methods. On account of its multiresolution analysis, the **2-D DWT** approach avoids operating over fixed-size windows, but does not guarantee stability in return. As for the **2-D OPT** focus measurement, its band-pass filter designed for offering robustess inevitably damages the sensitivity, a bit like 3-D approaches that favour robustness

In Fig. 7, the second simulated test studies most robust aforementioned focus measurements under various artificial noisy conditions. In view of the fact that the synthetic depth map

8

10

**2-D VAR 2-D TEN 2-D OPT 3-D DCT-PCA 3-D EIG** *K***=1 3-D NEIG** *K***=1**

0 0.02 0.04 0.06 0.08 0.1 0.1

Impulse noise density (c) Impulsive noise

(a)

0 0.03 0.06 0.09 0.12 0.15 0.1

artificial impulsive or Gaussian noises.

Gaussian noise standard deviation (b) Gaussian noise

to sensitivity.

RMSE

Fig. 8. Some individual 2-D image sections among the 42 constituting the image sequence of the grain of sand. This sequence was imaged in steps of 3.2 *μ*m through a reflected white-light microscope equipped with a × 20 / 0.46 NA objective in air immersion. Each image section is 766 × 573 pixels, representating 635 × 475 *μ*m.

Fig. 9. Some individual 2-D image sections among the 50 constituting the image sequence of the Vickers hardness test. This sequence was imaged in steps of 9 *μ*m through a reflected white-light microscope equipped with a × 10 / 0.3 NA objective in air immersion. Each image section is 766 × 573 pixels, representating 1262 × 944 *μ*m. The marked regions A and B will be used as sites for comparing the different restored textures.

Fig. 10. Some individual 2-D image sections among the 40 constituting the image sequence of the human *ex-vivo* corneal endothelium. This sequence was imaged in steps of 4.5 *μ*m through a transmitted white-light microscope equipped with a × 10 / 0.25 NA objective in air immersion. Each image section is 1040 × 772 pixels, representating 718 × 533 *μ*m. Note that both bottom left corner and right edge regions never appear in-focus throughout the sequence. The marked regions A, B and C will be used as sites for comparing the different restored textures. Some cell fragments present in the immersion biochemical solution are clearly visible on (a) and (b) as small dark spots, *e.g.* throughout the region B. Futhermore, some contrast reversals emerge: the endothelial cell borders, which are normally darker than the cell bodies, look brighter for a specific range of distances of defocus.

A

B

sharper.

regions for a better visibility.

**4.2.2 Results & discussion**

(a) **2-D TEN** (b) **2-D TEN** (c) **2-D OPT** (d) **2-D DWT** (e) **3-D**

Fig. 12. Details of the restored textures in the regions A and B for the image sequence of the grain of sand presented in Fig. 9 (*r* = 4 pixels). In the details (b) and (f) resulting from our suggested methods, there are less bright artefacts and the grain borders appear darker and

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 89

motorized stage, but through different configurations (using reflected or transmitted light) and magnifications. Related to various application fields, three samples with varying textural and topographical properties are regarded so as to rigorously test both selected and proposed methods. The first two ones are a Vickers hardness test6 performed on a polished aluminium plate surface and a grain of sand; their reflected white-light acquisitions are illustrated and described in Fig. 8 and Fig. 9, respectively. These real image sequences exhibit some difficult regions: *e.g.* around the sharp borders of the sand grain and at the bottom of the Vickers pyramid-shaped indentation, thus requiring a good sensitivity from the focus measurements. Moreover, they are necessarly degraded by some noisy data introduced by the imaging system during the acquisition, but in much lesser extent than the third one. This latter, illustrated and described in Fig. 10, images using transmitted white-light an human *ex-vivo* corneal endothelium<sup>7</sup> folded after storage of the graft in a specific preservation medium (Pels & Schuchard, 1983). Effectively, it appears very disturbed by intense contrast reversals and some cell fragments present in the graft immersion solution. For these real image sequences, the assessment will be only qualitative, *i.e.* by visually examining and comparing the restored depth maps and/or textures; these latter will be highlighted in some crucial

The topography of the Vickers hardness test reconstructed by the major part of the aforementioned focus measurements are shown in Fig. 11. Those related to proposed methods clearly exhibit less artefacts (*e.g.* wrong sharp peaks), notably at the bottom and the edges of

<sup>6</sup> The test of Vickers consists in examining the deformation of a material from a standard pyramid-shaped

<sup>7</sup> The endothelium is the innermost layer of the cornea and is constituted of a monolayer and hexagonal mosaic of cells. Given that those non-regenerative cells make keeping the cornea clear, the estimation

of its cell density is essential in the corneal transplant process (Thuret et al., 2004; 2003).

diamond indenter to deduce a measure of hardness (Tabor, 2000).

**DCT-PCA**

(f) **3-D NEIG**

*K***=1**

Fig. 11. Reconstructed depth maps for the image sequence of the Vickers hardness test presented in Fig. 8 (*r* = 3 pixels) after a median filtering (*r* = 2 pixels). The color *z*-scale is: **0** • • **30** • • **60** • • **90** • • 120 *μm* •. The depth maps recovered by our proposed methods (b),(e), (g) and (h) more reveal the pyramid-shaped structure of the sample.

exhibits neither sharp depth slopes nor discontinuities, we opt for a rather large neighborhood size *r* (*r* = 8 pixels), moreover necessary to average out noise. In presence of noise, the proposed **3-D EIG** and **3-D NEIG** methods with *K* set to 1 clearly outperform the state-of-the-art other ones. The adopted 3-D statistical strategies make possible a better discrimination of focus cues "drowned" in noise. Notice that the **3-D EIG** version offers a more robust behaviour than the normalized **3-D NEIG** one. As for the other 3-D focus measurement (**3-D DCT-PCA**), it shows the weakest performances by lack of sensitivity, the previous transformation being not sufficient to improve it.

#### **4.2 Results on experimental data**

We now illustrate the potential of the suggested focus measurements on real image sequence acquisitions.

#### **4.2.1 Experimental setup**

The real test dataset is made up of three image sequences exclusively acquired in conventional optical microscopy by gradually shifting the samples along the optical axis direction with a

Fig. 12. Details of the restored textures in the regions A and B for the image sequence of the grain of sand presented in Fig. 9 (*r* = 4 pixels). In the details (b) and (f) resulting from our suggested methods, there are less bright artefacts and the grain borders appear darker and sharper.

motorized stage, but through different configurations (using reflected or transmitted light) and magnifications. Related to various application fields, three samples with varying textural and topographical properties are regarded so as to rigorously test both selected and proposed methods. The first two ones are a Vickers hardness test6 performed on a polished aluminium plate surface and a grain of sand; their reflected white-light acquisitions are illustrated and described in Fig. 8 and Fig. 9, respectively. These real image sequences exhibit some difficult regions: *e.g.* around the sharp borders of the sand grain and at the bottom of the Vickers pyramid-shaped indentation, thus requiring a good sensitivity from the focus measurements. Moreover, they are necessarly degraded by some noisy data introduced by the imaging system during the acquisition, but in much lesser extent than the third one. This latter, illustrated and described in Fig. 10, images using transmitted white-light an human *ex-vivo* corneal endothelium<sup>7</sup> folded after storage of the graft in a specific preservation medium (Pels & Schuchard, 1983). Effectively, it appears very disturbed by intense contrast reversals and some cell fragments present in the graft immersion solution. For these real image sequences, the assessment will be only qualitative, *i.e.* by visually examining and comparing the restored depth maps and/or textures; these latter will be highlighted in some crucial regions for a better visibility.

#### **4.2.2 Results & discussion**

16 Will-be-set-by-IN-TECH

(a) **2-D VAR** (b) **2-D VAR**� (c) **2-D OPT**

(d) **2-D SML** (e) **2-D SML**� (f) **2-D DWT**

(g) **3-D EIG** *K***=1** (h) **3-D NEIG** *K***=1** (i) **3-D DCT-PCA**

exhibits neither sharp depth slopes nor discontinuities, we opt for a rather large neighborhood size *r* (*r* = 8 pixels), moreover necessary to average out noise. In presence of noise, the proposed **3-D EIG** and **3-D NEIG** methods with *K* set to 1 clearly outperform the state-of-the-art other ones. The adopted 3-D statistical strategies make possible a better discrimination of focus cues "drowned" in noise. Notice that the **3-D EIG** version offers a more robust behaviour than the normalized **3-D NEIG** one. As for the other 3-D focus measurement (**3-D DCT-PCA**), it shows the weakest performances by lack of sensitivity, the

We now illustrate the potential of the suggested focus measurements on real image sequence

The real test dataset is made up of three image sequences exclusively acquired in conventional optical microscopy by gradually shifting the samples along the optical axis direction with a

Fig. 11. Reconstructed depth maps for the image sequence of the Vickers hardness test presented in Fig. 8 (*r* = 3 pixels) after a median filtering (*r* = 2 pixels). The color *z*-scale is: **0** • • **30** • • **60** • • **90** • • 120 *μm* •. The depth maps recovered by our proposed methods (b),(e),

(g) and (h) more reveal the pyramid-shaped structure of the sample.

previous transformation being not sufficient to improve it.

**4.2 Results on experimental data**

**4.2.1 Experimental setup**

acquisitions.

The topography of the Vickers hardness test reconstructed by the major part of the aforementioned focus measurements are shown in Fig. 11. Those related to proposed methods clearly exhibit less artefacts (*e.g.* wrong sharp peaks), notably at the bottom and the edges of

<sup>6</sup> The test of Vickers consists in examining the deformation of a material from a standard pyramid-shaped diamond indenter to deduce a measure of hardness (Tabor, 2000).

<sup>7</sup> The endothelium is the innermost layer of the cornea and is constituted of a monolayer and hexagonal mosaic of cells. Given that those non-regenerative cells make keeping the cornea clear, the estimation of its cell density is essential in the corneal transplant process (Thuret et al., 2004; 2003).

ABC

(a) **2-D VAR**

(b) **2-D OPT**

(c) **3-D DCT-PCA**

(d) **3-D EIG K=1**

exhibit less artefacts attributed to both disturbances.

moving cell fragments and contrast reversals, respectively.

Fig. 14. Reconstructed depth maps (left) and details of the restored textures in the regions A, B and C (right) for the image sequence of the human *ex-vivo* corneal endothelium presented in Fig. 10 (*r* = 10 pixels). The color *z*-scale is: **0** • • 29.25 • • 58.5 • • 87.75 • • 117 • • 146.25 • • 175.5 *μ*m. The **3-D EIG K=1** depth map in (d) distincly contains fewer blue spots and over-red regions respectively caused by moving cell fragments and cell border contrast reversals, moreover attested by the details of its respective texture in (d) that noticeably

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 91

is not too much damaged by disturbances, like dark steaks and bright cell borders due to

Fig. 13. 3-D surface reconstructions (a) of the grain of sand using the **3-D NEIG** method and (b) of the Vickers hardness test using the **2-D VAR** method.

the pyramid-shaped indentation. Compared to classical **2-D VAR** and **2-D SML** methods, the respective psyschophysical LIP-based reinterpretations (**2-D VAR** and **2-D SML**) are able to deal with more difficult regions, such as poorly textured and/or ill-illuminated ones. Moreover, they offer a relative robustness to noise sufficient for most real usual cases. In same cases, the more sensitive normalized **3-D NEIG** method with *K* set to 1 will be preferred to **3-D EIG** *K***=1** one, except for much noisier acquisitions as encountered in the last example below. The normalization effectively provides some accuracy and stability to the analysis, up to a certain degree of noise in the image sequence. A 3-D reconstruction of the Vickers hardness test is shown in Fig. 13(a).

Concerning the grain of sand, the textures resulting from a more restricted set of aforementioned focus measurements are highlighted and compared in Fig. 12. First, the light-gray stains around the grain corner of the region A that designate false textural restorations are much less frequent with our suggested **2-D TEN** and, even more so, **3-D NEIG** *K***=1** methods. Second, the inspection of the grain borders within B clearly reveals marked improvements with the same **2-D TEN** and **3-D NEIG** *<sup>K</sup>***=1** methods. As previously, there are less bright artefacts in and around the borders, which moreover appear much darker and sharper. As previously, a 3-D reconstruction of the grain of sand is shown in Fig. 13(b), in which a binary mask is used so as to exclude the background from the reconstruction process (Niederöst et al., 2003).

In Fig. 14, we compare both depth map and texture obtained with the most robust studied focus measurements from the noisy and disturbed image sequence of the corneal endothelium. Contrary to above real examples, a larger neighborhood size (*r* = 10 pixels) is used, because of both wider textural content and noisier aspect of the image sequence. Moreover, this is here non-prejudicial in view of the complete absence of discontinuities and sharp slopes. First, the depth map recovered by the proposed **3-D EIG** *K***=1** method clearly exhibits less artefacts, anatomically impossible as the endothelial surface is necessarily continuous. Indeed, it distinctly contains less underestimated (over-red) and overestimated (blue) regions caused by cell fragments and contrast reversals, respectively. As for the restored textures, their inspection corroborates the above appreciation (moreover knowing that each of them is intimately related to its respective depth map). The texture tagged with **3-D EIG** *K***=1**

18 Will-be-set-by-IN-TECH

(a) (b)

Fig. 13. 3-D surface reconstructions (a) of the grain of sand using the **3-D NEIG** method and

the pyramid-shaped indentation. Compared to classical **2-D VAR** and **2-D SML** methods, the respective psyschophysical LIP-based reinterpretations (**2-D VAR** and **2-D SML**) are able to deal with more difficult regions, such as poorly textured and/or ill-illuminated ones. Moreover, they offer a relative robustness to noise sufficient for most real usual cases. In same cases, the more sensitive normalized **3-D NEIG** method with *K* set to 1 will be preferred to **3-D EIG** *K***=1** one, except for much noisier acquisitions as encountered in the last example below. The normalization effectively provides some accuracy and stability to the analysis, up to a certain degree of noise in the image sequence. A 3-D reconstruction of the Vickers hardness

Concerning the grain of sand, the textures resulting from a more restricted set of aforementioned focus measurements are highlighted and compared in Fig. 12. First, the light-gray stains around the grain corner of the region A that designate false textural restorations are much less frequent with our suggested **2-D TEN** and, even more so, **3-D NEIG** *K***=1** methods. Second, the inspection of the grain borders within B clearly reveals marked improvements with the same **2-D TEN** and **3-D NEIG** *<sup>K</sup>***=1** methods. As previously, there are less bright artefacts in and around the borders, which moreover appear much darker and sharper. As previously, a 3-D reconstruction of the grain of sand is shown in Fig. 13(b), in which a binary mask is used so as to exclude the background from the reconstruction process

In Fig. 14, we compare both depth map and texture obtained with the most robust studied focus measurements from the noisy and disturbed image sequence of the corneal endothelium. Contrary to above real examples, a larger neighborhood size (*r* = 10 pixels) is used, because of both wider textural content and noisier aspect of the image sequence. Moreover, this is here non-prejudicial in view of the complete absence of discontinuities and sharp slopes. First, the depth map recovered by the proposed **3-D EIG** *K***=1** method clearly exhibits less artefacts, anatomically impossible as the endothelial surface is necessarily continuous. Indeed, it distinctly contains less underestimated (over-red) and overestimated (blue) regions caused by cell fragments and contrast reversals, respectively. As for the restored textures, their inspection corroborates the above appreciation (moreover knowing that each of them is intimately related to its respective depth map). The texture tagged with **3-D EIG** *K***=1**

(b) of the Vickers hardness test using the **2-D VAR** method.

test is shown in Fig. 13(a).

(Niederöst et al., 2003).

Fig. 14. Reconstructed depth maps (left) and details of the restored textures in the regions A, B and C (right) for the image sequence of the human *ex-vivo* corneal endothelium presented in Fig. 10 (*r* = 10 pixels). The color *z*-scale is: **0** • • 29.25 • • 58.5 • • 87.75 • • 117 • • 146.25 • • 175.5 *μ*m. The **3-D EIG K=1** depth map in (d) distincly contains fewer blue spots and over-red regions respectively caused by moving cell fragments and cell border contrast reversals, moreover attested by the details of its respective texture in (d) that noticeably exhibit less artefacts attributed to both disturbances.

is not too much damaged by disturbances, like dark steaks and bright cell borders due to moving cell fragments and contrast reversals, respectively.

**7. References**

Agard, D. A. (1984). Optical sectioning microscopy: Cellular architecture in three dimensions,

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 93

Ahmad, M. B. & Choi, T. S. (2005). A heuristic approach for finding best focused shape, *IEEE Transactions on Circuits and Systems for Video Technology* 15(4): 566–574. Asif, M. & Choi, T. S. (2001). Shape from focus using multilayer feedforward neural networks,

Boddeke, F. R., Van Vliet, L. J., Netten, H. & Young, I. T. (1994). Autofocusing in microscopy

Born, M. & Wolf, E. (1991). *Principles of Optics - Electromagnetic Theory of Propagation Interference and Diffraction of Light*, 6th (corrected) edn, Pergamon Press, New York, USA. Brenner, J. F., Dew, B. S., Horton, J. B., King, T., Neurath, P. W. & Selles, W. D. (1976).

Brodatz, P. (1966). *Textures: A Photographic Album for Artists and Designers*, Dover Publications,

Burt, P. & Adelson, E. (1983). The laplacian pyramid as a compact image code, *IEEE*

Burt, P. J. & Kolczynski, R. J. (1993). Enhanced image capture through fusion, *Proceedings of the IEEE International Conference on Computer Vision*, Berlin, Germany, pp. 173–182. Darrell, T. & Wohn, K. (1988). Pyramid based depth from focus, *Proceedings of the*

Fernandes, M., Gavet, Y. & Pinoli, J.-C. (2011a). Improving focus measurements using

Fernandes, M., Gavet, Y. & Pinoli, J.-C. (2011b). Robust shape-from-focus by 3-D multivariate

Fernandes, M., Gavet, Y. & Pinoli, J.-C. (n.d.). Robust 3-D reconstruction of surfaces from

Forster, B., Van De Ville, D., Berent, J., Sage, D. & Unser, M. (2004). Complex wavelets for

Gonzalez, R. C. & Woods, R. E. (2008). *Digital Image Processing*, 3rd edn, Prentice-Hall, Inc.,

Groen, F. C. A., Young, I. T. & Ligthart, G. (1985). A comparison of different focus functions

Helmli, F. S. & Scherer, S. (2001). Adaptive shape from focus with an error estimation in

Jourlin, M. & Pinoli, J.-C. (1987). Logarithmic image processing, *Acta Stereologica* 6: 651–656. Jourlin, M. & Pinoli, J.-C. (1988). A model for logarithmic image processing, *Journal of*

Horn, B. K. P. (2001). *Robot Vision*, 13th edn, The MIT Press, Cambridge, MA, USA.

logarithmic image processing, *Journal of Microscopy* 242(3): 228–241.

Automated microscope for cytologic research: Preliminary evaluation, *Journal of*

*IEEE Conference on Computer Vision and Pattern Recognition*, Ann Arbor, MI , USA,

statistical analyses, *Proceedings of the IEEE International Conference on Image Processing*,

image focus by local cross-sectional multivariate statistical analyses: application to

extended depth-of-field: A new method for the fusion of multichannel microscopy

light microscopy, *Proceedings of the IEEE International Symposium on Image and Signal*

*Annual Review of Biophysics and Bioengineering* 13: 191–219.

*IEEE Transactions on Image Processing* 10(11): 1670–1675.

based on the otf and sampling, *Bioimaging* 2(4): 193–203.

*Histochemistry & Cytochemistry* 24(1): 100–111.

*Transactions on Communications* 31: 532–540.

Brussels, Belgium, pp. 2113–2116.

Upper Saddle River, NJ, USA.

*Microscopy* 149: 21–35.

human ex-vivo corneal endotheliums, Submitted.

images, *Microscopy Research and Technique* 65(1-2): 33–42.

for use in autofocus algorithms, *Cytometry* 6(2): 81–91.

*Processing and Analysis*, Pula , Croatia, pp. 188–193.

New York, USA.

pp. 504–509.

## **5. Conclusions**

This chapter has focused on image restoration of both topographical and textural information of an observed surface from a registered image sequence acquired by optical sectioning through the common concepts of Shape-From-Focus (SFF) and Extended Depth-of-Field (EDF). More particularly, the essential step of these complementary processes of restoration: the focus measurement, has been examined. After a brief specialized review, we have introduced novel evolved focus measurements that push the limits of state-of-the-art ones in terms of sensitivity and robustness, in order to cope with various frequently encountered acquisition issues.

On the one hand, reinterpretations with the LIP framework (**2-D VAR**, **2-D TEN** and **2-D SML**) of three traditional 2-D focus measurements (**2-D VAR**, **2-D TEN** and **2-D SML**) have been suggested. From a computational point of view, they involve some subtleties to succeed that, about human brightness perception, accordingly result in revisited focus measurements attempting to work in terms of brightness (intensity sensation from physical light stimuli). Firstly designed to deal with difficult ill-illuminated and poor textured parts of the obserbed surface, the strategy of using the LIP model effectively confers higher sensitivity to focus cues, at the expense of a slight loss of noise robustness that nevertheless remains sufficient in most usual cases. On the other hand, novel 3-D statistical focus measurements (**3-D EIG** and **3-D NEIG**) have been developed in order to conversely handle noisy and disturbed acquisitions. Contrary to 2-D sectional way adopted by the major part of the current methods, a 3-D strategy is originally achieved throughout the image sequence via multivariate statistical analyses within local stacks of collected 2-D sectional windows along the axial direction, thereby offering a strong robustness to noise while preserving a sufficient sensitivity (contrary to the state-of-the-art **3-D DCT-PCA** one). The efficiency of all proposed focus measurements have been clearly demonstrated on simulated data and real experimental acquisitions.

The concept of reinterpreting traditional focus measurements through the LIP framework is obviously restricted to neither image processing frameworks nor to specific focus measurements. While the studied focus measurements are illustrated in the context of conventional optical microscopy, they are also applicable to the wider range of imaging systems offering a limited depth-of-field, provided the acquired image sequence is previously registered. Morever, they can be used for all application issues requiring focus degree information (obviously after considering the focus measurement strategy and dimensionality), such as autofocusing. Finally, we believe that the use of adaptive windows instead of fixed-size ones for measuring degrees of focus could improve the restoration process, with a view to always capturing focus cues (whatever the textural content of the observed surface) while reducing the inherent smoothing effect (around sharp depth slopes and discontinuities of the observed surface).

## **6. Acknowledgments**

The authors would like to thank the different partners from the University Hospital Center of Saint-Etienne, and from the LPMG and PECM CNRS Units in France who have kindly supplied the different original images studied in this chapter.

#### **7. References**

20 Will-be-set-by-IN-TECH

This chapter has focused on image restoration of both topographical and textural information of an observed surface from a registered image sequence acquired by optical sectioning through the common concepts of Shape-From-Focus (SFF) and Extended Depth-of-Field (EDF). More particularly, the essential step of these complementary processes of restoration: the focus measurement, has been examined. After a brief specialized review, we have introduced novel evolved focus measurements that push the limits of state-of-the-art ones in terms of sensitivity and robustness, in order to cope with various frequently encountered

On the one hand, reinterpretations with the LIP framework (**2-D VAR**, **2-D TEN** and **2-D SML**) of three traditional 2-D focus measurements (**2-D VAR**, **2-D TEN** and **2-D SML**) have been suggested. From a computational point of view, they involve some subtleties to succeed that, about human brightness perception, accordingly result in revisited focus measurements attempting to work in terms of brightness (intensity sensation from physical light stimuli). Firstly designed to deal with difficult ill-illuminated and poor textured parts of the obserbed surface, the strategy of using the LIP model effectively confers higher sensitivity to focus cues, at the expense of a slight loss of noise robustness that nevertheless remains sufficient in most usual cases. On the other hand, novel 3-D statistical focus measurements (**3-D EIG** and **3-D NEIG**) have been developed in order to conversely handle noisy and disturbed acquisitions. Contrary to 2-D sectional way adopted by the major part of the current methods, a 3-D strategy is originally achieved throughout the image sequence via multivariate statistical analyses within local stacks of collected 2-D sectional windows along the axial direction, thereby offering a strong robustness to noise while preserving a sufficient sensitivity (contrary to the state-of-the-art **3-D DCT-PCA** one). The efficiency of all proposed focus measurements have been clearly demonstrated on simulated data and real experimental

The concept of reinterpreting traditional focus measurements through the LIP framework is obviously restricted to neither image processing frameworks nor to specific focus measurements. While the studied focus measurements are illustrated in the context of conventional optical microscopy, they are also applicable to the wider range of imaging systems offering a limited depth-of-field, provided the acquired image sequence is previously registered. Morever, they can be used for all application issues requiring focus degree information (obviously after considering the focus measurement strategy and dimensionality), such as autofocusing. Finally, we believe that the use of adaptive windows instead of fixed-size ones for measuring degrees of focus could improve the restoration process, with a view to always capturing focus cues (whatever the textural content of the observed surface) while reducing the inherent smoothing effect (around sharp depth slopes

The authors would like to thank the different partners from the University Hospital Center of Saint-Etienne, and from the LPMG and PECM CNRS Units in France who have kindly

**5. Conclusions**

acquisition issues.

acquisitions.

and discontinuities of the observed surface).

supplied the different original images studied in this chapter.

**6. Acknowledgments**


Pajares, G. & de la Cruz, J. M. (2004). A wavelet-based image fusion tutorial, *Pattern*

Surface Topography and Texture Restoration from Sectional Optical Imaging by Focus Analysis 95

Pels, E. & Schuchard, Y. (1983). Organ-culture preservation of human corneas, *Documenta*

Pentland, A. P. (1987). A new sense for depth of field, *IEEE Transactions on Pattern Analysis and*

Piella, G. (2003). A general framework for multiresolution image fusion: From pixels to

Pieper, R. J. & Korpel, A. (1983). Image processing for extended depth of field, *Applied Optics*

Pinoli, J.-C. (1997a). A general comparative study of the multiplicative homomorphic,

Pinoli, J.-C. (1997b). The logarithmic image processing model: Connections with human

Pradeep, K. S. & Rajagopalan, A. N. (2007). Improving shape from focus using defocus cue,

Subbarao, M. & Choi, T. (1995). Accurate recovery of three-dimensional shape from image focus, *IEEE Transactions on Pattern Analysis and Machine Intelligence* 17(3): 266–274. Subbarao, M., Choi, T. & Nikzad, A. (1993). Focusing techniques, *Optical Engineering*

Sugimoto, S. A. & Ichioka, Y. (1985). Digital composition of images with increased depth of focus considering depth information, *Applied Optics* 24(14): 2076–2080.

Thuret, G., Manissolle, C., Acquart, S., Garraud, O., Campos-Guyotat, L., Maugery, J. & Gain,

Thuret, G., Manissolle, C., Acquart, S., Le Petit, J. C., Maugery, J., Campos-Guyotat, L.,

Toet, A. (1989). Image fusion by a ratio of low-pass pyramid, *Pattern Recognition Letters*

Valdecasas, A. G., Marshall, D., Becerra, J. M. & Terrero, J. J. (2001). On the extended depth of

Vollath, D. (1987). Automatic focusing by correlative methods, *Journal of Microscopy*

Watanabe, M. & Nayar, S. K. (1997). Telecentric optics for focus analysis, *IEEE Transactions on*

Wee, C. Y. & Paramesran, R. (2007). Measure of image sharpness using eigenvalues,

Willson, R. G. (1994). Modeling and calibration of automated zoom lenses, *Technical Report*

*CMU-RI-TR-94-03*, The Robotics Institute, Carnegie Mellon University, Pittsburgh,

focus algorithms for bright field microscopy, *Micron* 32(6): 559–569.

*Pattern Analysis and Machine Intelligence* 19(12): 1360–1365.

P. (2004). Urgent need for normalization of corneal graft quality controls in french

Doughty, M. J. & Gain, P. (2003). Is manual counting of corneal endothelial cell density in eye banks still acceptable? The French experience, *British Journal of*

*IEEE Transactions on Image Processing* 16(7): 1920–1925.

Tabor, D. (2000). *The Hardness of Metals*, Oxford University Press, Oxford, UK.

eye banks, *Transplantation* 78(9): 1299–1302.

*Ophthalmology* 87(12): 1481–1486.

*Information Sciences* 177(12): 2533–2552.

log-ratio and logarithmic image processing approaches, *Signal Processing* 58(1): 11–45.

brightness perception and contrast estimators, *Journal of Mathematical Imaging and*

*Recognition* 37(9): 1855–1872.

22(10): 1449–1453.

*Vision* 7(4): 341–358.

32(11): 2824–2836.

9(4): 245–253.

147: 279–288.

PA, USA.

*Ophthalmologica* 56(1-2): 147–153.

*Machine Intelligence* 9(4): 523–531.

regions, *Information Fusion* 4(4): 259–280.


22 Will-be-set-by-IN-TECH

Jourlin, M. & Pinoli, J. C. (2001). Logarithmic image processing - the mathematical and

Khan, A., Mahmood, M. T. & Choi, T. S. (2010). A nonlinear transform based

Kristan, M., Perš, J., Perše, M. & Kovaˇciˇc, S. (2006). A bayes-spectral-entropy-based

Liu, Z., Tsukada, K., Hanasaki, K., Ho, Y. K. & Dai, Y. P. (2001). Image fusion by using steerable

Mahajan, V. N. (1998). *Optical Imaging and Aberrations Part I: Ray Geometrical Optics*, SPIE Press,

Mahajan, V. N. (2001). *Optical Imaging and Aberrations Part II: Wave Diffraction Optics*, SPIE

Mahmood, M. T. & Choi, T. S. (2008). A feature analysis approach to estimate 3D shape from

Mahmood, M. T. & Choi, T. S. (2010). 3D shape recovery from image focus using kernel

Mahmood, M. T., Choi, W. J. & Choi, T. S. (2008). PCA-based method for 3D shape recovery

Mahmood, M. T., Khan, A. & Choi, T. S. (2009). Shape from focus based on bilateral filtering

Mahmood, M. T., Shim, S. O. & Choi, T. S. (2009). Shape from focus using principal component analysis in discrete wavelet transform, *Optical Engineering* 48(5): 057203. Malik, A. S. & Choi, T. S. (2007). Consideration of illumination effects and optimization of

Malik, A. S. & Choi, T. S. (2008). A novel algorithm for estimation of depth map using

Meneses, J., Suarez, M. A., Braga, J. & Gharbi, T. (2008). Extended depth of field using

Minhas, R., Mohammed, A. A. & Wu, Q. M. J. (2011). Shape from focus using fast discrete

Nayar, S. K. & Nakagawa, Y. (1994). Shape from focus, *IEEE Transactions on Pattern Analysis*

Niederöst, M., Niederöst, J. & Ўcuˇcka, J. (2003). Automatic 3D reconstruction and visualization

of microscopic objects from a monoscopic multifocus image sequence, *International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, Vol.

shapelet-based image analysis, *Applied Optics* 47(2): 169–178.

curvelet transform, *Pattern Recognition* 44(4): 839–853.

*and Machine Intelligence* 16(8): 824–831.

regression in eigenspace, *Image and Vision Computing* 28(4): 634–643.

image focus, *Proceedings of the IEEE International Conference on Image Processing*, Vol.

of microscopic objects from image focus using discrete cosine transform, *Microscopy*

and principal component analysis, *Applications of Soft Computing: From Theory to*

window size for accurate calculation of depth map for 3D shape recovery, *Pattern*

image focus for 3D shape recovery in the presence of noise, *Pattern Recognition*

Krotkov, E. (1987). Focusing, *International Journal of Computer Vision* 1(3): 223–237.

*Advances in Imaging and Electron Physics* 115: 129–196.

*Recognition and Artificial Intelligence* 24(5): 719–736.

pyramid, *Pattern Recognition Letters* 22(9): 929–939.

27(13): 1431–1439.

*Praxis* 58: 453–462.

41(7): 2200–2225.

XXXIV-5/W10.

*Recognition* 40(1): 154–170.

Bellingham, Washington, USA.

Press, Bellingham, Washington, USA.

1–5, San Diego, CA , USA, pp. 3216–3219.

*Research and Technique* 71(12): 897–907.

physical framework for the representation and processing of transmitted images,

three-dimensional shape recovery from image focus, *International Journal of Pattern*

measure of camera focus using a discrete cosine transform, *Pattern Recognition Letters*


**1. Introduction**

Malik (1990)).

The problem of image restoration has deserved considerable attention in resent years. For the visual analysis of images, clarity and visibility of details are important factors, but for advanced processing, a high signal-to-noise ratio (SNR) is essential, as further processing steps (such as segmentation and classification) are sensitive to noise. Though the years, different techniques have been studied to improve the SNR or a degraded image. Techniques based on post-processing have the advantage of not affecting the acquisition process (Gonzalez & Woods (2001); Jain (1989); Weickert (1995)). More recently, the work of Tschumperle (2006) has explored the more extensive use of curve-preserving PDE's for restoration of images. The calculation of mean intensities over neighboring pixels, equivalent to isotropic diffusion, considerably increases the SNR, but degrades the quality of image features (edges, lines and dots). This effect can be reduced with non-linear filters. The median filter has the characteristic of maintaining these features, but details are lost, degrading the image resolution. Perhaps the most popular technique introduced in the last couple of years is anisotropic diffusion, initially proposed by Perona and Malik (Black et al. (1998); Perona &

**Image Restoration via Topological Derivative** 

*Networking Research Center on Bioengineering, Biomaterials and Nanomedicine* 

*National Institute of Science and Technology in Medicine Assisted by* 

*1Center for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB),* 

Ignacio Larrabide1,2 and Raúl A. Feijóo2

*(CIBER-BBN), Universitat Pompeu Fabra, Barcelona 2National Laboratory for Scientific Computation (LNCC),* 

*Scientific Computing (INCT-MACC), Petrópolis, RJ* 

**5**

*1Spain 2Brazil* 

This problem has motivated interdisciplinary research and the use of techniques actually born in other areas, as is the case of the Topological Derivative (TD). The TD has been originally conceived for the study of topology optimization and property identification problems. Since 1994 different works proposed this new paradigm for the study of such problems. The pioneer works of Eschenauer in 1994 (Eschenauer et al. (1994)) and Schumacher in 1995 (Schumacher (1995)) introduced a way to obtain the optimal shape and topology using Topological Sensitivity Analysis. In summary, this new concept called asymptotical topological expansion is posed as follows. Let J (Ω) = F(*u*(Ω)) be an arbitrary cost function that measures the "quality" associated to a given topology characterized by the state function *u*(Ω), which is restricted to be the solution of a variational equation defined in Ω. Given a sufficiently small positive number , a positive function *f*() that goes to zero with , we call


## **Image Restoration via Topological Derivative**

Ignacio Larrabide1,2 and Raúl A. Feijóo2

*1Center for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB), Networking Research Center on Bioengineering, Biomaterials and Nanomedicine (CIBER-BBN), Universitat Pompeu Fabra, Barcelona 2National Laboratory for Scientific Computation (LNCC), National Institute of Science and Technology in Medicine Assisted by Scientific Computing (INCT-MACC), Petrópolis, RJ 1Spain 2Brazil* 

## **1. Introduction**

24 Will-be-set-by-IN-TECH

96 Image Restoration – Recent Advances and Applications

Willson, R. G. & Shafer, S. A. (1991). Dynamic lens compensation for active color imaging and

Yap, P. T. & Raveendran, P. (2004). Image focus measure based on chebyshev moments, *IEE*

Yun, J. & Choi, T. S. (1999). Accurate 3-D shape recovery using curved window focus measure,

Zhang, Y., Zhang, Y. & Wen, C. Y. (2000). A new focus measure method using moments, *Image*

Zhang, Z. & Blum, R. S. (1999). A categorization of multiscale-decomposition-based

Institute, Carnegie Mellon University, Pittsburgh, PA, USA.

pp. 910–914.

*and Vision Computing* 18(12): 959–965.

*Proceedings of the IEEE* 87(8): 1315–1326.

*Proceedings - Vision Image and Signal Processing* 151(2): 128–136.

constant magnification focusing, *Technical Report CMU-RI-TR-91-26*, The Robotics

*Proceedings of the IEEE International Conference on Image Processing*, Vol. 3, Kobe , Japan,

image fusion schemes with a performance study for a digital camera application,

The problem of image restoration has deserved considerable attention in resent years. For the visual analysis of images, clarity and visibility of details are important factors, but for advanced processing, a high signal-to-noise ratio (SNR) is essential, as further processing steps (such as segmentation and classification) are sensitive to noise. Though the years, different techniques have been studied to improve the SNR or a degraded image. Techniques based on post-processing have the advantage of not affecting the acquisition process (Gonzalez & Woods (2001); Jain (1989); Weickert (1995)). More recently, the work of Tschumperle (2006) has explored the more extensive use of curve-preserving PDE's for restoration of images. The calculation of mean intensities over neighboring pixels, equivalent to isotropic diffusion, considerably increases the SNR, but degrades the quality of image features (edges, lines and dots). This effect can be reduced with non-linear filters. The median filter has the characteristic of maintaining these features, but details are lost, degrading the image resolution. Perhaps the most popular technique introduced in the last couple of years is anisotropic diffusion, initially proposed by Perona and Malik (Black et al. (1998); Perona & Malik (1990)).

This problem has motivated interdisciplinary research and the use of techniques actually born in other areas, as is the case of the Topological Derivative (TD). The TD has been originally conceived for the study of topology optimization and property identification problems. Since 1994 different works proposed this new paradigm for the study of such problems. The pioneer works of Eschenauer in 1994 (Eschenauer et al. (1994)) and Schumacher in 1995 (Schumacher (1995)) introduced a way to obtain the optimal shape and topology using Topological Sensitivity Analysis. In summary, this new concept called asymptotical topological expansion is posed as follows. Let J (Ω) = F(*u*(Ω)) be an arbitrary cost function that measures the "quality" associated to a given topology characterized by the state function *u*(Ω), which is restricted to be the solution of a variational equation defined in Ω. Given a sufficiently small positive number , a positive function *f*() that goes to zero with , we call

properties. For instance, the TD can be used to determine the diffusion tensor, which in the following we recall as *K*. To formalize this, we consider the image characterized by its intensity *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω) defined in a limited open domain <sup>Ω</sup> <sup>∈</sup> **<sup>R</sup>**<sup>2</sup> (the extension to **<sup>R</sup>**<sup>3</sup> is straightforward). For each restored image, characterized by the intensity *<sup>u</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω), we

Image Restoration via Topological Derivative 99

Ω

measuring the "quality" of the restoration given by the solution of the following variational

Different methods exist to remove noise and enhance edges of a degraded image. We can distinguish two types: based on the solution of a stationary problem and based on the solution of an evolutionary problem. Both types of methods are based on the application of non-linear/anisotropic diffusion on the image. In both cases, the diffusion coefficient or tensor is computed as a function of the local image gradient. This coefficient takes small values for high gradients (edges) thus stoping diffusion, and higher values when the gradient is small (homogeneous regions) promoting a higher diffusion. Both methods have two parameters. The stationary method has a parameter determining which gradients will be considered as edges and which ones not, and a second one characterizing the intensity of the diffusion to be applied. In the case of evolutionary methods, the first parameter is in some way similar to the one used by the stationary method to determine the threshold for gradients that are considered edges, and the number of iterations. For both methods, the parameter determining the gradient threshold can be estimated. But this does not happen for the other parameters, which need to be set depending on the noise type and intensity. In both cases, the the selection

Topological sensitivity analysis allows to characterize the sensitivity of a problem when the domain Ω where the problem is defined is perturbed in some way. This perturbation can be

• change in topology: the domain Ω is perturbed introducing a hole of an arbitrary shape *ω�* at the point **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup>, and the TD provides the sensitivity of a cost function when *�* <sup>→</sup> <sup>0</sup>

• change in material properties: a perturbation in the material properties at point **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> of an arbitrary shape *ω�* is introduced and the TD provides the sensitivity of the cost function when *�* → 0 (see Eq. (1)). By "material properties" is meant the coefficients that define the

• change in the forces/sources acting on Ω: similar to the previous case, but perturbing the

In the following, and for the sake of simplicity, only the first case will be considered, namely the perturbation of the domain by the introduction of a hole. The extension to the other two

∇*u* · ∇*u d*Ω ,

(*<sup>u</sup>* <sup>−</sup> *<sup>u</sup>*0)*<sup>η</sup> <sup>d</sup>*<sup>Ω</sup> <sup>=</sup> <sup>0</sup> <sup>∀</sup>*<sup>η</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω).

<sup>J</sup> (*u*) =

 Ω

can associate the cost function

problem: Determine *<sup>u</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω) such that Ω

*k*∇*u* · ∇*η d*Ω +

of this parameter will determine the quality of the result.

variational problem associated to the problem.

**2. Topological Derivative**

(see Eq. (1)).

forces/sources.

cases is straightforward.

a:

<sup>Ω</sup>*�* the perturbed domain after the inclusion at **<sup>x</sup>** of a hole of infinitesimal size governed by *�*. Therefore, the asymptotical topological expansion

$$\mathcal{J}(\Omega\_{\varepsilon}) = \mathcal{J}(\Omega) + f(\varepsilon)D\_T(\hat{\mathbf{x}}) + \mathcal{O}(f(\varepsilon)) \tag{1}$$

provides an estimate of the cost function value in the perturbed domain for a sufficiently small *�*, where *DT* is known as the TD. The Topological Derivative can then be defined as (Novotny (2003))

*A scalar function, defined in* <sup>Ω</sup>*, indicating in each point <sup>x</sup>* <sup>∈</sup> <sup>Ω</sup> *the sensitivity of a cost function when a hole of infinitesimal size � is introduced at that point.*

#### **1.1 Motivation for the use of Topological Derivative in image restoration**

In 2005 appeared the first papers using the TD in image processing: restoration by Belaid et al. (2007) and Larrabide *et al.* (Larrabide, Novotny, Feijóo & Taroco (2005; 2006)), where the objective was to recover an image that suffered some kind of degradation; segmentation by Larrabide *et al.* (Larrabide, Feijóo, Novotny, Taroco & Masmoudi (2005); Larrabide, Feijóo, Taroco & Novotny (2006)) and Hintermüler (2005), in medical images where the interest is in the identification of different organs for posterior reconstruction; and image classification by Auroux et al. (2006). He & Osher (2006) established a relation between the TD and other techniques broadly used in image processing as is the case of level sets. A remarkable feature of the TD is that it allows to compute the variation of a cost function with respect to a parameter that changes non-smoothly (e.g., characteristic function of the domain, material properties or non-continuous change of the forces acting on the problem). This derivative can be used to identify, according to some criterion, the characteristic function of an optimal domain, the material properties and their distribution in a given domain, or even the forces acting on a given domain and how they are distributed. This problem appears frequently in the context of image processing. As examples we can mention identification of edges, identification of objects, object tracking, decomposition of texture and geometry and reconstruction from projections, where the use of the TD appears as a natural way to solve these problems.

Regarding image restoration, the stationary heat equation has been used for this purpose (Kornprobst et al. (1997)). In this approach, the diffusion coefficient is usually given by

$$c(|\nabla u|) = \frac{\phi'(|\nabla u|)}{|\nabla u|} \, .$$

The problem consists in determining the function *φ* that allowed to remove noise preserving edges in the image. One property that characterized restoration methods based on non-linear isotropic diffusion was removing noise along the edges in the image. This unwanted property of non-linear diffusion, but still isotropic, was partially reduced when a non-linear anisotropic diffusion tensor was introduced (Frangakis & Hegerl (2001)). In this case, a diffusivity tensor **c**(|∇*u*|) was constructed from two eigenvectors and eigenvalues of tensor *J* = ∇*u* ⊗ ∇*u*. Still, as diffusion across edges is not completely stopped, heuristic criterion must be introduced to avoid the loss of image details.

Stoping the diffusion in the direction orthogonal to the intensity iso-lines of *u* is somehow equivalent to introducing a crack in the same direction. In this way, the use of the TD in image restoration appears naturally as it allows to analyze abrupt variations in the material 2 Will-be-set-by-IN-TECH

<sup>Ω</sup>*�* the perturbed domain after the inclusion at **<sup>x</sup>** of a hole of infinitesimal size governed by *�*.

provides an estimate of the cost function value in the perturbed domain for a sufficiently small *�*, where *DT* is known as the TD. The Topological Derivative can then be defined as (Novotny

*A scalar function, defined in* <sup>Ω</sup>*, indicating in each point <sup>x</sup>* <sup>∈</sup> <sup>Ω</sup> *the sensitivity of a cost function*

In 2005 appeared the first papers using the TD in image processing: restoration by Belaid et al. (2007) and Larrabide *et al.* (Larrabide, Novotny, Feijóo & Taroco (2005; 2006)), where the objective was to recover an image that suffered some kind of degradation; segmentation by Larrabide *et al.* (Larrabide, Feijóo, Novotny, Taroco & Masmoudi (2005); Larrabide, Feijóo, Taroco & Novotny (2006)) and Hintermüler (2005), in medical images where the interest is in the identification of different organs for posterior reconstruction; and image classification by Auroux et al. (2006). He & Osher (2006) established a relation between the TD and other techniques broadly used in image processing as is the case of level sets. A remarkable feature of the TD is that it allows to compute the variation of a cost function with respect to a parameter that changes non-smoothly (e.g., characteristic function of the domain, material properties or non-continuous change of the forces acting on the problem). This derivative can be used to identify, according to some criterion, the characteristic function of an optimal domain, the material properties and their distribution in a given domain, or even the forces acting on a given domain and how they are distributed. This problem appears frequently in the context of image processing. As examples we can mention identification of edges, identification of objects, object tracking, decomposition of texture and geometry and reconstruction from projections, where the use of the TD appears as a natural way to solve

Regarding image restoration, the stationary heat equation has been used for this purpose (Kornprobst et al. (1997)). In this approach, the diffusion coefficient is usually given by

The problem consists in determining the function *φ* that allowed to remove noise preserving edges in the image. One property that characterized restoration methods based on non-linear isotropic diffusion was removing noise along the edges in the image. This unwanted property of non-linear diffusion, but still isotropic, was partially reduced when a non-linear anisotropic diffusion tensor was introduced (Frangakis & Hegerl (2001)). In this case, a diffusivity tensor **c**(|∇*u*|) was constructed from two eigenvectors and eigenvalues of tensor *J* = ∇*u* ⊗ ∇*u*. Still, as diffusion across edges is not completely stopped, heuristic criterion must be introduced to

Stoping the diffusion in the direction orthogonal to the intensity iso-lines of *u* is somehow equivalent to introducing a crack in the same direction. In this way, the use of the TD in image restoration appears naturally as it allows to analyze abrupt variations in the material

(|∇*u*|) |∇*u*<sup>|</sup> .

*<sup>c</sup>*(|∇*u*|) = *<sup>φ</sup>*�

<sup>J</sup> (Ω*�*) = <sup>J</sup> (Ω) + *<sup>f</sup>*(*�*)*DT*(**<sup>x</sup>**) + <sup>O</sup>(*f*(*�*)) (1)

Therefore, the asymptotical topological expansion

*when a hole of infinitesimal size � is introduced at that point.*

**1.1 Motivation for the use of Topological Derivative in image restoration**

(2003))

these problems.

avoid the loss of image details.

properties. For instance, the TD can be used to determine the diffusion tensor, which in the following we recall as *K*. To formalize this, we consider the image characterized by its intensity *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω) defined in a limited open domain <sup>Ω</sup> <sup>∈</sup> **<sup>R</sup>**<sup>2</sup> (the extension to **<sup>R</sup>**<sup>3</sup> is straightforward). For each restored image, characterized by the intensity *<sup>u</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω), we can associate the cost function

$$\mathcal{J}(u) = \int\_{\Omega} \nabla u \cdot \nabla u \, d\Omega\_{\nu}$$

measuring the "quality" of the restoration given by the solution of the following variational problem: Determine *<sup>u</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω) such that

$$\int\_{\Omega} \, \boldsymbol{k} \nabla \boldsymbol{u} \cdot \nabla \eta \, d\Omega + \int\_{\Omega} (\boldsymbol{u} - \boldsymbol{u}\_{0}) \eta \, d\Omega = 0 \quad \forall \eta \in H^{1}(\Omega) \,.$$

Different methods exist to remove noise and enhance edges of a degraded image. We can distinguish two types: based on the solution of a stationary problem and based on the solution of an evolutionary problem. Both types of methods are based on the application of non-linear/anisotropic diffusion on the image. In both cases, the diffusion coefficient or tensor is computed as a function of the local image gradient. This coefficient takes small values for high gradients (edges) thus stoping diffusion, and higher values when the gradient is small (homogeneous regions) promoting a higher diffusion. Both methods have two parameters. The stationary method has a parameter determining which gradients will be considered as edges and which ones not, and a second one characterizing the intensity of the diffusion to be applied. In the case of evolutionary methods, the first parameter is in some way similar to the one used by the stationary method to determine the threshold for gradients that are considered edges, and the number of iterations. For both methods, the parameter determining the gradient threshold can be estimated. But this does not happen for the other parameters, which need to be set depending on the noise type and intensity. In both cases, the the selection of this parameter will determine the quality of the result.

## **2. Topological Derivative**

Topological sensitivity analysis allows to characterize the sensitivity of a problem when the domain Ω where the problem is defined is perturbed in some way. This perturbation can be a:


In the following, and for the sake of simplicity, only the first case will be considered, namely the perturbation of the domain by the introduction of a hole. The extension to the other two cases is straightforward.

**Theorem 1.** *Let f*(*�*) *be a function chosen such that* <sup>0</sup> <sup>&</sup>lt; <sup>|</sup>*DT*(*<sup>x</sup>*)<sup>|</sup> <sup>&</sup>lt; <sup>∞</sup>*, then the limit when �* <sup>→</sup> <sup>0</sup>

Image Restoration via Topological Derivative 101

1 *f* �(*�*)

In the previous expression is implicit the domain transformation (deformation) *χτ* : **x***�* ∈

where **v** is the velocity field characterizing the shape change and Ω*τ*|*τ*=<sup>0</sup> = Ω*�*. The field **v** is

For further information on this type of transformation see the work of Haug et al. (1986) and Haug & Céa (1981), Pironneau (1984), Sokolowski & Zolésio (1992) and Zolésio (1981).

From this theorem is naturally deduced the *Topological-Shape Sensitivity Method*, which will be explored in the following. The shape change derivative of the cost function in relation to the

> J*τ*(*uτ*) **subject to :** *a<sup>τ</sup>* (*uτ*, *η*) = *lτ*(*η*) ∀ *η* ∈ V*τ*.

where *a*Ω*<sup>τ</sup>* (·, ·) is given by *aτ*(·, ·), *l*Ω*<sup>τ</sup>* (·) with *lτ*(·) and where the notation J*τ*(*uτ*) evidences

To compute the derivative to change in shape considering the state equation as a restriction, the Lagrangian method is used (i.e., relaxing the restriction by the introduction of a Lagrange

*d*J (Ω*τ*) *dτ*

 *τ*=0

**v**(**x**) = −**n** ∀**x** ∈ *∂ω�* e **v**(**x**) = 0 ∀**x** ∈ *∂*Ω*�* \ *∂ω�*. (8)

L*τ*(*v*, *η*) = J*τ*(*v*) + *aτ*(*v*, *η*) − *lτ*(*η*) ∀ *η* ∈ V*<sup>τ</sup>* e *v* ∈ U*τ*. (10)

= J*τ*(*uτ*) ∀ *η* ∈ V*τ*. (11)

<sup>∀</sup> *<sup>d</sup><sup>η</sup> dτ* ∀ *η* ∈ V*<sup>τ</sup>*

. (12)

∈ V*<sup>τ</sup>* (13)

=0, solution of the state equation

 *dη dτ* 

**x***<sup>τ</sup>* = **x***�* + *τ***v**(**x***�*) (7)

(6)

(9)

*DT*(*<sup>x</sup>*) = lim

**Proof 1.** *The proof of this theorem can be found in the work of Novotny et al. (2003).*

*<sup>d</sup><sup>τ</sup> is the classical shape sensitivity.*

the dependency of the cost function on *u<sup>τ</sup>* and on Ω*τ*.

multiplier). The Lagrangian of this problem is written as

We compute the derivative with respect to *τ* in Eq. (10), then

*<sup>d</sup><sup>τ</sup>* <sup>=</sup> *<sup>∂</sup>*L*<sup>τ</sup>*

*d*L*τ*(*v*, *η*)

 *<sup>∂</sup>*L*<sup>τ</sup> ∂η* ; *dη dτ* = *a<sup>τ</sup> v*, *dη dτ* − *l<sup>τ</sup>*

**Compute :** *<sup>d</sup>*

*dτ*

<sup>L</sup>*τ*(*uτ*, *<sup>η</sup>*) = <sup>J</sup>*τ*(*uτ*) + *<sup>a</sup>τ*(*uτ*, *<sup>η</sup>*) <sup>−</sup> *<sup>l</sup>τ*(*η*)

*∂τ* <sup>+</sup>

 *<sup>∂</sup>*L*<sup>τ</sup> ∂v* ; *dv dτ* + *<sup>∂</sup>*L*<sup>τ</sup> ∂η* ; *dη dτ* 

We then work term by term on Eq. (12). Starting by the third term on the right-hand side we

*�*→0

*appearing in* (4)*, can be written as*

Ω*�* → **x***<sup>τ</sup>* ∈ Ω*<sup>τ</sup>* defined as

parameter *τ* can be written as

We verify, for *v* = *uτ*, that

have

*where <sup>d</sup>*<sup>J</sup> (Ω*τ*)

characterized by

Fig. 1. Topological derivative - Change in topology

#### **2.1 Topological shape sensitivity analysis**

Let a problem be defined in Ω where the quality/performance is characterized by a cost function <sup>J</sup> (Ω) = <sup>F</sup>(Ω, *<sup>u</sup>*(Ω)) where <sup>Ω</sup> <sup>⊂</sup> **<sup>R</sup>***n*, *<sup>n</sup>* <sup>=</sup> 2, 3, is a regular domain of open boundary *∂*Ω with exterior normal **n**. With the notation (Ω, *u*(Ω)) we empathize that F depends on Ω explicitly and implicitly through *u*(Ω), solution of the variational equation (state equation), that can be written in the abstract form as: determine *u* ∈ U = U(Ω) such that

$$a(\mu, \eta) = l(\eta) \quad \forall \eta \in \mathcal{V},\tag{2}$$

where U characterizes a set (usually a linear manifold of V) of admissible functions defined in Ω e V = V(Ω) the vector space of admissible variations. Also, *a*(., .) : U×V �→ **R** is a symmetrical bilinear form and *l*(.) : V �→ **R** a linear form. These forms also satisfy the properties of continuity and coercivity to warrant existence and uniqueness of solution *u*.

Let *ω* be a open domain arbitrarily shaped and of regular boundary *∂ω* containing the origin. Given *�* > 0 sufficiently small it can be defined for any point **x** ∈ Ω the domain *ω�* given by *ω�* <sup>=</sup> **<sup>x</sup>** <sup>+</sup> *�ω*. In this way, the introduction of a hole *ω�* centered in **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> allows to characterize the perturbed domain Ω*�* (Fig. 1) given by

$$
\Omega\_{\mathfrak{C}} = \Omega \backslash \overline{\omega\_{\mathfrak{C}}}.\tag{3}
$$

From Eq. (1), *DT* in **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> can be defined as

$$D\_T(\hat{\mathbf{x}}) = \lim\_{\epsilon \to 0} \frac{\mathcal{J}(\Omega\_\epsilon) - \mathcal{J}(\Omega)}{f(\epsilon)} \tag{4}$$

where *f*(*�*) is a positive monotone decreasing function (*f*(*�*) → 0 with *�* → 0). Furthermore, J (Ω*�*) = F(Ω*�*, *u�*(Ω*�*)), being *u�* the solution of the same state equation now defined in the perturbed domain, namely in Ω*�*: Determine *u�* ∈ U*�* = U(Ω*�*) such that

$$a\_{\Omega\_{\mathfrak{c}}}(u\_{\mathfrak{c}\prime}\eta) = l\_{\Omega\_{\mathfrak{c}}}(\eta) \quad \forall \eta \in \mathcal{V}\_{\mathfrak{c}} = \mathcal{V}(\Omega\_{\mathfrak{c}}).\tag{5}$$

In the work of Novotny et al. (2003) a relation between the TD and classical shape sensitivity analysis (Haug et al. (1986); Murat & Simon (1976)) is established. This result permits to use tools developed in classical sensitivity analysis for the computation of the TD. This new approach can be stated in the form of theorem as:

4 Will-be-set-by-IN-TECH

Let a problem be defined in Ω where the quality/performance is characterized by a cost function <sup>J</sup> (Ω) = <sup>F</sup>(Ω, *<sup>u</sup>*(Ω)) where <sup>Ω</sup> <sup>⊂</sup> **<sup>R</sup>***n*, *<sup>n</sup>* <sup>=</sup> 2, 3, is a regular domain of open boundary *∂*Ω with exterior normal **n**. With the notation (Ω, *u*(Ω)) we empathize that F depends on Ω explicitly and implicitly through *u*(Ω), solution of the variational equation (state equation),

where U characterizes a set (usually a linear manifold of V) of admissible functions defined in Ω e V = V(Ω) the vector space of admissible variations. Also, *a*(., .) : U×V �→ **R** is a symmetrical bilinear form and *l*(.) : V �→ **R** a linear form. These forms also satisfy the properties of continuity and coercivity to warrant existence and uniqueness of solution *u*.

Let *ω* be a open domain arbitrarily shaped and of regular boundary *∂ω* containing the origin. Given *�* > 0 sufficiently small it can be defined for any point **x** ∈ Ω the domain *ω�* given by *ω�* <sup>=</sup> **<sup>x</sup>** <sup>+</sup> *�ω*. In this way, the introduction of a hole *ω�* centered in **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> allows to

*a*(*u*, *η*) = *l*(*η*) ∀*η* ∈ V, (2)

Ω*�* = Ω \ *ω�*. (3)

*<sup>f</sup>*(*�*) (4)

that can be written in the abstract form as: determine *u* ∈ U = U(Ω) such that

*DT*(**<sup>x</sup>**) = lim

perturbed domain, namely in Ω*�*: Determine *u�* ∈ U*�* = U(Ω*�*) such that

*�*→0

where *f*(*�*) is a positive monotone decreasing function (*f*(*�*) → 0 with *�* → 0). Furthermore, J (Ω*�*) = F(Ω*�*, *u�*(Ω*�*)), being *u�* the solution of the same state equation now defined in the

In the work of Novotny et al. (2003) a relation between the TD and classical shape sensitivity analysis (Haug et al. (1986); Murat & Simon (1976)) is established. This result permits to use tools developed in classical sensitivity analysis for the computation of the TD. This new

J (Ω*�*) − J (Ω)

*a*Ω*�* (*u�*, *η*) = *l*Ω*�* (*η*) ∀*η* ∈ V*�* = V(Ω*�*). (5)

Fig. 1. Topological derivative - Change in topology

characterize the perturbed domain Ω*�* (Fig. 1) given by

From Eq. (1), *DT* in **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> can be defined as

approach can be stated in the form of theorem as:

**2.1 Topological shape sensitivity analysis**

**Theorem 1.** *Let f*(*�*) *be a function chosen such that* <sup>0</sup> <sup>&</sup>lt; <sup>|</sup>*DT*(*<sup>x</sup>*)<sup>|</sup> <sup>&</sup>lt; <sup>∞</sup>*, then the limit when �* <sup>→</sup> <sup>0</sup> *appearing in* (4)*, can be written as*

$$D\_T(\hat{\mathfrak{x}}) = \lim\_{\varepsilon \to 0} \frac{1}{f'(\varepsilon)} \left. \frac{d\mathcal{J}(\Omega\_{\mathbb{T}})}{d\tau} \right|\_{\tau=0} \tag{6}$$

*where <sup>d</sup>*<sup>J</sup> (Ω*τ*) *<sup>d</sup><sup>τ</sup> is the classical shape sensitivity.*

**Proof 1.** *The proof of this theorem can be found in the work of Novotny et al. (2003).*

In the previous expression is implicit the domain transformation (deformation) *χτ* : **x***�* ∈ Ω*�* → **x***<sup>τ</sup>* ∈ Ω*<sup>τ</sup>* defined as

$$\mathbf{x}\_{\mathsf{T}} = \mathbf{x}\_{\mathsf{c}} + \tau \mathbf{v}(\mathbf{x}\_{\mathsf{c}}) \tag{7}$$

where **v** is the velocity field characterizing the shape change and Ω*τ*|*τ*=<sup>0</sup> = Ω*�*. The field **v** is characterized by

$$\mathbf{v}(\mathbf{x}) = -\mathbf{n} \,\,\forall \mathbf{x} \in \partial \omega\_{\varepsilon} \,\,\mathbf{e} \,\, \mathbf{v}(\mathbf{x}) = 0 \,\,\forall \mathbf{x} \in \partial \Omega\_{\varepsilon} \,\, \bigvee \omega\_{\varepsilon} \,\,. \tag{8}$$

For further information on this type of transformation see the work of Haug et al. (1986) and Haug & Céa (1981), Pironneau (1984), Sokolowski & Zolésio (1992) and Zolésio (1981).

From this theorem is naturally deduced the *Topological-Shape Sensitivity Method*, which will be explored in the following. The shape change derivative of the cost function in relation to the parameter *τ* can be written as

$$\begin{cases} \mathbf{Compute} \colon \frac{d}{d\tau} \mathcal{I}\_{\tau}(u\_{\tau}) \\ \mathbf{subject to} \colon a\_{\tau} \left( u\_{\tau}, \eta \right) = l\_{\tau}(\eta) \quad \forall \, \eta \in \mathcal{V}\_{\tau} . \end{cases} \tag{9}$$

where *a*Ω*<sup>τ</sup>* (·, ·) is given by *aτ*(·, ·), *l*Ω*<sup>τ</sup>* (·) with *lτ*(·) and where the notation J*τ*(*uτ*) evidences the dependency of the cost function on *u<sup>τ</sup>* and on Ω*τ*.

To compute the derivative to change in shape considering the state equation as a restriction, the Lagrangian method is used (i.e., relaxing the restriction by the introduction of a Lagrange multiplier). The Lagrangian of this problem is written as

$$\mathcal{L}\_{\tau}(v,\eta) = \mathcal{J}\_{\tau}(v) + a\_{\tau}(v,\eta) - l\_{\tau}(\eta) \,\,\,\forall \,\eta \in \mathcal{V}\_{\tau} \,\,\,\mathbf{e} \,\, v \in \mathcal{U}\_{\tau}. \tag{10}$$

We verify, for *v* = *uτ*, that

$$\mathcal{L}\_{\tau}(\boldsymbol{u}\_{\tau},\boldsymbol{\eta}) = \mathcal{J}\_{\tau}(\boldsymbol{u}\_{\tau}) + \underbrace{\boldsymbol{a}\_{\tau}(\boldsymbol{u}\_{\tau},\boldsymbol{\eta}) - l\_{\tau}(\boldsymbol{\eta})}\_{=\text{0, solution of the state equation}} \quad \forall \, \boldsymbol{\eta} \in \mathcal{V}\_{\tau}$$

$$= \mathcal{J}\_{\tau}(\boldsymbol{u}\_{\tau}) \qquad \forall \, \boldsymbol{\eta} \in \mathcal{V}\_{\tau}. \tag{11}$$

We compute the derivative with respect to *τ* in Eq. (10), then

$$\frac{d\mathcal{L}\_{\tau}(v,\eta)}{d\tau} = \frac{\partial \mathcal{L}\_{\tau}}{\partial \tau} + \left\langle \frac{\partial \mathcal{L}\_{\tau}}{\partial v}; \frac{dv}{d\tau} \right\rangle + \left\langle \frac{\partial \mathcal{L}\_{\tau}}{\partial \eta}; \frac{d\eta}{d\tau} \right\rangle. \tag{12}$$

We then work term by term on Eq. (12). Starting by the third term on the right-hand side we have

$$\left\langle \frac{\partial \mathcal{L}\_{\tau}}{\partial \eta}; \frac{d\eta}{d\tau} \right\rangle = a\_{\tau} \left( v, \frac{d\eta}{d\tau} \right) - l\_{\tau} \left( \frac{d\eta}{d\tau} \right) \; \forall \, \frac{d\eta}{d\tau} \in \mathcal{V}\_{\tau} \tag{13}$$

**<sup>x</sup>** is obtained. As mentioned, this expression will only depend on the values of *<sup>u</sup>* and *<sup>q</sup>* and its derivatives at point **<sup>x</sup>**. The former has consequences from the computational point of view and in fact, once *u* and *q* are obtained, **the computation of the TD is just a post processing**.

Image Restoration via Topological Derivative 103

Then, for a given cost function F(Ω, *u*(Ω)) the *Topological-Shape Sensitivity Method* can be

1. Compute the shape change derivative for the cost function F(Ω*�*, *u�*(Ω*�*)), using the

2. Identify the Eshelby momentum tensor **Σ***�* and write the sensitivity expression as an

This is a general and systematic way to compute the *DT* for an arbitrary cost function. The particular case presented in this chapter is applied to the introduction of a perturbation in the domain Ω in the shape of a crack, which is then presented as part of an algorithm for the

The concept of the TD, allows to quantify the sensitivity of a performance measure of cost function when the problem definition domain is perturbed. Therefore, if the cost function is associated to the noise present in the image, it will be possible to use its TD to develop appropriate image restoration algorithms. In this context, two state equations are studied: the first one based on a stationary problem and the second one on the evolutionary problem. The TD will allow to determine the location and orientation of the cracks that should be introduced in the domain to minimize the cost function. From the point of view of the state equation, the cracks will stop diffusion in the orthogonal direction, only allowing diffusion in the tangent direction. In other words, the TD provides a procedure to compute the diffusion anisotropic

This approach is based in introducing cracks in an image under the effect of diffusion. These cracks are introduced at specific locations in the image using the information provided by the TD. For a small *�*, let Ω*�* = Ω \ *γ�* be the domain perturbed by the insertion of a crack *γ�* <sup>=</sup> **<sup>x</sup>** <sup>+</sup> *�γ*, where **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> and *<sup>γ</sup>*(**m**) is a straight crack, being **<sup>m</sup>** the normal direction of *<sup>γ</sup>* (Fig. 2). Let *u�* be the solution of the same variational problem in the perturbed domain Ω*�* and J (*u�*) a cost function associated to it. Then, we obtain the following asymptotic topological

<sup>J</sup>*�*(*u�*) = <sup>J</sup> (*u*) + *<sup>f</sup>*(*�*)*DT*(**<sup>x</sup>**, **<sup>m</sup>**) + <sup>O</sup>(*f*(*�*)),

*DT*(**<sup>x</sup>**, **<sup>m</sup>**) = *<sup>M</sup>* **<sup>m</sup>** · **<sup>m</sup>**

3. Do the asymptotical analysis to study the behavior of *u�* and *q�* when *�* → 0.

tensorial field that will be used to restore the image preserving the image edges.

summarized in the following steps:

5. Compute the *DT* using Eq. (17).

restoration of degraded images.

integral defined on the boundary *∂ω�*.

4. From the asymptotical analysis, chose *f*(*�*).

**3. Topological Derivative in image restoration**

**3.1 Continuous approach -** *RDT* **-continuous**

expansion of J*�*(*u�*) when *�* → 0, i.e.,

where

Lagrangian method.

Then, for the particular case *v* = *uτ*, Eq. (13) is zero. Considering the second term of Eq. (12), we have

$$
\left\langle \frac{\partial \mathcal{L}\_{\tau}}{\partial v}; \frac{dv}{d\tau} \right\rangle = \left\langle \frac{\partial \mathcal{J}\_{\tau}}{\partial v}; \frac{dv}{d\tau} \right\rangle + a\_{\tau} \left( \eta\_{\prime} \frac{dv}{d\tau} \right) \,\,\,\forall \, \frac{dv}{d\tau} \in \mathcal{V}\_{\tau} \tag{14}
$$

where the symmetry of *aτ*(·, ·) was used. In this expression, *η* can be chosen arbitrarily. In articular, it will be selected *η* = *qτ*, being *q<sup>τ</sup>* ∈ V*<sup>τ</sup>* the solution to the adjoint equation given by

$$\left. \left\langle \frac{\partial \mathcal{J}\_{\mathsf{T}}}{\partial v}; \frac{dv}{d\tau} \right\rangle \right|\_{v=u\_{\mathsf{T}}} + a\_{\mathsf{T}} \left( q\_{\mathsf{T}\prime} \frac{dv}{d\tau} \right) = 0 \,\,\,\forall \, \frac{dv}{d\tau} \in \mathcal{V}\_{\mathsf{T}}.\tag{15}$$

The previous equation is known as "(variational) adjoint equation", and its solution *q<sup>τ</sup>* (or *q�* and *q* if the adjoint equation is defined in the domain Ω*�* and Ω respectively) as "adjoint solution". We note that, because of the properties of *a*(·, ·), the adjoint equation is of the same kind as the state equation (Eq. (2), or its counterpart in the perturbed domain Eq. (5)). From the computational point of view, the former means that the same computational system used to compute the solution of the state equation *u* (or *u�*) can be used to compute *q* (or *q�*).

The total derivative with respect to parameter *τ* of the Lagrangian is given by

$$\begin{split} \frac{d\mathcal{J}(\Omega\_{\mathsf{T}})}{d\tau} \Big|\_{\tau=0} &= \left. \frac{\partial \mathcal{L}\_{\mathsf{T}}(v,\eta)}{\partial \tau} \right|\_{\substack{v=u\_{\mathsf{T}}\\ \eta=q\_{\mathsf{T}}}} \Big|\_{\tau=0} \\ &= \left. \left[ \frac{\partial \mathcal{J}\_{\mathsf{T}}(v)}{\partial \tau} + \frac{\partial a\_{\mathsf{T}}(v,\eta)}{\partial \tau} - \frac{\partial l\_{\mathsf{T}}(\eta)}{\partial \tau} \right]\_{\substack{\mathcal{V}=u\_{\mathsf{T}}\\ \eta=q\_{\mathsf{T}}}} \right|\_{\substack{\mathcal{V}=u\_{\mathsf{T}}\\ \tau=0}} . \end{split} \tag{16}$$

We notice that *uτ*|*τ*=<sup>0</sup> = *u�* and *qτ*|*τ*=<sup>0</sup> = *q�*. Therefore, the former expression results in a function of *u�* and *q�* and its derivatives. As we noted before, only the boundary *∂ω�* is perturbed by a uniform expansion (Eq. (8)). Then, the derivative to shape change results in an integral only defined on the boundary *∂ω�*. Therefore, the topological derivative is given by an expression of the form

$$D\_T(\hat{\mathbf{x}}) = -\lim\_{\varepsilon \to 0} \frac{1}{f'(\varepsilon)} \int\_{\partial \omega\_{\varepsilon}} \mathbf{E}\_{\varepsilon} \mathbf{n} \cdot \mathbf{n} \, d\partial \omega\_{\varepsilon} \tag{17}$$

where **Σ***�* depends on *u�* and *q�*, and it can be interpreted as a generalization of the Eshelby momentum energy tensor proposed by Eshelby (1985). The tensor **Σ***�* must be identified for each particular problem, which depends on the cost function adopted and the state equation associated to *u�*.

Finally, its necessary to compute the limit when *�* → 0 to obtain the *DT* expression. For this, it is necessary to study the behavior of solutions *u�* and *q�* when *�* → 0. This behavior can be obtained with asymptotical analysis on the neighborhood of the hole. At this point, different alternatives can be used depending on the problem under consideration (boundary conditions, type of perturbation, etc.). In all these cases, asymptotical analysis allows to express *<sup>u</sup>�* and *<sup>q</sup>�* as a function of *�*, *<sup>u</sup>*(**x**), *<sup>q</sup>*(**x**) and there derivatives in **x**, respectively. Namely, as a function of the solutions of the state and adjoint equation defined in the domain **without** perturbation providing as well the function *f*(*�*). In this way, substituting these equation in Eq. (17) and from the computation of the limit for *�* → 0 the final expression of the *DT* at point 6 Will-be-set-by-IN-TECH

Then, for the particular case *v* = *uτ*, Eq. (13) is zero. Considering the second term of Eq. (12),

where the symmetry of *aτ*(·, ·) was used. In this expression, *η* can be chosen arbitrarily. In articular, it will be selected *η* = *qτ*, being *q<sup>τ</sup>* ∈ V*<sup>τ</sup>* the solution to the adjoint equation given

The previous equation is known as "(variational) adjoint equation", and its solution *q<sup>τ</sup>* (or *q�* and *q* if the adjoint equation is defined in the domain Ω*�* and Ω respectively) as "adjoint solution". We note that, because of the properties of *a*(·, ·), the adjoint equation is of the same kind as the state equation (Eq. (2), or its counterpart in the perturbed domain Eq. (5)). From the computational point of view, the former means that the same computational system used to compute the solution of the state equation *u* (or *u�*) can be used to compute *q* (or *q�*).

> *v*=*u<sup>τ</sup> <sup>η</sup>*=*q<sup>τ</sup>*

We notice that *uτ*|*τ*=<sup>0</sup> = *u�* and *qτ*|*τ*=<sup>0</sup> = *q�*. Therefore, the former expression results in a function of *u�* and *q�* and its derivatives. As we noted before, only the boundary *∂ω�* is perturbed by a uniform expansion (Eq. (8)). Then, the derivative to shape change results in an integral only defined on the boundary *∂ω�*. Therefore, the topological derivative is given

> 1 *f* �(*�*)

where **Σ***�* depends on *u�* and *q�*, and it can be interpreted as a generalization of the Eshelby momentum energy tensor proposed by Eshelby (1985). The tensor **Σ***�* must be identified for each particular problem, which depends on the cost function adopted and the state equation

Finally, its necessary to compute the limit when *�* → 0 to obtain the *DT* expression. For this, it is necessary to study the behavior of solutions *u�* and *q�* when *�* → 0. This behavior can be obtained with asymptotical analysis on the neighborhood of the hole. At this point, different alternatives can be used depending on the problem under consideration (boundary conditions, type of perturbation, etc.). In all these cases, asymptotical analysis allows to express *<sup>u</sup>�* and *<sup>q</sup>�* as a function of *�*, *<sup>u</sup>*(**x**), *<sup>q</sup>*(**x**) and there derivatives in **x**, respectively. Namely, as a function of the solutions of the state and adjoint equation defined in the domain **without** perturbation providing as well the function *f*(*�*). In this way, substituting these equation in Eq. (17) and from the computation of the limit for *�* → 0 the final expression of the *DT* at point

 *∂ω�*

 *τ*=0

*∂aτ*(*v*, *η*)

*∂τ* <sup>−</sup> *<sup>∂</sup>lτ*(*η*) *∂τ*

*<sup>v</sup>*=*u<sup>τ</sup> <sup>η</sup>*=*q<sup>τ</sup>*

 *τ*=0

**Σ***�***n** · **n** *d∂ω�* (17)

+ *a<sup>τ</sup> qτ*, *dv dτ* 

The total derivative with respect to parameter *τ* of the Lagrangian is given by

<sup>=</sup> *<sup>∂</sup>*L*τ*(*v*, *<sup>η</sup>*) *∂τ*

= *∂*J*τ*(*v*) *∂τ* <sup>+</sup>

*DT*(**x**) = <sup>−</sup>lim

*�*→0

<sup>∀</sup> *dv dτ*

<sup>=</sup> <sup>0</sup> <sup>∀</sup> *dv dτ* ∈ V*<sup>τ</sup>* (14)

∈ V*τ*. (15)

. (16)

 *<sup>∂</sup>*J*<sup>τ</sup> ∂v* ; *dv dτ* + *a<sup>τ</sup> η*, *dv dτ* 

we have

by

 *<sup>∂</sup>*L*<sup>τ</sup> ∂v* ; *dv dτ* =

> *<sup>∂</sup>*J*<sup>τ</sup> ∂v* ; *dv dτ v*=*u<sup>τ</sup>*

*d*J (Ω*τ*) *dτ*

by an expression of the form

associated to *u�*.

 *τ*=0 **<sup>x</sup>** is obtained. As mentioned, this expression will only depend on the values of *<sup>u</sup>* and *<sup>q</sup>* and its derivatives at point **<sup>x</sup>**. The former has consequences from the computational point of view and in fact, once *u* and *q* are obtained, **the computation of the TD is just a post processing**.

Then, for a given cost function F(Ω, *u*(Ω)) the *Topological-Shape Sensitivity Method* can be summarized in the following steps:


This is a general and systematic way to compute the *DT* for an arbitrary cost function. The particular case presented in this chapter is applied to the introduction of a perturbation in the domain Ω in the shape of a crack, which is then presented as part of an algorithm for the restoration of degraded images.

### **3. Topological Derivative in image restoration**

The concept of the TD, allows to quantify the sensitivity of a performance measure of cost function when the problem definition domain is perturbed. Therefore, if the cost function is associated to the noise present in the image, it will be possible to use its TD to develop appropriate image restoration algorithms. In this context, two state equations are studied: the first one based on a stationary problem and the second one on the evolutionary problem. The TD will allow to determine the location and orientation of the cracks that should be introduced in the domain to minimize the cost function. From the point of view of the state equation, the cracks will stop diffusion in the orthogonal direction, only allowing diffusion in the tangent direction. In other words, the TD provides a procedure to compute the diffusion anisotropic tensorial field that will be used to restore the image preserving the image edges.

## **3.1 Continuous approach -** *RDT* **-continuous**

This approach is based in introducing cracks in an image under the effect of diffusion. These cracks are introduced at specific locations in the image using the information provided by the TD. For a small *�*, let Ω*�* = Ω \ *γ�* be the domain perturbed by the insertion of a crack *γ�* <sup>=</sup> **<sup>x</sup>** <sup>+</sup> *�γ*, where **<sup>x</sup>** <sup>∈</sup> <sup>Ω</sup> and *<sup>γ</sup>*(**m**) is a straight crack, being **<sup>m</sup>** the normal direction of *<sup>γ</sup>* (Fig. 2). Let *u�* be the solution of the same variational problem in the perturbed domain Ω*�* and J (*u�*) a cost function associated to it. Then, we obtain the following asymptotic topological expansion of J*�*(*u�*) when *�* → 0, i.e.,

$$\mathcal{J}\_{\boldsymbol{\varepsilon}}(\boldsymbol{\mu}\_{\boldsymbol{\varepsilon}}) = \mathcal{J}(\boldsymbol{\mu}) + f(\boldsymbol{\varepsilon})D\_{T}(\widehat{\mathbf{x}}, \mathbf{m}) + \mathcal{O}(f(\boldsymbol{\varepsilon}))\_{\boldsymbol{\varepsilon}}$$

where

*DT*(**<sup>x</sup>**, **<sup>m</sup>**) = *<sup>M</sup>* **<sup>m</sup>** · **<sup>m</sup>**

Fig. 2. Topological Derivative concept for cracks.

being *M* a symmetric tensor given by

$$M = -\left(\nabla\mu \otimes \nabla\mu + k(\nabla\mu \otimes\_s \nabla q)\right). \tag{18}$$

Fig. 3. Image restoration with the continuous TD algorithm *RDT* -Continuous (Iso) of the Lena image with *k*<sup>0</sup> = 2, upper row: *α* = 0.10, 0.20, lower row: *α* = 0.30 and TD value for each

Image Restoration via Topological Derivative 105

pixel.

**–** *K*(**x**) = *k*<sup>0</sup> *I* otherwise.

and *q* the solution of the adjoint equation

$$\int\_{\Omega} (k \nabla \eta \cdot \nabla \eta + q \eta) \, d\Omega = -\int\_{\Omega} \nabla u \cdot \nabla \eta \, d\Omega \, \, \forall \, \eta \in \mathcal{V}.$$

For any point **<sup>x</sup>**, *DT*(**<sup>x</sup>**, **<sup>m</sup>**) reaches its minimum when **<sup>m</sup>** is an eigenvector associated to the smallest eigenvalue *κmin* of *M*. Then, it is considered as the optimal direction of the crack *γ�*(**m**) the eigenvector corresponding to the eigenvalue *κmin*. This value will be adopted as the TD associated to the creation of a crack at the point **<sup>x</sup>**. In Fig. 3 is presented an example for the Lena image (SNR = 26dB).

As mentioned, for any **<sup>x</sup>** , *DT*(**<sup>x</sup>**, **<sup>m</sup>**) takes the minimal value when **<sup>m</sup>** is the eigenvector associated to the smallest eigenvalue *κmin* of *M*. Then, by considering the orientation of the crack *γ�*(**m**) the eigenvector corresponding to the eigenvalue *κmin*. This minimal value will be adopted as the TD associated to the creation of a crack at the point **<sup>x</sup>**. The algorithm proposed here consists in computing the TD and introducing small cracks in the locations where the derivative is smaller than a given value *DTLim*. Two algorithms are presented: isotropic and anisotropic.

To solve the numerical problem, the introduction of a small diffusion coefficient (or a conductivity tensor that acts on one direction) is interpreted as the presence of a crack. In the proposed algorithms, the tensor *K*(**x**) for the isotropic and anisotropic case are computed as a function of the TD, namely *K* = *K*(**x**, *DT*):

	- **–** *K*(**x**) = *k<sup>ε</sup> I* if *DT*(**x**) ≤ *DTLim*;
	- **–** *K*(**x**) = *k*<sup>0</sup> *I* otherwise.
	- **–** *K*(**x**) = *kε*(**n** ⊗ **n**) + *k*0(**t** ⊗ **t**) if *DT*(**x**) ≤ *DTLim*;

8 Will-be-set-by-IN-TECH

∇*u* ⊗ ∇*u* + *k*(∇*u* ⊗*<sup>s</sup>* ∇*q*)

 Ω

For any point **<sup>x</sup>**, *DT*(**<sup>x</sup>**, **<sup>m</sup>**) reaches its minimum when **<sup>m</sup>** is an eigenvector associated to the smallest eigenvalue *κmin* of *M*. Then, it is considered as the optimal direction of the crack *γ�*(**m**) the eigenvector corresponding to the eigenvalue *κmin*. This value will be adopted as the TD associated to the creation of a crack at the point **<sup>x</sup>**. In Fig. 3 is presented an example

As mentioned, for any **<sup>x</sup>** , *DT*(**<sup>x</sup>**, **<sup>m</sup>**) takes the minimal value when **<sup>m</sup>** is the eigenvector associated to the smallest eigenvalue *κmin* of *M*. Then, by considering the orientation of the crack *γ�*(**m**) the eigenvector corresponding to the eigenvalue *κmin*. This minimal value will be adopted as the TD associated to the creation of a crack at the point **<sup>x</sup>**. The algorithm proposed here consists in computing the TD and introducing small cracks in the locations where the derivative is smaller than a given value *DTLim*. Two algorithms are presented: isotropic and

To solve the numerical problem, the introduction of a small diffusion coefficient (or a conductivity tensor that acts on one direction) is interpreted as the presence of a crack. In the proposed algorithms, the tensor *K*(**x**) for the isotropic and anisotropic case are computed

∇*u* · ∇*η d*Ω ∀ *η* ∈ V.

. (18)

Fig. 2. Topological Derivative concept for cracks.

*M* = − 

(*k*∇*q* · ∇*η* + *qη*) *d*Ω = −

being *M* a symmetric tensor given by

and *q* the solution of the adjoint equation Ω

for the Lena image (SNR = 26dB).

as a function of the TD, namely *K* = *K*(**x**, *DT*):

**–** *K*(**x**) = *k<sup>ε</sup> I* if *DT*(**x**) ≤ *DTLim*;

**–** *K*(**x**) = *k*<sup>0</sup> *I* otherwise.

• Isotropic diffusion based on the TD (*RDT* -Continuous (Iso)):

**–** *K*(**x**) = *kε*(**n** ⊗ **n**) + *k*0(**t** ⊗ **t**) if *DT*(**x**) ≤ *DTLim*;

• Anisotropic diffusion based on the TD (*RDT* -Continuous (Aniso)):

anisotropic.

Fig. 3. Image restoration with the continuous TD algorithm *RDT* -Continuous (Iso) of the Lena image with *k*<sup>0</sup> = 2, upper row: *α* = 0.10, 0.20, lower row: *α* = 0.30 and TD value for each pixel.

**–** *K*(**x**) = *k*<sup>0</sup> *I* otherwise.

(a) Lena image. (b) Lena image with low contrast.

Image Restoration via Topological Derivative 107

(c) Topological derivative for the previous images.

approximate solutions *u<sup>h</sup>* of *u*, *q<sup>h</sup>* of *q* and *u*∗*<sup>h</sup>* of *u*<sup>∗</sup> can be easily obtained for any image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω). Using these solution, an approximation by finite elements *<sup>M</sup><sup>h</sup>* of the tensor *<sup>M</sup>* is

To find the restored image, three boundary value problems need to be computed. These correspond to the scalar fields of *u<sup>h</sup>* the isotropic problem, *q<sup>h</sup>* the adjoint problem and *u*∗*<sup>h</sup>* the problem with the diffusivity tensor *K*(**x**) computed using *RDT* -Continuous (Iso or Aniso).

<sup>∇</sup>*u<sup>h</sup>* ⊗ ∇*u<sup>h</sup>* <sup>+</sup> *<sup>k</sup>*(∇*u<sup>h</sup>* <sup>⊗</sup>*<sup>s</sup>* <sup>∇</sup>*qh*)

. (20)

Fig. 5. Lena image with different contrast. The second row corresponds to the TD for each

image using the same color scale in both cases.

*<sup>M</sup><sup>h</sup>* <sup>=</sup> <sup>−</sup>

given by

Fig. 4. Image restoration with the continuous TD algorithm *RDT* -Continuous (Aniso) for the Lena image with *k*<sup>0</sup> = 2, upper row: *α* = 0.10, 0.20, lower row: *α* = 0.30 and TD value for each pixel.

for *k�* � 1, *k*<sup>0</sup> a positive real number, **n** and **t** the normal ad tangent directions to the crack, respectively.

With this diffusion tensor the restored image is obtained by solving the following variational problem: Determine *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω) such that

$$\int\_{\Omega} \ K \nabla \boldsymbol{u}^\* \cdot \nabla \eta \, d\Omega + \int\_{\Omega} (\boldsymbol{u}^\* - \boldsymbol{u}\_0) \eta \, d\Omega = \boldsymbol{0} \,\,\,\forall \eta \in H^1(\Omega) \,. \tag{19}$$

As the solution *u*∗ of the variational problem given by Eq. (19) cannot be explicitly known in general, its necessary to compute an approximate solution. The Finite Element Method is used for this purpose Hughes (2000). Then, using the simplest finite element given by quadrilateral bilinear element (for the 2 dimensional case) or a trilinear parallelepiped (for the 3 dimensional case) whose nodal points coincide with the centers of the image elements, 10 Will-be-set-by-IN-TECH

Fig. 4. Image restoration with the continuous TD algorithm *RDT* -Continuous (Aniso) for the Lena image with *k*<sup>0</sup> = 2, upper row: *α* = 0.10, 0.20, lower row: *α* = 0.30 and TD value for

for *k�* � 1, *k*<sup>0</sup> a positive real number, **n** and **t** the normal ad tangent directions to the crack,

With this diffusion tensor the restored image is obtained by solving the following variational

As the solution *u*∗ of the variational problem given by Eq. (19) cannot be explicitly known in general, its necessary to compute an approximate solution. The Finite Element Method is used for this purpose Hughes (2000). Then, using the simplest finite element given by quadrilateral bilinear element (for the 2 dimensional case) or a trilinear parallelepiped (for the 3 dimensional case) whose nodal points coincide with the centers of the image elements,

(*u*<sup>∗</sup> <sup>−</sup> *<sup>u</sup>*0)*<sup>η</sup> <sup>d</sup>*<sup>Ω</sup> <sup>=</sup> <sup>0</sup> <sup>∀</sup>*<sup>η</sup>* <sup>∈</sup> *<sup>H</sup>*1(Ω). (19)

 Ω

each pixel.

respectively.

problem: Determine *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω) such that

*K*∇*u*<sup>∗</sup> · ∇*η d*Ω +

 Ω

(a) Lena image. (b) Lena image with low contrast.

(c) Topological derivative for the previous images.

Fig. 5. Lena image with different contrast. The second row corresponds to the TD for each image using the same color scale in both cases.

approximate solutions *u<sup>h</sup>* of *u*, *q<sup>h</sup>* of *q* and *u*∗*<sup>h</sup>* of *u*<sup>∗</sup> can be easily obtained for any image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω). Using these solution, an approximation by finite elements *<sup>M</sup><sup>h</sup>* of the tensor *<sup>M</sup>* is given by

$$M^{\hbar} = -\left(\nabla \boldsymbol{\mu}^{\hbar} \otimes \nabla \boldsymbol{\mu}^{\hbar} + k(\nabla \boldsymbol{\mu}^{\hbar} \otimes\_{\mathrm{s}} \nabla q^{\hbar})\right) . \tag{20}$$

To find the restored image, three boundary value problems need to be computed. These correspond to the scalar fields of *u<sup>h</sup>* the isotropic problem, *q<sup>h</sup>* the adjoint problem and *u*∗*<sup>h</sup>* the problem with the diffusivity tensor *K*(**x**) computed using *RDT* -Continuous (Iso or Aniso).

(a) Detail of degraded Lena image. (b) Detail of the TD.

Image Restoration via Topological Derivative 109

Fig. 6. Detail of Lena's image topological derivative, crack normal (black) and tangent

0.30) where used. Again, the SNR improves going from 26.92 in the degraded image to 28.47, 28.85 and 29.05 in the processed images, respectively. This time, and even if the SNR of the isotropic case are not reached, the noise along edges is more efficiently removed. Finally, in Fig. 6 is presented a detail of the TD, the vectors normal (in black) and tangent (in white) to

The discrete approach relies on an auxiliary transient heat equation. In this case, cracks are introduced in the image to stop the diffusion in given points and directions. In this way, the

information provided by the TD is used to determine the location of these cracks.

(d) Detail of the restored image.

(c) Detail of the crack normal and tangent

(white) directions and final result.

the cracks and the restored image.

**3.3 Discrete approach -** *RDT* **-discrete**

directions.

**Algorithm 1** Image restoration based in the continuous topological derivative - *RDT* -Continuous

**Require:** Degraded image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω), parameters *DTLim* <sup>e</sup> *<sup>k</sup>*0. **Ensure:** Restored image *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω). compute *u* and *q*, solutions of the state and adjoint equation, respectively,

compute the matrix 2 × 2 *M* and its minimal eigenvalue *κmin* for each point in Ω,

find *K* using *RDT* -Continuous (Iso or Aniso),

compute *u*∗, a restored image, using *K*(**x**) previously obtained.

This algorithm uses one parameter (*DTLim*) to select the elements in the image that will have their coefficients with a modified diffusivity. Then, depending on the TD value of the image being processed, this coefficient will be modified in a different number of points (e.g., two similar images with different contrast will produce a different TD, as it depends on ∇*u*, see Fig. 5). It is easy to verify that the parameter *DTLim* must be a value in the interval [*DT MIN*, 0) (being *DT MIN* the minimum value of the *DT*). *DT MIN* and the distribution of values of the TD in this interval can vary considerably for different images. This will require readjusting this parameter for different images making its estimation more complex.

A different alternative, presented in Larrabide, Feijóo, Novotny & Taroco (2008), is to modify the diffusivity coefficient using a fixed point algorithm. This consists in sorting in growing order the values of the TD and selecting a percentage *α* of the most negative values and use this value as threshold for the insertion of cracks in the image. We then define the set M*<sup>α</sup>* as

$$\begin{aligned} \mathcal{M}\_{\mathfrak{A}} &:= \{ D\_{\varGamma}(s) : D\_{\varGamma}(s) < 0 \\ &\text{and } D\_{\varGamma}(s) \text{ is in the } u\text{\textquotedblleft most negative values of the } D\_{\varGamma} \}. \end{aligned} \tag{21}$$

For *s* between one and the number of elements in the image. This alternative provides a better control on the algorithm (Algorithm 2).

**Algorithm 2** Image restoration based on the continuous TD II **Require:** Degraded image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω), parameters *<sup>α</sup>* <sup>e</sup> *<sup>k</sup>*0. **Ensure:** Restored image *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω). compute *u* and *q*, solutions of the state and adjoint equation, respectively, compute the matrix 2 × 2 *M* and its minimal eigenvalue *κmin* for each point in Ω, find *K* using *RDT* -Continuous (Iso or Aniso) and Eq. (21), compute *u*∗, the restored image, using *K*(**x**) previously obtained.

### **3.2 Continuous approach results**

It can be observed in Fig. 3 the result obtained with Algorithm 2 for *DT*-Iso using the tensor *K* for the Lena image degraded wit white Gaussian noise (degraded image is presented in Fig. 3). For all the experiments presented *k*<sup>0</sup> = 2 was used. Visually, we observe that the noise is removed in the 3 cases. The results obtained in the three experiments, namely *α* = 0.10, 0.20 and 0.30, present considerable improvement in the SNR, going from 26.92 in the degraded image to 29.57, 30.36 and 30.88 in the processed images, respectively. By analyzing in detail these images, we observe that as other non-linear isotropic methods, it has difficulties to remove noise along edges. In Fig. 4 are presented the results corresponding to *K* computed using *DT*-Aniso and the same Lena image. The same values of *α* (namely *α* = 0.10, 0.20 and 12 Will-be-set-by-IN-TECH

**Algorithm 1** Image restoration based in the continuous topological derivative -

This algorithm uses one parameter (*DTLim*) to select the elements in the image that will have their coefficients with a modified diffusivity. Then, depending on the TD value of the image being processed, this coefficient will be modified in a different number of points (e.g., two similar images with different contrast will produce a different TD, as it depends on ∇*u*, see Fig. 5). It is easy to verify that the parameter *DTLim* must be a value in the interval [*DT MIN*, 0) (being *DT MIN* the minimum value of the *DT*). *DT MIN* and the distribution of values of the TD in this interval can vary considerably for different images. This will require readjusting this

A different alternative, presented in Larrabide, Feijóo, Novotny & Taroco (2008), is to modify the diffusivity coefficient using a fixed point algorithm. This consists in sorting in growing order the values of the TD and selecting a percentage *α* of the most negative values and use this value as threshold for the insertion of cracks in the image. We then define the set M*<sup>α</sup>* as

For *s* between one and the number of elements in the image. This alternative provides a better

and *DT*(*s*) is in the *α*% most negative values of the *DT*}. (21)

**Require:** Degraded image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω), parameters *DTLim* <sup>e</sup> *<sup>k</sup>*0.

compute *u*∗, a restored image, using *K*(**x**) previously obtained.

parameter for different images making its estimation more complex.

M*<sup>α</sup>* := {*DT*(*s*) : *DT*(*s*) < 0

**Algorithm 2** Image restoration based on the continuous TD II

find *K* using *RDT* -Continuous (Iso or Aniso) and Eq. (21), compute *u*∗, the restored image, using *K*(**x**) previously obtained.

compute *u* and *q*, solutions of the state and adjoint equation, respectively,

compute the matrix 2 × 2 *M* and its minimal eigenvalue *κmin* for each point in Ω,

It can be observed in Fig. 3 the result obtained with Algorithm 2 for *DT*-Iso using the tensor *K* for the Lena image degraded wit white Gaussian noise (degraded image is presented in Fig. 3). For all the experiments presented *k*<sup>0</sup> = 2 was used. Visually, we observe that the noise is removed in the 3 cases. The results obtained in the three experiments, namely *α* = 0.10, 0.20 and 0.30, present considerable improvement in the SNR, going from 26.92 in the degraded image to 29.57, 30.36 and 30.88 in the processed images, respectively. By analyzing in detail these images, we observe that as other non-linear isotropic methods, it has difficulties to remove noise along edges. In Fig. 4 are presented the results corresponding to *K* computed using *DT*-Aniso and the same Lena image. The same values of *α* (namely *α* = 0.10, 0.20 and

**Require:** Degraded image *<sup>u</sup>*<sup>0</sup> <sup>∈</sup> *<sup>L</sup>*2(Ω), parameters *<sup>α</sup>* <sup>e</sup> *<sup>k</sup>*0.

control on the algorithm (Algorithm 2).

**Ensure:** Restored image *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω).

**3.2 Continuous approach results**

compute *u* and *q*, solutions of the state and adjoint equation, respectively,

compute the matrix 2 × 2 *M* and its minimal eigenvalue *κmin* for each point in Ω,

*RDT* -Continuous

**Ensure:** Restored image *<sup>u</sup>*<sup>∗</sup> <sup>∈</sup> *<sup>H</sup>*1(Ω).

find *K* using *RDT* -Continuous (Iso or Aniso),

(a) Detail of degraded Lena image. (b) Detail of the TD.

(c) Detail of the crack normal and tangent directions.

(d) Detail of the restored image.

Fig. 6. Detail of Lena's image topological derivative, crack normal (black) and tangent (white) directions and final result.

0.30) where used. Again, the SNR improves going from 26.92 in the degraded image to 28.47, 28.85 and 29.05 in the processed images, respectively. This time, and even if the SNR of the isotropic case are not reached, the noise along edges is more efficiently removed. Finally, in Fig. 6 is presented a detail of the TD, the vectors normal (in black) and tangent (in white) to the cracks and the restored image.

## **3.3 Discrete approach -** *RDT* **-discrete**

The discrete approach relies on an auxiliary transient heat equation. In this case, cracks are introduced in the image to stop the diffusion in given points and directions. In this way, the information provided by the TD is used to determine the location of these cracks.

**Algorithm 3** Image restoration based on a discrete version of the TD - *RDT* -Discrete

Image Restoration via Topological Derivative 111

*iso*, *s* = 1..*M* × *N*

*�* ∈ C*σ*}

*�*<sup>∗</sup> the diffusivity coefficient associated to *DT*(*s*)

To compute the value of *DT* for a determined pixel, its necessary to introduce a perturbation.

*<sup>t</sup>* · <sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

*<sup>t</sup>* + ∑ *p*∈*ns k s*,*p �* <sup>Δ</sup> *�us*,*<sup>p</sup>*

*�*) computed using Eq. (24) and <sup>Δ</sup> *�us*,*<sup>p</sup>*

*<sup>k</sup>s*,*p*<sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

*<sup>t</sup>* · <sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

*<sup>t</sup>* − ∑ *p*∈*ns*

*�*). Using this information is proposed the following discrete image restoration

*<sup>k</sup>s*,*p*<sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

For Eqs. (25) and (26) the total variation of the cost function <sup>J</sup> *<sup>d</sup>* due to the perturbation **<sup>k</sup>***<sup>s</sup>*

*<sup>t</sup>* · <sup>Δ</sup> *�us*,*<sup>p</sup>*

*iso* = {*k*0, *k*0, *k*0, *k*0}.

*�* ∈ C*σ*. Then, the cost function

*<sup>t</sup>* · <sup>Δ</sup> *�us*,*<sup>p</sup> <sup>t</sup>* ,

*<sup>t</sup>* <sup>=</sup> *<sup>u</sup><sup>p</sup>*

*<sup>t</sup>* . (27)

*�* that minimizes the value

(26)

*�* is

*<sup>t</sup>* <sup>−</sup>*� <sup>u</sup><sup>s</sup> t*,

**Require:** The 2D image *u*<sup>0</sup> ∈ U, a diffusivity coefficient *k*<sup>0</sup> and a parameter *α*.

*�*) following Eq. (27)

*�*), **<sup>k</sup>***<sup>s</sup>*

*�*<sup>∗</sup> {*DT*(*s*, **<sup>k</sup>***<sup>s</sup>*

*<sup>t</sup>*−1)<sup>|</sup> <sup>&</sup>gt; *tol* **then**

*<sup>t</sup>*,*s* = 1, ··· , *M* × *N*, stop = *TRUE*

The last case corresponds to isotropic diffusion and is defined as **k***<sup>s</sup>*

*<sup>t</sup>*) − ∑ *p*∈*ns*

*t*(**k***<sup>s</sup>*

As in the continuous case, it will be considered a perturbation **k***<sup>s</sup>*

This is done by changing, for one pixel *s*, the set **k***<sup>s</sup>* for **k***<sup>s</sup>*

*<sup>t</sup>* = *<sup>u</sup><sup>s</sup>*

*�*) = ∑ *p*∈*ns k s*,*p �* <sup>Δ</sup> *�us*,*<sup>p</sup>*

*<sup>t</sup>*) = <sup>J</sup> *<sup>d</sup>*(*u<sup>s</sup>*

*<sup>t</sup>*(**k***s*) and *�u<sup>s</sup>*

*DT*(*s*, **k***<sup>s</sup>*

) using Eq. (24).

• no diffusion with two neighbors sharing one vertex,

**Ensure:** The restored image *<sup>u</sup><sup>s</sup>* ∈ U. make t=1; stop = *FALSE*; **k***<sup>s</sup>* = **k***<sup>s</sup>*

> **for** each pixel *s* ∈ M*<sup>α</sup>* **do** make *DT*(*s*) = min

> > *t*(**k***<sup>s</sup>*

*<sup>t</sup>*) − J *<sup>d</sup>*(*�u<sup>s</sup>*

• no diffusion with one neighbor,

• no diffusion with three neighbors,

make **k***<sup>s</sup>* = **k***<sup>s</sup>*

*�* ∈ C*<sup>σ</sup>* **do** compute *DT*(*s*, **k***<sup>s</sup>*

**while** stop = *FALSE* **do for** each pixel *s* **do for** each **k***<sup>s</sup>*

> **end for end for**

**end for** compute *u<sup>s</sup>*

**if** |J *<sup>d</sup> � u<sup>s</sup>*

**else**

**end if end while**

J *d �* (*�u<sup>s</sup>*

for *u<sup>s</sup>*

*t* = *t* + 1

*u<sup>s</sup>* = *u<sup>s</sup>*

• diffuse on x direction, • diffuse in y direction, • diffuse in all directions.

*<sup>t</sup>*) takes the value

algorithm based on the TD 3).

J *d �* (*�u<sup>s</sup>*

*<sup>t</sup>* = *<sup>u</sup><sup>s</sup>*

respectively.

written as

of *DT*(*s*, **k***<sup>s</sup>*

In the discrete approach, the image is characterized by a matrix of pixels with an intensity associated to them. We consider a bi-dimensional image *u* given by a set of *M* × *N* pixels *s*. In each pixel *s*, the image *u* intensity will be denoted as *us*. Then, the image belongs to the space U*d*

$$\mathcal{U}^d := \{ \mu; \mu^s = \text{constant in } \omega^s, s = 1 \cdot \dots \cdot M \times N \}\,\,\,\tag{22}$$

with <sup>Ω</sup> <sup>=</sup> <sup>∪</sup>*sω<sup>s</sup>* , being *ω<sup>s</sup>* the domain of *s*. The set of neighbors *n<sup>s</sup>* of pixel *s* was defined as the four pixels1

The cost function adopted by the discrete approach is

$$\mathcal{J}^d(\mu\_t^s) = \sum\_{\mathbf{s}} \sum\_{p \in n^s} k^{s,p} \widehat{\Delta} \mu\_t^{s,p} \cdot \widehat{\Delta} \mu\_t^{s,p} \mu\_t^{s,p}$$

which can be interpreted as a discrete approximation of the energy norm of the field *u*. In this expression the term *ks*,*<sup>p</sup>* is the diffusion coefficient of pixel *s* with its neighbor *p*, *n<sup>s</sup>* = {*n*,*s*,*e*, *<sup>w</sup>*} are the neighbors of pixel *<sup>s</sup>* and <sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup> <sup>t</sup>* is defined as

$$
\widehat{\Delta}\mathfrak{u}\_t^{s,p} = \mathfrak{u}\_t^p - \mathfrak{u}\_t^s. \tag{23}
$$

In this case, *u<sup>s</sup> <sup>t</sup>* is explicitly computed as

$$u\_t^s(\mathbf{k}^s) = u\_{t-1}^s + \Delta t \sum\_{p \in n^s} k^{s,p} \widehat{\Delta} u\_{t-1}^{s,p} \tag{24}$$

where the index *<sup>t</sup>* <sup>≥</sup> 1 represents the iteration number, being *<sup>u</sup><sup>s</sup>* <sup>0</sup> the intensity at pixel *<sup>s</sup>*, **<sup>k</sup>***<sup>s</sup>* = {*ks*,*o*, *<sup>k</sup>s*,*<sup>l</sup>* , *<sup>k</sup>s*,*n*, *<sup>k</sup>s*,*s*} characterizes the set of coefficients associated to pixel *<sup>s</sup>*, <sup>Δ</sup>*<sup>t</sup>* is the artificial step size in time.

As opposed to the continuous case, because *u<sup>s</sup> <sup>t</sup>* is an explicit function and given the discrete set **k***s* , it is possible to compute the exact *total variation* of the cost function for each perturbation in *ks*,*p*. Also, we call **k***<sup>s</sup> �* the perturbed configuration of pixel *s* diffusivity coefficients. The value of the cost function when the perturbation is introduced is given by

$$\mathcal{J}^d(u\_t^s(\mathbf{k}\_\varepsilon^s)) = \mathcal{J}^d(u\_t^s(\mathbf{k}^s)) + D\_T(s, \mathbf{k}\_\varepsilon^s), \tag{25}$$

where *DT*(*s*, **k***<sup>s</sup> �*) represents the *total variation* of the cost function due to a perturbation in the diffusivity coefficients of pixel *s* characterized by the set **k***<sup>s</sup> �*. Likewise in the continuous case, the introduction of a perturbation to pixel *s* where *DT* is negative, will produce a decrease in the value of the cost function <sup>J</sup> *<sup>d</sup>*. Using this information we can select the best candidate pixels for perturbations.

We assume *<sup>k</sup>s*,*<sup>p</sup>* ∈ {*kε*, *<sup>k</sup>*0}, so the set of all possible configurations for **<sup>k</sup>***<sup>s</sup>* is defined as

$$\mathcal{L}(\mathbf{s}) := \{ \mathbf{k}^s = (k^{s,w}, k^{s,\varepsilon}, k^{s,\mathbf{n}}, k^{s,\mathbf{s}}); k^{s,p} \in \{ k\_\varepsilon, k\_0 \}, p = \{ \mathbf{w} \mathbf{e} \mathbf{n} \mathbf{s} \} \}.$$

We see that 16 possible combinations for **k***<sup>s</sup>* exist (values are *ks*,*<sup>p</sup>* = *k<sup>ε</sup>* or *ks*,*<sup>p</sup>* = *k*0, for each neighbor, then 24 = 16 cases are possible). The case *ks*,*<sup>w</sup>* = *ks*,*<sup>e</sup>* = *ks*,*<sup>n</sup>* = *ks*,*<sup>s</sup>* = *k<sup>ε</sup>* is not taken into account as it does not change the value of the cost function. The 15 remaining combinations are

<sup>1</sup> i.e. *north*, *south*, *east* e *west* of pixel *s*.

14 Will-be-set-by-IN-TECH

In the discrete approach, the image is characterized by a matrix of pixels with an intensity associated to them. We consider a bi-dimensional image *u* given by a set of *M* × *N* pixels *s*. In each pixel *s*, the image *u* intensity will be denoted as *us*. Then, the image belongs to the space

, being *ω<sup>s</sup>* the domain of *s*. The set of neighbors *n<sup>s</sup>* of pixel *s* was defined as

*<sup>k</sup>s*,*p*<sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

*<sup>t</sup>* is defined as

*<sup>t</sup>* <sup>−</sup> *<sup>u</sup><sup>s</sup>*

*p*∈*ns*

, *<sup>k</sup>s*,*n*, *<sup>k</sup>s*,*s*} characterizes the set of coefficients associated to pixel *<sup>s</sup>*, <sup>Δ</sup>*<sup>t</sup>* is the artificial

*<sup>t</sup>* · <sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup> <sup>t</sup>* ,

*<sup>k</sup>s*,*p*<sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

*�* the perturbed configuration of pixel *s* diffusivity coefficients. The value

)) + *DT*(*s*, **k***<sup>s</sup>*

); *<sup>k</sup>s*,*<sup>p</sup>* ∈ {*kε*, *<sup>k</sup>*0}, *<sup>p</sup>* <sup>=</sup> {w,e,n,s}}.

*�*) represents the *total variation* of the cost function due to a perturbation in the

,*s* = 1 ··· *M* × *N*} , (22)

*<sup>t</sup>* . (23)

*<sup>t</sup>* is an explicit function and given the discrete set

*<sup>t</sup>*−<sup>1</sup> (24)

<sup>0</sup> the intensity at pixel *<sup>s</sup>*, **<sup>k</sup>***<sup>s</sup>* =

*�*), (25)

*�*. Likewise in the continuous case,

<sup>U</sup>*<sup>d</sup>* :={*u*; *<sup>u</sup><sup>s</sup>* <sup>=</sup> constant in *<sup>ω</sup><sup>s</sup>*

*<sup>t</sup>*) = <sup>∑</sup>*<sup>s</sup>* <sup>∑</sup>

<sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup> <sup>t</sup>* <sup>=</sup> *<sup>u</sup><sup>p</sup>*

) = *u<sup>s</sup>*

*p*∈*ns*

which can be interpreted as a discrete approximation of the energy norm of the field *u*. In this expression the term *ks*,*<sup>p</sup>* is the diffusion coefficient of pixel *s* with its neighbor *p*, *n<sup>s</sup>* =

*<sup>t</sup>*−<sup>1</sup> <sup>+</sup> <sup>Δ</sup>*<sup>t</sup>* ∑

, it is possible to compute the exact *total variation* of the cost function for each perturbation in

*t*(**k***<sup>s</sup>*

the introduction of a perturbation to pixel *s* where *DT* is negative, will produce a decrease in the value of the cost function <sup>J</sup> *<sup>d</sup>*. Using this information we can select the best candidate

We see that 16 possible combinations for **k***<sup>s</sup>* exist (values are *ks*,*<sup>p</sup>* = *k<sup>ε</sup>* or *ks*,*<sup>p</sup>* = *k*0, for each neighbor, then 24 = 16 cases are possible). The case *ks*,*<sup>w</sup>* = *ks*,*<sup>e</sup>* = *ks*,*<sup>n</sup>* = *ks*,*<sup>s</sup>* = *k<sup>ε</sup>* is not taken into account as it does not change the value of the cost function. The 15 remaining

*�*)) = <sup>J</sup> *<sup>d</sup>*(*u<sup>s</sup>*

We assume *<sup>k</sup>s*,*<sup>p</sup>* ∈ {*kε*, *<sup>k</sup>*0}, so the set of all possible configurations for **<sup>k</sup>***<sup>s</sup>* is defined as

, *ks*,*n*, *ks*,*<sup>s</sup>*

The cost function adopted by the discrete approach is

{*n*,*s*,*e*, *<sup>w</sup>*} are the neighbors of pixel *<sup>s</sup>* and <sup>Δ</sup>*<sup>u</sup>s*,*<sup>p</sup>*

As opposed to the continuous case, because *u<sup>s</sup>*

*<sup>t</sup>* is explicitly computed as

*us t*(**k***<sup>s</sup>*

where the index *<sup>t</sup>* <sup>≥</sup> 1 represents the iteration number, being *<sup>u</sup><sup>s</sup>*

of the cost function when the perturbation is introduced is given by

<sup>J</sup> *<sup>d</sup>*(*u<sup>s</sup> t*(**k***<sup>s</sup>*

diffusivity coefficients of pixel *s* characterized by the set **k***<sup>s</sup>*

<sup>C</sup>(*s*) :={**k***<sup>s</sup>* = (*ks*,*w*, *<sup>k</sup>s*,*<sup>e</sup>*

<sup>J</sup> *<sup>d</sup>*(*u<sup>s</sup>*

U*d*

with <sup>Ω</sup> <sup>=</sup> <sup>∪</sup>*sω<sup>s</sup>*

the four pixels1

In this case, *u<sup>s</sup>*

{*ks*,*o*, *<sup>k</sup>s*,*<sup>l</sup>*

**k***s*

step size in time.

*ks*,*p*. Also, we call **k***<sup>s</sup>*

pixels for perturbations.

where *DT*(*s*, **k***<sup>s</sup>*

combinations are

<sup>1</sup> i.e. *north*, *south*, *east* e *west* of pixel *s*.

**Algorithm 3** Image restoration based on a discrete version of the TD - *RDT* -Discrete

**Require:** The 2D image *u*<sup>0</sup> ∈ U, a diffusivity coefficient *k*<sup>0</sup> and a parameter *α*. **Ensure:** The restored image *<sup>u</sup><sup>s</sup>* ∈ U. make t=1; stop = *FALSE*; **k***<sup>s</sup>* = **k***<sup>s</sup> iso*, *s* = 1..*M* × *N* **while** stop = *FALSE* **do for** each pixel *s* **do for** each **k***<sup>s</sup> �* ∈ C*<sup>σ</sup>* **do** compute *DT*(*s*, **k***<sup>s</sup> �*) following Eq. (27) **end for end for for** each pixel *s* ∈ M*<sup>α</sup>* **do** make *DT*(*s*) = min *�*<sup>∗</sup> {*DT*(*s*, **<sup>k</sup>***<sup>s</sup> �*), **<sup>k</sup>***<sup>s</sup> �* ∈ C*σ*} make **k***<sup>s</sup>* = **k***<sup>s</sup> �*<sup>∗</sup> the diffusivity coefficient associated to *DT*(*s*) **end for** compute *u<sup>s</sup> t*(**k***<sup>s</sup>* ) using Eq. (24). **if** |J *<sup>d</sup> � u<sup>s</sup> <sup>t</sup>*) − J *<sup>d</sup>*(*�u<sup>s</sup> <sup>t</sup>*−1)<sup>|</sup> <sup>&</sup>gt; *tol* **then** *t* = *t* + 1 **else** *u<sup>s</sup>* = *u<sup>s</sup> <sup>t</sup>*,*s* = 1, ··· , *M* × *N*, stop = *TRUE* **end if end while**


The last case corresponds to isotropic diffusion and is defined as **k***<sup>s</sup> iso* = {*k*0, *k*0, *k*0, *k*0}.

To compute the value of *DT* for a determined pixel, its necessary to introduce a perturbation. This is done by changing, for one pixel *s*, the set **k***<sup>s</sup>* for **k***<sup>s</sup> �* ∈ C*σ*. Then, the cost function J *d �* (*�u<sup>s</sup> <sup>t</sup>*) takes the value

$$\mathcal{J}\_{\varepsilon}^{d}(\mathfrak{e}\_{t}^{s}) = \mathcal{J}^{d}(\mathfrak{u}\_{t}^{s}) - \sum\_{p \in n^{s}} k^{s,p} \widehat{\Delta}\mathfrak{u}\_{t}^{s,p} \cdot \widehat{\Delta}\mathfrak{u}\_{t}^{s,p} + \sum\_{p \in n^{s}} k\_{\varepsilon}^{s,p} \widehat{\Delta}\_{\varepsilon}\mathfrak{u}\_{t}^{s,p} \cdot \widehat{\Delta}\_{\varepsilon}\mathfrak{u}\_{t}^{s,p},\tag{26}$$

for *u<sup>s</sup> <sup>t</sup>* = *<sup>u</sup><sup>s</sup> <sup>t</sup>*(**k***s*) and *�u<sup>s</sup> <sup>t</sup>* = *<sup>u</sup><sup>s</sup> t*(**k***<sup>s</sup> �*) computed using Eq. (24) and <sup>Δ</sup> *�us*,*<sup>p</sup> <sup>t</sup>* <sup>=</sup> *<sup>u</sup><sup>p</sup> <sup>t</sup>* <sup>−</sup>*� <sup>u</sup><sup>s</sup> t*, respectively.

For Eqs. (25) and (26) the total variation of the cost function <sup>J</sup> *<sup>d</sup>* due to the perturbation **<sup>k</sup>***<sup>s</sup> �* is written as

$$D\_T(s, \mathbf{k}\_\varepsilon^s) = \sum\_{p \in n^s} k\_\varepsilon^{s,p} \widehat{\Delta}\_\varepsilon u\_t^{s,p} \cdot \widehat{\Delta}\_\varepsilon u\_t^{s,p} - \sum\_{p \in n^s} k^{s,p} \widehat{\Delta} u\_t^{s,p} \cdot \widehat{\Delta} u\_t^{s,p} \,. \tag{27}$$

As in the continuous case, it will be considered a perturbation **k***<sup>s</sup> �* that minimizes the value of *DT*(*s*, **k***<sup>s</sup> �*). Using this information is proposed the following discrete image restoration algorithm based on the TD 3).

Fig. 7. Image restoration with the *RDT* -Discrete algorithm of the Lena image. Results correspond to *k*<sup>0</sup> = 1 and *α* = 0.05, 0.15 e 0.25 respectively.

#### **3.4 Discrete approach results**

The set M*<sup>α</sup>* is defined as in Eq. (21). As in the continuous case, the parameter *α* allows to control the values of the TD that will introduce changes in **k***<sup>s</sup>* . In all the cases it was used �*<sup>t</sup>* <sup>=</sup> <sup>1</sup> <sup>4</sup> e *k*<sup>0</sup> = 1, the maximum values that warrant the Courant-Friedrichs-Levy (stability) of the iterative solution algorithm. In Fig. 7 are presented some results obtained with this technique. The three images presented (corresponding to the results for values of *α* = 0.05, 0.15 e 0.25 respectively) present a considerable improvement of SNR (going from 26.91 in the degraded image to 28.49, 29.61 and 29.53 in the processed images, respectively).

In Figs. 8 are presented the results after different number of iterations. The edges introduced in the image are highlighted in white. We observe that after some iterations, the image has edges in almost all the edges. In this way, the variation of the cost function is practically null in two consecutive iterations, stopping the algorithm. The number of edges that are added to the image in each iteration is controlled by parameter *α*. In Fig. 9 is presented the detail of a region of the image before and after the processing.

Fig. 8. Detail of the crack configuration introduced to the image during the processing

Image Restoration via Topological Derivative 113

(*k*<sup>0</sup> = 1 and *α* = 0.20).

16 Will-be-set-by-IN-TECH

Fig. 7. Image restoration with the *RDT* -Discrete algorithm of the Lena image. Results

degraded image to 28.49, 29.61 and 29.53 in the processed images, respectively).

The set M*<sup>α</sup>* is defined as in Eq. (21). As in the continuous case, the parameter *α* allows to

In Figs. 8 are presented the results after different number of iterations. The edges introduced in the image are highlighted in white. We observe that after some iterations, the image has edges in almost all the edges. In this way, the variation of the cost function is practically null in two consecutive iterations, stopping the algorithm. The number of edges that are added to the image in each iteration is controlled by parameter *α*. In Fig. 9 is presented the detail of a

<sup>4</sup> e *k*<sup>0</sup> = 1, the maximum values that warrant the Courant-Friedrichs-Levy (stability) of the iterative solution algorithm. In Fig. 7 are presented some results obtained with this technique. The three images presented (corresponding to the results for values of *α* = 0.05, 0.15 e 0.25 respectively) present a considerable improvement of SNR (going from 26.91 in the

. In all the cases it was used

correspond to *k*<sup>0</sup> = 1 and *α* = 0.05, 0.15 e 0.25 respectively.

region of the image before and after the processing.

control the values of the TD that will introduce changes in **k***<sup>s</sup>*

**3.4 Discrete approach results**

�*<sup>t</sup>* <sup>=</sup> <sup>1</sup>

Fig. 8. Detail of the crack configuration introduced to the image during the processing (*k*<sup>0</sup> = 1 and *α* = 0.20).

**–** Black. *et al.* (Black et al. (1998)),

**–** Continuous *RDT* (Belaid et al. (2007)).

and for Continuous *RDT α* = 0.80.

results of similar quality to the existing methods.

**–** Discrete *RDT* . • Stationary methods:

case.

**–** Selective smoothing (Alvarez et al. (1992)) and

**–** Semi-quadratic minimization (Kornprobst et al. (1997)),

In Fig. 10 are presented results for classical and TD image restoration methods. The processed image corresponds to the artificially degraded image of Lena (see Fig. 3) with uniform noise (*r* = 20), with an approximate SNR of 27 dB. All the classical methods depend on a parameter *σ* with equivalent meaning. This parameter is used to control the diffusion on the edges of the image to preserve relevant characteristics. For the classical methods, *σ* was adjusted using the technique proposed by Black et al. (1998), which allows to estimate the value of *σ* as a function of the gradient of the image processed. In the case of the evolutionary methods, the number of iterations was fixed to 10, 20 and 30. The results for 20 iterations where found to shield the lowest SNR and, thus, these are reported. For the semi-quadratic minimization, the same analysis was performed for parameter *λ* and the best SNR was obtained with *λ* ≈ 10 in this

Image Restoration via Topological Derivative 115

Parameter selection in the case of TD methods is different. Both, the Discrete and Continuous *RDT* methods, depend on parameter *α* (a real value between 0 and 1), that determines the amount of cracks to be introduced in the image. This parameter determines the quantity (as a %) of the pixels that will be introduced in cracks to stop diffusion. As before, parameter *α* was analyzed to select the value that provides the best SNR (in the case of Discrete *RDT α* = 0.18

As presented, the proposed restoration methods use information of the cost function sensitivity to a change in the topology. This information is used to find the optimal domain topology that, in the presence of diffusion, will eliminate noise preserving image features. In the continuous case, we observe that this technique eliminates most of the noise but has difficulties to remove noise in regions of elevated gradients. For the Discrete *RDT* , we observe that the noise is removed from the whole image, even from the edges. This algorithm is also

In Table 1 are presented qualitative results for the Lena image. The different columns present the PSNR, SNR, *μ*(*e*) (mean error) and *σ*(*e*) (error standard deviation) between the processed image and the original one (i.e., without degradation). We observe similar results for the different methods, where the best performers were the evolutive method proposed by Black *et al.* with respect to PSNR (30.58dB) and SNR (30.37dB) and the one by Perona & Malik with respect to *σ*(*e*) (6.807). The proposed methods, Discrete *RDT* and Continuous *RDT* , provide

The proposed methods provide an intuitive tool for image restoration based on the concept of the TD. These methods are intended to modify the topology of the image by inserting cracks in selected location that, in the presence of diffusion, will improve the quality of a degraded image. The diffusion will eliminate noise while preserving edges and details in the image.

capable of improving the sharpness of the edges, enhancing the image features.

Fig. 9. Detail of Lena image before and after the processing (*k*<sup>0</sup> = 1 and *α* = 0.20).

(e) *RDT* -Discrete (f) *RDT* -Continuous

Fig. 10. Results for the Lena image restored using different methods.

## **4. Results**

minimization

The methods based on the TD have been compared to other methods proposed in the literature

	- **–** Perona & Malik (Perona & Malik (1990)),

18 Will-be-set-by-IN-TECH

Fig. 9. Detail of Lena image before and after the processing (*k*<sup>0</sup> = 1 and *α* = 0.20).

(a) Perona & Malik (b) Black *et al.* (c) Selective smoothing

The methods based on the TD have been compared to other methods proposed in the literature

(e) *RDT* -Discrete (f) *RDT* -Continuous

(d) Semi-quadratic

• Evolutionary methods:

Fig. 10. Results for the Lena image restored using different methods.

**–** Perona & Malik (Perona & Malik (1990)),

minimization

**4. Results**

	- **–** Semi-quadratic minimization (Kornprobst et al. (1997)),
	- **–** Continuous *RDT* (Belaid et al. (2007)).

In Fig. 10 are presented results for classical and TD image restoration methods. The processed image corresponds to the artificially degraded image of Lena (see Fig. 3) with uniform noise (*r* = 20), with an approximate SNR of 27 dB. All the classical methods depend on a parameter *σ* with equivalent meaning. This parameter is used to control the diffusion on the edges of the image to preserve relevant characteristics. For the classical methods, *σ* was adjusted using the technique proposed by Black et al. (1998), which allows to estimate the value of *σ* as a function of the gradient of the image processed. In the case of the evolutionary methods, the number of iterations was fixed to 10, 20 and 30. The results for 20 iterations where found to shield the lowest SNR and, thus, these are reported. For the semi-quadratic minimization, the same analysis was performed for parameter *λ* and the best SNR was obtained with *λ* ≈ 10 in this case.

Parameter selection in the case of TD methods is different. Both, the Discrete and Continuous *RDT* methods, depend on parameter *α* (a real value between 0 and 1), that determines the amount of cracks to be introduced in the image. This parameter determines the quantity (as a %) of the pixels that will be introduced in cracks to stop diffusion. As before, parameter *α* was analyzed to select the value that provides the best SNR (in the case of Discrete *RDT α* = 0.18 and for Continuous *RDT α* = 0.80.

As presented, the proposed restoration methods use information of the cost function sensitivity to a change in the topology. This information is used to find the optimal domain topology that, in the presence of diffusion, will eliminate noise preserving image features. In the continuous case, we observe that this technique eliminates most of the noise but has difficulties to remove noise in regions of elevated gradients. For the Discrete *RDT* , we observe that the noise is removed from the whole image, even from the edges. This algorithm is also capable of improving the sharpness of the edges, enhancing the image features.

In Table 1 are presented qualitative results for the Lena image. The different columns present the PSNR, SNR, *μ*(*e*) (mean error) and *σ*(*e*) (error standard deviation) between the processed image and the original one (i.e., without degradation). We observe similar results for the different methods, where the best performers were the evolutive method proposed by Black *et al.* with respect to PSNR (30.58dB) and SNR (30.37dB) and the one by Perona & Malik with respect to *σ*(*e*) (6.807). The proposed methods, Discrete *RDT* and Continuous *RDT* , provide results of similar quality to the existing methods.

The proposed methods provide an intuitive tool for image restoration based on the concept of the TD. These methods are intended to modify the topology of the image by inserting cracks in selected location that, in the presence of diffusion, will improve the quality of a degraded image. The diffusion will eliminate noise while preserving edges and details in the image.

Haug, E. J., Choi, K. K. & Komkov, V. (1986). *Design Sensitivity Analysis of Structural Systems*,

Image Restoration via Topological Derivative 117

He, L. & Osher, S. (2006). Solving the chan-vese model by multiphase level set algorithm based on the topological derivative, *Technical Report 56*, UCLA - CAM. Hintermüler, M. (2005). Fast level set based algorithms usind shape and topological

Hughes, T. J. R. (2000). *The Finite Element Method - Linear Static and Dynamic Finite Element*

Kornprobst, P., Deriche, R. & Aubert, G. (1997). Nonlinear operators in image restoration,

Larrabide, I. (2007). *Image processing via topological derivative and its applications to Human*

Larrabide, I., Feijóo, R. A., Novotny, A. A. & Taroco, E. A. (2008). Topological derivative: A

Larrabide, I., Feijóo, R. A., Novotny, A. A., Taroco, E. & Masmoudi, M. (2005). An image

Larrabide, I., Feijóo, R., Taroco, E. & Novotny, A. (2006). Configurational derivative as a

Larrabide, I., Novotny, A. A., Feijóo, R. A. & Taroco, E. (2005). A medical image enhancement

Larrabide, I., Novotny, A. A., Feijóo, R. A. & Taroco, E. (2006). Topological derivative as

Murat, F. & Simon, J. (1976). *Sur le Contrôle par un Domaine Géométrique*, PhD thesis, Université

Novotny, A. A. (2003). *Análise de Sensibilidade Topológica*, Phd thesis, Laboratorio Nacional de

Novotny, A. A., Feijóo, R. A., Taroco, E. & Padra, C. (2003). Topological sensitivity analysis,

Schumacher, A. (1995). *Topologieoptimierung von Bauteilstrukturen unter Verwendung von Lochpositionierungkriterien*, PhD thesis, Universität-Gesamthochschule-Siegen. Sokolowski, J. & Zolésio, J. P. (1992). *Introduction to Shape Optimization - Shape Sensitivity*

*Computer Methods in Applied Mechanics and Engineering* 192: 803–829. Perona, P. & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion, *IEEE*

Pironneau, O. (1984). *Optimal Shape Design for Elliptic Systems*, Springer-Verlag.

*Proceedings of the International Conference on Computer Vision and Pattern Recognition*,

*Cardiovascular System modelling and simulation*, PhD thesis, Laboratório Nacional de

segmentation method based on a discrete version of the topological derivative, *World Congress Structural and Multidisciplinary Optimization 6, Rio de Janeiro*, International

tool for image segmentation, *Proceedings of the ECCM 2006. LNEC - Lisbon, Portugal 5th-9th June*, European Committee of Computational Solid and Structural Mechanics

algorithm based on topological derivative and anisotropic diffusion, *Proceedings of the XXVI Iberian Latin-American Congress on Computational Methods in Engineering -*

a tool for image processing: Image segmentation, *Technical Report 15/2006*, LNCC -

sensitivity, *Control and Cybernetics* 34(1): 305–324.

URL: *citeseer.ist.psu.edu/kornprobst97nonlinear.html*

Jain, A. K. (1989). *Fundamentals of Digital Image Processing*, Prentice Hall.

Computação Científica - LNCC/MCT, Petrópolis - Brazil.

Society for Structural and Multidisciplinary Optimization.

*CILAMCE 2005 - Guarapari, Espírito Santo, Brazil*.

Laboratório Nacional de Computação Científica.

Computação Científica, Petrópolis - RJ - Brazil.

*Trans. Pattern Anal. Machine Intell.* 12(7): 629–639.

Pierre et Marie Curie, Paris - France.

*Analysis*, Springer-Verlag.

tool for image processing, *Computers & Structures* 86: 1386–1403.

Academic Press.

*Analysis*, Prentice-Hall.


IEEE, Puerto-Rico, pp. 325–331.


Table 1. Comparison between the proposed and classical image restoration methods.

## **5. Online material**

The computational implementation of these methods in Matlab is available online at Matlab Central 2. A more complete description of the mathematical and numerical methods used in this work can be found in the work of Larrabide (2007).

## **6. Acknowledgements**

Ignacio Larrabide was partly supported by the Brazilian agency CNPq (141336/2003-0) and CIBER-BBN, Barcelona, Spain. The authors would also like to acknowledge the contribution of Prof. Andre Novotny in the form of fruitful discussions that helped improving the quality of this work.

#### **7. References**


Haug, E. J. & Céa, J. (1981). Optimization of distributed parameters structures, *Iowa - USA*.

Gonzalez, C. R. & Woods, R. E. (2001). *Digital Image Processing - Second Edition*, Prentice Hall.

<sup>2</sup> http://www.mathworks.com/matlabcentral/fileexchange/14223-image-restoration-via-topologicalderivative

20 Will-be-set-by-IN-TECH

Degraded image 26.88 26.67 0.6005 11.535 Perona & Malik 30.36 30.15 1.1013 6.8077 Black *et al.* 30.58 30.37 1.1003 7.1221 Selective smoothing 29.68 29.27 1.0106 8.0220 Semi-quadratic minimization 30.15 30.04 1.0968 7.3185 Discrete *RDT* 29.98 29.73 1.1463 8.0455 Continuous *RDT* 29.93 29.89 1.0944 8.3446

Table 1. Comparison between the proposed and classical image restoration methods.

this work can be found in the work of Larrabide (2007).

The computational implementation of these methods in Matlab is available online at Matlab Central 2. A more complete description of the mathematical and numerical methods used in

Ignacio Larrabide was partly supported by the Brazilian agency CNPq (141336/2003-0) and CIBER-BBN, Barcelona, Spain. The authors would also like to acknowledge the contribution of Prof. Andre Novotny in the form of fruitful discussions that helped improving the quality

Alvarez, L., Guichard, F., Lions, P. & Morel, J. (1992). Axiomatisation et nouveaux operatéurs

Auroux, D., Masmoudi, M. & Belaid, L. (2006). Image restoration and classification by

Belaid, L. J., Jaoua, M., Masmoudi, M. & Siala, L. (2007). Application of the topological

Black, M. J., Sapiro, G., Marimont, D. H. & Heeger, D. (1998). Robust anisotropic diffusion,

Eschenauer, H. A., Kobelev, V. V. & Schumacher, A. (1994). Bubble method for topology and

Frangakis, A. S. & Hegerl, R. (2001). Noise reduction in electron tomographic reconstructions

<sup>2</sup> http://www.mathworks.com/matlabcentral/fileexchange/14223-image-restoration-via-topological-

shape optimization of structures, *Structural Optimization* 8: 42–51. Eshelby, J. D. (1985). The elastic energy-momentum tensor, *J. Elasticity 5* pp. 321–335.

using nonlinear anisotropic diffusion, *J. Struct. Biol.* 1(135): 239–250. Gonzalez, C. R. & Woods, R. E. (2001). *Digital Image Processing - Second Edition*, Prentice Hall. Haug, E. J. & Céa, J. (1981). Optimization of distributed parameters structures, *Iowa - USA*.

*Applications - CIMNE, Barcelona, Spain 2006 (In press)* .

*IEEE Transactions on Image Processing* 7(3): 421–423.

de la morphologie mathematique, *Comptes Rendus de l'Académie des Sciences* t. 315,

topological asymptotic expansion, *Variational Formulations in Mechanics: Theory and*

gradient to image restoration and edge detection, *To apper on Special Issue on "Shape and Topological Sensitivity Analysis: Theory and Application" of the Engineering Analysis*

**5. Online material**

**6. Acknowledgements**

Série I: 265–268.

*with Boundary Element Journal* .

of this work.

**7. References**

derivative

PSNR (dB) SNR (dB) *μ*(*e*) *σ*(*e*)

	- URL: *citeseer.ist.psu.edu/kornprobst97nonlinear.html*

**1. Introduction**

Image restoration or deconvolution of a blurred natural image is a mature research activity with a rich set of available techniques and algorithms, well-summarised in review articles, Banham & Katsaggelos (1997); Kundur & Hatzinakos (1996). Despite this history and volume of work, there is current research activity motivated by the desire to find yet superior methods to restore the ground truth image (GTI). Important performance metrics to assess the efficacy of restoration methods include: restoration accuracy, computational complexity and convergence speed. In this chapter we use these performance metrics in the development of restoration methods of greatest utility for real-world applications where complexity/speed is a major concern and the evaluation of image restoration needs to take into account the highly structured features of natural images and, to a lesser extent, the human visual system.

**Regularized Image Restoration** 

*The Australian National University, Canberra, ACT,* 

Pradeepa D. Samarasinghe and Rodney A. Kennedy

*Research School of Engineering, College of Engineering and Computer Science,* 

**6**

*Australia* 

The scope of this work focusses on non-blind image restoration where the point spread function (PSF) of the blur convolutional kernel is known. Blind deconvolution is, by its nature, a more challenging problem, Haykin (1994); Kundur & Hatzinakos (1996). However with effective and efficient PSF estimation techniques, Fergus et al. (2006); Joshi et al. (2008); Krahmer et al. (2006); Nayar & Ben-Ezra (2004); Oliveira et al. (2007), the research trend has been to handling blind deconvolution in two steps, with PSF estimation as the first step and image estimation as the second step, Levin et al. (2009). This motivates us to focus on efficient

In this chapter, we first analyze existing linear deterministic restoration models and develop a class of novel models with better performance. Then using regularization as the basis, we link linear deterministic and stochastic restoration models. By introducing a previously developed novel visual metric to image regularization analysis, we study the purported superior performance of stochastic prior models and demonstrate that those models are not superior to simpler linear deterministic prior models. In addition, we show that the high complexity "derivative likelihood" models under the maximum a posteriori (MAP) framework offer no

Image acquisition being an inverse problem can be modeled by a continuous model in an infinite dimensional space, which is categorized as a (linear) Fredholm integral equation of

algorithms for image restoration where the blur convolutional kernel is known.

significant advantage to a properly configured, efficient "normal likelihood" model.

**2. Quadratic regularization in image restoration**

**2.1 Regularization**


## **Regularized Image Restoration**

Pradeepa D. Samarasinghe and Rodney A. Kennedy

*Research School of Engineering, College of Engineering and Computer Science, The Australian National University, Canberra, ACT, Australia* 

## **1. Introduction**

22 Will-be-set-by-IN-TECH

118 Image Restoration – Recent Advances and Applications

Tschumperle, D. (2006). Fast anisotropic smoothing of multi-valued images using curvature-preserving pde's, *International Journal of Computer Vision* 68(1): 65–82. Weickert, J. (1995). Multiscale texture enhancement, *V. Hlavac, R. Sara (Eds.), Computer analysis*

Zolésio, J. P. (1981). *The material derivative (or speed) method for shape optimization, in:*

*Optimization of Distributed Parameter Structures*, Vol. II(50), NATO Adv. Study Inst.

*of images and patterns, Lecture Notes in Computer Science* 970: 230–237.

Ser. E, Appl. Sci., pp. 1089–1151.

Image restoration or deconvolution of a blurred natural image is a mature research activity with a rich set of available techniques and algorithms, well-summarised in review articles, Banham & Katsaggelos (1997); Kundur & Hatzinakos (1996). Despite this history and volume of work, there is current research activity motivated by the desire to find yet superior methods to restore the ground truth image (GTI). Important performance metrics to assess the efficacy of restoration methods include: restoration accuracy, computational complexity and convergence speed. In this chapter we use these performance metrics in the development of restoration methods of greatest utility for real-world applications where complexity/speed is a major concern and the evaluation of image restoration needs to take into account the highly structured features of natural images and, to a lesser extent, the human visual system.

The scope of this work focusses on non-blind image restoration where the point spread function (PSF) of the blur convolutional kernel is known. Blind deconvolution is, by its nature, a more challenging problem, Haykin (1994); Kundur & Hatzinakos (1996). However with effective and efficient PSF estimation techniques, Fergus et al. (2006); Joshi et al. (2008); Krahmer et al. (2006); Nayar & Ben-Ezra (2004); Oliveira et al. (2007), the research trend has been to handling blind deconvolution in two steps, with PSF estimation as the first step and image estimation as the second step, Levin et al. (2009). This motivates us to focus on efficient algorithms for image restoration where the blur convolutional kernel is known.

In this chapter, we first analyze existing linear deterministic restoration models and develop a class of novel models with better performance. Then using regularization as the basis, we link linear deterministic and stochastic restoration models. By introducing a previously developed novel visual metric to image regularization analysis, we study the purported superior performance of stochastic prior models and demonstrate that those models are not superior to simpler linear deterministic prior models. In addition, we show that the high complexity "derivative likelihood" models under the maximum a posteriori (MAP) framework offer no significant advantage to a properly configured, efficient "normal likelihood" model.

## **2. Quadratic regularization in image restoration**

## **2.1 Regularization**

Image acquisition being an inverse problem can be modeled by a continuous model in an infinite dimensional space, which is categorized as a (linear) Fredholm integral equation of

for some matrix *A* and *aij* is the (*i*, *j*) entry. The constrained minimization problem in (2) can be solved by the method of Lagrange multipliers, which is to determine *<sup>g</sup>*, an estimate of

Regularized Image Restoration 121

where *λ* > 0 is the Lagrange multiplier and is often called as the *regularization parameter*. As the regularization parameter, *λ*, controls the tradeoff between the solution accuracy �*b* − <sup>K</sup>*g*�<sup>2</sup> and its degree of regularity <sup>Φ</sup>(*g*), choosing a proper value for *<sup>λ</sup>* is important in image

The first term in (4), named as *data-fidelity term* fits to the data, while *stabilizing functional* incorporates "believed" properties of the GTI. Generally the *data-fidelity term* is a standard fixed choice. In contrast, the richness and variety of image restoration techniques comes down to different choices of the regularization term, reflecting different implicit models. As the choice of the *stabilizing functional* can take a variety of forms, in this chapter, we selected two widely used model classes for our analysis: the fast quadratic stabilizing functionals introduced in section 2.2, and Sparse and Laplacian prior methods in section 3.2. The latter model class can be developed from relating the *stabilizing functional* to the *priori* knowledge using a probabilistic viewpoint and is claimed to have better performance, Levin et al. (2007). When the stabilizing functional Φ(*g*) in (4) belongs to the class of nonnegative quadratic

where D is a bounded linear operator, Miller (1970) and is often called the *regularization operator* or *stabilizing operator*. It is shown in Hunt (1973), that the minimization problem in (5) can be formulated as a constrained least squares image restoration problem when the solution

<sup>K</sup>*T*<sup>K</sup> <sup>+</sup> *<sup>λ</sup>*(D*T*D)

We extend, in a trivial way, the above formulation by considering D as the combination of *R*

With the introduction of *R* Lagrange multipliers, the general form of (5) can be expressed as

�*<sup>b</sup>* − K*g*�<sup>2</sup> <sup>+</sup>

*<sup>λ</sup>r*(D*<sup>T</sup> <sup>r</sup>* D*r*) −<sup>1</sup>

*R* ∑ *r*=1 <sup>2</sup> ,..., <sup>D</sup>*<sup>T</sup> R T*

> *R* ∑ *r*=1

�*<sup>b</sup>* − K*g*�<sup>2</sup> <sup>+</sup> *<sup>λ</sup>* <sup>Φ</sup>(*g*), (4)

�*<sup>b</sup>* − K*g*�<sup>2</sup> <sup>+</sup> *<sup>λ</sup>* �D*g*�2, (5)

<sup>K</sup>*Tb*. (7)

*g*. (8)

*<sup>λ</sup>r*�D*rg*�2, (9)

<sup>K</sup>*Tb*. (10)

(K*T*<sup>K</sup> <sup>+</sup> *<sup>λ</sup>* <sup>D</sup>*T*D)*<sup>g</sup>* <sup>=</sup> <sup>K</sup>*Tb*. (6)

−<sup>1</sup>

ground truth *g*, by minimizing the functional

functionals, the minimization problem can be expressed as

*g* satisfies the necessary and sufficient condition of

component regularization operators, in the form of

for which, the closed form solution is given by

This leads to the closed form solution for (5) in the form

*<sup>g</sup>* <sup>=</sup> arg min *g*

*<sup>g</sup>* <sup>=</sup>

<sup>D</sup>*<sup>g</sup>*

KK*<sup>T</sup>* <sup>+</sup>

*<sup>g</sup>* <sup>=</sup> arg min *g*

*<sup>g</sup>* <sup>=</sup>

D*T* <sup>1</sup> , <sup>D</sup>*<sup>T</sup>*

restoration.

the first kind, Demoment (1989). In the sense of Hardamard, Hadamard (1952), a solution to a well-posed problem satisfies the conditions of existence, uniqueness and stability. As Fredholm integral equations of the first kind do not meet the criteria for a well-posed problem, image restoration belongs into the general class of problems which are classified as ill-posed problems, Tikhonov & Arsenin (1977). The ill-posed nature of image restoration problem implies that, small bounded perturbations in the data may lead to unbounded deviations in the solution, Phillips (1962).

For images defined on a discrete set, linear algebra can be used to find solutions for ill-posed problems such as image restoration. One of the simplest methods to restore images affected by a linear distortion is the use of the pseudo inverse, Albert (1972), for which the solution fulfils the first two conditions (existence and uniqueness) of Hardamard's well-posed problem, Hadamard (1952), but fails in meeting the stability condition. This motivates or leads to regularization as one of the most widely accepted and used techniques, in which the solution fulfils all three conditions of a well-posed problem. The concept underlying regularization is to find an acceptable solution from imperfect data, for which, the problem should be stated more completely by including some extra or priori information, Miller (1970); Tikhonov & Arsenin (1977).

Regularization approaches to image restoration are classified broadly in two ways: stochastic regularization which uses the knowledge of covariance matrices of the GTI and noise; and deterministic regularization which deems that most natural images are relatively featureless with limited high-frequency activity, Banham & Katsaggelos (1997). While stochastic regularization has been used extensively in the past, with important contributions to the field such as Wiener filter, Wiener (1942), recently, much emphasis has been on the use of derivative filters with deterministic regularization, Fergus et al. (2006); Levin et al. (2007). Thus, our contribution in this chapter relates to deterministic regularization and the term regularization, henceforth, refers to deterministic regularization techniques.

Among many regularization techniques, Tikhonov, Tikhonov & Arsenin (1977) regularization is one of the first and best-known techniques for stabilization. It was proposed in Tikhonov & Arsenin (1977), that the solution for

$$b = \pounds \mathfrak{g} + n\_\prime \tag{1}$$

where *b* is the measured data, *g* is the original data (ground truth), K is the distortion operator or the transformation and *n* represents additive random noise, can be achieved by constrained minimization of a functional Φ(*g*), which is called the *stabilizing functional*. Under the stabilizing functional approach, the image restoration problem is formulated as determining an estimate *<sup>g</sup>* of *<sup>g</sup>*, which minimizes the functional <sup>Φ</sup>(*g*) under the condition that the estimate *<sup>g</sup>* satisfies

$$\|\|b - \mathcal{K}\mathbf{g}\|\|^2 = \delta\_\prime \tag{2}$$

where *δ* is a positive constant and �·� denotes the Frobenius norm

$$||A|| = \sqrt{\sum\_{i=1}^{m\_1} \sum\_{j=1}^{m\_2} a\_{ij}^2} \tag{3}$$

2 Will-be-set-by-IN-TECH

the first kind, Demoment (1989). In the sense of Hardamard, Hadamard (1952), a solution to a well-posed problem satisfies the conditions of existence, uniqueness and stability. As Fredholm integral equations of the first kind do not meet the criteria for a well-posed problem, image restoration belongs into the general class of problems which are classified as ill-posed problems, Tikhonov & Arsenin (1977). The ill-posed nature of image restoration problem implies that, small bounded perturbations in the data may lead to unbounded deviations in

For images defined on a discrete set, linear algebra can be used to find solutions for ill-posed problems such as image restoration. One of the simplest methods to restore images affected by a linear distortion is the use of the pseudo inverse, Albert (1972), for which the solution fulfils the first two conditions (existence and uniqueness) of Hardamard's well-posed problem, Hadamard (1952), but fails in meeting the stability condition. This motivates or leads to regularization as one of the most widely accepted and used techniques, in which the solution fulfils all three conditions of a well-posed problem. The concept underlying regularization is to find an acceptable solution from imperfect data, for which, the problem should be stated more completely by including some extra or priori information, Miller (1970);

Regularization approaches to image restoration are classified broadly in two ways: stochastic regularization which uses the knowledge of covariance matrices of the GTI and noise; and deterministic regularization which deems that most natural images are relatively featureless with limited high-frequency activity, Banham & Katsaggelos (1997). While stochastic regularization has been used extensively in the past, with important contributions to the field such as Wiener filter, Wiener (1942), recently, much emphasis has been on the use of derivative filters with deterministic regularization, Fergus et al. (2006); Levin et al. (2007). Thus, our contribution in this chapter relates to deterministic regularization and the term regularization,

Among many regularization techniques, Tikhonov, Tikhonov & Arsenin (1977) regularization is one of the first and best-known techniques for stabilization. It was proposed in Tikhonov &

where *b* is the measured data, *g* is the original data (ground truth), K is the distortion operator or the transformation and *n* represents additive random noise, can be achieved by constrained minimization of a functional Φ(*g*), which is called the *stabilizing functional*. Under the stabilizing functional approach, the image restoration problem is formulated as determining an estimate *<sup>g</sup>* of *<sup>g</sup>*, which minimizes the functional <sup>Φ</sup>(*g*) under the condition that

> *m*1 ∑ *i*=1

*m*<sup>2</sup> ∑ *j*=1 *a*2

*b* = K*g* + *n*, (1)

�*<sup>b</sup>* − K*g*�<sup>2</sup> <sup>=</sup> *<sup>δ</sup>*, (2)

*ij* (3)

henceforth, refers to deterministic regularization techniques.

where *δ* is a positive constant and �·� denotes the Frobenius norm

�*A*� =

the solution, Phillips (1962).

Tikhonov & Arsenin (1977).

Arsenin (1977), that the solution for

the estimate *<sup>g</sup>* satisfies

for some matrix *A* and *aij* is the (*i*, *j*) entry. The constrained minimization problem in (2) can be solved by the method of Lagrange multipliers, which is to determine *<sup>g</sup>*, an estimate of ground truth *g*, by minimizing the functional

$$\|\|b - \mathcal{K}g\|\|^2 + \lambda \, \Phi(\mathcal{g})\_\prime \, \tag{4}$$

where *λ* > 0 is the Lagrange multiplier and is often called as the *regularization parameter*. As the regularization parameter, *λ*, controls the tradeoff between the solution accuracy �*b* − <sup>K</sup>*g*�<sup>2</sup> and its degree of regularity <sup>Φ</sup>(*g*), choosing a proper value for *<sup>λ</sup>* is important in image restoration.

The first term in (4), named as *data-fidelity term* fits to the data, while *stabilizing functional* incorporates "believed" properties of the GTI. Generally the *data-fidelity term* is a standard fixed choice. In contrast, the richness and variety of image restoration techniques comes down to different choices of the regularization term, reflecting different implicit models. As the choice of the *stabilizing functional* can take a variety of forms, in this chapter, we selected two widely used model classes for our analysis: the fast quadratic stabilizing functionals introduced in section 2.2, and Sparse and Laplacian prior methods in section 3.2. The latter model class can be developed from relating the *stabilizing functional* to the *priori* knowledge using a probabilistic viewpoint and is claimed to have better performance, Levin et al. (2007).

When the stabilizing functional Φ(*g*) in (4) belongs to the class of nonnegative quadratic functionals, the minimization problem can be expressed as

$$\widehat{\mathbf{g}} = \underset{\mathbf{g}}{\text{arg min}} \; \|\mathbf{b} - \mathcal{K}\mathbf{g}\|\|^2 + \lambda \left\|\mathcal{D}\mathbf{g}\right\|^2,\tag{5}$$

where D is a bounded linear operator, Miller (1970) and is often called the *regularization operator* or *stabilizing operator*. It is shown in Hunt (1973), that the minimization problem in (5) can be formulated as a constrained least squares image restoration problem when the solution *g* satisfies the necessary and sufficient condition of

$$(\mathcal{K}^T \mathcal{K} + \lambda \mathcal{D}^T \mathcal{D}) \mathcal{g} = \mathcal{K}^T \mathcal{b}. \tag{6}$$

This leads to the closed form solution for (5) in the form

$$
\widehat{\mathbf{g}} = \left(\boldsymbol{\mathcal{K}}^T \boldsymbol{\mathcal{K}} + \lambda (\boldsymbol{\mathcal{D}}^T \boldsymbol{\mathcal{D}})\right)^{-1} \boldsymbol{\mathcal{K}}^T \boldsymbol{b}.\tag{7}
$$

We extend, in a trivial way, the above formulation by considering D as the combination of *R* component regularization operators, in the form of

$$\mathcal{D}\mathfrak{g} \triangleq \left(\mathcal{D}\_1^T, \mathcal{D}\_2^T, \dots, \mathcal{D}\_R^T\right)^T \mathfrak{g}.\tag{8}$$

With the introduction of *R* Lagrange multipliers, the general form of (5) can be expressed as

$$\widehat{\mathbf{g}} = \operatorname\*{arg\,min}\_{\mathbf{g}} \|\boldsymbol{b} - \boldsymbol{\mathcal{K}}\boldsymbol{g}\|^2 + \sum\_{r=1}^{R} \lambda\_r \|\boldsymbol{\mathcal{D}}\_r \boldsymbol{g}\|^2 \boldsymbol{\lambda} \tag{9}$$

for which, the closed form solution is given by

$$\widehat{\boldsymbol{g}} = \left(\boldsymbol{\mathcal{K}}\boldsymbol{\mathcal{K}}^T + \sum\_{r=1}^{R} \lambda\_r (\boldsymbol{\mathcal{D}}\_r^T \boldsymbol{\mathcal{D}}\_r)\right)^{-1} \boldsymbol{\mathcal{K}}^T \boldsymbol{b}. \tag{10}$$

2. Non-isotropic SOPDO – The non-isotropic SOPDO is formed by

(1995).

**2.2.3 Mixed partial derivative operator**

the direction of the derivatives.

than one dimensional can be expressed as

**2.3 Non-blind image restoration through SOPDO**

**2.3.1 Noisy image deconvolution**

restoration techniques.

D*g* =

⎛ ⎝ *∂xx ∂xy ∂yy*

Regularized Image Restoration 123

As the edges and lines in images may occur in any direction, when the differential operator is isotropic it would give better results than a non-isotropic differential operator, Leung & Lu

In general, considering only even-order derivatives, the use of directional derivatives in more

where *p* is the order of the derivatives, *m* is the number of dimensions and *s*<sup>1</sup> to *sm* represent

Using the above general model, we introduce a new regularization operator, with different combinations of higher order derivative operators. In this discussion, we limit the use of higher order derivative operators up to the second order, and the new PDO is called first and

second order derivative operator (FSOPDO). With FSOPDO, D*g* in (11) takes the form

⎛

*∂x ∂y ∂xx ∂yy*

⎞

⎟⎟⎠ *g*.

⎜⎜⎝

These quadratic regularization functionals are compared in a new perspective with the widely

Although most previous image restoration algorithms have considered FOPDO as the regularization model, Levin et al. (2007), we claim that SOPDO has better performance in terms of the difference between the ground truth and the estimated data on images which are susceptible to noise. Here, we deal only with additive Gaussian noise, as it effectively models the noise in many different imaging scenarios. In this section, we study in detail a few simulation results which are used to do comparison evaluations with other existing image

We take non-isotropic SOPDO as the regularization operator for image restoration through least squares restoration as given in section 2.2. In order to compare the performance of the non-isotropic SOPDO prior model, we take two regularization models, FOPDO and a sparse stabilizing functional defined in Levin et al. (2007). Relating regularization to probability, the stabilizing functional in image restoration is also referred as prior model and

D*g* =

used prior models which are believed to result in better performance in section 3.5.

<sup>D</sup>*<sup>g</sup>* <sup>=</sup> � *∂ p s p* 1 , *∂ p s p* 2 ,..., *∂ p s p m* �

⎞ ⎠ *g*.

*g*. (11)

As the images are of limited support and when the corresponding hypothesis of uniformity on image edges can be made, the matrices K and D in (10) have a special structure and are called block circulant matrices, Hunt (1971). As circulant matrices can be diagonalized by the discrete Fourier transform, the minimization in (9) can be solved extremely quickly using the Fourier domain techniques, Hunt (1973).

## **2.2 Regularization operators as components in quadratic stabilizing functionals**

The generality of the regularization operator allows the development of a class of linear operators and the minimization in (9) will be the source of many regularizing solutions for (1) depending on the choice of the regularization operator. This choice is usually based on the known details of the image formation process and plays an important role in the regularization.

The simplest regularization operator is when D is an identity matrix, where D*g* = *g*, and the regularized solution for this was referred as *minimum norm restoration*, Hunt & Andrews (1977). In general, D often takes the form of a sparsifying operator such as a discrete approximation of a derivative operator. Through the experiments in Zhu & Mumford (1997), it was shown that even though the statistics of natural images vary from image to image, the histograms for the response of derivative filters are relatively consistent and scale invariant across the images. Taking these factors into consideration, in this section, we discuss a class of regularization operators based on the partial derivative operators (PDO), which could be used in the quadratic stabilizing functional.

#### **2.2.1 First order partial derivative operator**

When first order partial derivative operators (FOPDO) are considered as the regularization operators, D*g* in (8) can be expressed as

$$\mathcal{D}\mathcal{g} = \begin{pmatrix} \partial\_x \\ \partial\_y \end{pmatrix} \mathcal{g}\_\mathcal{H}$$

where *∂<sup>x</sup>* and *∂<sup>y</sup>* are any discrete space, spatially invariant linear operators that emulate first order derivative in *x* and *y* directions, respectively, Levin et al. (2007). This type of regularization uses two component regularization operators.

#### **2.2.2 Second order partial derivative operator**

Second order partial derivative operators (SOPDO), can be derived mainly in two forms.

1. Isotropic SOPDO – When the regularization operator takes the form

$$\mathcal{D}g = \begin{pmatrix} \partial\_{xx} \\ \partial\_{yy} \end{pmatrix} g\_{\prime}$$

it is called the isotropic SOPDO. Though the SOPDO defined above cannot be considered as a true isotropic differential operator, such as the continuous Laplacian operator, it gives the simplest possible isotropic operator with even-order derivatives, Leung & Lu (1995). Similar to FOPDO, *∂xx* and *∂yy* represent any discrete space, spatially invariant linear operators that emulate second order derivatives.

4 Will-be-set-by-IN-TECH

As the images are of limited support and when the corresponding hypothesis of uniformity on image edges can be made, the matrices K and D in (10) have a special structure and are called block circulant matrices, Hunt (1971). As circulant matrices can be diagonalized by the discrete Fourier transform, the minimization in (9) can be solved extremely quickly using the

The generality of the regularization operator allows the development of a class of linear operators and the minimization in (9) will be the source of many regularizing solutions for (1) depending on the choice of the regularization operator. This choice is usually based on the known details of the image formation process and plays an important role in the

The simplest regularization operator is when D is an identity matrix, where D*g* = *g*, and the regularized solution for this was referred as *minimum norm restoration*, Hunt & Andrews (1977). In general, D often takes the form of a sparsifying operator such as a discrete approximation of a derivative operator. Through the experiments in Zhu & Mumford (1997), it was shown that even though the statistics of natural images vary from image to image, the histograms for the response of derivative filters are relatively consistent and scale invariant across the images. Taking these factors into consideration, in this section, we discuss a class of regularization operators based on the partial derivative operators (PDO), which could be

When first order partial derivative operators (FOPDO) are considered as the regularization

 *∂<sup>x</sup> ∂y g*,

where *∂<sup>x</sup>* and *∂<sup>y</sup>* are any discrete space, spatially invariant linear operators that emulate first order derivative in *x* and *y* directions, respectively, Levin et al. (2007). This type of

Second order partial derivative operators (SOPDO), can be derived mainly in two forms.

D*g* =

 *∂xx ∂yy g*,

it is called the isotropic SOPDO. Though the SOPDO defined above cannot be considered as a true isotropic differential operator, such as the continuous Laplacian operator, it gives the simplest possible isotropic operator with even-order derivatives, Leung & Lu (1995). Similar to FOPDO, *∂xx* and *∂yy* represent any discrete space, spatially invariant linear

D*g* =

regularization uses two component regularization operators.

operators that emulate second order derivatives.

1. Isotropic SOPDO – When the regularization operator takes the form

**2.2 Regularization operators as components in quadratic stabilizing functionals**

Fourier domain techniques, Hunt (1973).

used in the quadratic stabilizing functional.

**2.2.1 First order partial derivative operator**

**2.2.2 Second order partial derivative operator**

operators, D*g* in (8) can be expressed as

regularization.

2. Non-isotropic SOPDO – The non-isotropic SOPDO is formed by

$$\mathcal{D}\mathcal{g} = \begin{pmatrix} \partial\_{xx} \\ \partial\_{xy} \\ \partial\_{yy} \end{pmatrix} \mathcal{g} \mathcal{A}$$

As the edges and lines in images may occur in any direction, when the differential operator is isotropic it would give better results than a non-isotropic differential operator, Leung & Lu (1995).

#### **2.2.3 Mixed partial derivative operator**

In general, considering only even-order derivatives, the use of directional derivatives in more than one dimensional can be expressed as

$$\mathcal{D}\mathbf{g} = (\eth\_{\sf s\_1^p}^p \eth\_{\sf s\_2^p}^p \dots \eth\_{\sf s\_m^p}^p)\mathbf{g}.\tag{11}$$

where *p* is the order of the derivatives, *m* is the number of dimensions and *s*<sup>1</sup> to *sm* represent the direction of the derivatives.

Using the above general model, we introduce a new regularization operator, with different combinations of higher order derivative operators. In this discussion, we limit the use of higher order derivative operators up to the second order, and the new PDO is called first and second order derivative operator (FSOPDO). With FSOPDO, D*g* in (11) takes the form

$$\mathcal{D}\mathcal{g} = \begin{pmatrix} \partial\_{\mathcal{X}} \\ \partial\_{y} \\ \partial\_{\mathcal{X}} \\ \partial\_{\mathcal{Y}} \end{pmatrix} \mathcal{g} \cdot$$

These quadratic regularization functionals are compared in a new perspective with the widely used prior models which are believed to result in better performance in section 3.5.

#### **2.3 Non-blind image restoration through SOPDO**

#### **2.3.1 Noisy image deconvolution**

Although most previous image restoration algorithms have considered FOPDO as the regularization model, Levin et al. (2007), we claim that SOPDO has better performance in terms of the difference between the ground truth and the estimated data on images which are susceptible to noise. Here, we deal only with additive Gaussian noise, as it effectively models the noise in many different imaging scenarios. In this section, we study in detail a few simulation results which are used to do comparison evaluations with other existing image restoration techniques.

We take non-isotropic SOPDO as the regularization operator for image restoration through least squares restoration as given in section 2.2. In order to compare the performance of the non-isotropic SOPDO prior model, we take two regularization models, FOPDO and a sparse stabilizing functional defined in Levin et al. (2007). Relating regularization to probability, the stabilizing functional in image restoration is also referred as prior model and

Restoration Technique Efficiency in seconds

Regularized Image Restoration 125

Restoration Image size Kernel size Efficiency Technique in pixels in pixels in seconds

Levin Sparse deconv. <sup>484</sup> <sup>×</sup> <sup>752</sup> <sup>19</sup> <sup>×</sup> <sup>27</sup> <sup>576</sup> (50 iterations) <sup>99</sup> <sup>×</sup> <sup>99</sup> <sup>556</sup>

Non-isotropic SOPDO <sup>484</sup> <sup>×</sup> <sup>752</sup> <sup>19</sup> <sup>×</sup> <sup>27</sup> 2.34 regularization <sup>99</sup> <sup>×</sup> <sup>99</sup> 2.44

non-isotropic SOPDO regularization model results in the best speed performance when

Further, we tested for the robustness of non-isotropic SOPDO regularization by using different sized images with varying sized kernels. The detailed results are shown in Table 3. All the images used for this experiment are color images, having separate *rgb* (red, green, blue) channels. The image and kernel sizes are given in pixels and the efficiency was measured in seconds. The results clearly show the robustness and the efficiency of the non-isotropic

Several computational experiments were carried out in order to compare non-isotropic SOPDO regularization with "Levin Sparse deconvolution" and "Shan executable". The performance of these deconvolution techniques on a naturally blurred, highly textured image, given in Shan et al. (2008), are shown in Fig. 1 and Fig. 2. The blur kernel used in this experiment was retrieved through the blind deconvolution package of, Shan et al. (2008) distributed online. Closer visual inspection of the image results show that non-isotropic SOPDO technique best shows the tree branches and leaves while the other techniques have a blurring effect still remaining on the estimated result. This fact is further discussed and

compared with "Levin Sparse deconvolution" and "Shan executable" methods.

SOPDO regularization model with respect to different scales of image and kernel.

evidenced by evaluating the deconvolution in a new perspective in section 3.5.

910 × 903 99 × 99 1240 1107 × 1694 99 × 99 2429

<sup>484</sup> <sup>×</sup> <sup>752</sup> <sup>19</sup> <sup>×</sup> <sup>27</sup> <sup>40</sup>

910 × 903 99 × 99 166 1107 × 1694 99 × 99 Error

910 × 903 99 × 99 5.54 1107 × 1694 99 × 99 13.17

99 × 99 73

Levin Sparse deconvolution (50 iterations) 556 Levin Sparse deconvolution (10 iterations) 124 Shan executable 39 Non-isotropic SOPDO deconvolution 2

Table 2. Efficiency in non-blind image deconvolution

Shan executable

Table 3. Efficiency results on scaling

**2.3.3 Performance in deconvolution**


Table 1. Comparison of stabilizing functional model.

a detailed discussion of the Bayesian interpretation to regularization, including the Sparse prior model, is covered in section 3.2. The deconvolution with non-isotropic SOPDO and FOPDO regularization lead to closed form solutions with highly efficient computation, while the Sparse prior cannot be minimized in closed form, Levin et al. (2007). In all the simulations discussed in this section we use

$$
\lambda\_I = \lambda\_\prime \quad \forall \, r \in 1 \ldots R,\tag{12}
$$

for the quadratic regularization functionals, where the value of *R* depends on the respective model such as *R* = 2 for FOPDO and *R* = 3 for non-isotropic SOPDO model.

We claim, using non-isotropic SOPDO prior gives better results for images which are susceptible to noise over the FOPDO. When comparing the non-isotropic SOPDO regularization with the Sparse prior, we found that the non-isotropic SOPDO regularization outperforms Sparse prior significantly in speed. These results are shown in Table 1.

For the experiment in Table 1, we added "Gaussian" noise to the original "Picasso" image, Shan et al. (2008) with a standard deviation of 0.0001 (relative to the image value range of 0 to 1). The original colored image was first converted to greyscale with the pixel values resulting in the range from 0 to 1 and the original image was considered to be periodic. The term MSE stands for mean square error and for a two dimensional image, MSE is defined as

$$MSE \stackrel{\triangle}{=} \frac{1}{L\_1 L\_2} \sum\_{\ell\_1=1}^{L\_1} \sum\_{\ell\_2=1}^{L\_2} \left( \hat{\mathbf{g}}(\ell\_1, \ell\_2) - \hat{\mathbf{g}}(\ell\_1, \ell\_2) \right)^2 \tag{13}$$

where *<sup>g</sup>* and *<sup>g</sup>* represent GTI and the estimated GTI, respectively while *<sup>L</sup>*<sup>1</sup> and *<sup>L</sup>*<sup>2</sup> represent the size of the image in *x* and *y* directions, respectively. The MSE values in Table 1 are in multiples of 10−<sup>4</sup> while the time is given in seconds. The results show that the non-isotropic SOPDO outperforms FOPDO on MSE and has a significant advantage over Sparse prior on speed performance.

#### **2.3.2 Efficiency in deconvolution**

As SOPDO can use frequency domain deconvolution techniques, it can be implemented highly efficiently than most of the recent non-blind deblurring techniques. The comparison was done with the Sparse deconvolution algorithm in Levin et al. (2007) named as "Levin Sparse deconvolution" and the non blind deconvolution of, Shan et al. (2008) (distributed online) named as "Shan executable". The results in Table 2 support the claim that 6 Will-be-set-by-IN-TECH

thing *MSE* Time *MSE* Time Itera- *MSE* Time weight <sup>×</sup>10−<sup>4</sup> in seconds <sup>×</sup>10−<sup>4</sup> in seconds tions <sup>×</sup>10−<sup>4</sup> in seconds 0.1 4.978 0.43 5.168 0.44 10 8.050 27

0.01 6.043 0.42 7.561 0.44 10 4.289 27

a detailed discussion of the Bayesian interpretation to regularization, including the Sparse prior model, is covered in section 3.2. The deconvolution with non-isotropic SOPDO and FOPDO regularization lead to closed form solutions with highly efficient computation, while the Sparse prior cannot be minimized in closed form, Levin et al. (2007). In all the simulations

for the quadratic regularization functionals, where the value of *R* depends on the respective

We claim, using non-isotropic SOPDO prior gives better results for images which are susceptible to noise over the FOPDO. When comparing the non-isotropic SOPDO regularization with the Sparse prior, we found that the non-isotropic SOPDO regularization

For the experiment in Table 1, we added "Gaussian" noise to the original "Picasso" image, Shan et al. (2008) with a standard deviation of 0.0001 (relative to the image value range of 0 to 1). The original colored image was first converted to greyscale with the pixel values resulting in the range from 0 to 1 and the original image was considered to be periodic. The term MSE stands for mean square error and for a two dimensional image, MSE is defined as

model such as *R* = 2 for FOPDO and *R* = 3 for non-isotropic SOPDO model.

*MSE* <sup>1</sup>

*L*1*L*<sup>2</sup>

outperforms Sparse prior significantly in speed. These results are shown in Table 1.

*L*1 ∑ <sup>1</sup>=1

*L*2 ∑ <sup>2</sup>=1 

where *<sup>g</sup>* and *<sup>g</sup>* represent GTI and the estimated GTI, respectively while *<sup>L</sup>*<sup>1</sup> and *<sup>L</sup>*<sup>2</sup> represent the size of the image in *x* and *y* directions, respectively. The MSE values in Table 1 are in multiples of 10−<sup>4</sup> while the time is given in seconds. The results show that the non-isotropic SOPDO outperforms FOPDO on MSE and has a significant advantage over Sparse prior on

As SOPDO can use frequency domain deconvolution techniques, it can be implemented highly efficiently than most of the recent non-blind deblurring techniques. The comparison was done with the Sparse deconvolution algorithm in Levin et al. (2007) named as "Levin Sparse deconvolution" and the non blind deconvolution of, Shan et al. (2008) (distributed online) named as "Shan executable". The results in Table 2 support the claim that

*<sup>g</sup>*(1, <sup>2</sup>) <sup>−</sup> *<sup>g</sup>*(1, <sup>2</sup>)

4.646 0.42 4.730 0.44 10 6.899 27

50 6.153 117 100 6.128 231

<sup>2</sup> (13)

*λ<sup>r</sup>* = *λ*, ∀ *r* ∈ 1... *R*, (12)

Smoo- Non-isotropic SOPDO FOPDO Sparse

0.05

speed performance.

**2.3.2 Efficiency in deconvolution**

discussed in this section we use

Table 1. Comparison of stabilizing functional model.


Table 2. Efficiency in non-blind image deconvolution


Table 3. Efficiency results on scaling

non-isotropic SOPDO regularization model results in the best speed performance when compared with "Levin Sparse deconvolution" and "Shan executable" methods.

Further, we tested for the robustness of non-isotropic SOPDO regularization by using different sized images with varying sized kernels. The detailed results are shown in Table 3. All the images used for this experiment are color images, having separate *rgb* (red, green, blue) channels. The image and kernel sizes are given in pixels and the efficiency was measured in seconds. The results clearly show the robustness and the efficiency of the non-isotropic SOPDO regularization model with respect to different scales of image and kernel.

## **2.3.3 Performance in deconvolution**

Several computational experiments were carried out in order to compare non-isotropic SOPDO regularization with "Levin Sparse deconvolution" and "Shan executable". The performance of these deconvolution techniques on a naturally blurred, highly textured image, given in Shan et al. (2008), are shown in Fig. 1 and Fig. 2. The blur kernel used in this experiment was retrieved through the blind deconvolution package of, Shan et al. (2008) distributed online. Closer visual inspection of the image results show that non-isotropic SOPDO technique best shows the tree branches and leaves while the other techniques have a blurring effect still remaining on the estimated result. This fact is further discussed and evidenced by evaluating the deconvolution in a new perspective in section 3.5.

(a) 27 × 27 Blur Kernel scaled upwards (b) Blurred Image

(a) Shan executable (b) Levin Sparse deconvolution – 10 iterations

Regularized Image Restoration 127

(d) non-isotropic SOPDO

*p*(*g*|*b*). (15)

. (16)

regularization

*p*(*g*|*b*) 

(ML) estimate may be viewed as a special case of MAP where no prior distribution is

Considering the non-blind image deconvolution process, we convert (15) to an energy

Different likelihood and prior models on the ground truth have been applied for image restoration in literature. An analysis of existing prior models can be found in Mignotte (2006).

*<sup>g</sup>* <sup>=</sup> arg max *g*

*<sup>E</sup>*(*g*) <sup>−</sup> log

(c) Levin Sparse deconvolution –

Under the MAP technique, estimation of the GTI simplifies to

minimization problem, where the energy is defined as

50 iterations

used, Hunt (1977).

Fig. 2. Image results for a highly textured image

Fig. 1. Image results for a highly textured image

## **3. Comparison of sparse prior models to quadratic regularization**

#### **3.1 Key issues**

While the development of regularized solutions for ill-posed problems is widely discussed in the signal processing literature, recently by looking at the ill-posed image restoration problem from a probabilistic view point, some researchers claim that the Sparse prior model, Fergus et al. (2006); Levin et al. (2007) (discussed in more detail below in section 3.2.1) outperforms quadratic regularization models (discussed in section 2.2). The analytical study in this section addresses the following problems:


#### **3.2 Regularization – Bayesian interpretation**

Inverse problems such as image restoration are seen as probabilistic inference problems, where lack of information is compensated by assumptions. Therefore, it is not surprising, when the nature of the regularization detailed above is taken into consideration, to see that there is a close relationship between regularization and Bayesian estimation. Applying Bayes theorem to the image restoration problem in (1), for a known blur kernel, the posterior distribution can be written as

$$p(\mathbf{g}|b) \propto p(b|\mathbf{g}) \, p(\mathbf{g}) \, \tag{14}$$

where *p*(*b*|*g*) represents the likelihood and *p*(*g*) represents the prior for the ground truth image. The estimation of the GTI based on posterior distribution can be classified in several ways. The minimum mean-square error estimate represents the mean of the posterior density, the MAP estimate stands for the mode of the posterior density while the maximum likelihood 8 Will-be-set-by-IN-TECH

(a) 27 × 27 Blur Kernel scaled upwards (b) Blurred Image

While the development of regularized solutions for ill-posed problems is widely discussed in the signal processing literature, recently by looking at the ill-posed image restoration problem from a probabilistic view point, some researchers claim that the Sparse prior model, Fergus et al. (2006); Levin et al. (2007) (discussed in more detail below in section 3.2.1) outperforms quadratic regularization models (discussed in section 2.2). The analytical study in this section

Inverse problems such as image restoration are seen as probabilistic inference problems, where lack of information is compensated by assumptions. Therefore, it is not surprising, when the nature of the regularization detailed above is taken into consideration, to see that there is a close relationship between regularization and Bayesian estimation. Applying Bayes theorem to the image restoration problem in (1), for a known blur kernel, the posterior

where *p*(*b*|*g*) represents the likelihood and *p*(*g*) represents the prior for the ground truth image. The estimation of the GTI based on posterior distribution can be classified in several ways. The minimum mean-square error estimate represents the mean of the posterior density, the MAP estimate stands for the mode of the posterior density while the maximum likelihood

*p*(*g*|*b*) ∝ *p*(*b*|*g*) *p*(*g*), (14)

**3. Comparison of sparse prior models to quadratic regularization**

1. Are sparse prior models superior to quadratic regularization models? 2. What is the source of better performance of sparse prior models?

3. Are fast quadratic regularization models good enough for image restoration?

Fig. 1. Image results for a highly textured image

addresses the following problems:

distribution can be written as

**3.2 Regularization – Bayesian interpretation**

**3.1 Key issues**

(a) Shan executable (b) Levin Sparse deconvolution – 10 iterations

(c) Levin Sparse deconvolution – 50 iterations (d) non-isotropic SOPDO regularization

Fig. 2. Image results for a highly textured image

(ML) estimate may be viewed as a special case of MAP where no prior distribution is used, Hunt (1977).

Under the MAP technique, estimation of the GTI simplifies to

$$
\widehat{\mathbf{g}} = \underset{\mathbf{g}}{\text{arg}\,\mathbf{max}} \, p(\mathbf{g}|\boldsymbol{b}).\tag{15}
$$

Considering the non-blind image deconvolution process, we convert (15) to an energy minimization problem, where the energy is defined as

$$E(\mathbf{g}) \stackrel{\Delta}{=} -\log\left(p(\mathbf{g}|\mathbf{b})\right). \tag{16}$$

Different likelihood and prior models on the ground truth have been applied for image restoration in literature. An analysis of existing prior models can be found in Mignotte (2006).

Recently, much attention has been paid in solving *L*<sup>1</sup> norm regularization problems through compressed sensing. in Kim et al. (2007), an efficient method for optimizing a solution to a

Regularized Image Restoration 129

When *α* = 2, minimization in (17) is called the Gaussian prior deconvolution in Levin et al. (2007) and is equivalent to the quadratic regularization problem in (9). Thus, in this chapter, we use the terms Gaussian prior and quadratic (specifically isotropic SOPDO) regularization

For all the restoration performance analysis and comparisons in this paper, we use a recently developed visual metric called SSIM (Structured SIMilarity) index, Wang et al. (2004), which has not been used for the comparison of prior models in the image restoration literature to date. The approach of SSIM is motivated by the highly structured characteristics of the natural image, where the strong neighborhood dependencies carry important information about the

Assuming *x* and *y* are local image patches representing the same patch in the original and estimated images, the local SSIM index measures the similarities of three elements of the image patches: the similarity (*x*, *y*) of the local patch luminances (brightness values), the similarity *c*(*x*, *y*) of the local patch contrasts, and the similarity *s*(*x*, *y*) of the local patch structures. These local similarities are expressed using simple, easily computed statistics, and

problem similar to (20) was discussed when D*<sup>r</sup>* are invertible.

structures of the objects in the visual scene, Wang & Bovik (2009).

combined together to form local SSIM, *S*(*x*, *y*), Wang & Bovik (2009).

*S*(*x*, *y*) = (*x*, *y*) · *c*(*x*, *y*) · *s*(*x*, *y*) <sup>=</sup> <sup>2</sup>*μxμ<sup>y</sup>* <sup>+</sup> *<sup>C</sup>*<sup>1</sup> *μ*2 *<sup>x</sup>* + *μ*<sup>2</sup>

*<sup>y</sup>* + *C*<sup>1</sup>

 · <sup>2</sup>*σxσ<sup>y</sup>* <sup>+</sup> *<sup>C</sup>*<sup>2</sup>

*<sup>y</sup>* + *C*<sup>2</sup>

 ·  *<sup>σ</sup>xy* <sup>+</sup> *<sup>C</sup>*<sup>3</sup> *σxσ<sup>y</sup>* + *C*<sup>3</sup>

, (21)

*σ*2 *<sup>x</sup>* + *σ*<sup>2</sup>

where *μ<sup>x</sup>* and *μ<sup>y</sup>* are the local sample means of *x* and *y*, respectively, *σ<sup>x</sup>* and *σ<sup>y</sup>* are the local sample standard deviations of *x* and *y*, respectively, and *σxy* is the sample cross correlation of *x* and *y* after removing their means. The items *C*1, *C*2, and *C*<sup>3</sup> are small positive constants that stabilize each term, so that near zero sample means, variances or correlations do not lead to

Due to the fact that the underlying principle of SSIM is to extract the structural information which complies with the human visual system, SSIM maps are asserted to be a better signal fidelity measurement over MSE, Wang & Bovik (2009). In evaluating images through MSE, all image pixels are treated equally and content dependent variations in image fidelity are not accounted for. The two main indicators in SSIM evaluations, mean SSIM (MSSIM) and SSIM maps have values in the range from 0 to 1, where 1 indicates the best restoration. Although MSSIM and SSIM maps are generally used as visual fidelity metrics, we evaluate image restoration with the histogram of the SSIM map, as it provides an accurate view of the

**3.2.3 Gaussian prior model**

**3.3 Image restoration evaluation 3.3.1 Visual metric for evaluation**

interchangeably.

numerical instability.

local restoration.

Considering the fact that for a given *g*, the variation in *b* is due to the noise *n*, Hunt (1977), together with the above definitions, non-blind image restoration problem can be recast as seeking the unknown GTI, *g*(*i*, *j*), that minimizes the functional

$$\left\|\left\|\boldsymbol{b}-\mathcal{K}\boldsymbol{g}\right\|\right\|^2 + \sum\_{r=1}^{R} \sum\_{i\boldsymbol{j}} \lambda\_r \,\rho\left(\mathcal{D}\_r\boldsymbol{g}\left(\boldsymbol{i},\boldsymbol{j}\right)\right),\tag{17}$$

where D*<sup>r</sup>* is the *r*th of *R* linear operators, *i*, *j* are pixel indices, *λ<sup>r</sup>* > 0 are the regularization parameters, · stands for the Frobenius norm and *ρ*(·) is a scalar memoryless nonlinear mapping, generally taking the form

$$\rho(z) \stackrel{\Delta}{=} |z|^{\alpha} \tag{18}$$

for judicious choice of real parameter *α* (not necessarily integer).

Many techniques belong to this class and differ only in: the set of linear operators D*r*, *r* = 1, 2, . . . , *R*, and the nonlinear mapping *ρ*(*z*) (or choice of *α*). Numerous image restoration techniques have been developed under this framework from the early work, Geman & Geman (1984); Greig et al. (1989) to the most recent research, Fergus et al. (2006); Levin et al. (2007; 2009); Shan et al. (2008).

#### **3.2.1 Sparse prior model**

In recent literature, it is shown that, when derivative filters are applied to natural images, the filter outputs tend to be sparse, Olshausen et al. (1996); Simoncelli (1997). That is, the histogram of the derivative filtered image peaks at zero and falls off much faster than a Gaussian distribution. These heavy tailed natural image priors are used in a number of applications in image processing literature, such as denoising, Roth & Black (2005); Simoncelli (1999), reflection separation, Levin & Weiss (2007); Weiss (2001) and deconvolution, Levin (2007); Shan et al. (2008) in which, they are implemented in various ways such as student-*t* distributions, Roth & Black (2005) and scale mixtures of Gaussian distributions, Fergus et al. (2006); Portilla et al. (2003).

In Levin et al. (2007), sparsity is incorporated by having D*<sup>r</sup>* as the derivative filters and *α* = 0.8 in (18) as the prior term, which results in

$$\left\|\left\|\boldsymbol{b}-\mathcal{K}\boldsymbol{g}\right\|\right\|^2 + \sum\_{r=1}^{R} \sum\_{i\boldsymbol{j}} \lambda\_r \left(\mathcal{D}\_r \boldsymbol{g}(\boldsymbol{i}, \boldsymbol{j})\right)^{0.8} . \tag{19}$$

This can be solved in the spatial domain using the Conjugate Gradient algorithm, Barrett et al. (1994).

#### **3.2.2 Laplacian prior model**

Although not as close as the Sparse prior to the natural image priors, Laplacian prior with *α* = 1 in (18) is expected to result in a less smooth solution than the Gaussian prior. With the Laplacian prior, the optimization becomes

$$\left\|\left\|\boldsymbol{\boldsymbol{b}}-\boldsymbol{\mathcal{K}}\boldsymbol{g}\right\|\right\|^{2}+\sum\_{r=1}^{R}\lambda\_{r}\left\|\left\|\mathcal{D}\_{r}\boldsymbol{g}\right\|\right\|\_{1}.\tag{20}$$

10 Will-be-set-by-IN-TECH

Considering the fact that for a given *g*, the variation in *b* is due to the noise *n*, Hunt (1977), together with the above definitions, non-blind image restoration problem can be recast as

where D*<sup>r</sup>* is the *r*th of *R* linear operators, *i*, *j* are pixel indices, *λ<sup>r</sup>* > 0 are the regularization

*ρ*(*z*) |*z*|

Many techniques belong to this class and differ only in: the set of linear operators D*r*, *r* = 1, 2, . . . , *R*, and the nonlinear mapping *ρ*(*z*) (or choice of *α*). Numerous image restoration techniques have been developed under this framework from the early work, Geman & Geman (1984); Greig et al. (1989) to the most recent research, Fergus et al. (2006); Levin et al. (2007;

In recent literature, it is shown that, when derivative filters are applied to natural images, the filter outputs tend to be sparse, Olshausen et al. (1996); Simoncelli (1997). That is, the histogram of the derivative filtered image peaks at zero and falls off much faster than a Gaussian distribution. These heavy tailed natural image priors are used in a number of applications in image processing literature, such as denoising, Roth & Black (2005); Simoncelli (1999), reflection separation, Levin & Weiss (2007); Weiss (2001) and deconvolution, Levin (2007); Shan et al. (2008) in which, they are implemented in various ways such as student-*t* distributions, Roth & Black (2005) and scale mixtures of Gaussian distributions, Fergus et al.

In Levin et al. (2007), sparsity is incorporated by having D*<sup>r</sup>* as the derivative filters and *α* = 0.8

This can be solved in the spatial domain using the Conjugate Gradient algorithm, Barrett et al.

Although not as close as the Sparse prior to the natural image priors, Laplacian prior with *α* = 1 in (18) is expected to result in a less smooth solution than the Gaussian prior. With the

> <sup>2</sup> + *R* ∑ *r*=1

D*rg*(*i*, *j*)

D*rg*(*i*, *j*)

stands for the Frobenius norm and *ρ*(·) is a scalar memoryless nonlinear

, (17)

*<sup>α</sup>* (18)

0.8. (19)

*λr*�D*rg*�1. (20)

seeking the unknown GTI, *g*(*i*, *j*), that minimizes the functional

for judicious choice of real parameter *α* (not necessarily integer).

 <sup>2</sup> + *R* ∑ *r*=1 ∑ *ij λr ρ* 

 *b* − K*g*

parameters,

 · 

2009); Shan et al. (2008).

**3.2.1 Sparse prior model**

(2006); Portilla et al. (2003).

**3.2.2 Laplacian prior model**

(1994).

in (18) as the prior term, which results in

Laplacian prior, the optimization becomes

 *b* − K*g*

> *b* − K*g*

 <sup>2</sup> + *R* ∑ *r*=1 ∑ *ij λr* 

mapping, generally taking the form

Recently, much attention has been paid in solving *L*<sup>1</sup> norm regularization problems through compressed sensing. in Kim et al. (2007), an efficient method for optimizing a solution to a problem similar to (20) was discussed when D*<sup>r</sup>* are invertible.

## **3.2.3 Gaussian prior model**

When *α* = 2, minimization in (17) is called the Gaussian prior deconvolution in Levin et al. (2007) and is equivalent to the quadratic regularization problem in (9). Thus, in this chapter, we use the terms Gaussian prior and quadratic (specifically isotropic SOPDO) regularization interchangeably.

## **3.3 Image restoration evaluation**

#### **3.3.1 Visual metric for evaluation**

For all the restoration performance analysis and comparisons in this paper, we use a recently developed visual metric called SSIM (Structured SIMilarity) index, Wang et al. (2004), which has not been used for the comparison of prior models in the image restoration literature to date. The approach of SSIM is motivated by the highly structured characteristics of the natural image, where the strong neighborhood dependencies carry important information about the structures of the objects in the visual scene, Wang & Bovik (2009).

Assuming *x* and *y* are local image patches representing the same patch in the original and estimated images, the local SSIM index measures the similarities of three elements of the image patches: the similarity (*x*, *y*) of the local patch luminances (brightness values), the similarity *c*(*x*, *y*) of the local patch contrasts, and the similarity *s*(*x*, *y*) of the local patch structures. These local similarities are expressed using simple, easily computed statistics, and combined together to form local SSIM, *S*(*x*, *y*), Wang & Bovik (2009).

$$\begin{split} S(\mathbf{x}, y) &= \ell(\mathbf{x}, y) \cdot \mathbf{c}(\mathbf{x}, y) \cdot \mathbf{s}(\mathbf{x}, y) \\ &= \left( \frac{2\mu\_{\mathbf{x}}\mu\_{y} + \mathbf{C}\_{1}}{\mu\_{\mathbf{x}}^{2} + \mu\_{\mathbf{y}}^{2} + \mathbf{C}\_{1}} \right) \cdot \left( \frac{2\sigma\_{\mathbf{x}}\sigma\_{\mathbf{y}} + \mathbf{C}\_{2}}{\sigma\_{\mathbf{x}}^{2} + \sigma\_{\mathbf{y}}^{2} + \mathbf{C}\_{2}} \right) \cdot \left( \frac{\sigma\_{\mathbf{x}\mathbf{y}} + \mathbf{C}\_{3}}{\sigma\_{\mathbf{x}}\sigma\_{\mathbf{y}} + \mathbf{C}\_{3}} \right), \end{split} \tag{21}$$

where *μ<sup>x</sup>* and *μ<sup>y</sup>* are the local sample means of *x* and *y*, respectively, *σ<sup>x</sup>* and *σ<sup>y</sup>* are the local sample standard deviations of *x* and *y*, respectively, and *σxy* is the sample cross correlation of *x* and *y* after removing their means. The items *C*1, *C*2, and *C*<sup>3</sup> are small positive constants that stabilize each term, so that near zero sample means, variances or correlations do not lead to numerical instability.

Due to the fact that the underlying principle of SSIM is to extract the structural information which complies with the human visual system, SSIM maps are asserted to be a better signal fidelity measurement over MSE, Wang & Bovik (2009). In evaluating images through MSE, all image pixels are treated equally and content dependent variations in image fidelity are not accounted for. The two main indicators in SSIM evaluations, mean SSIM (MSSIM) and SSIM maps have values in the range from 0 to 1, where 1 indicates the best restoration. Although MSSIM and SSIM maps are generally used as visual fidelity metrics, we evaluate image restoration with the histogram of the SSIM map, as it provides an accurate view of the local restoration.

0.5 0.6 0.7 0.8 0.9 <sup>1</sup> <sup>0</sup>

SSIM map values

Regularization operator *r λ<sup>r</sup>*

Table 4. Choice of regularization parameter (*λ*) values for different quadratic regularization

the evaluation of image restoration should be carried out excluding these artifacts to properly

In order to achieve the objective of studying the performance of different operators in the quadratic regularization as detailed in section 2.2, we carried out some simulations, where we avoided the effect of "Model artifacts" by taking a boundary strip off from the blurred image. In our evaluations, we used FOPDO, SOPDO and FSOPDO models to compare the performance. From this point onwards the term SOPDO refers to isotropic-SOPDO unless

The simulations, for which the results are demonstrated in Fig. 6, are executed in the same environment as the simulation for Fig. 5, but with quadratic regularization models. We evaluated the performance of the regularization models under varying regularization parameter (*λ*) values as discussed in section 2.2. While the choice of parameters representing *λ* for FOPDO, SOPDO and FSOPDO are given in Table 4, the actual values for the respective

While the overall SSIM values for few of the simulation results under varying *λ* values are shown in Table 5, the histogram distribution representing the first line of Table 5 is shown in Fig. 7. Overall, by analyzing these results, we claim that, in the presence of "Process artifacts", a better performance could be achieved with FSOPDO over FOPDO and SOPDO

FOPDO 1, 2 *β*<sup>1</sup> SOPDO 3, 4 *β*<sup>2</sup> FSOPDO 1, 2 *<sup>β</sup>*<sup>1</sup>

3, 4 *β*<sup>2</sup>

2

Fig. 5. Image restoration results with prior models.

operators used in the simulations of Table 5

stated otherwise.

parameters are given in Table 5.

assess the performance of any image restoration method.

**3.4 Performance of quadratic regularization operators**

4

6

Log distribution

8

10

Sparse prior model Laplacian prior model

Gaussian prior model − IRLS processing Gaussian prior model − FFT processing

Regularized Image Restoration 131

12

Fig. 3. Image restoration model, where *g* is the ground truth image, *b* is the distorted image, <sup>K</sup> is the blur operator, <sup>L</sup> is the deblur process and *<sup>g</sup>* is the estimated image.

Fig. 4. Simulated image restoration model.

#### **3.3.2 Image restoration models**

Ignoring the presence of noise in image acquisition represented by (1), general image restoration could be represented by the model shown in Fig. 3, where L represents the deblur process. The notation in (1) and the representation in Fig. 3 may be an over-simplification. From physical intuition, we could see that even though *g* is continuous by nature, image recording imposes limitations on the spatial extent of *g* and *b*, leading to artifacts which impact on the final estimate of image restoration.

As illustrated in Fig. 4, we categorize these spatial artifacts in two ways. The "Model artifacts" are those, which are not present on naturally blurred images, but introduced in blur simulations as a result of sharp intensity transitions at the boundary of a finite image. Generating a blur image from a finite GTI causes unnatural blur distortions in the vicinity of the boundary of the image. Suppression of these "Model artifacts" could be accomplished by preprocessing the observed degraded image with techniques such as truncation and reducing the size of the blurred image. On the other hand, "Process artifacts" come along with the deblur process L due to finite *b*, which affect the performance of most deconvolution algorithms.

In order to show the effect of "Process artifacts", we restore an image, originally, of size 255 × 255 pixels, but truncated in order to remove the "Model artifacts" introduced by a 13 × 13 pixels blur kernel, making the final image of size 242 × 242. The results of restoration with Sparse, Laplacian and Gaussian priors are shown in Fig. 5. In this experiment, deconvolution with Sparse and Laplacian priors were carried out using iterative re-weighted least squares (IRLS) method, Meer (2004), through the code available online, Levin et al. (2007), while the Gaussian prior is processed with both IRLS and fast Fourier techniques (FFT) separately. In our simulations, we processed IRLS for 150 iterations beyond which there were no further improvements. Analyzing the results of the performance of the Gaussian prior model with the FFT and IRLS techniques, we see that process artifacts are better handled by the IRLS technique than the FFT and this result is justified by the IRLS processing of Sparse and Laplacian prior models.

Both the "Model" and "Process" artifacts discussed above are not part of natural images, but are imposed artificially by the image modeling and processing techniques. Thus we claim that 12 Will-be-set-by-IN-TECH

<sup>g</sup> <sup>K</sup> <sup>b</sup> <sup>L</sup> <sup>g</sup><sup>ˆ</sup>

Fig. 3. Image restoration model, where *g* is the ground truth image, *b* is the distorted image,

<sup>g</sup> <sup>K</sup> <sup>+</sup> <sup>b</sup> <sup>L</sup> + ˆ<sup>g</sup>

Ignoring the presence of noise in image acquisition represented by (1), general image restoration could be represented by the model shown in Fig. 3, where L represents the deblur process. The notation in (1) and the representation in Fig. 3 may be an over-simplification. From physical intuition, we could see that even though *g* is continuous by nature, image recording imposes limitations on the spatial extent of *g* and *b*, leading to artifacts which impact

As illustrated in Fig. 4, we categorize these spatial artifacts in two ways. The "Model artifacts" are those, which are not present on naturally blurred images, but introduced in blur simulations as a result of sharp intensity transitions at the boundary of a finite image. Generating a blur image from a finite GTI causes unnatural blur distortions in the vicinity of the boundary of the image. Suppression of these "Model artifacts" could be accomplished by preprocessing the observed degraded image with techniques such as truncation and reducing the size of the blurred image. On the other hand, "Process artifacts" come along with the deblur process L due to finite *b*, which affect the performance of most deconvolution

In order to show the effect of "Process artifacts", we restore an image, originally, of size 255 × 255 pixels, but truncated in order to remove the "Model artifacts" introduced by a 13 × 13 pixels blur kernel, making the final image of size 242 × 242. The results of restoration with Sparse, Laplacian and Gaussian priors are shown in Fig. 5. In this experiment, deconvolution with Sparse and Laplacian priors were carried out using iterative re-weighted least squares (IRLS) method, Meer (2004), through the code available online, Levin et al. (2007), while the Gaussian prior is processed with both IRLS and fast Fourier techniques (FFT) separately. In our simulations, we processed IRLS for 150 iterations beyond which there were no further improvements. Analyzing the results of the performance of the Gaussian prior model with the FFT and IRLS techniques, we see that process artifacts are better handled by the IRLS technique than the FFT and this result is justified by the IRLS processing of Sparse

Both the "Model" and "Process" artifacts discussed above are not part of natural images, but are imposed artificially by the image modeling and processing techniques. Thus we claim that

Model artifacts Process artifacts

<sup>K</sup> is the blur operator, <sup>L</sup> is the deblur process and *<sup>g</sup>* is the estimated image.

Fig. 4. Simulated image restoration model.

on the final estimate of image restoration.

**3.3.2 Image restoration models**

algorithms.

and Laplacian prior models.

Fig. 5. Image restoration results with prior models.


Table 4. Choice of regularization parameter (*λ*) values for different quadratic regularization operators used in the simulations of Table 5

the evaluation of image restoration should be carried out excluding these artifacts to properly assess the performance of any image restoration method.

## **3.4 Performance of quadratic regularization operators**

In order to achieve the objective of studying the performance of different operators in the quadratic regularization as detailed in section 2.2, we carried out some simulations, where we avoided the effect of "Model artifacts" by taking a boundary strip off from the blurred image. In our evaluations, we used FOPDO, SOPDO and FSOPDO models to compare the performance. From this point onwards the term SOPDO refers to isotropic-SOPDO unless stated otherwise.

The simulations, for which the results are demonstrated in Fig. 6, are executed in the same environment as the simulation for Fig. 5, but with quadratic regularization models. We evaluated the performance of the regularization models under varying regularization parameter (*λ*) values as discussed in section 2.2. While the choice of parameters representing *λ* for FOPDO, SOPDO and FSOPDO are given in Table 4, the actual values for the respective parameters are given in Table 5.

While the overall SSIM values for few of the simulation results under varying *λ* values are shown in Table 5, the histogram distribution representing the first line of Table 5 is shown in Fig. 7. Overall, by analyzing these results, we claim that, in the presence of "Process artifacts", a better performance could be achieved with FSOPDO over FOPDO and SOPDO

<sup>g</sup> <sup>K</sup> <sup>b</sup> <sup>L</sup> <sup>z</sup> <sup>P</sup><sup>M</sup> (z) ˆ<sup>g</sup>

Regularized Image Restoration 133

Fig. 8. Image restoration model for a naturally blurred image, where K is the blur process, L is the deblur process, *<sup>g</sup>*, *<sup>b</sup>*, *<sup>z</sup>*, *<sup>g</sup>* stand for GTI, blur image, deblurred image with artifacts, and the final estimated GTI respectively. The process *PM*(*z*) decouples "Model" and "Process"

These models form a method of regularization in image restoration. This section is devoted for the comparison of these models. The comparison in this section will guide us for making recommendations for the appropriate regularization technique and is discussed at the end of

As the objective of our simulations is to evaluate the contribution of the regularization models towards image restoration, we use the restoration model shown in Fig. 8, where we decouple

To be consistent with the SSIM map region in Fig. 5, we take a large image of support 1024 × 1024 and project the final image to a 242 × 242 region within the inner region of the estimated image, which is least affected by the artifacts. The restoration was carried out with FFT processing of the Gaussian prior and IRLS processing of Sparse and Laplacian priors. The comparison of the performance of the priors is shown in Fig. 9. In it we note that Gaussian prior with FFT processing has performed as well as or better to the Sparse and Laplacian prior

As the literature claims that iterative algorithms such as conjugate gradient algorithms suppress noise and perform better in noisy blur image restoration, we simulated a noisy blurred restoration under the same conditions given for Fig. 9, but with different regularization parameter values, as more weight should now be given to the prior over data. The noise added was Gaussian with zero mean and 0.01 variance. The optimal results we obtained for varying *λ* are shown in Fig. 10. With these results, we claim that Gaussian prior

Thus, these results pave a new path of thinking and we claim that quadratic regularization with SOPDO model, when appropriately configured and used in a realistic context, free from unnatural artifacts, is comparable to Sparse prior model in terms of image restoration

As the optimization problem in least-squares regularization is convex and as the fast Fourier techniques could be applied for the computation, for an image of size *L* × *L* pixels, the restoration through least-squares regulation has a complexity of *O*(*L* log *L*) operations. In

*z*(*i*, *j*), if *i*, *j* ∈ *M*

0, otherwise (22)

artifact effects from restoration by projecting the estimated image with

where *M* is a region without "Model" and "Process" artifacts.

*PM*(*z*)(*i*, *<sup>j</sup>*) =

handles noisy images as better as the Sparse and Laplacian prior models.

artifacts from the deblurred image .

**3.5.1 SSIM performance comparison**

performance under the SSIM criterion.

**3.5.2 Efficiency comparison**

this section.

models.

Fig. 6. Image restoration results with quadratic regularization models.


Table 5. Performance of quadratic regularization operators under varying regularization parameter values

Fig. 7. Image restoration results for simulations in Table 5

models. In the next section we compare the performance of these quadratic regularization models by removing the "Process" and "Model" artifacts.

#### **3.5 Regularization model performance comparison**

As shown earlier in section 2.2, we modeled the regularization of image restoration based on the quadratic regularization terms (sometimes called as the least squares regularization) and in section 3.2, we discussed the existing probabilistic models under a MAP framework. 14 Will-be-set-by-IN-TECH

First Order Quadratic Model Second Order Quadratic Model First & Second Order Quadratic Model

0.965 0.97 0.975 0.98 0.985 0.99 0.995 <sup>1</sup> 1.005 <sup>0</sup>

SSIM map values

Regularization parameter FOPDO SOPDO FSOPDO *β*<sup>1</sup> = *β*<sup>2</sup> = 0.001 0.9412 0.9597 0.9626 *β*<sup>1</sup> = 0.001, *β*<sup>2</sup> = 0.003 0.9596 0.9597 0.9657 *β*<sup>1</sup> = 0.003, *β*<sup>2</sup> = 0.001 0.9412 0.9674 0.968

Table 5. Performance of quadratic regularization operators under varying regularization

Log distribution

models. In the next section we compare the performance of these quadratic regularization

As shown earlier in section 2.2, we modeled the regularization of image restoration based on the quadratic regularization terms (sometimes called as the least squares regularization) and in section 3.2, we discussed the existing probabilistic models under a MAP framework.

MSSIM values for

First Order Quadratic Model Second Order Quadratic Model First & Second Order Quadratic Model

0.93 0.94 0.95 0.96 0.97 0.98 0.99 <sup>1</sup> <sup>0</sup>

(b) Result for row 3

SSIM map values

2

0.93 0.94 0.95 0.96 0.97 0.98 0.99 <sup>1</sup> <sup>0</sup>

(a) Result for row 2

First Order Quadratic Model Second Order Quadratic Model First & Second Order Quadratic Model

SSIM map values

Fig. 7. Image restoration results for simulations in Table 5

models by removing the "Process" and "Model" artifacts.

**3.5 Regularization model performance comparison**

Fig. 6. Image restoration results with quadratic regularization models.

4

6

Log distribution

parameter values

Log distribution

8

10

12

<sup>g</sup> <sup>K</sup> <sup>b</sup> <sup>L</sup> <sup>z</sup> <sup>P</sup><sup>M</sup> (z) ˆ<sup>g</sup>

Fig. 8. Image restoration model for a naturally blurred image, where K is the blur process, L is the deblur process, *<sup>g</sup>*, *<sup>b</sup>*, *<sup>z</sup>*, *<sup>g</sup>* stand for GTI, blur image, deblurred image with artifacts, and the final estimated GTI respectively. The process *PM*(*z*) decouples "Model" and "Process" artifacts from the deblurred image .

These models form a method of regularization in image restoration. This section is devoted for the comparison of these models. The comparison in this section will guide us for making recommendations for the appropriate regularization technique and is discussed at the end of this section.

#### **3.5.1 SSIM performance comparison**

As the objective of our simulations is to evaluate the contribution of the regularization models towards image restoration, we use the restoration model shown in Fig. 8, where we decouple artifact effects from restoration by projecting the estimated image with

$$P\_M(z)(i,j) = \begin{cases} z(i,j), & \text{if } i, j \in M \\ 0, & \text{otherwise} \end{cases} \tag{22}$$

where *M* is a region without "Model" and "Process" artifacts.

To be consistent with the SSIM map region in Fig. 5, we take a large image of support 1024 × 1024 and project the final image to a 242 × 242 region within the inner region of the estimated image, which is least affected by the artifacts. The restoration was carried out with FFT processing of the Gaussian prior and IRLS processing of Sparse and Laplacian priors. The comparison of the performance of the priors is shown in Fig. 9. In it we note that Gaussian prior with FFT processing has performed as well as or better to the Sparse and Laplacian prior models.

As the literature claims that iterative algorithms such as conjugate gradient algorithms suppress noise and perform better in noisy blur image restoration, we simulated a noisy blurred restoration under the same conditions given for Fig. 9, but with different regularization parameter values, as more weight should now be given to the prior over data. The noise added was Gaussian with zero mean and 0.01 variance. The optimal results we obtained for varying *λ* are shown in Fig. 10. With these results, we claim that Gaussian prior handles noisy images as better as the Sparse and Laplacian prior models.

Thus, these results pave a new path of thinking and we claim that quadratic regularization with SOPDO model, when appropriately configured and used in a realistic context, free from unnatural artifacts, is comparable to Sparse prior model in terms of image restoration performance under the SSIM criterion.

#### **3.5.2 Efficiency comparison**

As the optimization problem in least-squares regularization is convex and as the fast Fourier techniques could be applied for the computation, for an image of size *L* × *L* pixels, the restoration through least-squares regulation has a complexity of *O*(*L* log *L*) operations. In

Image size Time for Quadratic Time for Sparse prior in pixels regularization with FFT 50 iterations 100 iterations

Regularized Image Restoration 135

grey-scale 255 × 255 0.08 30 60 grey-scale 1024 × 1024 1.1 303 596 colored 484 × 752 1.7 371 730 colored 1107 × 1694 9.0 2180 4292 Table 6. Efficiency of regularization operators. The times taken for restoration of grey-scale

In addition to lower efficiency and not-superior performance, Sparse prior models lack in proper theoretical guidelines for selecting the best regularization parameter. In contrast, the quadratic regularization models can use well-established methods such as L-curve criterion, Hansen (1998) and the Generalized Cross Validation criterion, Hansen (1998) for choosing the value of *λ*. Difficulties in selecting the optimal converging point in non-convex

According to the theoretical and experimental details provided above, we propose that if we could decouple image restoration and "Process artifact" handling, then the use of quadratic regularization models will result in more efficient and effective image restoration in comparison to Sparse and Laplacian prior models. The decoupling of image restoration and "Process" artifact handling could be achieved through techniques such as tiling, Liu & Jia

1. Sparse prior models are not superior to quadratic regularization models in terms of

2. In terms of efficiency, Sparse prior models are significantly inferior to quadratic

3. The performance through Sparse prior model increases over quadratic regularization models when boundary effects are not addressed and processing artifacts are not

4. Quadratic regularization models provide the best image restoration for large images in terms of efficiency and effectiveness while they provide a good enough solution for other

Analyzing the above items further, if the improvements of the Sparse prior model are in

"Do more complicated prior models such as Sparse, which are asserted be better matched to natural images, actually help image restoration in terms of restoring natural image

"If those complicated prior models hold no significant advantage, is it worth the effort spend on them compared to simple and efficient prior models which restore closer or better than

artifact handling, not in image restoration, we can pose the following questions.

and colored images are given in seconds for each of the regularization operators.

(2008), which enables the uses of the efficient least squares regularization. Thus, coming back to our problem formulation in section 3.1, we claim that:

**3.5.3 Regularization recommendations**

performance in image restoration.

regularization models.

compensated for.

features?"

those prior models?"

minimization techniques such as IRLS also is an issue.

images when the boundary artifacts are taken care of.

Fig. 9. Image restoration results for the system in Fig. 8 with FFT processing of Gaussian Prior and IRLS processing of Sparse and Laplacian prior models.

Fig. 10. Image restoration results for a noisy image under the same environment in Fig. 9.

contrast, when a Sparse prior is used, the optimization problem is no longer convex and cannot be minimized in closed form. Using the conjugate gradient method, Barrett et al. (1994), or IRLS method, the optimization can be solved in *O*(*L i*max) where *i*max represent the maximum number of iterations.

A few simulation results on efficiency are shown in Table 6, where all the values are in seconds and represent the time taken for the restoration using each of the respective model. While the quadratic regularization deconvolution was carried out using Fourier domain techniques, the Sparse deconvolution was carried out using the IRLS method. Under the IRLS algorithm, it is experienced that in order to achieve an acceptable result, the number of iterations should be at least 50 and better results could be achieved when the number of iterations are above 100. From the results shown, it is evident that when the size of the image increases, the relative efficiency of the restoration through Sparse prior model becomes extremely low.

16 Will-be-set-by-IN-TECH

Gaussian prior model − FFT processing

Sparse prior model Laplacian prior model

0.94 0.95 0.96 0.97 0.98 0.99 <sup>1</sup> <sup>0</sup>

Fig. 9. Image restoration results for the system in Fig. 8 with FFT processing of Gaussian

Gaussian prior model − FFT processing

SSIM map values

<sup>0</sup> 0.2 0.4 0.6 0.8 <sup>1</sup> <sup>0</sup>

SSIM map values

contrast, when a Sparse prior is used, the optimization problem is no longer convex and cannot be minimized in closed form. Using the conjugate gradient method, Barrett et al. (1994), or IRLS method, the optimization can be solved in *O*(*L i*max) where *i*max represent

A few simulation results on efficiency are shown in Table 6, where all the values are in seconds and represent the time taken for the restoration using each of the respective model. While the quadratic regularization deconvolution was carried out using Fourier domain techniques, the Sparse deconvolution was carried out using the IRLS method. Under the IRLS algorithm, it is experienced that in order to achieve an acceptable result, the number of iterations should be at least 50 and better results could be achieved when the number of iterations are above 100. From the results shown, it is evident that when the size of the image increases, the relative

efficiency of the restoration through Sparse prior model becomes extremely low.

Fig. 10. Image restoration results for a noisy image under the same environment in Fig. 9.

2

the maximum number of iterations.

Log distribution

Prior and IRLS processing of Sparse and Laplacian prior models.

Sparse prior model Laplacian prior model

4

6

Log distribution

8

10

12


Table 6. Efficiency of regularization operators. The times taken for restoration of grey-scale and colored images are given in seconds for each of the regularization operators.

## **3.5.3 Regularization recommendations**

In addition to lower efficiency and not-superior performance, Sparse prior models lack in proper theoretical guidelines for selecting the best regularization parameter. In contrast, the quadratic regularization models can use well-established methods such as L-curve criterion, Hansen (1998) and the Generalized Cross Validation criterion, Hansen (1998) for choosing the value of *λ*. Difficulties in selecting the optimal converging point in non-convex minimization techniques such as IRLS also is an issue.

According to the theoretical and experimental details provided above, we propose that if we could decouple image restoration and "Process artifact" handling, then the use of quadratic regularization models will result in more efficient and effective image restoration in comparison to Sparse and Laplacian prior models. The decoupling of image restoration and "Process" artifact handling could be achieved through techniques such as tiling, Liu & Jia (2008), which enables the uses of the efficient least squares regularization.

Thus, coming back to our problem formulation in section 3.1, we claim that:


Analyzing the above items further, if the improvements of the Sparse prior model are in artifact handling, not in image restoration, we can pose the following questions.


*r ∂<sup>r</sup> ∂rn ω*(*∂r*) 1 *∂*<sup>1</sup> *∂xn* 1 2 *∂*<sup>2</sup> *∂yn* 1 3 *∂*<sup>3</sup> *∂xxn* 2 4 *∂*<sup>4</sup> *∂xyn* 2 5 *∂*<sup>5</sup> *∂yyn* 2

Regularized Image Restoration 137

Out of the various likelihood models introduced in the literature of image restoration, we consider two recent approaches in Levin et al. (2007) and, Shan et al. (2008) for our analysis. In Levin et al. (2007), the single image likelihood conforms to (23) and is explicitly given by

<sup>−</sup> <sup>1</sup>

In Shan et al. (2008), the likelihood is defined with different orders of partial derivatives, denoted by operator *∂r*, of a single degraded image. For ease of understanding, we represent

For example, in Shan et al. (2008), the set Θ has the elements {*∂x*, *∂y*, *∂xx*, *∂xy*, *∂yy*}, in which, *∂<sup>x</sup>* is the first order derivative in *x* direction and *∂<sup>y</sup>* is the first order derivative in *y* direction

Further, Shan et al. (2008) shows that the partial derivatives of *n* also follow Normal distributions with standard deviation values based on the order of the partial derivative

where *ω*(*∂r*) represents the order of the partial derivative operator *∂r*. Few example elements

As there was no analysis presented behind using the higher order partial derivatives of noise in Shan et al. (2008) leading to (29), we provide an interpretation of formula, based on our new general likelihood model for a group of degraded images of a common ground truth *g* in (27). Guided by the likelihood expression (27), we can define a virtual group of images for the

2)*ω*(*∂r*)

operator. The standard deviations of the partial derivatives are specified in the form *σω*(*∂r*) = (<sup>√</sup>

of the set Θ in (30) with the respective standard deviation values are given in Table 7.

<sup>2</sup>*σ*<sup>2</sup> �K*g*−*b*�<sup>2</sup>

*∂rn*(1, <sup>2</sup>)|0, *σω*(*∂r*)

Θ {*∂*1, *∂*2, *∂*3,..., *∂R*}. (30)

*br* = *∂rb*, *r* = 1, 2, 3, . . . , *R* (32)

, (28)

*σ*, (31)

, *r* = 1, 2, 3, . . . , *R* (29)

*p*(*b*|*g*) ∝ *e*

 *N*

Table 7. An example of set Θ in (30) with *R* = 5

**4.1.3 Likelihood models for analysis**

where �·� stands for the Frobenius norm.

*L*1 ∏ <sup>1</sup>=1

*L*2 ∏ <sup>2</sup>=1

*N*

where Θ is a set of partial derivative operators given by

and similar interpretations hold for higher order derivatives.

*n*(1, <sup>2</sup>)|0, *σ*

their model in the form

*<sup>p</sup>*(*b*|*g*) = ∏

*∂r*∈Θ

likelihood model in (29) as

### **4. Likelihood model analysis**

Different likelihood models in the prior model in (14) have been studied in various ways. The fact that most of these models are not justified with proper theoretical foundations encouraged us to analyze and understand the variations and the validity and accuracy of the (implicit) underlying assumptions, which could explain the different performances.

This investigation guides our development of a new scheme for the multiple image likelihood model described in section 4.1.2. The likelihood model analysis is carried out using this new model and the theoretical analysis is corroborated by the computational experiments detailed in section 4.3.

#### **4.1 Likelihood models in image restoration**

#### **4.1.1 Likelihood model for a single image**

In image restoration literature, the likelihood for a single image is defined by modeling the image noise (*n*) as a set of independently and identically distributed (i.i.d.) random variables following a Gaussian distribution for all pixels, which is given by

$$p(b|\mathbf{g}) = \prod\_{\ell\_1=1}^{L\_1} \prod\_{\ell\_2=1}^{L\_2} N(n(\ell\_1, \ell\_2)|0, \sigma) \,. \tag{23}$$

where *<sup>N</sup>*(·|*μ*, *<sup>σ</sup>*) denotes a Gaussian distribution with mean *<sup>μ</sup>* and variance *<sup>σ</sup>*2, while *<sup>L</sup>*<sup>1</sup> and *L*<sup>2</sup> represent the image support.

#### **4.1.2 Likelihood model for multiple images**

Based on the above likelihood model for a single image, we develop a new model for the likelihood of multiple images as detailed below.

Given a set of *R* degraded images of a common GTI *g*, the posterior distribution for the GTI can be derived by extending (14), resulting in

$$p(\mathbf{g}|b\_1, b\_2, \dots, b\_R) \propto p(b\_1, b\_2, \dots, b\_R|\mathbf{g}) \, p(\mathbf{g}) \, \tag{24}$$

where, generalizing (1),

$$b\_r = \pounds\_r \emptyset + n\_r \quad r = 1,2,3,\dots,R \tag{25}$$

and K*<sup>r</sup>* are operators representing possibly different but known blurs, and *nr* are noise images. Under the assumption that *np* is independent of *nq* for all *p* �= *q*, the likelihood in (24) is

$$p(b\_1, b\_2, \dots, b\_R | \mathcal{g}) = \prod\_{r=1}^{R} N(n\_r). \tag{26}$$

Thus, for a group of *R* images satisfying the noise independency condition in (26), the likelihood can be modeled as

$$p(b\_1, b\_2, \dots, b\_R | \mathcal{g}) = \prod\_{r=1}^{R} \prod\_{\ell\_1=1}^{L\_1} \prod\_{\ell\_2=1}^{L\_2} N(n\_r(\ell\_1, \ell\_2) | 0, \sigma\_r) \tag{27}$$

where *σ<sup>r</sup>* represent the standard deviation of the Gaussian distribution for *nr*. This new model for the likelihood of multiple images will be used for the analysis of likelihood models in the next section.

18 Will-be-set-by-IN-TECH

Different likelihood models in the prior model in (14) have been studied in various ways. The fact that most of these models are not justified with proper theoretical foundations encouraged us to analyze and understand the variations and the validity and accuracy of the (implicit)

This investigation guides our development of a new scheme for the multiple image likelihood model described in section 4.1.2. The likelihood model analysis is carried out using this new model and the theoretical analysis is corroborated by the computational experiments detailed

In image restoration literature, the likelihood for a single image is defined by modeling the image noise (*n*) as a set of independently and identically distributed (i.i.d.) random variables

*N*

where *<sup>N</sup>*(·|*μ*, *<sup>σ</sup>*) denotes a Gaussian distribution with mean *<sup>μ</sup>* and variance *<sup>σ</sup>*2, while *<sup>L</sup>*<sup>1</sup> and

Based on the above likelihood model for a single image, we develop a new model for the

Given a set of *R* degraded images of a common GTI *g*, the posterior distribution for the GTI

and K*<sup>r</sup>* are operators representing possibly different but known blurs, and *nr* are noise images. Under the assumption that *np* is independent of *nq* for all *p* �= *q*, the likelihood in (24) is

Thus, for a group of *R* images satisfying the noise independency condition in (26), the

*L*1 ∏ <sup>1</sup>=1

where *σ<sup>r</sup>* represent the standard deviation of the Gaussian distribution for *nr*. This new model for the likelihood of multiple images will be used for the analysis of likelihood models in the

*L*2 ∏ <sup>2</sup>=1

*N*

*R* ∏*r*=1

*p*(*b*1, *b*2,..., *bR*|*g*) =

*R* ∏*r*=1

*p*(*b*1, *b*2,..., *bR*|*g*) =

*n*(1, <sup>2</sup>)|0, *σ*

*p*(*g*|*b*1, *b*2,..., *bR*) ∝ *p*(*b*1, *b*2,..., *bR*|*g*) *p*(*g*), (24)

*br* = K*rg* + *nr*, *r* = 1, 2, 3, . . . , *R* (25)

*nr*(1, <sup>2</sup>)|0, *σ<sup>r</sup>*

, (23)

*N*(*nr*). (26)

, (27)

*L*2 ∏ <sup>2</sup>=1

underlying assumptions, which could explain the different performances.

following a Gaussian distribution for all pixels, which is given by

*p*(*b*|*g*) =

*L*1 ∏ <sup>1</sup>=1

**4. Likelihood model analysis**

**4.1 Likelihood models in image restoration 4.1.1 Likelihood model for a single image**

*L*<sup>2</sup> represent the image support.

where, generalizing (1),

likelihood can be modeled as

next section.

**4.1.2 Likelihood model for multiple images**

likelihood of multiple images as detailed below.

can be derived by extending (14), resulting in

in section 4.3.


Table 7. An example of set Θ in (30) with *R* = 5

#### **4.1.3 Likelihood models for analysis**

Out of the various likelihood models introduced in the literature of image restoration, we consider two recent approaches in Levin et al. (2007) and, Shan et al. (2008) for our analysis.

In Levin et al. (2007), the single image likelihood conforms to (23) and is explicitly given by

$$p(b|g) \propto e^{-\frac{1}{2\sigma^2} \|\mathcal{K}g - b\|^2} \, \Big|\, \tag{28}$$

where �·� stands for the Frobenius norm.

In Shan et al. (2008), the likelihood is defined with different orders of partial derivatives, denoted by operator *∂r*, of a single degraded image. For ease of understanding, we represent their model in the form

$$p(b|\mathbf{g}) = \prod\_{\partial\_r \in \Theta} \prod\_{\ell\_1=1}^{L\_1} \prod\_{\ell\_2=1}^{L\_2} N\left( n(\ell\_1, \ell\_2) | 0, \sigma \right) N \left( \partial\_r n(\ell\_1, \ell\_2) | 0, \sigma\_{\omega(\partial\_r)} \right), \quad r = 1, 2, 3, \dots, R \tag{29}$$

where Θ is a set of partial derivative operators given by

$$\Theta \stackrel{\Rightarrow}{=} \{\eth\_1, \eth\_2, \eth\_3, \dots, \eth\_R\}. \tag{30}$$

For example, in Shan et al. (2008), the set Θ has the elements {*∂x*, *∂y*, *∂xx*, *∂xy*, *∂yy*}, in which, *∂<sup>x</sup>* is the first order derivative in *x* direction and *∂<sup>y</sup>* is the first order derivative in *y* direction and similar interpretations hold for higher order derivatives.

Further, Shan et al. (2008) shows that the partial derivatives of *n* also follow Normal distributions with standard deviation values based on the order of the partial derivative operator. The standard deviations of the partial derivatives are specified in the form

$$
\sigma\_{\omega(\partial\_{\mathbf{r}})} = (\sqrt{2})^{\omega(\partial\_{\mathbf{r}})} \sigma\_{\mathbf{r}} \tag{31}
$$

where *ω*(*∂r*) represents the order of the partial derivative operator *∂r*. Few example elements of the set Θ in (30) with the respective standard deviation values are given in Table 7.

As there was no analysis presented behind using the higher order partial derivatives of noise in Shan et al. (2008) leading to (29), we provide an interpretation of formula, based on our new general likelihood model for a group of degraded images of a common ground truth *g* in (27).

Guided by the likelihood expression (27), we can define a virtual group of images for the likelihood model in (29) as

$$b\_r = \partial\_r b, \quad r = 1, 2, 3, \dots, R \tag{32}$$

Minimizing the energy in (38) and solving for estimated *G* denoted as *G* results in

*<sup>n</sup>* <sup>=</sup> *<sup>B</sup> <sup>K</sup>*

Regularized Image Restoration 139

Ψ = F(*∂x*) F(*∂x*) + F(*∂y*) F(*∂y*),

*<sup>n</sup>* is the Fourier transform of the estimated GTI under "normal likelihood", (·) stands for the complex conjugate and the division is performed element-wise. The estimated ground truth

With the above derivations, it is evident that Fourier domain expression used to estimate the

�F(*∂r*) *<sup>G</sup> <sup>K</sup>* − F(*∂r*) *<sup>B</sup>*�<sup>2</sup>

<sup>2</sup>*ω*(*∂r*) <sup>F</sup>(*∂r*) <sup>F</sup>(*∂r*).

where, *∂<sup>r</sup>* is a matrix convolution operator representing a partial order derivative operator

*<sup>d</sup>* <sup>=</sup> *<sup>B</sup> <sup>K</sup>* <sup>Ω</sup>

1

By taking the inverse Fourier transforms of (42), we could get the estimated GTI, *<sup>g</sup><sup>d</sup>*, under

*<sup>K</sup> <sup>K</sup>* <sup>+</sup> *<sup>λ</sup>* <sup>Ψ</sup>, (39)

*<sup>n</sup>*.

<sup>2</sup>*ω*(*∂r*) �*∂r*(*<sup>g</sup>* <sup>∗</sup> *<sup>k</sup>*) <sup>−</sup> *<sup>∂</sup>rb*�<sup>2</sup> <sup>+</sup> *<sup>λ</sup>* <sup>Φ</sup>(*g*). (40)

<sup>+</sup> *<sup>λ</sup>* <sup>F</sup>

*<sup>K</sup> <sup>K</sup>* <sup>Ω</sup> <sup>+</sup> *<sup>λ</sup>* <sup>Ψ</sup>, (42)

Φ(*g*) 

, (41)

*G*

image *gn* can be derived by taking the inverse Fourier transform of *<sup>G</sup>*

We now give an analogous derivation for the "derivative likelihood".

*∂r*∈Θ

1 2*ω*(*∂r*) 

By minimizing the energy (41), we compute the estimated *G*,

*G*

Ω ∑ *∂r*∈Θ

"derivative likelihood" model similar to "normal likelihood" model.

The energy functional in this case is derived similar to (37),

*Ed*(*g*) = ∑

Transforming (40) into the frequency domain results in

*∂r*∈Θ

2. amenable to Fast Fourier Techniques leading to a highly efficient solution.

1

1. simple and leads to a closed form solution and

**4.2.2 Derivative likelihood deconvolution**

F(*Ed*(*g*)) = ∑

and F(*∂r*) denotes its Fourier transformation.

where

GTI is:

where

*G*

and in order to align with model (25), define

$$\mathcal{K}\_r \underline{\mathfrak{g}} \stackrel{\triangle}{=} \partial\_r (\underline{\mathfrak{g}} \* k), \quad r = 1, 2, 3, \dots, R \tag{33}$$

$$n\_r \triangleq \partial\_r n\_r \quad r = 1, 2, 3, \dots, R \tag{34}$$

where ∗ stands for the convolution operator and *k* is the blur kernel.

From this, we infer that the likelihood (29) implicitly assumes *∂pn* is independent of *∂qn* for all *p* � *q*. Since all virtual images are derived from a single degraded image, we can infer this is a strong assumption made to simplify the likelihood expression. In principle, it should be possible to formulate a model without recourse to the derivative images which add limited new information. We corroborate this claim in section 4.3 with experiments.

#### **4.2 Frequency domain deconvolution**

In this section we approach image deconvolution with FOPDO regularization and with different likelihood models discussed above. For our analysis, we consider the likelihood models of (23) and (29) using terminology "normal likelihood" and "derivative likelihood" with the notation using subscripts "*n*" and "*d*", respectively. With our experiments, we limit the set Θ in (30), going up to second order partial derivative operators in (29) and we take elements of Θ from the following values

$$
\Theta = \{\eth\_{\mathbf{x}}, \eth\_{\mathbf{y}}, \eth\_{\mathbf{xx}}, \eth\_{\mathbf{xy}}, \eth\_{\mathbf{yy}}\}.\tag{35}
$$

#### **4.2.1 Normal likelihood deconvolution**

Under FOPDO regularization as detailed in section 2.2, the stabilizing functional Φ(*g*) takes the form

$$\Phi(\mathcal{g}) \stackrel{\Delta}{=} \|\partial\_{\mathbf{x}} \mathcal{g}\|^2 + \|\partial\_{\mathbf{y}} \mathcal{g}\|^2. \tag{36}$$

Applying this stabilizing functional to the MAP framework detailed in section 4, the energy functional under "normal likelihood", can be derived as

$$E\_n(\mathbf{g}) = \|\mathbf{g} \ast k - b\|^2 + \lambda \,\Phi(\mathbf{g}).\tag{37}$$

According to the convolution theorem, the convolution operation in the spatial domain becomes an element-wise product in the frequency domain making F(*g* ∗ *k*) = *G K* where F(·) stands for discrete Fourier transform, *G* for F(*g*), *K* for F(*k*) and "" denotes element-wise product. Based on the above property, transforming (37) into frequency domain and applying Plancherels theorem, Bracewell & Kahn (1966), we derive the energy in the frequency domain for (37) as follows.

$$\mathcal{F}(E\_n(\mathbf{g})) = \|G \star K - B\|^2 + \lambda \mathcal{F}(\Phi(\mathbf{g})),\tag{38}$$

where

$$\mathcal{F}(\Phi(\mathcal{g})) = \|\mathcal{F}(\partial\_{\mathcal{X}}) \star G\|^2 + \|\mathcal{F}(\partial\_{\mathcal{Y}}) \star G\|^2 \lambda$$

*B* stands for F(*b*) and given *∂<sup>x</sup>* takes the form of a (convolution) matrix, then F(*∂x*) denotes its Fourier transform.

Minimizing the energy in (38) and solving for estimated *G* denoted as *G* results in

$$
\widehat{\mathbf{G}\_n} = \frac{B \star \overline{K}}{K \star \overline{K} + \lambda \,\mathbf{Y}'} \,\tag{39}
$$

where

20 Will-be-set-by-IN-TECH

From this, we infer that the likelihood (29) implicitly assumes *∂pn* is independent of *∂qn* for all *p* � *q*. Since all virtual images are derived from a single degraded image, we can infer this is a strong assumption made to simplify the likelihood expression. In principle, it should be possible to formulate a model without recourse to the derivative images which add limited

In this section we approach image deconvolution with FOPDO regularization and with different likelihood models discussed above. For our analysis, we consider the likelihood models of (23) and (29) using terminology "normal likelihood" and "derivative likelihood" with the notation using subscripts "*n*" and "*d*", respectively. With our experiments, we limit the set Θ in (30), going up to second order partial derivative operators in (29) and we take

Under FOPDO regularization as detailed in section 2.2, the stabilizing functional Φ(*g*) takes

Applying this stabilizing functional to the MAP framework detailed in section 4, the energy

According to the convolution theorem, the convolution operation in the spatial domain becomes an element-wise product in the frequency domain making F(*g* ∗ *k*) = *G K* where F(·) stands for discrete Fourier transform, *G* for F(*g*), *K* for F(*k*) and "" denotes element-wise product. Based on the above property, transforming (37) into frequency domain and applying Plancherels theorem, Bracewell & Kahn (1966), we derive the energy in the

<sup>F</sup>(Φ(*g*)) = �F(*∂x*) *<sup>G</sup>*�<sup>2</sup> <sup>+</sup> �F(*∂y*) *<sup>G</sup>*�2, *B* stands for F(*b*) and given *∂<sup>x</sup>* takes the form of a (convolution) matrix, then F(*∂x*) denotes

K*rg ∂r*(*g* ∗ *k*), *r* = 1, 2, 3, . . . , *R* (33) *nr ∂rn*, *r* = 1, 2, 3, . . . , *R* (34)

Θ = {*∂x*, *∂y*, *∂xx*, *∂xy*, *∂yy*}. (35)

<sup>Φ</sup>(*g*) �*∂<sup>x</sup> <sup>g</sup>*�<sup>2</sup> <sup>+</sup> �*∂<sup>y</sup> <sup>g</sup>*�2. (36)

*En*(*g*) = �*<sup>g</sup>* <sup>∗</sup> *<sup>k</sup>* <sup>−</sup> *<sup>b</sup>*�<sup>2</sup> <sup>+</sup> *<sup>λ</sup>* <sup>Φ</sup>(*g*). (37)

<sup>F</sup>(*En*(*g*)) = �*<sup>G</sup> <sup>K</sup>* <sup>−</sup> *<sup>B</sup>*�<sup>2</sup> <sup>+</sup> *<sup>λ</sup>* <sup>F</sup>(Φ(*g*)), (38)

and in order to align with model (25), define

**4.2 Frequency domain deconvolution**

elements of Θ from the following values

**4.2.1 Normal likelihood deconvolution**

frequency domain for (37) as follows.

functional under "normal likelihood", can be derived as

the form

where

its Fourier transform.

where ∗ stands for the convolution operator and *k* is the blur kernel.

new information. We corroborate this claim in section 4.3 with experiments.

$$\Psi = \mathcal{F}(\partial\_{\mathcal{X}}) \star \overline{\mathcal{F}(\partial\_{\mathcal{X}})} + \mathcal{F}(\partial\_{\mathcal{Y}}) \star \overline{\mathcal{F}(\partial\_{\mathcal{Y}})},$$

*G <sup>n</sup>* is the Fourier transform of the estimated GTI under "normal likelihood", (·) stands for the complex conjugate and the division is performed element-wise. The estimated ground truth image *gn* can be derived by taking the inverse Fourier transform of *<sup>G</sup> <sup>n</sup>*.

With the above derivations, it is evident that Fourier domain expression used to estimate the GTI is:


#### **4.2.2 Derivative likelihood deconvolution**

We now give an analogous derivation for the "derivative likelihood".

The energy functional in this case is derived similar to (37),

$$E\_d(\mathbf{g}) = \sum\_{\partial\_r \in \Theta} \frac{1}{2^{\omega(\partial\_r)}} \left\| \partial\_r (\mathbf{g} \ast \mathbf{k}) - \partial\_r \mathbf{b} \right\|^2 + \lambda \left\| \Phi(\mathbf{g}).\tag{40}$$

Transforming (40) into the frequency domain results in

$$\mathcal{F}(E\_d(g)) = \sum\_{\partial\_l \in \Theta} \frac{1}{2^{\omega(\partial\_l)}} \left( \|\mathcal{F}(\partial\_r) \star G \star K - \mathcal{F}(\partial\_r) \star B\|^2 \right) + \lambda \, \mathcal{F}(\Phi(g)),\tag{41}$$

where, *∂<sup>r</sup>* is a matrix convolution operator representing a partial order derivative operator and F(*∂r*) denotes its Fourier transformation.

By minimizing the energy (41), we compute the estimated *G*,

$$
\widehat{G\_d} = \frac{B \star \overline{K} \star \Omega}{K \star \overline{K} \star \Omega + \lambda \,\Psi'} \,\tag{42}
$$

where

$$
\Omega \stackrel{\triangle}{=} \sum\_{\partial\_{\prime} \in \Theta} \frac{1}{\mathfrak{L}^{\omega(\partial\_{\prime})}} \mathcal{F}(\partial\_{r}) \star \overline{\mathcal{F}(\partial\_{r})}.
$$

By taking the inverse Fourier transforms of (42), we could get the estimated GTI, *<sup>g</sup><sup>d</sup>*, under "derivative likelihood" model similar to "normal likelihood" model.

(a) Ground Truth Image (b) 19 × 27 Blur kernel scaled upwards

Regularized Image Restoration 141

(c) Normal likelihood (*λ* = 0.001) (d) Derivative likelihood (*λ* = 0.001)

(e) Normal likelihood (*λ* = 0.8) (f) Derivative likelihood (*λ* = 0.8)

In this chapter, we have contributed to regularization based image restoration techniques in

1. We have developed a general class of quadratic regularization models based on partial derivative operators (PDO), section 2.2. Out of those models, we have shown that the Second Order Partial Derivative Operator (SOPDO) model performs better than First Order Partial Derivative Operator (FOPDO) model for images susceptible to noise, while the novel First and Second Order Partial Derivative Operator (FSOPDO) model performs

Fig. 11. Noise maps for Likelihood models

better than both FOPDO and SOPDO models.

**5. Contributions**

the following:


Table 8. Comparison of likelihood models

## **4.3 Likelihood model analysis**

In order to come up with the most effective and efficient restoration algorithm, we investigate the contribution of each of the likelihood models for estimating the GTI: (39) corresponding to "normal likelihood" and (42) corresponding to "derivative likelihood", respectively.

We used the same "Picasso" image which was used in Shan et al. (2008) for experiments using the likelihood model in (29). The ground truth images are estimated using the Fourier domain techniques, specifically applying (39) and (42) for the "Normal" and "Derivative" likelihood models respectively. The experiment results are given in Table 8. In order to eliminate the "Model" and "Process" artifacts as discussed in section 3.3.2, in all our simulations, the blurring was carried out under the assumption that the images are periodic.

The MSE values in the table are given as multiples of 10−4, while the value of *λ* is given in multiples of 10−5. The values in bold in Table 8 refer to the optimal MSE values the respective likelihood model could reach for varying *λ*. As the results show clearly, the "normal likelihood" model has a better estimate for the GTI than the "derivative likelihood" model, we claim that applying "normal likelihood" in the image restoration algorithm results in a better restoration.

Our investigation was further extended to analyze whether higher order derivatives of noise contribute to the spatial randomness of noise as claimed in Shan et al. (2008). The noise maps given in Fig. 11 are computed for different values of *λ* in (39) and (42).

As per the results Fig. 11(c) and Fig. 11(d), when the effect of the prior becomes smaller (i.e., the weight on the data fitting term or the likelihood becomes larger), the noise estimate is more spatially random, but with the increase in the weight of the prior, the noise estimate becomes structured (signal dependant), see Fig. 11(e) and Fig. 11(f). We experienced these results regardless of the likelihood model we used. Based on the above results, we claim that using higher order partial derivatives in the likelihood model for non-blind deconvolution does not result in a better noise map estimation while the same noise map estimation can be achieved through the normal likelihood model with the appropriate Lagrange multiplier.

Hence, through the likelihood model analysis based on benchmark image, we conclude that higher order derivatives in the likelihood model are not required for better performance whereas applying single image likelihood model with appropriate regularization results in a more effective non-blind image restoration.

22 Will-be-set-by-IN-TECH

In order to come up with the most effective and efficient restoration algorithm, we investigate the contribution of each of the likelihood models for estimating the GTI: (39) corresponding to "normal likelihood" and (42) corresponding to "derivative likelihood", respectively.

We used the same "Picasso" image which was used in Shan et al. (2008) for experiments using the likelihood model in (29). The ground truth images are estimated using the Fourier domain techniques, specifically applying (39) and (42) for the "Normal" and "Derivative" likelihood models respectively. The experiment results are given in Table 8. In order to eliminate the "Model" and "Process" artifacts as discussed in section 3.3.2, in all our simulations, the

The MSE values in the table are given as multiples of 10−4, while the value of *λ* is given in multiples of 10−5. The values in bold in Table 8 refer to the optimal MSE values the respective likelihood model could reach for varying *λ*. As the results show clearly, the "normal likelihood" model has a better estimate for the GTI than the "derivative likelihood" model, we claim that applying "normal likelihood" in the image restoration algorithm results in a better

Our investigation was further extended to analyze whether higher order derivatives of noise contribute to the spatial randomness of noise as claimed in Shan et al. (2008). The noise maps

As per the results Fig. 11(c) and Fig. 11(d), when the effect of the prior becomes smaller (i.e., the weight on the data fitting term or the likelihood becomes larger), the noise estimate is more spatially random, but with the increase in the weight of the prior, the noise estimate becomes structured (signal dependant), see Fig. 11(e) and Fig. 11(f). We experienced these results regardless of the likelihood model we used. Based on the above results, we claim that using higher order partial derivatives in the likelihood model for non-blind deconvolution does not result in a better noise map estimation while the same noise map estimation can be achieved through the normal likelihood model with the appropriate Lagrange multiplier.

Hence, through the likelihood model analysis based on benchmark image, we conclude that higher order derivatives in the likelihood model are not required for better performance whereas applying single image likelihood model with appropriate regularization results in

blurring was carried out under the assumption that the images are periodic.

given in Fig. 11 are computed for different values of *λ* in (39) and (42).

a more effective non-blind image restoration.

Normal likelihood Derivative likelihood

*<sup>λ</sup>* MSE MSE

<sup>×</sup>10−<sup>5</sup> <sup>×</sup>10−<sup>4</sup> <sup>×</sup>10−<sup>4</sup> 100 2.8448 4.4371 5 2.4148 **4.0862** 0.25 **2.2292** 4.1852

Table 8. Comparison of likelihood models

**4.3 Likelihood model analysis**

restoration.

(a) Ground Truth Image (b) 19 × 27 Blur kernel scaled upwards

(c) Normal likelihood (*λ* = 0.001) (d) Derivative likelihood (*λ* = 0.001)

(e) Normal likelihood (*λ* = 0.8) (f) Derivative likelihood (*λ* = 0.8)

Fig. 11. Noise maps for Likelihood models

### **5. Contributions**

In this chapter, we have contributed to regularization based image restoration techniques in the following:

1. We have developed a general class of quadratic regularization models based on partial derivative operators (PDO), section 2.2. Out of those models, we have shown that the Second Order Partial Derivative Operator (SOPDO) model performs better than First Order Partial Derivative Operator (FOPDO) model for images susceptible to noise, while the novel First and Second Order Partial Derivative Operator (FSOPDO) model performs better than both FOPDO and SOPDO models.

Hunt, B. R. (1977). Bayesian methods in nonlinear digital image restoration, *IEEE Transactions*

Regularized Image Restoration 143

Hunt, B. R. & Andrews, H. C. (1977). *Digital Image Restoration*, Prentice Hall Professional

Joshi, N., Szeliski, R. & Kriegman, D. J. (2008). PSF estimation using sharp edge prediction, *IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008*, pp. 1–8. Kim, S. J., Koh, K., Lustig, M. & Boyd, S. (2007). An efficient method for compressed sensing, *International Conference on Image Processing, ICIP 2007*, Vol. 3, pp. 117–120. Krahmer, F., Lin, Y., McAdoo, B., Ott, K., Wang, J. & Widemann, D. (2006). Blind image

Kundur, D. & Hatzinakos, D. (1996). Blind image deconvolution, *IEEE Signal Processing*

Leung, C. M. & Lu, W. S. (1995). On the use of discrete Laplacian operators in image

Levin, A. (2007). Blind motion deblurring using image statistics, *Advances in Neural Information*

Levin, A., Fergus, R., Durand, F. & Freeman, W. T. (2007). Image and depth from

Levin, A. & Weiss, Y. (2007). User assisted separation of reflections from a single image

Levin, A., Weiss, Y., Durand, F. & Freeman, W. T. (2009). Understanding and evaluating blind

Liu, R. & Jia, J. Y. (2008). Reducing boundary artifacts in image deconvolution, *IEEE International Conference on Image Processing, ICIP'08*, Vol. 1-5, pp. 505–508. Meer, P. (2004). Robust techniques for computer vision, *Emerging topics in computer vision*

Mignotte, M. (2006). A segmentation-based regularization term for image deconvolution,

Miller, K. (1970). Least squares methods for ill-posed problems with a prescribed bound, *SIAM*

Nayar, S. K. & Ben-Ezra, M. (2004). Motion-based motion deblurring, *IEEE Transactions on*

Oliveira, J. P., Figueiredo, M. A. & Bioucas-Dias, J. M. (2007). Blind estimation of motion

Olshausen, B. A. et al. (1996). Emergence of simple-cell receptive field properties by learning

Phillips, D. L. (1962). A technique for the numerical solution of certain integral equations of

blur parameters for image deconvolution, *IbPRIA '07: Proceedings of the 3rd Iberian conference on Pattern Recognition and Image Analysis, Part II*, Springer-Verlag, Berlin,

*IEEE Transactions on Image Processing* 15(7): 1973–1984.

*Pattern Analysis and Machine Intelligence* 26(6): 689–698.

a sparse code for natural images, *Nature* 381(6583): 607–609.

the first kind, *Journal of the ACM (JACM)* 9(1): 84–97.

*Journal on Mathematical Analysis* 1(1): 52–74. URL: *http://link.aip.org/link/?SJM/1/52/1*

URL: *http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.110.2995*

deconvolution: motion blur estimation, *University of Minnesota, URL: http://www. ima.*

restoration, *IEEE Pacific Rim Conference on Communications, Computers, and Signal*

a conventional camera with a coded aperture, *ACM Transactions on Graphics*

using a sparsity prior, *IEEE Transactions on Pattern Analysis and Machine Intelligence*

deconvolution algorithms, *IEEE Conference on Computer Vision and Pattern Recognition,*

*on Computers* 26: 219–229.

Technical Reference.

*Magazine* 13(3): 43–64.

*Processing*, pp. 411–415.

29(9): 1647–1654.

pp. 107–190.

*CVPR 2009*, pp. 1964–1971.

Heidelberg, pp. 604–611.

*Processing Systems* 19: 841–848.

*(Proceedings of SIGGRAPH)* 26(3): 70–79.

*umn. edu* .


### **6. References**

	- URL: *http://dx.doi.org/10.1080/02664769300000058*

24 Will-be-set-by-IN-TECH

2. We have used the Structured Similarity index (SSIM) map, Mean SSIM (MSSIM) value and histograms of SSIM maps as novel visual metrics for comparison and evaluation of

3. We have critically evaluated Sparse and Laplacian prior models against Quadratic regularization models using the novel visual metrics discussed in section 3.5. By eliminating the effects of processing and modeling artifacts, not present when capturing actual blurred natural images, we have shown that Sparse and Laplacian derivative prior models, which are claimed to be consistent with natural images, do not significantly contribute in restoring natural image features and have significantly slower relative

4. Finally, we have analyzed and evaluated multiple derivative operator based restoration methods under MAP/ML framework with a novel model to represent the likelihood based on multiple images, section 4.1.2. By using this novel model, we demonstrate that complex higher order derivative likelihood models are not required for better performance in image

Albert, A. E. (1972). *Regression and the Moore-Penrose Pseudoinverse*, Mathematics in science and

Banham, M. R. & Katsaggelos, A. K. (1997). Digital image restoration, *IEEE Signal Processing*

Barrett, R., Berry, M., Chan, T. F., Demmel, J., Donato, J. M., Dongarra, J., Eijkhout, V., Romine,

Bracewell, R. & Kahn, P. B. (1966). The Fourier transform and its applications, *American Journal*

Demoment, G. (1989). Image reconstruction and restoration: overview of common estimation

Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T. & Freeman, W. T. (2006). Removing camera

Geman, S. & Geman, D. (1984). Stochastic relaxation, gibbs distributions and the bayesian

Greig, D. M., Porteous, B. T. & Seheult, A. H. (1989). Exact maximum a posteriori estimation

Hadamard, J. (1952). *Lectures on Cauchy's problem in linear partial differential equations*, New

Hansen, P. C. (1998). *Rank-deficient and discrete ill-posed problems: numerical aspects of linear inversion*, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.

Hunt, B. R. (1971). Biased estimation for nonparametric identification of linear systems,

Hunt, B. R. (1973). The application of constrained least squares estimation to image restoration

by digital computer, *IEEE Transactions on Computers* 22(9): 805–812.

shake from a single photograph, *ACM Trans. Graph.* 25(3): 787–794.

URL: *http://dx.doi.org/10.1080/02664769300000058*

Haykin, S. (1994). *Blind Deconvolution*, Prentice-Hall: Englewood Cliffs, NJ.

Haven, CT: Yale University Press.

*Mathematical Biosciences* 10(3-4): 215–237.

*Blocks for Iterative Methods*, Society for Industrial Mathematics.

R. P. C. & Vorst, H. V. D. (1994). *Templates for the Solution of Linear Systems: Building*

structures and problems, *IEEE Transactions on Acoustics, Speech and Signal Processing*

restoration of images, *IEEE Transactions on Pattern Analysis and Machine Intelligence*

for binary images, *Journal of the Royal Statistical Society. Series B (Methodological)*

regularization models in image restoration, section 3.3.2.

engineering, New York : Academic Press.

restoration performance.

*Magazine* 14(2): 24–41.

*of Physics* .

37(12): 2024–2036.

6(6): 721–741.

51(2): 271–279.

restoration.

**6. References**


URL: *http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.110.2995*


**7** 

*Spain* 

**Iterative Restoration Methods** 

*1Polythecnic University of Madrid, Department of Signals,* 

 **of Regularized Solutions** 

 *Systems and Radiocommunications* 

 **to Loose Estimations Dependency** 

Miguel A. Santiago1, Guillermo Cisneros1 and Emiliano Bernués2

*2University of Zaragoza, Department of Electronic Engineering and Communications* 

Image restoration is a classical area of digital image processing, appearing in many applications such as remote sensing, medical imaging, astronomy or computerized tomography (González & Woods, 2007). Simply put, the aim is to recover an original image which has been degraded due to the imperfections in the acquisition system: blurring and noise. Restoring this degradation leads to an ill-posed problem since the simple inverse using least-squares yields highly noise-sensitive solutions. A large number of techniques have been developed to cope with this issue, most of them under the regularization or the Bayesian frameworks (a complete review can found in Banham & Katsaggelos, 1997; Bovik,

Mathematical regularization is used to include prior knowledge about the original image in the restoration process which allows stabilizing the solution in the face of noise. However, two main problems arise for such a regularization approach. First, the non-local property of the underlying convolution implies that part of the blurred image near the boundary integrates information of the original scenery outside the field of view. However, this information is not available in the deconvolution process and may cause strong ringing artifacts on the restored image, i.e., the well-known boundary problem (Woods et al., 1985). Typical methods to counteract the boundary effect is to make assumptions about the behavior of the original image outside the field of view such as Dirichlet, Neuman, periodic or other recent conditions in Calvetti & Somersalo, 2005; Martinelli et al., 2006; Liu & Jia, 2008. Secondly, restoration methods depend on a wide set of parameters which can be roughly grouped into three categories: parameters with respect to the degradation process, the noise and the original image. All parameters require an accurate prior estimation because small errors in their values lead to important deviations in the restoration results. In fact, classical restoration methods tend to improve the estimation of those parameters without prior knowledge about the real scenery, which is known as blind deconvolution (Campisi & Egiazarian, 2007; Molina et al., 2006). The boundary problem and the sensitivity to estimations are the issues to solve in this chapter by means of two iterative algorithms.

**1. Introduction** 

2005; Chan & Shen, 2005).


## **Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions**

Miguel A. Santiago1, Guillermo Cisneros1 and Emiliano Bernués2 *1Polythecnic University of Madrid, Department of Signals, Systems and Radiocommunications 2University of Zaragoza, Department of Electronic Engineering and Communications Spain* 

## **1. Introduction**

26 Will-be-set-by-IN-TECH

144 Image Restoration – Recent Advances and Applications

Portilla, J., Strela, V., Wainwright, M. J. & Simoncelli, E. P. (2003). Image denoising using scale

Roth, S. & Black, M. (2005). Fields of experts: A framework for learning image priors,

Shan, Q., Jia, J. & Agarwala, A. (2008). High-quality motion deblurring from a single image,

Simoncelli, E. P. (1997). Statistical models for images: compression, restoration and synthesis,

Simoncelli, E. P. (1999). Bayesian denoising of visual images in the wavelet domain, *Lecture*

Tikhonov, A. N. & Arsenin, V. Y. (1977). *Solutions of ill-posed problems*, John Wiley, New York. Wang, Z. & Bovik, A. C. (2009). Mean squared error: love it or leave it? - A new look at signal

Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. (2004). Image quality assessment:

Weiss, Y. (2001). Deriving intrinsic images from image sequences, *9th International Conference*

Wiener, N. (1942). Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Zhu, S. C. & Mumford, D. (1997). Prior learning and Gibbs reaction-diffusion, *IEEE Transactions on Pattern Analysis and Machine Intelligence* 19(11): 1236–1250.

fidelity measures, *IEEE Signal Processing Magazine* 26(1): 98–117.

*on Computer Vision, ICCV*, IEEE Computer Society, pp. 68–75.

*2008*, Vol. 27, ACM, New York, NY, USA, pp. 1–10.

*Computers*, Vol. 1, pp. 673–678.

*Notes in Statistics* 141: 291–308.

12(11): 1338–1351.

pp. 860–867.

13(4): 600–612.

mixtures of Gaussians in the wavelet domain, *IEEE Transactions on Image processing*

*Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2*, IEEE Computer Society, Washington, DC, USA,

*International Conference on Computer Graphics and Interactive Techniques SIGGRAPH*

*Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and*

From error visibility to structural similarity, *IEEE Transactions on Image Processing*

Image restoration is a classical area of digital image processing, appearing in many applications such as remote sensing, medical imaging, astronomy or computerized tomography (González & Woods, 2007). Simply put, the aim is to recover an original image which has been degraded due to the imperfections in the acquisition system: blurring and noise. Restoring this degradation leads to an ill-posed problem since the simple inverse using least-squares yields highly noise-sensitive solutions. A large number of techniques have been developed to cope with this issue, most of them under the regularization or the Bayesian frameworks (a complete review can found in Banham & Katsaggelos, 1997; Bovik, 2005; Chan & Shen, 2005).

Mathematical regularization is used to include prior knowledge about the original image in the restoration process which allows stabilizing the solution in the face of noise. However, two main problems arise for such a regularization approach. First, the non-local property of the underlying convolution implies that part of the blurred image near the boundary integrates information of the original scenery outside the field of view. However, this information is not available in the deconvolution process and may cause strong ringing artifacts on the restored image, i.e., the well-known boundary problem (Woods et al., 1985). Typical methods to counteract the boundary effect is to make assumptions about the behavior of the original image outside the field of view such as Dirichlet, Neuman, periodic or other recent conditions in Calvetti & Somersalo, 2005; Martinelli et al., 2006; Liu & Jia, 2008. Secondly, restoration methods depend on a wide set of parameters which can be roughly grouped into three categories: parameters with respect to the degradation process, the noise and the original image. All parameters require an accurate prior estimation because small errors in their values lead to important deviations in the restoration results. In fact, classical restoration methods tend to improve the estimation of those parameters without prior knowledge about the real scenery, which is known as blind deconvolution (Campisi & Egiazarian, 2007; Molina et al., 2006). The boundary problem and the sensitivity to estimations are the issues to solve in this chapter by means of two iterative algorithms.

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 147

digitalizing images. The statistical descriptors of the noise are likewise assumed to not vary

Let *xi j* (, ) be the unknown gray-scaled original image of size *L L* 1 2 , degraded by a PSF *hi j* (, ) and corrupted by a noise sample *ni j* (, ) . Therefore, we can express the observed

where represents the two dimensional convolution operator. In order to simplify expressions, we shall use lexicographic notation by stacking the columns of a matrix in a

defined by the original image **x** of length *LL L* 1 2 , whereas the degraded image **y** is a *L* sized vector bigger than the original image as result of the non-local property of the convolution operation (see 2.1). In terms of blurring, **H** is known as the convolution matrix of size *L L* built from the PSF and using the so-called boundary conditions that we will

Another way to represent the equation (1) is through its spectral equivalence. By applying

( , ) ( , )( , ) ( , ) *Y HX N*

Fourier transforms. In the frequency domain it is assumed that the observed image is a circular period that wraps around the edges, what it is not physically true but typically used

In view of the above equations, image restoration is defined as an inverse problem that tries to estimate the original image **x**ˆ from the observed image **y** using the blurring model **H** . However, a simple least-squares solution is not possible since the presence of noise or the singularity of the matrix **H** make it an ill-conditioned problem. Thus, a regularization method is needed to control the high sensitivity to noise as explained in Banham & Katsaggelos, 1997. Quite a few examples have been presented in the literature by means of

> <sup>1</sup> <sup>ˆ</sup> arg min 2 2

 

*<sup>j</sup>* are the spatial frequency coordinates, and the capital letters represent

 *<sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup>* (3)

> 2 2 2 2

**<sup>x</sup> x y Hx Dx** (4)

**<sup>z</sup>** *z* denotes the 2 norm, **x**ˆ is the restored image and **D** is the

regularization operator, built on the basis of a high pass filter mask **d** of size *NN N* 1 2 . The first term in (4) is the 2 residual norm appearing in the least-squares approach and

discrete Fourier transform (González & Woods, 2007) to that expression, we obtain

 

 *i* 

the classical Tikhonov regularization which establishes

*<sup>n</sup>* , which also represents the quantization error coming from

*y*(, ) (, ) (, ) (, ) *i j hi j xi j ni j* (1)

**y Hx n** (2)

noise *ni j* (, ) of variance <sup>2</sup>

spatially.

image *yi j* (, ) as

discuss later.

where ( , ) *i* 

where <sup>2</sup> <sup>2</sup>

2 *i i*

for computational convenience.

vector. Then, the equation (1) is rewritten as

The first algorithm copes with the boundary problem taking a blurred image defined in the field of view, but with neither any image information nor prior assumption on the boundary conditions. Furthermore, the objective is not only to reduce the ringing artifacts on the whole image, but also reconstruct the missed boundaries of the original image which becomes a significant step of the research. Neural networks are very well suited to combine both processes in the same restoration algorithm and thus we provide a solution based on a Multilayer Perceptron (MLP) in line with a backpropagation strategy. Other neural-netbased restoration techniques (Paik & Katsaggelos, 1992; Sun, 2000; Han & Wu, 2004) have been proposed in the literature with the Hopfield's model, but they are typically timeconsuming and large scaled. In the light of the good results of the total variation (TV) regularizer in recent deconvolution (Wang et al., 2005; Wu et al., 2007; Bioucas-Dias et al., 2006; Oliveira et al., 2009; Molina et al., 2006), we have used it to set the minimization mechanism of the net. The proposed scheme is then an iterative method which performs repeatedly a cycle of two steps: forward and backward, simulating respectively restoration and degradation processes at each iteration.

Following the same iterative concept of restoration-degradation, we present a second algorithm in the frequency domain to reduce the dependency on the estimation of parameters. Hence, a novel desensitized restoration filter is designed by applying an iterative algorithm over the original filter. Analyzing the sensitivity properties of this filter and setting a criterion to choose the number of iterations, we come up with an expression for the desensitized algorithm for traditional filters such as Wiener and Tikhonov (González & Woods, 2007). The results of this algorithm pretend to increase the robustness of the restoration methods when estimating parameters such as the noise variance or degradation related parameters.

The chapter is organized as follows. In the next section, we provide a detailed formulation of the two restoration problems of the chapter, establishing naming conventions and the mathematical basis of the respective algorithms. In Sec. 3, we present the architecture of the iterative methods under analysis: MLP and desensitized filter, going into details about the adjustment of the synaptic weights of the net in every layer and the computation of the number of iterations for the desensitized scheme respectively. We present some experimental results in Sec. 4 and, finally, concluding remarks are given in Sec. 5.

## **2. Problem formulation**

To start with image restoration a better understanding of the acquisition system is required. Because of physical limitations or human errors in operating imaging systems, the observed image is actually a degraded version of the original scene. For instance, deterministic degradations such as motion blurs, out of focus lens or effects of atmospheric turbulence in remote sensing cause a bandwidth reduction of the original image. In a linear acquisition scenario this distortion is mathematically described as a point spread function (PSF) denoted by *hi j* (, ) , which represents a two dimensional filter mask of size *M*1 2 *M* . For sake of simplicity we consider spatially invariant functions such that the degradation is independent of the position. In addition to blurring, noise is always present in the observed image due to stochastic variations in the process of image formation, the transmission medium or the recording system. We assume a common additive zero-mean Gaussian white 146 Image Restoration – Recent Advances and Applications

The first algorithm copes with the boundary problem taking a blurred image defined in the field of view, but with neither any image information nor prior assumption on the boundary conditions. Furthermore, the objective is not only to reduce the ringing artifacts on the whole image, but also reconstruct the missed boundaries of the original image which becomes a significant step of the research. Neural networks are very well suited to combine both processes in the same restoration algorithm and thus we provide a solution based on a Multilayer Perceptron (MLP) in line with a backpropagation strategy. Other neural-netbased restoration techniques (Paik & Katsaggelos, 1992; Sun, 2000; Han & Wu, 2004) have been proposed in the literature with the Hopfield's model, but they are typically timeconsuming and large scaled. In the light of the good results of the total variation (TV) regularizer in recent deconvolution (Wang et al., 2005; Wu et al., 2007; Bioucas-Dias et al., 2006; Oliveira et al., 2009; Molina et al., 2006), we have used it to set the minimization mechanism of the net. The proposed scheme is then an iterative method which performs repeatedly a cycle of two steps: forward and backward, simulating respectively restoration

Following the same iterative concept of restoration-degradation, we present a second algorithm in the frequency domain to reduce the dependency on the estimation of parameters. Hence, a novel desensitized restoration filter is designed by applying an iterative algorithm over the original filter. Analyzing the sensitivity properties of this filter and setting a criterion to choose the number of iterations, we come up with an expression for the desensitized algorithm for traditional filters such as Wiener and Tikhonov (González & Woods, 2007). The results of this algorithm pretend to increase the robustness of the restoration methods when estimating parameters such as the noise variance or degradation

The chapter is organized as follows. In the next section, we provide a detailed formulation of the two restoration problems of the chapter, establishing naming conventions and the mathematical basis of the respective algorithms. In Sec. 3, we present the architecture of the iterative methods under analysis: MLP and desensitized filter, going into details about the adjustment of the synaptic weights of the net in every layer and the computation of the number of iterations for the desensitized scheme respectively. We present some

To start with image restoration a better understanding of the acquisition system is required. Because of physical limitations or human errors in operating imaging systems, the observed image is actually a degraded version of the original scene. For instance, deterministic degradations such as motion blurs, out of focus lens or effects of atmospheric turbulence in remote sensing cause a bandwidth reduction of the original image. In a linear acquisition scenario this distortion is mathematically described as a point spread function (PSF) denoted by *hi j* (, ) , which represents a two dimensional filter mask of size *M*1 2 *M* . For sake of simplicity we consider spatially invariant functions such that the degradation is independent of the position. In addition to blurring, noise is always present in the observed image due to stochastic variations in the process of image formation, the transmission medium or the recording system. We assume a common additive zero-mean Gaussian white

experimental results in Sec. 4 and, finally, concluding remarks are given in Sec. 5.

and degradation processes at each iteration.

related parameters.

**2. Problem formulation** 

noise *ni j* (, ) of variance <sup>2</sup> *<sup>n</sup>* , which also represents the quantization error coming from digitalizing images. The statistical descriptors of the noise are likewise assumed to not vary spatially.

Let *xi j* (, ) be the unknown gray-scaled original image of size *L L* 1 2 , degraded by a PSF *hi j* (, ) and corrupted by a noise sample *ni j* (, ) . Therefore, we can express the observed image *yi j* (, ) as

$$y(i,j) = l(i,j) \* \* \ge (i,j) + n(i,j) \tag{1}$$

where represents the two dimensional convolution operator. In order to simplify expressions, we shall use lexicographic notation by stacking the columns of a matrix in a vector. Then, the equation (1) is rewritten as

$$\mathbf{y} = \mathbf{H}\mathbf{x} + \mathbf{n} \tag{2}$$

defined by the original image **x** of length *LL L* 1 2 , whereas the degraded image **y** is a *L* sized vector bigger than the original image as result of the non-local property of the convolution operation (see 2.1). In terms of blurring, **H** is known as the convolution matrix of size *L L* built from the PSF and using the so-called boundary conditions that we will discuss later.

Another way to represent the equation (1) is through its spectral equivalence. By applying discrete Fourier transform (González & Woods, 2007) to that expression, we obtain

$$X(\varpi\_i, \varpi\_j) = H(\varpi\_i, \varpi\_j) X(\varpi\_i, \varpi\_j) + N(\varpi\_i, \varpi\_j) \tag{3}$$

where ( , ) *i <sup>j</sup>* are the spatial frequency coordinates, and the capital letters represent Fourier transforms. In the frequency domain it is assumed that the observed image is a circular period that wraps around the edges, what it is not physically true but typically used for computational convenience.

In view of the above equations, image restoration is defined as an inverse problem that tries to estimate the original image **x**ˆ from the observed image **y** using the blurring model **H** . However, a simple least-squares solution is not possible since the presence of noise or the singularity of the matrix **H** make it an ill-conditioned problem. Thus, a regularization method is needed to control the high sensitivity to noise as explained in Banham & Katsaggelos, 1997. Quite a few examples have been presented in the literature by means of the classical Tikhonov regularization which establishes

$$\hat{\mathbf{x}} = \arg\min\_{\mathbf{x}} \left\{ \frac{1}{2} \left\| \mathbf{y} - \mathbf{H} \mathbf{x} \right\|\_{2}^{2} + \frac{\lambda}{2} \left\| \mathbf{D} \mathbf{x} \right\|\_{2}^{2} \right\} \tag{4}$$

where <sup>2</sup> <sup>2</sup> 2 *i i* **<sup>z</sup>** *z* denotes the 2 norm, **x**ˆ is the restored image and **D** is the regularization operator, built on the basis of a high pass filter mask **d** of size *NN N* 1 2 . The first term in (4) is the 2 residual norm appearing in the least-squares approach and

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 149

where **T** has a Toeplitz structure and **B** is often structured, sparse and low rank, and specifically defined for every BC. Common cases are the *Zero* (Bertero & Bocacci, 1998), *Periodic* (Bertero & Bocacci, 1998), *Reflective* (Ng et al., 1999) or *Anti-reflective* (Martinelly et

As a result of the convolution, it can be easily demonstrated (see Fig. 1) that the degraded

where *B*1 and *B*2 are the respective horizontal and vertical bandwidths of the PSF, then the length of **h** is *MM M B B* 12 1 2 2 12 1 . We have gray colored the pixels affected by the boundary conditions which are not actually present in a real observation. Therefore, a real observed image **yreal** is a truncated version of the convolution process to the region

�

as

FOV

*L*

*LL B L B* <sup>1122</sup> 2 2 (8)

FOV 2 2 *LBLB L* 11 22 (9)

*B*1

*B*

<sup>1</sup> *<sup>L</sup>* <sup>2</sup> *<sup>B</sup>*

*B*1

*x*ˆ

image **y** increases its size with respect to the original image **x** from *L* to *L*

FOV

FOV

*L*

*L* ~

Fig. 1. Real observed image defined in the field of view (left). Restored image which

neither any image information nor prior assumption on the boundary conditions.

operator that removes (zero-fixes) the pixels of the boundary region, that is to say

Common deblurring methods deal with this real image **yreal** and try to restore it minimizing the boundary ringing as much as possible using BCs on the model **H** like (7). However, the restored image is only obtained within the FOV domain, that is smaller than the original image size *L* . Our goal is to not only improve the restoration on the whole image but also reconstruct the boundaries that are missed in the observation, without

Let us define an image **ytru** which represents this observed image **yreal** using a trunc

<sup>1</sup> *<sup>L</sup> real <sup>y</sup>* <sup>2</sup> *<sup>B</sup>* <sup>2</sup> *<sup>B</sup>*

*ytru*

indicates the boundary reconstruction area (right).

*B*1 *B*1

al., 2006) boundary conditions.

called field of view

� <sup>2</sup> *<sup>L</sup>*

ensures fidelity to data. The second term is the so-called *regularizer* which captures prior knowledge about **x** through an additional 2 penalty term involving the image. The hyperparameter (or regularization parameter) is a critical value which measures the trade-off between a good fit and a regularized solution.

Alternatively, the total variation (TV) regularization proposed by Rudin et al., 1992, has become very popular in recent research as it achieves to preserve edges in the restored image. A discrete version of the TV deblurring problem is given by

$$\hat{\mathbf{x}} = \arg\min\_{\mathbf{x}} \left\{ \frac{1}{2} \left\| \mathbf{y} - \mathbf{H} \mathbf{x} \right\|\_{2}^{2} + \mathcal{L} \left\| \nabla \mathbf{x} \right\|\_{1} \right\} \tag{5}$$

where 1 **z** denotes the 1 norm (i.e., the sum of the absolute value of the elements) and stands for the discrete gradient operator. The operator is defined by the matrices **<sup>ξ</sup> D** and **<sup>μ</sup> D** as

$$\nabla \mathbf{x} = \left| \mathbf{D}^{\xi} \mathbf{x} \right| + \left| \mathbf{D}^{\mu} \mathbf{x} \right| \tag{6}$$

built on the basis of the respective masks **<sup>ξ</sup> d** and **<sup>μ</sup> d** of size *NN N* 1 2 , which turn out the horizontal and vertical first order differences of the image. Compared to the expression (4), the TV regularization provides a 1 penalty term which can be thought as a measure of signal variability. Once again, is the critical regularization parameter to control the weight assigned to the regularizer with respect to the data mist term.

Significant amount of work has been addressed to solve any of the above regularizations and mainly the TV deblurring in recent times (Chan & Shen, 2005). However, there are two important issues in those algorithms which require making assumptions and constraining the regularized solution: boundary conditions and parameters estimations. This chapter provides two novel iterative methods aimed to loose this dependency and achieve a more robust solution in terms of estimations. Let us analyze each problem separately.

#### **2.1 Boundary conditions**

As defined in González & Woods, 2007, the convolution operator of equation (1) integrates a portion of the original scenery **x** into a single point by weighting the nearby pixels by a 180 degrees rotated mask **h** . When computing the pixels near the boundary and depending on the size of the PSF, many pixels of **y** contain information coming from the original scenery outside the field of view (FOV) which is indeterminate. We refer to this phenomenon and to its consequences as *boundary effect*. It is well known that if the boundary effect is not properly taken into account, it may cause strong ringing artifacts on the deconvolved image. For that reason, various methods of the literature try to counteract this effect by selecting appropriate boundaries conditions (BCs). These boundary conditions are included in the model of **H** used in deconvolution as

$$\mathbf{H} = \mathbf{T} + \mathbf{B} \tag{7}$$

148 Image Restoration – Recent Advances and Applications

ensures fidelity to data. The second term is the so-called *regularizer* which captures prior knowledge about **x** through an additional 2 penalty term involving the image. The hyper-

Alternatively, the total variation (TV) regularization proposed by Rudin et al., 1992, has become very popular in recent research as it achieves to preserve edges in the restored

where 1 **z** denotes the 1 norm (i.e., the sum of the absolute value of the elements) and stands for the discrete gradient operator. The operator is defined by the matrices **<sup>ξ</sup> D** and

built on the basis of the respective masks **<sup>ξ</sup> d** and **<sup>μ</sup> d** of size *NN N* 1 2 , which turn out the horizontal and vertical first order differences of the image. Compared to the expression (4), the TV regularization provides a 1 penalty term which can be thought as a measure of

Significant amount of work has been addressed to solve any of the above regularizations and mainly the TV deblurring in recent times (Chan & Shen, 2005). However, there are two important issues in those algorithms which require making assumptions and constraining the regularized solution: boundary conditions and parameters estimations. This chapter provides two novel iterative methods aimed to loose this dependency and achieve a more

As defined in González & Woods, 2007, the convolution operator of equation (1) integrates a portion of the original scenery **x** into a single point by weighting the nearby pixels by a 180 degrees rotated mask **h** . When computing the pixels near the boundary and depending on the size of the PSF, many pixels of **y** contain information coming from the original scenery outside the field of view (FOV) which is indeterminate. We refer to this phenomenon and to its consequences as *boundary effect*. It is well known that if the boundary effect is not properly taken into account, it may cause strong ringing artifacts on the deconvolved image. For that reason, various methods of the literature try to counteract this effect by selecting appropriate boundaries conditions (BCs). These boundary conditions are included in the

2

2 1

**<sup>x</sup> x y Hx x** (5)

**ξ μ x Dx Dx** (6)

is the critical regularization parameter to control the

**HTB** (7)

is a critical value which measures the trade-off

image. A discrete version of the TV deblurring problem is given by

weight assigned to the regularizer with respect to the data mist term.

robust solution in terms of estimations. Let us analyze each problem separately.

<sup>1</sup> <sup>ˆ</sup> arg min <sup>2</sup>

parameter (or regularization parameter)

signal variability. Once again,

**2.1 Boundary conditions** 

model of **H** used in deconvolution as

**<sup>μ</sup> D** as

between a good fit and a regularized solution.

where **T** has a Toeplitz structure and **B** is often structured, sparse and low rank, and specifically defined for every BC. Common cases are the *Zero* (Bertero & Bocacci, 1998), *Periodic* (Bertero & Bocacci, 1998), *Reflective* (Ng et al., 1999) or *Anti-reflective* (Martinelly et al., 2006) boundary conditions.

As a result of the convolution, it can be easily demonstrated (see Fig. 1) that the degraded image **y** increases its size with respect to the original image **x** from *L* to *L* as

$$\tilde{L} = \left(L\_1 + \mathcal{D}B\_1\right) \times \left(L\_2 + \mathcal{D}B\_2\right) \tag{8}$$

where *B*1 and *B*2 are the respective horizontal and vertical bandwidths of the PSF, then the length of **h** is *MM M B B* 12 1 2 2 12 1 . We have gray colored the pixels affected by the boundary conditions which are not actually present in a real observation. Therefore, a real observed image **yreal** is a truncated version of the convolution process to the region called field of view

FOV 2 2 *LBLB L* 11 22 (9)

Fig. 1. Real observed image defined in the field of view (left). Restored image which indicates the boundary reconstruction area (right).

Common deblurring methods deal with this real image **yreal** and try to restore it minimizing the boundary ringing as much as possible using BCs on the model **H** like (7). However, the restored image is only obtained within the FOV domain, that is smaller than the original image size *L* . Our goal is to not only improve the restoration on the whole image but also reconstruct the boundaries that are missed in the observation, without neither any image information nor prior assumption on the boundary conditions.

Let us define an image **ytru** which represents this observed image **yreal** using a trunc operator that removes (zero-fixes) the pixels of the boundary region, that is to say

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 151

common algorithms of the literature defined in the Fourier space: Wiener and Tikhonov (Bovik, 2005). Both methods are completely linear so described by a restoration filter as

ˆ( , ) ( , )( , ) *X GY*

be removed from all formulae throughout the remainder of the chapter. They are differentiated from the variables of the boundary problem because those are in bold. Besides, it must be taken into account that all mathematical expressions involving matrices in the Fourier Transform domain are scalar computations for each frequency component

> 2 *nn xx*

2 2

*H D*

Let us symbolize the restoration filter as *G*ˆ when calculated by estimations (not real values)

Therefore, we shall define an iterative method which achieves a filter *G* based on the

In this section we propose two algorithms to cope with the aforementioned constraints of a restoration problem: boundary conditions and estimation dependency. Both methods are iterative and lead to various restoration-degradation processes repeated a certain number of

Restore a real observed image **yreal** without neither any image information nor prior

where *H* represents the complex conjugate of *<sup>H</sup>* , *Sxx* and *Snn* are the respective spectral

*<sup>H</sup> <sup>G</sup>*

where *D* is the Fourier transform of the regularization operator **D** in (4).

as well as the rest of variables involved in (14) and (15) such as *H*ˆ , ˆ

times. A detailed analysis is devoted to each algorithm in the following sections.

*S*

*<sup>H</sup> <sup>G</sup> <sup>S</sup> <sup>H</sup>*

 

*<sup>j</sup>* denotes the Fourier transform of the restoration filter. In order to simplify

*<sup>j</sup> <sup>i</sup> <sup>j</sup> <sup>i</sup> <sup>j</sup>* (13)

*<sup>j</sup>* of the matrices in the frequency domain will

(14)

(15)

*Sxx* , <sup>ˆ</sup>

*Syy* and <sup>ˆ</sup>

.

 

 *i* 

 *i* 

notation, the reference to the element ( , )

From González & Woods, 2007, it is demonstrated that

densities of the original image **x** and the noise **n** .

original *G*ˆ with less sensitivity to wrong estimations.

The main issues addressed by this algorithm are

assumption on the boundary conditions.

where ( , ) *G i* 

Wiener Filter

Tikhonov Filter

**3. Iterative methods** 

**3.1 MLP approach** 

(, ) *i <sup>j</sup>* .

$$\mathbf{y}\_{tm}(i,j) = \text{trunc}\left(\mathbf{H}\_{\mathbf{a}}\mathbf{x} + \mathbf{n}\big|\_{(i,j)}\right) = \begin{cases} \mathbf{y}\_{\text{real}} = \mathbf{H}\_{\mathbf{a}}\mathbf{x} + \mathbf{n}\big|\_{(i,j)} & \forall (i,j) \in \text{FOV} \\ 0 & otherwise \end{cases} \tag{10}$$

where **Ha** denotes the Toeplitz matrix when not using boundary conditions (aperiodic model). Therefore, we aim to restore this truncated image **ytru** in spite of the discontinuity at the boundaries and reconstruct the region *B* depicted in Fig. 1

$$B = L - FOV\tag{11}$$

whose area is calculated by *BLB B* 11 1 4 , if we consider square dimensions such that *B B* 1 2 and *L L* 1 2 .

Particulary, we intend to study an iterative algorithm using the TV regularizer which loose the dependency on the boundary conditions. So we redefine the restoration problem (5) including the trunc operator as

$$\hat{\mathbf{x}} = \arg\min\_{\mathbf{x}} \left\{ \frac{1}{2} \left\| \mathbf{y} - \text{trunc} \left( \mathbf{H}\_{\mathbf{a}} \mathbf{x} \right) \right\|\_{2}^{2} + \lambda \left\| \text{trunc} \left( \left\| \mathbf{D}\_{\mathbf{a}}^{\xi} \mathbf{x} \right\| + \left\| \mathbf{D}\_{\mathbf{a}}^{\mathbf{u}} \mathbf{x} \right\| \right) \right\} \tag{12}$$

where the subscript **a** denotes the aperiodic formulation of every matrix operator. An equivalent analysis for the Tikhonov regularizer can be found in Santiago et al., 2010.

#### **2.2 Estimations dependency**

If we have a look to any restoration method of the literature, we come up with their dependency on a wide set of parameters which must be estimated a priori. We can group them basically into three classes


In terms of blurring, the convolution matrix **H** is not always available in the restoration process and thus it is required to make assumptions about its parameters, such as the length of motion or the radius of out-of-focus among others. We can find a lot of articles devoted to estimate the PSF which are normally referred to as blind deconvolution. Regarding noise we have assumed a Gaussian white noise from the very beginning, so the concrete parameter is just the variance <sup>2</sup> *<sup>n</sup>* . Finally, the parameters related to the original image have to do with the regularization term of the equations (4) or (5) and, in turn, with the regularization parameter .

Blind deconvolution methods try to obtain the more accurate parameters but deal with a problem known as sensitivity to estimations, that is to say, relatively small deviations from the real (unknown) values have a severe impact on the restoration quality. Therefore, we aim to define an algorithm that improves the results of a restoration scheme when having wrong estimates of the said parameters, namely, a desensitization process.

We shall work in the frequency domain for this issue so we take for granted the circular boundary conditions of the previous section. In particular, our goal is to desensitize two common algorithms of the literature defined in the Fourier space: Wiener and Tikhonov (Bovik, 2005). Both methods are completely linear so described by a restoration filter as

$$
\hat{X}(\boldsymbol{\varpi}\_{i'}\boldsymbol{\varpi}\_{\boldsymbol{j}}) = \mathbf{G}(\boldsymbol{\varpi}\_{i'}\boldsymbol{\varpi}\_{\boldsymbol{j}})Y(\boldsymbol{\varpi}\_{i'}\boldsymbol{\varpi}\_{\boldsymbol{j}})\tag{13}
$$

where ( , ) *G i <sup>j</sup>* denotes the Fourier transform of the restoration filter. In order to simplify notation, the reference to the element ( , ) *i <sup>j</sup>* of the matrices in the frequency domain will be removed from all formulae throughout the remainder of the chapter. They are differentiated from the variables of the boundary problem because those are in bold. Besides, it must be taken into account that all mathematical expressions involving matrices in the Fourier Transform domain are scalar computations for each frequency component (, ) *i <sup>j</sup>* .

From González & Woods, 2007, it is demonstrated that

Wiener Filter

150 Image Restoration – Recent Advances and Applications

*(i, j) y ij otherwise*

where **Ha** denotes the Toeplitz matrix when not using boundary conditions (aperiodic model). Therefore, we aim to restore this truncated image **ytru** in spite of the discontinuity

whose area is calculated by *BLB B* 11 1 4 , if we consider square dimensions such that

Particulary, we intend to study an iterative algorithm using the TV regularizer which loose the dependency on the boundary conditions. So we redefine the restoration problem (5)

where the subscript **a** denotes the aperiodic formulation of every matrix operator. An equivalent analysis for the Tikhonov regularizer can be found in Santiago et al., 2010.

If we have a look to any restoration method of the literature, we come up with their dependency on a wide set of parameters which must be estimated a priori. We can group

In terms of blurring, the convolution matrix **H** is not always available in the restoration process and thus it is required to make assumptions about its parameters, such as the length of motion or the radius of out-of-focus among others. We can find a lot of articles devoted to estimate the PSF which are normally referred to as blind deconvolution. Regarding noise we have assumed a Gaussian white noise from the very beginning, so the concrete parameter is

the regularization term of the equations (4) or (5) and, in turn, with the regularization

Blind deconvolution methods try to obtain the more accurate parameters but deal with a problem known as sensitivity to estimations, that is to say, relatively small deviations from the real (unknown) values have a severe impact on the restoration quality. Therefore, we aim to define an algorithm that improves the results of a restoration scheme when having

We shall work in the frequency domain for this issue so we take for granted the circular boundary conditions of the previous section. In particular, our goal is to desensitize two

wrong estimates of the said parameters, namely, a desensitization process.

**<sup>a</sup> a a <sup>x</sup>**

( , ) trunc <sup>0</sup>

<sup>1</sup> <sup>ˆ</sup> arg min trunc trunc

2

Parameters with respect to the blurring process.

Parameters with respect to the original image.

Parameters with respect to the noise.

**a**

at the boundaries and reconstruct the region *B* depicted in Fig. 1

*tru (i,j)*

*B B* 1 2 and *L L* 1 2 .

including the trunc operator as

**2.2 Estimations dependency** 

them basically into three classes

just the variance <sup>2</sup>

.

parameter

(,) FOV

 <sup>2</sup> <sup>2</sup> <sup>1</sup>

**x y Hx Dx Dx** (12)

*<sup>n</sup>* . Finally, the parameters related to the original image have to do with

 **real a**

*i j*

**y Hx n Hx n** (10)

*B L FOV* (11)

**ξ μ**

$$G = \frac{H^\*}{\left|H\right|^2 + \frac{S\_{\text{nn}}}{S\_{\text{xx}}}} \tag{14}$$

where *H* represents the complex conjugate of *<sup>H</sup>* , *Sxx* and *Snn* are the respective spectral densities of the original image **x** and the noise **n** .

Tikhonov Filter

$$G = \frac{H^\*}{\left|H\right|^2 + \mathcal{L}\left|D\right|^2} \tag{15}$$

where *D* is the Fourier transform of the regularization operator **D** in (4).

Let us symbolize the restoration filter as *G*ˆ when calculated by estimations (not real values) as well as the rest of variables involved in (14) and (15) such as *H*ˆ , ˆ *Sxx* , <sup>ˆ</sup> *Syy* and <sup>ˆ</sup> . Therefore, we shall define an iterative method which achieves a filter *G* based on the original *G*ˆ with less sensitivity to wrong estimations.

## **3. Iterative methods**

In this section we propose two algorithms to cope with the aforementioned constraints of a restoration problem: boundary conditions and estimation dependency. Both methods are iterative and lead to various restoration-degradation processes repeated a certain number of times. A detailed analysis is devoted to each algorithm in the following sections.

#### **3.1 MLP approach**

The main issues addressed by this algorithm are

 Restore a real observed image **yreal** without neither any image information nor prior assumption on the boundary conditions.

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 153

A restored image is therefore obtained in real conditions on the basis of a global energy minimization strategy, with reconstructed borders while adapting the center of the image to the optimum solution and thus making the ringing artifact negligible. Finally, we recall that

Let us define each layer of Fig. 2 as an input vector **p** of size *R* 1 , a synaptic weight matrix **W** of *S R* in size, and a 1 *S* output vector **z** of the layer. We utilize a log-sigmoid

denote the number of layer, but it will be removed when deduced by context. So we can redraw our MLP as depicted in Fig. 3 where we have symbolized the variation of the

and a null bias vector. A superscript is used to

*W2*

**Layer 1 Layer 2** *<sup>L</sup>*<sup>2</sup>

*S*<sup>1</sup>1 *S*<sup>1</sup>1 *L*1

inputs 1 *S L* neurons

*2 v L*1 *<sup>z</sup> <sup>x</sup> <sup>2</sup>* <sup>ˆ</sup>

**<sup>W</sup>** (16)

(19)

*L*1

<sup>1</sup> *SL*

*2 p*

*th m 1* iteration. Then,

*i E m*

indicates the learning speed. Defining the vectors **e**( ) *m* and **r**( ) *m* for the

*m*

( ) trunc ( ) *m m* ˆ **<sup>a</sup> e y Hx** (17)

**a a r D x Dx** (18)

the input to the net is always the image **y***tru* as no net training is required.

*W1*

*1 v* 1 <sup>1</sup> *S* *<sup>1</sup> <sup>z</sup>*

A variant of the well-known algorithm of back-propagation is used to adjust those matrixes with the truncated cost function of (12). Let ( 1) *<sup>i</sup>* **W** *m* be the correction applied to the

( ) ( 1) ( )

where *E m*( ) stands for the cost error function after *m* iterations at the output of the net and

( ) trunc ( ) ( ) *m mm* ˆ ˆ **ξ μ**

2

<sup>1</sup> () () () <sup>2</sup> *Em m m* **e r**

2 1

neurons 1 *S*

*i*

**W**

*m*

respective error and regularization terms at the output layer after *m* iterations

*LS*<sup>1</sup> <sup>~</sup>

*p y <sup>1</sup>* <sup>1</sup> <sup>~</sup> *L*

Fig. 3. MLP algorithm with matrix-vector notation.

inputs ~ *L*

weight matrix **W***<sup>i</sup>* of the layer *i* at the

we can rewrite the restoration error from (12)

**3.1.1 Adjustment of the neural net** 

expression for the transfer function

synaptic matrixes of every layer.

2*BL* <sup>22</sup>

the constant

<sup>1</sup> 2*BL* <sup>1</sup>


To go around this problem we know that neural networks are particularly well-suited as their ability to nonlinear mapping and self-adaptiveness. In fact, the Hopfield network has been used in the literature to solve the optimization problem (4) and recently some neural network solutions as in Wang, 2005 and Wu, 2007 deal with the TV regularization (5).

Our proposal is a MLP (Multiplayer Perceptron) with back-propagation as illustrated inFig. 2. The input layer of the net consists of *L* neurons with inputs 1 2 , ,..., *<sup>L</sup> <sup>y</sup> y y* being respectively the *L* pixels of the truncated image **ytru** . At any generic iteration *<sup>m</sup>* , the output layer is defined by *L* neurons whose outputs 1 2 ˆˆ ˆ ( ), ( ),..., ( ) *<sup>L</sup> xmxm x m* are respectively the *L* pixels of an approach *x*ˆ*(m)* to the restored image. After *m* iterations, the neural net outcomes the actual restored image **x x** ˆ ˆ *(m)* . On the other hand, the hidden layer consists of only two neurons, although being enough to achieve good restoration results while keeping low complexity of the network.

Fig. 2. MLP scheme adopted for image restoration.

The neural network undertakes two processes iteratively: forward and backward. The former is the result of applying from left to right the equations of every layer. It is actually the restoration step. The latter is the back-propagation process where the network must minimize a regularized error function which we will set to the expression (12). It means to adjust the synaptic coefficients of every single neuron from right to left and can be thought as a reblurring step. Since the trunc operator is involved in all those expressions, the truncation of the boundaries is performed at every iteration but also their reconstruction as indicated by the *L* size at the output. What deserves attention is that no a priori knowledge, assumption or estimation concerning the unknown borders is needed to perform the regeneration. Generally speaking it could be explained by the neural net nature which is able to learn about the degradation model.

A restored image is therefore obtained in real conditions on the basis of a global energy minimization strategy, with reconstructed borders while adapting the center of the image to the optimum solution and thus making the ringing artifact negligible. Finally, we recall that the input to the net is always the image **y***tru* as no net training is required.

#### **3.1.1 Adjustment of the neural net**

152 Image Restoration – Recent Advances and Applications

Reconstruct the boundary region *B* so that the restored image has the same size *L* as

To go around this problem we know that neural networks are particularly well-suited as their ability to nonlinear mapping and self-adaptiveness. In fact, the Hopfield network has been used in the literature to solve the optimization problem (4) and recently some neural network solutions as in Wang, 2005 and Wu, 2007 deal with the TV regularization (5).

Our proposal is a MLP (Multiplayer Perceptron) with back-propagation as illustrated inFig.

output layer is defined by *L* neurons whose outputs 1 2 ˆˆ ˆ ( ), ( ),..., ( ) *<sup>L</sup> xmxm x m* are respectively the *L* pixels of an approach *x*ˆ*(m)* to the restored image. After *m* iterations, the neural net outcomes the actual restored image **x x** ˆ ˆ *(m)* . On the other hand, the hidden layer consists of only two neurons, although being enough to achieve good restoration results while

*tru y* ˆˆ )(*mtotal xx*

*backward*

The neural network undertakes two processes iteratively: forward and backward. The former is the result of applying from left to right the equations of every layer. It is actually the restoration step. The latter is the back-propagation process where the network must minimize a regularized error function which we will set to the expression (12). It means to adjust the synaptic coefficients of every single neuron from right to left and can be thought as a reblurring step. Since the trunc operator is involved in all those expressions, the truncation of the boundaries is performed at every iteration but also their reconstruction as indicated by the *L* size at the output. What deserves attention is that no a priori knowledge, assumption or estimation concerning the unknown borders is needed to perform the regeneration. Generally speaking it could be explained by the neural net nature which is

*forward*

*L L* outputs

pixels of the truncated image **ytru** . At any generic iteration *<sup>m</sup>* , the

neurons with inputs 1 2 , ,..., *<sup>L</sup> <sup>y</sup> y y* being

ˆ )(1 *mx*

ˆ )(2 *mx*

ˆ *mx* )(*<sup>L</sup>*

Remove boundary ringing in spite of the discontinuity at the boundaries.

the original image.

respectively the *L*

Make use of the TV regularizer.

2. The input layer of the net consists of *L*

keeping low complexity of the network.

2 *y*

inputs <sup>~</sup>

1 *y*

*L y*~

able to learn about the degradation model.

Fig. 2. MLP scheme adopted for image restoration.

Let us define each layer of Fig. 2 as an input vector **p** of size *R* 1 , a synaptic weight matrix **W** of *S R* in size, and a 1 *S* output vector **z** of the layer. We utilize a log-sigmoid expression for the transfer function and a null bias vector. A superscript is used to denote the number of layer, but it will be removed when deduced by context. So we can redraw our MLP as depicted in Fig. 3 where we have symbolized the variation of the synaptic matrixes of every layer.

Fig. 3. MLP algorithm with matrix-vector notation.

A variant of the well-known algorithm of back-propagation is used to adjust those matrixes with the truncated cost function of (12). Let ( 1) *<sup>i</sup>* **W** *m* be the correction applied to the weight matrix **W***<sup>i</sup>* of the layer *i* at the *th m 1* iteration. Then,

$$
\Delta \mathbf{W}^i(m+1) = -\eta \frac{\partial E(m)}{\partial \mathbf{W}^i(m)} \tag{16}
$$

where *E m*( ) stands for the cost error function after *m* iterations at the output of the net and the constant indicates the learning speed. Defining the vectors **e**( ) *m* and **r**( ) *m* for the respective error and regularization terms at the output layer after *m* iterations

$$\mathbf{e}(m) = \mathbf{y} - \text{trunc}\left(\mathbf{H}\_{\mathbf{a}}\hat{\mathbf{x}}(m)\right) \tag{17}$$

$$\mathbf{r}(m) = \text{trunc}\left( \left| \mathbf{D}\_{\mathbf{a}}^{\xi} \hat{\mathbf{x}}(m) \right| + \left| \mathbf{D}\_{\mathbf{a}}^{\mu} \hat{\mathbf{x}}(m) \right| \right) \tag{18}$$

we can rewrite the restoration error from (12)

$$E(m) = \frac{1}{2} \left\| \mathbf{e}(m) \right\|\_2^2 + \mathcal{A} \left\| \mathbf{r}(m) \right\|\_1 \tag{19}$$

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 155

regularizer, but lately researches aim to provide solutions for the TV regularization.

However, we still need to investigate a particular algorithm for the MLP since those Bayesian approaches work only for circulant degradation models, but not for the truncated

Desensitize the restoration filter (assumed linear) with respect to wrong parameter

Counteract the effects of mistaking parameters in order to achieve a better restoration

Alternative to classic restoration approaches which focus on obtaining accurate

Let us define an expression for the desensitized filter *G* based on the original *G*ˆ in the frequency domain. Again our approach is an iterative algorithm as illustrated in Fig. 4.

)(ˆ ),...,1(ˆ *mYY*

*Y m*( 1) after going through the restoration filter *G*ˆ and the estimated transfer

*<sup>j</sup>* and, in fact, we put forward that the number of iterations is also a

the output image ˆ ˆ *X Xm* ( ) . This algorithm is somehow based on the same iterative concept of restoration-degradation processes of the MLP but applied to the Fourier domain. Let us recall that the mathematical expressions for this algorithm are particular for each frequency

ˆ

*G*'

*Y*(0) is equal to the degraded image *Y* and, after the

*Y m*( ) is restored again by the filter *G*ˆ leading to the

*mm* 1

. In fact, its main purpose is to speed up or slow down the convergence of

with solutions coming from the Bayesian state-of-art.

which optimizes the results.

shows lower sensitivity

for the images of

*XmXX* ˆ )(ˆ ),...,1(ˆ

*Y m*( ) computed by its previous

Specifically, the Bayesian framework leads to successful approaches in this respect.

the algorithm. Then, for the sake of simplicity, we shall assume 2

image of this chapter. So we shall compute yet a hand-tuned

As for learning speed it was already demonstrated that

The second of our methods go around the following issues

quality compared to that without desensitization.

ˆ *H*ˆ )0( *YY*ˆ *G*

The input at any iteration *m* ( *m m* 1,2,..., ) is an image ˆ

 *i <sup>j</sup>* .

It can be easily demonstrated that the filter *G* of Fig. 4 is expressed as

Particularization to Wiener and Tikhonov filters

*G*

Fig. 4. Desensitized restoration scheme.

function *H*ˆ . In a first step the image ˆ

total number of iterations, the image ˆ

function of these elements, i.e., (,) *m*

In Santiago et al., 2010 we adjusted

**3.2 Desensitization approach** 

compared to

256 256 in size.

estimations.

estimations.

iteration ˆ

component ( , ) *i* 

Now we aim to compute the so-called gradient matrix ( ) ( ) *<sup>i</sup> E m m* **W** in the layers of the MLP. A

high detailed analysis can be found in Santiago et al., 2010 based on the algorithm of majorization-minimization developed by Bioucas-Dias et al., 2006 when facing a TV regularization problem like (5). Let us summarize the main results below:

$$
\Delta \mathbf{W}^i(m+1) = -\eta \mathbf{\hat{G}}^i(m) \Big(\mathbf{z}^{i-1}(m)\Big)^T \tag{20}
$$

where **δ**( ) *m* stands for the local gradient vector and is defined for a MLP of *J* layers as:

Output layer ( *i J* )

$$\mathbf{\dot{\mathfrak{G}}}(m) = \boldsymbol{\varphi}\{\mathbf{v}(m)\} \circ \left(-\mathbf{H}\_{\mathbf{a}}^{T}\mathbf{e}(m) + \lambda \mathbf{D}\_{\mathbf{a}}^{T}\mathbf{\mathfrak{Q}}(m)\mathbf{r}(m)\right) \tag{21}$$

where denotes the Hadamard (elementwise) product, **Da** is a composition of the matrices **<sup>ξ</sup> Da** and **<sup>μ</sup> Da** as *<sup>T</sup> T T* **ξ μ DDD aaa** and **Ω**( ) *m* represents a weigh matrix which controls the influence of regions with high intensity variation

$$\begin{aligned} \mathbf{\dot{\Delta}}(m) &= \begin{bmatrix} \mathbf{\dot{A}}(m) & 0 \\ 0 & \mathbf{A}(m) \end{bmatrix} \\ \text{with } \mathbf{\dot{A}}(m) &= \text{diag}\left(\frac{1}{2\sqrt{\left(\mathbf{D\_{\dot{A}}^{\xi}\hat{\mathbf{x}}(m)\right)^{2} + \left(\mathbf{D\_{\dot{A}}^{\mu}\hat{\mathbf{x}}(m)\right)^{2} + \varepsilon}}}\right) \\ &= \text{diag}\left(\frac{1}{2\sqrt{\left(\mathbf{D\_{\dot{A}}^{\xi}\hat{\mathbf{x}}(m)\right)^{2} + \left(\mathbf{D\_{\dot{A}}^{\mu}\hat{\mathbf{x}}(m)\right)^{2} + \varepsilon}}}\right) \end{aligned} \tag{22}$$

Any hidden layer ( *i J* )

$$\mathbf{b}^{i}(m) = \text{diag}\left(\boldsymbol{\varphi}\left\{\mathbf{v}^{i}(m)\right\}\right)\left(\mathbf{W}^{i+1}(m)\right)^{T}\mathbf{b}^{i+1}(m) \tag{23}$$

#### **3.1.2 Algorithm parameters**

Due to the iterative nature of the algorithm the first parameter to establish has to do with the stop rule. It is a condition such that either the number of iterations is more than a maximum; or the error *E m*( ) converges and, thus, the error change *E m*( ) is less than a threshold; or, even, this error *E m*( ) starts to increase. If one of these conditions comes true, the algorithm concludes and the final outgoing image is the restored image **x x** ˆ ˆ ( ) *m* .

In the image restoration field it is remarked the importance of the parameter . Low values of yield oscillatory solutions because of the presence of noise or discontinuities; high values of yield over smoothed results though. For that reason, the literature has given significant attention to it with popular approaches such as the unbiased predictive risk estimator (UPRE), the generalized cross validation (GCV), or the L-curve method; see Vogel, 2002 for an overview and references. Most of them were particularized for a Tikhonov regularizer, but lately researches aim to provide solutions for the TV regularization. Specifically, the Bayesian framework leads to successful approaches in this respect.

In Santiago et al., 2010 we adjusted with solutions coming from the Bayesian state-of-art. However, we still need to investigate a particular algorithm for the MLP since those Bayesian approaches work only for circulant degradation models, but not for the truncated image of this chapter. So we shall compute yet a hand-tuned which optimizes the results.

As for learning speed it was already demonstrated that shows lower sensitivity compared to . In fact, its main purpose is to speed up or slow down the convergence of the algorithm. Then, for the sake of simplicity, we shall assume 2 for the images of 256 256 in size.

## **3.2 Desensitization approach**

154 Image Restoration – Recent Advances and Applications

high detailed analysis can be found in Santiago et al., 2010 based on the algorithm of majorization-minimization developed by Bioucas-Dias et al., 2006 when facing a TV

> <sup>1</sup> ( 1) ( ) ( ) *<sup>T</sup> i ii m mm*

where **δ**( ) *m* stands for the local gradient vector and is defined for a MLP of *J* layers as:

( ) ( ) ( ) ( )( ) *T T* **δ** *m m m mm*

where denotes the Hadamard (elementwise) product, **Da** is a composition of the matrices

() 0 ( ) 0 ()

**Λ**

*m*

 1 1 ( ) diag () () () *<sup>T</sup> i iii m m mm* **<sup>δ</sup>**

Due to the iterative nature of the algorithm the first parameter to establish has to do with the stop rule. It is a condition such that either the number of iterations is more than a maximum; or the error *E m*( ) converges and, thus, the error change *E m*( ) is less than a threshold; or, even, this error *E m*( ) starts to increase. If one of these conditions comes true, the algorithm concludes and the final outgoing image is the restored image **x x** ˆ ˆ ( ) *m* .

yield oscillatory solutions because of the presence of noise or discontinuities; high

significant attention to it with popular approaches such as the unbiased predictive risk estimator (UPRE), the generalized cross validation (GCV), or the L-curve method; see Vogel, 2002 for an overview and references. Most of them were particularized for a Tikhonov

yield over smoothed results though. For that reason, the literature has given

 

<sup>1</sup> with ( ) diag

In the image restoration field it is remarked the importance of the parameter

*m*

**Ω**

 

**ξ μ DDD aaa** and **Ω**( ) *m* represents a weigh matrix which controls

2 2

 **ξ μ a a**

*m m ε*

**v W δ** (23)

*m*

**Λ**

 

2 () () ˆ ˆ

**Dx Dx**

( ) *i E m m*

**W δ z** (20)

**v He D a aΩ r** (21)

**W** in the layers of the MLP. A

(22)

. Low values

Now we aim to compute the so-called gradient matrix ( )

*<sup>T</sup> T T*

*m*

**Λ**

the influence of regions with high intensity variation

Output layer ( *i J* )

**<sup>ξ</sup> Da** and **<sup>μ</sup> Da** as

Any hidden layer ( *i J* )

**3.1.2 Algorithm parameters** 

of 

values of

regularization problem like (5). Let us summarize the main results below:

The second of our methods go around the following issues


Let us define an expression for the desensitized filter *G* based on the original *G*ˆ in the frequency domain. Again our approach is an iterative algorithm as illustrated in Fig. 4.

Fig. 4. Desensitized restoration scheme.

The input at any iteration *m* ( *m m* 1,2,..., ) is an image ˆ *Y m*( ) computed by its previous iteration ˆ *Y m*( 1) after going through the restoration filter *G*ˆ and the estimated transfer function *H*ˆ . In a first step the image ˆ *Y*(0) is equal to the degraded image *Y* and, after the total number of iterations, the image ˆ *Y m*( ) is restored again by the filter *G*ˆ leading to the the output image ˆ ˆ *X Xm* ( ) . This algorithm is somehow based on the same iterative concept of restoration-degradation processes of the MLP but applied to the Fourier domain. Let us recall that the mathematical expressions for this algorithm are particular for each frequency component ( , ) *i <sup>j</sup>* and, in fact, we put forward that the number of iterations is also a function of these elements, i.e., (,) *m i <sup>j</sup>* .

It can be easily demonstrated that the filter *G* of Fig. 4 is expressed as

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 157

previous analysis is that the trend of both errors is contrary for the desensitized filter *G* . Whereas the noise-dependent error is lower as the number of iterations increases, the ringing component gets higher. Consequently, we need to look for a trade-off between the

Since the relative sensitivity function *Z m*( ) does not have a local minimum as viewed in Fig. 5, let us optimize another *Z m*( ) property which also fulfills the desensitization criteria. In particular, we shall look for a maximum of efficiency for the complexity introduced in the restoration process by increasing the number of iterations from *m* to 1 *m* . In other words, let us seek a value of *m* from that on the improvement on desensitization is lower than the incremental complexity. In mathematical terms we can express this efficiency change as the second derivative of *Z m*( ) denoted by *Rm Z m* () () . It can be easily derived from (27) that

The purpose is to maximize this function as well as constrain it to the desensitization condition of *Z m*() 1 . In Santiago et al., 2007 we came up to a number of iterations as

Finally, let us compute some numeric results of the main variables of the desensitization

*m round*

subject to a constraint on the regularization term ˆ ˆ 0.14 0.84 *GH* .

algorithm for different regularization products *GH*ˆ ˆ : *m* , *Z m*( ) , ( )

<sup>3</sup> 1 ˆ <sup>ˆ</sup> ln( )

*GH*

ˆˆ ˆ ˆˆ ˆ ( ) ( ) ln( ) 2 ( 1)ln( ) *<sup>m</sup> R m GH GH m GH* (28)

(29)

 *<sup>r</sup> m* and ( ) 

*<sup>n</sup> m* , where

error components while keeping the desensitization criteria true.

Fig. 5. Relative sensitivity function *Z m*( ) .

**3.2.2 Number of iterations** 

follows

$$\mathbf{G}' = \hat{\mathbf{G}} \left( \hat{\mathbf{G}} \hat{H} \right)^{\mathrm{m}} \tag{24}$$

where *GH*ˆ ˆ is known as the regularization product. In Santiago et al., 2007 we verified that the higher the regularization is, the lower the product *GH*ˆ ˆ becomes with a dynamic range ˆ ˆ 0 1 *GH* .

#### **3.2.1 Sensitivity criteria**

So far we have referred to sensitivity as a concept, but now we put it on mathematical expressions. Let us consider that the restoration filter *G* depends on a set of parameters 1 2 , ,..., *PP Pr* which can be grouped into the three groups of Section 2.2: blurring, noise and original image. Then we can define the sensitivity *SG* regarding the filter *G* as

$$S\_G = \frac{\partial G}{\partial P\_1}dP\_1 + \frac{\partial G}{\partial P\_2}dP\_2 + \dots + \frac{\partial G}{\partial P\_r}dP\_r \tag{25}$$

Analogously, the sensitivity concerning the proposed filter *G* can be expressed as follows

$$S\_{G'} = \frac{\partial G'}{\partial P\_1}dP\_1 + \frac{\partial G'}{\partial P\_2}dP\_2 + \dots + \frac{\partial G'}{\partial P\_r}dP\_r \tag{26}$$

To compare the sensitivity of both filters we make use of a relative function *ZS S G G* which sets the desensitization criteria as *Z* 1 . After differentiating the filter *G* of (24) with respect to *G* we come up with an expression for the relative sensitivity function (see Santiago et al., 2007 for further details)

$$Z(m) = \frac{S\_{G'}}{S\_G} = (m+1)(\hat{\mathbf{G}}\hat{H})^m < 1\tag{27}$$

As ˆ ˆ ˆ ˆ 0 1 *<sup>m</sup> GH GH* we can foresee that the function *Z m*( ) of (27) is neither monotonically increasing nor decreasing with the number of iterations *m* , but it may show a relative maximum extreme depending on the value of the term *GH*ˆ ˆ for a particular pair (, ) *i <sup>j</sup>* . This is illustrated in Fig. 5 for several regularization values

Looking into this plot we can observe that the expected maximum extremes of *Z m*( ) depend on the value of *GH*ˆ ˆ . The lower the product *GH*ˆ ˆ is, the less iterations *m* are required to reach the maximum; even high regularization conditions make *Z m*( ) strictly decreasing monotonic. In any case, the main conclusion has to do with the sensitivity condition (27) illustrated by the straight line of the figure. Regardless of the value of the product *GH*ˆ ˆ , *G* is less sensitive than *G* if the number of iterations *m* is high enough. We might therefore increase the value of *m* as needed to prevent poor restoration results of wrong estimates. However, that is not possible as the restoration error is significantly affected as demonstrated in Santiago et al., 2007.

In González & Woods, 2007 the restoration error is divided into the ringing (or imagedependent) component and the noise-dependent component. What we found out in our previous analysis is that the trend of both errors is contrary for the desensitized filter *G* . Whereas the noise-dependent error is lower as the number of iterations increases, the ringing component gets higher. Consequently, we need to look for a trade-off between the error components while keeping the desensitization criteria true.

Fig. 5. Relative sensitivity function *Z m*( ) .

### **3.2.2 Number of iterations**

156 Image Restoration – Recent Advances and Applications

where *GH*ˆ ˆ is known as the regularization product. In Santiago et al., 2007 we verified that the higher the regularization is, the lower the product *GH*ˆ ˆ becomes with a dynamic range

So far we have referred to sensitivity as a concept, but now we put it on mathematical expressions. Let us consider that the restoration filter *G* depends on a set of parameters 1 2 , ,..., *PP Pr* which can be grouped into the three groups of Section 2.2: blurring, noise and

> 1 2 1 2

Analogously, the sensitivity concerning the proposed filter *G* can be expressed as follows

1 2 1 2

*GG G S dP dP dP PP P* 

To compare the sensitivity of both filters we make use of a relative function *ZS S G G* which sets the desensitization criteria as *Z* 1 . After differentiating the filter *G* of (24) with respect to *G* we come up with an expression for the relative sensitivity function (see

ˆ ˆ ( ) ( 1)( ) 1 *<sup>G</sup> <sup>m</sup>*

Looking into this plot we can observe that the expected maximum extremes of *Z m*( ) depend on the value of *GH*ˆ ˆ . The lower the product *GH*ˆ ˆ is, the less iterations *m* are required to reach the maximum; even high regularization conditions make *Z m*( ) strictly decreasing monotonic. In any case, the main conclusion has to do with the sensitivity condition (27) illustrated by the straight line of the figure. Regardless of the value of the product *GH*ˆ ˆ , *G* is less sensitive than *G* if the number of iterations *m* is high enough. We might therefore increase the value of *m* as needed to prevent poor restoration results of wrong estimates. However, that is not possible as the restoration error is significantly

In González & Woods, 2007 the restoration error is divided into the ringing (or imagedependent) component and the noise-dependent component. What we found out in our

 *GH GH* we can foresee that the function *Z m*( ) of (27) is neither monotonically increasing nor decreasing with the number of iterations *m* , but it may show a relative maximum extreme depending on the value of the term *GH*ˆ ˆ for a particular pair

*G <sup>S</sup> Z m m GH S*

*<sup>j</sup>* . This is illustrated in Fig. 5 for several regularization values

... *G r*

... *G r*

*GG G S dP dP dP PP P* 

*r*

*r*

(25)

(26)

(27)

original image. Then we can define the sensitivity *SG* regarding the filter *G* as

ˆ ˆ 0 1 *GH* .

**3.2.1 Sensitivity criteria** 

Santiago et al., 2007 for further details)

affected as demonstrated in Santiago et al., 2007.

As ˆ ˆ ˆ ˆ 0 1 *<sup>m</sup>*

(, ) *i* 

ˆ ˆ <sup>ˆ</sup> *<sup>m</sup>*

*G G GH* (24)

Since the relative sensitivity function *Z m*( ) does not have a local minimum as viewed in Fig. 5, let us optimize another *Z m*( ) property which also fulfills the desensitization criteria. In particular, we shall look for a maximum of efficiency for the complexity introduced in the restoration process by increasing the number of iterations from *m* to 1 *m* . In other words, let us seek a value of *m* from that on the improvement on desensitization is lower than the incremental complexity. In mathematical terms we can express this efficiency change as the second derivative of *Z m*( ) denoted by *Rm Z m* () () . It can be easily derived from (27) that

$$R(m) = (\hat{G}\hat{H})^m \ln(\hat{G}\hat{H}) \left[2 + (m+1)\ln(\hat{G}\hat{H})\right] \tag{28}$$

The purpose is to maximize this function as well as constrain it to the desensitization condition of *Z m*() 1 . In Santiago et al., 2007 we came up to a number of iterations as follows

$$\overline{m} = round\left\{-\left[1 + \frac{3}{\ln(\hat{G}\hat{H})}\right]\right\}\tag{29}$$

subject to a constraint on the regularization term ˆ ˆ 0.14 0.84 *GH* .

Finally, let us compute some numeric results of the main variables of the desensitization algorithm for different regularization products *GH*ˆ ˆ : *m* , *Z m*( ) , ( ) *<sup>r</sup> m* and ( ) *<sup>n</sup> m* , where

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 159

tuning basis. Regarding the interconnection weights, they do not require any network training so the weigh matrices are all initialized to zero. Finally, we set the stopping criteria as a maximum number of 500 iterations (though never reached) or when the relative difference of the restoration error *E m*( ) falls below a threshold of <sup>3</sup> 10 in a temporal

Fig. 6. Degraded and truncated image by diagonal motion blur (right) and the expected

particularize the standard deviation to the pixels of the boundary region *B* .

In order to measure the performance of our algorithm, we compute the standard deviation

 *<sup>e</sup>* of the error image **exx** ˆ since it does not depend on the blurred image **y** as in the ISNR (Banham and Katsaggelos, 1997). Regarding the boundary reconstruction process we

Our first experiment takes the Lena image degraded by several motion and uniform blurs. Regarding the motion blur, we establish 45º of inclination and the length of pixels is varied between 5 and 15. We have used the approximation of Matlab to construct the filter of motion which leads to masks between 5 5 and 11 11 in size. Analogously, the uniform blur is defined with odd sizes between 5 5 and 11 11 . In terms of Gaussian noise we set

The results of the MLP are shown in Table 2. As presumable, the quality of restoration is getting worse as the size of the blur increases, but let us remark that the boundary reconstruction area is also expanding. If we compare the results between blurs we can observe that the uniform mask achieves better values at the boundaries, but lower in the center for the same size. It can be thought of a spatial varying restoration process of the MLP

To visually assess the performance of the MLP we select some of the results indicated in the previous table. On the left of Fig. 7 we depict the Lena restored image for a diagonal motion blur of 10 pixels. The restored boundary area is 252 16 in size marked by a white broken line and reveals how the borders are successfully regenerated without neither any image

window of 10 iterations.

boundary region to be reconstructed (left).

in the center with respect to the boundaries.

**4.1.1 Experiment 1** 

a ratio of BSNR 20 dB .


these delta functions are respectively the relative error components (ringing and noise) expressed in dB.

Table 1. Numeric results for the main functions of the desensitized filter.

Looking at the figures of Table 1 we can see that the improvements achieved for ( ) *<sup>n</sup> m* are greater than the impairments obtained from ( ) *<sup>r</sup> m* , always satisfying the desensitization condition *Z m*() 1 . For that reason, we may expect to have good restoration results with a rough estimation of noise in a very wide range, much better than other kind of wrong parameters.

## **4. Experimental results**

In this section we aim to validate the properties of the previous algorithms using a variety of experiments with very well-known 256 256 sized images such as Lena, Barbara or Cameraman, or PSFs widely used in the field as the motion, Gaussian or uniforms blurs. Furthermore, we shall compare the results with classic approaches of image restoration to ensure the good performance of our iterative methods.

## **4.1 MLP experiments**

Let us see our problem formulation by means of an example. Fig. 6 depicts the original Barbara image blurred by a motion blur of 15 pixels and 45º of inclination, which turns out a PSF mask of 11 11 in size ( 1 2 *B B* 5 ). We have represented the truncated image **y***tru* on the right which reflects the zeros at the boundaries and the size of *L* 266 266 . A real model would consist of the FOV 246 246 region of this image which we named as **y***real* so far*.* Most recent algorithms deal with this real image but making assumptions about the boundaries and yielding a restored image of 246 246 . Consequently, the boundaries marked with the white broken line on the left are never restored. In contrast, our MLP outcomes a 256 256 sized image **x**ˆ reconstructing the boundary area 251 20 *B* .

To resolve this sort of problems we have implemented the MLP according to the following parameters. In the light of the expression (18) we have used the horizontal and vertical Sobel masks ( 3 3 *N* ) of Bovik, 2005 for the filters **<sup>ξ</sup> d** and **<sup>μ</sup> d** . We already commented that the learning speed of the net is set to 2 and the regularization parameter relies on a hand tuning basis. Regarding the interconnection weights, they do not require any network training so the weigh matrices are all initialized to zero. Finally, we set the stopping criteria as a maximum number of 500 iterations (though never reached) or when the relative difference of the restoration error *E m*( ) falls below a threshold of <sup>3</sup> 10 in a temporal window of 10 iterations.

Fig. 6. Degraded and truncated image by diagonal motion blur (right) and the expected boundary region to be reconstructed (left).

In order to measure the performance of our algorithm, we compute the standard deviation *<sup>e</sup>* of the error image **exx** ˆ since it does not depend on the blurred image **y** as in the ISNR (Banham and Katsaggelos, 1997). Regarding the boundary reconstruction process we particularize the standard deviation to the pixels of the boundary region *B* .

## **4.1.1 Experiment 1**

158 Image Restoration – Recent Advances and Applications

these delta functions are respectively the relative error components (ringing and noise)

0.20 1 0.40 9.15 -13.98 0.30 1 0.60 8.41 -10.46 0.40 2 0.48 9.43 -15.92 0.50 3 0.50 9.66 -18.06 0.60 5 0.47 9.97 -22.18 0.70 7 0.66 9.94 -21.69 0.80 12 0.89 10.03 -23.26

 *<sup>r</sup> m* ( ) *<sup>n</sup> m*

> *<sup>n</sup> m* are

*<sup>r</sup> m* , always satisfying the desensitization

relies on a hand

*GH*ˆ ˆ *m Z m*( ) ( )

Table 1. Numeric results for the main functions of the desensitized filter.

greater than the impairments obtained from ( )

ensure the good performance of our iterative methods.

Looking at the figures of Table 1 we can see that the improvements achieved for ( )

condition *Z m*() 1 . For that reason, we may expect to have good restoration results with a rough estimation of noise in a very wide range, much better than other kind of wrong

In this section we aim to validate the properties of the previous algorithms using a variety of experiments with very well-known 256 256 sized images such as Lena, Barbara or Cameraman, or PSFs widely used in the field as the motion, Gaussian or uniforms blurs. Furthermore, we shall compare the results with classic approaches of image restoration to

Let us see our problem formulation by means of an example. Fig. 6 depicts the original Barbara image blurred by a motion blur of 15 pixels and 45º of inclination, which turns out a PSF mask of 11 11 in size ( 1 2 *B B* 5 ). We have represented the truncated image **y***tru* on the right which reflects the zeros at the boundaries and the size of *L* 266 266 . A real model would consist of the FOV 246 246 region of this image which we named as **y***real* so far*.* Most recent algorithms deal with this real image but making assumptions about the boundaries and yielding a restored image of 246 246 . Consequently, the boundaries marked with the white broken line on the left are never restored. In contrast, our MLP

outcomes a 256 256 sized image **x**ˆ reconstructing the boundary area 251 20 *B* .

To resolve this sort of problems we have implemented the MLP according to the following parameters. In the light of the expression (18) we have used the horizontal and vertical Sobel masks ( 3 3 *N* ) of Bovik, 2005 for the filters **<sup>ξ</sup> d** and **<sup>μ</sup> d** . We already commented that the

and the regularization parameter

expressed in dB.

parameters.

**4. Experimental results** 

**4.1 MLP experiments** 

learning speed of the net is set to 2

Our first experiment takes the Lena image degraded by several motion and uniform blurs. Regarding the motion blur, we establish 45º of inclination and the length of pixels is varied between 5 and 15. We have used the approximation of Matlab to construct the filter of motion which leads to masks between 5 5 and 11 11 in size. Analogously, the uniform blur is defined with odd sizes between 5 5 and 11 11 . In terms of Gaussian noise we set a ratio of BSNR 20 dB .

The results of the MLP are shown in Table 2. As presumable, the quality of restoration is getting worse as the size of the blur increases, but let us remark that the boundary reconstruction area is also expanding. If we compare the results between blurs we can observe that the uniform mask achieves better values at the boundaries, but lower in the center for the same size. It can be thought of a spatial varying restoration process of the MLP in the center with respect to the boundaries.

To visually assess the performance of the MLP we select some of the results indicated in the previous table. On the left of Fig. 7 we depict the Lena restored image for a diagonal motion blur of 10 pixels. The restored boundary area is 252 16 in size marked by a white broken line and reveals how the borders are successfully regenerated without neither any image

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 161

view). We can observe that the restored images for each boundary condition are all 250 250 sized images which miss the information of the boundaries up to 256 256 . Furthermore, a remarkable boundary ringing can be appreciated for the periodic BCs as result of the discontinuity of the image in the boundaries. As demonstrated in Martinelli et al., 2006 the reflexive and the anti-reflexive conditions perform considerably better

The restored image of our MLP algorithm is shown on the bottom-right of Fig. 8 and makes obvious the good performance of the neural net. First, the boundary ringing is negligible without prior assumption on the boundary condition. Moreover, the visual aspect is better compared to the others which recalls the good properties of the TV regularizer. To

*<sup>e</sup>* for the reflexive BCs). Finally, the MLP is able to reconstruct the 253 12

Fig. 8. Restored images with HyBR under periodic (upper-left), reflective (upper-right) and

anti-reflective (bottom-left) BCs. Restored image with our MLP (bottom-right).

sized boundary region and outcomes the original image size of 256 256 .

*<sup>e</sup>* which is notably lower to the values of the HyBR algorithm

*<sup>e</sup>* of the MLP is measured only in the FOV

removing that boundary effect.

region. It leads to 12.47 *F*

(e.g. 12.99 *F*

numerically contrast the results, the parameter



information nor prior assumption on the boundary conditions. Likewise, we illustrate the restored image with a uniform blur of 7 7 on the right and a boundary region of 253 12 .

Table 2. Numeric values of *<sup>e</sup>* and *B<sup>e</sup>* for different sizes of degradation.

Fig. 7. Restored images of the MLP when using motion (left) and uniform (right) blurs.

## **4.1.2 Experiment 2**

This experiment aims to compare the performance of the MLP with other restoration algorithms which need BCs to deal with a realistic capture model: zero, periodic, reflective and anti-reflective as commented in Section 2.1. We have used the RestoreTools, 2007 library patched with the anti-reflective modification which implements the matrix-vector operations for every boundary condition. In particular, we have selected an algorithm of this library named as HyBR (hybrid bidiagonalization regularization) that is a modified version of the Tikhonov regularization.

Let us consider the Barbara image degraded by a 7 7 Gaussian blur and the same additive white noise of the previous experiments with BSNR 20 dB . Fig. 8 shows the restored images of the HyBR method from a real acquisition of FOV 250 250 in size (field of 160 Image Restoration – Recent Advances and Applications

information nor prior assumption on the boundary conditions. Likewise, we illustrate the restored image with a uniform blur of 7 7 on the right and a boundary region of 253 12 .

*<sup>e</sup>*

*<sup>e</sup> B*

*e*

7 7 7 10.35 27.23 **Uniform**

9 7 7 10.26 20.96 5 5 8.90 17.29 10 9 9 11.62 26.04 7 7 11.32 19.64 11 9 9 11.50 23.36 9 9 13.20 20.64 12 9 9 11.51 20.85 11 11 14.69 22.27

*<sup>e</sup>* for different sizes of degradation.

8 7 7 10.25 24.05 *Size*

Fig. 7. Restored images of the MLP when using motion (left) and uniform (right) blurs.

This experiment aims to compare the performance of the MLP with other restoration algorithms which need BCs to deal with a realistic capture model: zero, periodic, reflective and anti-reflective as commented in Section 2.1. We have used the RestoreTools, 2007 library patched with the anti-reflective modification which implements the matrix-vector operations for every boundary condition. In particular, we have selected an algorithm of this library named as HyBR (hybrid bidiagonalization regularization) that is a modified version

Let us consider the Barbara image degraded by a 7 7 Gaussian blur and the same additive white noise of the previous experiments with BSNR 20 dB . Fig. 8 shows the restored images of the HyBR method from a real acquisition of FOV 250 250 in size (field of

**Motion** 

5 5 5 8.70 24.59 6 5 5 8.70 20.58

13 11 11 12.78 25.85 14 11 11 12.61 23.15 15 11 11 12.63 21.10

> *<sup>e</sup>* and *B*

*<sup>e</sup> B*

*Length Size* 

Table 2. Numeric values of

**4.1.2 Experiment 2** 

of the Tikhonov regularization.

view). We can observe that the restored images for each boundary condition are all 250 250 sized images which miss the information of the boundaries up to 256 256 . Furthermore, a remarkable boundary ringing can be appreciated for the periodic BCs as result of the discontinuity of the image in the boundaries. As demonstrated in Martinelli et al., 2006 the reflexive and the anti-reflexive conditions perform considerably better removing that boundary effect.

The restored image of our MLP algorithm is shown on the bottom-right of Fig. 8 and makes obvious the good performance of the neural net. First, the boundary ringing is negligible without prior assumption on the boundary condition. Moreover, the visual aspect is better compared to the others which recalls the good properties of the TV regularizer. To numerically contrast the results, the parameter *<sup>e</sup>* of the MLP is measured only in the FOV region. It leads to 12.47 *F <sup>e</sup>* which is notably lower to the values of the HyBR algorithm (e.g. 12.99 *F <sup>e</sup>* for the reflexive BCs). Finally, the MLP is able to reconstruct the 253 12 sized boundary region and outcomes the original image size of 256 256 .

Fig. 8. Restored images with HyBR under periodic (upper-left), reflective (upper-right) and anti-reflective (bottom-left) BCs. Restored image with our MLP (bottom-right).

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 163

*Uniform* 13.23 17.43 12.99 13.53 15.05 13.45 *Gaussian* 12.49 17.79 12.18 12.33 14.13 12.24 *Motion* 11.37 17.63 10.97 11.33 12.58 11.27

Let us visually assess the performance of both methods for some experiments of Table 3. In particular, we have used two 250 250 sized images degraded by uniform and Gaussian blurs of 7 7 . The restored images appear in Fig. 9 with 256 256 in size and thus

for the Bishop's method in the uniform blur, we can observe that the subjective quality of the MLP output is better. As for the Gaussian blur the restored images look similar although

In this case our experiments aim to compare the performance of the desensitization filter *G* with respect to the classical filters *G* Wiener and Tikhonov when having errors on the estimations. So let us define a way to measure the deviations from the real value of the

be the relative error of a generic parameter *P* defined as follows

(30)

, even though we only consider the significant values ranged

). On the other hand, we shall focus on the motion blur so that we can

 *i*  *<sup>j</sup>* due to its dependence on the

is also extended

*<sup>n</sup>* of a Gaussian

100 *real estimated*

*real*

where *Preal* and *Pestimated* stand for the respective real and estimated values of the parameter

The types of parameters for these experiments have to do with the noise and blurs of

In terms of implementation let us recall that the proposed desensitization algorithm yields a

product *GH*ˆ ˆ . By using the expression (29) we obtain a value of *m* for those pairs whose regularization term *GH*ˆ ˆ is within the range ˆ ˆ 0.14 0.84 *GH* . Thus, a criterion will be adopted for choosing a number of iterations for the rest of frequencies. Owing to the increasing trend of *m* with respect to *GH*ˆ ˆ (see Table 1), all pairs whose corresponding regularization value exceeds 0.84 are associated to the upper bound of iterations and

 ( ).

*P P P*

**Blur** 

the value of

parameters. Let *<sup>P</sup>*

**4.2 Desensitization experiments** 

along the range *<sup>P</sup>*

between 100 and 100 %.

additive sample (

 

observe the effects of mistaking the angle

different number of iterations *m* for every pair ( , )

likewise the minimum value (cero) if *GH*ˆ ˆ is below 0.14.

*<sup>e</sup> B*

*<sup>e</sup>* is in favor of the neural net.

 *<sup>e</sup> <sup>e</sup> F*

Table 3. Comparison between Bishop's method and MLP for various PSFs

*P*

*P* . Provided that these parameters are real variables, the relative error *<sup>P</sup>*

previous experiments. As for the noise we shall deal with the variance <sup>2</sup>

reconstructing the boundary area 253 12 *B* . Despite the fact that the value of

**Bishop MLP** 

*<sup>e</sup> B*

 *<sup>e</sup> <sup>e</sup> F*

*<sup>e</sup>* is lower

## **4.1.3 Experiment 3**

Finally, let us compare with other algorithms of the literature which deal with the boundary problem in a different sense from the typical BCs and that reconstruct the area *B* bordering the field of view. In recent research Bishop, 2008 proposed a method based on the Bayesian model and treated the truncation effect as modeling error. To make a better comparison we have updated the MLP to leverage the concept of extended image of this method by removing the operator trunc from all formulae of Section 3.1 and setting the observed image **yreal** at the input of the MLP instead of the truncated image **ytru** .

Fig. 9. Restored images with Bishop's method: uniform (upper-left) and Gaussian (bottomleft) blurs. Likewise for MLP: uniform (upper-right) and Gaussian (bottom-right).

Looking at Table 3 we find out that the values of *<sup>e</sup>* are quite similar for both methods, being the MLP which outperforms in the Gaussian and motion blurs. But what really deserves attention are the results in the boundary region *B* . The MLP is considerably better reconstructing the missed boundaries as indicated by the lower values of *B <sup>e</sup>* . Then, it reveals the outstanding properties of the neural net in terms of learning about the unknown image.


Table 3. Comparison between Bishop's method and MLP for various PSFs

Let us visually assess the performance of both methods for some experiments of Table 3. In particular, we have used two 250 250 sized images degraded by uniform and Gaussian blurs of 7 7 . The restored images appear in Fig. 9 with 256 256 in size and thus reconstructing the boundary area 253 12 *B* . Despite the fact that the value of *<sup>e</sup>* is lower for the Bishop's method in the uniform blur, we can observe that the subjective quality of the MLP output is better. As for the Gaussian blur the restored images look similar although the value of *<sup>e</sup>* is in favor of the neural net.

## **4.2 Desensitization experiments**

162 Image Restoration – Recent Advances and Applications

Finally, let us compare with other algorithms of the literature which deal with the boundary problem in a different sense from the typical BCs and that reconstruct the area *B* bordering the field of view. In recent research Bishop, 2008 proposed a method based on the Bayesian model and treated the truncation effect as modeling error. To make a better comparison we have updated the MLP to leverage the concept of extended image of this method by removing the operator trunc from all formulae of Section 3.1 and setting the observed

Fig. 9. Restored images with Bishop's method: uniform (upper-left) and Gaussian (bottom-

the MLP which outperforms in the Gaussian and motion blurs. But what really deserves attention are the results in the boundary region *B* . The MLP is considerably better

the outstanding properties of the neural net in terms of learning about the unknown image.

*<sup>e</sup>* are quite similar for both methods, being

*<sup>e</sup>* . Then, it reveals

left) blurs. Likewise for MLP: uniform (upper-right) and Gaussian (bottom-right).

reconstructing the missed boundaries as indicated by the lower values of *B*

Looking at Table 3 we find out that the values of

image **yreal** at the input of the MLP instead of the truncated image **ytru** .

**4.1.3 Experiment 3** 

In this case our experiments aim to compare the performance of the desensitization filter *G* with respect to the classical filters *G* Wiener and Tikhonov when having errors on the estimations. So let us define a way to measure the deviations from the real value of the parameters. Let *<sup>P</sup>* be the relative error of a generic parameter *P* defined as follows

$$
\omega\_P = \frac{P\_{real} - P\_{estimated}}{P\_{real}} \cdot 100 \tag{30}
$$

where *Preal* and *Pestimated* stand for the respective real and estimated values of the parameter *P* . Provided that these parameters are real variables, the relative error *<sup>P</sup>* is also extended along the range *<sup>P</sup>* , even though we only consider the significant values ranged between 100 and 100 %.

The types of parameters for these experiments have to do with the noise and blurs of previous experiments. As for the noise we shall deal with the variance <sup>2</sup> *<sup>n</sup>* of a Gaussian additive sample ( ). On the other hand, we shall focus on the motion blur so that we can observe the effects of mistaking the angle ( ).

In terms of implementation let us recall that the proposed desensitization algorithm yields a different number of iterations *m* for every pair ( , ) *i <sup>j</sup>* due to its dependence on the product *GH*ˆ ˆ . By using the expression (29) we obtain a value of *m* for those pairs whose regularization term *GH*ˆ ˆ is within the range ˆ ˆ 0.14 0.84 *GH* . Thus, a criterion will be adopted for choosing a number of iterations for the rest of frequencies. Owing to the increasing trend of *m* with respect to *GH*ˆ ˆ (see Table 1), all pairs whose corresponding regularization value exceeds 0.84 are associated to the upper bound of iterations and likewise the minimum value (cero) if *GH*ˆ ˆ is below 0.14.

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 165

affected by the small error in noise estimation with a significant noise-dependent error on the textures of Barbara and Lena. However, our algorithm is able to counteract this effect and provide a restored image with a better visual aspect. The numeric figures of ISNR also make evidence of this situation. In Barbara we obtain a value of ISNR 2.46 dB for the Tikhnov filter whereas the desensitization clearly improves it with ISNR 2.38 dB . Analogously, we end up with ISNR 3.53 dB and ISNR 2.45 dB in the Lena example.

Fig. 10. ISNR for errors on estimations <sup>2</sup>

*<sup>n</sup>* and of Wiener filter.

In view of the expressions (14) and (15) let us do some remarks. First, the spectral density of the Gaussian white noise is just its variance <sup>2</sup> *Snn n* . As for the spectral density *Sxx* it is commonly estimated by means of the spectral density of the observed image *Syy* , which in turn is estimated by the periodogram approximation (Marple, 1987)

$$S\_{yy} = \frac{1}{L^2} \|Y\|^2 \tag{31}$$

Finally, the parameter of (15) is typically computed by the discrepancy principle (Bonesky, 2009) which establishes that

$$\left\|\mathbf{y} - \mathbf{H}\hat{\mathbf{x}}\right\|\_{2}^{2} = \left\|\mathbf{n}\right\|\_{2}^{2} = L\sigma\_{n}^{2} \tag{32}$$

In these experiments we do use the common ISNR (improvements on the signal-to-noise ratio) as the objective metric.

#### **4.2.1 Experiment 1**

In a first simulation we shall execute the desensitization filter for the whole range 100 100 *<sup>P</sup> ε* of the relative error of the parameters <sup>2</sup> *<sup>n</sup>* and . The original motion blur is described by a length of 15 pixels and an angle of 45 degrees in a counter-clockwise direction. And the Gaussian noise is added according to a specific BSNR of 20 dB . This experiment is computed for the two original filters Wiener and Tikhonov when facing a degraded image Cameraman.

In Fig. 10 we can observe the regions of desensitization for the Wiener filter. As for the noise estimation the desensitization filter outperforms from a specific value on. Regarding the angle estimation our method achieves better results outside a bandwidth. In Santiago, 2007 it is demonstrated that the desensitization method may completely outperform in case of high enough noise conditions.

If we look into the results of the Tikhonov filter in Fig. 11 we come up with better results as it is required a lower value of to be in the desensitization region (with less than 10%). This situation may be typical in a method of estimation of the noise variance and therefore our iterative scheme means a successful solution. With regard to the blur estimation the region of desensitization is practically the same as in the Wiener example, so it reveals the better behavior of our algorithm in case of the noise.

#### **4.2.2 Experiment 2**

Finally, we devote this section to visually analyze the results of the desensitization filter for the optimum case: noise estimation and Tikhonov filter. We shall use the Barbara and Lena images degraded by a Gaussian blur of size 10 10 , and we keep the same noise level as in previous experiments with BSNR 20 dB . The estimation error is fixed to 10%.

We have printed in Fig. 12 the restored images obtained by the Tikhonov and the desensitization filter in each case. It is remarkable how the Tikhonov algorithm is highly 164 Image Restoration – Recent Advances and Applications

In view of the expressions (14) and (15) let us do some remarks. First, the spectral density of

commonly estimated by means of the spectral density of the observed image *Syy* , which in

2 2 <sup>1</sup> *S Y yy <sup>L</sup>*

2 2 2

*<sup>n</sup>* and

our method achieves better results outside a bandwidth. In Santiago,

to be in the desensitization region (with less than 10%).

 

is fixed to 10%.

2 2 <sup>ˆ</sup> **y Hx n** *<sup>L</sup>*

In these experiments we do use the common ISNR (improvements on the signal-to-noise

In a first simulation we shall execute the desensitization filter for the whole range

is described by a length of 15 pixels and an angle of 45 degrees in a counter-clockwise direction. And the Gaussian noise is added according to a specific BSNR of 20 dB . This experiment is computed for the two original filters Wiener and Tikhonov when facing a

In Fig. 10 we can observe the regions of desensitization for the Wiener filter. As for the noise

2007 it is demonstrated that the desensitization method may completely outperform in case

If we look into the results of the Tikhonov filter in Fig. 11 we come up with better results as

This situation may be typical in a method of estimation of the noise variance and therefore

region of desensitization is practically the same as in the Wiener example, so it reveals the

Finally, we devote this section to visually analyze the results of the desensitization filter for the optimum case: noise estimation and Tikhonov filter. We shall use the Barbara and Lena images degraded by a Gaussian blur of size 10 10 , and we keep the same noise level as in

We have printed in Fig. 12 the restored images obtained by the Tikhonov and the desensitization filter in each case. It is remarkable how the Tikhonov algorithm is highly

our iterative scheme means a successful solution. With regard to the blur estimation

. As for the spectral density *Sxx* it is

*<sup>n</sup>* (32)

 

. The original motion blur

on. Regarding the

 the

(31)

of (15) is typically computed by the discrepancy principle

the Gaussian white noise is just its variance <sup>2</sup> *Snn n*

100 100 *<sup>P</sup> ε* of the relative error of the parameters <sup>2</sup>

estimation the desensitization filter outperforms from a specific value

 

previous experiments with BSNR 20 dB . The estimation error

better behavior of our algorithm in case of the noise.

(Bonesky, 2009) which establishes that

Finally, the parameter

ratio) as the objective metric.

degraded image Cameraman.

 

of high enough noise conditions.

it is required a lower value of

**4.2.1 Experiment 1** 

angle estimation

**4.2.2 Experiment 2** 

turn is estimated by the periodogram approximation (Marple, 1987)

affected by the small error in noise estimation with a significant noise-dependent error on the textures of Barbara and Lena. However, our algorithm is able to counteract this effect and provide a restored image with a better visual aspect. The numeric figures of ISNR also make evidence of this situation. In Barbara we obtain a value of ISNR 2.46 dB for the Tikhnov filter whereas the desensitization clearly improves it with ISNR 2.38 dB . Analogously, we end up with ISNR 3.53 dB and ISNR 2.45 dB in the Lena example.

Fig. 10. ISNR for errors on estimations <sup>2</sup> *<sup>n</sup>* and of Wiener filter.

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 167

Fig. 12. Restored images with Tikhonov filter (upper-left and bottom-left) compared to the

This chapter has addressed two well-known problems of the regularization solutions in image restoration: dependency of boundary conditions and sensitivity to parameters estimations. Following a similar iterative concept of restoration-degradation we have

On the one hand, we have presented a neural network which aims to restore a real observed image where the borders outside the field of view (FOV) have been truncated. The idea is to apply a TV-based regularization function in an iterative minimization of a MLP (Multilayer perceptron) according to a backpropagation strategy. It achieves to not only restore the center of the image following the optimum linear solution (the ringing artifact thus being

restored images of the desensitization filter (upper-right and bottom-right).

provided two algorithms in the spatial and frequency domain respectively.

negligible), but also reconstruct the boundary area without any prior.

**5. Conclusion** 

Fig. 11. ISNR for errors on estimations <sup>2</sup> *<sup>n</sup>* and of Tikhonov filter.

Fig. 12. Restored images with Tikhonov filter (upper-left and bottom-left) compared to the restored images of the desensitization filter (upper-right and bottom-right).

## **5. Conclusion**

166 Image Restoration – Recent Advances and Applications

Fig. 11. ISNR for errors on estimations <sup>2</sup>

*<sup>n</sup>* and of Tikhonov filter.

This chapter has addressed two well-known problems of the regularization solutions in image restoration: dependency of boundary conditions and sensitivity to parameters estimations. Following a similar iterative concept of restoration-degradation we have provided two algorithms in the spatial and frequency domain respectively.

On the one hand, we have presented a neural network which aims to restore a real observed image where the borders outside the field of view (FOV) have been truncated. The idea is to apply a TV-based regularization function in an iterative minimization of a MLP (Multilayer perceptron) according to a backpropagation strategy. It achieves to not only restore the center of the image following the optimum linear solution (the ringing artifact thus being negligible), but also reconstruct the boundary area without any prior.

Iterative Restoration Methods to Loose Estimations Dependency of Regularized Solutions 169

Liu, R. & Jia, J. (2008). Reducing boundary artifacts in image deconvolution, *Proceedings of*

Marple, S. L. (1987). *Digital Spectra Analysis: With Applications*, Prentice-Hall, ISBN 0-132-

Martinelli, A.; Donatelli, M.; Estatico, C. & Serra-Capizzano, S. (2006). Improved image

Molina, R.; Mateos, J. & Katsaggelos, A. K. (1999). Bayesian and regularization methods for

Molina, R.; Mateos, J. & Katsaggelos, A. K. (2006). Blind deconvolution using a variational

Ng, M. K.; Chan, R. H. & Wun-Cheung, T. (1999). A fast algorithm for deblurring models

Oliveira, J.; Bioucas-Dias, J. & Figueiredo, M. (2009). Adaptive total variation image

Osher, S.; Rudin, L. & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. *Physica D: Nonlinear Phenomena*, Vol. 60, pp. 259-268, ISSN 0167-2789 Paik, J. K. & Katsaggelos, A. K. (1992). Image restoration using a modied hopeld network. *IEEE Transactions on Image Processing*, Vol. 1, No. 1, pp. 49–63, ISSN 1057-7149

Santiago, M. A.; Cisneros, G. & Bernués, E. (2007). Iterative Desensitisation of Image

Santiago, M. A.; Cisneros, G. & Bernués, E. (2010). An MLP Neural Net with L1 and L2

Vogel, C. R. (2002). *Computational methods for inverse problems,* SIAM, ISBN 0-89871-550-4,

Wang, J.; Liao, X. & Yi, Z. (2005). Image restoration using hopfield neural network based on

*Signal Processing*, Vol. 2010, Article ID 394615, 18 pages, ISSN 1687-6172 Sun, Y. (2000). Hopeld neural network based algorithms for image restoration and

*Processing*, Vol. 48, No. 7, pp. 2119–2131, ISSN 1053-587X

10, pp. 692–695, ISBN 1-4244-0469-X, Tampa, Florida

Restoration Filters under Wrong PSF and Noise Estimates. *EURASIP Journal on Advances in Signal Processing*, Vol. 2007, Article ID 72658, 18 pages, ISSN 1687-

Regularizers for Real Conditions of Deblurring. *EURASIP Journal on Advances in* 

reconstruction – Part I: algorithms and simulations. *IEEE Transactions on Signal* 

total variational model, *Proceedings of the Second international conference on Advances in neural networks*, Vol. 2, pp. 735–740, ISBN 3-540-25913-9, Chongqing, China Woods, J.; Biemond, J. & Tekalp, A. (1985). Boundary Value Problem in Image Restoration,

*Proceedings of International Conference on Acoustics, Speech and Signal Processing*, Vol.

*Problems*, Vol. 22, No. 6, pp. 2035–2053, ISSN 0266-5611

*Processing*, Vol. 15, No. 12, pp. 3715–3727, ISSN 1057-7149

Restore Tools (2007). In: *Emory University*, 14.10.2011, Available from http://www.mathcs.emory.edu/~nagy/RestoreTools/

San Diego, USA

14149-3, Upper Saddle River, USA.

*Processing*, Vol. 8, No. 2, pp. 231-246

No. 3, pp. 851-866, ISSN 1064-8275

pp. 2479–2493, ISSN 0165-1684

6172

Philadelphia, USA

*International Conference on Image Processing*, pp. 505–508, ISBN 978-1-4244-1765-0,

deblurring with anti-reective boundary conditions and re-blurring. *Inverse* 

hyperparameter estimation in image restoration. *IEEE Transactions on Image* 

approach to parameter, image and blur estimation. *IEEE Transactions on Image* 

with Neumann boundary conditions. *SIAM Journal on Scientific Computing*, Vol. 21,

deblurring: a majorization-minimization approach. *Signal Processing*, Vol. 89, No. 9,

The proposed restoration scheme has been validated by means of several tests. As a result, we can conclude the ability of our neural net to deal with the non-linearity of border truncation and its learning properties about the degradation model so as to regenerate the missed boundaries. In fact, it clearly outperforms when comparing with other methods of the state-of-the-art which also try to inpaint the boundary area.

The second algorithm of this chapter outcomes a frequency-based restoration filter which desensitizes an original method when having errors on its parameters. By means of an iterative sequence of restoration-degradation processes for each frequency pair we come up with a trade-off between desensitization and restoration error. In particular, the noisedependent error is more robust to estimations than the ringing error which gets higher as the iterations increase.

Various tests demonstrate that the region of desensitization is located from a low value of parameters errors, being more evident in the noise variance and using the Tikhonov filter. We observed the undesirable effects on the original filter in spite of the low error, while our desensitized filter counteract this noise error with successful results.

## **6. References**


168 Image Restoration – Recent Advances and Applications

The proposed restoration scheme has been validated by means of several tests. As a result, we can conclude the ability of our neural net to deal with the non-linearity of border truncation and its learning properties about the degradation model so as to regenerate the missed boundaries. In fact, it clearly outperforms when comparing with other methods of

The second algorithm of this chapter outcomes a frequency-based restoration filter which desensitizes an original method when having errors on its parameters. By means of an iterative sequence of restoration-degradation processes for each frequency pair we come up with a trade-off between desensitization and restoration error. In particular, the noisedependent error is more robust to estimations than the ringing error which gets higher as

Various tests demonstrate that the region of desensitization is located from a low value of parameters errors, being more evident in the noise variance and using the Tikhonov filter. We observed the undesirable effects on the original filter in spite of the low error, while our

Banham, M. R. & Katsaggelos, A. K. (1997). Digital Image Restoration. *IEEE Signal Processing* 

Bertero, M. & Bocacci, P. (1998). *Introduction to Inverse Problems in Imaging*, Institute of

Bishop, T. E. (2008). *Bayesian image deblurring and boundary effects*, PhD Thesis, University of

Bioucas-Dias, J.; Figueiredo, M. & Oliveira, J. P. (2006). Total variation-based image

Bonesky, T. (2009). Morozov's discrepancy principle and Tikhonov-type functionals. *Inverse Problems*, Vol. 25, No. 1, Article ID 015015, 11 pages, ISSN 0266-5611 Bovik, A. (2005). *Handbook of Image & Video Processing* (2nd edition), Elsevier, ISBN 0-12-

Calvetti, D. & Somersalo, E. (2005). Statistical elimination of boundary artifacts in image deblurring. *Inverse Problems*, Vol. 21, No. 5, pp. 1697–1714, ISSN 0266-5611 Campisi, P. & Egiazarian, K. (2007). *Blind image deconvolution: theory and applications*, CRC

Chan, T.F. & Shen, J. (2005). *Image Processing and Analysis Variational, PDE, Wavelet and* 

González, R. C. & Woods, R. E. (2007). *Digital Image Processing* (3rd edition)*,* Prentice Hall,

Han, Y. B. & Wu, L. N. (2004). Image restoration using a modied hopeld neural network

of continuous state change. *Signal Processing*, Vol. 12, No. 3, pp.431–435, ISSN 1003-

*Stochastic Methods*, SIAM, ISBN 0-89871-589-X, Philadelphia, USA

deconvolution: a majorization–minimization approach, *Proceedings of International Conference on Acoustics, Speech and Signal Processing*, Vol. 2, pp. 861–864, ISBN 1-

the state-of-the-art which also try to inpaint the boundary area.

desensitized filter counteract this noise error with successful results.

*Magazine*, Vol. 14, No. 2, pp. 24–41, ISSN 1053-5888

Physics Publishing, ISBN 0750304359

4244-0469-X, Tolouse, France

119792-1, Burlington, USA

ISBN 0-13-168728-X

0530

Press, ISBN 0-8493-7367-0, New York, USA

the iterations increase.

**6. References** 

Edimburg, UK


**0**

**8**

Liyun Su

*China*

**Defocused Image Restoration with Local**

*School of Mathematics and Statistics, Chongqing University of Technology*

Shooting a real world image with a camera through an optical device gives a 2-D image where at least some parts are affected by a blur and noise. Images can be blurred by atmospheric turbulence, relative motion between sensors and objects, longer exposures, and so on, but the exact cause of blurring may be unknown. Restoration of blurred noisy images (Spiros et al., 2009; 2010; Su et al., 2007) is one of the main topics in many processing. The literatures Alonso et al. (2008; 2005); Bar et al. (2006) have given good methods to improve image qualities. The purpose of image restoration is to reconstruct an unobservable true image from a degraded observation. An observed image can be written, ignoring additive noise, as the two-dimensional (2-D) convolution of the true image with a linear space-invariant (*LSI*) blur, known as the *PSF*. Restoration in the case of known blur, assuming the linear degradation model, is called linear image restoration and it has been presented extensively in the last three decades giving rise to a variety of solutions Chen et al. (2000); Suyash et al. (2006); Gu et al. (2009); Lu et al. (2009) . In many practical situations, however, the blur is unknown. Hence, both blur identification and image restoration must be performed from the degraded image. Restoration in the case of unknown blur is called blind image restoration Filip et al. (2003); Mario et al. (2003); Liao et al. (2005) . Existing blind restoration methods can be categorized into two main groups: (i) those which estimate the *PSF* a priori independent of the true image so as to use it later with one of the linear image restoration methods, such as zero sheet separation, generalized cross validation, and maximum likelihood and expectation maximization based on the *ARMA* image model Chang et al., (1991); Reeves et al. (1992); Lagendijk et al. (1990) , and (ii) those which estimate the *PSF* and the true image simultaneously, such as nonnegative sand support constraints recursive inverse filtering, maximum likelihood and conjugate gradient minimization, and simulated annealing Kundur et al. (1998); Katsaggelos et al. (1991) . Algorithms belonging to the first class are computationally simple, but they are limited to situations in which the *PSF* has a special form, and the true image has certain features. Algorithms belonging to the second class, which are computationally more complex, must be used for more general situations. In this paper, a kind of semi-blind image restoration algorithm is proposed in case of known the

In general, discrete model for a linear degradation caused by blurring can be given by the

*y*(*i*, *j*) = *h*(*i*, *j*) ∗ *f*(*i*, *j*) + *n*(*i*, *j*) (1)

**1. Introduction**

blur type (defocused blurring).

following equation

**Polynomial Regression and IWF**

Wu, Y. D.; Sun, Y.; Zhang, H. Y. & Sun, S. X. (2007). Variational PDE based image restoration using neural network. *IET Image Processing*, Vol. 1, No. 1, pp. 85–93, ISSN 1751-9659

## **Defocused Image Restoration with Local Polynomial Regression and IWF**

Liyun Su

*School of Mathematics and Statistics, Chongqing University of Technology China*

#### **1. Introduction**

170 Image Restoration – Recent Advances and Applications

Wu, Y. D.; Sun, Y.; Zhang, H. Y. & Sun, S. X. (2007). Variational PDE based image restoration

using neural network. *IET Image Processing*, Vol. 1, No. 1, pp. 85–93, ISSN 1751-9659

Shooting a real world image with a camera through an optical device gives a 2-D image where at least some parts are affected by a blur and noise. Images can be blurred by atmospheric turbulence, relative motion between sensors and objects, longer exposures, and so on, but the exact cause of blurring may be unknown. Restoration of blurred noisy images (Spiros et al., 2009; 2010; Su et al., 2007) is one of the main topics in many processing. The literatures Alonso et al. (2008; 2005); Bar et al. (2006) have given good methods to improve image qualities. The purpose of image restoration is to reconstruct an unobservable true image from a degraded observation. An observed image can be written, ignoring additive noise, as the two-dimensional (2-D) convolution of the true image with a linear space-invariant (*LSI*) blur, known as the *PSF*. Restoration in the case of known blur, assuming the linear degradation model, is called linear image restoration and it has been presented extensively in the last three decades giving rise to a variety of solutions Chen et al. (2000); Suyash et al. (2006); Gu et al. (2009); Lu et al. (2009) . In many practical situations, however, the blur is unknown. Hence, both blur identification and image restoration must be performed from the degraded image. Restoration in the case of unknown blur is called blind image restoration Filip et al. (2003); Mario et al. (2003); Liao et al. (2005) . Existing blind restoration methods can be categorized into two main groups: (i) those which estimate the *PSF* a priori independent of the true image so as to use it later with one of the linear image restoration methods, such as zero sheet separation, generalized cross validation, and maximum likelihood and expectation maximization based on the *ARMA* image model Chang et al., (1991); Reeves et al. (1992); Lagendijk et al. (1990) , and (ii) those which estimate the *PSF* and the true image simultaneously, such as nonnegative sand support constraints recursive inverse filtering, maximum likelihood and conjugate gradient minimization, and simulated annealing Kundur et al. (1998); Katsaggelos et al. (1991) . Algorithms belonging to the first class are computationally simple, but they are limited to situations in which the *PSF* has a special form, and the true image has certain features. Algorithms belonging to the second class, which are computationally more complex, must be used for more general situations. In this paper, a kind of semi-blind image restoration algorithm is proposed in case of known the blur type (defocused blurring).

In general, discrete model for a linear degradation caused by blurring can be given by the following equation

$$(y(i,j) = h(i,j) \* f(i,j) + n(i,j) \tag{1}$$

by a factor of 2. The *l* level wavelet decomposition of an image *I* results in an approximation image *Xl* and three detail images *Hl*, *Vl*, and *Dl* in horizontal, vertical, and diagonal directions respectively. Decomposition into *l* levels of an original image results in a down sampled image

Defocused Image Restoration with Local Polynomial Regression and IWF 173

When an image is defocused, edged in it are smoothed and widened. The amount of high

In order to denote the relationship between wavelet coefficients and defocused radius *R*, we

where |·|*<sup>s</sup>* represents the summation of all coefficients' absolute value, *num*{·} is total number

An original image is blurred artificially by a uniform defocus *PSF* with *R* whose value ranging from 1 to 20. The relationship between *v*1, *v*2, *v*3, *v*4, *v*<sup>5</sup> and *R* are shown in Fig.1, where the curves are normalized in [0,1] interval. When *R* increases, *v*2, *v*3, *v*<sup>4</sup> and *v*<sup>5</sup> decrease

cameraman

v1 v2 v3 v4 v5

v1 v2 v3 v4 v5

0 5 10 15 20

rice

0 5 10 15 20

In order to estimate defocus parameter *R*, only known the roughly similar relationship is not enough. As shown in Fig. 2, every image has monotonous curve between *v*2, *v*<sup>5</sup> and *R*, but they are not superposition. For a degraded unknown *PSF* image, *R* can not be calculated

(4)

*v*<sup>1</sup> = |*V*2|*s*/|*H*2|*<sup>s</sup> v*<sup>2</sup> = |*H*2|*s*/|*X*2|*<sup>s</sup> v*<sup>3</sup> = |*H*1|*s*/*num*{*H*1} *v*<sup>4</sup> = |*H*2|*s*/*num*{*H*2} *v*<sup>5</sup> = |*D*1|*s*/*num*{*D*1}

frequency band decreased, and that corresponding to low frequency band increases.

of resolution 2*<sup>l</sup>* with respect to the image as well as detail images.

⎧ ⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎩

define five variables named *v*1, *v*2, *v*3, *v*4, and *v*<sup>5</sup> as:

0 0.2 0.4 0.6 0.8 1

0 0.2 0.4 0.6 0.8 1

Fig. 1. Relationship Between *v*1−<sup>5</sup> and *R*

of coefficients.

monotonously.

where \* indicates two-dimensional convolution, *f*(*i*, *j*) represents on original image, *y*(*i*, *j*) is the degraded image, *h*(*i*, *j*) represents the two-dimensional *PSF*, and *n*(*i*, *j*) is the additive noise. In this article, we deal only with additive Gaussian noise, as it effectively models the noise in many different imaging scenarios. The difficulty in solving the restoration problem with a spatially varying blur commonly motivates the use of a stationary model for the blur. This leads to the following expression for the degradation system,

$$y(i,j) = h(i,j) \* f(i,j) + n(i,j) = \sum\_{k=1}^{M} \sum\_{l=1}^{N} h(i-k, j-l)f(k,l) + n(i,j) \tag{2}$$

The use of linear techniques for solving the restoration problem is facilitated by using space-invariant model. Models that utilize space-variant degradations are also common, but lead to more complex solutions. As for defocused blur, *PSF* is modeled as a uniform intensity distribution within a circular disk,

$$h(i,j) = \begin{cases} \frac{1}{\pi R^2} & \text{if } \sqrt{i^2 + j^2} \le R\\ 0 & \text{otherwise} \end{cases} \tag{3}$$

where disk radius *R* is the only unknown parameter for this type of blur.

Many existing image restoration algorithms assume that the *PSF* is known, but in practical it is not always the case. The restoration without knowing of the *PSF* is called blind image restoration. Fourier methods can be used to estimate the defocused parameter *R* through calculating a ratio of power of high frequencies portion to that of low frequencies portion. However, a main drawback of the method is its bad noise immunity. To solve this problem, a novel algorithm is proposed to overcome this shortcoming based on *RBF* neural network and iterative Wiener filtering. The *RBF* neural network is applied to fit *R*. This scheme has good fitting, but bad prediction. To avoid the weak generalization ability, a more efficient method for estimating parameter *R* is also proposed. The prediction ability of these two methods is compared with the trained five images. The steps of the presented algorithm in this chapter is as follows: Firstly we construct feature vectors of several blurred images with known defocused radius *R* in wavelet domain, then a *RBF* neural network or a multivariate local polynomial estimation model is trained using the vectors as inputs and defocused parameters as outputs. After the model is trained, the new defocused images are applied to the trained model for predicting the parameter *R*. For a semi-blind defocused image, *R* can be estimated through calculating the feature vectors and using it as input of the trained model. With known radius *R*, many traditional algorithm could be applied to restore the degraded image. In this chapter, iterative Wiener filtering(*IWF*) is adopted to image restoration.

#### **2. Relationship between wavelet coefficients and** *R*

The wavelet transform provides a powerful and versatile framework for image processing. It is widely used in the fields of image de-noising, compression, fusion, image restoration Patrick et al. (2004); Zhou et al. (2007); Guo et al. (2007), etc.

The two-dimensional discrete wavelet transform (*DWT*) Li et al. (2009; 2010) hierarchically decompose an input image into a series of successively lower resolution images and their associated detail images. *DWT* is implemented by a set of filters, which are convolved with the image rows and columns. An image is convolved with low-pass and high-pass filters and the odd samples of the filtered outputs are discarded resulting in down sampling the image 2 Image Restoration

where \* indicates two-dimensional convolution, *f*(*i*, *j*) represents on original image, *y*(*i*, *j*) is the degraded image, *h*(*i*, *j*) represents the two-dimensional *PSF*, and *n*(*i*, *j*) is the additive noise. In this article, we deal only with additive Gaussian noise, as it effectively models the noise in many different imaging scenarios. The difficulty in solving the restoration problem with a spatially varying blur commonly motivates the use of a stationary model for the blur.

> *M* ∑ *k*=1

The use of linear techniques for solving the restoration problem is facilitated by using space-invariant model. Models that utilize space-variant degradations are also common, but lead to more complex solutions. As for defocused blur, *PSF* is modeled as a uniform intensity

Many existing image restoration algorithms assume that the *PSF* is known, but in practical it is not always the case. The restoration without knowing of the *PSF* is called blind image restoration. Fourier methods can be used to estimate the defocused parameter *R* through calculating a ratio of power of high frequencies portion to that of low frequencies portion. However, a main drawback of the method is its bad noise immunity. To solve this problem, a novel algorithm is proposed to overcome this shortcoming based on *RBF* neural network and iterative Wiener filtering. The *RBF* neural network is applied to fit *R*. This scheme has good fitting, but bad prediction. To avoid the weak generalization ability, a more efficient method for estimating parameter *R* is also proposed. The prediction ability of these two methods is compared with the trained five images. The steps of the presented algorithm in this chapter is as follows: Firstly we construct feature vectors of several blurred images with known defocused radius *R* in wavelet domain, then a *RBF* neural network or a multivariate local polynomial estimation model is trained using the vectors as inputs and defocused parameters as outputs. After the model is trained, the new defocused images are applied to the trained model for predicting the parameter *R*. For a semi-blind defocused image, *R* can be estimated through calculating the feature vectors and using it as input of the trained model. With known radius *R*, many traditional algorithm could be applied to restore the degraded image. In this

The wavelet transform provides a powerful and versatile framework for image processing. It is widely used in the fields of image de-noising, compression, fusion, image restoration

The two-dimensional discrete wavelet transform (*DWT*) Li et al. (2009; 2010) hierarchically decompose an input image into a series of successively lower resolution images and their associated detail images. *DWT* is implemented by a set of filters, which are convolved with the image rows and columns. An image is convolved with low-pass and high-pass filters and the odd samples of the filtered outputs are discarded resulting in down sampling the image

*N* ∑ *l*=1

*<sup>π</sup>R*<sup>2</sup> if *<sup>i</sup>*<sup>2</sup> <sup>+</sup> *<sup>j</sup>*<sup>2</sup> <sup>≤</sup> *<sup>R</sup>*

*h*(*i* − *k*, *j* − *l*)*f*(*k*, *l*) + *n*(*i*, *j*) (2)

0 otherwise (3)

This leads to the following expression for the degradation system,

*h*(*i*, *j*) =

where disk radius *R* is the only unknown parameter for this type of blur.

chapter, iterative Wiener filtering(*IWF*) is adopted to image restoration.

**2. Relationship between wavelet coefficients and** *R*

Patrick et al. (2004); Zhou et al. (2007); Guo et al. (2007), etc.

1

*y*(*i*, *j*) = *h*(*i*, *j*) ∗ *f*(*i*, *j*) + *n*(*i*, *j*) =

distribution within a circular disk,

by a factor of 2. The *l* level wavelet decomposition of an image *I* results in an approximation image *Xl* and three detail images *Hl*, *Vl*, and *Dl* in horizontal, vertical, and diagonal directions respectively. Decomposition into *l* levels of an original image results in a down sampled image of resolution 2*<sup>l</sup>* with respect to the image as well as detail images.

When an image is defocused, edged in it are smoothed and widened. The amount of high frequency band decreased, and that corresponding to low frequency band increases.

In order to denote the relationship between wavelet coefficients and defocused radius *R*, we define five variables named *v*1, *v*2, *v*3, *v*4, and *v*<sup>5</sup> as:

$$\begin{cases} v\_1 = |V\_2|\_s / |H\_2|\_s\\ v\_2 = |H\_2|\_s / |X\_2|\_s\\ v\_3 = |H\_1|\_s / num\{H\_1\}\\ v\_4 = |H\_2|\_s / num\{H\_2\}\\ v\_5 = |D\_1|\_s / num\{D\_1\} \end{cases} \tag{4}$$

where |·|*<sup>s</sup>* represents the summation of all coefficients' absolute value, *num*{·} is total number of coefficients.

An original image is blurred artificially by a uniform defocus *PSF* with *R* whose value ranging from 1 to 20. The relationship between *v*1, *v*2, *v*3, *v*4, *v*<sup>5</sup> and *R* are shown in Fig.1, where the curves are normalized in [0,1] interval. When *R* increases, *v*2, *v*3, *v*<sup>4</sup> and *v*<sup>5</sup> decrease monotonously.

Fig. 1. Relationship Between *v*1−<sup>5</sup> and *R*

In order to estimate defocus parameter *R*, only known the roughly similar relationship is not enough. As shown in Fig. 2, every image has monotonous curve between *v*2, *v*<sup>5</sup> and *R*, but they are not superposition. For a degraded unknown *PSF* image, *R* can not be calculated

Fig. 3. Defocus Parameter Estimation Process

*<sup>ϕ</sup>j*(� *<sup>χ</sup>* <sup>−</sup> *<sup>μ</sup><sup>j</sup>* �) = *exp*[<sup>−</sup> <sup>1</sup>

feature vectors are constructed using variables *p*1−<sup>5</sup> of each image:

(0, 0, 0, 0, 0, 1)*T*; for *R* = 3, *t* = (0, 0, 0, 0, 1, 0)*T*, and so on.

*T*Φ†, Φ† is pseudo-inverse of Φ, where Φ is a matrix:

Φ =

⎛

⎜⎜⎜⎝

. . . . . 2*γ*<sup>2</sup> *j*

where *μ<sup>j</sup>* is the center and *γ<sup>j</sup>* is the standard deviation of the Gaussian function, respectively. Sixteen original images are chosen to train the *RBF* net. The images are defocused artificially with *R* whose value ranging from 2 to 7. So the total number of training samples are 96. Then

Defocused Image Restoration with Local Polynomial Regression and IWF 175

For the network output vector, we use one-of-*k* encoding method, that is, for *R* =2, *t* =

1 ··· 1 *ϕ*(||*χ*<sup>1</sup> − *μ*1||) ··· *ϕ*(||*χ*<sup>96</sup> − *μ*1||)

*ϕ*(||*χ*<sup>1</sup> − *μα*||) ··· *ϕ*(||*χ*<sup>96</sup> − *μα*||)

After obtaining weights matrix *W*, the defocused parameter *R* can be calculated using the

. .

. .

� *<sup>χ</sup>* <sup>−</sup> *<sup>μ</sup><sup>j</sup>* �2] (6)

*χ* = (*p*1, *p*2, *p*3, *p*4, *p*5) (7)

*<sup>i</sup>*=<sup>1</sup> are given, the weights matrix *W* can be obtained as *W* =

⎞

⎟⎟⎟⎠

(8)

functions are chosen as of Gaussian type:

When training samples {*χi*, *ti*}<sup>96</sup>

and *T* = (*t*1, *t*2, ··· , *t*96).

trained *RBF* network.

Fig. 2. Curve *v*2, and *v*<sup>5</sup> of Different Images

because the curve of the given image is not known. For example, if *v*<sup>2</sup> of image "rice" has been calculated, and then we estimate *R* according curve if "ic" in Fig. 2, wrong results are obtained obviously. To solve this problem, one of the methods is to choose neural networks. Computational artificial neural networks are known to have the capability for performing complex mappings between input and output data, but neural network method has bad generalization ability. Here we also propose another multivariate local polynomial regression model to estimate *R*. The variables *v*1−<sup>5</sup> are chosen to train the *RBF* neural network and multivariate local polynomial estimation model. Prediction Comparisons are made to verify the advantages of multivariate local polynomial fitting.

#### **3. Training** *RBF* **neural network and multivariate local polynomial estimation model**

#### **3.1** *RBF* **neural network for defocused parameter**

We propose and implement a parameter estimation technique in this section. Fig. 3 shows the description of this technique. In the first phase a *RBF* neural network is designed and trained. In the second phase *R* can be estimated using the trained neural network. A brief description of this technique is given in the following paragraphs.

RBF neural network is a most commonly-used feed-forward network. It usually has one hidden layer, and the basis function is radial symmetry. The output of the network looks like:

$$y\_k(\chi) = \sum\_{j=1}^{n} w\_{kj} \rho\_j(\chi) + w\_{k0} \Leftrightarrow y(\chi) = W\rho(\chi) \tag{5}$$

where *χ* is a put vector, *wk*<sup>0</sup> is a set of bias constants, *ϕ*0(� *χ* − *μ<sup>j</sup>* �) ≡ 1, *α* is the number of *RBF* hidden neurons and *W* holds both weights and bias. In the experiments, the radial basis 4 Image Restoration

because the curve of the given image is not known. For example, if *v*<sup>2</sup> of image "rice" has been calculated, and then we estimate *R* according curve if "ic" in Fig. 2, wrong results are obtained obviously. To solve this problem, one of the methods is to choose neural networks. Computational artificial neural networks are known to have the capability for performing complex mappings between input and output data, but neural network method has bad generalization ability. Here we also propose another multivariate local polynomial regression model to estimate *R*. The variables *v*1−<sup>5</sup> are chosen to train the *RBF* neural network and multivariate local polynomial estimation model. Prediction Comparisons are made to verify

**3. Training** *RBF* **neural network and multivariate local polynomial estimation model**

We propose and implement a parameter estimation technique in this section. Fig. 3 shows the description of this technique. In the first phase a *RBF* neural network is designed and trained. In the second phase *R* can be estimated using the trained neural network. A brief description

RBF neural network is a most commonly-used feed-forward network. It usually has one hidden layer, and the basis function is radial symmetry. The output of the network looks

where *χ* is a put vector, *wk*<sup>0</sup> is a set of bias constants, *ϕ*0(� *χ* − *μ<sup>j</sup>* �) ≡ 1, *α* is the number of *RBF* hidden neurons and *W* holds both weights and bias. In the experiments, the radial basis

*wkjϕj*(*χ*) + *wk*<sup>0</sup> ⇔ *y*(*χ*) = *Wϕ*(*χ*) (5)

Fig. 2. Curve *v*2, and *v*<sup>5</sup> of Different Images

the advantages of multivariate local polynomial fitting.

**3.1** *RBF* **neural network for defocused parameter**

of this technique is given in the following paragraphs.

*yk*(*χ*) =

*α* ∑ *j*=1

like:

Fig. 3. Defocus Parameter Estimation Process

functions are chosen as of Gaussian type:

$$\log(\left|\left\|\left.\chi-\mu\_{j}\right\|\right|)=\exp[-\frac{1}{2\gamma\_{j}^{2}}\left\|\left.\chi-\mu\_{j}\right\|\right\|^{2}]\tag{6}$$

where *μ<sup>j</sup>* is the center and *γ<sup>j</sup>* is the standard deviation of the Gaussian function, respectively.

Sixteen original images are chosen to train the *RBF* net. The images are defocused artificially with *R* whose value ranging from 2 to 7. So the total number of training samples are 96. Then feature vectors are constructed using variables *p*1−<sup>5</sup> of each image:

$$\chi = (p\_1, p\_2, p\_3, p\_4, p\_5) \tag{7}$$

For the network output vector, we use one-of-*k* encoding method, that is, for *R* =2, *t* = (0, 0, 0, 0, 0, 1)*T*; for *R* = 3, *t* = (0, 0, 0, 0, 1, 0)*T*, and so on.

When training samples {*χi*, *ti*}<sup>96</sup> *<sup>i</sup>*=<sup>1</sup> are given, the weights matrix *W* can be obtained as *W* = *T*Φ†, Φ† is pseudo-inverse of Φ, where Φ is a matrix:

$$\Phi = \begin{pmatrix} 1 & \cdots & 1 \\ \varphi(||\chi\_1 - \mu\_1||) \ \cdots \ \varphi(||\chi\_{96} - \mu\_1||) \\ \vdots & \vdots & \vdots \\ \varphi(||\chi\_1 - \mu\_a||) \ \cdots \ \varphi(||\chi\_{96} - \mu\_a||) \end{pmatrix} \tag{8}$$

and *T* = (*t*1, *t*2, ··· , *t*96).

After obtaining weights matrix *W*, the defocused parameter *R* can be calculated using the trained *RBF* network.

The localization scheme at a point *x* assigns the weight

**3.2.2 Multivariate predictor with local polynomial fitting**

Our purpose is to obtain the estimation *R*ˆ = ˆ

*<sup>f</sup>*(*V*) ≈ ∑

∑ 0≤|*j*|≤*d*

> *A* ∑ *a*=1

*fi* is determined by minimizing

= *d* ∑ |*j*|=0 ( |*j*| ∑ *j*1=0

> *∂v j*1 <sup>1</sup> *∂v j*2 <sup>2</sup> ··· *∂v*

[*Ra* − ∑

0≤|*j*|≤*d*

For the weighted least squared problem, a matrix form can be described by

*<sup>D</sup>*(*j*) *fi*(*VT*) = *<sup>∂</sup>*|*j*<sup>|</sup>

where

where

0≤|*j*|≤*d*

1 *j*!

*KH*(*Xi* − *x*), *with KH*(*x*) = |*H*|

designed into *H* = *hIm* (*Im* denoting the identity matrix of order *m*).

where |*H*| is the determinant of the matrix *H*. The bandwidth matrix is introduced to accommodate the dependent structure in the independent variables. For practical problems, the bandwidth matrix *H* is taken to be a diagonal matrix. The different independent variables will be accommodated into different scales. For simplification, the bandwidth matrix is

Defocused Image Restoration with Local Polynomial Regression and IWF 177

Suppose that the input vector is *V* = (*v*1, *v*2, *v*3, *v*4, *v*5). The model is fitted by the function

order multivariate local polynomial *f*(*V*) to predict the defocused parameter *RT* value based

*<sup>j</sup>* <sup>=</sup> ∑ 0≤|*j*|≤*d*

, *V<sup>j</sup>* = *v*

<sup>|</sup>*V*=*VT* , *bj*(*VT*) = <sup>1</sup>

*j* ]

*j*1 1 *v j*2 <sup>2</sup> ··· *v jm*

*j*!

*<sup>W</sup>*1/2 · *<sup>Y</sup>* <sup>=</sup> *<sup>W</sup>*1/2 · *<sup>X</sup>* · *<sup>B</sup>* <sup>+</sup> *<sup>ε</sup>* (21)

*<sup>Y</sup>* = (*y*1, *<sup>y</sup>*2, ··· , *yA*)*T*, *ya* <sup>=</sup> *Ra*, (22) *<sup>B</sup>* = (*b*0(*VT*), *<sup>b</sup>*1(*VT*), ··· , *bd*(*VT*))*T*, (23)

on the point *VT* of the test image. The polynomial function can be described as

*<sup>D</sup>*(*j*) *fi*(*VT*)(*<sup>V</sup>* <sup>−</sup> *VT*)

*m* = 5, *j* = (*j*1, *j*2, ··· , *jm*), *j*! = *j*1!*j*2! ··· *jm*!, |*j*| =



*fi*(*V*)


*jm m*

In the multivariate prediction method, *VTa* (*a* = 1, 2, ··· , *A*) denoting the trained image feature vectors. Using *A* pairs of (*VTa* , *Ra*), for which the values are already known, the coefficients of

*bj*(*VT*)(*VTa* − *VT*)

<sup>−</sup>1*K*(*H*−1*x*), (14)

*R* = *f*(*V*). (15)

*f*(*V*) of function *f* . This paper, we use the *d*th

*bj*(*VT*)(*V* − *VT*)

*m* ∑ *l*=1 *<sup>j</sup>* (16)

*jl*, (17)

*<sup>m</sup>* , (18)

*<sup>D</sup>*(*j*) *fi*(*VT*). (19)

<sup>2</sup> · *KH*(*VTa* <sup>−</sup> *VT*) (20)

#### **3.2 Multivariate local polynomial regression for defocused parameter**

Multivariate local polynomial fitting is an attractive method both from theoretical and practical point of view. Multivariate local polynomial method has a small mean squared error compared with the *Nadaraya* − *Watson* estimator which leads to an undesirable form of the bias and the *Gasser* − *Muller* estimator which has to pay a price in variance when dealing with a random design model. Multivariate local polynomial fitting also has other advantages. The method adapts to various types of designs such as random and fixed designs, highly clustered and nearly uniform designs. Furthermore, there is an absence of boundary effects: the bias at the boundary stays automatically of the same order as the interior, without use of specific boundary kernels. The local polynomial approximation approach is appealing on general scientific grounds: the least squares principle to be applied opens the way to a wealth of statistical knowledge and thus easy generalizations. In this Section, we briefly outline and review the idea of the extension of multivariate local polynomial fitting Kantz et al. (1997); Fan et al. (1996); Su (2010) to the parameter *R* of defoused *PSF*.

#### **3.2.1 Multivariate kernel function**

To localize data in the *m*-dimension, we need a multi kernel function. Generally speaking, a multivariate kernel function refers to a *m*-variate function satisfying

$$\int\_{-\infty}^{+\infty} \cdots \int\_{-\infty}^{+\infty} K(\underline{x}) d\underline{x} = 1 \tag{9}$$

Here and hereafter, we use to indicate multivariate integration over the *m*-dimensional Euclidean space.

There are two common methods for constructing multivariate kernel functions. For a univariate kernel *k*(*x*), the product kernel is given by

$$K(\underline{x}) = \prod\_{i=1}^{m} k(x\_i)\_{\prime} \tag{10}$$

and the spherically symmetric kernel is defined as

$$K(\underline{\mathbf{x}}) = c\_{K,m} K(\|\underline{\mathbf{x}}\|). \tag{11}$$

where *cK*,*<sup>m</sup>* = { *<sup>K</sup>*(�*x*�)*dx*}−<sup>1</sup> is a normalization constant and �*x*� = (*x*<sup>2</sup> <sup>1</sup> <sup>+</sup> *<sup>x</sup>*<sup>2</sup> <sup>2</sup> + ··· + *x*2 *<sup>m</sup>*)−1/2. Popular choices of *K*(*x*) include the standard *d*-variate normal density

$$K(\underline{\mathbf{x}}) = (2\pi)^{-m/2} \exp(-||\underline{\mathbf{x}}||^2/2) \tag{12}$$

and the spherical *Epanechnikov* kernel

$$K(\underline{\mathbf{x}}) = \{d(d+2)\Gamma(m/2)/(4\pi^{m/2})\}(1-||\underline{\mathbf{x}}||^2)\_+\tag{13}$$

The latter is the optimal kernel, according to Fan et al Fan et al. (1996); Su (2010).

The localization in multivariate nonparametric regression is frequently carried out by the kernel weighting. Let *H* be a symmetric positive-definite matrix called a bandwidth matrix. The localization scheme at a point *x* assigns the weight

$$K\_{\underline{H}}(\underline{X\_i} - \underline{x}), \quad \text{with} \quad K\_{\underline{H}}(\underline{x}) = |\underline{H}|^{-1} K(\underline{H}^{-1}\underline{x}), \tag{14}$$

where |*H*| is the determinant of the matrix *H*. The bandwidth matrix is introduced to accommodate the dependent structure in the independent variables. For practical problems, the bandwidth matrix *H* is taken to be a diagonal matrix. The different independent variables will be accommodated into different scales. For simplification, the bandwidth matrix is designed into *H* = *hIm* (*Im* denoting the identity matrix of order *m*).

#### **3.2.2 Multivariate predictor with local polynomial fitting**

Suppose that the input vector is *V* = (*v*1, *v*2, *v*3, *v*4, *v*5). The model is fitted by the function

$$R = f(\underline{V}).\tag{15}$$

Our purpose is to obtain the estimation *R*ˆ = ˆ *f*(*V*) of function *f* . This paper, we use the *d*th order multivariate local polynomial *f*(*V*) to predict the defocused parameter *RT* value based on the point *VT* of the test image. The polynomial function can be described as

$$f(\underline{V}) \approx \sum\_{0 \le |\underline{j}| \le d} \frac{1}{\underline{j}!} D^{(\underline{j})} f\_{\vec{i}}(\underline{V\_{T}}) (\underline{V} - \underline{V\_{T}})^{\underline{j}} = \sum\_{0 \le |\underline{j}| \le d} \underline{b\_{\vec{i}}}(\underline{V\_{T}}) (\underline{V} - \underline{V\_{T}})^{\underline{j}} \tag{16}$$

where

6 Image Restoration

Multivariate local polynomial fitting is an attractive method both from theoretical and practical point of view. Multivariate local polynomial method has a small mean squared error compared with the *Nadaraya* − *Watson* estimator which leads to an undesirable form of the bias and the *Gasser* − *Muller* estimator which has to pay a price in variance when dealing with a random design model. Multivariate local polynomial fitting also has other advantages. The method adapts to various types of designs such as random and fixed designs, highly clustered and nearly uniform designs. Furthermore, there is an absence of boundary effects: the bias at the boundary stays automatically of the same order as the interior, without use of specific boundary kernels. The local polynomial approximation approach is appealing on general scientific grounds: the least squares principle to be applied opens the way to a wealth of statistical knowledge and thus easy generalizations. In this Section, we briefly outline and review the idea of the extension of multivariate local polynomial fitting Kantz et al. (1997);

To localize data in the *m*-dimension, we need a multi kernel function. Generally speaking, a

 +∞ −∞

Here and hereafter, we use to indicate multivariate integration over the *m*-dimensional

There are two common methods for constructing multivariate kernel functions. For a

*m* ∏ *i*=1

*<sup>K</sup>*(�*x*�)*dx*}−<sup>1</sup> is a normalization constant and �*x*� = (*x*<sup>2</sup>

*K*(*x*) =

*<sup>m</sup>*)−1/2. Popular choices of *K*(*x*) include the standard *d*-variate normal density

The latter is the optimal kernel, according to Fan et al Fan et al. (1996); Su (2010).

The localization in multivariate nonparametric regression is frequently carried out by the kernel weighting. Let *H* be a symmetric positive-definite matrix called a bandwidth matrix.

*K*(*x*)*dx* = 1 (9)

*k*(*xi*), (10)

<sup>1</sup> <sup>+</sup> *<sup>x</sup>*<sup>2</sup>

<sup>2</sup> + ··· +

*K*(*x*) = *cK*,*mK*(�*x*�). (11)

*<sup>K</sup>*(*x*)=(2*π*)−*m*/2*exp*(−�*x*�2/2) (12)

*<sup>K</sup>*(*x*) = {*d*(*<sup>d</sup>* <sup>+</sup> <sup>2</sup>)Γ(*m*/2)/(4*πm*/2)}(<sup>1</sup> − �*x*�<sup>2</sup>)+ (13)

**3.2 Multivariate local polynomial regression for defocused parameter**

Fan et al. (1996); Su (2010) to the parameter *R* of defoused *PSF*.

multivariate kernel function refers to a *m*-variate function satisfying

univariate kernel *k*(*x*), the product kernel is given by

and the spherically symmetric kernel is defined as

and the spherical *Epanechnikov* kernel

 +∞ −∞ ···

**3.2.1 Multivariate kernel function**

Euclidean space.

where *cK*,*<sup>m</sup>* = {

*x*2

$$m = \mathbf{5}, \quad \underline{\mathbf{j}} = (\mathbf{j}\_1, \mathbf{j}\_2, \dots, \mathbf{j}\_m), \quad \underline{\mathbf{j}} = \mathbf{j}\_1! \mathbf{j}\_2! \cdots \mathbf{j}\_m! , \quad |\underline{\mathbf{j}}| = \sum\_{l=1}^m \mathbf{j}\_{l'} \tag{17}$$

$$\sum\_{\substack{0 \le |\underline{j}| \le d \\ 0 \le |\underline{j}| = 0}} = \sum\_{\substack{|\underline{j}| = 0 \\ \underline{j} = 0}}^d (\sum\_{\substack{j\_1 = 0 \\ j\_2 = 0}}^{|\underline{j}|} \cdots \sum\_{\substack{j\_m = 0 \\ j\_m = 0}}^{|\underline{j}|}), \quad \underline{V}^{\underline{j}} = \upsilon\_1^{j\_1} \upsilon\_2^{j\_2} \cdots \upsilon\_m^{j\_m} \tag{18}$$

$$D^{(\vec{j})}f\_{\vec{i}}(\underline{V\_{T}}) = \frac{\partial^{\left[\vec{j}\right]}f\_{\vec{i}}(\underline{V})}{\partial v\_{1}^{\vec{j}}\partial v\_{2}^{\vec{j}}\cdot\cdots\partial v\_{m}^{\vec{j}\_{w}}}|\_{\underline{V}=\underline{V\_{T}}\prime} \,\,\,\,\underline{\mathfrak{b}}\_{\vec{l}}(\underline{V\_{T}}) = \frac{1}{\underline{l}!}D^{(\vec{j})}f\_{\vec{i}}(\underline{V\_{T}}).\tag{19}$$

In the multivariate prediction method, *VTa* (*a* = 1, 2, ··· , *A*) denoting the trained image feature vectors. Using *A* pairs of (*VTa* , *Ra*), for which the values are already known, the coefficients of *fi* is determined by minimizing

$$\sum\_{a=1}^{A} \left[ R\_d - \sum\_{0 \le |\underline{j}| \le d} \underline{b\_{\underline{j}}}(\underline{V\_{\underline{T}}}) (\underline{V\_{\underline{T}\_a}} - \underline{V\_{\underline{T}}})^{\underline{\dot{i}}} \right]^2 \cdot \mathcal{K}\_{\underline{H}}(\underline{V\_{\underline{T}\_a}} - \underline{V\_{\underline{T}}}) \tag{20}$$

For the weighted least squared problem, a matrix form can be described by

$$
\underline{W}^{1/2} \cdot \underline{Y} = \underline{W}^{1/2} \cdot \underline{X} \cdot \underline{B} + \underline{\varepsilon} \tag{21}
$$

where

$$\underline{Y} = (y\_1, y\_2, \dots, y\_A)^T, \quad y\_a = R\_{a\nu} \tag{22}$$

$$\underline{B} = \begin{pmatrix} \underline{b}\_0(\underline{V\_T})\_\prime \underline{b}\_1(\underline{V\_T})\_\prime \cdots \; \underline{b}\_d(\underline{V\_T}) \end{pmatrix}^T,\tag{23}$$

Another issue in multivariate local polynomial fitting is the choice of the order of the polynomial. Since the modeling bias is primarily controlled by the bandwidth, this issue is less crucial however. For a given bandwidth *h*, a large value of *d* would expectedly reduce the modeling bias, but would cause a large variance and a considerable computational cost. Since the bandwidth is used to control the modeling complexity, and due to the sparsity of local data in multi-dimensional space, a higher-order polynomial is rarely used. We use the local quadratic regression to indicate the flavor of the multivariate local polynomial fitting, that is

Defocused Image Restoration with Local Polynomial Regression and IWF 179

The third issue is the selection of the kernel function. In this paper, of course, we choose the optimal spherical *Epanechnikov* kernel function, which minimizes the asymptotic *MSE* of the

Twenty original images are chosen to train the model. The images are defocused artificially with *R* whose value ranging from 2 to 7. So the total number of training samples are 120. Then

relationship between the *V* and *R*, then the defocused parameter *R* can be calculated using

Wiener filtering (minimizing mean square error ) is commonly used to restore linearly-degraded images. To obtain optimal results,there must be accurate knowledge of the covariance of the ideal image. In this section, the so-called iterative Wiener filter Su et al.

The imaging system H is assumed to be linear shift invariant with additive, independent, white noise processes of known variance. the model for the observed image y is given in

where f is the ideal image. The optimal linear minimum mean-squared error, or Wiener

autocorrelation of ideal image *f* . However, in practical situations *f* is usually not available and only a single copy of the blurred image to be restored, *y*, is provided. In the absence of a more accurate knowledge of the ideal image *f* , the blurred image *y* is often used in its place simply because there is no other information about *f* readily available. The signal *y* is subsequently used to compute an estimate of *Rf f* and this estimate is used in place of *Rf f* in

ˆ

*V* = (*v*1, *v*2, *v*3, *v*4, *v*5) (30)

*<sup>a</sup>*=<sup>1</sup> are given, obtaining weights matrix *B*, according to the

*y* = *H f* + *n* (31)

*f* = *By* (32)

<sup>−</sup>1, requires accurate knowledge of *Rf f* , the

resulting multivariate local polynomial estimators, as our kernel function.

feature vectors are constructed using variables *v*1−<sup>5</sup> of each image:

(2008); Allen et al. (1990)is used to restore the original image.

The following summarizes the iterative Wiener filtering procedure.

**3.2.4 Estimating the defocused parameter**

The defocused parameters *R* is the model output.

When training samples {*VTa* , *Ra*}<sup>120</sup>

the trained model.

matrix notation by

Equation (32).

restoration filter given by

where *B* = *Rf f HT*[*HRf f H<sup>T</sup>* + *Rnn*]

**4. Iterative Wiener filter**

to say, *d* = 2.

$$\underline{\mathsf{W}} = \operatorname{diag} \{ K\_{\underline{H}} (\underline{V\_{T\_1}} - \underline{V\_{T}}), K\_{\underline{H}} (\underline{V\_{T\_2}} - \underline{V\_{T}}), \dots, K\_{\underline{H}} (\underline{V\_{T\_A}} - \underline{V\_{T}}) \}\tag{24}$$
  $\text{and } \underline{\mathsf{X}} \text{ is the } A \times S \quad (S = \sum\_{\substack{\alpha < |\vec{\alpha}| \ \alpha < \frac{\alpha}{\epsilon}}} \frac{|\vec{j}|}{\vec{j}!})$ 

$$\underline{X} = \begin{pmatrix} 1 \ (\underline{V\_{T\_1}} - \underline{V\_T})^1 \ \cdots \ (\underline{V\_{T\_1}} - \underline{V\_T})^d \\ 1 \ (\underline{V\_{T\_2}} - \underline{V\_T})^1 \ \cdots \ (\underline{V\_{T\_2}} - \underline{V\_T})^d \\ \vdots \ \vdots \ \ddots \ \vdots \\ 1 \ (\underline{V\_{T\_A}} - \underline{V\_T})^1 \ \cdots \ (\underline{V\_{T\_A}} - \underline{V\_T})^d \end{pmatrix} \tag{25}$$

We then have the least squared solution with multivariate local polynomial fitting.

$$
\underline{\hat{B}} = (\underline{W}^{1/2}\underline{X})^\dagger \underline{Y} \tag{26}
$$

or, when *XTWX* is inverse, the estimation can be written by

0≤|*j*|≤*d*

$$
\underline{\hat{B}} = (\underline{\underline{X}}^T \underline{\underline{WX}})^{-1} \underline{\underline{X}}^T \underline{\underline{WY}} \tag{27}
$$

then, we can get the estimation *R*ˆ *<sup>T</sup>* = ˆ *f*(*VT*)

$$\hat{\mathcal{R}}\_{T} = \hat{f}(\underline{V\_{T}}) = \underline{E\_{1}}(\underline{X}^{T}\underline{WX})^{-1}\underline{X}^{T}\underline{WY} \tag{28}$$

where *<sup>E</sup>*<sup>1</sup> = (1, 0, 0, ··· , 0)1×*S*.

Computing the *B*ˆ will suffer from large computational cost. we can use the recursive least squared method to reduce the computation complexity, and it is very powerful especially in the real time prediction problems. There are several important issues about the bandwidth, the order of multivariate local polynomial function and the kernel function which have to be discussed. The three problems will be presented in Section 3.2.3.

#### **3.2.3 Parameters selections**

For the multivariate local polynomial predictor, there are three important problems which have significant influence to the prediction accuracy and computational complexity. First of all, there is the choice of the bandwidth matrix, which plays a rather crucial role. The bandwidth matrix *H* is taken to be a diagonal matrix. For simplification, the bandwidth matrix is designed into *H* = *hIm*. So the most important thing is to find the bandwidth *h*. A too big bandwidth under-parameterizes the regression function, causing a large modeling bias, while a too small bandwidth over-parameterizes the unknown function and results in noisy estimates. In theory, there exists a optimal bandwidth *hopt* in the meaning of mean squared error, such that

$$h\_{opt} = \arg\min\_{h} \int (f(\underline{x}) - \hat{f}(\underline{x}))^2 d\underline{x} \tag{29}$$

But the optimal bandwidth can not be solved directly. So we discuss how to get the asymptotically optimal bandwidth. There are quite a few important techniques for selecting the bandwidth. such as cross-validation and plug-in bandwidth selectors. a conceptually simple technique, with theoretical justification and good empirical performance , is the plug-in technique.

Another issue in multivariate local polynomial fitting is the choice of the order of the polynomial. Since the modeling bias is primarily controlled by the bandwidth, this issue is less crucial however. For a given bandwidth *h*, a large value of *d* would expectedly reduce the modeling bias, but would cause a large variance and a considerable computational cost. Since the bandwidth is used to control the modeling complexity, and due to the sparsity of local data in multi-dimensional space, a higher-order polynomial is rarely used. We use the local quadratic regression to indicate the flavor of the multivariate local polynomial fitting, that is to say, *d* = 2.

The third issue is the selection of the kernel function. In this paper, of course, we choose the optimal spherical *Epanechnikov* kernel function, which minimizes the asymptotic *MSE* of the resulting multivariate local polynomial estimators, as our kernel function.

## **3.2.4 Estimating the defocused parameter**

Twenty original images are chosen to train the model. The images are defocused artificially with *R* whose value ranging from 2 to 7. So the total number of training samples are 120. Then feature vectors are constructed using variables *v*1−<sup>5</sup> of each image:

$$
\underline{V} = (v\_{1\prime}v\_{2\prime}v\_{3\prime}v\_{4\prime}v\_{5\prime})\tag{30}
$$

The defocused parameters *R* is the model output.

When training samples {*VTa* , *Ra*}<sup>120</sup> *<sup>a</sup>*=<sup>1</sup> are given, obtaining weights matrix *B*, according to the relationship between the *V* and *R*, then the defocused parameter *R* can be calculated using the trained model.

## **4. Iterative Wiener filter**

8 Image Restoration

<sup>1</sup> (*VT*<sup>1</sup> <sup>−</sup> *VT*)<sup>1</sup> ··· (*VT*<sup>1</sup> <sup>−</sup> *VT*)*<sup>d</sup>* <sup>1</sup> (*VT*<sup>2</sup> <sup>−</sup> *VT*)<sup>1</sup> ··· (*VT*<sup>2</sup> <sup>−</sup> *VT*)*<sup>d</sup>*

. ... .

<sup>1</sup> (*VTA* <sup>−</sup> *VT*)<sup>1</sup> ··· (*VTA* <sup>−</sup> *VT*)*<sup>d</sup>*

We then have the least squared solution with multivariate local polynomial fitting.

*f*(*VT*)

Computing the *B*ˆ will suffer from large computational cost. we can use the recursive least squared method to reduce the computation complexity, and it is very powerful especially in the real time prediction problems. There are several important issues about the bandwidth, the order of multivariate local polynomial function and the kernel function which have to be

For the multivariate local polynomial predictor, there are three important problems which have significant influence to the prediction accuracy and computational complexity. First of all, there is the choice of the bandwidth matrix, which plays a rather crucial role. The bandwidth matrix *H* is taken to be a diagonal matrix. For simplification, the bandwidth matrix is designed into *H* = *hIm*. So the most important thing is to find the bandwidth *h*. A too big bandwidth under-parameterizes the regression function, causing a large modeling bias, while a too small bandwidth over-parameterizes the unknown function and results in noisy estimates. In theory, there exists a optimal bandwidth *hopt* in the meaning of mean

�

But the optimal bandwidth can not be solved directly. So we discuss how to get the asymptotically optimal bandwidth. There are quite a few important techniques for selecting the bandwidth. such as cross-validation and plug-in bandwidth selectors. a conceptually simple technique, with theoretical justification and good empirical performance , is the plug-in

(*f*(*x*) <sup>−</sup> <sup>ˆ</sup>

and *X* is the *A* × *S* (*S* = ∑

0≤|*j*|≤*d*

*X* =

or, when *XTWX* is inverse, the estimation can be written by

*R*ˆ *<sup>T</sup>* = ˆ

discussed. The three problems will be presented in Section 3.2.3.

*hopt* <sup>=</sup> *arg minh*

then, we can get the estimation *R*ˆ *<sup>T</sup>* = ˆ

where *<sup>E</sup>*<sup>1</sup> = (1, 0, 0, ··· , 0)1×*S*.

**3.2.3 Parameters selections**

squared error, such that

technique.


⎛

⎜⎜⎜⎜⎝

. . . . .

*W* = *diag*{*KH*(*VT*<sup>1</sup> − *VT*), *KH*(*VT*<sup>2</sup> − *VT*), ··· , *KH*(*VTA* − *VT*)} (24)

. . ⎞

⎟⎟⎟⎟⎠

*B*ˆ = (*W*1/2*X*)†*Y* (26)

*B*ˆ = (*XTWX*)−1*XTWY* (27)

*f*(*VT*) = *E*1(*XTWX*)−1*XTWY* (28)

*f*(*x*))2*dx* (29)

(25)

Wiener filtering (minimizing mean square error ) is commonly used to restore linearly-degraded images. To obtain optimal results,there must be accurate knowledge of the covariance of the ideal image. In this section, the so-called iterative Wiener filter Su et al. (2008); Allen et al. (1990)is used to restore the original image.

The imaging system H is assumed to be linear shift invariant with additive, independent, white noise processes of known variance. the model for the observed image y is given in matrix notation by

$$y = Hf + n\tag{31}$$

where f is the ideal image. The optimal linear minimum mean-squared error, or Wiener restoration filter given by

$$
\hat{f} = \mathcal{B}y\tag{32}
$$

where *B* = *Rf f HT*[*HRf f H<sup>T</sup>* + *Rnn*] <sup>−</sup>1, requires accurate knowledge of *Rf f* , the autocorrelation of ideal image *f* . However, in practical situations *f* is usually not available and only a single copy of the blurred image to be restored, *y*, is provided. In the absence of a more accurate knowledge of the ideal image *f* , the blurred image *y* is often used in its place simply because there is no other information about *f* readily available. The signal *y* is subsequently used to compute an estimate of *Rf f* and this estimate is used in place of *Rf f* in Equation (32).

The following summarizes the iterative Wiener filtering procedure.

prediction errors are shown in Table1. From Table 1, we can conclude that the prediction results of *MLPR* predictor are significantly better than the RBF neural network method in the

Defocused Image Restoration with Local Polynomial Regression and IWF 181

training image different methods *eMSE*

Fig. 4. RBFNN method result of *Coins*. True image(left); blurred image(middle); estimated

Fig. 5. RBFNN method result for *Tire*. True image(left); blurred image (middle); restored

Fig. 6. Result of *Moon*. True image(left); blurred image(middle); estimated image(right),

blurred images and estimated true images are depicted in the left, middle and right column,

Figures 4 and 5, in which the true images, blurred images and estimated true images are depicted in the left, middle and right column, respectively, illustrate how the method behaves in *Coins* and *Tire* images. It is clear from Figs. 4 and 5 that performance of the RBFNN method is effective in different images. Figures 6, 7, 8 and 9, in which the true images,

Moon RBFNN 4.81 <sup>×</sup>10−<sup>6</sup> Moon MLPR 4.13 <sup>×</sup>10−<sup>8</sup> Coins RBFNN 5.06 <sup>×</sup>10−<sup>6</sup> Coins MLPR 3.97 <sup>×</sup>10−<sup>8</sup> Saturn RBFNN 6.62 <sup>×</sup>10−<sup>6</sup> Saturn MLPR 5.65 <sup>×</sup>10−<sup>9</sup> Tire RBFNN 8.04 <sup>×</sup>10−<sup>6</sup> Tire MLPR 7.19 <sup>×</sup>10−<sup>8</sup>

same simulated data.

Table 1. MSE using both methods

image(right), BSNR=12.35, ISNR=22.56

image(right), BSNR=11.22, ISNR=23.14

BSNR=12.35, ISNR=22.56

step 1 Initialization: Use *y* to compute an initial (i=0) estimate of *Rf f* by

$$R\_{ff}(0) = R\_{yy} = E\{yy^T\} \tag{33}$$

Step 2 Filter construction: Use *Rf f*(*i*), the i*th* estimate of *Rf f* to construct the (*i* + 1)*th* restoration filter *B*(*i* + 1) given by

$$B\_{i+1} = \mathcal{R}\_{ff} H^T [H \mathcal{R}\_{ff} H^T + \mathcal{R}\_{nn}]^{-1} \tag{34}$$

Step 3 Restoration: Restore *y* by the *B*(*i* + 1) filter to obtain ˆ *f*(*i* + 1), the (*i* + 1)*th* estimate of *f*

$$
\hat{f}(i+1) = B(i+1)y\tag{35}
$$

Step 4 Update: Use ˆ *f*(*i* + 1) to compute an improved estimate of *Rf f* , given by

$$R\_{ff}(i+1) = E\{f(i+1)f^T(i+1)\}\tag{36}$$

Step 5 Iteration: Increment *i* and repeat steps 2,3,4, and 5.

#### **5. Experimental results and analysis**

The experiments are carried out by using the Matlab image processing toolbox. The performance of the proposed image restoration algorithm has been evaluated using the classical gray-scale *Moon* image, *Coins* image, *Saturn* image, and *Tire* image in Matlab toolbox. To verify the good ability of restoration of the proposed algorithm, one real blurred image is used for the deconvolution procedure. The results show our method is very successful for this kind of blurred image.

In image restoration studies, the degradation modelled by blurring and additive noise is referred to in terms of the metric blurred signal-to-noise ratio (*BSNR*). This metric for a zero-mean *M* × *N* image is given by

$$BSNR = 10\log\_{10}\{\frac{\frac{1}{\text{MIN}}\sum\_{m=1}^{M}\sum\_{n=1}^{N}z^2(m,n)}{\sigma\_v^2}\}\tag{37}$$

where *z*(*m*, *n*) is the noise free blurred image and *σ*<sup>2</sup> *<sup>v</sup>* is the additive noise variance.

For the purpose of objectively testing the performance of linear image restoration algorithms, the improvement in signal-to-noise ratio (*ISNR*) is often used. *ISNR* is defined as

$$ISNR = 10\log\_{10}\{\frac{\sum\_{m=1}^{M}\sum\_{n=1}^{N}[f(m,n) - y(m,n)]^2}{\sum\_{m=1}^{M}\sum\_{n=1}^{N}[f(m,n) - f(m,n)]^2}\}\tag{38}$$

where *f*(*m*, *n*) and *y*(*m*, *n*) are the original and degraded image pixel intensity values and ˆ *f*(*m*, *n*) is the restored true image pixel intensity value. *ISNR* cannot be used when the true image is unknown, but it can be used to compare different methods in simulations when the true image is known.

In order to find the good performance of the proposed multivariate local polynomial Regression method (MLPR) compared with the RBF neural network algorithm (RBFNN) Su et al. (2008), the same defocused blurred images are used for the experiments. Mean squared prediction errors are shown in Table1. From Table 1, we can conclude that the prediction results of *MLPR* predictor are significantly better than the RBF neural network method in the same simulated data.


Table 1. MSE using both methods

10 Image Restoration

Step 2 Filter construction: Use *Rf f*(*i*), the i*th* estimate of *Rf f* to construct the (*i* + 1)*th*

*Bi*<sup>+</sup><sup>1</sup> = *Rf f HT*[*HRf f H<sup>T</sup>* + *Rnn*]

*f*(*i* + 1) to compute an improved estimate of *Rf f* , given by

*f*(*i* + 1) ˆ

The experiments are carried out by using the Matlab image processing toolbox. The performance of the proposed image restoration algorithm has been evaluated using the classical gray-scale *Moon* image, *Coins* image, *Saturn* image, and *Tire* image in Matlab toolbox. To verify the good ability of restoration of the proposed algorithm, one real blurred image is used for the deconvolution procedure. The results show our method is very

In image restoration studies, the degradation modelled by blurring and additive noise is referred to in terms of the metric blurred signal-to-noise ratio (*BSNR*). This metric for a

> 1 *MN* <sup>∑</sup>*<sup>M</sup>*

For the purpose of objectively testing the performance of linear image restoration algorithms,

where *f*(*m*, *n*) and *y*(*m*, *n*) are the original and degraded image pixel intensity values and

*f*(*m*, *n*) is the restored true image pixel intensity value. *ISNR* cannot be used when the true image is unknown, but it can be used to compare different methods in simulations when the

In order to find the good performance of the proposed multivariate local polynomial Regression method (MLPR) compared with the RBF neural network algorithm (RBFNN) Su et al. (2008), the same defocused blurred images are used for the experiments. Mean squared

*<sup>m</sup>*=<sup>1</sup> <sup>∑</sup>*<sup>N</sup>*

the improvement in signal-to-noise ratio (*ISNR*) is often used. *ISNR* is defined as

∑*<sup>M</sup> <sup>m</sup>*=<sup>1</sup> <sup>∑</sup>*<sup>N</sup>* *<sup>m</sup>*=<sup>1</sup> <sup>∑</sup>*<sup>N</sup>*

*σ*2 *v*

*<sup>n</sup>*=1[ *<sup>f</sup>*(*m*, *<sup>n</sup>*) <sup>−</sup> <sup>ˆ</sup>

*<sup>n</sup>*=<sup>1</sup> *<sup>z</sup>*2(*m*, *<sup>n</sup>*)

*<sup>n</sup>*=1[ *<sup>f</sup>*(*m*, *<sup>n</sup>*) <sup>−</sup> *<sup>y</sup>*(*m*, *<sup>n</sup>*)]<sup>2</sup>

*<sup>v</sup>* is the additive noise variance.

*Rf f*(0) = *Ryy* <sup>=</sup> *<sup>E</sup>*{*yyT*} (33)

*f*(*i* + 1) = *B*(*i* + 1)*y* (35)

<sup>−</sup><sup>1</sup> (34)

*f*(*i* + 1), the (*i* + 1)*th* estimate of *f*

*<sup>f</sup> <sup>T</sup>*(*<sup>i</sup>* <sup>+</sup> <sup>1</sup>)} (36)

} (37)

*<sup>f</sup>*(*m*, *<sup>n</sup>*)]<sup>2</sup> } (38)

step 1 Initialization: Use *y* to compute an initial (i=0) estimate of *Rf f* by

ˆ

*Rf f*(*<sup>i</sup>* <sup>+</sup> <sup>1</sup>) = *<sup>E</sup>*{ <sup>ˆ</sup>

Step 3 Restoration: Restore *y* by the *B*(*i* + 1) filter to obtain ˆ

Step 5 Iteration: Increment *i* and repeat steps 2,3,4, and 5.

*BSNR* = 10*log*10{

*ISNR* <sup>=</sup> <sup>10</sup>*log*10{ <sup>∑</sup>*<sup>M</sup>*

where *z*(*m*, *n*) is the noise free blurred image and *σ*<sup>2</sup>

**5. Experimental results and analysis**

successful for this kind of blurred image.

zero-mean *M* × *N* image is given by

ˆ

true image is known.

restoration filter *B*(*i* + 1) given by

Step 4 Update: Use ˆ

Fig. 4. RBFNN method result of *Coins*. True image(left); blurred image(middle); estimated image(right), BSNR=12.35, ISNR=22.56

Fig. 5. RBFNN method result for *Tire*. True image(left); blurred image (middle); restored image(right), BSNR=11.22, ISNR=23.14

Figures 4 and 5, in which the true images, blurred images and estimated true images are depicted in the left, middle and right column, respectively, illustrate how the method behaves in *Coins* and *Tire* images. It is clear from Figs. 4 and 5 that performance of the RBFNN method is effective in different images. Figures 6, 7, 8 and 9, in which the true images,

Fig. 6. Result of *Moon*. True image(left); blurred image(middle); estimated image(right), BSNR=12.35, ISNR=22.56

blurred images and estimated true images are depicted in the left, middle and right column,

simulation and real image. Defocused image parameter can be successfully estimated by using trained model. Experimental results show the proposed schemes are reliable and robust for defocused blurred image restoration. Comparisons are made to verify the advantages of

Defocused Image Restoration with Local Polynomial Regression and IWF 183

This work was supported by Chongqing CSTC foundation of China (CSTC2010BB2310, CSTC2011jjA40033), Chongqing CMEC foundation of China (KJ080614,KJ100810,KJ100818),

Spiros C., Vasilios N. K., and Dimitrios P. (2009). Applications of the Moore-Penrose Inverse in Digital Image Restoration. *Mathematical Problems in Engineering*, 2009:1-12. Spiros C., Vasilios N. K., and Dimitrios P. Digital Image Reconstruction in the Spectral

Su L., Ma H., Li Z., and Ju S. Blind image restoration based on constant modulus with

Alonso M., and Adjouadi M., Digital image inverse filtering for improving visual acuity for

Alonso M., Cremades J. G., Jacko J., and Adjouadi M., Image Pre-compensation to facilitate

Bar L., Sochen N., and Kiryati N., Semi-blind image restoration via Mumford-Shah

Chen W., Chen M., and Zhou J., Adaptively Regularized Constrained Total Least-Squares

Suyash P. A. and Ross T. W., Unsupersived, Information-Theoretic, Adaptive Image Filtering

Gu X. and Li G., A new method for parameter estimation of edge-preserving regularization in image restoration. *Journal of Computational and Applied Mathematics*, 225(2):478-486. Lu L., Michael K. N., and Lin F., Approximation BFGS methods for nonlinear image restoration. *Journal of Computational and Applied Mathematics*, 226(1):84-91. Filip S. and Jan F., Multichannel Blind Iterative Image Restoration. *IEEE Trans. on Image*

Mario A. T. F., Robert D. N., An EM Algorithm for Wavelet-Based Image Restoration. *IEEE*

Yehong Liao and Xueyin Lin, "Blind Image Restoration with Eigen-Face Subspace," *IEEE Trans.*

M. M. Chang, A. M. Tekalp, and A. T. Erdem, "Blur Identification using the Bi-Spectrum," *IEEE*

Reeves S. J. and Mersereau R. M. , Blur Identification by the Method of Generalized

Domain Utilizing the Moore-Penrose Inverse. *Mathematical Problems in Engineering*,

averaging and ANFIS. *in Proceedings of Fourth International Conference on Image and*

computer users with visual aberrations. *Inverse Problems in Science and Engineering*,

computer access for users with refractive errors. *Behaviour & Information Technology*,

for Image Restoration. *IEEE Trans. on Pattern Analysis and Machine Intellgence*,

multivariate local polynomial regression based method.

**7. Acknowledgment**

**8. References**

CQUT foundation of China (2007ZD16).

16(8): 957-966.

24(3):161-173.

28(3):364-375.

*Processing*, 12(9):1094-1106.

*Trans. on Image Processing*, 12(8):906-916.

2010, Article ID 750352, 14 pages.

*Graphics (ICIG'07)*, pp.143-148, Chengdu, China.

regularization. *IEEE Trans. Image Process*, 15(2):483-493.

*on Image Processing*, vol. 14, no. 11, pp. 1766-1772, 2005.

*Trans. on Image Processing*, vol. 39, no. 10, pp.2323-2325, 1991.

Cross-Validation. *IEEE trans. on Image Processing*, 1(7):301-311.

Image Restoration. *IEEE Trans. on Image Processing*, 9(4):588-596.

Fig. 7. Result for *Coins*. True image(left); blurred image (middle); restored image(right), BSNR=11.22, ISNR=23.14

Fig. 8. Result for *Saturn*. True image(left); blurred image (middle); restored image(right), BSNR=13.17, ISNR=24.31

Fig. 9. Result for *Tire*. True image(left); blurred image (middle); restored image(right), BSNR=11.56, ISNR=22.09

Fig. 10. Result for real blurred image. blurred image (left); restored image(right)

respectively, illustrate how the method behaves in *Moon*, *Coins*, *Saturn* and *Tire* images. It is clear from Figs.6-9 that performance of the new method is effective in different images. Figure 10 also shows that the presented MLPR algorithm is good for real blurred image. And from the BSNR and ISNR in Figures 4, 5, 7, 9 we can see that the MLPR defocused image restoration method is better than RBFNN algorithm.

## **6. Conclusions**

Two new methods that are based on RBF neural network, multivariate local polynomial regression model and iterative Wiener filtering for semi-blind restoration of blurred noisy images were proposed in this chapter. Defocused parameter was estimated by a RBF neural network or multivariate local polynomial regression model trained in wavelet domain. The main advantages of the proposed techniques are that they are not only robust to noise because wavelet transform have an excellent de-noising ability, but also effective to artificially and practically defocused blurred image. Restoration is successfully realized by the iterative Wiener filter, resulting in improved the image quality. The algorithm was justified via simulation and real image. Defocused image parameter can be successfully estimated by using trained model. Experimental results show the proposed schemes are reliable and robust for defocused blurred image restoration. Comparisons are made to verify the advantages of multivariate local polynomial regression based method.

## **7. Acknowledgment**

This work was supported by Chongqing CSTC foundation of China (CSTC2010BB2310, CSTC2011jjA40033), Chongqing CMEC foundation of China (KJ080614,KJ100810,KJ100818), CQUT foundation of China (2007ZD16).

## **8. References**

12 Image Restoration

Fig. 7. Result for *Coins*. True image(left); blurred image (middle); restored image(right),

Fig. 8. Result for *Saturn*. True image(left); blurred image (middle); restored image(right),

Fig. 9. Result for *Tire*. True image(left); blurred image (middle); restored image(right),

Fig. 10. Result for real blurred image. blurred image (left); restored image(right)

respectively, illustrate how the method behaves in *Moon*, *Coins*, *Saturn* and *Tire* images. It is clear from Figs.6-9 that performance of the new method is effective in different images. Figure 10 also shows that the presented MLPR algorithm is good for real blurred image. And from the BSNR and ISNR in Figures 4, 5, 7, 9 we can see that the MLPR defocused image restoration

Two new methods that are based on RBF neural network, multivariate local polynomial regression model and iterative Wiener filtering for semi-blind restoration of blurred noisy images were proposed in this chapter. Defocused parameter was estimated by a RBF neural network or multivariate local polynomial regression model trained in wavelet domain. The main advantages of the proposed techniques are that they are not only robust to noise because wavelet transform have an excellent de-noising ability, but also effective to artificially and practically defocused blurred image. Restoration is successfully realized by the iterative Wiener filter, resulting in improved the image quality. The algorithm was justified via

BSNR=11.22, ISNR=23.14

BSNR=13.17, ISNR=24.31

BSNR=11.56, ISNR=22.09

method is better than RBFNN algorithm.

**6. Conclusions**


**9** 

*1Russia 2Mexico* 

**Image Restoration Using** 

*2Computer Science Department, CICESE,* 

 **Two-Dimensional Variations** 

*1Institute for Information Transmission Problems of RAS,* 

Olga Milukova1, Vitaly Kober1,2 and Victor Karnaukhov1

In many applications (consumer and commercial imaging, medical imaging, robotics, space research, and etc.) observed images are often degraded due to atmospheric turbulence, relative motion between a scene and a camera, nonuniform illumination, wrong focus, etc. Image restoration refers to the problem of estimating the ideal image from its observed degraded one. Numerous restoration techniques (linear, nonlinear, deterministic, stochastic, etc.) optimized with respect to different were introduced (Banham & Katsaggelos, 1997 ; Jain, 1989; Sezan & Tekalp, 1990; Bovik, 2005; Gonzalez & Woods 2008). The amount of *a priori* information about degradation such as the size and shape of blurs, noise level determines how mathematically ill-posed the problem is. *A priori* information can be used in a variety of ways in modeling and algorithm development. The information about the nature of blur (e.g., linear or nonlinear and space-variant or space-invariant) and noise (additive or multiplicative) is used in modeling the input-output relation of imaging systems. In blur modeling, when the type of blur is known (e.g., out of focus, motion, turbulence), the blurring operator can be parameterized using only a few parameters. In image modeling, the ideal image can be modeled, for instance, on the basis of *a priori* Markovian assumption. In algorithm development, *a priori* information is used in defining constraints on the solution and in defining a criterion or a quantitative description of the solution. The blind and non-blind deconvolutions were extensively studied, and many techniques were proposed for their solution (Kundur & Hatzinakos, 1996; Bertero & Boccacci, 1998; Biemond et al., 1990; Sroubek & Flusser, 2003). They usually involve some regularization which assures various statistical properties of the image or constrains on the estimated image and restoration filter according to some assumptions. This regularization is required to guarantee a unique solution and stability against noise and some model discrepancies. One of the most popular fundamental techniques is a linear minimum mean square error method. It finds the linear estimate of the ideal image for which the mean square error between the estimate and the ideal image is minimal. This linear operator acting on the observed image to determine the estimate on the base of *a priori* second-order statistical information about the image and noise processes. For images with sharp changes of intensity, the appropriate regularization is based on variational integrals (Rudin, et al.,

**1. Introduction** 


## **Image Restoration Using Two-Dimensional Variations**

Olga Milukova1, Vitaly Kober1,2 and Victor Karnaukhov1 *1Institute for Information Transmission Problems of RAS, 2Computer Science Department, CICESE, 1Russia 2Mexico* 

## **1. Introduction**

14 Image Restoration

184 Image Restoration – Recent Advances and Applications

Lagendijk R. L., J. Biemond, and B. E. Boekee, Identification and Restoration of Noisy Blurred

D. Kundur and D. Hatzinakos, A Novel Blind Deconvolution Scheme for Image Restoration using Recurisive Filtering. *IEEE Trans. on Signal Processing*, 46(2):375-390. A. K. Katsaggelos and K. T. Lay, Maximum Likelihood Blur Identification and Image

Su L., Li F., Xu F., and Liu Y., Defocused Image Restoration Using RBF Network and Iterative

Fan J and I. Gijbels, *Local polynomial modelling and its applications*, Chapman and Hall, London,

Su L., Prediction of multivariate chaotic time series with local polynomial fitting. *Computers* &

Allen D. Hillery and Roland T. Chin, Iterative Wiener Filters for Image Restoration. *in*

Patrick L. Combettes and Jean-Christophe Pesquet, Wavelet-constrained image restoration.

Zhou Huiyu, Liu Tangwei, Lin Faquan, Pang Yusheng, and Wu Ji, Image restoration and

Guo P., Li H., and Michael T. L., Blind Image restoration by combining wavelet transform and

Li M. and Wu Y., Qiang ZhangSAR image segmentation based on mixture context and wavelet

Li P. and Fang Y., A Wavelet Interpolation Galerkin Method for the Simulation of MEMS

Su L., Liu R. (2011). Blind Image Restoration with Modified CMA. *International Journal of Image*

Su L., Li F., Li J., Chen B. (2011). A Novel Digital Image Covert Communication Scheme based

Su L. (2011). Multivariate local polynomial regression with application to Shenzhen component index. *Discrete Dynamics in Nature and Society*, 2011:1-11. Liu, R., Su, L., Li, F. (2011). Image reconstruction based on the TV and Mumford-Shah-Euler

Su L. and Li F. (2010). Deconvolution of Defocused Image with Multivariate Local Polynomial

*Engineering*, vol. 2010, Article ID 586718, 25 pages, 2010.

model. *Advanced Materials Research*, 186:198-202.

*and signal processing(CISP'08)*c ˇ ˇnpp. 311-315, Sanya, Hainan, China, 2008. Kantz H and Schreiber T, *Nonlinear Time Series Analysis*, Cambridge University Press,

*Speech, Signal Processing*, 38(7):1180-1191.

*Mathematics with Applications*, 59(2):737-744.

pp. 1901-1904, Albuquerque, NM , USA, 1990.

Cambridge, UK, 1997.

UK, 1996.

2(4):371-389.

7(2):497-514.

57(6):961-969.

*Processing*, 5(1):371-389.

*and Graphics*, 11(3): 403-413.

*Engineering*, 2010:1-14.

Images using the Expectation-Maximization Algorithm. *IEEE Trans. on Acoustics,*

Restoration using the EM Algorithm," *IEEE Trans. on Signal Processing*, 39(3):729-733.

Wiener Filter in Wavelet Domain. *in Proceedings of 2008 International congress on image*

*Proceedings of The international conference on Acoustics, Speech, and Signal Processing*,

*International Journal of Wavelets, Multiresolution and Information Processing*,

detail preservation by Bayesian estimation. *International Journal of Image and Graphics*,

RBF neural network. *International Journal of Wavelets, Multiresolution and Information*

hidden-class-label Markov random field. *Computers* & *Mathematics with Applications*,

Devices under the Effect of Squeeze Film Damping. *Mathematical Problems in*

on Generalized FCM in DCT Domain. *Fuzzy Information and Engineering*, 3(2):127-136.

Regression and Iterative Wiener Filtering in DWT domain. *Mathematical Problems in*

In many applications (consumer and commercial imaging, medical imaging, robotics, space research, and etc.) observed images are often degraded due to atmospheric turbulence, relative motion between a scene and a camera, nonuniform illumination, wrong focus, etc. Image restoration refers to the problem of estimating the ideal image from its observed degraded one. Numerous restoration techniques (linear, nonlinear, deterministic, stochastic, etc.) optimized with respect to different were introduced (Banham & Katsaggelos, 1997 ; Jain, 1989; Sezan & Tekalp, 1990; Bovik, 2005; Gonzalez & Woods 2008). The amount of *a priori* information about degradation such as the size and shape of blurs, noise level determines how mathematically ill-posed the problem is. *A priori* information can be used in a variety of ways in modeling and algorithm development. The information about the nature of blur (e.g., linear or nonlinear and space-variant or space-invariant) and noise (additive or multiplicative) is used in modeling the input-output relation of imaging systems. In blur modeling, when the type of blur is known (e.g., out of focus, motion, turbulence), the blurring operator can be parameterized using only a few parameters. In image modeling, the ideal image can be modeled, for instance, on the basis of *a priori* Markovian assumption. In algorithm development, *a priori* information is used in defining constraints on the solution and in defining a criterion or a quantitative description of the solution. The blind and non-blind deconvolutions were extensively studied, and many techniques were proposed for their solution (Kundur & Hatzinakos, 1996; Bertero & Boccacci, 1998; Biemond et al., 1990; Sroubek & Flusser, 2003). They usually involve some regularization which assures various statistical properties of the image or constrains on the estimated image and restoration filter according to some assumptions. This regularization is required to guarantee a unique solution and stability against noise and some model discrepancies. One of the most popular fundamental techniques is a linear minimum mean square error method. It finds the linear estimate of the ideal image for which the mean square error between the estimate and the ideal image is minimal. This linear operator acting on the observed image to determine the estimate on the base of *a priori* second-order statistical information about the image and noise processes. For images with sharp changes of intensity, the appropriate regularization is based on variational integrals (Rudin, et al.,

Image Restoration Using Two-Dimensional Variations 187

restoration lead to optimization problems, which are similar to that of Eq. (3). For instance, using Bayes' strategy (Kay, 1993) we can obtain the optimal estimation in the following form:

2 1 inf{ ln ( ) ln ( )}

additive noise . *n Az v* The main difference between the regularization method of image restoration in Eq. (3) and the statistical method in Eq. (4) is the regularization parameter

can be chosen from the set of solutions using, for instance, a subjective criterion. If the space Q in Eq. (3) is the Euclidian space with the norm (*v, Bv*), where B is a positive defined

> \* <sup>2</sup> inf{ ( )} *<sup>B</sup> z Z z Az v z*

It is commonly assumed that the original image is a smooth function with respect to the Sobolev space (Adams, 1975), and the stabilization functional in Eq. (5) is ( ) *<sup>p</sup>*

Quadratic forms can be used in order to avoid nonlinear restoration algorithms. Note that a Gaussian image model leads to minimization of a quadratic form. In discrete case it corresponds to the Sobolev norm for *p* 2 in Eq. (5). On the other hand, the use of quadratic forms in image restoration often yields undesirable results because of real images are not Gaussian. Now suppose that an image to be restored is a function of bounded

> \* inf{ ( , ) ( )} *<sup>Q</sup> z Z z Az v Var z*

2 ( ) sup ( ) ( )

It can be shown, that if the image *zxy* (,) , (,) *xy D* consists of 1D functions of bounded variation along its rows and columns then the image is also a 2D function of bounded variation. Different multidimensional variations were proposed such as variations of Arzela, Vitali, Tonelly, etc. (Vitushkin, 1955). A different approach was suggested by Kronrod, who introduced two functionals in order to describe an image as a function of two variables

, and 2 1 () ( )*<sup>t</sup> d z m e dt*

where *te* is *t* - level set of the function *zxy* (,) , i.e. a set of points (,) *x y* with function values equal to *t*, 0( ) *m et* is the number of components of *te* , and 1( ) *m et* is the length of the set *te* .

*k k <sup>a</sup> x x <sup>k</sup> Vf fx fx* 

 

*b n*

1....

*n*

The variation of a 1D function *f x x ab* ( ), [ , ] is defined as follows:

1 0 () ( )*<sup>t</sup> d z m e dt* 

(Kronrod, 1950). The functionals are given as follows:

\*

This leads to a family of solutions as a function of the parameter

is a parameter of regularization. Note that the statistical methods used in image

*z Z z p Az v p z* , (4)

. (5)

. (6)

. (7)

, (8)

1

.

*q q <sup>W</sup> z z* .

. The best restored image

are *a priori* probability densities of the original image *z* and

where

where 1 *p* ( )

operator, we obtain,

 and 2 *p* ( ) 

variations. Therefore, it may be written as

1992; Perona & Malik, 1990; Chan & Wong, 1998). Minimization of the variational integrals preserves edges and fine details in the image. It is obvious that quality of the restored image depends on accuracy of mathematical model of image formation process. In particular, a good estimation of distortion parameters such as speed of camera movement, transparency of atmosphere or water, etc. is very important for restoration (Biemond et al., 1990).

Recently, restoration methods based on image variations were proposed (Milukova et al., 2010a, 2010b, 2011). In this chapter, image restoration with two-dimensional variations is presented. The restored image minimizes two-dimensional variations defined by Kronrod (Kronrod, 1950). We also consider the identification of distortion operator and estimation of its parameters. It is assumed that a monochrome stationary image distorted by homogeneous integrated transformation. Various physical problems can be modeled by such transformation. The spectral method of identification of distortion parameters uses only degraded image. Computer simulation results illustrate the performance of the proposed method for restoration of blurred images.

## **2. Restoration of linear degraded images with variation methods**

#### **2.1 Variation concept for image restoration**

Image restoration problem is usually formulated as follows. Undistorted (original) image *z* is recovered from the given equation:

$$
\upsilon = Az + n\quad\,,\tag{1}
$$

where *A* :Z Q ( Z,Q are metric spaces) is linear or nonlinear operator, *z*Ζ , *n* is noise, *v* is observed distorted image. A general approach for image restoration can be formulated using statistical estimation methods and the theory of solving of ill-posed problems (Tikhonov & Arsenin, 1977). The restoration problem is a typical inverse problem of mathematical physics, and, therefore, it can be correctly solved on the base of mathematical methods. The restored image can be obtained by minimization of the following functional:

$$\left(z\right)^{\*} = \inf\_{z\neq Z} \rho\_{\mathbb{Q}}(Az, v) \; \; \; \; \; \tag{2}$$

where *<sup>Q</sup>* is a metric in Q . Note that various definitions of a distance *<sup>Q</sup>* between two images may be used. It is easy to show that the solution of the optimization problem in Eq. (2) is not unique even when the operator *A* and the distorted image *v* are exactly known, and no additive noise. *A priori* information about the original image should be used to obtain a unique and stable solution from the set of solutions. The simplest way to guarantee uniqueness and stability of the solution is to describe the image model with a functional ( ) *z* that possesses stabilizing properties. In this case the image restoration problem can be reduced to conditional or unconditional optimization problem, in particular to the Tikhonov's minimization (Tikhonov & Arsenin, 1977),

$$\begin{aligned} z \stackrel{\*}{=} \inf\_{z \in \mathcal{Z}} \{ \rho\_{\mathcal{Q}}(Az, \upsilon) + \alpha \Omega(z) \}\, \end{aligned} \tag{3}$$

186 Image Restoration – Recent Advances and Applications

1992; Perona & Malik, 1990; Chan & Wong, 1998). Minimization of the variational integrals preserves edges and fine details in the image. It is obvious that quality of the restored image depends on accuracy of mathematical model of image formation process. In particular, a good estimation of distortion parameters such as speed of camera movement, transparency

Recently, restoration methods based on image variations were proposed (Milukova et al., 2010a, 2010b, 2011). In this chapter, image restoration with two-dimensional variations is presented. The restored image minimizes two-dimensional variations defined by Kronrod (Kronrod, 1950). We also consider the identification of distortion operator and estimation of its parameters. It is assumed that a monochrome stationary image distorted by homogeneous integrated transformation. Various physical problems can be modeled by such transformation. The spectral method of identification of distortion parameters uses only degraded image. Computer simulation results illustrate the performance of the

Image restoration problem is usually formulated as follows. Undistorted (original) image *z*

where *A* :Z Q ( Z,Q are metric spaces) is linear or nonlinear operator, *z*Ζ , *n* is noise, *v* is observed distorted image. A general approach for image restoration can be formulated using statistical estimation methods and the theory of solving of ill-posed problems (Tikhonov & Arsenin, 1977). The restoration problem is a typical inverse problem of mathematical physics, and, therefore, it can be correctly solved on the base of mathematical methods. The restored image can be obtained by minimization of the

> \* inf ( , ) *<sup>Q</sup> z Z z Az v*

images may be used. It is easy to show that the solution of the optimization problem in Eq. (2) is not unique even when the operator *A* and the distorted image *v* are exactly known, and no additive noise. *A priori* information about the original image should be used to obtain a unique and stable solution from the set of solutions. The simplest way to guarantee uniqueness and stability of the solution is to describe the image model with a functional ( ) *z* that possesses stabilizing properties. In this case the image restoration problem can be reduced to conditional or unconditional optimization problem, in particular to the

> \* inf{ ( , ) ( )} *<sup>Q</sup> z Z z Az v z*

 

*<sup>Q</sup>* is a metric in Q . Note that various definitions of a distance

*v Az n* , (1)

, (2)

, (3)

*<sup>Q</sup>* between two

of atmosphere or water, etc. is very important for restoration (Biemond et al., 1990).

**2. Restoration of linear degraded images with variation methods** 

proposed method for restoration of blurred images.

Tikhonov's minimization (Tikhonov & Arsenin, 1977),

**2.1 Variation concept for image restoration** 

is recovered from the given equation:

following functional:

where

where is a parameter of regularization. Note that the statistical methods used in image restoration lead to optimization problems, which are similar to that of Eq. (3). For instance, using Bayes' strategy (Kay, 1993) we can obtain the optimal estimation in the following form:

$$\text{tr}^\* = \inf\_{z \in \mathbb{Z}} \{-\ln p\_2(Az - v) - \ln p\_1(z)\}\,\,\,\,\tag{4}$$

where 1 *p* ( ) and 2 *p* ( ) are *a priori* probability densities of the original image *z* and additive noise . *n Az v* The main difference between the regularization method of image restoration in Eq. (3) and the statistical method in Eq. (4) is the regularization parameter . This leads to a family of solutions as a function of the parameter . The best restored image can be chosen from the set of solutions using, for instance, a subjective criterion. If the space Q in Eq. (3) is the Euclidian space with the norm (*v, Bv*), where B is a positive defined operator, we obtain,

$$\|z\|^\* = \inf\_{z \le Z} \{ \left\| Az - v \right\|\_{\mathcal{B}}^2 + \alpha \Omega(z) \}\,. \tag{5}$$

It is commonly assumed that the original image is a smooth function with respect to the Sobolev space (Adams, 1975), and the stabilization functional in Eq. (5) is ( ) *<sup>p</sup> q q <sup>W</sup> z z* . Quadratic forms can be used in order to avoid nonlinear restoration algorithms. Note that a Gaussian image model leads to minimization of a quadratic form. In discrete case it corresponds to the Sobolev norm for *p* 2 in Eq. (5). On the other hand, the use of quadratic forms in image restoration often yields undesirable results because of real images are not Gaussian. Now suppose that an image to be restored is a function of bounded variations. Therefore, it may be written as

$$\left|z\right|^{\*} = \inf\_{z \in Z} \{\rho\_{\mathbb{Q}}(Az, v) + aVar(z)\}\,. \tag{6}$$

The variation of a 1D function *f x x ab* ( ), [ , ] is defined as follows:

$$\overset{b}{V}\_{a}^{b}(f) = \sup\_{\mathbf{x}\_{1\dots x}} \sum\_{k=2}^{n} \left| f(\mathbf{x}\_{k}) - f(\mathbf{x}\_{k-1}) \right|. \tag{7}$$

It can be shown, that if the image *zxy* (,) , (,) *xy D* consists of 1D functions of bounded variation along its rows and columns then the image is also a 2D function of bounded variation. Different multidimensional variations were proposed such as variations of Arzela, Vitali, Tonelly, etc. (Vitushkin, 1955). A different approach was suggested by Kronrod, who introduced two functionals in order to describe an image as a function of two variables (Kronrod, 1950). The functionals are given as follows:

$$d\_1(z) = \bigcap\_{-\infty}^{\infty} m\_0(e\_t)dt \text{ , and } \; d\_2(z) = \bigcap\_{-\infty}^{\infty} m\_1(e\_t)dt \; , \tag{8}$$

where *te* is *t* - level set of the function *zxy* (,) , i.e. a set of points (,) *x y* with function values equal to *t*, 0( ) *m et* is the number of components of *te* , and 1( ) *m et* is the length of the set *te* .

Image Restoration Using Two-Dimensional Variations 189

where *zz z L L* = ,..., 1 -1 , *zz z RL N* = ,..., , *M=N-L+1*. A general solution of Eq. (13) can be

1

*m m*

*B p*

*m*

particular solution of linear system of equations in Eq. (13). Suppose that 0 *Lz* , the

<sup>1</sup>

 <sup>1</sup> 2 1 , *m m*

1 11 1 1 1

It is of interest to note that the basis of kernel *A* contains vertical columns of unities; therefore, a general solution of Eq. (13) could contain a periodic structure with the period of blurring (Buades et al., 2006). In our computer simulation we compared three methods: 1) first, substituting Eq. (15) into Eq. (10) and minimizing the functional with respect to

2) the second method minimizes the same functional subject to the first Kronrod's variation given in Eq. (8), which is taken close to that of the original image; 3) minimum norm image restoration from Eq. (10) without considering variations. Actually, if the inverse of the blur operator exists, it can be applied to the observed image to obtain an estimate. This is called inverse filtering. The estimate differs from the actual image by the additional error of amplified noise, and depending on the nature of the blur operator and the noise, it may drastically obscure the desired image information. Hence, inverse filtering is extremely noise sensitive. If the inverse operator does not exist, a solution can be found on the basis of a least squares criterion. A least squares solution minimizes the norm of the residual signal *Az-*v. The least squares solution with minimum norm (energy) is called also the pseudoinverse filtering (Jain, 1989). The first tested method is referred to as Grad method, the second one is called Grad-conditional method, and the last method is named Min-norm method. Fig. 1(a) shows a test input image used in experiments. The size of the image is

1 11 1 1 1 .... .. .. ..

11 11 11

 

\*

*z z*

1 2 *Az Dz Dz v L R* + , (13)

**<sup>e</sup>** , (14)

<sup>2</sup> 0, *<sup>p</sup> z Dv* . (15)

*mL L e D De* **<sup>e</sup>** , (16)

is the Kronecker delta function. The basis of kernel *A* has

.

*<sup>m</sup>* ;

2 1 , *<sup>p</sup> L L z z D v Dz* is a

*<sup>m</sup>* are real variables, <sup>1</sup>

and the linear system of equations in Eq. (1) is rewritten as

where **e***m* is a basis of kernel *A*,

particular solution can be expressed as

The basis **e***m* can be found from Eq. (13) as

 , *ij*

where 12 1 , ,..., *<sup>m</sup> eL m m Lm* 

the following form:

written as

The class of functions of bounded variations given in Eq. (8) is very extensive. These functions possess the following attractive properties: they are differentiable almost everywhere and their Fourier series are convergent almost everywhere. Note that numerous attempts to create a mathematical image model with the help of one functional were unsatisfactory. It can be done on the base of two (independent in a certain way) functionals. It is interesting to point out, that the first variation *d1* in Eq. (8) is a topological characteristic of the image. If the original image is a continuous differentiable function, then the second variation can be represented as

$$d\_2\left(z\right) = \bigcap\_{a$$

If only the second variation is used, the image restoration can be carried out as follows (Perona & Malik,1990):

$$z^\* = \inf\_{z \ge \overline{Z}} \{ \left| Az - v \right|\_B^2 + \alpha \prod\_{a < \overline{c}}^{b \cdot \overline{\mathcal{S}}} \left| \left| \operatorname{grad} \left( z(\mathbf{x}, y) \right) \right| \, \mathbf{dx} \, dy \right\} \,. \tag{10}$$

where *grad(.)* is a gradient operator.

It is of interest to note that this nonlinear method of image restoration minimizes the functional that is identical to the Kronrod's second variation. We propose to minimize the functional in Eq. (10) subject to constraint on the first Kronrod's variation of the image. This approach is referred to as conditional variation approach (Milukova et al., 2010). Next with the help of computer simulation we illustrate the difference in the performance of two variation methods: minimization of the functional in Eq. (10) and conditional minimization of the same functional. Additionally, the performance of minimum norm image restoration from Eq. (10) without considering variations is also provided.

#### **2.2 Restoration of uniformly blurred image with spatial variations**

The impulse response of the 1D uniform blur can be expressed as follows:

$$h\_L(\mathbf{x}) = \begin{cases} \mathbf{1}/L, & \text{if } \; 0 \le \mathbf{x} \le L - 1 \\ \mathbf{0}, & \text{otherwise} \end{cases} \tag{11}$$

where *L* determines a blur extension. It is known that point spread functions for such blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings (Cannon, 1976; Gennery, 1973). If the original image is blurred and noiseless then the blur matrix *A* in Eq. (1) is given as

$$A = \mathbf{1} / L \begin{vmatrix} 1 & 1 & \dots \vert 1 \\ & 1 & 1 \vert \dots & 1 \\ & & 1 \vert 1 & \dots & 1 \\ & & & 1 & \dots & 1 \\ & & & & 1 & \dots & 1 \\ & & & & & \cdot & \cdot & \cdot \\ & & & & 1 & 1 & \dots & 1 \end{vmatrix}\_{\mathsf{M} \models \mathsf{N}} \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \begin{pmatrix} \mathsf{1} \\ & \mathsf{2} \\ & & \mathsf{3} \\ & & & \mathsf{3} \\ & & & \mathsf{4} \\ & & & \mathsf{5} \\ & & & \mathsf{6} \end{pmatrix}\_{\mathsf{M} \not\equiv \mathsf{N}} \tag{12}$$

188 Image Restoration – Recent Advances and Applications

The class of functions of bounded variations given in Eq. (8) is very extensive. These functions possess the following attractive properties: they are differentiable almost everywhere and their Fourier series are convergent almost everywhere. Note that numerous attempts to create a mathematical image model with the help of one functional were unsatisfactory. It can be done on the base of two (independent in a certain way) functionals. It is interesting to point out, that the first variation *d1* in Eq. (8) is a topological characteristic of the image. If the original image is

a continuous differentiable function, then the second variation can be represented as

*a c*

*<sup>B</sup> z Z a c*

from Eq. (10) without considering variations is also provided.

noiseless then the blur matrix *A* in Eq. (1) is given as

*A L*

**2.2 Restoration of uniformly blurred image with spatial variations**  The impulse response of the 1D uniform blur can be expressed as follows:

1

*D*

1 1 ... 1

1 1 ... 1

1 / 1 1 ... 1

1 1 ... 1

(Perona & Malik,1990):

where *grad(.)* is a gradient operator.

<sup>2</sup> () (, ) *b g*

If only the second variation is used, the image restoration can be carried out as follows

 \* <sup>2</sup> inf { (,) } *b g*

It is of interest to note that this nonlinear method of image restoration minimizes the functional that is identical to the Kronrod's second variation. We propose to minimize the functional in Eq. (10) subject to constraint on the first Kronrod's variation of the image. This approach is referred to as conditional variation approach (Milukova et al., 2010). Next with the help of computer simulation we illustrate the difference in the performance of two variation methods: minimization of the functional in Eq. (10) and conditional minimization of the same functional. Additionally, the performance of minimum norm image restoration

> 1 , if 0 1 0, otherwise *<sup>L</sup> L xL h x*

where *L* determines a blur extension. It is known that point spread functions for such blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings (Cannon, 1976; Gennery, 1973). If the original image is blurred and

2

, (12)

*D*

*MxN*

... . . 1 1 ... 1

*z Az v grad z x y dx dy* 

*d z grad z x y dx dy* . (9)

, (10)

, (11)

and the linear system of equations in Eq. (1) is rewritten as

$$Az = \mathbf{D}\_1 \mathbf{z}\_L \ \mathbf{+} \mathbf{D}\_2 \mathbf{z}\_R = \mathbf{v} \ , \tag{13}$$

where *zz z L L* = ,..., 1 -1 , *zz z RL N* = ,..., , *M=N-L+1*. A general solution of Eq. (13) can be written as

$$z^\* = z^p + \sum\_{m=1}^B \beta\_m \mathbf{e}\_m \tag{14}$$

where **e***m* is a basis of kernel *A*, *<sup>m</sup>* are real variables, <sup>1</sup> 2 1 , *<sup>p</sup> L L z z D v Dz* is a particular solution of linear system of equations in Eq. (13). Suppose that 0 *Lz* , the particular solution can be expressed as

$$z^{p^v} = \left\{ 0, D\_2^{-1} v \right\}.\tag{15}$$

The basis **e***m* can be found from Eq. (13) as

$$\mathbf{e}\_m = \left\{ e\_L^m \, \! \! -D\_2^{-1} D\_1 e\_L^m \right\} \, \! \, \tag{16}$$

where 12 1 , ,..., *<sup>m</sup> eL m m Lm* , *ij* is the Kronecker delta function. The basis of kernel *A* has the following form:


It is of interest to note that the basis of kernel *A* contains vertical columns of unities; therefore, a general solution of Eq. (13) could contain a periodic structure with the period of blurring (Buades et al., 2006). In our computer simulation we compared three methods: 1) first, substituting Eq. (15) into Eq. (10) and minimizing the functional with respect to *<sup>m</sup>* ; 2) the second method minimizes the same functional subject to the first Kronrod's variation given in Eq. (8), which is taken close to that of the original image; 3) minimum norm image restoration from Eq. (10) without considering variations. Actually, if the inverse of the blur operator exists, it can be applied to the observed image to obtain an estimate. This is called inverse filtering. The estimate differs from the actual image by the additional error of amplified noise, and depending on the nature of the blur operator and the noise, it may drastically obscure the desired image information. Hence, inverse filtering is extremely noise sensitive. If the inverse operator does not exist, a solution can be found on the basis of a least squares criterion. A least squares solution minimizes the norm of the residual signal *Az-*v. The least squares solution with minimum norm (energy) is called also the pseudoinverse filtering (Jain, 1989). The first tested method is referred to as Grad method, the second one is called Grad-conditional method, and the last method is named Min-norm method. Fig. 1(a) shows a test input image used in experiments. The size of the image is

Image Restoration Using Two-Dimensional Variations 191

(a) (b)

(c) Fig. 2. Image restoration with (a) Grad method, (b) Grad-conditional method, and (c) Min-

Figs. 3(a), 3(b), and 3(c) show differences between the original image and that of restored with the Grad algorithm, the Grad-conditional algorithm, and the Min-norm algorithm, respectively. We see that the second algorithm, which takes into account two Kronrod's variations yields the best recognition performance. A quantitative comparison is given by

10 \*

*z z*

. (17)

\*

<sup>255</sup> *PSNR z z* ( , ) 20log

norm method.

the peak signal-to-noise ratio (PSNR),

256×256 pixels, *N*=256. The signal range is [0-255]. The test input scene is homogeneously blurred with *L*=7. The blurred image is shown in Fig. 1(b).

(a)

Fig. 1. (a) Test image, (b) test image uniformly blurred in horizontal direction with L=7.

The original image has the following values of the Kronrod variations: *d*1=2105, *d*2=12210. The results of image restoration with the variation methods are shown in Figs. 2(a) and 2(b), respectively. The restoration result with the Min-norm algorithm is shown in Fig. 2(c). Subjective visual criterion is defined as an enhanced difference between original and restored images. A pixel is displayed as gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white (with intensity values of 0 and 255, respectively).

190 Image Restoration – Recent Advances and Applications

256×256 pixels, *N*=256. The signal range is [0-255]. The test input scene is homogeneously

(a)

(b)

The original image has the following values of the Kronrod variations: *d*1=2105, *d*2=12210. The results of image restoration with the variation methods are shown in Figs. 2(a) and 2(b), respectively. The restoration result with the Min-norm algorithm is shown in Fig. 2(c). Subjective visual criterion is defined as an enhanced difference between original and restored images. A pixel is displayed as gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white

Fig. 1. (a) Test image, (b) test image uniformly blurred in horizontal direction with L=7.

(with intensity values of 0 and 255, respectively).

blurred with *L*=7. The blurred image is shown in Fig. 1(b).

Fig. 2. Image restoration with (a) Grad method, (b) Grad-conditional method, and (c) Minnorm method.

Figs. 3(a), 3(b), and 3(c) show differences between the original image and that of restored with the Grad algorithm, the Grad-conditional algorithm, and the Min-norm algorithm, respectively. We see that the second algorithm, which takes into account two Kronrod's variations yields the best recognition performance. A quantitative comparison is given by the peak signal-to-noise ratio (PSNR),

$$PSNR\left(z, z^\*\right) = 20\log\_{10}\left(\frac{255}{\left\|z - z^\*\right\|}\right). \tag{17}$$

Image Restoration Using Two-Dimensional Variations 193

So, in order to achieve a good restoration, it is important take into account topological characteristics of the image to be restored. These characteristics can be described by the first

**3. Identification of degradation operators and estimation parameters in the** 

In this section, identification of operator degradation type and estimation of distortion parameters is discussed. Monochrome stationary image distorted by homogeneous linear transformation is considered. Various physical problems can be modeled by such degradations (Bertero & Boccacci, 1998). Identification of the distortion operator is carried out using the Fourier spectrum of the distorted image. Automatic image restoration is performed in three steps, that is, i) identification of distortion operator, ii) estimation of distortion parameters, and iii) image restoration with estimated parameters. Certain types of distortion operators are completely characterized by attributes such as the location of frequency-domain zeros. The techniques (Cannon, 1976; Gennery, 1973) make the following two assumptions: (i) the blurring produces zero crossings in the frequency domain and it can be completely characterized by the location of these zero crossings, and (ii) the location of zero crossings can be determined from the Fourier transform or power cepstrum (the logarithm of the power spectrum) of the observed image. These methods are very simple to use and they can successfully applied in many real-life situations. It is indeed true that the models for motion and focus blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings. On the other hand, blurring models that do not have zero crossings in the frequency domain (e.g., Gaussian modeling atmospheric turbulence) cannot be identified by these techniques. Furthermore, the identification of the zero crossings from the observed image may be quite difficult due to the presence of strong observation noise. Almost all practical implementations of the restoration algorithms assume that the observation noise is a zeromean, white Gaussian process that is uncorrelated to the image signal. In this case, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local region of the observed image

Let us consider an observed image degraded with a linear spatially invariant system and

 

where *hxy* (,) is the impulse response of the system. The Fourier spectrum of the observed

realization, respectively. Several typical examples of linear degradations are provided (Jain,

 

is the frequency response of the liner degradation filter, 1 2 *V*

(18)

 12 12 12 12 , , , ,, (19)

are spectra of the observed image, original image, and noise

 ,,

*V HZ N*

 

= ( , ) ( , ) ( , ), *v Az n h x y z d d n x y*

Kronrod's variation.

**Fourier domain** 

(Yaroslavsky & Eden, 1996).

additive sensor noise, that is,

image is given by

1 2 ,

, and *N*

1 2 ,

where *H*

 1 2 *Z* ,

Fig. 3. Differences between the original image and restored with (a) Grad method, (b) Gradconditional method, and (c) Min-norm method.

The image restored with the Grad method has d1=2050, d2=12015, the image restored with Grad-conditional method has d1=2103, d2=12120, and finally, the image restored by the Min-norm method possesses d1=2218, d2=12343. Table 1 shows the restoration performance of the tested methods in terms of the PSNR versus the blur extension.


Table 1. PSNR (dB) results for the tested methods.

192 Image Restoration – Recent Advances and Applications

(a) (b)

(c) Fig. 3. Differences between the original image and restored with (a) Grad method, (b) Grad-

The image restored with the Grad method has d1=2050, d2=12015, the image restored with Grad-conditional method has d1=2103, d2=12120, and finally, the image restored by the Min-norm method possesses d1=2218, d2=12343. Table 1 shows the restoration performance

> L Min-norm Grad Grad-conditional 3 28.3 34.5 35.2 5 25.1 30.2 31.3 7 22.2 28.5 29.4 9 18.4 26.2 27.3

of the tested methods in terms of the PSNR versus the blur extension.

conditional method, and (c) Min-norm method.

Table 1. PSNR (dB) results for the tested methods.

So, in order to achieve a good restoration, it is important take into account topological characteristics of the image to be restored. These characteristics can be described by the first Kronrod's variation.

## **3. Identification of degradation operators and estimation parameters in the Fourier domain**

In this section, identification of operator degradation type and estimation of distortion parameters is discussed. Monochrome stationary image distorted by homogeneous linear transformation is considered. Various physical problems can be modeled by such degradations (Bertero & Boccacci, 1998). Identification of the distortion operator is carried out using the Fourier spectrum of the distorted image. Automatic image restoration is performed in three steps, that is, i) identification of distortion operator, ii) estimation of distortion parameters, and iii) image restoration with estimated parameters. Certain types of distortion operators are completely characterized by attributes such as the location of frequency-domain zeros. The techniques (Cannon, 1976; Gennery, 1973) make the following two assumptions: (i) the blurring produces zero crossings in the frequency domain and it can be completely characterized by the location of these zero crossings, and (ii) the location of zero crossings can be determined from the Fourier transform or power cepstrum (the logarithm of the power spectrum) of the observed image. These methods are very simple to use and they can successfully applied in many real-life situations. It is indeed true that the models for motion and focus blurs do have zeros in the frequency domain, and they can be uniquely identified by the location of these zero crossings. On the other hand, blurring models that do not have zero crossings in the frequency domain (e.g., Gaussian modeling atmospheric turbulence) cannot be identified by these techniques. Furthermore, the identification of the zero crossings from the observed image may be quite difficult due to the presence of strong observation noise. Almost all practical implementations of the restoration algorithms assume that the observation noise is a zeromean, white Gaussian process that is uncorrelated to the image signal. In this case, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local region of the observed image (Yaroslavsky & Eden, 1996).

Let us consider an observed image degraded with a linear spatially invariant system and additive sensor noise, that is,

$$v = Az + n = \int\_{-\infty}^{\psi} \int h(\mathbf{x} - \underline{\xi}, y - \eta) \mathbf{z}(\underline{\xi}, \eta) d\underline{\xi} d\eta + n(\mathbf{x}, y), \tag{18}$$

where *hxy* (,) is the impulse response of the system. The Fourier spectrum of the observed image is given by

$$V(o\_1, o\_2) = H(o\_1, o\_2) Z(o\_1, o\_2) + N(o\_1, o\_2) \, , \tag{19}$$

where *H*1 2 , is the frequency response of the liner degradation filter, 1 2 *V* , , 1 2 *Z* , , and *N*1 2 , are spectra of the observed image, original image, and noise realization, respectively. Several typical examples of linear degradations are provided (Jain,

Image Restoration Using Two-Dimensional Variations 195

It is of interest to compare the spectra of the original and degraded images. From numerous experiments it is known, that the spectral magnitude of a realistic image is not very informative. It contains information about distribution of the signal energy in the frequency domain. For instance, if we exchange the spectral magnitudes of two similar images belonging to the same class and perform the inverse Fourier transform, then the difference in visual appearance of the original and obtained images will be negligible (Yaroslavsky & Eden, 1996). However, the difference between the spectral magnitudes of the original and degraded images may be significant due to the spectral magnitude of the frequency response of the linear system. Figs. 4(a) and 4(b) show a test original image and its spectral

(a) (b)

In order to identify the distortion operator a database containing various images of spectral magnitudes was created. Training elements of database were obtained on the base of mathematical modeling or computer simulation. In practice, the number of degradation operators is not very large. Next, the spectral magnitude of a degraded image is matched to those of the database. This simple recognition system works well to identify the type of degradation operator for common blurs. Figs. 5 illustrate spectral magnitudes for different common blurring operators. One can observe that spectral magnitudes of distorted images contain mainly the information about distortion operators such as zero crossings on the

Fig. 4. (a)Test original image, (b) spectral magnitude of the original image.

where *a*x*b* is the rectangular aperture.

magnitude.

plane 1 2 , .

1989). Motion blur occurs when there is relative motion between the object and camera during exposure. In this case the impulse response and the frequency response of the linear system of horizontal motion blur are given, respectively, as

$$h(x, y) = \frac{1}{L} \text{rect}\left(\frac{x}{L} - \frac{1}{2}\right) \delta(y) \tag{20}$$

$$H(o\_1, o\_2) = e^{-i\pi w\_1 L} \text{sinc}(w\_1 L) \, , \tag{21}$$

where *L* is the motion path, *rect x if x else* 1, 0,1 , 0, y is the Dirac delta function.

Atmospheric turbulence is a common blur in remote sensing and aerial imaging. For long term exposure through the atmosphere Gaussian model is used. So, the impulse response and the frequency response of the linear system of turbulence blur are given, respectively, as

 22 2 , *a x <sup>y</sup> hxy e* , (22)

$$H(o\_1, o\_2) = \frac{1}{a^2} e^{\frac{-\pi \left(o\_1^{\alpha\_1^2} + o\_2^{\alpha\_2^2}\right)}{a^2}},\tag{23}$$

where *α* is the parameter that determines the severity of blur.

Defocusing is another common type of blurring owing to the finite size of the camera aperture. When the defocusing blur is large, the following uniform model is used. The impulse response and the frequency response of the linear system can be expressed, respectively, as

 222 2 <sup>1</sup> , , 0, elsewhere *xyr hxy r* , (24)

$$H\left(o\_1, o\_2\right) = \frac{f\_1\left(rp\right)}{rp}, \; p = o\_1^{-2} + o\_2^{-2},\tag{25}$$

where *J1* is the first-order Bessel function.

Image blurring also occurs in image acquisition by scanners in which the image pixels are integrated over the scanning aperture. Example of such degradations can be found in image capturing by radar, beam-forming arrays, and display systems using television raster. The impulse response and the frequency response of the linear system can be written, respectively, as

$$h(x, y) = \operatorname{rect}\left(\frac{x}{a}, \frac{y}{b}\right) \; , \tag{26}$$

$$H(o\_1, o\_2) = ab \text{sinc}(ao\_1) \text{sinc}(bo\_2),\tag{27}$$

where *a*x*b* is the rectangular aperture.

194 Image Restoration – Recent Advances and Applications

1989). Motion blur occurs when there is relative motion between the object and camera during exposure. In this case the impulse response and the frequency response of the linear

> 1 1 , , <sup>2</sup> *<sup>x</sup> h x <sup>y</sup> rect <sup>y</sup> L L*

Atmospheric turbulence is a common blur in remote sensing and aerial imaging. For long term exposure through the atmosphere Gaussian model is used. So, the impulse response and the frequency response of the linear system of turbulence blur are given, respectively, as

> 22 2 , *a x <sup>y</sup> hxy e*

Defocusing is another common type of blurring owing to the finite size of the camera aperture. When the defocusing blur is large, the following uniform model is used. The impulse response

> 2 <sup>1</sup> , ,

 <sup>1</sup> 2 2 1 2 1 2 , , *J rp H p rp*

Image blurring also occurs in image acquisition by scanners in which the image pixels are integrated over the scanning aperture. Example of such degradations can be found in image capturing by radar, beam-forming arrays, and display systems using television raster. The impulse response and the frequency response of the linear system can be written,

> , , , *<sup>x</sup> <sup>y</sup> h x y rect a b*

> >

 1 2 , sinc sinc , 1 2 (27)

*H ab a b*

*xyr hxy r* 

 

 <sup>1</sup> 1 2 <sup>1</sup> , sinc( ) *i wL H e wL* 

 

and the frequency response of the linear system can be expressed, respectively, as

 

 

where *J1* is the first-order Bessel function.

respectively, as

where *α* is the parameter that determines the severity of blur.

1 2 2 <sup>1</sup> ,

 *H e <sup>a</sup> a*

 2 2 1 2 2

222

 

0, elsewhere

  (20)

y is the Dirac delta function.

, (24)

, (25)

(26)

, (21)

, (22)

, (23)

system of horizontal motion blur are given, respectively, as

where *L* is the motion path, *rect x if x else* 1, 0,1 , 0,

It is of interest to compare the spectra of the original and degraded images. From numerous experiments it is known, that the spectral magnitude of a realistic image is not very informative. It contains information about distribution of the signal energy in the frequency domain. For instance, if we exchange the spectral magnitudes of two similar images belonging to the same class and perform the inverse Fourier transform, then the difference in visual appearance of the original and obtained images will be negligible (Yaroslavsky & Eden, 1996). However, the difference between the spectral magnitudes of the original and degraded images may be significant due to the spectral magnitude of the frequency response of the linear system. Figs. 4(a) and 4(b) show a test original image and its spectral magnitude.

Fig. 4. (a)Test original image, (b) spectral magnitude of the original image.

In order to identify the distortion operator a database containing various images of spectral magnitudes was created. Training elements of database were obtained on the base of mathematical modeling or computer simulation. In practice, the number of degradation operators is not very large. Next, the spectral magnitude of a degraded image is matched to those of the database. This simple recognition system works well to identify the type of degradation operator for common blurs. Figs. 5 illustrate spectral magnitudes for different common blurring operators. One can observe that spectral magnitudes of distorted images contain mainly the information about distortion operators such as zero crossings on the plane 1 2 , .

Image Restoration Using Two-Dimensional Variations 197

(g) (h) Fig. 5. Spectral magnitudes of the test original image shown in Fig. 4(a) degraded by: (a) atmosphere turbulence defocusing (Eq. 23) with *α* = 2, (b) atmosphere turbulence defocusing (Eq. (23) with *α* = 4, (c) isotropic defocusing (Eq. 25) with *r* = 4, (d) isotropic defocusing (Eq. 25) with *r* = 6, (e) 1D motion blur (Eq. 21) with *L* = 4, (f) 1D motion blur (Eq. 21) with *L*= 6, (g) convolution with rectangular aperture (Eq. 27) with *a*=4, *b*=2, and (h) convolution with

Actually, composite degradations can be considered as a combination of the basic distortion operators. In this case, the Fourier spectrum of a new composite operator is the product of the spectra of used basic operators. Figs. 6(a) and 6(b) show the spectral magnitudes of the

(a) (b)

Fig. 6. Spectral magnitudes of the test original image shown in Fig. 4(a) with composite degradations: (a) isotropic defocusing (Eq. 25) with *r* = 6 and 1D motion blur (Eq. 21) with *L* = 4, (b) isotropic defocusing (Eq.25) with *r* = 6 and 1D motion blur (Eq. 21) with *L* = 6.

test original image degraded with isotropic blur and horizontal motion.

rectangular aperture (Eq. 27) with *a*=8, *b*=4.

196 Image Restoration – Recent Advances and Applications

(a) (b)

(c) (d)

(e) (f)

Fig. 5. Spectral magnitudes of the test original image shown in Fig. 4(a) degraded by: (a) atmosphere turbulence defocusing (Eq. 23) with *α* = 2, (b) atmosphere turbulence defocusing (Eq. (23) with *α* = 4, (c) isotropic defocusing (Eq. 25) with *r* = 4, (d) isotropic defocusing (Eq. 25) with *r* = 6, (e) 1D motion blur (Eq. 21) with *L* = 4, (f) 1D motion blur (Eq. 21) with *L*= 6, (g) convolution with rectangular aperture (Eq. 27) with *a*=4, *b*=2, and (h) convolution with rectangular aperture (Eq. 27) with *a*=8, *b*=4.

Actually, composite degradations can be considered as a combination of the basic distortion operators. In this case, the Fourier spectrum of a new composite operator is the product of the spectra of used basic operators. Figs. 6(a) and 6(b) show the spectral magnitudes of the test original image degraded with isotropic blur and horizontal motion.

Fig. 6. Spectral magnitudes of the test original image shown in Fig. 4(a) with composite degradations: (a) isotropic defocusing (Eq. 25) with *r* = 6 and 1D motion blur (Eq. 21) with *L* = 4, (b) isotropic defocusing (Eq.25) with *r* = 6 and 1D motion blur (Eq. 21) with *L* = 6.

Image Restoration Using Two-Dimensional Variations 199

If noise fluctuation in Eq. (19) is small, the phase of the distortion operator is equal to the difference between the phases of the degraded and original images. If the distortion operator is Gaussian, its phase is zero, and the phases of the distorted and original images coincide If the distortion operator is a finite function, e.g. (,) *xy W* , then the phase of the distorted image may differ from the phase of the original image by *±π*, and points at which the phase jumps by *±π* coincide with the location of zeros of the spectral magnitude of the degraded image. These zeros, for even functions are all located on the real axis. So, the phase of the original image either coincides with that of the distorted image or differs from that of the distorted image by *±π.* Fig. 7 shows the differences between the phases of the distorted and original images for rectangular and circular aperture blurs. Therefore, under certain conditions, we can identify the type of the distortion operator and estimate its

In this chapter we treated the problem of restoring linearly degraded image using twodimensional image variations. The restored image minimizes the objective functional subject to the Kronrod's variations. In order to achieve a good restoration, it is important take into account topological characteristics of the original image, which are well described by the first Kronrod's variation. The first step in restoring a degraded image is the identification of the type of degradation operator. It can be done by matching of the spectral magnitude of the degraded image with those of created database. Under certain conditions, the phase of the distortion operator may be also estimated from the distorted image. Extensive testing it was shown that the original image can be automatically restored by proper choice of the

This work was supported by the Russian Foundation for Basic Research, project no.

Adams, R. A. (1975). *Sobolev Spaces*, Academic Press, ISBN 978-0-12-044150-1, Boston, USA Banham, M., & Katsaggelos, A. (1997). Digital image restoration. *IEEE Signal Processing* 

Buades, A., Coll, B. & Morel, J.M. (2006). The staircasing effect in neighborhood filters and

Bertero, M. & Boccacci, P. (1998). *Introduction to inverse problems in imaging,* Institute of

Biemond, J., Lagendijk, R.L. & Mersereau, R.M. (1990). Iterative methods for image deblurring. *Proc. IEEE. Vol. 78*, No. 5, (May 1990), pp. 856-883, ISSN 0018-9219 Bovik, A. (2005). *Handbook of image and video processing* (2nd ed.), Academic Press, ISBN 0-12-

its solution. *IEEE Trans. Image Process*., Vol. 15, No. 6, (June 2006), pp 1499-1505,

*Magazine,* Vol. 14, No.2, (March 1997), pp. 24-41, ISSN 1053-5888

Physics Publishing, ISBN 0-7503-0435-9, Bristol, UK

spectral phase from the observed degraded image.

parameters of the proposed method.

ISSN 1057-7149

119792-1, NJ, USA

**5. Acknowledgment** 

11\_07\_00361.

**6. References** 

**4. Conclusion** 

One can observe that since one of the distortion operators dominates, the spectral magnitudes of composite degradations cannot be synthesized by a simple combination of those of the basic degradation operators. This means that the number of training elements of a matching system should be drastically increased.

In the Fourier representation of images, spectral magnitude and phase tend to play different roles (Oppenheim & Lim, 1981). For instance, in some situations many of the important features of an image are preserved if only the phase is retained. Furthermore, under a variety of conditions, phase information alone is sufficient to completely reconstruct an image to within a scale factor.

Fig. 7. (a) Cepstrum of the original image degraded by convolution with rectangular aperture (Eq. 27) with *a*=2, *b*=4, (b) the difference of phases of the original and distorted (Fig. 7(a)) images, (c) cepstrum of the original image degraded by convolution with circular aperture with radius of 4, and (d) the difference of phases of the original and distorted (Fig. 7(c)) images.

If noise fluctuation in Eq. (19) is small, the phase of the distortion operator is equal to the difference between the phases of the degraded and original images. If the distortion operator is Gaussian, its phase is zero, and the phases of the distorted and original images coincide If the distortion operator is a finite function, e.g. (,) *xy W* , then the phase of the distorted image may differ from the phase of the original image by *±π*, and points at which the phase jumps by *±π* coincide with the location of zeros of the spectral magnitude of the degraded image. These zeros, for even functions are all located on the real axis. So, the phase of the original image either coincides with that of the distorted image or differs from that of the distorted image by *±π.* Fig. 7 shows the differences between the phases of the distorted and original images for rectangular and circular aperture blurs. Therefore, under certain conditions, we can identify the type of the distortion operator and estimate its spectral phase from the observed degraded image.

## **4. Conclusion**

198 Image Restoration – Recent Advances and Applications

One can observe that since one of the distortion operators dominates, the spectral magnitudes of composite degradations cannot be synthesized by a simple combination of those of the basic degradation operators. This means that the number of training elements of

In the Fourier representation of images, spectral magnitude and phase tend to play different roles (Oppenheim & Lim, 1981). For instance, in some situations many of the important features of an image are preserved if only the phase is retained. Furthermore, under a variety of conditions, phase information alone is sufficient to completely reconstruct an

(a) (b)

(c) (d)

aperture (Eq. 27) with *a*=2, *b*=4, (b) the difference of phases of the original and distorted (Fig. 7(a)) images, (c) cepstrum of the original image degraded by convolution with circular aperture with radius of 4, and (d) the difference of phases of the original and distorted (Fig.

Fig. 7. (a) Cepstrum of the original image degraded by convolution with rectangular

a matching system should be drastically increased.

image to within a scale factor.

7(c)) images.

In this chapter we treated the problem of restoring linearly degraded image using twodimensional image variations. The restored image minimizes the objective functional subject to the Kronrod's variations. In order to achieve a good restoration, it is important take into account topological characteristics of the original image, which are well described by the first Kronrod's variation. The first step in restoring a degraded image is the identification of the type of degradation operator. It can be done by matching of the spectral magnitude of the degraded image with those of created database. Under certain conditions, the phase of the distortion operator may be also estimated from the distorted image. Extensive testing it was shown that the original image can be automatically restored by proper choice of the parameters of the proposed method.

## **5. Acknowledgment**

This work was supported by the Russian Foundation for Basic Research, project no. 11\_07\_00361.

## **6. References**

Adams, R. A. (1975). *Sobolev Spaces*, Academic Press, ISBN 978-0-12-044150-1, Boston, USA


**0**

**10**

*Japan*

**for Blind Restoration**

Rachel Mabanag Chong and Toshihisa Tanaka *Tokyo University of Agriculture and Technology*

*g*(*x*, *y*) = *f*(*x*, *y*) ∗ *h*(*x*, *y*) + *n*(*x*, *y*) (1)

**Harnessing the Potentials of Image Extrema**

Images contain a wealth of information. The advances in modern technology makes it easier to deal with different kinds of images. As a result, its applications have been increasing and spreading out to different fields of research. Despite these, degradations are unavoidable owing to the fact that the acquisition systems are imperfect and the environment can highly vary. One of the most studied type of degradation is the blur. The process of removing this from an image is known as image deconvolution, restoration or reconstruction. We illustrate its importance in the field of image processing with an example in Fig. 1. In Fig. 1(a), an image is acquired but degradations make it difficult for further processing. Reconstruction is applied and the resulting image in Fig. 1(b) can now be used in order to obtain an accurate representation of the original bar code as shown in Fig. 1(c). In essence, reconstruction is necessary as a preprocessing step for degraded images in order to extract more information from it. As a result, this has been studied in various fields of applications such as bar code interpretation (Choksi & van Gennip, 2010; Esedoglu, 2004; Yahyanejad & Strom, 2010), fingerprint identification (Cappelli et al., 2007), iris recognition (Kang & Park, 2007), face identification (Chu, Yang & Chen, 2010; Nishiyama et al., 2010; Xin et al., 2003), among others. In the two-dimensional domain, degradation is mathematically modelled as (Lagendijk &

where the symbol ∗ is the two-dimensional convolution process. On the other hand, the variables *g*(*x*, *y*), *f*(*x*, *y*), *h*(*x*, *y*), and *n*(*x*, *y*) represent the degraded image, original image,

(a) degraded image (b) reconstructed image (c) reconstructed code

Fig. 1. Reconstruction of 2D bar code images in (Chu, Yang, Pan & Chen, 2010).

**1. Introduction**

Biemond, 2005):


## **Harnessing the Potentials of Image Extrema for Blind Restoration**

Rachel Mabanag Chong and Toshihisa Tanaka *Tokyo University of Agriculture and Technology Japan*

#### **1. Introduction**

200 Image Restoration – Recent Advances and Applications

Cannon, M. (1976). Blind deconvolution of spatially invariant image blurs with phase. *IEEE* 

Chan, T. & Wong, C. (1998). Total variation blind deconvolution. *IEEE Trans. Image Process*.,

Gennery, D.B. (1973). Determination of optical transfer function by inspection of frequency-

González, R. & Woods, R. (2008). Digital image processing (3rd ed.), Prentice Hall, ISBN 0-

Jain, A.K. (1989). *Fundamentals of digital image processing*, Prentice Hall, ISBN 0-13-332578-4,

Kay, S. (1993). *Fundamentals of statistical signal processing: estimation theory*, Prentice Hall,

Kundur, D., & Hatzinakos, D. (1996). Blind image deconvolution. *IEEE Signal Processing* 

Kronrod, A. S. (1950). On functions of two variables. *Uspekhi Mat. Nauk*, Vol. 5, No. 1,

Milukova, O., Kober, V., Karnaukhov, V. & Ovseevich, I.A. (2010). Restoration of blurred

Milukova, O., Kober, V., Karnaukhov, V. & Ovseevich, I.A. (2010). Spectral analysis of

Milukova, O., Kober, V., Karnaukhov, V. & Ovseevich, I.A. (2011). Iterative global and local

Oppenheim, A.V. & Lim, J.S. (1981). The importance of phase in signals. *Proceedings of the* 

Perona, P. & Malik, J. (1990). Scale-space and edge detection using anisotropic diffusion.

Rudin, L., Osher, S. & Fatemi, E. (1992). Nonlinear total variation based noise removal

Sezan, M.I. & Tekalp, A.M. (1990). Survey of recent developments in digital image restoration. *Optical Engineering*, Vol. 29, No. 5, (May 1990), pp. 393-404, ISSN 0091-3286 Sroubek, F. & Flusser, J. (2003). Multichannel blind iterative image restoration. *IEEE Trans. Image Process*., Vol. 12, No. 9, (September 2003), pp. 1094–1106, ISSN 1057-7149 Tikhonov, A.N. & Arsenin, V.Y. (1977). *Solution of ill-posed problems*, Winston & Sons, ISBN 0-

Vitushkin, A.G. (1955). *On multidimensional variations*, GITTL, Moscow, Russia (in Russian) Yaroslavsky, L. P. & Eden, M. (1996). *Fundamentals of digital optics*, Birkhauser, ISBN 3-7643-

images with conditional total variation method. *Pattern Recognition and Image* 

distorted images in restoration problems. *Pattern Recognition and Image Analysis* Vol.

methods of image restoration. *Pattern Recognition and Image Analysis* Vol. 21, No. 2,

*IEEE Transaction on Pattern Analysis and Machine Intelligence*, Vol. 12, No. 7, (July

*Magazine,* Vol. 13, No. 3, (May 1996), pp. 73-76, ISSN 1053-5888

*Analysis* Vol. 20, No. 2, (May 2010), pp. 179-184, ISSN 1054-6618

*IEEE*, Vol. 69, No. 5, (May 1981), pp. 529 - 541, ISSN 0018-9219

algorithms. *Phys. D*, Vol. 60, pp. 259–268, ISSN 0167-278

20, No. 3, (September 2010), pp. 335-340, ISSN 1054-6618

Vol. 7, No. 3, (March 1998), pp 370–375, ISSN 1057-7149

(1976), ISSN: 0096-3518

ISSN 1084-7529

NY, USA

13-1687288, NJ, USA

ISBN 0-13-345711-7, NJ, USA

(January 1950), pp. 24–134, ISSN 0042-1316

(May 2011), pp. 309-311, ISSN 1054-6618

1990), pp. 629-639, ISSN 0162-8828

470-99124-0, Washington, USA

3833-9, Boston, USA

*Trans. Acoust. Speech Sig. Proc*., Vol. ASSP-24, No. 1, (Fenruary 1976), pp. 58—63

domain plot. *J. Opt. Soc. Am*., Vol. 63, No. 12, (December 1973), pp. 1571—1577,

Images contain a wealth of information. The advances in modern technology makes it easier to deal with different kinds of images. As a result, its applications have been increasing and spreading out to different fields of research. Despite these, degradations are unavoidable owing to the fact that the acquisition systems are imperfect and the environment can highly vary. One of the most studied type of degradation is the blur. The process of removing this from an image is known as image deconvolution, restoration or reconstruction. We illustrate its importance in the field of image processing with an example in Fig. 1. In Fig. 1(a), an image is acquired but degradations make it difficult for further processing. Reconstruction is applied and the resulting image in Fig. 1(b) can now be used in order to obtain an accurate representation of the original bar code as shown in Fig. 1(c). In essence, reconstruction is necessary as a preprocessing step for degraded images in order to extract more information from it. As a result, this has been studied in various fields of applications such as bar code interpretation (Choksi & van Gennip, 2010; Esedoglu, 2004; Yahyanejad & Strom, 2010), fingerprint identification (Cappelli et al., 2007), iris recognition (Kang & Park, 2007), face identification (Chu, Yang & Chen, 2010; Nishiyama et al., 2010; Xin et al., 2003), among others.

In the two-dimensional domain, degradation is mathematically modelled as (Lagendijk & Biemond, 2005):

$$\log(\mathbf{x}, \mathbf{y}) = f(\mathbf{x}, \mathbf{y}) \* h(\mathbf{x}, \mathbf{y}) + n(\mathbf{x}, \mathbf{y}) \tag{1}$$

where the symbol ∗ is the two-dimensional convolution process. On the other hand, the variables *g*(*x*, *y*), *f*(*x*, *y*), *h*(*x*, *y*), and *n*(*x*, *y*) represent the degraded image, original image,

Fig. 1. Reconstruction of 2D bar code images in (Chu, Yang, Pan & Chen, 2010).

blur point spread function (PSF), and noise, respectively. In most cases, it is assumed that noise is negligible and the model is simplified to a purely convolutional process. Thus, the vector-matrix form of the model in the image domain is:

$$\mathbf{g} = H\mathbf{f} \tag{2}$$

and

$$\mathbf{g} = \mathbf{F}\mathbf{h} \tag{3}$$

pixel values

50 60 70 80 90 100 110 120

Fig. 2. Sample plots of extrema values in a row.

**2.2 Detection and identification technique**

1. dispersion, *dispS*

plateaus.

image columns (a) Unblurred

The plots in Fig. 2 show a row of pixel values where the extrema are marked. Fig.2(a) is from the unblurred image while its Gaussian blurred version is shown in Fig. 2(b). These illustrate that there are more extrema values in an unblurred image than its blurred version. Thus, the presence of blur makes some extrema disappear resulting to a decrease in its number. Aside from this, the unblurred images have pixel values that are highly separated from each other. In other words, the extrema are non-neighbouring pixels with distances that are only small. On the other hand, the blurred images have more neighbouring extrema because the presence of blur flattened the image pixels. This results to extrema values that are more clustered with greater separation from each other. This is illustrated in Fig. 3. It can be seen that the minima and maxima values have the same behaviour in the presence of blur. Similarly, these characteristics can also be observed not only in the image's rows but also in its columns.

Taking into account the above-mentioned observations, blur detection and identification can be accomplished by extracting the features and using a classifier. Consider a blur classification problem with *K* categories consisting of unblurred images and *K* − 1 types of blurs. For each extrema in an image, the distances and plateaus are counted by rows then by columns. In the context of this work, we define distance as the number of pixels between extrema while plateau as the number of consecutive extrema values. Let *hS*(*i*) be an extrema histogram of *S* with *S* ∈ {dn, dx, pn, px} and the letters *i*, d, p, n, and x stand for histogram index, distance, plateau, minima, and maxima, respectively. Since there are two possible directions in populating the histogram, *S* is appended with \_c or \_r to indicate column-wise or row-wise,

This reflects the closeness between extrema for distances. On the other hand, it also quantifies the number of standalone extrema or those that do not have neighbours for

> *hS*(1) = *hS*(1) *mS* ∑ *i*=1

*hS*(*i*)

(4)

respectively. For each extrema histogram, the feature values are defined as follows:

*dispS* = <sup>ˆ</sup>

pixel values

Harnessing the Potentials of Image Extrema for Blind Restoration 203

50 60 70 80 90 100 110 120

image columns (b) Gaussian blurred

in the blur domain. The small letters are vectors that represent their respective quantities, which are lexicographically ordered. On the other hand, the capital letters are Toeplitz matrices constructed from their corresponding quantities. We adapt the terms image and blur domain as used in (He et al., 2009) to indicate the quantity being estimated and the direction to which the reconstruction cost function is being projected. In this way, derivations can be more succinct. Based on these equations, the unblurred image can be easily computed if the models for the degraded image and PSF are known. However, this is not the case in actual applications. Images cannot be modelled in a straightforward manner thus, their features and properties are usually utilized. For some applications, probability models are created based on the imaging conditions and type of scenes (Simoncelli, 2005). On the other hand, blurring functions can be mathematically modelled (Lagendijk & Biemond, 2005). Exploiting the characteristics of these models can decrease the complexity of determining the unblurred image.

The reconstruction of a degraded image undergoes the following major steps: blur detection and identification; reference PSF (RPSF) determination; deconvolution; and image quality assessment. In this chapter, we will explore the various characteristics of image extrema that make it useful in each step. These are tested on numerous natural color images wherein synthetic and actual blurs are also considered. Experimental data will illustrate the effectiveness of the methods.

## **2. Blur detection and identification**

Reconstruction algorithms assume that blurs are always present. However, subjecting an unblurred image to this will only waste resources. To avoid this, preprocessing the image with a blur detection method is a must. Some methods are based on edge information (Chung et al., 2004; Marziliano et al., 2002; Rooms et al., 2002; Tong et al., 2004) or frequency domain characteristics (Aizenberg et al., 2002; n.d.; 2006; 2008). A downside to these is its restrictiveness towards image size and orientation. Aside from this, most edge-based methods are only limited to detection and are not capable of identifying the type of degradation. Transform-based methods have promising results but are mostly applicable to non-Gaussian types since these exploit the null patterns. These limitations can be overcome by using the characteristics of image extrema (Chong & Tanaka, 2008; 2009). This method can be applied to images with different sizes and orientations. Additional parameters or settings are not necessary and different types of degradations can be included.

### **2.1 Images and their extrema**

The presence of blurs will lessen the perception of details in an image. The method herein will show that the loss of details is not only obvious in edges but also in the extrema. For this reason, we can generally call this as image extrema analysis (IEXA)(Chong & Tanaka, 2009).

2 Will-be-set-by-IN-TECH

blur point spread function (PSF), and noise, respectively. In most cases, it is assumed that noise is negligible and the model is simplified to a purely convolutional process. Thus, the

in the blur domain. The small letters are vectors that represent their respective quantities, which are lexicographically ordered. On the other hand, the capital letters are Toeplitz matrices constructed from their corresponding quantities. We adapt the terms image and blur domain as used in (He et al., 2009) to indicate the quantity being estimated and the direction to which the reconstruction cost function is being projected. In this way, derivations can be more succinct. Based on these equations, the unblurred image can be easily computed if the models for the degraded image and PSF are known. However, this is not the case in actual applications. Images cannot be modelled in a straightforward manner thus, their features and properties are usually utilized. For some applications, probability models are created based on the imaging conditions and type of scenes (Simoncelli, 2005). On the other hand, blurring functions can be mathematically modelled (Lagendijk & Biemond, 2005). Exploiting the characteristics of these models can decrease the complexity of determining the unblurred

The reconstruction of a degraded image undergoes the following major steps: blur detection and identification; reference PSF (RPSF) determination; deconvolution; and image quality assessment. In this chapter, we will explore the various characteristics of image extrema that make it useful in each step. These are tested on numerous natural color images wherein synthetic and actual blurs are also considered. Experimental data will illustrate the

Reconstruction algorithms assume that blurs are always present. However, subjecting an unblurred image to this will only waste resources. To avoid this, preprocessing the image with a blur detection method is a must. Some methods are based on edge information (Chung et al., 2004; Marziliano et al., 2002; Rooms et al., 2002; Tong et al., 2004) or frequency domain characteristics (Aizenberg et al., 2002; n.d.; 2006; 2008). A downside to these is its restrictiveness towards image size and orientation. Aside from this, most edge-based methods are only limited to detection and are not capable of identifying the type of degradation. Transform-based methods have promising results but are mostly applicable to non-Gaussian types since these exploit the null patterns. These limitations can be overcome by using the characteristics of image extrema (Chong & Tanaka, 2008; 2009). This method can be applied to images with different sizes and orientations. Additional parameters or settings are not

The presence of blurs will lessen the perception of details in an image. The method herein will show that the loss of details is not only obvious in edges but also in the extrema. For this reason, we can generally call this as image extrema analysis (IEXA)(Chong & Tanaka, 2009).

*g* = *H f* (2)

*g* = *Fh* (3)

vector-matrix form of the model in the image domain is:

and

image.

effectiveness of the methods.

**2.1 Images and their extrema**

**2. Blur detection and identification**

necessary and different types of degradations can be included.

Fig. 2. Sample plots of extrema values in a row.

The plots in Fig. 2 show a row of pixel values where the extrema are marked. Fig.2(a) is from the unblurred image while its Gaussian blurred version is shown in Fig. 2(b). These illustrate that there are more extrema values in an unblurred image than its blurred version. Thus, the presence of blur makes some extrema disappear resulting to a decrease in its number. Aside from this, the unblurred images have pixel values that are highly separated from each other. In other words, the extrema are non-neighbouring pixels with distances that are only small.

On the other hand, the blurred images have more neighbouring extrema because the presence of blur flattened the image pixels. This results to extrema values that are more clustered with greater separation from each other. This is illustrated in Fig. 3. It can be seen that the minima and maxima values have the same behaviour in the presence of blur. Similarly, these characteristics can also be observed not only in the image's rows but also in its columns.

#### **2.2 Detection and identification technique**

Taking into account the above-mentioned observations, blur detection and identification can be accomplished by extracting the features and using a classifier. Consider a blur classification problem with *K* categories consisting of unblurred images and *K* − 1 types of blurs. For each extrema in an image, the distances and plateaus are counted by rows then by columns. In the context of this work, we define distance as the number of pixels between extrema while plateau as the number of consecutive extrema values. Let *hS*(*i*) be an extrema histogram of *S* with *S* ∈ {dn, dx, pn, px} and the letters *i*, d, p, n, and x stand for histogram index, distance, plateau, minima, and maxima, respectively. Since there are two possible directions in populating the histogram, *S* is appended with \_c or \_r to indicate column-wise or row-wise, respectively. For each extrema histogram, the feature values are defined as follows:

1. dispersion, *dispS*

This reflects the closeness between extrema for distances. On the other hand, it also quantifies the number of standalone extrema or those that do not have neighbours for plateaus.

$$\hat{d}isp\_S = \hat{h}\_S(1) = \frac{h\_S(1)}{\frac{m\_S}{m\_S}}\tag{4}$$

where *mS* is the maximum histogram index in *S*.

This is the density of the highest count with respect to the total count either for distances

Harnessing the Potentials of Image Extrema for Blind Restoration 205

*mS* ∑ *i*=1 ˆ

This measures the range of distances or plateaus where its counts are considered

*hS*(*i*) <sup>≥</sup> ¯

*hS*}

*hwS* = max{*iw*} − min{*iw*} (7)

*hS*(*i*) : 1 ≤ *i* ≤ *mS*} (5)

*hS*(*i*). (6)

(8)

*ctnS* <sup>=</sup> max{<sup>ˆ</sup>

¯ *hS* <sup>=</sup> <sup>1</sup> *mS*

significant. The histogram indices with significant counts are determined by:

*iw* ∈ {*<sup>i</sup>* : <sup>ˆ</sup>

*mS* ∑ *i*=1 ˆ *hS*(*i*) <sup>−</sup> ¯ *hS* 2

In summary, there are 5 feature values for each *S* where the directions for counting the distances (or plateaus) are row- and column-wise in order to completely account the extrema behaviour. Thus, each image must have a total of 40 features. Finally, all quantities are normalized with their respective maximum feature values for the training and testing sets. The blur classification is accomplished by using nearest neighbour (Cover & Hart, 1967).It is noteworthy to mention that other more sophisticated classifiers are also applicable that may yield better performance. In this section, the discriminative power of the proposed features will be demonstrated despite the simplicity of the classifier. Through the experimental data,

The images in the experiments were coloured however, we prefer to use the green channel. This is based on the fact that cameras have twice as much green sensors than the red or blue and that computations will be lesser as opposed to using the three channels. As a result, the experiment starts with the extraction of green component for each image. A database of different coloured natural images is used. This is composed of 300 unblurred images with sizes that randomly varies between 640 × 480 and 480 × 640. These images are then synthetically blurred using (1). We consider the following models for the synthetic blurs

*hS* This is the average distance or plateau count for a given histogram.

2. concentration, *ctnS*

4. histogram width, *hwS*

5. variance of the counts, *σ*<sup>2</sup>

**2.3 Experimental results**

**2.3.1 Experiment descriptions**

3. arithmetic mean of the counts, ¯

thus, *hwS* can be computed as follows:

*S* This is the measure of count dispersion of a histogram.

it will be shown that high values of accuracy are attainable.

(Banham & Katsaggelos, 1997; Lagendijk & Biemond, 2005):

*σ*2 *<sup>S</sup>* <sup>=</sup> <sup>1</sup> *mS*

or plateaus.

Fig. 3. Example of an unblurred image with its Gaussian blurred version.

where *mS* is the maximum histogram index in *S*.

2. concentration, *ctnS*

4 Will-be-set-by-IN-TECH

(a) Unblurred image (b) Blurred image

(c) Minima values of 3(a) (d) Minima values of 3(b)

(e) Maxima values of 3(a) (f) Maxima values of 3(b)

Fig. 3. Example of an unblurred image with its Gaussian blurred version.

This is the density of the highest count with respect to the total count either for distances or plateaus.

$$\mathcal{Cat}\_{\mathcal{S}} = \max \{ \hat{h}\_{\mathcal{S}}(i) : 1 \le i \le m\_{\mathcal{S}} \} \tag{5}$$

3. arithmetic mean of the counts, ¯ *hS*

This is the average distance or plateau count for a given histogram.

$$
\tilde{h}\_S = \frac{1}{m\_S} \sum\_{i=1}^{m\_S} \hat{h}\_S(i). \tag{6}
$$

4. histogram width, *hwS*

This measures the range of distances or plateaus where its counts are considered significant. The histogram indices with significant counts are determined by:

$$i\_w \in \{ i : \hat{h}\_S(i) \ge \bar{h}\_S \} $$

thus, *hwS* can be computed as follows:

$$hw\_S = \max\{i\_w\} - \min\{i\_w\} \tag{7}$$

5. variance of the counts, *σ*<sup>2</sup> *S*

This is the measure of count dispersion of a histogram.

$$
\sigma\_S^2 = \frac{1}{m\_S} \sum\_{i=1}^{m\_S} \left( \hat{h}\_S(i) - \bar{h}\_S \right)^2 \tag{8}
$$

In summary, there are 5 feature values for each *S* where the directions for counting the distances (or plateaus) are row- and column-wise in order to completely account the extrema behaviour. Thus, each image must have a total of 40 features. Finally, all quantities are normalized with their respective maximum feature values for the training and testing sets.

The blur classification is accomplished by using nearest neighbour (Cover & Hart, 1967).It is noteworthy to mention that other more sophisticated classifiers are also applicable that may yield better performance. In this section, the discriminative power of the proposed features will be demonstrated despite the simplicity of the classifier. Through the experimental data, it will be shown that high values of accuracy are attainable.

#### **2.3 Experimental results**

#### **2.3.1 Experiment descriptions**

The images in the experiments were coloured however, we prefer to use the green channel. This is based on the fact that cameras have twice as much green sensors than the red or blue and that computations will be lesser as opposed to using the three channels. As a result, the experiment starts with the extraction of green component for each image. A database of different coloured natural images is used. This is composed of 300 unblurred images with sizes that randomly varies between 640 × 480 and 480 × 640. These images are then synthetically blurred using (1). We consider the following models for the synthetic blurs (Banham & Katsaggelos, 1997; Lagendijk & Biemond, 2005):

Input No. of Accuracy ( % )

Harnessing the Potentials of Image Extrema for Blind Restoration 207

Unblurred 167 99.40 99.40 98.80 Gaussian 1,169 100.00 100.00 NA HM 1,002 100.00 100.00 13.17 OOF 1,002 99.90 100.00 98.80 TOTAL 3,340 **99.94 99.97** 59.28

Input No. of Accuracy ( % )

Unblurred 167 **99.40 99.40** Gaussian 1,169 90.16 **95.21** HM 1,002 **99.40** 98.90 OOF 1,002 93.61 **96.01** TOTAL 3,340 94.43 **96.77**

The blur detection performance in this section is compared with the work in (Tong et al., 2004), which detects blur with Haar wavelet transform (BDHWT). This involves the creation of an edge map based on a three-level decomposition of an image. The different edge types are then identified and counted by using a threshold and comparative conditions. The resulting ratio

The accuracy of the detection algorithms are shown in Table 1. The NA in the table indicates not applicable since the method in (Tong et al., 2004) is only for unblurred, motion, and OOF blurs. It can be observed that accuracies are consistently higher when extrema analysis is used. Furthermore, the result after feature selection yielded a better performance than using

The classification accuracies for IEXA1 and IEXA2 are in Table 2. Only the accuracy for the unblurred images did not change after feature selection. Motion classification performance slightly decreased however, Gaussian and OOF have higher values. As a result, the decrease

This section shows the effectiveness of using extrema features for blur detection and identification. Furthermore, the accuracy is further improved by feature selection. In the

The previous section shows that maxima and minima can be exploited to determine and identify the blur in an image. In contrast, this section will only use maxima in order to determine a quantity that closely resembles the PSF that is present in an image. We call this quantity as reference PSF (RPSF) (Chong & Tanaka, 2010a). Its purpose is to yield a rough idea regarding the blurring function by exploiting the effect of blurs on the maxima values and locations. Additional uses of RPSF will be discussed in the next section. In comparison

between the counts is used to detect the blur by comparing it with another threshold.

Images IEXA1 IEXA2

Table 1. Comparison of blur detection accuracy.

Table 2. Comparison of blur classification accuracy.

became negligible and the overall accuracy improved.

**3. RPSF: Characteristics and extraction methods**

next section, maxima and RPSF will be presented.

**2.3.2 Data and results**

all the features.

Images IEXA1 IEXA2 BDHWT

1. Gaussian or atmospheric turbulence blur:

$$h(x, y) = K \exp\left(-\frac{x^2 + y^2}{2\sigma^2}\right) \tag{9}$$

where *K* is a normalizing constant and *σ* is the variance. This is generally used to model a variety of imaging devices as well as long-term atmospheric exposure.

2. Horizontal motion (HM) blur:

$$h(b,d) = \begin{cases} \frac{1}{L}, & b = 0, |d| \le \frac{L-1}{2} \\ 0, & \text{otherwise} \end{cases} \tag{10}$$

where *L* is the length of motion. The variables *b* and *d* represent the PSF coordinates. This models the effect on the acquired image when the camera or object is horizontally moving faster than the camera's exposure period.

3. uniform out-of-focus (OOF) blur:

$$h(b,d) = \begin{cases} \frac{1}{\pi R^2}, \sqrt{b^2 + d^2} \le R^2\\ 0, \text{ otherwise} \end{cases} \tag{11}$$

where *R* is the blur radius. This blur is observable as defocus in images and is caused by the finite size of camera aperture.

The Gaussian blurs have standard deviations of *σ* ∈ {1, 1.33, 1.66, 2, 2.33, 2.66, 3} with a spread determined by 6*σ*. The uniform HM are set to have lengths of *L* ∈ {3, 5, 7, 9, 11, 13}. The radii for the OOF blur are *R* ∈ {1, 2, . . . , 6}.

The synthetically degraded images are subjected to extrema extraction and histogram creation. The required features can then be computed from the histograms. The training set consisted of 2,660 images that is composed of 133 images with their corresponding 20 variations. The testing set also has 20 variations for 167 images for a total of 3,340 images. In the classification process, the Euclidean distances from the test image to all of the training images are first calculated. The training image with the minimum distance from the test image is considered as the nearest neighbour. The unknown object's class is then identified by the class of this neighbour.

There are two IEXA experiments being compared. The first (IEXA1) used the complete 40 features while IEXA2 had only 20. The reduction in number is achieved by applying forward selection (Theodoridis & Koutroumbas, 2006). Since this is computationally expensive and time consuming, a naive method is first applied in order to determine the order of features to be added. The resulting required features are:


It is important to note that features from dn\_c and pn\_c are not included so these need not be determined.


Table 1. Comparison of blur detection accuracy.


Table 2. Comparison of blur classification accuracy.

## **2.3.2 Data and results**

6 Will-be-set-by-IN-TECH

where *K* is a normalizing constant and *σ* is the variance. This is generally used to model a

where *L* is the length of motion. The variables *b* and *d* represent the PSF coordinates. This models the effect on the acquired image when the camera or object is horizontally moving

where *R* is the blur radius. This blur is observable as defocus in images and is caused by

The Gaussian blurs have standard deviations of *σ* ∈ {1, 1.33, 1.66, 2, 2.33, 2.66, 3} with a spread determined by 6*σ*. The uniform HM are set to have lengths of *L* ∈ {3, 5, 7, 9, 11, 13}. The radii

The synthetically degraded images are subjected to extrema extraction and histogram creation. The required features can then be computed from the histograms. The training set consisted of 2,660 images that is composed of 133 images with their corresponding 20 variations. The testing set also has 20 variations for 167 images for a total of 3,340 images. In the classification process, the Euclidean distances from the test image to all of the training images are first calculated. The training image with the minimum distance from the test image is considered as the nearest neighbour. The unknown object's class is then identified by the

There are two IEXA experiments being compared. The first (IEXA1) used the complete 40 features while IEXA2 had only 20. The reduction in number is achieved by applying forward selection (Theodoridis & Koutroumbas, 2006). Since this is computationally expensive and time consuming, a naive method is first applied in order to determine the order of features to

It is important to note that features from dn\_c and pn\_c are not included so these need not be

<sup>−</sup> *<sup>x</sup>*<sup>2</sup> <sup>+</sup> *<sup>y</sup>*<sup>2</sup> 2*σ*<sup>2</sup>

*<sup>L</sup>* , *<sup>b</sup>* <sup>=</sup> 0, <sup>|</sup>*d*| ≤ *<sup>L</sup>*−<sup>1</sup>

<sup>√</sup>*b*<sup>2</sup> <sup>+</sup> *<sup>d</sup>*<sup>2</sup> <sup>≤</sup> *<sup>R</sup>*<sup>2</sup>

2

0, otherwise (10)

0, *otherwise* (11)

(9)

*h*(*x*, *y*) = *K* exp

1

 1 *<sup>π</sup>R*<sup>2</sup> ,

variety of imaging devices as well as long-term atmospheric exposure.

*h*(*b*, *d*) =

*h*(*b*, *d*) =

1. Gaussian or atmospheric turbulence blur:

faster than the camera's exposure period.

2. Horizontal motion (HM) blur:

3. uniform out-of-focus (OOF) blur:

the finite size of camera aperture.

for the OOF blur are *R* ∈ {1, 2, . . . , 6}.

be added. The resulting required features are:

1. *dS* : *S* ∈{dx\_c, dn\_r, dx\_r, pn\_r, px\_r} 2. *cS* : *S* ∈{dx\_c, px\_c, dn\_r, dx\_r, pn\_r, px\_r}

*hS* : *S* ∈{dx\_c, dn\_r, dx\_r} 4. *hwS* : *S* ∈{dx\_c, dn\_r, dx\_r}

*<sup>S</sup>* : *S* ∈{dn\_r, dx\_r, pn\_r}

class of this neighbour.

3. ˆ

5. *σ*<sup>2</sup>

determined.

The blur detection performance in this section is compared with the work in (Tong et al., 2004), which detects blur with Haar wavelet transform (BDHWT). This involves the creation of an edge map based on a three-level decomposition of an image. The different edge types are then identified and counted by using a threshold and comparative conditions. The resulting ratio between the counts is used to detect the blur by comparing it with another threshold.

The accuracy of the detection algorithms are shown in Table 1. The NA in the table indicates not applicable since the method in (Tong et al., 2004) is only for unblurred, motion, and OOF blurs. It can be observed that accuracies are consistently higher when extrema analysis is used. Furthermore, the result after feature selection yielded a better performance than using all the features.

The classification accuracies for IEXA1 and IEXA2 are in Table 2. Only the accuracy for the unblurred images did not change after feature selection. Motion classification performance slightly decreased however, Gaussian and OOF have higher values. As a result, the decrease became negligible and the overall accuracy improved.

This section shows the effectiveness of using extrema features for blur detection and identification. Furthermore, the accuracy is further improved by feature selection. In the next section, maxima and RPSF will be presented.

## **3. RPSF: Characteristics and extraction methods**

The previous section shows that maxima and minima can be exploited to determine and identify the blur in an image. In contrast, this section will only use maxima in order to determine a quantity that closely resembles the PSF that is present in an image. We call this quantity as reference PSF (RPSF) (Chong & Tanaka, 2010a). Its purpose is to yield a rough idea regarding the blurring function by exploiting the effect of blurs on the maxima values and locations. Additional uses of RPSF will be discussed in the next section. In comparison

(a) Blurred with *L* = 5 and *θ* = 0◦ (b) Blurred with *L* = 5 and *θ* = 45◦

Harnessing the Potentials of Image Extrema for Blind Restoration 209

(c) Horizontal scanning of 4(a) (d) Horizontal scanning of 4(b)

(e) Vertical scanning of 4(a) (f) Vertical scanning of 4(b)

Fig. 4. Blurry images and their maxima locations.

with the previous section, this does not require a classifier or image features. It is extracted from the given image assuming only a PSF size.

#### **3.1 Motion blurs and maxima**

The presence of blur has the effect of flattening the pixel values. As a result, the locations of the maxima values are influenced by the blurring function. In (Chong & Tanaka, 2010a; 2011b) the extraction of RPSF has been tested on motion and OOF blurred images. However, the most commonly encountered type of blur is motion. Thus, more focus will be given to this type. In this case we will consider the general model of the motion blur given by:

$$h(b,d;L,\theta) = \begin{cases} \frac{1}{L'}\sqrt{b^2 + d^2} \le \frac{L}{2} \text{ and } \frac{b}{d} = -\tan\theta\\ 0, \text{ otherwise} \end{cases} \tag{12}$$

where the parameter *θ* represents motion angle in degrees.

Figure 4 shows images blurred by motion in different directions. Each degraded image consists of two maxima images based on the direction of scanning. These are shown as binary images in Figs. 4(c)-4(f) where the white pixels indicate the maxima. It can be observed that the locations are influenced by the direction of motion that is present. As a result, the maxima locations tend to be arranged in the direction of motion. Considering a small window in the image where the maxima are present, it is obvious that this will give us an idea regarding the direction of motion. This is indicated by the larger pixel values located at the direction of motion. To detect the direction Hough transform has been utilized in (Chong & Tanaka, 2010b; 2011a). In image processing, this is commonly used to detect lines and circles in binary images. For a straight line, all of its points will intersect in the parameter space [(Gonzales & Woods, 2008)]. The parameters (*ζ* and *θ*) are specified from the representation of a line in the PSF space domain:

$$d\cos\theta + d\sin\theta = \mathbb{Z}\tag{13}$$

where parameter *θ* can be interpreted as the direction of motion. Since the PSFs are discrete and have sizes smaller than the image, the parameter space can be easily subdivided. For practical purposes, we set *θ* = {0◦, 45◦, 90◦, 135◦}. The accumulator for a certain combination of *ζ* and *θ* is incremented based on the point located at (*b*, *d*). The detection of the motion direction is accomplished by selecting the accumulator with the maximum number of points.

#### **3.2 RPSF extraction**

The RPSF can be extracted from motion blurred images by the following steps:


$$
\omega w = \omega \sum\_{z=1}^{Z} w\_z. \tag{14}
$$

8 Will-be-set-by-IN-TECH

with the previous section, this does not require a classifier or image features. It is extracted

The presence of blur has the effect of flattening the pixel values. As a result, the locations of the maxima values are influenced by the blurring function. In (Chong & Tanaka, 2010a; 2011b) the extraction of RPSF has been tested on motion and OOF blurred images. However, the most commonly encountered type of blur is motion. Thus, more focus will be given to this

<sup>√</sup>*b*<sup>2</sup> <sup>+</sup> *<sup>d</sup>*<sup>2</sup> <sup>≤</sup> *<sup>L</sup>*

Figure 4 shows images blurred by motion in different directions. Each degraded image consists of two maxima images based on the direction of scanning. These are shown as binary images in Figs. 4(c)-4(f) where the white pixels indicate the maxima. It can be observed that the locations are influenced by the direction of motion that is present. As a result, the maxima locations tend to be arranged in the direction of motion. Considering a small window in the image where the maxima are present, it is obvious that this will give us an idea regarding the direction of motion. This is indicated by the larger pixel values located at the direction of motion. To detect the direction Hough transform has been utilized in (Chong & Tanaka, 2010b; 2011a). In image processing, this is commonly used to detect lines and circles in binary images. For a straight line, all of its points will intersect in the parameter space [(Gonzales & Woods, 2008)]. The parameters (*ζ* and *θ*) are specified from the representation of a line in the

where parameter *θ* can be interpreted as the direction of motion. Since the PSFs are discrete and have sizes smaller than the image, the parameter space can be easily subdivided. For practical purposes, we set *θ* = {0◦, 45◦, 90◦, 135◦}. The accumulator for a certain combination of *ζ* and *θ* is incremented based on the point located at (*b*, *d*). The detection of the motion direction is accomplished by selecting the accumulator with the maximum number of points.

1. Scan the green channel of the image for local maxima locations in the horizontal and

2. Determine the number of locations for each scanning direction and let *Z* be the total count. 3. If *z* is the index of the maxima locations and assuming a blur support size of *s* × *s*, create

*w* = *ω*

*Z* ∑ *z*=1

The RPSF can be extracted from motion blurred images by the following steps:

<sup>2</sup> and *<sup>b</sup>*

*<sup>d</sup>* = − tan *θ* 0, otherwise (12)

*b* cos *θ* + *d* sin *θ* = *ζ* (13)

*<sup>z</sup>*=<sup>1</sup> of the same size whose elements are the pixel values with the

*wz*. (14)

type. In this case we will consider the general model of the motion blur given by:

 1 *L* ,

from the given image assuming only a PSF size.

*h*(*b*, *d*; *L*, *θ*) =

where the parameter *θ* represents motion angle in degrees.

**3.1 Motion blurs and maxima**

PSF space domain:

**3.2 RPSF extraction**

vertical directions.

a set of windows {*wz*}*<sup>Z</sup>*

maxima value at the center.

4. Compute the sum of these windows by

(a) Blurred with *L* = 5 and *θ* = 0◦ (b) Blurred with *L* = 5 and *θ* = 45◦

Fig. 4. Blurry images and their maxima locations.

angle no. of Accuracy (%)

Table 3. Accuracy (%) in determining the motion direction with different PSF sizes.

◦) images 3 5 7 9 11 13 300 **100.0 100.0 100.0 100.0 100.0 100.0** 300 97.0 87.0 94.3 96.7 94.0 96.3 300 **99.7 99.7 99.7 99.7 100.0 100.0** 300 97.7 86.7 96.0 97.0 95.0 96.3

Harnessing the Potentials of Image Extrema for Blind Restoration 211

In this section, we consider the deconvolution of a degraded image. This involves the estimation of *f* and *h* given only *g*. Since it is highly ill-posed, we use the alternating minimization (AM) technique. This enables the incorporation of information about *f* and *h* so that the solution is more stable. AM is a technique that is used in the minimization of a cost function involving two variables. In each cycle, one variable is set constant while the other is being solved. The roles are then reversed and the cycle repeats until a criterion is

The estimation process involves the minimization of a cost function generally defined by:

*<sup>f</sup>*)*T*(*<sup>g</sup>* <sup>−</sup> *<sup>H</sup>*<sup>ˆ</sup> <sup>ˆ</sup>

ˆ· represents an estimated quantity. The first term is known as the fidelity term and is a basic term in most reconstruction cost functions. This was proposed in (Yang et al., 1994) wherein minimization was accomplished by projection-based blind deconvolution. The results showed good convergence properties and high flexibility for incorporation of prior knowledge. The image smoothness term is based on the concept of total variation (TV)

The work of (Chan et al., 1999) added a variable to avoid its non-differentiability. Thus, the

TV in vector-matrix form was derived in (He et al., 2009) to make the minimization process simpler. We will be employing here the TV norm in equation (20) instead of the commonly

where *ε* is an auxiliary variable, *ε* = |∇ *f* |. By following the fixed point scheme in (Chan & Wong, 1998), *ε* can be computed using the value from the previous iteration of the AM loop.

 |∇ *f* | 2

<sup>2</sup> <sup>=</sup> ||*V f* ||<sup>2</sup>


*JTV* = Ω

*J<sup>γ</sup>* = Ω 

(*f*(*x* + 1, *y*) − *f*(*x*, *y*))

For simplicity, consider the gradients of the image in the *x* direction

*f*) + *J* <sup>ˆ</sup>

*<sup>f</sup>* + *J*<sup>ˆ</sup> *h* 

*<sup>h</sup>* is the PSF characterization term. The symbol



*<sup>ε</sup>* (22)

<sup>2</sup> = *<sup>f</sup> TVTV f* (23)

(19)

(*<sup>g</sup>* <sup>−</sup> *<sup>H</sup>*<sup>ˆ</sup> <sup>ˆ</sup>

*<sup>J</sup>* <sup>=</sup> <sup>1</sup> 2 

proposed in (Rudin et al., 1992), which can be modelled by:

used *L*<sup>2</sup> norm. This can be reformulated as

∑ ∀(*x*,*y*)

*<sup>f</sup>* is the image smoothness term and *J*<sup>ˆ</sup>

(

**4. Image deconvolution**

achieved.

where *J* <sup>ˆ</sup>

**4.1 Cost function**

equation becomes

Determine the value of the constant *ω* that will normalize *w* such that:

$$\sum\_{\forall (b,d)} w(b,d) = 1 \tag{15}$$

where (*b*, *d*) ∈ {0, 1, . . . ,*s* − 1}.

5. Create a binary matrix (*o*) that will mask out unnecessary values

$$o(b,d) = \begin{cases} 1, \, w(b,d) \ge t \\ 0, \, \text{otherwise} \end{cases} \tag{16}$$

where *t* is the threshold that can be determined by the mean of the central and outermost elements of *w*.

6. Non-zero elements of *o* must satisfy the symmetry condition:

$$
\sigma(b,d) = \sigma(-b,-d). \tag{17}
$$

	- (a) Create *η* binary masks, *oη*,*<sup>θ</sup>* , that correspond with *θ* represented by the accumulator.
	- (b) For each *η*, compute:

$$a\_{\eta,\theta} = w \cdot o\_{\eta,\theta} \cdot$$

(c) The direction is selected by:

$$\theta = \arg\max\_{\theta} \{a\_{\eta,\theta}\},$$

8. Determine the matrix form of the RPSF (*r*) by

$$r(b,d) = w(b,d)o(b,d)\tag{18}$$

where the matrix must satisfy the conditions in equations (15) and (17).

#### **3.3 Experimental results**

#### **3.3.1 Experiment descriptions**

The experiment herein uses 300 unblurred images with sizes that may be 640 × 480 or 480 × 640. These are then motion blurred with *L* ∈ {3, 5, 7, 9, 11, 13} and *θ* ∈ {0◦, 45◦, 90◦, 135◦}. The detection process was also applied only to the green channels to minimize computational time.

#### **3.3.2 Data and results**

The data in Table 3 shows the accuracy of the method in detecting the direction of motion. It can be seen that horizontal and vertical motions have consistently high values for different PSF sizes. On the other hand, slanting directions (45◦ and 135◦) yielded lower values, which are fluctuating as the PSF size increases. These are attributed by the direction in scanning for the maxima. When the motion has the same direction as the scanning, the chances of preserving the blur's effects are higher.

These results show that RPSF is capable of giving us an idea of the PSF's characteristics using only the maxima of the degraded image. Although only one colour channel was processed, a high accuracy in detecting the motion direction was manifested in the experiment with various natural images. Due to these we can also use it for image deconvolution, which will be discussed in the next section.


Table 3. Accuracy (%) in determining the motion direction with different PSF sizes.

#### **4. Image deconvolution**

10 Will-be-set-by-IN-TECH

1, *<sup>w</sup>*(*b*, *<sup>d</sup>*) <sup>≥</sup> *<sup>t</sup>*

where *t* is the threshold that can be determined by the mean of the central and outermost

7. Apply Hough transform on binary matrix *<sup>o</sup>* using *<sup>θ</sup>* <sup>=</sup> {0*o*, 45*o*, 90*o*, 135*o*}. When there are

(a) Create *η* binary masks, *oη*,*<sup>θ</sup>* , that correspond with *θ* represented by the accumulator.

The experiment herein uses 300 unblurred images with sizes that may be 640 × 480 or 480 × 640. These are then motion blurred with *L* ∈ {3, 5, 7, 9, 11, 13} and *θ* ∈ {0◦, 45◦, 90◦, 135◦}. The detection process was also applied only to the green channels to minimize computational

The data in Table 3 shows the accuracy of the method in detecting the direction of motion. It can be seen that horizontal and vertical motions have consistently high values for different PSF sizes. On the other hand, slanting directions (45◦ and 135◦) yielded lower values, which are fluctuating as the PSF size increases. These are attributed by the direction in scanning for the maxima. When the motion has the same direction as the scanning, the chances of

These results show that RPSF is capable of giving us an idea of the PSF's characteristics using only the maxima of the degraded image. Although only one colour channel was processed, a high accuracy in detecting the motion direction was manifested in the experiment with various natural images. Due to these we can also use it for image deconvolution, which will

*aη*,*<sup>θ</sup>* = *w* · *oη*,*<sup>θ</sup>* .

*<sup>θ</sup>* <sup>=</sup> arg max*<sup>θ</sup>* {*aη*,*θ*}

*w*(*b*, *d*) = 1 (15)

0, otherwise (16)

*o*(*b*, *d*) = *o*(−*b*, −*d*). (17)

*r*(*b*, *d*) = *w*(*b*, *d*)*o*(*b*, *d*) (18)

Determine the value of the constant *ω* that will normalize *w* such that:

*o*(*b*, *d*) =

where the matrix must satisfy the conditions in equations (15) and (17).

5. Create a binary matrix (*o*) that will mask out unnecessary values

6. Non-zero elements of *o* must satisfy the symmetry condition:

*η* accumulators with maximum number of points:

where (*b*, *d*) ∈ {0, 1, . . . ,*s* − 1}.

elements of *w*.

(b) For each *η*, compute:

**3.3 Experimental results**

**3.3.2 Data and results**

time.

**3.3.1 Experiment descriptions**

preserving the blur's effects are higher.

be discussed in the next section.

(c) The direction is selected by:

8. Determine the matrix form of the RPSF (*r*) by

∑ ∀(*b*,*d*)

> In this section, we consider the deconvolution of a degraded image. This involves the estimation of *f* and *h* given only *g*. Since it is highly ill-posed, we use the alternating minimization (AM) technique. This enables the incorporation of information about *f* and *h* so that the solution is more stable. AM is a technique that is used in the minimization of a cost function involving two variables. In each cycle, one variable is set constant while the other is being solved. The roles are then reversed and the cycle repeats until a criterion is achieved.

#### **4.1 Cost function**

The estimation process involves the minimization of a cost function generally defined by:

$$J = \frac{1}{2}\left( (\mathbf{g} - \hat{H}\hat{f})^T (\mathbf{g} - \hat{H}\hat{f}) + J\_{\hat{f}} + J\_{\hat{h}} \right) \tag{19}$$

where *J* <sup>ˆ</sup> *<sup>f</sup>* is the image smoothness term and *J*<sup>ˆ</sup> *<sup>h</sup>* is the PSF characterization term. The symbol ˆ· represents an estimated quantity. The first term is known as the fidelity term and is a basic term in most reconstruction cost functions. This was proposed in (Yang et al., 1994) wherein minimization was accomplished by projection-based blind deconvolution. The results showed good convergence properties and high flexibility for incorporation of prior knowledge. The image smoothness term is based on the concept of total variation (TV) proposed in (Rudin et al., 1992), which can be modelled by:

$$J\_{TV} = \int\_{\Omega} |\nabla f| \,. \tag{20}$$

The work of (Chan et al., 1999) added a variable to avoid its non-differentiability. Thus, the equation becomes

$$J\_{\gamma} = \int\_{\Omega} \sqrt{|\nabla f|^2 + \gamma^2}. \tag{21}$$

TV in vector-matrix form was derived in (He et al., 2009) to make the minimization process simpler. We will be employing here the TV norm in equation (20) instead of the commonly used *L*<sup>2</sup> norm. This can be reformulated as

$$\int |\nabla f| = \int \frac{|\nabla f|^2}{\varepsilon} \tag{22}$$

where *ε* is an auxiliary variable, *ε* = |∇ *f* |. By following the fixed point scheme in (Chan & Wong, 1998), *ε* can be computed using the value from the previous iteration of the AM loop. For simplicity, consider the gradients of the image in the *x* direction

$$\sum\_{\forall(\mathbf{x},y)} \left( f(\mathbf{x}+1,y) - f(\mathbf{x},y) \right)^2 = ||Vf||\_2^2 = f^T V^T V f \tag{23}$$

terms have the same derivations. These are employed for the reason that the TV norm is known for edge preservation and robustness in the presence of noise. The use of RPSF will yield a rough idea on the blurring function using only the degraded image. As a result, the need for a training set is eliminated, the learning model is determined only once and it is

Harnessing the Potentials of Image Extrema for Blind Restoration 213

*<sup>h</sup>* is in equation 28. As a whole, this is not convex however, iterative projection into <sup>ˆ</sup>

*h* will result in convex functions. Thus, we use the AM technique to solve the two unknowns.

*H*ˆ *TH*ˆ + *λT*<sup>ˆ</sup>

*f* at (*n* + 1)-th iteration:

*<sup>n</sup> <sup>H</sup>*<sup>ˆ</sup> *<sup>n</sup>* + *<sup>λ</sup>T*<sup>ˆ</sup>

*<sup>f</sup>*(*ε*) ˆ

*fn* (*ε*) −<sup>1</sup> *H*ˆ *<sup>T</sup>*

> ˆ *h* −

−<sup>1</sup> *F*ˆ*T <sup>n</sup>*+1*g* + *βr*

*h* involve *λ*, *α*, and *β*, which are collectively called as regularization

*<sup>H</sup>TH f* <sup>−</sup> *<sup>H</sup>Tg* <sup>+</sup> *<sup>λ</sup>Tf*(*ε*)*<sup>f</sup>* <sup>=</sup> <sup>0</sup> (33)

*c* = *Aχ* (35)

*<sup>F</sup><sup>T</sup> Fh* <sup>−</sup> *<sup>F</sup>Tg* <sup>+</sup> *<sup>α</sup>Th*(*ε*)*<sup>h</sup>* <sup>+</sup> *<sup>β</sup>*(*<sup>h</sup>* <sup>−</sup> *<sup>r</sup>*) = 0, (34)

*F*ˆ*Tg* + *βr*

*<sup>h</sup>*(*ε*) + *βI*

(*ε*) + *βI*

*<sup>f</sup>* is in equation 26

*f* where ˆ

. (32)

*<sup>f</sup>* <sup>−</sup> *<sup>H</sup>*<sup>ˆ</sup> *Tg*. (29)

*<sup>n</sup> g*. (30)

*f* and

*h* is set

(31)

The cost function that will be minimized is defined in equation 19 where *J* <sup>ˆ</sup>

The first step is the partial differentiation of the cost function with respect to ˆ

 *H*ˆ *<sup>T</sup>*

*F*ˆ*<sup>T</sup> F*ˆ + *αT*<sup>ˆ</sup>

*<sup>n</sup>*+<sup>1</sup> + *αT*<sup>ˆ</sup>

*hn*

To summarize, the AM technique involves solving equations 30 and 32 alternately until

parameters. These are positive values that measure the trade off between a good fit and the regularity of the solutions. These can be accurately determined using many methods. However, these are usually computationally expensive and time consuming. To make the selection process less complicated, we will follow the concept used in (Sroubek & Flusser, 2003; You & Kaveh, 1996). The idea is based on the fact that the partial derivatives of the cost function are zero assuming that we are given the correct values of *f* and *h*. Thus equations 29

respectively. The resulting equations are overdetermined system of linear equations (OSLE).

In general, an OSLE with *E* linear equations and *N* unknowns can be expressed as:

*∂J*( ˆ *f* , ˆ *h*) *∂* ˆ *<sup>f</sup>* <sup>=</sup>

> ˆ *fn*+<sup>1</sup> =

 *F*ˆ*T <sup>n</sup>*+1*F*<sup>ˆ</sup>

convergence or when the desired number of iterations is reached.

*∂J*( ˆ *f* , ˆ *h*) *∂*ˆ *<sup>h</sup>* <sup>=</sup>

*f* and ˆ

independent from other estimated values.

Equating this with zero will yield ˆ

where *I* is an identity matrix. Thus,

**4.2 Regularization parameters**

The computations of ˆ

and 31 will become:

and

ˆ *hn*+<sup>1</sup> =

Reversing the roles will yield:

and *J*<sup>ˆ</sup>

as constant:

ˆ

where

$$V = \begin{bmatrix} 1 \ -1 & 0 & \cdots & 0 \\ 0 & 1 & -1 & \cdots & \vdots \\ \vdots & \vdots & \vdots & \ddots & -1 \\ 0 & 0 & 0 & \vdots & 1 \end{bmatrix}.$$

Applying the concept of equation (22) to equation (23)

$$\sum\_{\forall(\mathbf{x},\mathbf{y})} \left( f(\mathbf{x}+1,\mathbf{y}) - f(\mathbf{x},\mathbf{y}) \right)^2 = \sum\_{\forall(\mathbf{x},\mathbf{y})} \frac{1}{\varepsilon(\overline{\mathbf{x}})} \left( f(\mathbf{x}+1,\mathbf{y}) - f(\mathbf{x},\mathbf{y}) \right)^2$$

$$= f^T V^T W V f \tag{24}$$

where

$$W = \text{diag}\left\{\frac{1}{\varepsilon(\stackrel{\rightarrow}{x})}\right\}.$$

For practical purposes, we also implement the following conditions:

$$\frac{1}{\varepsilon(\stackrel{\rightarrow}{\mathfrak{x}})} = \begin{cases} \frac{1}{\varepsilon(\stackrel{\rightarrow}{\mathfrak{x}})} & , \varepsilon(\stackrel{\rightarrow}{\mathfrak{x}}) \neq 0\\ 0 & , \text{otherwise} \end{cases}.$$

From equation (24), let *Tx*(*ε*) = *VTWV*. The same derivation will follow for the *y* direction. The result of using the two directions will be *T*(*ε*) = *Tx*(*ε*) + *Ty*(*ε*) and equation (24) can be rewritten as

$$\sum\_{\forall(\mathbf{x},y)} \left( f(\mathbf{x}+1,y) - f(\mathbf{x},y) \right)^2 = f^T T(\varepsilon) f. \tag{25}$$

Thus, the image smoothness term will be

$$J(\not f) = \lambda f^T T\_{\not f}(\varepsilon) f \tag{26}$$

where *λ* is the image regularization parameter. On the other hand, the PSF characterization term in (He et al., 2008; 2009) uses a reinforcement blur estimation (RBE) framework. This assumes that real-life blurs satisfy a certain degree of parametric structure. Mathematically, this can be modelled by

$$J\_{\hat{h}\\_RBE} = \left(\hat{h} - t\right)^T \left(\hat{h} - t\right) \tag{27}$$

where *t* is the best-fit parametric PSF selected from a training set. This is highly dependent on the current estimate as well as the contents of the training set. As a result, failure of reinforced learning is possible when the estimate changes as the iteration progresses. In some cases, learning can also be erroneous when the selected model does not match the actual PSF. Furthermore, the learning set must be exhaustive to ensure an accurate model selection. To overcome these problems, RPSF can be used in its place. Thus,

$$J(\hat{h}) = a\hbar^T T\_{\hat{h}}(\varepsilon)\hat{h} + \beta(\hat{h} - r)^T(\hat{h} - r) \tag{28}$$

where *α* and *β* are the regularization parameters and *r* is RPSF. The first term is the PSF smoothness term while the second is PSF learning term. The image and PSF smoothness terms have the same derivations. These are employed for the reason that the TV norm is known for edge preservation and robustness in the presence of noise. The use of RPSF will yield a rough idea on the blurring function using only the degraded image. As a result, the need for a training set is eliminated, the learning model is determined only once and it is independent from other estimated values.

The cost function that will be minimized is defined in equation 19 where *J* <sup>ˆ</sup> *<sup>f</sup>* is in equation 26 and *J*<sup>ˆ</sup> *<sup>h</sup>* is in equation 28. As a whole, this is not convex however, iterative projection into <sup>ˆ</sup> *f* and ˆ *h* will result in convex functions. Thus, we use the AM technique to solve the two unknowns. The first step is the partial differentiation of the cost function with respect to ˆ *f* where ˆ *h* is set as constant:

$$\frac{\partial J(\vec{f}, \hat{h})}{\partial \hat{f}} = \left(\hat{H}^T \hat{H} + \lambda T\_{\hat{f}}(\varepsilon)\right) \hat{f} - \hat{H}^T \hat{g}. \tag{29}$$

Equating this with zero will yield ˆ *f* at (*n* + 1)-th iteration:

$$\boldsymbol{f}\_{n+1} = \left(\hat{\boldsymbol{H}}\_{n}^{T}\hat{\boldsymbol{H}}\_{n} + \lambda\boldsymbol{T}\_{\boldsymbol{f}\_{n}}(\boldsymbol{\varepsilon})\right)^{-1}\hat{\boldsymbol{H}}\_{n}^{T}\boldsymbol{g}.\tag{30}$$

Reversing the roles will yield:

12 Will-be-set-by-IN-TECH

1 −1 0 ··· 0 0 1 <sup>−</sup><sup>1</sup> ··· .

00 0 .

<sup>2</sup> <sup>=</sup> ∑ ∀(*x*,*y*)

> � 1 *ε*( *x* )

From equation (24), let *Tx*(*ε*) = *VTWV*. The same derivation will follow for the *y* direction. The result of using the two directions will be *T*(*ε*) = *Tx*(*ε*) + *Ty*(*ε*) and equation (24) can be

> *f TT*<sup>ˆ</sup> *<sup>f</sup>*(*ε*) <sup>ˆ</sup>

> > �*<sup>T</sup>* � ˆ *h* − *t* �

where *λ* is the image regularization parameter. On the other hand, the PSF characterization term in (He et al., 2008; 2009) uses a reinforcement blur estimation (RBE) framework. This assumes that real-life blurs satisfy a certain degree of parametric structure. Mathematically,

where *t* is the best-fit parametric PSF selected from a training set. This is highly dependent on the current estimate as well as the contents of the training set. As a result, failure of reinforced learning is possible when the estimate changes as the iteration progresses. In some cases, learning can also be erroneous when the selected model does not match the actual PSF. Furthermore, the learning set must be exhaustive to ensure an accurate model selection. To

*h* + *β*(ˆ

where *α* and *β* are the regularization parameters and *r* is RPSF. The first term is the PSF smoothness term while the second is PSF learning term. The image and PSF smoothness

*<sup>h</sup>* <sup>−</sup> *<sup>r</sup>*)*T*(<sup>ˆ</sup>

� ˆ *h* − *t*

(*f*(*x* + 1, *y*) − *f*(*x*, *y*))

*J*( ˆ *f*) = *λ* ˆ

*hTT*<sup>ˆ</sup> *h*(*ε*)<sup>ˆ</sup>

*J*ˆ *<sup>h</sup>*\_*RBE* =

overcome these problems, RPSF can be used in its place. Thus,

*J*(ˆ *h*) = *α*ˆ . . 1

1 *ε*( *x* )

> � .

. . ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

.

(*f*(*x* + 1, *y*) − *f*(*x*, *y*))

= *f TVTWVf* (24)

<sup>2</sup> = *f TT*(*ε*)*f* . (25)

*f* (26)

*h* − *r*) (28)

(27)

2

*V* =

Applying the concept of equation (22) to equation (23)

(*f*(*x* + 1, *y*) − *f*(*x*, *y*))

For practical purposes, we also implement the following conditions:

∑ ∀(*x*,*y*)

Thus, the image smoothness term will be

1 *ε*( *x* ) = � 1 *ε*( *<sup>x</sup>* ) ,*ε*( *x* ) �= 0 0 , otherwise .

∑ ∀(*x*,*y*) ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣

. . . . . . . . . ... <sup>−</sup><sup>1</sup>

*W* = diag

where

where

rewritten as

this can be modelled by

$$\frac{\partial I(\boldsymbol{f}, \boldsymbol{\hat{h}})}{\partial \boldsymbol{\hat{h}}} = \left(\boldsymbol{\hat{F}^{T}}\boldsymbol{\hat{F}} + \boldsymbol{\alpha}\boldsymbol{T}\_{\hat{h}}(\boldsymbol{e}) + \beta\boldsymbol{I}\right)\boldsymbol{\hat{h}} - \left(\boldsymbol{\hat{F}^{T}}\boldsymbol{g} + \beta\boldsymbol{r}\right) \tag{31}$$

where *I* is an identity matrix. Thus,

$$
\hat{\boldsymbol{h}}\_{n+1} = \left(\hat{\mathbf{f}}\_{n+1}^T \hat{\mathbf{f}}\_{n+1} + \boldsymbol{\pi} \boldsymbol{T}\_{\hat{\mathbf{h}}\_n}(\boldsymbol{\varepsilon}) + \beta \boldsymbol{I}\right)^{-1} \left(\hat{\mathbf{f}}\_{n+1}^T \mathbf{g} + \beta \boldsymbol{r}\right). \tag{32}
$$

To summarize, the AM technique involves solving equations 30 and 32 alternately until convergence or when the desired number of iterations is reached.

#### **4.2 Regularization parameters**

The computations of ˆ *f* and ˆ *h* involve *λ*, *α*, and *β*, which are collectively called as regularization parameters. These are positive values that measure the trade off between a good fit and the regularity of the solutions. These can be accurately determined using many methods. However, these are usually computationally expensive and time consuming. To make the selection process less complicated, we will follow the concept used in (Sroubek & Flusser, 2003; You & Kaveh, 1996). The idea is based on the fact that the partial derivatives of the cost function are zero assuming that we are given the correct values of *f* and *h*. Thus equations 29 and 31 will become:

$$H^T H f - H^T g + \lambda T\_f(\varepsilon) f = 0 \tag{33}$$

and

$$\boldsymbol{F}^T \boldsymbol{F} \boldsymbol{h} - \boldsymbol{F}^T \boldsymbol{g} + \mathfrak{a} T\_h(\boldsymbol{\varepsilon}) \boldsymbol{h} + \boldsymbol{\beta}(\boldsymbol{h} - \boldsymbol{r}) = \boldsymbol{0},\tag{34}$$

respectively. The resulting equations are overdetermined system of linear equations (OSLE). In general, an OSLE with *E* linear equations and *N* unknowns can be expressed as:

$$
\mathfrak{c} = A \mathfrak{x} \tag{35}
$$

Motion Regularization Parameters Direction *λ α β*

Harnessing the Potentials of Image Extrema for Blind Restoration 215

◦ 0.47 3.35 <sup>×</sup> 103 9.06 <sup>×</sup> <sup>10</sup><sup>4</sup> ◦ 1.21 2.59 <sup>×</sup> 104 7.26 <sup>×</sup> <sup>10</sup><sup>5</sup> ◦ 1.13 1.09 <sup>×</sup> 104 2.13 <sup>×</sup> <sup>10</sup><sup>5</sup> ◦ 1.05 1.85 <sup>×</sup> 104 4.34 <sup>×</sup> <sup>10</sup><sup>5</sup>

◦ 0.79 1.06 <sup>×</sup> 104 2.45 <sup>×</sup> <sup>10</sup><sup>5</sup> ◦ 1.83 3.25 <sup>×</sup> 103 1.64 <sup>×</sup> <sup>10</sup><sup>6</sup> ◦ 1.16 1.69 <sup>×</sup> 104 3.68 <sup>×</sup> <sup>10</sup><sup>5</sup> ◦ 2.03 2.62 <sup>×</sup> 103 1.77 <sup>×</sup> <sup>10</sup><sup>6</sup>

◦ 1.55 1.61 <sup>×</sup> 104 6.56 <sup>×</sup> <sup>10</sup><sup>5</sup> ◦ 1.68 3.63 <sup>×</sup> 103 1.79 <sup>×</sup> <sup>10</sup><sup>6</sup> ◦ 2.44 1.91 <sup>×</sup> 103 3.58 <sup>×</sup> <sup>10</sup><sup>6</sup> ◦ 1.84 1.84 <sup>×</sup> 103 3.08 <sup>×</sup> <sup>10</sup><sup>6</sup>

(MSE). To demonstrate the effectiveness of using the cost function in equation 26, three techniques are compared. These have cost functions wherein the image term uses TV while the PSF term is varied. The first technique uses TV on PSF as proposed in (Chan & Wong, 1998). The second uses reinforced blur estimation (RBE) as in (He et al., 2009) with a training set containing the same number of parameters as previously mentioned. The third,

The PSNRs of the estimated images with various motion directions and PSF sizes are shown in Table 5. It can be observed that MXB values are mostly higher in contrast with the other methods. Furthermore, the horizontal and vertical directions tend to have higher PSNRs than the diagonal motions. This is attributed by the fact that the gradients used in the TV prior are also in the same directions. The PSF estimation errors for the three methods can be compared in Table 6. The lower values demonstrate the method's effectiveness in estimating the PSF in different conditions. Notice that when the error is low in Table 6, a high PSNR in Table 5 can be observed. A sample image reconstructed by MXB is shown in Fig. 5. It shows the method's

This section shows the effects of integrating the RPSF during image deconvolution. The PSF smoothness and learning terms resulted to a lesser PSF estimation error with a larger PSNR value. Furthermore, we can see the importance of monitoring the image quality. This is very useful in selecting the regularization parameters as well as the estimated image. In the next section, we consider the case wherein the unblurred image is unknown making it impossible

maxima-based deconvolution (MXB), uses the cost function in equation 28.

(*s* = 3)

(*s* = 5)

(*s* = 7)

Table 4. Selected regularization parameters.

capability in recovering the details of the image.

for the computation of the PSNR.

**4.3.2 Data and results**

where *<sup>c</sup>* is a vector of given values with *<sup>c</sup>* ∈ �*E*×<sup>1</sup> , *<sup>A</sup>* is a matrix of coefficients with *<sup>A</sup>* ∈ �*E*×*N*, and *<sup>χ</sup>* is a vector of unknown variables with *<sup>χ</sup>* ∈ �*N*×1. Using minimum sum of squared error approximation (Cadzow, 2002), the solution can be determined by:

$$\chi = (A^T A)^{-1} A^T c. \tag{36}$$

Based on equation 33, *λ* can then be computed by:

$$\lambda = (A\_f^T A\_f)^{-1} A\_f^T \mathbf{c}\_f \tag{37}$$

where *Af* <sup>=</sup> <sup>−</sup>*Tf*(*ε*)*<sup>f</sup>* and *<sup>c</sup> <sup>f</sup>* <sup>=</sup> *<sup>H</sup>TH f* <sup>−</sup> *<sup>H</sup>Tg*. Similarly, *<sup>α</sup>* and *<sup>β</sup>* can be solved from equation 34:

$$\mathbb{E}\left[\mathfrak{a}\,\,\mathfrak{B}\right]^{T} = (A\_{\hbar}^{T}A\_{\hbar})^{-1}A\_{\hbar}^{T}\mathfrak{c}\_{\hbar} \tag{38}$$

where *Ah* <sup>=</sup> <sup>−</sup>[*Th*(*ε*)*<sup>h</sup>* (*<sup>h</sup>* <sup>−</sup> *<sup>r</sup>*)] and *ch* <sup>=</sup> *<sup>F</sup><sup>T</sup> Fh* <sup>−</sup> *<sup>F</sup>Tg*.

In practical applications, the correct values of *f* and *h* are unknown. The estimated values, ˆ *f* and ˆ *h*, may be utilized in lieu of the unknown correct values then equations 37 and 38 can be used to compute approximate values of the regularization parameters. In this case, parameter tuning is done iteratively with a stopping criterion that may be based on the maximum PSNR; a fixed iteration count; or the minimum cost function, among others.

#### **4.3 Experimental results**

#### **4.3.1 Experiment descriptions**

The experiment in this section will investigate the effects of integrating the RPSF during the deconvolution of synthetically blurred images. An unblurred image shown in Fig. 5(c) has a size of 60 × 60 pixels. This is synthetically blurred with a PSF defined by equation 10 with the following parameters: *L* ∈ {3, 5, 7} and *θ* ∈ {0◦, 45◦, 90◦, 135◦}. The unblurred image and PSF are known thus, the results can be easily monitored and verified. In this way, a reference can be established for cases when both the unblurred image and PSF are unknown.

The regularization parameters are computed for 50 iterations. The selection is based on the maximum peak signal-to-noise ratio (PSNR) in dB, which can be computed as follows:

$$\text{PSNR}\_{\text{dB}} = 10 \log\_{10} \frac{\left(2^b - 1\right)^2}{\text{MSE}} \tag{39}$$

where MSE is defined by

$$\text{MSE} = \frac{1}{XY} \sum\_{\forall(\mathbf{x}, \mathbf{y})} \left( f(\mathbf{x}, \mathbf{y}) - \hat{f}(\mathbf{x}, \mathbf{y}) \right)^2 \tag{40}$$

and *b* is the number of bits used to represent a pixel value. The selected parameters are shown in Table 4. Using these values, ˆ *f* and ˆ *h* are computed for 20 iterations. The initial estimated image is equated to the given degraded image while the PSF is composed of positive real random numbers.

The quality of the estimated image is quantitatively evaluated by computing its PSNR. On the other hand, the estimated PSF is compared with the actual PSF by mean squared error 14 Will-be-set-by-IN-TECH

where *<sup>c</sup>* is a vector of given values with *<sup>c</sup>* ∈ �*E*×<sup>1</sup> , *<sup>A</sup>* is a matrix of coefficients with *<sup>A</sup>* ∈ �*E*×*N*, and *<sup>χ</sup>* is a vector of unknown variables with *<sup>χ</sup>* ∈ �*N*×1. Using minimum sum of squared error

*<sup>f</sup> Af*)−1*A<sup>T</sup>*

*<sup>h</sup> Ah*)−1*A<sup>T</sup>*

where *Af* <sup>=</sup> <sup>−</sup>*Tf*(*ε*)*<sup>f</sup>* and *<sup>c</sup> <sup>f</sup>* <sup>=</sup> *<sup>H</sup>TH f* <sup>−</sup> *<sup>H</sup>Tg*. Similarly, *<sup>α</sup>* and *<sup>β</sup>* can be solved from equation

In practical applications, the correct values of *f* and *h* are unknown. The estimated values, ˆ

The experiment in this section will investigate the effects of integrating the RPSF during the deconvolution of synthetically blurred images. An unblurred image shown in Fig. 5(c) has a size of 60 × 60 pixels. This is synthetically blurred with a PSF defined by equation 10 with the following parameters: *L* ∈ {3, 5, 7} and *θ* ∈ {0◦, 45◦, 90◦, 135◦}. The unblurred image and PSF are known thus, the results can be easily monitored and verified. In this way, a reference

The regularization parameters are computed for 50 iterations. The selection is based on the maximum peak signal-to-noise ratio (PSNR) in dB, which can be computed as follows:

and *b* is the number of bits used to represent a pixel value. The selected parameters are shown

image is equated to the given degraded image while the PSF is composed of positive real

The quality of the estimated image is quantitatively evaluated by computing its PSNR. On the other hand, the estimated PSF is compared with the actual PSF by mean squared error

 <sup>2</sup>*<sup>b</sup>* <sup>−</sup> <sup>1</sup> 2

*<sup>f</sup>*(*x*, *<sup>y</sup>*) <sup>−</sup> <sup>ˆ</sup>

*f*(*x*, *y*) 2

*h* are computed for 20 iterations. The initial estimated

can be established for cases when both the unblurred image and PSF are unknown.

PSNRdB = 10 log10

*XY* ∑ ∀(*x*,*y*)

MSE <sup>=</sup> <sup>1</sup>

*f* and ˆ

*h*, may be utilized in lieu of the unknown correct values then equations 37 and 38 can be used to compute approximate values of the regularization parameters. In this case, parameter tuning is done iteratively with a stopping criterion that may be based on the maximum PSNR;

*<sup>T</sup>* = (*A<sup>T</sup>*

*λ* = (*A<sup>T</sup>*

[ *α β* ]

a fixed iteration count; or the minimum cost function, among others.

*χ* = (*A<sup>T</sup> A*)−1*ATc*. (36)

*<sup>f</sup> c <sup>f</sup>* (37)

*<sup>h</sup> ch* (38)

MSE (39)

*f*

(40)

approximation (Cadzow, 2002), the solution can be determined by:

Based on equation 33, *λ* can then be computed by:

where *Ah* <sup>=</sup> <sup>−</sup>[*Th*(*ε*)*<sup>h</sup>* (*<sup>h</sup>* <sup>−</sup> *<sup>r</sup>*)] and *ch* <sup>=</sup> *<sup>F</sup><sup>T</sup> Fh* <sup>−</sup> *<sup>F</sup>Tg*.

34:

and ˆ

**4.3 Experimental results**

where MSE is defined by

random numbers.

in Table 4. Using these values, ˆ

**4.3.1 Experiment descriptions**


Table 4. Selected regularization parameters.

(MSE). To demonstrate the effectiveness of using the cost function in equation 26, three techniques are compared. These have cost functions wherein the image term uses TV while the PSF term is varied. The first technique uses TV on PSF as proposed in (Chan & Wong, 1998). The second uses reinforced blur estimation (RBE) as in (He et al., 2009) with a training set containing the same number of parameters as previously mentioned. The third, maxima-based deconvolution (MXB), uses the cost function in equation 28.

## **4.3.2 Data and results**

The PSNRs of the estimated images with various motion directions and PSF sizes are shown in Table 5. It can be observed that MXB values are mostly higher in contrast with the other methods. Furthermore, the horizontal and vertical directions tend to have higher PSNRs than the diagonal motions. This is attributed by the fact that the gradients used in the TV prior are also in the same directions. The PSF estimation errors for the three methods can be compared in Table 6. The lower values demonstrate the method's effectiveness in estimating the PSF in different conditions. Notice that when the error is low in Table 6, a high PSNR in Table 5 can be observed. A sample image reconstructed by MXB is shown in Fig. 5. It shows the method's capability in recovering the details of the image.

This section shows the effects of integrating the RPSF during image deconvolution. The PSF smoothness and learning terms resulted to a lesser PSF estimation error with a larger PSNR value. Furthermore, we can see the importance of monitoring the image quality. This is very useful in selecting the regularization parameters as well as the estimated image. In the next section, we consider the case wherein the unblurred image is unknown making it impossible for the computation of the PSNR.

(a) degraded (b) estimated (c) unblurred

Harnessing the Potentials of Image Extrema for Blind Restoration 217

As mentioned in the previous section, image reconstruction is highly ill-posed. This results to the solution being solved through an iterative method wherein several images and PSFs will be produced. In most cases, the image quality is given more priority. Comparison between them can be easily achieved if the unblurred image is known or given. However, this is not the case in practical applications. It is, therefore, crucial to establish a method of evaluating

The most common approach of evaluating an image is through a measure based on the edges. An image is composed of different types of edges whose number is affected by the type of blur that is present. Haar Wavelet Transforms (HWT) can discriminate these edges and can be used for blur extent measurement as proposed in (Tong et al., 2004). Another technique is to employ edge active measure on 8 × 8 blocks of the wavelet coefficients as proposed in (Xin et al., 2003). A downside to these is that it requires a transformation with at least three levels of decompositions. Thus, it greatly increases the computational cost when considering an iterative reconstruction method. A technique that does not require a transform has been proposed in (Marziliano et al., 2002; 2004) where the blur metric is computed using the edge points and their local extrema. The metric's accuracy is dependent on the edges that can be detected. However, when blur increases edge points will decrease and in turn result to

In (Li et al., 2005) selection of the best deblurred image with Wiener filter is achieved by kurtosis minimization. It has been shown that the results are similar to those of PSNR maximization. However, it can be empirically shown that this is not consistent in natural images (Chong & Tanaka, 2011c). A different objective criterion using variance and kurtosis

The variance and kurtosis will be considered for the characterization of images. The variance

*<sup>f</sup>*(*x*, *<sup>y</sup>*) <sup>−</sup> ¯ *f*

<sup>2</sup> (41)

*<sup>v</sup>* <sup>=</sup> <sup>1</sup>

*<sup>N</sup>* ∑ ∀(*x*,*y*)

Fig. 5. An image degraded with *L* = 5 and *θ* = 90◦.

an image that is independent of its unblurred version.

**5. Quality assessment**

inaccurate values.

will be discussed next.

can be computed by

**5.1 Maxima and image quality**


Table 5. Comparison of the image PSNR (dB)for different methods.



(a) degraded (b) estimated (c) unblurred

Fig. 5. An image degraded with *L* = 5 and *θ* = 90◦.

## **5. Quality assessment**

16 Will-be-set-by-IN-TECH

Motion PSNR (dB) Direction TV RBE MXB

◦ 28.82 28.22 **31.63** ◦ 25.32 24.85 **26.65** ◦ 28.76 28.64 **29.51** ◦ 25.47 24.75 **26.76**

◦ 22.99 19.22 **26.43** ◦ 22.29 21.45 **23.65** ◦ 21.96 20.64 **25.06** ◦ 22.57 23.66 **23.68**

◦ 19.30 17.19 **23.97** ◦ 20.74 20.47 **21.81** ◦ 21.56 **23.34** 23.32 ◦ 19.60 18.15 **21.83**

(*s* = 3)

(*s* = 5)

(*s* = 7)

Motion Method

Direction TV RBE MXB

◦ 1.8 <sup>×</sup> <sup>10</sup>−<sup>2</sup> 2.0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> **<sup>9</sup>**.**<sup>9</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 7.0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> 4.1 <sup>×</sup> <sup>10</sup>−<sup>2</sup> **<sup>7</sup>**.**<sup>3</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>3</sup>** ◦ 1.9 <sup>×</sup> <sup>10</sup>−<sup>2</sup> 1.0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> **<sup>4</sup>**.**<sup>2</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>3</sup>** ◦ 7.1 <sup>×</sup> <sup>10</sup>−<sup>2</sup> 4.1 <sup>×</sup> <sup>10</sup>−<sup>2</sup> **<sup>4</sup>**.**<sup>2</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>3</sup>**

◦ 5.9 <sup>×</sup> <sup>10</sup>−<sup>3</sup> 8.4 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>4</sup>**.**<sup>4</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 5.4 <sup>×</sup> <sup>10</sup>−<sup>3</sup> 7.2 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>2</sup>**.**<sup>2</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 1.1 <sup>×</sup> <sup>10</sup>−<sup>2</sup> 1.0 <sup>×</sup> <sup>10</sup>−<sup>2</sup> **<sup>9</sup>**.**<sup>2</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 7.6 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>1</sup>**.**<sup>9</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** 2.0 <sup>×</sup> <sup>10</sup>−<sup>4</sup>

◦ 4.8 <sup>×</sup> <sup>10</sup>−<sup>3</sup> 4.2 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>1</sup>**.**<sup>5</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 2.1 <sup>×</sup> <sup>10</sup>−<sup>3</sup> 3.1 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>1</sup>**.**<sup>3</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>4</sup>** ◦ 3.9 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>3</sup>**.**<sup>9</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>5</sup> <sup>3</sup>**.**<sup>9</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>5</sup>** ◦ 3.3 <sup>×</sup> <sup>10</sup>−<sup>3</sup> 4.4 <sup>×</sup> <sup>10</sup>−<sup>3</sup> **<sup>2</sup>**.**<sup>4</sup>** <sup>×</sup> **<sup>10</sup>**−**<sup>5</sup>**

Table 5. Comparison of the image PSNR (dB)for different methods.

(*s* = 3)

(*s* = 5)

(*s* = 7)

Table 6. Comparison of the PSF estimation error.

As mentioned in the previous section, image reconstruction is highly ill-posed. This results to the solution being solved through an iterative method wherein several images and PSFs will be produced. In most cases, the image quality is given more priority. Comparison between them can be easily achieved if the unblurred image is known or given. However, this is not the case in practical applications. It is, therefore, crucial to establish a method of evaluating an image that is independent of its unblurred version.

The most common approach of evaluating an image is through a measure based on the edges. An image is composed of different types of edges whose number is affected by the type of blur that is present. Haar Wavelet Transforms (HWT) can discriminate these edges and can be used for blur extent measurement as proposed in (Tong et al., 2004). Another technique is to employ edge active measure on 8 × 8 blocks of the wavelet coefficients as proposed in (Xin et al., 2003). A downside to these is that it requires a transformation with at least three levels of decompositions. Thus, it greatly increases the computational cost when considering an iterative reconstruction method. A technique that does not require a transform has been proposed in (Marziliano et al., 2002; 2004) where the blur metric is computed using the edge points and their local extrema. The metric's accuracy is dependent on the edges that can be detected. However, when blur increases edge points will decrease and in turn result to inaccurate values.

In (Li et al., 2005) selection of the best deblurred image with Wiener filter is achieved by kurtosis minimization. It has been shown that the results are similar to those of PSNR maximization. However, it can be empirically shown that this is not consistent in natural images (Chong & Tanaka, 2011c). A different objective criterion using variance and kurtosis will be discussed next.

## **5.1 Maxima and image quality**

The variance and kurtosis will be considered for the characterization of images. The variance can be computed by

$$v = \frac{1}{N} \sum\_{\forall(\mathbf{x}, \mathbf{y})} \left( f(\mathbf{x}, \mathbf{y}) - f \right)^2 \tag{41}$$


Table 7. Conditions for experiment 1.

where ¯ *f* is the mean of the image and *N* represents the total number of pixels. On the other hand, kurtosis can be represented by

$$k = \frac{\frac{1}{N} \sum\_{\forall(\mathbf{x}, y)} \left( f(\mathbf{x}, y) - f \right)^4}{v^2} - 3. \tag{42}$$

These quantities can be used to describe the distribution of the values for a given image. To demonstrate this, the unblurred image shown in Fig. 6 is synthetically blurred with OOF. The PSF sizes are {7 × 7, 5 × 5, 3 × 3} and these are indexed by an integer number where the smallest index represents the largest size. Thus, a large index number indicates a smaller PSF size and an image with better quality. For each image, the maxima values are scanned from left to right and from top to bottom. The variance and kurtosis are computed using the pixel and maxima values. This means that every degraded image will have four quantities. Fig. 7 shows that the variance of the pixel values monotonically increases as the size becomes smaller or as the index becomes larger. This trend is also the same for the kurtosis of the maxima values. Although kurtosis of the pixel values decreased, this trend is not monotonic. Therefore, the decrease in blur can be described by the increase of pixel variance (*v*p) and maxima kurtosis (*k*m). The subscripts are used to indicate the values that are used i.e., p for pixel values and m for maxima values. These characteristics are empirically proven to be consistent in many images as will be shown by the data. Due to this, the trends of *v*p and *k*m can be used for the selection of the regularization parameters and the deblurred image. For a method with *n* iterations, *v*p and *k*m will be composed of a set with *n* values. Each set is first scaled such that the minimum value is zero and the maximum is one. The average (*q*) for each iteration is then computed as

$$q = \frac{v\_{\rm P} + k\_{\rm m}}{2}.\tag{43}$$

Fig. 6. An unblurred image.

The second experiment is the evaluation of the parameter selection and the resulting reconstructed images. For practical purposes, the image size is 60 × 60 and degradations are the same with the previous experiment but the support sizes are {3 × 3, 5 × 5}. The regularization parameters are computed for 100 iterations using the PSF sizes {3 × 3, 5 × 5, 7 × 7}. RGB images are used so that subtle changes in the images can be seen easily. As a result, the modified cost function in (Chong & Tanaka, 2010b) is used. The final PSF size and parameters are selected using the maximum value of *q*. These values are then used for the reconstruction of the degraded image computed for 50 iterations. Although this experiment

Harnessing the Potentials of Image Extrema for Blind Restoration 219

The third experiment involves the selection of the motion blur parameters, *L* and *θ*, through image deconvolution with Wiener filter. We follow the procedure in (Li et al., 2005) where deconvolution is computed over a set of parameters. In this experiment, the values are *L* = {3, 5, 7, 9, 11, 13, 15} and *θ* = {0◦, 45◦, 90◦, 135◦}. The selection is based on a criterion

deals with RGB, the images are evaluated using only the green channel.

specifically, maximum PSNR, minimum *kp*, and maximum *q*.

Thus, the criterion for selection is the maximum *q* among the *n* values.

#### **5.2 Experimental results**

#### **5.2.1 Experiment descriptions**

There are three experiments that are being considered here. The first will investigate the effect of the different PSF sizes to the distribution of the pixel and maxima values. The latter are extracted by scanning the green channel from left to right and from top to bottom. The variances and kurtoses are compared for different conditions (Cond) as shown in Table 7. The subscript *z* ∈ {1, 2, ..., 4} is the index of the PSF support size where *z* = 4 indicates an unblurred image. The support sizes are {7 × 7, 5 × 5, 3 × 3}. There are 300 unblurred images with natural scenes that may be 640 × 480 or 480 × 640 pixels in size. Synthetic blurs include horizontal motion (HM), vertical motion (VM), and OOF.

18 Will-be-set-by-IN-TECH

Name Conditions Cond A *vp*,*z*� <sup>4</sup> < *vp*,*z*=<sup>4</sup> Cond B *km*,*z*� <sup>4</sup> < *km*,*z*=<sup>4</sup> Cond C *vp*,*<sup>z</sup>* < *vp*,*z*+<sup>1</sup> Cond D *km*,*<sup>z</sup>* < *km*,*z*+<sup>1</sup>

These quantities can be used to describe the distribution of the values for a given image. To demonstrate this, the unblurred image shown in Fig. 6 is synthetically blurred with OOF. The PSF sizes are {7 × 7, 5 × 5, 3 × 3} and these are indexed by an integer number where the smallest index represents the largest size. Thus, a large index number indicates a smaller PSF size and an image with better quality. For each image, the maxima values are scanned from left to right and from top to bottom. The variance and kurtosis are computed using the pixel and maxima values. This means that every degraded image will have four quantities. Fig. 7 shows that the variance of the pixel values monotonically increases as the size becomes smaller or as the index becomes larger. This trend is also the same for the kurtosis of the maxima values. Although kurtosis of the pixel values decreased, this trend is not monotonic. Therefore, the decrease in blur can be described by the increase of pixel variance (*v*p) and maxima kurtosis (*k*m). The subscripts are used to indicate the values that are used i.e., p for pixel values and m for maxima values. These characteristics are empirically proven to be consistent in many images as will be shown by the data. Due to this, the trends of *v*p and *k*m can be used for the selection of the regularization parameters and the deblurred image. For a method with *n* iterations, *v*p and *k*m will be composed of a set with *n* values. Each set is first scaled such that the minimum value is zero and the maximum is one. The average (*q*) for each iteration is then

*<sup>q</sup>* <sup>=</sup> *<sup>v</sup>*<sup>p</sup> <sup>+</sup> *<sup>k</sup>*<sup>m</sup>

There are three experiments that are being considered here. The first will investigate the effect of the different PSF sizes to the distribution of the pixel and maxima values. The latter are extracted by scanning the green channel from left to right and from top to bottom. The variances and kurtoses are compared for different conditions (Cond) as shown in Table 7. The subscript *z* ∈ {1, 2, ..., 4} is the index of the PSF support size where *z* = 4 indicates an unblurred image. The support sizes are {7 × 7, 5 × 5, 3 × 3}. There are 300 unblurred images with natural scenes that may be 640 × 480 or 480 × 640 pixels in size. Synthetic blurs include

Thus, the criterion for selection is the maximum *q* among the *n* values.

horizontal motion (HM), vertical motion (VM), and OOF.

*f* is the mean of the image and *N* represents the total number of pixels. On the other

*<sup>f</sup>*(*x*, *<sup>y</sup>*) <sup>−</sup> ¯ *f* 4

*<sup>v</sup>*<sup>2</sup> <sup>−</sup> 3. (42)

<sup>2</sup> . (43)

Table 7. Conditions for experiment 1.

hand, kurtosis can be represented by

*k* = 1 *<sup>N</sup>* ∑∀(*x*,*y*)

where ¯

computed as

**5.2 Experimental results**

**5.2.1 Experiment descriptions**

Fig. 6. An unblurred image.

The second experiment is the evaluation of the parameter selection and the resulting reconstructed images. For practical purposes, the image size is 60 × 60 and degradations are the same with the previous experiment but the support sizes are {3 × 3, 5 × 5}. The regularization parameters are computed for 100 iterations using the PSF sizes {3 × 3, 5 × 5, 7 × 7}. RGB images are used so that subtle changes in the images can be seen easily. As a result, the modified cost function in (Chong & Tanaka, 2010b) is used. The final PSF size and parameters are selected using the maximum value of *q*. These values are then used for the reconstruction of the degraded image computed for 50 iterations. Although this experiment deals with RGB, the images are evaluated using only the green channel.

The third experiment involves the selection of the motion blur parameters, *L* and *θ*, through image deconvolution with Wiener filter. We follow the procedure in (Li et al., 2005) where deconvolution is computed over a set of parameters. In this experiment, the values are *L* = {3, 5, 7, 9, 11, 13, 15} and *θ* = {0◦, 45◦, 90◦, 135◦}. The selection is based on a criterion specifically, maximum PSNR, minimum *kp*, and maximum *q*.

PSF maximum *q* Regularization Parameters Type 3 × 3 5 × 5 7 × 7 *λ α β*

Harnessing the Potentials of Image Extrema for Blind Restoration 221

HM 0.56 0.69 **0.84** 0.60 3.68 <sup>×</sup> <sup>10</sup><sup>5</sup> 5.17 <sup>×</sup> 106 VM **0.87** 0.74 0.64 0.22 3.09 <sup>×</sup> <sup>10</sup><sup>2</sup> 4.51 <sup>×</sup> 105 OOF **0.94** 0.89 0.75 0.51 2.46 <sup>×</sup> <sup>10</sup><sup>4</sup> 1.32 <sup>×</sup> 106

HM 0.72 **0.73** 0.72 0.50 9.40 <sup>×</sup> <sup>10</sup><sup>4</sup> 3.42 <sup>×</sup> 106 VM **0.89** 0.73 0.68 0.13 4.51 <sup>×</sup> <sup>10</sup><sup>2</sup> 2.33 <sup>×</sup> 105 OOF **0.72** 0.70 0.66 0.41 2.26 <sup>×</sup> <sup>10</sup><sup>4</sup> 7.38 <sup>×</sup> 105

For the second experiment, the PSF size and regularization parameters are chosen by the maximum value of *q*. This is shown in Table 9 for each PSF size. The emphasized number is the maximum considering a certain degraded image. The underlined number indicates correct identification of the PSF size. Based on the emphasized values, the resulting regularization parameters are also shown in Table 9. It can be observed that when *s* = 3, the correct size is selected for those blurred with VM and OOF. For HM, the selected size is larger than the actual. In contrast, this reverses when *s* = 5. Only HM is correct while VM and OOF have smaller sizes than the actual. The parameters are employed during reconstruction and the image quality measures are computed and shown in Table 10. The PSNR and *q* values are compared when one of these is the maximum. The rows with emphasized values are those with correct PSF sizes. It can be seen that when the size is incorrect the PSNRs are approximately the same. These are consistent despite the fact that when *s* = 3 the selected size is larger and when *s* = 5 the size is smaller. The same can be observed with the values of *q* except for VM when *s* = 5. On the other hand, when the size is correct both values (PSNR and *q*) are different. If we consider maximum PSNR then *q* is smaller and vice versa. To illustrate the difference in values, the corresponding images, when *s* = 3, are shown in Fig. 8. It can be observed, from the first row in Fig. 8, that reconstruction with an incorrect PSF size will yield images with a higher contrast. The second and third rows used correct PSF sizes. The reconstructed images show more details than the degraded versions. The difference is with the perceptible focus of the image. When PSNR is maximum the image appears smoother. However, when *q* is maximum the image appears sharper. The same observations are true even if the blur types are different. Thus, a *q* with higher value indicates a sharper image and low indicates a smoother image. This is due to the fact that the statistical distributions of the

The data in Table 11 show the selected parameters using different criteria. It can be observed that only the angle, *θ* = 0◦, are correctly identified by the maximum PSNR criterion. In contrast, *θ* = 90◦ and *L* = 3 are selected by minimum *kp*. Finally, for maximum *q* all

This section shows the effects of blur on the pixel variance and maxima kurtosis. The values are highest when the image is unblurred and these decrease as the PSF size increases. A new criterion for comparing images is also discussed. This does not require the unblurred image, which makes it useful for the reconstruction of images. By exploiting the pixel variance and maxima kurtosis, perceptible focus can be quantized. Higher values have sharper images

parameters are correct for small PSFs while only *θ* are correct for larger PSFs.

Table 9. Selection of the PSF size and regularization parameters using maximum *q*.

(*s* = 3)

(*s* = 5)

pixels and maxima are taken into consideration.

while lower ones have smoother images.

Fig. 7. Variances and kurtoses for OOF degraded versions of Fig. 6.


Table 8. Condition consistency with *vp* and *km* (%).

#### **5.2.2 Data and results**

Table 8 shows the consistency of specific conditions using image pixels and maxima values. The consistency here refers to the trueness of a condition when tested to a number of images. The letter s indicates sets where each set consists of four images with increasing PSF size. Cond A shows that all *vp* of the degraded images are lower than the unblurred image. This means that a high *vp* is expected in the absence of blur. The same is observable for *km* as shown by the data for Cond B. The monotonicity of the increase in *vp* is evaluated by Cond C. The percentage indicates that this is true for all the tested images. This shows that the *vp* value consistently increases as the PSF size decreases. On the other hand, monotonicity of *km* is not true for all the tested images. However, these only affect a few images as manifested by the high consistency values in row Cond D. With reference to the conditions in Table 7, it can be observed that these indicate that both *vp* and *kp* will increase as blur decreases. Since their behaviour are similar, integration will be easier in order to produce a single quantity that can reflect an image's quality.

20 Will-be-set-by-IN-TECH

1 1.5 2 2.5 3 3.5 4

pixel variance maxima variance pixel kurtosis maxima kurtosis

PSF size index

Setting Number of Accuracy (%) Total Images HM VM OOF Cond A 900 100.0 100.0 100.0 100.0 Cond B 900 100.0 100.0 100.0 100.0 Cond C 300s 100.0 100.0 100.0 100.0 Cond D 300s 98.7 98.3 95.3 97.4

Table 8 shows the consistency of specific conditions using image pixels and maxima values. The consistency here refers to the trueness of a condition when tested to a number of images. The letter s indicates sets where each set consists of four images with increasing PSF size. Cond A shows that all *vp* of the degraded images are lower than the unblurred image. This means that a high *vp* is expected in the absence of blur. The same is observable for *km* as shown by the data for Cond B. The monotonicity of the increase in *vp* is evaluated by Cond C. The percentage indicates that this is true for all the tested images. This shows that the *vp* value consistently increases as the PSF size decreases. On the other hand, monotonicity of *km* is not true for all the tested images. However, these only affect a few images as manifested by the high consistency values in row Cond D. With reference to the conditions in Table 7, it can be observed that these indicate that both *vp* and *kp* will increase as blur decreases. Since their behaviour are similar, integration will be easier in order to produce a single quantity that can

Fig. 7. Variances and kurtoses for OOF degraded versions of Fig. 6.

Table 8. Condition consistency with *vp* and *km* (%).

0.2

**5.2.2 Data and results**

reflect an image's quality.

0.3

0.4

0.5

0.6

normalized amplitude

0.7

0.8

0.9

1


Table 9. Selection of the PSF size and regularization parameters using maximum *q*.

For the second experiment, the PSF size and regularization parameters are chosen by the maximum value of *q*. This is shown in Table 9 for each PSF size. The emphasized number is the maximum considering a certain degraded image. The underlined number indicates correct identification of the PSF size. Based on the emphasized values, the resulting regularization parameters are also shown in Table 9. It can be observed that when *s* = 3, the correct size is selected for those blurred with VM and OOF. For HM, the selected size is larger than the actual. In contrast, this reverses when *s* = 5. Only HM is correct while VM and OOF have smaller sizes than the actual. The parameters are employed during reconstruction and the image quality measures are computed and shown in Table 10. The PSNR and *q* values are compared when one of these is the maximum. The rows with emphasized values are those with correct PSF sizes. It can be seen that when the size is incorrect the PSNRs are approximately the same. These are consistent despite the fact that when *s* = 3 the selected size is larger and when *s* = 5 the size is smaller. The same can be observed with the values of *q* except for VM when *s* = 5. On the other hand, when the size is correct both values (PSNR and *q*) are different. If we consider maximum PSNR then *q* is smaller and vice versa. To illustrate the difference in values, the corresponding images, when *s* = 3, are shown in Fig. 8. It can be observed, from the first row in Fig. 8, that reconstruction with an incorrect PSF size will yield images with a higher contrast. The second and third rows used correct PSF sizes. The reconstructed images show more details than the degraded versions. The difference is with the perceptible focus of the image. When PSNR is maximum the image appears smoother. However, when *q* is maximum the image appears sharper. The same observations are true even if the blur types are different. Thus, a *q* with higher value indicates a sharper image and low indicates a smoother image. This is due to the fact that the statistical distributions of the pixels and maxima are taken into consideration.

The data in Table 11 show the selected parameters using different criteria. It can be observed that only the angle, *θ* = 0◦, are correctly identified by the maximum PSNR criterion. In contrast, *θ* = 90◦ and *L* = 3 are selected by minimum *kp*. Finally, for maximum *q* all parameters are correct for small PSFs while only *θ* are correct for larger PSFs.

This section shows the effects of blur on the pixel variance and maxima kurtosis. The values are highest when the image is unblurred and these decrease as the PSF size increases. A new criterion for comparing images is also discussed. This does not require the unblurred image, which makes it useful for the reconstruction of images. By exploiting the pixel variance and maxima kurtosis, perceptible focus can be quantized. Higher values have sharper images while lower ones have smoother images.

(a) degraded by HM (b) PSNR=18.58, *q* = 0.92 (c) PSNR=18.56, *q* = 0.93

PSF maximum PSNR maximum *q* Type PSNR *q* PSNR *q*

Harnessing the Potentials of Image Extrema for Blind Restoration 223

HM 18.58 0.92 18.56 0.93 VM **22.95 0.53 20.55 0.84** OOF **27.54 0.17 22.88 1.00**

HM **20.05 0.64 19.53 0.86** VM 18.26 0.54 17.59 0.80 OOF 21.87 0.99 21.87 1.00

Type of Criterion

Motion PSNRmax *k*p\_min *q*max *L* = 3 *L* = 15 *L* = 3 *L* = **3** *θ* = 0◦ *θ* = **0**◦ *θ* = 90◦ *θ* = **0**◦ *L* = 3 *L* = 15 *L* = **3** *L* = **3** *θ* = 90◦ *θ* = 0◦ *θ* = **90**◦ *θ* = **90**◦ *L* = 5 *L* = 15 *L* = 15 *L* = 3 *θ* = 0◦ *θ* = **0**◦ *θ* = 135◦ *θ* = **0**◦ *L* = 5 *L* = 15 *L* = 3 *L* = 3 *θ* = 90◦ *θ* = 0◦ *θ* = **90**◦ *θ* = **90**◦

This chapter explores the different aspects of image deconvolution and the uses of extrema. For an efficient usage of resources, the whole deconvolution process is divided into several tasks. This begins with the detection and identification of blurs followed by the determination of RPSF. The resulting quantity is integrated into the cost function that is used to estimate the unblurred image. Since the method is iterative, several images will result and these must be

Section 1 introduces image deconvolution and its importance in different fields of applications. Some variations of its mathematical model are also mentioned as a backgrounder. The next section discusses image extrema and its exploitation for blur detection and identification. The concept is based on how blur changes the extrema locations and distributions. A histogram is first created and from which, specific features are computed. By using a classifier, the presence of the blur can be detected. Aside from this, it is also possible to identify its type. This has been tested to have high accuracy for Gaussian, motion and OOF

Degraded images must be processed further in order for them to yield more information. An initial step is to extract some details regarding its blurring function. This is accomplished by exploiting the maxima and computing for the RPSF as shown in section 3. This is especially applicable for motion blurs because the maxima locations also align with the motion direction. By employing the Hough transform, a single direction is detected and the resulting RPSF will be more similar to the actual PSF. Testing the accuracy of detecting the direction involved various natural images that are blurred synthetically with different PSF sizes. High values

(*s* = 3)

(*s* = 5)

Table 10. Comparison between PSNR and *q* (dB).

Table 11. Motion parameter selection with a Wiener filter.

compared to determine the best one.

**6. Summary**

blurs.

(d) degraded by VM (e) PSNR=22.95, *q* = 0.53 (f) PSNR=20.55, *q* = 0.84

(g) degraded by OOF (h) PSNR=27.54, *q* = 0.17 (i) PSNR=22.88, *q* = 1.00

Fig. 8. Degraded images and their reconstructed versions based on Table 10 when *s* = 3.



Table 11. Motion parameter selection with a Wiener filter.

## **6. Summary**

22 Will-be-set-by-IN-TECH

(a) degraded by HM (b) PSNR=18.58, *q* = 0.92 (c) PSNR=18.56, *q* = 0.93

(d) degraded by VM (e) PSNR=22.95, *q* = 0.53 (f) PSNR=20.55, *q* = 0.84

(g) degraded by OOF (h) PSNR=27.54, *q* = 0.17 (i) PSNR=22.88, *q* = 1.00

Fig. 8. Degraded images and their reconstructed versions based on Table 10 when *s* = 3.

This chapter explores the different aspects of image deconvolution and the uses of extrema. For an efficient usage of resources, the whole deconvolution process is divided into several tasks. This begins with the detection and identification of blurs followed by the determination of RPSF. The resulting quantity is integrated into the cost function that is used to estimate the unblurred image. Since the method is iterative, several images will result and these must be compared to determine the best one.

Section 1 introduces image deconvolution and its importance in different fields of applications. Some variations of its mathematical model are also mentioned as a backgrounder. The next section discusses image extrema and its exploitation for blur detection and identification. The concept is based on how blur changes the extrema locations and distributions. A histogram is first created and from which, specific features are computed. By using a classifier, the presence of the blur can be detected. Aside from this, it is also possible to identify its type. This has been tested to have high accuracy for Gaussian, motion and OOF blurs.

Degraded images must be processed further in order for them to yield more information. An initial step is to extract some details regarding its blurring function. This is accomplished by exploiting the maxima and computing for the RPSF as shown in section 3. This is especially applicable for motion blurs because the maxima locations also align with the motion direction. By employing the Hough transform, a single direction is detected and the resulting RPSF will be more similar to the actual PSF. Testing the accuracy of detecting the direction involved various natural images that are blurred synthetically with different PSF sizes. High values

Aizenberg, I., Paliy, D. V., Zurada, J. M. & Astola, J. T. (2008). Blur identification by

Harnessing the Potentials of Image Extrema for Blind Restoration 225

Banham, M. R. & Katsaggelos, A. K. (1997). Digital image restoration, *IEEE Signal Processing*

Cadzow, J. A. (2002). Minimum 1, 2, and <sup>∞</sup> norm approximate solutions to an overdetermined system of linear equations, *Digital Signal Processing* 12(4): 524–560. Cappelli, R., Lumini, A., Maio, D. & Maltoni, D. (2007). Fingerprint image reconstruction

Chan, T. F., Golub, G. H. & Mulet, P. (1999). A nonlinear primal-dual method

Chan, T. F. & Wong, C. K. (1998). Total variation blind deconvolution, *IEEE Transactions on*

Choksi, R. & van Gennip, Y. (2010). Deblurring of one dimensional bar codes via total variation

Chong, R. M. & Tanaka, T. (2008). Image extrema analysis and blur detection with

Chong, R. M. & Tanaka, T. (2010a). Blur identification based on maxima locations for

Chong, R. M. & Tanaka, T. (2010b). Motion blur identification using maxima locations for

Chong, R. M. & Tanaka, T. (2011a). Detection of motion blur direction based on maxima

Chong, R. M. & Tanaka, T. (2011b). Maxima exploitation for reference blurring function in motion deconvolution, *IEICE Transactions on Fundamentals* E94-A(3): 921–928. Chong, R. M. & Tanaka, T. (2011c). An objective criterion for the comparison of reconstructed

Chu, C. H., Yang, D. N. & Chen, M. S. (2010). Single image deblurring for a real-time

Chu, C. H., Yang, D. N., Pan, Y. L. & Chen, M. S. (2010). Stabilization and extraction of 2D

Chung, Y. C., Wang, J. M., Bailey, R. R., Chen, S. W. & Chang, S. L. (2004). A non-parametric

Cover, T. M. & Hart, P. E. (1967). Nearest neighbor pattern classification, *IEEE Transactions on*

Esedoglu, S. (2004). Blind deconvolution of bar code signals, *Inverse Problems* 20(1): 121–135. Gonzales, R. C. & Woods, R. E. (2008). *Digital Image Processing*, third edn, Pearson Education,

*Internet Based System (SITIS2008)*, Vol. 1, Bali, Indonesia, pp. 320–326. Chong, R. M. & Tanaka, T. (2009). Detection and classification of invariant blurs, *IEICE*

energy minimisation, *SIAM J. on Imaging Sciences* 3-4: 735–764.

*Transactions on Fundamentals* E92-A(12): 3313–3320.

*Ubiquitous Engineering (MUE2010)*, Cebu, Philippines.

blind color image restoration, *Journal of Convergence* 1(1): 49–56.

*Signal Processing (ICICS2011)*, Singapore. to be published.

barcodes for camera phones, *Multimedia Systems* pp. 1–21.

*Electronics Society (IECON 2010)*, Glendale, AZ, pp. 1185–1192.

*Networks* 19(5): 883–898.

*Magazine* 14(2): 24–41.

29(9): 1489–11503.

20(6): 1964–1977.

be published.

Singapore, pp. 356–360.

USA.

*Information Theory* 13(1): 21–27.

*Image Processing* 7(3): 370–375.

multilayer neural network based on multivalued neurons, *IEEE Trans. on Neural*

from standard templates, *IEEE Transactions on Pattern Analysis and Machine Intelligence*

for total variation-based image restoration, *SIAM Journal on Scientific Computing*

identification, *Proceedings of the International Conference on Signal Image Technology and*

color image restoration, *Proceedings of the International Conference on Multimedia and*

locations for blind deconvolution, *Proceedings of the IS&T/SPIE Electronic Imaging*. to

images, *Proceedings of the International Conference on Information, Communications and*

face recognition system, *Proceedings of the 36th Annual Conference on IEEE Industrial*

blur measure based on edge analysis for image processing applications, *Proceedings of the IEEE International Conference on Cybernetics and Intelligent Systems (CIS2004)*, Vol. 1,

indicate the efficacy of the method. An advantage of this is that it can reveal the nature of the embedded PSF using only the degraded image. It does not require a training set to determine the direction of motion.

RPSF can also be used during the estimation of the unblurred image. In section 4 the classical cost function for deconvolution is reformulated. It contains the terms for fidelity, image smoothness , and PSF characterization, which consists of the PSF smoothness and learning terms. The smoothness terms are included due to the fact that images as well as piecewise PSFs have edges. The PSF learning term integrates the RPSF during deconvolution. In this way, RPSF provides a way of reinforcement without the need for a training set. During the estimation process, RPSF also eliminates the dependence of learning on the current computed PSF. The cost function requires three regularization parameters that can be solved using the OSLEs. All equations have been expressed in the vector-matrix form to facilitate easier derivations. The resulting data show consistently high PSNRs for different motion directions and PSF sizes. This means that the image quality are improved. Furthermore, the MSEs of the PSFs are also very low, which indicates a high accuracy in estimating the PSF.

Lastly, a method to compare images is discussed in section 5. Iterative methods for deconvolution will naturally result in several estimated images and PSFs. When image quality is given more importance, an assessment technique is needed to choose the best one. Classical methods require the unblurred version however, this is not applicable in practical applications. Thus, the pixel variance and maxima kurtosis have been exploited to overcome this problem. It has been shown that both quantities are low when the blur is high and these monotonically increase as the blur decreases. Their combination resulted to a single value that can be used to compare several images. Experimental results have shown that this is high when an image is sharp. In contrast, when the value is low the image is smooth.

The extrema considered in this chapter are extracted by horizontal and vertical scanning. The effects of using other scanning directions are still unknown. Additionally, the degraded images are all invariantly blurred. This means that the PSF is assumed to be uniform all throughout the image. In practical applications, the variant case is also possible. The results for this case remain an open problem at this time. Finally, the image comparison technique is capable of differentiating images based on the sharpness of the details without the unblurred image. However, it cannot indicate the presence of distortions such as ringing and color artefacts. Quantizing these is not only important in image deconvolution but also in other applications of image processing.

## **7. References**


24 Will-be-set-by-IN-TECH

indicate the efficacy of the method. An advantage of this is that it can reveal the nature of the embedded PSF using only the degraded image. It does not require a training set to determine

RPSF can also be used during the estimation of the unblurred image. In section 4 the classical cost function for deconvolution is reformulated. It contains the terms for fidelity, image smoothness , and PSF characterization, which consists of the PSF smoothness and learning terms. The smoothness terms are included due to the fact that images as well as piecewise PSFs have edges. The PSF learning term integrates the RPSF during deconvolution. In this way, RPSF provides a way of reinforcement without the need for a training set. During the estimation process, RPSF also eliminates the dependence of learning on the current computed PSF. The cost function requires three regularization parameters that can be solved using the OSLEs. All equations have been expressed in the vector-matrix form to facilitate easier derivations. The resulting data show consistently high PSNRs for different motion directions and PSF sizes. This means that the image quality are improved. Furthermore, the MSEs of the

Lastly, a method to compare images is discussed in section 5. Iterative methods for deconvolution will naturally result in several estimated images and PSFs. When image quality is given more importance, an assessment technique is needed to choose the best one. Classical methods require the unblurred version however, this is not applicable in practical applications. Thus, the pixel variance and maxima kurtosis have been exploited to overcome this problem. It has been shown that both quantities are low when the blur is high and these monotonically increase as the blur decreases. Their combination resulted to a single value that can be used to compare several images. Experimental results have shown that this is high

The extrema considered in this chapter are extracted by horizontal and vertical scanning. The effects of using other scanning directions are still unknown. Additionally, the degraded images are all invariantly blurred. This means that the PSF is assumed to be uniform all throughout the image. In practical applications, the variant case is also possible. The results for this case remain an open problem at this time. Finally, the image comparison technique is capable of differentiating images based on the sharpness of the details without the unblurred image. However, it cannot indicate the presence of distortions such as ringing and color artefacts. Quantizing these is not only important in image deconvolution but also in other

Aizenberg, I., Butakoff, C., Karnaukhov, V., Merzlyakov, N. & Milukova, O. (2002). Type

Aizenberg, I., Butakoff, C., Karnaukhov, V., Merzlyakov, N. & Milukova, O. (n.d.). Blurred

Aizenberg, I., Paliy, D., Moraga, C. & Astola, J. (2006). Blur identification using neural

*Networks (ICANN2002)*, Madrid, Spain, pp. 1231–1236.

neural network, *Proceedings of the SPIE*, Vol. 4667.

*Days*, Dortmund, Germany, pp. 441–455.

of blur and blur parameters identification using neural network and its application to image restoration, *Proceedings of the International Conference on Artificial Neural*

image restoration using the type of blur and blur parameters identification on the

network for image restoration, *Proceedings of the International Conference 9th Fuzzy*

PSFs are also very low, which indicates a high accuracy in estimating the PSF.

when an image is sharp. In contrast, when the value is low the image is smooth.

the direction of motion.

applications of image processing.

**7. References**


**Part 2** 

**Applications** 


## **Part 2**

## **Applications**

26 Will-be-set-by-IN-TECH

226 Image Restoration – Recent Advances and Applications

He, Y., Yap, K. H., Chen, L. & Chau, L. P. (2008). A novel hybrid model framework to blind

He, Y., Yap, K. H., Chen, L. & Chau, L. P. (2009). A soft MAP framework for blind super-resolution image reconstruction, *Image and Vision Computing* 27(4): 364–373. Kang, B. J. & Park, K. R. (2007). Real-time image restoration for iris recognition systems, *IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics* 37(6): 1555–1566. Lagendijk, R. L. & Biemond, J. (2005). Basic methods for image restoration and identification,

Li, D., Mersereau, R. M. & Simske, S. (2005). Blur identification based on kurtosis

Marziliano, P., Dufaux, F., Winkler, S. & Ebrahimi, T. (2002). A no-reference perceptual blur

Marziliano, P., Dufaux, F., Winkler, S. & Ebrahimi, T. (2004). Perceptual blur and ringing

Nishiyama, M., Hadid, A., Takeshima, H., Shotton, J., Kozakaya, T. & Yamaguchi, O. (2010).

Rooms, F., Pizurica, A. & Philips, W. (2002). Estimating image blur in the wavelet

Rudin, L. I., Osher, S. & Fatemi, E. (1992). Nonlinear total variation based noise removal

Simoncelli, E. P. (2005). Statistical modeling of photographic images, *in* A. Bovik (ed.),

Sroubek, F. & Flusser, J. (2003). Multichannel blind iterative image restoration, *IEEE*

Theodoridis, S. & Koutroumbas, K. (2006). *Pattern Recognition*, third edn, Elsevier Academic

Tong, H., Li, M., Zhang, H. & Zhang, C. (2004). Blur detection for digital images using

Xin, F., Qi, Z., Dequn, L. & Ling, Z. (2003). Face image restoration based on statistical prior

Yahyanejad, S. & Strom, J. (2010). Removing motion blur from barcode images, *IEEE Computer*

Yang, Y., Galatsanos, N. P. & Stark, H. (1994). Projection-based blind deconvolution, *J. Opt.*

You, Y. & Kaveh, M. (1996). A regularization approach to joint blur identification and image

restoration, *IEEE Transactions on Image Processing* 5(3): 416–428.

*Transactions on Pattern Analysis and Machine Intelligence* PP(99): 1–1.

*Processing (ICASSP2002)*, Vol. 4, Orlando, FL, pp. IV–4190.

*Transactions on Image Processing* 12(9): 1094–1106.

*Expo (ICME2004)*, Vol. 1, Taipei, Taiwan, pp. 17–20.

*and Expo (ICME 2003)*, Vol. 3, pp. III–297–300.

*2010*, San Francisco, CA, pp. 41–46.

*Soc. Am. A* 11(9): 2401–2409.

*A: Systems and Humans* 38(4): 867–880.

Press, Orlando, FL, chapter 3.5, pp. 167–181.

*(ICIP2005)*, Genoa, Italy, pp. I–905–8.

Vol. 3, Rochester, NY, pp. 57–60.

algorithms, *Physica D* 60: 259–268.

FL, chapter 4.7, pp. 431–441.

Press, USA.

color image deconvolution, *IEEE Transactions on Systems, Man, and Cybernetics–Part*

*in* A. Bovik (ed.), *Handbook of Image & Video Processing*, second edn, Elsevier Academic

minimization, *Proceedings of the IEEE International Conference on Image Processing*

metric, *Proceedings of the IEEE International Conference on Image Processing (ICIP2002)*,

metrics: application to JPEG2000, *Signal Processing: Image Communication* 19: 163–172.

Facial deblur inference using subspace analysis for recognition of blurred faces, *IEEE*

domain, *Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal*

*Handbook of Image & Video Processing*, second edn, Elsevier Academic Press, Orlando,

wavelet transform, *Proceedings of the IEEE International Conference on Multimedia and*

and image blur measure, *Proceedings of the IEEE International Conference on Multimedia*

*Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*

**11** 

Min-Cheng Pan

*New Taipei City* 

 *Taiwan* 

**Image Restoration for** 

 **Long-Wavelength Imaging Systems** 

Basically, the quality of an image can be evaluated on its spatial and spatial-frequency resolutions, image interpolation and superresolution are perhaps the way to respectively produce high spatial and spatial-frequency resolutions of images especially for a single downsampled image. For convenience, the term "hyper-resolution" used here represents the approach to enhancing both the spatial and the spatial-frequency resolutions of an image.

As known, the process of decimation or down-sampling is an effective way often used to reduce image sizes, thus, reducing the amount of information transmitted through the communication channels and the local storage requirements, while trying to preserve as much as possible the image quality. Conversely, the reverse procedure of this, referred to as interpolation or up-sampling, is useful in restoring the original high resolution image from its decimated version or for resizing or zooming a digital image. Decimation and interpolation are used for several purposes in many practical applications, such as progressive image transmission systems, multimedia applications, and so forth. A number of conventional interpolation techniques [Hou & Andrews, 1978; Jain, 1989; Keys, 1981] have been proposed to increase the spatial resolution of an image. Undoubtedly, these techniques

Furthermore, images may be corrupted by degradation such as blurring distortion, noise, and blocking artifacts. These sources of degradation may arise during image capture or processing and have a direct bearing on visual quality. Various methods of restoration have been described in the literature; this diversity reveals the importance of the problem and its great difficulty. The purpose of image deconvolution or restoration is to recover degraded images by removing noise, highlighting image contrast, and preserving edge features of image.

Image superresolution was developed in 1950s to improve image quality and pilot research of this field is derived from the early work (Toraldo di Francia, 1952, 1955) where the term "superresolution" was used in the paper. Following that, clear definition, description and some of the obvious contribution to this field can be found in the work (Gerchberg, 1974; Hunt & Sementilli, 1992) in which their work, superresolution, was meant to seek to recover

**1. Introduction 1.1 Overview** 

degrade the quality of the magnified image.

*Department of Electronic Engineering, Tungnan University* 

## **Image Restoration for Long-Wavelength Imaging Systems**

Min-Cheng Pan *Department of Electronic Engineering, Tungnan University New Taipei City Taiwan* 

## **1. Introduction**

## **1.1 Overview**

Basically, the quality of an image can be evaluated on its spatial and spatial-frequency resolutions, image interpolation and superresolution are perhaps the way to respectively produce high spatial and spatial-frequency resolutions of images especially for a single downsampled image. For convenience, the term "hyper-resolution" used here represents the approach to enhancing both the spatial and the spatial-frequency resolutions of an image.

As known, the process of decimation or down-sampling is an effective way often used to reduce image sizes, thus, reducing the amount of information transmitted through the communication channels and the local storage requirements, while trying to preserve as much as possible the image quality. Conversely, the reverse procedure of this, referred to as interpolation or up-sampling, is useful in restoring the original high resolution image from its decimated version or for resizing or zooming a digital image. Decimation and interpolation are used for several purposes in many practical applications, such as progressive image transmission systems, multimedia applications, and so forth. A number of conventional interpolation techniques [Hou & Andrews, 1978; Jain, 1989; Keys, 1981] have been proposed to increase the spatial resolution of an image. Undoubtedly, these techniques degrade the quality of the magnified image.

Furthermore, images may be corrupted by degradation such as blurring distortion, noise, and blocking artifacts. These sources of degradation may arise during image capture or processing and have a direct bearing on visual quality. Various methods of restoration have been described in the literature; this diversity reveals the importance of the problem and its great difficulty. The purpose of image deconvolution or restoration is to recover degraded images by removing noise, highlighting image contrast, and preserving edge features of image.

Image superresolution was developed in 1950s to improve image quality and pilot research of this field is derived from the early work (Toraldo di Francia, 1952, 1955) where the term "superresolution" was used in the paper. Following that, clear definition, description and some of the obvious contribution to this field can be found in the work (Gerchberg, 1974; Hunt & Sementilli, 1992) in which their work, superresolution, was meant to seek to recover

Image Restoration for Long-Wavelength Imaging Systems 231

image processing techniques can improve the image quality of tomographic images between

The technique using image restoration gradually becomes popular for an mm-wave or an NIR DOT imaging system; the difference of both imaging systems is that the former is post-

There are two categories of restoration methods for improving image quality: (i) noniterative restoration such as the inverse filter, the Wiener filter (Wiener, 1942) and (ii) nonlinear iterative restoration/superresolution techniques such as Lorentzian restoration method (Lettington & Hong, 1995), maximum *a posteriori* (MAP) (Hunt & Sementilli, 1992), Richardson-Lucy (RL) deconvolution method (Richardson, 1972; Lucy, 1974), maximum entropy (Frieden, 1972), projection onto convex sets (Sezan & Tekalp, 1988), Gerchberg error energy reduction process (Gerchberg, 1974), and edge-preserving regularization (Teboul et al., 1998). In these methods, it is essential to use the adequate blurring function (a low-pass

In this section, we have described a number of terms such as spatial resolution, spatialfrequency resolution, interpolation, restoration, superresolution, hyper-resolution, interprocessing, and post-processing. In addition, advantages and drawbacks of long wavelength imaging systems were addressed and general description of restoration algorithms was made. It is worth emphasizing that long wavelength imaging systems have the same problem to be

Following this introduction, this chapter is organized as follows. Section 2 describes mathematical model of image formation; image restoration algorithms and further consideration on image restoration are explained in Sec. 3 and Sec. 4, respectively. Subsequently, Sec. 5 demonstrates related applications of image restoration. Finally,

In this section, imaging systems, image formation model, and forward problem and inverse

Usually, the imaging process of a common imaging system is formed as follows. Suppose we have a scene of interest that is going to pass through a common imaging system where it has been corrupted by a linear blurring function and some additive noise*.* The blurring function *h* accounts for the imperfectness of the imaging system including optical lens or the human factors in shooting the images. Some typical examples are a diffraction-limited or defocused lens and camera motion or shaking during the exposure. The noise arises from

dealt with so image restoration can be used to improve such an imaging system.

iterations of image reconstruction.

filter) to restore a degraded image.

conclusion is drawn in Sec. 6.

**2.1 Imaging systems** 

**2. Mathematical model of Image formation** 

problem are described in the following.

**2.1.1 Common imaging system** 

**1.4 Remark** 

processing and the latter is inter-processing.

**1.3 Varied algorithms of Image restoration** 

object information from above the spatial-frequency limit of diffraction. Originally, superresolution referred to a technique of one-frame-to-one-frame and its interest was in the spatial-frequency domain, neither for multi-frames-to-one-frame nor for interpolation. Since then, signal/image restoration/superresolution has been concerned for the spatialfrequency domain from one low-resolution frame to one high-resolution frame; basically, the distinct nature of those algorithms is iterative and nonlinear. A process of interpolation along with restoration/superresolution was used with one frame to enhance the spatial and spatialfrequency resolution of the frame (Pan, 2006). Else, the processing of multi-frame-to-one-frame has been quite concerned (Gillette et al., 1995; Ng et al., 2003; Segal et al., 2003), where a single high-resolution frame was reconstructed from multiple low-resolution frames.

## **1.2 Long-wavelength imaging system**

Image restoration is able to be applied to the long-wavelength imaging systems, millimeterwave (mm-wave) and near-infrared diffuse optical tomography (NIR DOT) imaging systems, shown as Fig. 1.1.. The advantage of long-wavelength imaging systems is to provide special information with no radioactive characteristics but the physical property of long wavelength with diffraction or scattering results in lower spatial-frequency resolution images, however, which can be improved using image restoration to enhance its applicability.

Fig. 1.1. Long Wavelength Imaging Systems v.s. Image Restoration

Images acquired from millimeter-wave imaging system for the fog or rain weather can be applied to navigation; its image resolution of 2D image can be improved with the technology of image restoration. NIR DOT imaging system provides computed tomography (CT) images of the human body or biological tissue/organ, used in medical diagnosis; image processing techniques can improve the image quality of tomographic images between iterations of image reconstruction.

The technique using image restoration gradually becomes popular for an mm-wave or an NIR DOT imaging system; the difference of both imaging systems is that the former is postprocessing and the latter is inter-processing.

## **1.3 Varied algorithms of Image restoration**

There are two categories of restoration methods for improving image quality: (i) noniterative restoration such as the inverse filter, the Wiener filter (Wiener, 1942) and (ii) nonlinear iterative restoration/superresolution techniques such as Lorentzian restoration method (Lettington & Hong, 1995), maximum *a posteriori* (MAP) (Hunt & Sementilli, 1992), Richardson-Lucy (RL) deconvolution method (Richardson, 1972; Lucy, 1974), maximum entropy (Frieden, 1972), projection onto convex sets (Sezan & Tekalp, 1988), Gerchberg error energy reduction process (Gerchberg, 1974), and edge-preserving regularization (Teboul et al., 1998). In these methods, it is essential to use the adequate blurring function (a low-pass filter) to restore a degraded image.

## **1.4 Remark**

230 Image Restoration – Recent Advances and Applications

object information from above the spatial-frequency limit of diffraction. Originally, superresolution referred to a technique of one-frame-to-one-frame and its interest was in the spatial-frequency domain, neither for multi-frames-to-one-frame nor for interpolation. Since then, signal/image restoration/superresolution has been concerned for the spatialfrequency domain from one low-resolution frame to one high-resolution frame; basically, the distinct nature of those algorithms is iterative and nonlinear. A process of interpolation along with restoration/superresolution was used with one frame to enhance the spatial and spatialfrequency resolution of the frame (Pan, 2006). Else, the processing of multi-frame-to-one-frame has been quite concerned (Gillette et al., 1995; Ng et al., 2003; Segal et al., 2003), where a single

Image restoration is able to be applied to the long-wavelength imaging systems, millimeterwave (mm-wave) and near-infrared diffuse optical tomography (NIR DOT) imaging systems, shown as Fig. 1.1.. The advantage of long-wavelength imaging systems is to provide special information with no radioactive characteristics but the physical property of long wavelength with diffraction or scattering results in lower spatial-frequency resolution images,

> **Near infrared diffusive Optical tomography**

\*imaging system \*breast diagnosis

**Medical Science** : Tissue optics Clinical Medicine

> **Software System:** Reconstruction Inter-processing

**inter-processing(1D signals2D image)**

**Hardware System:** Optoelectronics Scanning device 1**D signals**

however, which can be improved using image restoration to enhance its applicability.

**Long wavelength**

Non-radioactive

diffraction diffusion low resolution

**Hardware System: 2D image**

Images acquired from millimeter-wave imaging system for the fog or rain weather can be applied to navigation; its image resolution of 2D image can be improved with the technology of image restoration. NIR DOT imaging system provides computed tomography (CT) images of the human body or biological tissue/organ, used in medical diagnosis;

high-resolution frame was reconstructed from multiple low-resolution frames.

**1.2 Long-wavelength imaging system** 

**mm-wave imaging system**

**(2D image) post-processing**

**2D image restoration**

Fig. 1.1. Long Wavelength Imaging Systems v.s. Image Restoration

\*imaging system \*fog or drizzle weather

Optoelectronics Scanning device **2D image**

In this section, we have described a number of terms such as spatial resolution, spatialfrequency resolution, interpolation, restoration, superresolution, hyper-resolution, interprocessing, and post-processing. In addition, advantages and drawbacks of long wavelength imaging systems were addressed and general description of restoration algorithms was made. It is worth emphasizing that long wavelength imaging systems have the same problem to be dealt with so image restoration can be used to improve such an imaging system.

Following this introduction, this chapter is organized as follows. Section 2 describes mathematical model of image formation; image restoration algorithms and further consideration on image restoration are explained in Sec. 3 and Sec. 4, respectively. Subsequently, Sec. 5 demonstrates related applications of image restoration. Finally, conclusion is drawn in Sec. 6.

## **2. Mathematical model of Image formation**

In this section, imaging systems, image formation model, and forward problem and inverse problem are described in the following.

## **2.1 Imaging systems**

## **2.1.1 Common imaging system**

Usually, the imaging process of a common imaging system is formed as follows. Suppose we have a scene of interest that is going to pass through a common imaging system where it has been corrupted by a linear blurring function and some additive noise*.* The blurring function *h* accounts for the imperfectness of the imaging system including optical lens or the human factors in shooting the images. Some typical examples are a diffraction-limited or defocused lens and camera motion or shaking during the exposure. The noise arises from

Image Restoration for Long-Wavelength Imaging Systems 233

In a common imaging system, the image is formed as the above description in which finding an estimated original signal/image (f) is an inverse problem for a given corrupted signal/image (g) while the reverse process is a forward problem. In tomographic imaging, the reconstruction of optical-property images is done iteratively using a Newton method, requiring inversion of a highly ill-posed and ill-conditioned matrix. The goal of DOT is to estimate the distribution of the optical properties in tissue from non-invasive boundary measurements. For the purpose of determining the optical properties (the absorption coefficient and the diffusion/scattering coefficient) from measurement data, which is an inverse problem in DOT, a forward model is needed to describe the physical relation between the boundary measurements of tissue and the optical properties that characterize the tissue.

In general, such a forward model of NIR DOT that gives the description of this physical

 , ,, *<sup>a</sup> <sup>i</sup> <sup>S</sup> c* 

the optical absorption and diffusion coefficients, respectively. In addition, the finite element method (FEM) and a Robin (type-III) [Brendel & Nielsen, 2009; Holboke et al., 2000] boundary condition are applied on Eq. (2.2) to solve this forward problem, i.e., calculating

Owing to the non-linearity with respect to the optical properties, an analytic solution to the inverse problem in DOT is absent. Instead, the numerical way of obtaining the inverse solution is to iteratively minimize the difference between the measured diffusion photon density data, **Φ***<sup>M</sup>* , around the tissue and the calculated model data, **Φ***<sup>C</sup>* , from solving the forward problem with the current estimated optical properties. This data-model misfit

<sup>2</sup> <sup>2</sup>

,

*a* **Φ Φ ΦΦ μ κ**

*C M i i*

*C C*

*a*

**μ κ**

1

*i*

By means of the first order Taylor series to expand **Φ** , one can get Eq. (2.4),

*M C*

*NM*

 

**rr r r r** (2.2)

 

(2.3)

(2.4)

is the light modulation frequency.

*a* and  denote

is the isotropic source term and *c* is the speed of light in tissue.

is the photon density at position **r** and

the photon density for a given set of optical property within the tissue.

**2.3 Forward problem & inverse problem** 

**2.3.1 Forward problem in DOT** 

relation is the diffusion equation,

where **r**,

*S***r**, **2.3.2 Inverse problem in DOT** 

difference is typically defined as follows,

where *NM* is the number of measurements.

the inherent characteristics of the recording media, e.g., electronic noise and quantization noise when the images are digitized (or discretized).

In practice, the available blurred image not only follows exactly the above description but also is constrained with the film size, in most cases the images have to be truncated at the boundaries. Instead*,* what is available now becomes a windowed blurred image where a rectangular window is usually accounting for the film aperture shape and size. One inherent problem with this is that many ringing artifacts are introduced into the restored image when the linear or nonlinear filter is applied directly to the truncated blurred image.

## **2.1.2 Medical imaging system**

Here, we use NIR DOT imaging system as an example. Basically, an NIR DOT imaging system is composed of a measuring instrument associated with image reconstruction scheme for the purpose of reconstructing the NIR optical-property tomographic images of phantoms/tissue of interest. The reconstructed images reveal the NIR optical properties of tissue computed by using measured radiances emitted from the circumference of the object. A schematic diagram of the NIR DOT measuring system in the frequency domain is shown in Fig. 2.1.

Fig. 2.1. Schematic diagram of NIR DOT measuring system in the frequency domain.

## **2.2 Image formation model**

The image formation is modelled as

$$\mathbf{g} = f \otimes \mathbf{h} + \mathbf{n} \tag{2.1}$$

where *f* is the original scene, *h* is the point-spread function (p.s.f.) of the imaging system, is the convolution operator, *n* is the noise, and *g* is the corrupted image. Subsequently, the corrupted image is windowed due to the film size/support area and sampled for digitization.

Aliasing is arising, which causes different signals to become indistinguishable when sampled. It also refers to the distortion or artifact that results when the signal reconstructed from samples is different from the original continuous signal.

#### **2.3 Forward problem & inverse problem**

232 Image Restoration – Recent Advances and Applications

the inherent characteristics of the recording media, e.g., electronic noise and quantization

In practice, the available blurred image not only follows exactly the above description but also is constrained with the film size, in most cases the images have to be truncated at the boundaries. Instead*,* what is available now becomes a windowed blurred image where a rectangular window is usually accounting for the film aperture shape and size. One inherent problem with this is that many ringing artifacts are introduced into the restored image when

Here, we use NIR DOT imaging system as an example. Basically, an NIR DOT imaging system is composed of a measuring instrument associated with image reconstruction scheme for the purpose of reconstructing the NIR optical-property tomographic images of phantoms/tissue of interest. The reconstructed images reveal the NIR optical properties of tissue computed by using measured radiances emitted from the circumference of the object. A schematic diagram

the linear or nonlinear filter is applied directly to the truncated blurred image.

of the NIR DOT measuring system in the frequency domain is shown in Fig. 2.1.

Fig. 2.1. Schematic diagram of NIR DOT measuring system in the frequency domain.

where *f* is the original scene, *h* is the point-spread function (p.s.f.) of the imaging system, is the convolution operator, *n* is the noise, and *g* is the corrupted image. Subsequently, the corrupted image is windowed due to the film size/support area and sampled for

Aliasing is arising, which causes different signals to become indistinguishable when sampled. It also refers to the distortion or artifact that results when the signal reconstructed

from samples is different from the original continuous signal.

*g f hn* (2.1)

noise when the images are digitized (or discretized).

**2.1.2 Medical imaging system** 

**2.2 Image formation model** 

digitization.

The image formation is modelled as

In a common imaging system, the image is formed as the above description in which finding an estimated original signal/image (f) is an inverse problem for a given corrupted signal/image (g) while the reverse process is a forward problem. In tomographic imaging, the reconstruction of optical-property images is done iteratively using a Newton method, requiring inversion of a highly ill-posed and ill-conditioned matrix. The goal of DOT is to estimate the distribution of the optical properties in tissue from non-invasive boundary measurements. For the purpose of determining the optical properties (the absorption coefficient and the diffusion/scattering coefficient) from measurement data, which is an inverse problem in DOT, a forward model is needed to describe the physical relation between the boundary measurements of tissue and the optical properties that characterize the tissue.

#### **2.3.1 Forward problem in DOT**

In general, such a forward model of NIR DOT that gives the description of this physical relation is the diffusion equation,

$$\nabla \cdot \mathbf{\kappa}(\mathbf{r}) \nabla \Phi(\mathbf{r}, \alpha) - \left[ \mu\_a(\mathbf{r}) - \frac{i\alpha}{c} \right] \Phi(\mathbf{r}, \alpha) = -\mathbf{S}(\mathbf{r}, \alpha) \tag{2.2}$$

where **r**, is the photon density at position **r** and is the light modulation frequency. *S***r**, is the isotropic source term and *c* is the speed of light in tissue. *a* and denote the optical absorption and diffusion coefficients, respectively. In addition, the finite element method (FEM) and a Robin (type-III) [Brendel & Nielsen, 2009; Holboke et al., 2000] boundary condition are applied on Eq. (2.2) to solve this forward problem, i.e., calculating the photon density for a given set of optical property within the tissue.

#### **2.3.2 Inverse problem in DOT**

Owing to the non-linearity with respect to the optical properties, an analytic solution to the inverse problem in DOT is absent. Instead, the numerical way of obtaining the inverse solution is to iteratively minimize the difference between the measured diffusion photon density data, **Φ***<sup>M</sup>* , around the tissue and the calculated model data, **Φ***<sup>C</sup>* , from solving the forward problem with the current estimated optical properties. This data-model misfit difference is typically defined as follows,

$$\mathcal{X}^2 = \sum\_{i=1}^{N\_M} \left[ \Phi\_i^{\mathbb{C}} - \Phi\_i^{\mathbb{M}} \right]^2 \tag{2.3}$$

where *NM* is the number of measurements.

By means of the first order Taylor series to expand **Φ** , one can get Eq. (2.4),

$$
\left(\boldsymbol{\Phi}^{\mathcal{M}}\right) \approx \left(\boldsymbol{\Phi}^{\mathcal{C}}\right) + \left[\frac{\partial \boldsymbol{\Phi}^{\mathcal{C}}}{\partial \boldsymbol{\mu}\_{a}}\right] \left(\Delta \boldsymbol{\mu}\_{a}\right) + \left[\frac{\partial \boldsymbol{\Phi}^{\mathcal{C}}}{\partial \mathbf{x}}\right] \left(\Delta \mathbf{x}\right),\tag{2.4}
$$

Image Restoration for Long-Wavelength Imaging Systems 235

p.s.f., and a user-defined weight (α). Ψ can have various forms derived from the different algorithms. As known, there are several approaches to enhancing image quality including non-iterative restoration algorithms such as a Gaussian filter and non-linear iterative

Non-iterative restoration algorithms are described in this sub-section such as the inverse and Wiener filters usually recovering the spatial-frequencies below the diffraction limit.

Inverse Filter = <sup>1</sup>

However, Eq. (3.4) is not able to be directly implemented; usually, one uses a so called

Pseudo-inverse Filter = <sup>1</sup>

Wiener Filter =

*H* 

n f

\* [ /]

*H H* 

2

where *H* is the modulation transfer function (MTF) of p.s.f.; the superscript asterisk (\*) denotes the complex conjugate; [n/f], the ratio of noise-to-signal. n and f represent the power spectral densities for noise and the true images, respectively. Apparently, applying the Wiener filter to the restoration problem has to know the power spectral densities for the noise and the original image (or more precisely, their ratio). Unfortunately, this *a priori*  knowledge is not available in most cases. Nevertheless, the noise-to-signal ratio (NSR), [n/f], is usually approximated by a small constant ε. In such a case, the Wiener filter

> 2 *H* \* *H*

Wiener filtering achieves a compromise between the improvement obtained by boosting the amplitude of spatial-frequency coefficients up to the diffraction limit and the degradation that occurs because of the noise amplification of the inverse filtering. Noise propagation tends to be reduced by the convolution with p.s.f.; this has a smoothing effect in the result.

This technique is briefly described here; further, a more detailed description of the implementation of this algorithm can be found in the literature [Kundur & Hatzinakos, 1998].

This fact reveals that Wiener filtering is more immune to noise than inverse filtering.

*H* (3.4)

(3.5)

(3.6)

(3.7)

algorithms such as Poisson maximum *a posteriori* superresolution algorithm.

Filters in the Fourier domain are respectively given by the following expressions:

pseudo-inverse filter with a small constant ε as below.

Wiener filter is described as Eq. (3.6) in the following.

**3.1 Non-iterative methods** 

becomes

**3.2 Iterative methods** 

**3.2.1 Recursive wiener filter** 

since the goal is to reach **Φ***<sup>M</sup>* from the current **Φ***<sup>C</sup>* , and, thus, **Φ***<sup>M</sup>* and **Φ***<sup>C</sup>* have been used in the left and right parts of Eq. (2.4), respectively. As well, the vector **μ***<sup>a</sup>* and **κ** denote the updates respectively for *a* and with dimension *NN* , the number of total nodes in the finite element mesh, and the dimension of the matrices *<sup>C</sup> <sup>a</sup>* **Φ μ** or *<sup>C</sup>* **Φ κ** is *N N <sup>M</sup> <sup>N</sup>* . From Eq. (2.4), the inverse problem in DOT can be formulated as

$$
\left\lfloor \frac{\partial \mathbf{\bar{O}}^{\mathbb{C}}}{\partial \boldsymbol{\upmu}\_{a}} \right\rfloor \xrightarrow[\widehat{\mathcal{O}}\mathbf{K}]{} \left\lfloor \frac{\Lambda \mathbf{\bar{\mu}}\_{a}}{\Delta \mathbf{x}} \right\rfloor = \left(\boldsymbol{\upPhi}^{\mathcal{M}} - \boldsymbol{\upPhi}^{\mathbb{C}}\right),
\tag{2.5}
$$

or simply denoted as **Jχ Φ** , where *C C <sup>a</sup>* **<sup>J</sup> Φμ Φκ** is the Jacobian matrix, the rate of change of model data with respect to optical parameters.

However, solving this linearized inverse problem from Eq. (2.5) usually runs into difficulty with an ill-conditioned problem which typically happens as the number of model parameters increases, so as to solve the inverse problem by means of regularization to remedy such a drawback.

#### **2.4 Remark**

In this section, we have explained a common imaging system which includes the operation of convolution, support area, sampling, and noise as well as a medical imaging system of which the optical-property images are formed with the reconstruction algorithm from 1D signals.

### **3. Image restoration algorithms**

This section will discuss non-iterative, iterative and statistical methods; in addition, regularization is also used frequently in image restoration algorithms. More descriptions are explained in the following.

As known, the image degradation is basically modelled as

$$
\mathbf{g} = f \otimes \mathbf{h} + \mathbf{n} \tag{3.1}
$$

where *f* is the original scene, *h* is the point-spread function (p.s.f.) of the imaging system, is the convolution operator, *n* is the noise, and *g* is the corrupted image.

Generally, the non-linear iterative restoration algorithms (Archer & Titterington, 1995; Hunt, 1994; Meinel, 1986; Singh et al., 1986; Stewart & Durrani, 1986) to enhance image quality by restoring the high frequency spectrum of the corrupted images can be simply modelled as the following form:

$$f\_n \sim f\_{n-1} + \Delta f\_n \tag{3.2}$$

$$
\Delta f\_n = \Psi(f\_{n-1}, \mathbf{g}, h, a) \tag{3.3}
$$

where the subscript n is the n-th iteration, Eqs. (3.2) and (3.3) represent that a new update (*f*n) is equal to a previous one (*f*n-1) plus an updating increment ( *<sup>n</sup> f* ). Furthermore, the update ( *<sup>n</sup> f* ) is related to the function (Ψ) of the previous update, the corrupted image, p.s.f., and a user-defined weight (α). Ψ can have various forms derived from the different algorithms. As known, there are several approaches to enhancing image quality including non-iterative restoration algorithms such as a Gaussian filter and non-linear iterative algorithms such as Poisson maximum *a posteriori* superresolution algorithm.

### **3.1 Non-iterative methods**

234 Image Restoration – Recent Advances and Applications

since the goal is to reach **Φ***<sup>M</sup>* from the current **Φ***<sup>C</sup>* , and, thus, **Φ***<sup>M</sup>* and **Φ***<sup>C</sup>* have been used in the left and right parts of Eq. (2.4), respectively. As well, the vector **μ***<sup>a</sup>* and **κ**

*C C <sup>a</sup> M C*

However, solving this linearized inverse problem from Eq. (2.5) usually runs into difficulty with an ill-conditioned problem which typically happens as the number of model parameters increases, so as to solve the inverse problem by means of regularization to

In this section, we have explained a common imaging system which includes the operation of convolution, support area, sampling, and noise as well as a medical imaging system of which the optical-property images are formed with the reconstruction algorithm from 1D signals.

This section will discuss non-iterative, iterative and statistical methods; in addition, regularization is also used frequently in image restoration algorithms. More descriptions are

where *f* is the original scene, *h* is the point-spread function (p.s.f.) of the imaging system,

Generally, the non-linear iterative restoration algorithms (Archer & Titterington, 1995; Hunt, 1994; Meinel, 1986; Singh et al., 1986; Stewart & Durrani, 1986) to enhance image quality by restoring the high frequency spectrum of the corrupted images can be simply modelled as

<sup>1</sup> ( ,,, ) *n n f f gh*

where the subscript n is the n-th iteration, Eqs. (3.2) and (3.3) represent that a new update (*f*n) is equal to a previous one (*f*n-1) plus an updating increment ( *<sup>n</sup> f* ). Furthermore, the update ( *<sup>n</sup> f* ) is related to the function (Ψ) of the previous update, the corrupted image,

,

**μ κ <sup>κ</sup>** (2.5)

*<sup>a</sup>* **<sup>J</sup> Φμ Φκ** is the Jacobian matrix, the

*g f hn* (3.1)

*nn n* ~ <sup>1</sup> *f f f* (3.2)

(3.3)

**Φ κ** is *N N <sup>M</sup> <sup>N</sup>* . From Eq. (2.4), the inverse problem in DOT can be formulated as

 **Φ Φ <sup>μ</sup> Φ Φ**

with dimension *NN* , the number of total

*<sup>a</sup>* **Φ μ** or

*a* and

*a*

or simply denoted as **Jχ Φ** , where *C C*

As known, the image degradation is basically modelled as

is the convolution operator, *n* is the noise, and *g* is the corrupted image.

rate of change of model data with respect to optical parameters.

nodes in the finite element mesh, and the dimension of the matrices *<sup>C</sup>*

denote the updates respectively for

remedy such a drawback.

explained in the following.

the following form:

**3. Image restoration algorithms** 

**2.4 Remark** 

*<sup>C</sup>*

Non-iterative restoration algorithms are described in this sub-section such as the inverse and Wiener filters usually recovering the spatial-frequencies below the diffraction limit. Filters in the Fourier domain are respectively given by the following expressions:

$$\text{Inverse Filter} = \frac{1}{H} \tag{3.4}$$

However, Eq. (3.4) is not able to be directly implemented; usually, one uses a so called pseudo-inverse filter with a small constant ε as below.

$$\text{PSendo-inverse Filter} = \frac{1}{H + \varepsilon} \tag{3.5}$$

Wiener filter is described as Eq. (3.6) in the following.

$$\text{Wiener Filter} = \frac{H^\*}{\left|H\right|^2 + \left[\phi\_\mathbf{n}/\phi\_\mathbf{f}\right]} \tag{3.6}$$

where *H* is the modulation transfer function (MTF) of p.s.f.; the superscript asterisk (\*) denotes the complex conjugate; [n/f], the ratio of noise-to-signal. n and f represent the power spectral densities for noise and the true images, respectively. Apparently, applying the Wiener filter to the restoration problem has to know the power spectral densities for the noise and the original image (or more precisely, their ratio). Unfortunately, this *a priori*  knowledge is not available in most cases. Nevertheless, the noise-to-signal ratio (NSR), [n/f], is usually approximated by a small constant ε. In such a case, the Wiener filter becomes

$$\frac{H^\*}{\left|H\right|^2 + \varepsilon} \tag{3.7}$$

Wiener filtering achieves a compromise between the improvement obtained by boosting the amplitude of spatial-frequency coefficients up to the diffraction limit and the degradation that occurs because of the noise amplification of the inverse filtering. Noise propagation tends to be reduced by the convolution with p.s.f.; this has a smoothing effect in the result. This fact reveals that Wiener filtering is more immune to noise than inverse filtering.

#### **3.2 Iterative methods**

#### **3.2.1 Recursive wiener filter**

This technique is briefly described here; further, a more detailed description of the implementation of this algorithm can be found in the literature [Kundur & Hatzinakos, 1998].

Image Restoration for Long-Wavelength Imaging Systems 237

1

*f h* 

*C* can be regard as the correction term during the iterative restoration process. In terms of the operation of the Poisson MAP, it is an iterative algorithm where successive estimate of the restored image is obtained by multiplication of the current estimate by a quantity close to one. The quantity close to one is a function of the detected image divided by a convolution of the current estimate with p.s.f.. Indeed, one can replace the exponential in Eq. (3.12) by the first order approximation ex ~ 1+x because of low contrast in a blurred

> 1 ~ 1 1 *n*

*f h*

1 1 1 1 ~ 1 *nn n n n*

Equation (3.14) shows that the Poisson MAP superresolution is consistent with Eq. (3.2). Experience reveals that when implemented for simple point objects, the Poisson MAP algorithm is able to expand the bandwidth much more than done for more complex objects and the Poisson MAP superresolution algorithm requires hundreds of iterations for a final

Following that, the Poisson MAP can be improved by itself by operating upon the edge map with a re-blurring technique; that is, the *g* and *f*n-1 of the Poisson MAP are replaced by the corresponding gradients of the *g* ⊗ *h* and *f*n-1 along with the integrated p.s.f. (*h* ⊗ *h*).

> 1 ( )' ( )' ( 1( ) ( )' ( ) *n n n g h <sup>f</sup> <sup>f</sup> h h f hh*

Thus, the final hyper-resolved image *f* can be obtained by integrating (*f*n)'. The whole process of this improved Poisson MAP includes re-blurring, differentiation, restoration, integration, and then correction for a DC offset. More details concerning this algorithm can

Regularization presents a very general methodology for image restoration. The main technique of a regularization procedure is to transform this ill-posed problem into a wellposed one. Roughly speaking, restoration problem with regularization comes down to the

1

be found in the author's previous work [Pan, 2003].

minimization problem [Chen et al., 2000; Landi, 2007].

*<sup>g</sup> <sup>f</sup> <sup>f</sup> hf f f h*

 

*n*

*n*

signal/image to achieve Eq. (3.13).

solution.

**3.3.2 Improved P-MAP** 

**3.4 Regularization** 

Mathematically, it is shown that

Thus, Eq. (3.11) can approach to Eq. (3.14).

1

*C exp* (3.12)

*C* (3.13)

*)'exp* (3.15)

*<sup>f</sup>* (3.14)

*<sup>g</sup> <sup>h</sup>*

*<sup>g</sup> <sup>h</sup>*

 

Briefly, such a recursive Wiener-like filtering operation in the Fourier domain can be expressed as Eqs. (3.8) and (3.9).

$$
\hat{H}\_n = \frac{G \times \hat{F}\_{n-1}^\*}{\left| \hat{F}\_{n-1} \right|^2 + \alpha \left/ \left| \hat{H}\_{n-1} \right|^2 \right.} \tag{3.8}
$$

$$\hat{F}\_n = \frac{\mathbf{G} \times \hat{H}\_{n-1}^\*}{\left| \hat{H}\_{n-1} \right|^2 + \alpha \left/ \left| \hat{F}\_{n-1} \right|^2 \right.} \tag{3.9}$$

The real constant α represents the energy of the additive noise and is determined by prior knowledge of the noise contamination level, if available. The algorithm is run for a specified number of iterations or until the estimates begin to converge. The method is popular for its low computational complexity. The major drawback of the method is its lack of reliability. The uniqueness and convergence properties are, as yet, uncertain.

#### **3.2.2 Lucy-Richardson method**

The Richardson–Lucy algorithm, also known as Lucy–Richardson deconvolution, is an iterative procedure for recovering a latent image that has been blurred by a known point spread function.

The Richardson-Lucy (RL) algorithm has been widely used for the data from astronomical imaging. The RL algorithm (Richardson, 1972; Lucy, 1974) generates a restored image through an iterative method, which is derived using a Bayesian statistical approach to guess the original image (*f* ), to convolute it (*f*n-1) with the p.s.f. (*h*) and to compare the result with the real image (*g*). Usually the guessed image for the first iteration is the blurred image. It uses such an iterative approach:

$$\,\_{1}f\_{n} = f\_{n-1} \left[ \frac{\mathcal{g}}{f\_{n-1} \otimes h} \otimes h^\* \right] \tag{3.10}$$

#### **3.3 Statistical methods**

#### **3.3.1 Poisson MAP algorithm**

The Poisson MAP superresolution algorithm begins with Bayes' law associated with Poisson models for the statistics of image and object to estimate the object by finding the maximum probability on the object (*f*) given the image (g). Mathematically, the Poisson MAP (Hunt & Sementilli, 1992) is given by

$$f\_n = f\_{n-1} \exp\left[\left(\frac{\mathcal{g}}{f\_{n-1} \otimes h} - 1\right) \* h\right] = f\_{n-1} \mathcal{C} \tag{3.11}$$

where represents a convolution; \*, a correlation; *nf* , the restored signal/image; g is the blurred signal/image; *h*, p.s.f.; 0*f* , the initial guess signal/image; subscript *n*, the iteration number. Here,

236 Image Restoration – Recent Advances and Applications

Briefly, such a recursive Wiener-like filtering operation in the Fourier domain can be

ˆ ˆ /

*F H* 

ˆ ˆ / *<sup>n</sup> <sup>n</sup> <sup>n</sup>*

*H F* 

The real constant α represents the energy of the additive noise and is determined by prior knowledge of the noise contamination level, if available. The algorithm is run for a specified number of iterations or until the estimates begin to converge. The method is popular for its low computational complexity. The major drawback of the method is its lack of reliability.

The Richardson–Lucy algorithm, also known as Lucy–Richardson deconvolution, is an iterative procedure for recovering a latent image that has been blurred by a known point

The Richardson-Lucy (RL) algorithm has been widely used for the data from astronomical imaging. The RL algorithm (Richardson, 1972; Lucy, 1974) generates a restored image through an iterative method, which is derived using a Bayesian statistical approach to guess the original image (*f* ), to convolute it (*f*n-1) with the p.s.f. (*h*) and to compare the result with the real image (*g*). Usually the guessed image for the first iteration is the blurred image. It

1

*n n*

1

*n <sup>g</sup> <sup>f</sup> <sup>h</sup> f h* 

The Poisson MAP superresolution algorithm begins with Bayes' law associated with Poisson models for the statistics of image and object to estimate the object by finding the maximum probability on the object (*f*) given the image (g). Mathematically, the Poisson MAP (Hunt &

1 1

1 1 *n n <sup>n</sup> n <sup>g</sup> <sup>f</sup> <sup>f</sup> h fC f h* 

where represents a convolution; \*, a correlation; *nf* , the restored signal/image; g is the blurred signal/image; *h*, p.s.f.; 0*f* , the initial guess signal/image; subscript *n*, the iteration

*n n*

*G F*

<sup>ˆ</sup> . <sup>ˆ</sup>

1

<sup>ˆ</sup> . <sup>ˆ</sup>

*G H <sup>F</sup>*

*n*

*H*

The uniqueness and convergence properties are, as yet, uncertain.

1 \* 2 2 1 1

(3.8)

(3.9)

*n*

\* 1 2 2

1

\*

*f* (3.10)

*exp* (3.11)

*n*

expressed as Eqs. (3.8) and (3.9).

**3.2.2 Lucy-Richardson method** 

uses such an iterative approach:

**3.3 Statistical methods** 

**3.3.1 Poisson MAP algorithm** 

Sementilli, 1992) is given by

number. Here,

spread function.

$$C = \exp\left[\left(\frac{\mathcal{g}}{f\_{n-1}\otimes h} - 1\right) \* h\right] \tag{3.12}$$

*C* can be regard as the correction term during the iterative restoration process. In terms of the operation of the Poisson MAP, it is an iterative algorithm where successive estimate of the restored image is obtained by multiplication of the current estimate by a quantity close to one. The quantity close to one is a function of the detected image divided by a convolution of the current estimate with p.s.f.. Indeed, one can replace the exponential in Eq. (3.12) by the first order approximation ex ~ 1+x because of low contrast in a blurred signal/image to achieve Eq. (3.13).

$$C \sim 1 + \left[ \left( \frac{\mathcal{g}}{f\_{n-1} \otimes h} - 1 \right) \* h \right] \tag{3.13}$$

Thus, Eq. (3.11) can approach to Eq. (3.14).

$$f\_n \sim f\_{n-1} + f\_{n-1} \left[ \left( \frac{\mathcal{S}}{f\_{n-1} \otimes h} - 1 \right) \* h \right] = f\_{n-1} + \Delta f\_n \tag{3.14}$$

Equation (3.14) shows that the Poisson MAP superresolution is consistent with Eq. (3.2). Experience reveals that when implemented for simple point objects, the Poisson MAP algorithm is able to expand the bandwidth much more than done for more complex objects and the Poisson MAP superresolution algorithm requires hundreds of iterations for a final solution.

#### **3.3.2 Improved P-MAP**

Following that, the Poisson MAP can be improved by itself by operating upon the edge map with a re-blurring technique; that is, the *g* and *f*n-1 of the Poisson MAP are replaced by the corresponding gradients of the *g* ⊗ *h* and *f*n-1 along with the integrated p.s.f. (*h* ⊗ *h*). Mathematically, it is shown that

$$\mathbf{r}(f\_n)' = (f\_{n-1})' \exp\left[ \left( \frac{(\mathbf{g} \otimes h)'}{(f\_{n-1})' \otimes (h \otimes h)} - 1 \right) \* (h \otimes h) \right] \tag{3.15}$$

Thus, the final hyper-resolved image *f* can be obtained by integrating (*f*n)'. The whole process of this improved Poisson MAP includes re-blurring, differentiation, restoration, integration, and then correction for a DC offset. More details concerning this algorithm can be found in the author's previous work [Pan, 2003].

#### **3.4 Regularization**

Regularization presents a very general methodology for image restoration. The main technique of a regularization procedure is to transform this ill-posed problem into a wellposed one. Roughly speaking, restoration problem with regularization comes down to the minimization problem [Chen et al., 2000; Landi, 2007].

Image Restoration for Long-Wavelength Imaging Systems 239

*n n* ~ hp *f* 

of the symmetry of p.s.f.) have been introduced. Furthermore, *h*hp can be designed as a highpass filter such as *h*lp1 – *h*lp2 in general or *δ*- *h*lp in the extreme case where *h*lp1,2 are low-pass

into the left part of Eq. (3.21) and use the projection of the right pat in Eq. (3.21) on Δ*f* n for the purpose of true value invariance. Consequently, the new relation function, Eq. (3.23), can

( ) <sup>ˆ</sup> *n hp n*

*fh f <sup>f</sup> <sup>f</sup>* 

adopted for *h*lp1 and *h*lp2 in numerical simulation, respectively. Equations (3.23)–(3.25) show that the restored signal/image can be obtained from the increment iteratively updated using the projection of the high frequency spectra of the increment. As discussed, *h*hp is defined as the difference of a delta function and a Gaussian function; in addition, an edge operator like a Laplacian operator defined as Eq. (3.26) is adopted for *h*hp in the following experimental

*n*

in Eq. (3.25), normally, is equal to *h* but it is chosen as a user-guess p.s.f. when

In this section, we have established a framework of image restoration/superresolution including (pseudo) inverse filter, Wiener filter, recursive Wiener filter, Lucy-Richardson method, Poisson MAP algorithm, and improved P-MAP algorithm. Of restoring image

*n*

operator =

*h f* (3.21)

*nnn* <sup>1</sup> *fff* (3.22)

<sup>1</sup> <sup>ˆ</sup> *nn n ff f* (3.23)

*n n* <sup>1</sup> *f gf <sup>h</sup>* (3.25)

, where a delta function and a Gaussian function

(3.24)

(3.26)

*h* - *h* ( ˆ ˆ *h h* \* and *h h* \* because

Following that, an approximate equation is obtained as Eq. (3.21)

where 1 1 <sup>ˆ</sup> *nn n f gf h or g f <sup>h</sup>* , *α ~ 1/λ* and *h*hp = <sup>ˆ</sup>

be achieved for our novel method and expressed as

h is unknown. Here, *h*hp is chosen as *δ* – *h*

filters.

where

Note that *h*

verification.

**3.5 Remark** 

Subsequently, we substitute

In our real life, one cannot get the whole blurred and noisy images but only can get part of blurred and noisy images because of the limited support size. According to the part of blurred and noisy image, ones want to reconstruct an approximate true image by deconvolving the part of blurred and noisy image. Thus, noise (*n*) in general meaning should include both additive noise (*n*add) and the effect of the limited support size (*n*limited) at least.

Fig. 3.1. A schematic diagram of forming a real image and proposing our algorithm.

To develop the novel algorithm with regularization, we plot a schematic diagram, Fig. 3.1, to show the mechanism of the concept proposed here and thus define the following functions, Equation (3.16).

$$\left\|Q\_1 = \left\|g - f\_{n-1}\otimes h\right\|^2 \le n^2 \text{ and } Q\_2 = \left\|g - f\_{n-1}\otimes \hat{h}\right\|^2 \tag{3.16}$$

Normally, Q1 is usually used with a true *h* which is, however, not known and optimal, whereas Q2 is expected to be used with an ˆ *h* , which is supposed to be optimal in practice. Here, Q2 is proposed for the purpose of reducing the error energy coming from noise and ringing artifacts while only Q1 is considered. Thus, a new objective function combines Q1 with Q2, and its regularization term is <sup>2</sup> *<sup>n</sup> f* ; it is approaching to null when iteration is increasing. Finally, we define an objective function as Eq. (3.17)

$$Q \equiv \|Q\_1 - Q\_2 + \mathcal{X}\| \Delta f\_n\|^2 \tag{3.17}$$

where *λ* is the regularization parameter and then minimize Eq. (3.17) with respect to *f*n-1; i. e. ,

$$\min\_{f\_{n-1}} \{ Q \: \} = \; 0 \tag{3.18}$$

thus,

$$\min\_{f\_{n-1}} \{ \left. Q\_1 - Q\_2 + \lambda \left\| \Delta f\_n \right\|^2 \} \right\} = 0 \tag{3.19}$$

Then, we can find Eq. (3.20)

$$\mathcal{Z}(\hat{h})^{\*} \otimes (\mathcal{g} - f\_{n-1} \otimes \hat{h}) - \mathcal{Z}h^{\*} \otimes (\mathcal{g} - f\_{n-1} \otimes h) - \mathcal{Z}\mathcal{A}(\Delta f\_{n}) = 0 \tag{3.20}$$

Following that, an approximate equation is obtained as Eq. (3.21)

$$
\Delta f\_n \simeq \alpha \, h\_{\rm hp} \otimes \Delta f\_n \tag{3.21}
$$

where 1 1 <sup>ˆ</sup> *nn n f gf h or g f <sup>h</sup>* , *α ~ 1/λ* and *h*hp = <sup>ˆ</sup> *h* - *h* ( ˆ ˆ *h h* \* and *h h* \* because of the symmetry of p.s.f.) have been introduced. Furthermore, *h*hp can be designed as a highpass filter such as *h*lp1 – *h*lp2 in general or *δ*- *h*lp in the extreme case where *h*lp1,2 are low-pass filters.

Subsequently, we substitute

$$
\Delta f\_n = f\_n - f\_{n-1} \tag{3.22}
$$

into the left part of Eq. (3.21) and use the projection of the right pat in Eq. (3.21) on Δ*f* n for the purpose of true value invariance. Consequently, the new relation function, Eq. (3.23), can be achieved for our novel method and expressed as

$$f\_n := f\_{n-1} + \Delta \hat{f}\_n \tag{3.23}$$

where

238 Image Restoration – Recent Advances and Applications

In our real life, one cannot get the whole blurred and noisy images but only can get part of blurred and noisy images because of the limited support size. According to the part of blurred and noisy image, ones want to reconstruct an approximate true image by deconvolving the part of blurred and noisy image. Thus, noise (*n*) in general meaning should include both

additive noise (*n*add) and the effect of the limited support size (*n*limited) at least.

Fig. 3.1. A schematic diagram of forming a real image and proposing our algorithm.

<sup>2</sup> <sup>2</sup> *<sup>Q</sup>*1 1 *<sup>n</sup> g f h n* and

functions, Equation (3.16).

Then, we can find Eq. (3.20)

thus,

whereas Q2 is expected to be used with an ˆ

with Q2, and its regularization term is <sup>2</sup>

increasing. Finally, we define an objective function as Eq. (3.17)

To develop the novel algorithm with regularization, we plot a schematic diagram, Fig. 3.1, to show the mechanism of the concept proposed here and thus define the following

*f h* ⊕ *g*

ˆ *h* *n*add

*n*limited

Normally, Q1 is usually used with a true *h* which is, however, not known and optimal,

Here, Q2 is proposed for the purpose of reducing the error energy coming from noise and ringing artifacts while only Q1 is considered. Thus, a new objective function combines Q1

1 2 *Q QQ <sup>n</sup>*

where *λ* is the regularization parameter and then minimize Eq. (3.17) with respect to *f*n-1; i. e. ,

<sup>2</sup> min{ 1 2 } 0

1 1 ˆ ˆ 2( ) ( ) 2 ( ) 2 ( ) 0 *n nn h gf h h gf h f*

*<sup>n</sup> <sup>f</sup> QQ f*

1 min{ } 0

*nf Q* 

1

*n*

\* \*

2 2 1 <sup>ˆ</sup> *Q gf h <sup>n</sup>* (3.16)

*h* , which is supposed to be optimal in practice.

*<sup>n</sup> f* ; it is approaching to null when iteration is

*f* (3.17)

(3.18)

(3.20)

(3.19)

2

( ) <sup>ˆ</sup> *n hp n n n fh f <sup>f</sup> <sup>f</sup>* (3.24)

$$
\Delta \mathbf{f}\_n = \mathbf{g} - f\_{n-1} \otimes \tilde{h} \tag{3.25}
$$

Note that *h* in Eq. (3.25), normally, is equal to *h* but it is chosen as a user-guess p.s.f. when h is unknown. Here, *h*hp is chosen as *δ* – *h* , where a delta function and a Gaussian function adopted for *h*lp1 and *h*lp2 in numerical simulation, respectively. Equations (3.23)–(3.25) show that the restored signal/image can be obtained from the increment iteratively updated using the projection of the high frequency spectra of the increment. As discussed, *h*hp is defined as the difference of a delta function and a Gaussian function; in addition, an edge operator like a Laplacian operator defined as Eq. (3.26) is adopted for *h*hp in the following experimental verification.

$$\begin{aligned} \text{operator} &= \begin{bmatrix} 0 & -\frac{1}{4} & 0\\ -\frac{1}{4} & 1 & -\frac{1}{4} \\ 0 & -\frac{1}{4} & 0 \end{bmatrix} \end{aligned} \tag{3.26}$$

#### **3.5 Remark**

In this section, we have established a framework of image restoration/superresolution including (pseudo) inverse filter, Wiener filter, recursive Wiener filter, Lucy-Richardson method, Poisson MAP algorithm, and improved P-MAP algorithm. Of restoring image

Image Restoration for Long-Wavelength Imaging Systems 241

In this section, we address an approach to simultaneous image interpolation and smoothing by exploiting the probability filter [Pan & Lettington, 1998] coupled with a pyramidal decomposition, thereby extending the conventional applications of the probability filter originally designed for noise removal. Then, the improved Poisson maximum *a posteriori*  (MAP) superresolution [Pan & Lettington, 1999; Pan, 2002] is performed to reconstruct the high spatial-frequency spectrum of the interpolated image. Thus, the hybrid scheme shown in Fig. 4.1(b) is proposed for enhancing the spatial and the spatial-frequency resolutions of a down-sampled image. For more detailed description and examples, readers can refer to the previous work [Pan, 2006]. To illustrate the performance of this proposed scheme, comparisons are shown among the superresolution coupled with different interpolators as

Fig. 4.3. Demonstration of hyper-resolution for a single down-sampled noisy gray-level image. (a) Down-sampled image, (b) hyperresolved image incorporated with bilinear interpolation, (c) hyper-resolved image incorporated with cubic spline interpolation, and (d)

As known, restoration/superresolution or the reconstruction of optical-property images with an iteration procedure is usually computed off-line and computationally expensive. Most of studies, however, focused mainly on improving the spatial and spatial-frequency resolutions. If a real-time resolution processing is required, dedicated reconstruction hardwares or specialized computers are mandatory. Moreover, fast reconstruction algorithms should also be considered to reduce the computation load. It is worth emphasizing that our proposed method can reduce computation time with the regularization term which is designed on the viewpoint of the update characteristics in the iteration procedure but not utilizing any spatial/spectral *a priori* knowledge or constraints; some results can be found in the author's work [M.-Cheng & M.-Chun Pan, 2010]. Here, we

hyper-resolved image incorporated with probability-filtering-based interpolation.

the following examples, Fig. 4.2 and Fig. 4.3.

**4.2 Rapid convergence** 

quality and reducing ringing artifacts, the error-energy-reduction-based regularization algorithm has been proposed here for long-wavelength imaging systems as well.

## **4. Further consideration on image restoration**

In this section, the topics of improvement of spatial resolution, rapid convergence, and inverse pitfall for image restoration are described.

## **4.1 Improvement of spatial resolution**

Usually, hyper-resolution of a noisy image is considered as an interpolation followed with restoration/superresolution; generally, the procedure for processing noisy images is shown in Fig. 4.1(a), that is, noise removal, interpolation, and then superresolution, whereas the proposed scheme is dealing with interpolation and noise removal simultaneously, as shown in Fig. 4.1(b).

Fig. 4.1. The block diagram of hyper-resolution for a noisy image. (a) Conventional approach and (b) proposed approach.

Fig. 4.2. Demonstration of hyper-resolution for a single down-sampled gray-level image. (a) Down-sampled image, (b) hyper-resolved image incorporated with bilinear interpolation, (c) hyper-resolved image incorporated with cubic spline interpolation, and (d) hyperresolved image incorporated with probability-filtering-based interpolation.

240 Image Restoration – Recent Advances and Applications

quality and reducing ringing artifacts, the error-energy-reduction-based regularization

In this section, the topics of improvement of spatial resolution, rapid convergence, and

Usually, hyper-resolution of a noisy image is considered as an interpolation followed with restoration/superresolution; generally, the procedure for processing noisy images is shown in Fig. 4.1(a), that is, noise removal, interpolation, and then superresolution, whereas the proposed scheme is dealing with interpolation and noise removal simultaneously, as shown

**Image** 

**superresolution Hyper-resolved** 

**image** 

Fig. 4.1. The block diagram of hyper-resolution for a noisy image. (a) Conventional

**Image interpola**tion

Fig. 4.2. Demonstration of hyper-resolution for a single down-sampled gray-level image. (a) Down-sampled image, (b) hyper-resolved image incorporated with bilinear interpolation, (c) hyper-resolved image incorporated with cubic spline interpolation, and (d) hyper-

resolved image incorporated with probability-filtering-based interpolation.

algorithm has been proposed here for long-wavelength imaging systems as well.

**4. Further consideration on image restoration** 

inverse pitfall for image restoration are described.

**Noise removal** 

**Interpolation based on Probability filter** 

**4.1 Improvement of spatial resolution** 

approach and (b) proposed approach.

**(a)** 

 **(b)** 

in Fig. 4.1(b).

**Noisy image**  In this section, we address an approach to simultaneous image interpolation and smoothing by exploiting the probability filter [Pan & Lettington, 1998] coupled with a pyramidal decomposition, thereby extending the conventional applications of the probability filter originally designed for noise removal. Then, the improved Poisson maximum *a posteriori*  (MAP) superresolution [Pan & Lettington, 1999; Pan, 2002] is performed to reconstruct the high spatial-frequency spectrum of the interpolated image. Thus, the hybrid scheme shown in Fig. 4.1(b) is proposed for enhancing the spatial and the spatial-frequency resolutions of a down-sampled image. For more detailed description and examples, readers can refer to the previous work [Pan, 2006]. To illustrate the performance of this proposed scheme, comparisons are shown among the superresolution coupled with different interpolators as the following examples, Fig. 4.2 and Fig. 4.3.

Fig. 4.3. Demonstration of hyper-resolution for a single down-sampled noisy gray-level image. (a) Down-sampled image, (b) hyperresolved image incorporated with bilinear interpolation, (c) hyper-resolved image incorporated with cubic spline interpolation, and (d) hyper-resolved image incorporated with probability-filtering-based interpolation.

## **4.2 Rapid convergence**

As known, restoration/superresolution or the reconstruction of optical-property images with an iteration procedure is usually computed off-line and computationally expensive. Most of studies, however, focused mainly on improving the spatial and spatial-frequency resolutions. If a real-time resolution processing is required, dedicated reconstruction hardwares or specialized computers are mandatory. Moreover, fast reconstruction algorithms should also be considered to reduce the computation load. It is worth emphasizing that our proposed method can reduce computation time with the regularization term which is designed on the viewpoint of the update characteristics in the iteration procedure but not utilizing any spatial/spectral *a priori* knowledge or constraints; some results can be found in the author's work [M.-Cheng & M.-Chun Pan, 2010]. Here, we

Image Restoration for Long-Wavelength Imaging Systems 243

It is known that to solve Eq. (4.4) is an ill-posed problem. Tikhonov regularization is a method stabilizing the inverse problem through incorporating *a priori* assumptions to constraint the desired solution. It is able to convert an ill-posed problem into a well-posed one, and further to improve an ill-conditioned problem. The regularization term (penalty term) introduced in the process regularizes the problem and makes the update stable. It also strengthens the robustness of algorithm to noisy data with the adequate design of the regularization term. Generally, Tikhonov regularization is to optimize this ill-conditioned

subject to ( )

to be an energy bound. Applying Lagrange optimization technique, we seek a solution to

<sup>2</sup> *J* 

<sup>2</sup> min min *<sup>J</sup>*

2( ) 0 *<sup>T</sup>*

equals to zero. It is noted that the minus sign in Eq. (4.6), the objective function, corresponds to the regularization term proposed here as the term is constrained to an energy bound.

> <sup>2</sup> ( ) *L*

where *L* can be the identity matrix (*I*) or the discrete Laplacian matrix [Pogue et al., 1999;

<sup>1</sup> ( -) *T T*

   

where λ is referred to as the regularization parameter. A solution to Eq. (4.7) is given by

( ) <sup>2</sup> *T T J J J* 

   

, (4.7)

*E* (4.5)

(4.6)

, (4.8)

(4.9)

, but becomes an unconstrained one when λ

(4.10)

*JJ I J* . (4.11)

, and *E* is a quantity confining the constraint

<sup>2</sup> min *J*

where Eq. (4.9) is a constrained estimate of

**4.2.2 Constraints on the spatial domain** 

*J J* 

A constraint on the spatial domain can generally be expressed as

If *L* is the identity matrix (*I*), a solution to Eq. (4.9) is given by

*)* is a constraint on the estimate

the constrained objective function

problem as

where *(*

with the condition

and equivalently

Davis et al., 2007].

show how to speed up the computation to find an inverse solution for reconstructing optical-property images by using regularization with an iteration domain technique; similarly, this proposed method is capable of being applied to image restoration/superresolution for other imaging systems.

#### **4.2.1 Algorithm of rapid convergence**

Image reconstruction tasks contain forward modeling and inverse problem. The forward computation consists in obtaining the intensity out of a subject under investigation for a given source, and the initial-guess (or iterated result) on scattering and absorption coefficients. The inverse computation is to compute the scattering and absorption coefficients for a known light source and measured intensities in an iterative manner.

Since we utilize cw light illumination or DC data, the physical process of NIR light illuminating through a highly-scattering medium can be approximated by the steady-state diffusion equation

$$
\nabla \cdot D(\mathbf{r}) \nabla \Phi(\mathbf{r}) - \mu\_a(\mathbf{r}) \Phi(\mathbf{r}) = -S(\mathbf{r}) \,, \tag{4.1}
$$

where *S*( )**r** and ( )**r** denote the source and the intensity, respectively, as well as ( ) *<sup>a</sup>* **r** , *c* and *D*( )**r** are the absorption coefficient and the diffusion coefficient, respectively. For solving Eq. (4.1), the boundary condition, *D n Flux* ˆ , and finite element method are employed. Thus, the following discrete equations can be obtained [Paulsen and Jiang, 1995]

$$A\Phi = \mathbb{C} \ , \tag{4.2}$$

where *A* and *C* are matrices dependent on the optical properties and the source-detection locations, respectively. The forward solution, , can be explicitly evaluated by Eq. (4.2). Partially differentiating Eq. (4.2) with *<sup>D</sup>* and , respectively, yields

$$
\boldsymbol{\Phi}^{\uparrow} = -\boldsymbol{A}^{-1}\boldsymbol{A}^{\uparrow}\boldsymbol{\Phi} + \boldsymbol{A}^{-1}\boldsymbol{\mathsf{C}}^{\uparrow}.\tag{4.3}
$$

With an approximation to applying the Newton-Raphson method and ignoring higher order terms, we obtain

$$
\Delta \mathbb{X} = \Delta \Phi \tag{4.4}
$$

where the Jacobian matrix *J* denotes the matrix consisting of *<sup>b</sup> Dk* and *<sup>b</sup> l* , is the vector composed of *Dk* and *l*, and is the vector with differences between calculated intensities (*Φcal.*) and measured intensities (*Φmeas.*). Also, *Dk* for *k* = 1, 2, …, K and *μl* for *l* = 1, 2, …, L are the reconstruction parameters for the optical-property profile. The opticalproperty image reconstruction is actually a process of successively updating the distribution of optical coefficients so as to minimize the difference between measured intensities and computed ones from the forward process. More details can be found in [Paulsen and Jiang, 1995] where the Levenberg-Marquardt procedure was adopted to update the diffusion and absorption coefficients iteratively.

It is known that to solve Eq. (4.4) is an ill-posed problem. Tikhonov regularization is a method stabilizing the inverse problem through incorporating *a priori* assumptions to constraint the desired solution. It is able to convert an ill-posed problem into a well-posed one, and further to improve an ill-conditioned problem. The regularization term (penalty term) introduced in the process regularizes the problem and makes the update stable. It also strengthens the robustness of algorithm to noisy data with the adequate design of the regularization term. Generally, Tikhonov regularization is to optimize this ill-conditioned problem as

$$\min\_{\Delta \mathcal{X}} \left\| \left\| f \Delta \mathcal{X} - \Delta \Phi \right\| \right\|^2 \text{ subject to } \Psi(\Delta \mathcal{X}) \le E \tag{4.5}$$

where *()* is a constraint on the estimate , and *E* is a quantity confining the constraint to be an energy bound. Applying Lagrange optimization technique, we seek a solution to the constrained objective function

$$\mathbf{O} = \left\| \left[ \boldsymbol{\Delta} \boldsymbol{\chi} - \boldsymbol{\Delta} \boldsymbol{\Phi} \right] \right\|^2 - \boldsymbol{\lambda} \boldsymbol{\Psi} \tag{4.6}$$

with the condition

242 Image Restoration – Recent Advances and Applications

show how to speed up the computation to find an inverse solution for reconstructing optical-property images by using regularization with an iteration domain technique; similarly, this proposed method is capable of being applied to image

Image reconstruction tasks contain forward modeling and inverse problem. The forward computation consists in obtaining the intensity out of a subject under investigation for a given source, and the initial-guess (or iterated result) on scattering and absorption coefficients. The inverse computation is to compute the scattering and absorption coefficients for a known light source and measured intensities in an iterative manner.

Since we utilize cw light illumination or DC data, the physical process of NIR light illuminating through a highly-scattering medium can be approximated by the steady-state

> () () () () () **r r rr r** *D S*

and *D*( )**r** are the absorption coefficient and the diffusion coefficient, respectively. For

are employed. Thus, the following discrete equations can be obtained [Paulsen and Jiang,

where *A* and *C* are matrices dependent on the optical properties and the source-detection locations, respectively. The forward solution, , can be explicitly evaluated by Eq. (4.2).

With an approximation to applying the Newton-Raphson method and ignoring higher order

intensities (*Φcal.*) and measured intensities (*Φmeas.*). Also, *Dk* for *k* = 1, 2, …, K and *μl* for *l* = 1, 2, …, L are the reconstruction parameters for the optical-property profile. The opticalproperty image reconstruction is actually a process of successively updating the distribution of optical coefficients so as to minimize the difference between measured intensities and computed ones from the forward process. More details can be found in [Paulsen and Jiang, 1995] where the Levenberg-Marquardt procedure was adopted to update the diffusion and

*J*

 and

where *S*( )**r** and ( )**r** denote the source and the intensity, respectively, as well as ( )

*<sup>a</sup>* , (4.1)

*A C* , (4.2)

(4.4)

and *<sup>b</sup>*

*l* , 

, and finite element method

, respectively, yields

' 1' 1' *A A AC* . (4.3)

*Dk* 

is the vector with differences between calculated

*<sup>a</sup>* **r** , *c*

is the vector

restoration/superresolution for other imaging systems.

solving Eq. (4.1), the boundary condition, *D n Flux* ˆ

where the Jacobian matrix *J* denotes the matrix consisting of *<sup>b</sup>*

*l*, and

**4.2.1 Algorithm of rapid convergence** 

Partially differentiating Eq. (4.2) with *<sup>D</sup>*

diffusion equation

terms, we obtain

composed of

*Dk* and

absorption coefficients iteratively.

1995]

$$\min\_{\Delta \mathcal{X}} \{ \mathcal{O} \} = \min\_{\Delta \mathcal{X}} \left( \left\| \left\| \mathcal{J} \Delta \mathcal{X} - \Delta \Phi \right\| \right\|^2 - \mathcal{X} \Psi \right),\tag{4.7}$$

where λ is referred to as the regularization parameter. A solution to Eq. (4.7) is given by

$$
\lambda \mathcal{D}^T (\mathsf{J} \Delta \mathcal{X} - \Delta \Phi \ ) - \lambda \frac{\partial \Psi}{\partial \Delta \mathcal{X}} = 0 \,\,\,\,\tag{4.8}
$$

and equivalently

$$\left(\int^{T} \mathbf{J} \, d\boldsymbol{\chi} - \frac{\mathcal{A}}{2} \frac{\partial \boldsymbol{\Psi}}{\partial \boldsymbol{\Delta} \boldsymbol{\chi}}\right) = \boldsymbol{\int}^{T} \boldsymbol{\Delta} \boldsymbol{\Phi} \tag{4.9}$$

where Eq. (4.9) is a constrained estimate of , but becomes an unconstrained one when λ equals to zero. It is noted that the minus sign in Eq. (4.6), the objective function, corresponds to the regularization term proposed here as the term is constrained to an energy bound.

#### **4.2.2 Constraints on the spatial domain**

A constraint on the spatial domain can generally be expressed as

$$\mathbb{E}\left\|\Psi(\Delta\boldsymbol{\chi}) = \left\|\mathrm{L}\Delta\boldsymbol{\chi}\right\|^2\right\|\tag{4.10}$$

where *L* can be the identity matrix (*I*) or the discrete Laplacian matrix [Pogue et al., 1999; Davis et al., 2007].

If *L* is the identity matrix (*I*), a solution to Eq. (4.9) is given by

$$
\Delta \mathcal{X} = (\mathbf{J}^T \mathbf{J} - \mathcal{X}\mathbf{I})^{-1} \mathbf{J}^T \Delta \Phi \,. \tag{4.11}
$$

Image Restoration for Long-Wavelength Imaging Systems 245

In addition, as the shape of the histogram would be affected, it is smooth with a big value of γ and sharp with a small value of γ. Thus, Lorentzian distributed function can characterize the nature of Δ*χ* in the iterative process as the distribution from a smooth to a sharp

Lorentzian Distributed Function

small big

The ill-posed nature of inverse problems means that any restoration or reconstruction algorithm will have limitations on what images it can accurately reconstruct and that the images degrade with noise in the data. When developing a restoration or reconstruction algorithm it is usual to test it initially on simulated data. Moreover, the restoration or reconstruction algorithm typically incorporates a forward solver. A natural first test is to use the same forward model to generate simulated data with no simulated noise and to then find that the simulated data can be recovered fairly well. If one is fortunate enough to have a good data collection system and phantom, and someone skilled enough to make some accurate measurements with the system, one could then progress to attempting to reconstruct images from experimental data. However, more often the next stage is to test further with simulated data and it at this stage that one must take care not to cheat and commit a so-called inverse pitfall or inverse crime. Simply to say, inverse pitfall or inverse crime arises from the reason of 'limited for infinite', e.g., limited support area for infinite scenery, finite elements for continuous zone, or given noise for unknown noise. The best practice is to use a forward model independent of an inverse model. For example, in the case of a finite element forward model one would use a much finer mesh while a coarse

seen, it has a smooth distribution for a big γ and a sharp distribution as γ is small.

/ 

) at various γ. As can be

 

Fig. 4.4. Charts of the Lorentzian distributed functions ( <sup>2</sup> <sup>2</sup> ( )

distribution to be used as a constraint for the purpose of speeding up computation.

Histogram

**4.3 Inverse pitfall** 

mesh is used in the inverse model.

On the other hand, if *L* is the discrete Laplacian matrix, substituting Eq. (4.10) into Eq. (4.9), the corresponding solution is

$$
\Delta \mathbb{X} = \left(\int^T \mathbf{J} \cdot \mathbf{\mathcal{X}}^T \mathbf{L}\right)^{-1} \int^T \Delta \boldsymbol{\Phi} \,. \tag{4.12}
$$

Equation (4.11) is usually a primary inverse solution to optical-property image reconstruction, which is also Levenberg's contribution to the inverse problem; and Eq. (4.12) is a constrained inverse solution implemented to improve the quality of the reconstructed NIR DOT images, which is identical to Marquardt's work.

#### **4.2.3 Constraints on the iteration domain**

In NIR DOT, it is also crucial to accelerate the computation. But, up to now, speeding up the computation in the iteration domain has not been explored yet. Here we consider this issue through the use of a Lorentzian distributed function taking a natural logarithm computation as a constraint, i.e.

$$\Psi(\Delta \mathcal{X}) = \sum\_{p=1}^{K+L} \ell n \frac{\gamma/\pi}{\left(\Delta \mathcal{X}\right)\_p^2 + \gamma^2} \quad , \tag{4.13}$$

where *p* is the calculated nodes in the subject under investigation and *γ* is a user defined positive parameter. As can be seen, <sup>1</sup> ( ) ln( ) *p* , , meets the requirement of Eq.

(4.5). Performing the differentiation indicated in Eq. (4.9), we can obtain the solution in an iterative formality

$$(\Delta \mathbf{y})\_n = \left(\mathbf{J}^T \mathbf{J} + \frac{\lambda \mathbf{I}}{(\Delta \mathbf{y})\_{n-1}^2 + \boldsymbol{\gamma}\_n^2 \mathbf{I}}\right)^{-1} \mathbf{J}^T \Delta \boldsymbol{\Phi} \,. \tag{4.14}$$

For further inspection in Eqs. (4.13) and (4.14), as known, *μ<sup>a</sup>* and *D* are generally searched in a range of [10-3:10-1] mm-1and mm, respectively; and thus Δ*χ* is much smaller than a unit. It can be proven that even the use of the natural logarithm in the constraint Ψ(Δ*χ*) still makes it a positive and finite value. The other reason to use *ln* is because the regularization term in Eq. (4.14) still remains in a form of the Lorentzian distributed function derived from the constraint associated with the Lorentzian distributed function in Eq. (4.13).

The Lorentzian distributed function, as depicted in Fig. 4.4, is employed here owing to its following two characteristics:


244 Image Restoration – Recent Advances and Applications

On the other hand, if *L* is the discrete Laplacian matrix, substituting Eq. (4.10) into Eq. (4.9),

<sup>1</sup> (- ) *TT T*

Equation (4.11) is usually a primary inverse solution to optical-property image reconstruction, which is also Levenberg's contribution to the inverse problem; and Eq. (4.12) is a constrained inverse solution implemented to improve the quality of the reconstructed

In NIR DOT, it is also crucial to accelerate the computation. But, up to now, speeding up the computation in the iteration domain has not been explored yet. Here we consider this issue through the use of a Lorentzian distributed function taking a natural logarithm computation

> 1 / ( ) ( ) *K L*

 

() + ( ) *<sup>n</sup>*

constraint associated with the Lorentzian distributed function in Eq. (4.13).

*n*

smooth distribution in the beginning stage of iteration.

*p p n*

where *p* is the calculated nodes in the subject under investigation and *γ* is a user defined

*p*

(4.5). Performing the differentiation indicated in Eq. (4.9), we can obtain the solution in an

2 2

1

, (4.13)

, meets the requirement of Eq.

. (4.14)

 

,

2 2 1

 

*T T*

*n <sup>I</sup> J J <sup>J</sup> <sup>I</sup>* 

For further inspection in Eqs. (4.13) and (4.14), as known, *μ<sup>a</sup>* and *D* are generally searched in a range of [10-3:10-1] mm-1and mm, respectively; and thus Δ*χ* is much smaller than a unit. It can be proven that even the use of the natural logarithm in the constraint Ψ(Δ*χ*) still makes it a positive and finite value. The other reason to use *ln* is because the regularization term in Eq. (4.14) still remains in a form of the Lorentzian distributed function derived from the

The Lorentzian distributed function, as depicted in Fig. 4.4, is employed here owing to its

a. Lorentzian distributed function has a sharp peak with a long tail, describing the histogram distribution of Δ*χ*, many of Δ*χ* (~0) at its peak and a small rest of Δ*χ*

b. its histogram distribution can be further tuned with the parameter (γ) as iteration increasing. Related to the consideration in convergence, the updated quantity, Δ*χ*, decreases, ranging from the peak to the tail, as the iteration increases whereas it has a

 

*J J LL J* . (4.12)

 

NIR DOT images, which is identical to Marquardt's work.

positive parameter. As can be seen, <sup>1</sup> ( ) ln( )

**4.2.3 Constraints on the iteration domain** 

the corresponding solution is

as a constraint, i.e.

iterative formality

following two characteristics:

distributing along its long tail, and

In addition, as the shape of the histogram would be affected, it is smooth with a big value of γ and sharp with a small value of γ. Thus, Lorentzian distributed function can characterize the nature of Δ*χ* in the iterative process as the distribution from a smooth to a sharp distribution to be used as a constraint for the purpose of speeding up computation.

Lorentzian Distributed Function

Fig. 4.4. Charts of the Lorentzian distributed functions ( <sup>2</sup> <sup>2</sup> ( ) / ) at various γ. As can be seen, it has a smooth distribution for a big γ and a sharp distribution as γ is small.

#### **4.3 Inverse pitfall**

The ill-posed nature of inverse problems means that any restoration or reconstruction algorithm will have limitations on what images it can accurately reconstruct and that the images degrade with noise in the data. When developing a restoration or reconstruction algorithm it is usual to test it initially on simulated data. Moreover, the restoration or reconstruction algorithm typically incorporates a forward solver. A natural first test is to use the same forward model to generate simulated data with no simulated noise and to then find that the simulated data can be recovered fairly well. If one is fortunate enough to have a good data collection system and phantom, and someone skilled enough to make some accurate measurements with the system, one could then progress to attempting to reconstruct images from experimental data. However, more often the next stage is to test further with simulated data and it at this stage that one must take care not to cheat and commit a so-called inverse pitfall or inverse crime. Simply to say, inverse pitfall or inverse crime arises from the reason of 'limited for infinite', e.g., limited support area for infinite scenery, finite elements for continuous zone, or given noise for unknown noise. The best practice is to use a forward model independent of an inverse model. For example, in the case of a finite element forward model one would use a much finer mesh while a coarse mesh is used in the inverse model.

Image Restoration for Long-Wavelength Imaging Systems 247

Figure 5.1(a) and (c) present an original signal containing 256 pixels and a blurred version of

5.1(b), which approximates an atmospheric blur. Figure 5.1(d)-(f) show a comparison between the results obtained from the implementation of Gaussian filtering, the RL deconvolution method and our proposed algorithm, the MSEs of which are 188.29, 210.23, and 184.50, respectively. The resulting Wiener-filtered restored signal (with *ε* = 0.001) is shown in Fig. 5.1(d). It is clear that this restored signal is considerably better than the blurred signal shown in Fig. 5.1(c) whereas the restored signal using the RL method reveals lots of ringing artifacts. Figure 5.1(f) shows that the result using the proposed algorithm

0 128 256

(e)

0 128 256

Fig. 5.1. Comparison among the deconvolution for 1-D signal. (a) Original signal, (b) p.s.f. ( *σ*h = 1.5), (c) blurred signal, and restored signals by using (d) Gaussian filter, (e) the RL

Following the above discussion, Fig. 5.2 shows the iterations used by the RL method and the proposed algorithm satisfying with the stopping criterion. In the case of 1-D signal, our algorithm usually converges within fewer iterations than the RL method, the former using

<sup>100</sup> <sup>200</sup> 0 %

<sup>25</sup> <sup>50</sup> 0%

Fig. 5.2. Convergence rate vs. iteration no. of Fig. 5.1 for (a) the RL algorithm, and (b) our

(a)

(b)

0

250

0

algorithm and *σ*h = 1.5, and (f) our proposed algorithm with *δ* – *h*

0.5

1

=1.5) presents higher contrast and less ringing artifact than other

0

250

0

250

0 128 256

(f)

0 128 256

 and *<sup>h</sup>* =1.5.

(c)

*<sup>h</sup>* equal to 1.5, Fig.

**5.1.1 Results for synthetically blurred signal and image** 

 ( *<sup>h</sup>* 

(a)

0 128 256

(d)

0 128 256

34 iterations and the latter using 187 iterations.

50% 100%

50% 100%

with *h*hp equal to *δ* – *h*

0

250

0

proposed algorithm.

250

two methods.

this signal, obtained by convolving it with a Gaussian function with

## **4.4 Remark**

In this section, we have proposed some extra points about image restoration. Interpolator with noise removal, design of regularization term for reducing computational burden, and inverse pitfall/crime have been illustrated and discussed.

## **5. Related application**

In this section, application to a mm-wave imaging system or near infrared diffuse optical tomography using image restoration is demonstrated for post-processing or interprocessing. To verify the proposed method in the previous section (Sec. 3.4), a computergenerated signal/image and an image of real scene were tested.

## **5.1 Post-processing: Application to a millimeter-wave imaging system [Pan, 2010]**

A 1-D noiseless signal and a 2-D noisy image were used, originally blurred with a p.s.f. of Gaussian function plus additive white Gaussian noise. White Gaussian noise is defined with a zero mean and variance, *σ*2, specified by a blurred signal-to-noise ratio (BSNR). Recall that

$$BSNR = 10\log\_{10}\left(\frac{\sum\_{i,j=1}^{M.N} \frac{\left(\left(f \cdot \otimes h\right)\_{i,j} - \overline{f \cdot \otimes h}\right)^2}{\sigma^2}}{MN}\right) \quad \text{(in dB)}\tag{5.1}$$

where *M*, *N* are the dimension of the processed image and i, j are the indexes of a pixel and *X* means the average value of *X*. In many practical situations, the blur is often unknown and little information is available about the true image; therefore, several *<sup>h</sup>* of the Gaussian blur around the true *σ*h were tested in the following examples; *f*o and α are chosen

to g and *g* , respectively. In this work, the stopping criterion is 0 *nf f* 0.01% (for 1-D

signal) or 1% (for 2-D image). The mean square error (MSE) of the restored signal/image relative to the original signal/image is provided here for the evaluation of image quality, thus supporting the visual assessment.

The proposed algorithm was applied to a 1-D signal as well as both simulated and real atmospherically degraded images, one of a simulated blur and one of a real blur. The purpose of the simulation was to enable a comparative evaluation of the results given the original signal/image and to explain the algorithm characteristics. In the real-blur example shown here, a 256 × 400 pixel millimeter-wave image was tested and the image was captured at 94 GHz by the Defence Evaluation and Research Agency, Malvern, UK.

For a comparison purpose, non-iterative Gaussian filtering was used in the case of 1-D signal and the common Richardson–Lucy (RL) deconvolution method was implemented using a built-in MATLAB function deconvlucy in the cases of both 1-D signal and 2-D images. This RL method employs an iterative procedure to estimates the original signal/image, and therefore requires an initial guess of it as well.

246 Image Restoration – Recent Advances and Applications

In this section, we have proposed some extra points about image restoration. Interpolator with noise removal, design of regularization term for reducing computational burden, and

In this section, application to a mm-wave imaging system or near infrared diffuse optical tomography using image restoration is demonstrated for post-processing or interprocessing. To verify the proposed method in the previous section (Sec. 3.4), a computer-

**5.1 Post-processing: Application to a millimeter-wave imaging system [Pan, 2010]** 

A 1-D noiseless signal and a 2-D noisy image were used, originally blurred with a p.s.f. of Gaussian function plus additive white Gaussian noise. White Gaussian noise is defined with a zero mean and variance, *σ*2, specified by a blurred signal-to-noise ratio (BSNR). Recall that

, ,

where *M*, *N* are the dimension of the processed image and i, j are the indexes of a pixel and *X* means the average value of *X*. In many practical situations, the blur is often unknown

Gaussian blur around the true *σ*h were tested in the following examples; *f*o and α are chosen

signal) or 1% (for 2-D image). The mean square error (MSE) of the restored signal/image relative to the original signal/image is provided here for the evaluation of image quality,

The proposed algorithm was applied to a 1-D signal as well as both simulated and real atmospherically degraded images, one of a simulated blur and one of a real blur. The purpose of the simulation was to enable a comparative evaluation of the results given the original signal/image and to explain the algorithm characteristics. In the real-blur example shown here, a 256 × 400 pixel millimeter-wave image was tested and the image was

For a comparison purpose, non-iterative Gaussian filtering was used in the case of 1-D signal and the common Richardson–Lucy (RL) deconvolution method was implemented using a built-in MATLAB function deconvlucy in the cases of both 1-D signal and 2-D images. This RL method employs an iterative procedure to estimates the original

captured at 94 GHz by the Defence Evaluation and Research Agency, Malvern, UK.

and little information is available about the true image; therefore, several *<sup>h</sup>*

*M N i j*

, 1

*i j*

 

*MN*

*fh fh*

2

2

(in dB) (5.1)

0.01% (for 1-D

0 *nf f* 

of the

inverse pitfall/crime have been illustrated and discussed.

generated signal/image and an image of real scene were tested.

<sup>10</sup> 10log

to g and *g* , respectively. In this work, the stopping criterion is

signal/image, and therefore requires an initial guess of it as well.

*BSNR*

thus supporting the visual assessment.

**4.4 Remark** 

**5. Related application** 

### **5.1.1 Results for synthetically blurred signal and image**

Figure 5.1(a) and (c) present an original signal containing 256 pixels and a blurred version of this signal, obtained by convolving it with a Gaussian function with *<sup>h</sup>* equal to 1.5, Fig. 5.1(b), which approximates an atmospheric blur. Figure 5.1(d)-(f) show a comparison between the results obtained from the implementation of Gaussian filtering, the RL deconvolution method and our proposed algorithm, the MSEs of which are 188.29, 210.23, and 184.50, respectively. The resulting Wiener-filtered restored signal (with *ε* = 0.001) is shown in Fig. 5.1(d). It is clear that this restored signal is considerably better than the blurred signal shown in Fig. 5.1(c) whereas the restored signal using the RL method reveals lots of ringing artifacts. Figure 5.1(f) shows that the result using the proposed algorithm with *h*hp equal to *δ* – *h* ( *<sup>h</sup>* =1.5) presents higher contrast and less ringing artifact than other two methods.

Fig. 5.1. Comparison among the deconvolution for 1-D signal. (a) Original signal, (b) p.s.f. ( *σ*h = 1.5), (c) blurred signal, and restored signals by using (d) Gaussian filter, (e) the RL algorithm and *σ*h = 1.5, and (f) our proposed algorithm with *δ* – *h* and *<sup>h</sup>* =1.5.

Following the above discussion, Fig. 5.2 shows the iterations used by the RL method and the proposed algorithm satisfying with the stopping criterion. In the case of 1-D signal, our algorithm usually converges within fewer iterations than the RL method, the former using 34 iterations and the latter using 187 iterations.

Fig. 5.2. Convergence rate vs. iteration no. of Fig. 5.1 for (a) the RL algorithm, and (b) our proposed algorithm.

Image Restoration for Long-Wavelength Imaging Systems 249

quality can also be improved by reducing most of the ringing artifact and preserving more edge information. Also, it can be seen that our method with a Laplacian filter still works

(a) (b)

(c) (d) (e)

Fig. 5.4. Comparison among the deconvolution for 2-D image. (a) Original image, (b) blurred image (*σ*h = 2.5), and restored images by using (c) the RL algorithm and *σ*h = 2.5, and our

Corresponding to Fig. 5.4(c)-(e), Fig. 5.5 shows the iterations used by the RL method and the proposed algorithm where fewer iterations was used in the RL method than our algorithm, the former using 46 iterations and the latter two using about 200 iterations. It should be noted that the proposed algorithm is considerably more computationally expensive than the RL method. However, in our experiments we did not find any significant improvement but even more

<sup>25</sup> <sup>50</sup> 0 %

(b)

<sup>100</sup> <sup>200</sup> 0%

(c)

<sup>100</sup> <sup>200</sup> 0 %

and (c) 2-D Laplacian filter, respectively.

Fig. 5.5. Convergence rate vs. iteration no. of Fig. 5.4 for (a) the RL algorithm, and our

(a)

=2.5 and (f) 2-D Laplacian filter and *<sup>h</sup>*

=2.5,

 and *<sup>h</sup>* 

ringing artifacts when the RL method was employed for a further iteration number.

well, shown in Fig. 5.4(e).

proposed algorithm with (d) *δ* – *h*

50 % 100%

50% 100%

50% 100%

proposed algorithm with (b) *δ* – *h*

respectively.

Figure 5.3 shows that the nature of our proposed method possesses the ability to reconstructing frequency spectrum beyond the diffraction limit, where a 1-D noiseless signal was used. Figure 5.3(a)-(c) shows the original signal, p.s.f. and its modulation transfer function (MTF); the degraded (*σ*h = 1.5) and the restored signals are shown in Fig. 5.3(d)–(f) with *<sup>h</sup>* equal to 1.2, 1.5, and 1.8, respectively; and the MTFs of the original and the restored signals are depicted in Fig. 5.3(g)–(i). The restored signals in Fig. 5.3(e) and (f) display the performance of high resolution and the two peaks are separated in Fig. 5.3(d) even with a small *<sup>h</sup>* . Compared with that of the original signal, high-frequency information of the restored signals was definitely generated beyond the diffraction limit as shown between the two dashed lines in Fig. 5.3(g)–(i), explaining that the proposed method possesses the highresolution ability.

Fig. 5.3. Demonstration of the high resolution of the proposed algorithm. (a) Original signal, (b) Gaussian form (solid line) and MTF (dotted line) of the blurring function (*σ*h = 1.5), (c) blurred signal, (d)–(f) restored signals with *δ* – *h* and *<sup>h</sup>* =1.2, 1.5, and 1.8, respectively, and (g)–(i) MTFs of the blurred (solid line) and the restored (dotted line) signals. Note that the region between two dashed lines is the high frequency beyond the diffraction limit.

Figure 5.4 represents an image (256 × 256) of clown which is a built-in image in MatLab. Figure 5.4 displays a comparison between the results obtained from the implementation of the RL deconvolution method and our proposed algorithm. Figure 5.4(a) shows the original image, convolving it with a 2-D Gaussian function with *<sup>h</sup>* equal to 2.5 to obtain a blurred image shown in Fig. 5.4(b). Figure 5.4(c)-(e) show the images restored with the RL deconvolution method and our proposed algorithm with δ- *h* and the Laplacian filter where *h h*, =2.5 was used; the MSEs of these three results are 181.17, 49.45, and 52.61, respectively. These three restored images demonstrate high quality but Fig. 5.4(c) still shows ringing artifact especially in the boundary of the image. In Fig. 5.4(d), simultaneously, the image 248 Image Restoration – Recent Advances and Applications

Figure 5.3 shows that the nature of our proposed method possesses the ability to reconstructing frequency spectrum beyond the diffraction limit, where a 1-D noiseless signal was used. Figure 5.3(a)-(c) shows the original signal, p.s.f. and its modulation transfer function (MTF); the degraded (*σ*h = 1.5) and the restored signals are shown in Fig. 5.3(d)–(f)

 equal to 1.2, 1.5, and 1.8, respectively; and the MTFs of the original and the restored signals are depicted in Fig. 5.3(g)–(i). The restored signals in Fig. 5.3(e) and (f) display the performance of high resolution and the two peaks are separated in Fig. 5.3(d) even with a

 . Compared with that of the original signal, high-frequency information of the restored signals was definitely generated beyond the diffraction limit as shown between the two dashed lines in Fig. 5.3(g)–(i), explaining that the proposed method possesses the high-

0 128 <sup>256</sup> <sup>0</sup>

0 128 <sup>256</sup> <sup>0</sup>

<sup>0</sup> <sup>128</sup> <sup>256</sup> <sup>0</sup>

 and *<sup>h</sup>* 

Fig. 5.3. Demonstration of the high resolution of the proposed algorithm. (a) Original signal, (b) Gaussian form (solid line) and MTF (dotted line) of the blurring function (*σ*h = 1.5), (c)

(g)–(i) MTFs of the blurred (solid line) and the restored (dotted line) signals. Note that the region between two dashed lines is the high frequency beyond the diffraction limit.

Figure 5.4 represents an image (256 × 256) of clown which is a built-in image in MatLab. Figure 5.4 displays a comparison between the results obtained from the implementation of the RL deconvolution method and our proposed algorithm. Figure 5.4(a) shows the original

image shown in Fig. 5.4(b). Figure 5.4(c)-(e) show the images restored with the RL

 =2.5 was used; the MSEs of these three results are 181.17, 49.45, and 52.61, respectively. These three restored images demonstrate high quality but Fig. 5.4(c) still shows ringing artifact especially in the boundary of the image. In Fig. 5.4(d), simultaneously, the image

0 128 <sup>256</sup> <sup>0</sup>

0 128 <sup>256</sup> <sup>0</sup>

0 128 <sup>256</sup> <sup>0</sup>

=1.2, 1.5, and 1.8, respectively, and

*<sup>h</sup>* equal to 2.5 to obtain a blurred

and the Laplacian filter where

50 100 150

50 100 150

500 1000

<sup>200</sup> (c)

<sup>200</sup> (f)

<sup>1500</sup> (i)

0.5

50 100 150

500 1000

<sup>1</sup> (b)

<sup>200</sup> (e)

<sup>1500</sup> (h)

with *<sup>h</sup>* 

small *<sup>h</sup>* 

*h h*, 

resolution ability.

0 128 <sup>256</sup> <sup>0</sup>

0 128 <sup>256</sup> <sup>0</sup>

0 128 <sup>256</sup> <sup>0</sup>

image, convolving it with a 2-D Gaussian function with

deconvolution method and our proposed algorithm with δ- *h*

blurred signal, (d)–(f) restored signals with *δ* – *h*

50 100 150

50 100 150

500 1000

<sup>200</sup> (a)

<sup>200</sup> (d)

<sup>1500</sup> (g)

quality can also be improved by reducing most of the ringing artifact and preserving more edge information. Also, it can be seen that our method with a Laplacian filter still works well, shown in Fig. 5.4(e).

Fig. 5.4. Comparison among the deconvolution for 2-D image. (a) Original image, (b) blurred image (*σ*h = 2.5), and restored images by using (c) the RL algorithm and *σ*h = 2.5, and our proposed algorithm with (d) *δ* – *h* and *<sup>h</sup>* =2.5 and (f) 2-D Laplacian filter and *<sup>h</sup>* =2.5, respectively.

Corresponding to Fig. 5.4(c)-(e), Fig. 5.5 shows the iterations used by the RL method and the proposed algorithm where fewer iterations was used in the RL method than our algorithm, the former using 46 iterations and the latter two using about 200 iterations. It should be noted that the proposed algorithm is considerably more computationally expensive than the RL method. However, in our experiments we did not find any significant improvement but even more ringing artifacts when the RL method was employed for a further iteration number.

Fig. 5.5. Convergence rate vs. iteration no. of Fig. 5.4 for (a) the RL algorithm, and our proposed algorithm with (b) *δ* – *h* and (c) 2-D Laplacian filter, respectively.

Image Restoration for Long-Wavelength Imaging Systems 251

(a) (b)

(c) (d)

Fig. 5.7. Comparison among image deconvolution for a 94 GHz millimeter-wave image. (a) Real degraded image, and restored images by using (b) the RL algorithm and *σ*h =3, and our

(a)

(b)

(c)

Fig. 5.8. Convergence rate vs. iteration no. of Fig. 5.7 for (a) the RL algorithm, and our

=3 and (d) 2-D Laplacian filter and *<sup>h</sup>*

20 40

80 160

80 160

and (c) 2-D Laplacian filter, respectively.

=3,

 and *<sup>h</sup>* 

proposed algorithm with (c) *δ* – *h*

0 %

100%

0% 50%

0%

proposed algorithm with (b) *δ* – *h*

50% 100%

50%

100%

respectively.

For further inspection into our proposed algorithm, we investigated the effect of this algorithm using the high pass filter, *δ*- *h* , with varied *<sup>h</sup>* . Figure 5.6 demonstrates this case where the original and the noisy (*σ*h = 2.5 and BSNR=30 dB) images are displayed in Fig. 5.6(a), (b), and the restored images are shown in Fig. 5.6(c)–(e) obtained with the use of *<sup>h</sup>* equal to 2, 2.5, and 3, respectively. The MSEs of these results are 163.77, 76.82, and 97.97, respectively. Of all the restored images, Fig. 5.6(c) shows a worse image quality than the others, in which noise was intensively produced and hard to be removed although the contrast of the restored image was enhanced. Figure 5.6(d) and (e) show the promising results where high contrast was generated and noise was suppressed. As a result, it is recommended that a small *<sup>h</sup>* <sup>~</sup> , together with adequate iterations, should be avoided to use in the restoration process of the proposed algorithm.

Fig. 5.6. Demonstration of the deconvolution for a 2-D image using the proposed algorithm. (a) Original image, (b) noisy image (*σ*h = 2.5 and BSNR=30 dB), and restored images by using our proposed algorithm incorporating *δ* – *h* with (c) *<sup>h</sup>* = 2, (d) *<sup>h</sup>* = 2.5 and (f) *<sup>h</sup>* = 3, respectively.

### **5.1.2 Results for a real degraded image**

It is always expected that a novel algorithm can be implemented on a real image; Fig. 5.7(a) presents a real degraded image captured by an mm-wave imaging system. Figure 5.7(b) was restored using the RL method and Fig. 5.7(c), (d) were obtained by using our proposed method where Fig. 5.7(b)-(d) were obtained with *h h*, equal to 3. It is obvious that the restored images, Fig. 5.7(c) reveals sharp edges, high contrast and much more details like a number 2, two cars, and three lamps of the floodlight, etc., but Fig. 5.7(b) has shown ringing artifact spreading through the whole image. Furthermore, it is worth mentioning that Fig. 5.7(d) also shows a good image quality which was achieved with the use of a 3 × 3 Laplacian operator.

Corresponding to Fig. 5.7(c)-(e), Fig. 5.8 shows the iterations used by the RL method and the proposed algorithm satisfying with the stopping criterion. In the case of 2-D image, the RL method used less iteration than our algorithm, the former using 35 iterations and the latter two using about 150 iterations.

250 Image Restoration – Recent Advances and Applications

For further inspection into our proposed algorithm, we investigated the effect of this

where the original and the noisy (*σ*h = 2.5 and BSNR=30 dB) images are displayed in Fig. 5.6(a), (b), and the restored images are shown in Fig. 5.6(c)–(e) obtained with the use of *<sup>h</sup>*

equal to 2, 2.5, and 3, respectively. The MSEs of these results are 163.77, 76.82, and 97.97, respectively. Of all the restored images, Fig. 5.6(c) shows a worse image quality than the others, in which noise was intensively produced and hard to be removed although the contrast of the restored image was enhanced. Figure 5.6(d) and (e) show the promising results where high contrast was generated and noise was suppressed. As a result, it is

(a) (b)

(c) (d) (e)

Fig. 5.6. Demonstration of the deconvolution for a 2-D image using the proposed algorithm. (a) Original image, (b) noisy image (*σ*h = 2.5 and BSNR=30 dB), and restored images by

It is always expected that a novel algorithm can be implemented on a real image; Fig. 5.7(a) presents a real degraded image captured by an mm-wave imaging system. Figure 5.7(b) was restored using the RL method and Fig. 5.7(c), (d) were obtained by using our proposed

images, Fig. 5.7(c) reveals sharp edges, high contrast and much more details like a number 2, two cars, and three lamps of the floodlight, etc., but Fig. 5.7(b) has shown ringing artifact spreading through the whole image. Furthermore, it is worth mentioning that Fig. 5.7(d) also shows a good image quality which was achieved with the use of a 3 × 3 Laplacian operator. Corresponding to Fig. 5.7(c)-(e), Fig. 5.8 shows the iterations used by the RL method and the proposed algorithm satisfying with the stopping criterion. In the case of 2-D image, the RL method used less iteration than our algorithm, the former using 35 iterations and the latter

with (c) *<sup>h</sup>*

= 2, (d) *<sup>h</sup>*

equal to 3. It is obvious that the restored

= 2.5 and (f) *<sup>h</sup>*

= 3,

, with varied *<sup>h</sup>*

<sup>~</sup> , together with adequate iterations, should be avoided to use

. Figure 5.6 demonstrates this case

algorithm using the high pass filter, *δ*- *h*

in the restoration process of the proposed algorithm.

using our proposed algorithm incorporating *δ* – *h*

method where Fig. 5.7(b)-(d) were obtained with *h h*,

**5.1.2 Results for a real degraded image** 

two using about 150 iterations.

respectively.

recommended that a small *<sup>h</sup>*

Fig. 5.7. Comparison among image deconvolution for a 94 GHz millimeter-wave image. (a) Real degraded image, and restored images by using (b) the RL algorithm and *σ*h =3, and our proposed algorithm with (c) *δ* – *h* and *<sup>h</sup>* =3 and (d) 2-D Laplacian filter and *<sup>h</sup>* =3, respectively.

Fig. 5.8. Convergence rate vs. iteration no. of Fig. 5.7 for (a) the RL algorithm, and our proposed algorithm with (b) *δ* – *h* and (c) 2-D Laplacian filter, respectively.

Image Restoration for Long-Wavelength Imaging Systems 253

corresponding 1D circular transection profiles to reveal noticeable differences. Basically, there is a better separation resolution but a lower intensity owing to a highly suppressed signal by a hard prior rather than a soft prior. Additionally, Fig. 5.9 (m-o) exhibits good

The phantoms employed for justifying our proposed technique (Sec. 3.4) incorporate two or three inclusions with various sizes, locations and separations, illustrated in Fig. 5.10, where *R* denotes radius in the unit of *mm*. Of the phantom, the background absorption (*μa*) and

maximum absorption and reduced scattering for the inclusion are 0.025 mm-1 and 2.5 mm-1, thereby assuming the contrast ratio of the inclusion to background 10:1, because high contrast results in much more overlapping effects than low contrast although a contrast of

(a) (b) Fig. 5.10. Schematic diagram for the dimensions of two different test cases in simulation. (a)

As depicted in Fig. 5.10, Case 1, 2, respectively, have two inclusions separated with a similar distance but different sizes. As the separation resolution of inclusions is examined, several (two or three) embedded inclusions are necessary, and different inclusion sizes are considered as well. For the convenience in discussion latter, we denote M0-4 as the reconstructions with the schemes using non-filtering, δ-g2 (*σ*2=1.5), g1-g2(*σ*1=0.75, *σ*2=1.5), wavelet (a dilated factor *a*=0.5), and Laplacian high-pass filter (HPF) in their 2D form, respectively. Currently, absorption-coefficient images are presented for our continuous

In FEM-based image reconstruction, the homogeneous background (*μa* = 0.0025mm-1, *μ'*

0.25mm-1) was adopted as an initial guess. Thirty-iteration assignment was employed for

*<sup>s</sup>*=

2

, reaches smaller

*n n* 1 *n* 

and (b) are Case 1, 2, respectively, where R is radius in the unit of mm.

each case as the normalized increasing rate, i.e. mean value of

*<sup>s</sup>* ) values are about 0.0025 mm-1 and 0.25 mm-1, respectively, while the

convergences obtained by using both soft and hard priors.

**5.2.2 Image restoration applied to NIR DOT** 

2~10 were used throughout other published works.

wave image reconstruction algorithm.

than 10-2.

reduced scattering (*μ'*

#### **5.2 Inter-processing: Application to near infrared diffuse optical tomography**

#### **5.2.1 Rapid convergence algorithm applied to NIR DOT**

Corresponding to Eq. (4.14), some parameters are chosen as

$$\mathcal{A} = 0.75 \max \{ \boldsymbol{I}^T \boldsymbol{I} \} \; \; \; \; \left( \boldsymbol{\Delta} \boldsymbol{\chi} \right)\_0^2 = \boldsymbol{I} \; \; \; \; \; \boldsymbol{\gamma}\_n = \boldsymbol{\mathcal{A}} \; \; \; \; or \; \; \; \; \mathcal{B} \; \; \; \; \; \; \tag{5.2}$$

where the subscript *n* is the *n*-th iteration, "max" means the maximum value, and the superscript *T* denotes a transposition operation. One way to improve the convergence rate is using *<sup>n</sup>* as the Type-1 soft prior and using <sup>2</sup> 2.5 *<sup>n</sup> n e* , an exponentially decreasing form, as the Type-2 soft prior, where Type-1 is a parameter related to the system function (Jacobian matrix) and Type-2 is a user-defined parameter. Both values of *<sup>n</sup>* have been respectively employed to seek an inverse solution for comparison.

Fig. 5.9. Reconstruction data through various priors with intensity signals corrupted by Gaussian white noise (SNR=20 dB). Left column: constrained inverse solution with soft prior 1; middle column: constrained inverse solution with soft prior 2; right column: constrained inverse solution with hard prior.

Figure 5.9 illustrates the comparisons between constrained solutions using soft priors (Type 1 and 2) and a hard prior, where the left, middle and right columns are the constrained inverse solutions with soft prior 1, soft prior 2, and hard prior [M.-Cheng & M.-Chun Pan, 2010], respectively. Figure 5.9 (a-f) shows the 2D reconstructions of phantoms with two and three inclusions, where slight discrepancy can be observed. Figure 5.9 (g-l) depicts their 252 Image Restoration – Recent Advances and Applications

**5.2 Inter-processing: Application to near infrared diffuse optical tomography** 

<sup>2</sup> ( ) 

where the subscript *n* is the *n*-th iteration, "max" means the maximum value, and the superscript *T* denotes a transposition operation. One way to improve the convergence rate is

form, as the Type-2 soft prior, where Type-1 is a parameter related to the system function

(a) (b) (c)

(d) (e) (f)

0 360

(k)

0 360

(m) (n) (o)

0.01 Convergence Rate

0 10 20 30

iteration no.

Figure 5.9 illustrates the comparisons between constrained solutions using soft priors (Type 1 and 2) and a hard prior, where the left, middle and right columns are the constrained inverse solutions with soft prior 1, soft prior 2, and hard prior [M.-Cheng & M.-Chun Pan, 2010], respectively. Figure 5.9 (a-f) shows the 2D reconstructions of phantoms with two and three inclusions, where slight discrepancy can be observed. Figure 5.9 (g-l) depicts their

Fig. 5.9. Reconstruction data through various priors with intensity signals corrupted by Gaussian white noise (SNR=20 dB). Left column: constrained inverse solution with soft prior 1; middle column: constrained inverse solution with soft prior 2; right column: constrained

(h)

*n e*

 *I* , <sup>2</sup> 2.5 *<sup>n</sup> n* 

> 0 0.05 0.1

> 0 0.05 0.1

> > 0 0.05 0.1

> > 0 0.05 0.1

> > > 0

0.005

MSE(%)

2 incls. 3 incls.

*or e* , (5.2)

, an exponentially decreasing

0 360

(l)

0 360

2 incls. 3 incls.

0.01 Convergence Rate

0 10 20 30

iteration no.

(i)

*<sup>n</sup>* have been

0 0.05 0.1

0 0.05 0.1

**5.2.1 Rapid convergence algorithm applied to NIR DOT**  Corresponding to Eq. (4.14), some parameters are chosen as

0 360

(j)

0 360

0.01 Convergence Rate

0 10 20 30

iteration no.

inverse solution with hard prior.

(g)

0 0.05 0.1

0 0.05 0.1

0

0.005

MSE(%)

using *<sup>n</sup>*  0.75max{ } *<sup>T</sup>*

respectively employed to seek an inverse solution for comparison.

0 0.05 0.1

0 0.05 0.1

> 0 0.05 0.1

> 0 0.05 0.1

> > 0

0.005

MSE(%)

2 incls. 3 incls.

*J J* , 0

as the Type-1 soft prior and using <sup>2</sup> 2.5 *<sup>n</sup>*

(Jacobian matrix) and Type-2 is a user-defined parameter. Both values of

corresponding 1D circular transection profiles to reveal noticeable differences. Basically, there is a better separation resolution but a lower intensity owing to a highly suppressed signal by a hard prior rather than a soft prior. Additionally, Fig. 5.9 (m-o) exhibits good convergences obtained by using both soft and hard priors.

## **5.2.2 Image restoration applied to NIR DOT**

The phantoms employed for justifying our proposed technique (Sec. 3.4) incorporate two or three inclusions with various sizes, locations and separations, illustrated in Fig. 5.10, where *R* denotes radius in the unit of *mm*. Of the phantom, the background absorption (*μa*) and reduced scattering (*μ' <sup>s</sup>* ) values are about 0.0025 mm-1 and 0.25 mm-1, respectively, while the maximum absorption and reduced scattering for the inclusion are 0.025 mm-1 and 2.5 mm-1, thereby assuming the contrast ratio of the inclusion to background 10:1, because high contrast results in much more overlapping effects than low contrast although a contrast of 2~10 were used throughout other published works.

Fig. 5.10. Schematic diagram for the dimensions of two different test cases in simulation. (a) and (b) are Case 1, 2, respectively, where R is radius in the unit of mm.

As depicted in Fig. 5.10, Case 1, 2, respectively, have two inclusions separated with a similar distance but different sizes. As the separation resolution of inclusions is examined, several (two or three) embedded inclusions are necessary, and different inclusion sizes are considered as well. For the convenience in discussion latter, we denote M0-4 as the reconstructions with the schemes using non-filtering, δ-g2 (*σ*2=1.5), g1-g2(*σ*1=0.75, *σ*2=1.5), wavelet (a dilated factor *a*=0.5), and Laplacian high-pass filter (HPF) in their 2D form, respectively. Currently, absorption-coefficient images are presented for our continuous wave image reconstruction algorithm.

In FEM-based image reconstruction, the homogeneous background (*μa* = 0.0025mm-1, *μ' <sup>s</sup>*= 0.25mm-1) was adopted as an initial guess. Thirty-iteration assignment was employed for

each case as the normalized increasing rate, i.e. mean value of 2 *n n* 1 *n* , reaches smaller

than 10-2.

Image Restoration for Long-Wavelength Imaging Systems 255

(c)

0 180 <sup>360</sup> <sup>0</sup>

Degree

Fig. 5.12. Case 2- 2D reconstructed absorption images (a) without HPF (M0) and (b)-(e) with M1, M2, M3, M4 filtering, respectively; (f)-(j) are 1D circular profiles corresponding to (a)- (e), where solid lines are the designed, and dotted lines represent the reconstructed.

In terms of the optical properties within the inclusion and background, it is worth noted that the image reconstruction is not only pursuing qualitative correctness but also obtaining favorably quantitative information about the optical properties of either the inclusions or background. Parameters of interest such as size, contrast and location variations associated image quantification measures are most frequently investigated and discussed. Readers can

In this section, we have demonstrated the performance of our proposed image restoration algorithms exactly applied in the imaging process for 'inter-processing' and to corrupted

In this chapter, we have explained the background and the mathematical model of image formation and image restoration for long-wavelength imaging systems; as well, image restoration algorithms, further consideration on image restoration, and their related application have been described and demonstrated. In the meanwhile, a promising method to restore images has been proposed. As discussed in this chapter, the proposed algorithm was applied to both simulated and real atmospherically degraded images. Restoration results show significantly improved images. Especially, the restored millimeter-wave image highlights the superior performance of the proposed method in reality. The main novelty here is that error energy resulting from noise and ringing artifact is highly suppressed with

(h)

0

0.01

0.02

0.01

0.02

(d)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(i)

0

0.01

0.02

0.01

0.02

(e)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(j)

0

0.01

0.02

(b)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(g)

0.01

0.02

0

0.01

0.02

0

0.01

0.02

0.01

0.02

0 180 <sup>360</sup> <sup>0</sup>

Degree

**5.2.4 Performance investigation** 

images for 'post-processing.'

refer to the research work [Pan et al., 2008].

(f)

(a)

0.01

Absorption coefficient (mm-1)

**5.3 Remark** 

**6. Conclusions** 

**6.1 Concluding remark** 

0.02

## **5.2.3 Examples illustration**

## **5.2.3.1 Case 1**

This case was designed as a phantom with three smaller inclusions. Several improved images were obtained by using appropriate filtering, as shown in Fig. 5.11(b-e) of 1D circular profiles passing through the centers of inclusions. Likewise, M2 resulted in worse resolved image than others with HP filtering. Negative artifacts occurred in each reconstructed image, as depicted in Fig. 5.11(g-j). It is well noted that M4 overestimated the inclusion amplitudes, which yields a higher inclusion-to-background contrast.

Fig. 5.11. Case 1- 2D reconstructed absorption images (a) without HPF (M0) and (b)-(e) with M1, M2, M3, M4 filtering, respectively; (f)-(j) are 1D circular profiles corresponding to (a)- (e), where solid lines are the designed, and dotted lines represent the reconstructed.

## **5.2.3.2 Case 2**

In this highly challenging case, a phantom with two closest-separation inclusions was designed. As shown in Fig. 5.12(a-e), all reconstructed images underestimated inclusions, and offered relatively bad resolution for two separate inclusions. It is rather competitive for these employed filters. Based upon a quantitative comparison, as depicted in Fig. 5.12(i) and (j), M3 and M4 schemes demonstrate better resolution discrimination to separate bigger and closer inclusions in comparison of Case 1.

From the results of Case 1 and 2 for a phantom with inclusions of both small size and close separation, it can be concluded that the wavelet-like HP filtering (M3) demonstrates the best spatial-frequency resolution capability to the inclusions.

It evidently shows that the enhancement of reconstruction through the incorporation of our proposed HPF approach can effectively improve computed images. As illustrated above, the wavelet-like HP filtering schemes (M3, M4) further yields better results than the LPFcombined HP filtering schemes (M1, M2). In the aspects of sensitivity and stability of evaluation, M3 yielded results closest to the true absorption property than other schemes. However, M4 visually characterizes the inclusion-to-background contrast best.

Fig. 5.12. Case 2- 2D reconstructed absorption images (a) without HPF (M0) and (b)-(e) with M1, M2, M3, M4 filtering, respectively; (f)-(j) are 1D circular profiles corresponding to (a)- (e), where solid lines are the designed, and dotted lines represent the reconstructed.

## **5.2.4 Performance investigation**

In terms of the optical properties within the inclusion and background, it is worth noted that the image reconstruction is not only pursuing qualitative correctness but also obtaining favorably quantitative information about the optical properties of either the inclusions or background. Parameters of interest such as size, contrast and location variations associated image quantification measures are most frequently investigated and discussed. Readers can refer to the research work [Pan et al., 2008].

## **5.3 Remark**

254 Image Restoration – Recent Advances and Applications

This case was designed as a phantom with three smaller inclusions. Several improved images were obtained by using appropriate filtering, as shown in Fig. 5.11(b-e) of 1D circular profiles passing through the centers of inclusions. Likewise, M2 resulted in worse resolved image than others with HP filtering. Negative artifacts occurred in each reconstructed image, as depicted in Fig. 5.11(g-j). It is well noted that M4 overestimated the

(c)

0 180 <sup>360</sup> <sup>0</sup>

Degree

Fig. 5.11. Case 1- 2D reconstructed absorption images (a) without HPF (M0) and (b)-(e) with M1, M2, M3, M4 filtering, respectively; (f)-(j) are 1D circular profiles corresponding to (a)- (e), where solid lines are the designed, and dotted lines represent the reconstructed.

In this highly challenging case, a phantom with two closest-separation inclusions was designed. As shown in Fig. 5.12(a-e), all reconstructed images underestimated inclusions, and offered relatively bad resolution for two separate inclusions. It is rather competitive for these employed filters. Based upon a quantitative comparison, as depicted in Fig. 5.12(i) and (j), M3 and M4 schemes demonstrate better resolution discrimination to separate bigger and

From the results of Case 1 and 2 for a phantom with inclusions of both small size and close separation, it can be concluded that the wavelet-like HP filtering (M3) demonstrates the best

It evidently shows that the enhancement of reconstruction through the incorporation of our proposed HPF approach can effectively improve computed images. As illustrated above, the wavelet-like HP filtering schemes (M3, M4) further yields better results than the LPFcombined HP filtering schemes (M1, M2). In the aspects of sensitivity and stability of evaluation, M3 yielded results closest to the true absorption property than other schemes.

However, M4 visually characterizes the inclusion-to-background contrast best.

(h)

0

0.01

0.02

0.01

0.02

(d)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(i)

0

0.01

0.02

0.01

0.02

(e)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(j)

0

0.01

0.02

inclusion amplitudes, which yields a higher inclusion-to-background contrast.

0

0.01

0.02

0.01

0.02

(b)

0 180 <sup>360</sup> <sup>0</sup>

Degree

(g)

0.01

0.02

0

0.01

0.02

**5.2.3 Examples illustration** 

0 180 <sup>360</sup> <sup>0</sup>

Degree

closer inclusions in comparison of Case 1.

spatial-frequency resolution capability to the inclusions.

(f)

(a)

0.01

Absorption coefficient (mm-1)

**5.2.3.2 Case 2** 

0.02

**5.2.3.1 Case 1** 

In this section, we have demonstrated the performance of our proposed image restoration algorithms exactly applied in the imaging process for 'inter-processing' and to corrupted images for 'post-processing.'

## **6. Conclusions**

## **6.1 Concluding remark**

In this chapter, we have explained the background and the mathematical model of image formation and image restoration for long-wavelength imaging systems; as well, image restoration algorithms, further consideration on image restoration, and their related application have been described and demonstrated. In the meanwhile, a promising method to restore images has been proposed. As discussed in this chapter, the proposed algorithm was applied to both simulated and real atmospherically degraded images. Restoration results show significantly improved images. Especially, the restored millimeter-wave image highlights the superior performance of the proposed method in reality. The main novelty here is that error energy resulting from noise and ringing artifact is highly suppressed with

Image Restoration for Long-Wavelength Imaging Systems 257

[7] Gillette, J. C.; Stadtmiller, T. M. & Hardie, R. C. (1995). Aliasing Reduction in Staring Infrared Imagers Utilizing Subpixel Techniques. *Opt. Eng.*, Vol. 34, pp. 3130-3137 [8] Holboke, M. J.; Tromberg, B. J.; Li, X.; Shah, N.; Fishkin, J.; Kidney, D.; Butler, J.; Chance,

[9] Hou, H. & Andrews, H. (1978). Cubic Splines for Image Interpolation and Digital

[10] Hunt, B. R. & Sementilli, P. J. (1992). Description of a Poisson Imagery Superresolution

[11] Hunt, B. R. (1994). Prospects for Image Restoration, *Int. J. Mod. Phys.*, C 5, pp. 151-178 [12] Jain, A. K. (1989).*Fundamentals of Digital image Processing*, Prentice- Hall, Englewood

[13] Keys, R. (1981). Cubic Convolution Interpolation for Digital Image Processing. *IEEE* 

[14] Kundur, D. & Hatzinakos, D. A (1998). Novel Blind Deconvolution Scheme for Image

[15] Landi, G. (2007). A Fast Truncated Lagrange Method for Large-Scale Image Restoration Problems, Applied Mathematics and Computation, Vol. 186, pp. 1075–1082. [16] Lettington, A. H. & Hong, Q. H. (1995). Image Restoration Using a Lorentzian

[17] Lucy, L. B. (1974). An Iterative Technique for the Rectification of Observed

[18] Meinel, E. S. (1986). Origins of Linear and Nonlinear Recursive Restoration Algorithms.

[19] Ng, M. K.; Molina, R. & Bose, N. K. (2003). Mathematical Analysis of Super-Resolution

[20] Pan, M. C. & Lettington, A. H. (1998). Smoothing Images by a Probability Filter. *Proc. IEEE International Joint Symposia on Intelligence and Systems*, pp.343-346. [21] Pan, M. C. & Lettington, A. H. (1999). Efficient Method for Improving Poisson MAP

[22] Pan, M. C. (2003). A Novel Blind Super-resolution Algorithm for Restoring Gaussian

[23] Pan, M. C. (2006). Improving a Single Down-sampled Image Using Probability -

[24] Pan, M.-Cheng; Chen, C. H.; Chen, L.Y.; Pan, M.-Chun & Shyr, Y. M. (2008). Highly-

Blurred Images. *The International Journal of Imaging Systems & Technology*, Vol. 12,

filtering-based Interpolation and Improved Poisson Maximum *a posteriori* Superresolution. *EURASIP Journal on Applied Signal Processing*, Vol. 2006, Article ID 97492

resolved Diffuse Optical Tomography: A Systematical Approach Using High-pass Filtering for Value-preserved Images. *Journal of Biomedical Optics*, Vol. 13, No. 02,

Probability Model. *J. Mod. Opt.* , Vol. 42, pp. 1367-1376

Methodology, *IEEE Signal Processing Mag.* , Vol. 20, pp. 62-74

Super-resolution. *Electronics Letters*, Vol. 35, No. 10, pp. 803-805

Distributions. *The Astron. J.*, Vol. 79, pp. 745-765

*J. Opt. Soc. Am.*, A 3, pp. 787-799

pp. Article ID 024022 (1-14)

237-247

pp. 508–517

25, pp. 196-199

Cliffs, NJ, USA

46, No. 2, pp375-390

pp. 239-246

(1-9)

B. & Yodh, A. G. (2000). Three-dimensional Diffuse Optical Mammography with Ultrasound Localization in a Human Subject. *Journal of Biomedical Optics*, Vol. 5, pp.

Filtering. *IEEE Transactions on Acoustics, Speech, and Signal Processing*, Vol. 26, No. 6,

Algorithm, *Astronomical Data Analysis Software and System I., A.S.P. Conf. Ser.* , Vol.

*Transactions on Acoustics, Speech, and Signal Processing*, Vol. 29, No. 6, pp. 1153–1160

Restoration Using Recursive Filtering. *IEEE Transactions on Signal Processing*, Vol.

the algorithm proposed in this chapter. Also, we have used such a resolution-enhancing technique with HP filtering incorporated with the FEM-based inverse computation to obtain highly resolved tomographic images of optical-property.

In addition, we have developed and realized the schemes for expediting NIR DOT image reconstruction through the inverse solution regularized with the constraint of a Lorentzian distributed function. Substantial improvements in reconstruction have been achieved without incurring additional hardware cost. With the introduction of constraints having a form of the Lorentzian distributed function, rapid convergence can be achieved owing to the fact that decreasing Δχ results in the increase of *λ* as the iteration process proceeds, and vice versa. It behaves like a criterion in the sense of a rapid convergence that the optimal iteration number is founded as seeking an inverse solution regularized with the Lorentzian distributed function.

## **6.2 Future work**

It is anticipated that of regularizing mean square error (residual term) with error energy reduction and rapid convergence (*a priori* terms) an algorithm is explored to restore images effectively and efficiently. In addition, it is no doubt that image restoration for interdiscipline application is the focus in the future research.

## **7. Acknowledgements**

In collaboration with the DASD laboratory at the Department of Mechanical Engineering of National Central University, the author wish to thank group leader and members to develop the software (NIR.FD\_PC) and validate the instrumentation for NIR DOT imaging system. Also, the author would like to acknowledge the funding support from the grants by the National Science Council (NSC 93-2213-E-236-002, NSC 95-2221-E-236-002, NSC 98-2221-E-236-013, NSC 99-2221-E-236-014, NSC 100-2221-E-236-012) in Taiwan (ROC).

## **8. References**


256 Image Restoration – Recent Advances and Applications

the algorithm proposed in this chapter. Also, we have used such a resolution-enhancing technique with HP filtering incorporated with the FEM-based inverse computation to obtain

In addition, we have developed and realized the schemes for expediting NIR DOT image reconstruction through the inverse solution regularized with the constraint of a Lorentzian distributed function. Substantial improvements in reconstruction have been achieved without incurring additional hardware cost. With the introduction of constraints having a form of the Lorentzian distributed function, rapid convergence can be achieved owing to the fact that decreasing Δχ results in the increase of *λ* as the iteration process proceeds, and vice versa. It behaves like a criterion in the sense of a rapid convergence that the optimal iteration number is founded as seeking an inverse solution regularized with the Lorentzian

It is anticipated that of regularizing mean square error (residual term) with error energy reduction and rapid convergence (*a priori* terms) an algorithm is explored to restore images effectively and efficiently. In addition, it is no doubt that image restoration for inter-

In collaboration with the DASD laboratory at the Department of Mechanical Engineering of National Central University, the author wish to thank group leader and members to develop the software (NIR.FD\_PC) and validate the instrumentation for NIR DOT imaging system. Also, the author would like to acknowledge the funding support from the grants by the National Science Council (NSC 93-2213-E-236-002, NSC 95-2221-E-236-002, NSC 98-2221-E-

[1] Archer, G. & Titterington, D. M. (1995). On Some Bayesian/Regularization Methods for Image Restoration. *IEEE Trans. on Image Process*ing, Vol. 4, pp. 307-310 [2] Brendel, B. & Nielsen, T. (2009). Selection of Optimal Wavelengths for Spectral

[3] Chen, W.; Chen, M. & Zhou, J. (2000). Adaptively Regularized Constrained Total Least-Squares Image Restoration. *IEEE Trans. on Image Processing*, Vol. 9, pp. 588-596 [4] Davis, S. C.; Dehghani, H.; Wang, J.; Jiang, S.; Pogue, B. W. & Paulsen, K. D. (2007).

[5] Frieden, B. R. (1972). Restoring with Maximum Likelihood and Maximum Entropy, *J.* 

[6] Gerchberg, R. W. (1974). Super-Resolution through Error Energy Reduction. *Opt. Acta*,

Laplacian-type Regularization. *Opt. Exp.* , Vol. 15, pp. 4066-4082

Reconstruction in Diffuse Optical Tomography. *Journal of Biomedical Optics*, Vol. 14,

Image-guided Diffuse Optical Fluorescence Tomography Implemented with

236-013, NSC 99-2221-E-236-014, NSC 100-2221-E-236-012) in Taiwan (ROC).

highly resolved tomographic images of optical-property.

discipline application is the focus in the future research.

*Opt. Soc. Am.*, Vol. 62, pp. 511-518

Vol. 21, pp. 709-720

distributed function.

**7. Acknowledgements** 

**6.2 Future work** 

**8. References** 

034041


**12** 

*China* 

**Super–Resolution Restoration and** 

 **Passive Millimeter Wave Imaging** 

According to blackbody radiation theory [1], all substances at a finite absolute temperature will radiate electromagnetic energy. Passive millimeter-wave (PMMW) imaging system forms images by detecting the millimeter-wave radiation energy from the scene and utilizing the differences of the radiation intensity[2,3]. Although such imaging has been performed for decades (or more, if one includes microwave radiometric imaging), new sensor technology in the millimeter-wave regime has enabled the generation of PMMW imaging at video rates and has renewed interest in this area. Clouds and fog are effectively transparent to millimeter-wave and the cold sky is reflected by metallic objects on the ground making PMMW images similar to infrared (IR) and visible images. Due to being able to perform well under adverse weather conditions, PMMW imaging offers advantages over IR and visible imaging. It is widely used in airport security, scene monitoring, plane

However, the obtained images usually have the inherent problem of poor resolution, which is caused by limited aperture dimensions and the consequent diffraction limit. Images acquired from practical sensing operations usually suffer from poor resolution due to the finite size limitations of the antenna, or the lens, and the consequent imposition of diffraction limits. The fundamental operation underlying the sensing operation is the "low-pass" filtering effect due to the finite size of the antenna lens. The image at the output of the imaging system is a lowpass filtered version of the original scene. There is no useful signal beyond the cut-off frequency in the measured data, and the information lost by the imaging system are the fine details, i.e. high-frequency spectral components. In order to restore the details and improve the resolution of the image, some methods of image restoration will be needed. As is well-known that the problem of image restoration is a inverse problem and inverse problem is always singular or ill-posed. Traditional image restoration methods based on de-convolution approaches principally try to restore the information of the pass band and eliminate the effect of additive noise components. Therefore, these methods have merely limited resolution enhancement capabilities. Greater resolution improvements can only be achieved through a class of more sophisticated algorithms, called super-resolution algorithmsor image reconstruction algorithms, before the PMMW imaging can be employed.Some studies have

blind landing, medical diagnosis and environmental detection, et al [4-8].

**1. Introduction** 

 **Image Reconstruction for** 

Liangchao Li, Jianyu Yang and Chengyue Li *University of Electronic Science and Technology of China* 


## **Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging**

Liangchao Li, Jianyu Yang and Chengyue Li *University of Electronic Science and Technology of China China* 

## **1. Introduction**

258 Image Restoration – Recent Advances and Applications

[25] Pan, M.-Cheng & Pan, M.-Chun (2010). Rapid Convergence to the Inverse Solution

[26] Pan, M. C. (2010). Image Restoration Through Regularization Based on Error Energy

[27] Paulsen, K. D. & Jiang, H. (1995). Spatially Varying Optical Property Reconstruction

[28] Pogue, B. W.; McBride, T. O.; Prewitt, J.; Österberg, U. L. & Paulsen, K. D. (1999).

[29] Richardson, W. H. (1972). Bayesian-based Iterative Method of Image Restoration. *J. Opt.* 

[30] Segal, C.; Molina, R. & Katsaggelos, K. (2003). High-resolution Images from Low-Resolution Compressed Video. *IEEE Signal Processing Mag.* , Vol. 20, pp. 37-48 [31] Sezan, M. I. & Tekalp, A. M. (1988). Iterative Image Restoration with Ringing

[32] Singh, S.; Tandon, S. N. & Gupta, H. M. (1986). An Iterative Restoration Technique.

[33] Stewart, K. & Durrani, T. S. (1986). Constrained Signal Reconstruction--A Unified

[34] Teboul, S.; Blanc-Feraud, L.; Aubert, G. & Barlaud, M. (1998). Variational Approach for

[35] Toraldo di Francia, G. (1952). Supergain Antennas and Optical Resolving Power. *Nuovo* 

[36] Toraldo di Francia, G. (1955). Resolving Power and Information. *J. Opt. Soc. Am.* , Vol.

[37] Wiener, N. (1942). *Extrapolation, Interpolation, and Smoothing of Stationary Time Series*,

Approach. *EURASIP Signal Processing* III, pp. 1423-1426

15, 016014(1-11)

Vol. 38, pp. 2950-2961

*Soc. Am.*, Vol. 62, pp. 55-59

*Processing*, pp. 1300-1303

*Signal Processing*, Vol. 11, pp. 1-11

*Processing*, Vol. 7, pp. 387-397

45, pp. 497-501

MIT Press, USA

*Cimento Suppl*, Vol. 9, pp. 426-435

pp. 308-315

691-701

Regularized with the Lorentzian Distributed Function for Near-infrared Continuous Wave Diffuse Optical Tomography," *Journal of Biomedical Optics*, Vol.

Minimization. *International Journal of Imaging Systems & Technology*, Vol. 20, No. 4,

Using a Finite Element Diffusion Equation Approximation. *Med. Phys.*, Vol. 22, pp.

Spatially Variant Regularization Improves Diffuse Optical Tomography. *Appl. Opt.*,

Suppression Using POCS. *Proceedings of IEEE Int. Conf. Acoust., Speech, Signals* 

Edge-Preserving Regularization Using Coupled PDE's. *IEEE Transactions on Image* 

According to blackbody radiation theory [1], all substances at a finite absolute temperature will radiate electromagnetic energy. Passive millimeter-wave (PMMW) imaging system forms images by detecting the millimeter-wave radiation energy from the scene and utilizing the differences of the radiation intensity[2,3]. Although such imaging has been performed for decades (or more, if one includes microwave radiometric imaging), new sensor technology in the millimeter-wave regime has enabled the generation of PMMW imaging at video rates and has renewed interest in this area. Clouds and fog are effectively transparent to millimeter-wave and the cold sky is reflected by metallic objects on the ground making PMMW images similar to infrared (IR) and visible images. Due to being able to perform well under adverse weather conditions, PMMW imaging offers advantages over IR and visible imaging. It is widely used in airport security, scene monitoring, plane blind landing, medical diagnosis and environmental detection, et al [4-8].

However, the obtained images usually have the inherent problem of poor resolution, which is caused by limited aperture dimensions and the consequent diffraction limit. Images acquired from practical sensing operations usually suffer from poor resolution due to the finite size limitations of the antenna, or the lens, and the consequent imposition of diffraction limits. The fundamental operation underlying the sensing operation is the "low-pass" filtering effect due to the finite size of the antenna lens. The image at the output of the imaging system is a lowpass filtered version of the original scene. There is no useful signal beyond the cut-off frequency in the measured data, and the information lost by the imaging system are the fine details, i.e. high-frequency spectral components. In order to restore the details and improve the resolution of the image, some methods of image restoration will be needed. As is well-known that the problem of image restoration is a inverse problem and inverse problem is always singular or ill-posed. Traditional image restoration methods based on de-convolution approaches principally try to restore the information of the pass band and eliminate the effect of additive noise components. Therefore, these methods have merely limited resolution enhancement capabilities. Greater resolution improvements can only be achieved through a class of more sophisticated algorithms, called super-resolution algorithmsor image reconstruction algorithms, before the PMMW imaging can be employed.Some studies have

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 261

resolution and rate near to video when lens or reflectors are used in focal plane real-time

In Fig.2, the propagation process from object-plane to data-plane is two-dimensional (2D) spatial Fourier transformation (FT), while the propagation process from data-plane to focalplane is 2D spatial inverse Fourier transformation (IFT). FPA system forms images in the image-plane by receiving object-radiation energy, without 2D FT and IFT in post-data

According to Fresnel and Fraunhofer diffraction theory [20], the incoherent image is given by

where *fxy* (,) denotes the object's intensity function on the region (,) *x y* , *gxy* (,) denotes the gray level function on the region (,) *u v* , *h uvxy* <sup>1</sup> ,,, denotes the PSF (Point Spread function) of the imaging sensor, and *nuv* (,) is the noise of image plane, *S* denotes the non-

*Object plane Data plane Focal plane Image plane Image plane*

*Diffraction- Aperture* 

*Object plane Data plane Focal plane Image plane Image plane*

Practically, the system model above can be improved appropriately according to certain conditions. For example, if the gain consistency of the channels is well, the model can be

*Multiple-beam antenna Array receiver Super-resolution* 

*Energy distribution iinitial data image result* 

*Space-variant of the imaging process Non-uniform of beam Non-uniform of channel*

<sup>1</sup> *g u v S f x y h u v x y dxdy n u v* (,) [ (,) (,;,) ] (,) (2)

'

*Super- Resolution* 

'

imaging.

processing. Thus, the system is high real-time.

uniformity of the antenna beams and inconsistencies of channels.

Brightness temperature

Fig. 1. Space-variant-model of Focal Plane Array imaging

Fig. 2. Principle of FPA imaging

expressed space-variant model

Focal Plane Array Imaging

indicated that the cost of an imager increases as (1/Resolution) raised to the power 2.5. Hence, a possible two-fold improvement in resolution by super resolution processing, roughly translates into a cost reduction of an imager by more than 5 times.

Super-resolution algorithms can be classified into iterative and non-iterative algorithms. Iterative algorithms are generally the preferred approach due to their numerous advantages and also since the iteration can be terminated once a solution of a reasonable quality is achieved. Non-iterative algorithms nvolves convolution operations in the spatial domain, direct inverse methods, regularizedpseudo inverse techniques [9].The existing iterative super-resolution methods include Lucy-Richardson method [10-11], MAP method [12], steepest descent, conjugate gradients[13] and projection on convex set (POCS) method[14].

This chapter considers the general problem of super-resolution restoration and image reconstruction. While our focus will be on application to PMMW imaging. This chapter is based on work presented in [15], portions of which appeared in [13,15-19]. To solve the inherent problem of poor resolution which is caused by limited aperture dimensions and the consequent diffraction limit, this chapter presents system model, analyses the theoretical research results and design specifically for PMMW imaging. Firstly, we estimate the PSF of the PMMW imaging system by a variational Bayesian blind restoration algorithm. Secondly, we focus on mainly four algorithms, including Conjugate-Gradient algorithm (CG), Adaptive Projected Landweber super-resolution algorithm(APL), and Undedicated Steerable Pyramid Transform Projected Landweber algorithm (USPTPL), and Two-step Projection Iteration Thresholding (tw-PIT) supper resolution using compressed sensing architecture. Finally, we have verified the system model and super-resolution algorithms by experiment in different plane using differentsystem.

## **2. PMMW image formulation model**

Passive millimeter-wave (PMMW) imaging is a method of forming images through the passive detection ofnaturally occurring millimeter-wave radiation from a scene.According toblackbody radiation theory[1], all substances with a finite absolute temperature will radiate electromagnetic energy. The radiated energy spectral intensitycan be described as a Brightness Temperature B*<sup>f</sup>*

$$B\_f = \frac{2lf^3}{c} \left(\frac{1}{e^{l\circ fKT} - 1}\right) \tag{1}$$

where <sup>34</sup> *h* 6.63 10 (J) is the Plank's Constant; *f* (Hz) is the frequency; <sup>23</sup> *K* 1.38 10 (J.K-1) is Boltzmann's Constant; *c* is velocity of light and the *T* represents the absolute temperature. The brightness temperature can be assigned a gray level to generate a millimeter wave image. For the convenience of analyzing, we introduce a simplified focal plane model to calculate the imaging process, as illustrated in Fig.1 and Fig.2. The noise power can be received by sensor in the data Plane or in the image plane. Thus the millimeter wave image can be formed in the data plane or in the image plane.

Fig.1 shows the imaging process of PMMW Focal Plane Array (FPA) imaging. The process is Space-variant by the non-uniformity of the antenna beam and inconsistencies of channels. Fig.2 shows the principle of imaging in different planes. System will obtain the higher resolution and rate near to video when lens or reflectors are used in focal plane real-time imaging.

In Fig.2, the propagation process from object-plane to data-plane is two-dimensional (2D) spatial Fourier transformation (FT), while the propagation process from data-plane to focalplane is 2D spatial inverse Fourier transformation (IFT). FPA system forms images in the image-plane by receiving object-radiation energy, without 2D FT and IFT in post-data processing. Thus, the system is high real-time.

According to Fresnel and Fraunhofer diffraction theory [20], the incoherent image is given by

$$\log(\mu, \upsilon) = \text{S}[\bigcup f(\mathbf{x}, y)h\_1(\mathbf{u}, \upsilon; \mathbf{x}, y) \text{dx} dy] + n(\mu, \upsilon) \tag{2}$$

where *fxy* (,) denotes the object's intensity function on the region (,) *x y* , *gxy* (,) denotes the gray level function on the region (,) *u v* , *h uvxy* <sup>1</sup> ,,, denotes the PSF (Point Spread function) of the imaging sensor, and *nuv* (,) is the noise of image plane, *S* denotes the nonuniformity of the antenna beams and inconsistencies of channels.

Fig. 1. Space-variant-model of Focal Plane Array imaging

Fig. 2. Principle of FPA imaging

260 Image Restoration – Recent Advances and Applications

indicated that the cost of an imager increases as (1/Resolution) raised to the power 2.5. Hence, a possible two-fold improvement in resolution by super resolution processing, roughly

Super-resolution algorithms can be classified into iterative and non-iterative algorithms. Iterative algorithms are generally the preferred approach due to their numerous advantages and also since the iteration can be terminated once a solution of a reasonable quality is achieved. Non-iterative algorithms nvolves convolution operations in the spatial domain, direct inverse methods, regularizedpseudo inverse techniques [9].The existing iterative super-resolution methods include Lucy-Richardson method [10-11], MAP method [12], steepest descent, conjugate gradients[13] and projection on convex set (POCS) method[14]. This chapter considers the general problem of super-resolution restoration and image reconstruction. While our focus will be on application to PMMW imaging. This chapter is based on work presented in [15], portions of which appeared in [13,15-19]. To solve the inherent problem of poor resolution which is caused by limited aperture dimensions and the consequent diffraction limit, this chapter presents system model, analyses the theoretical research results and design specifically for PMMW imaging. Firstly, we estimate the PSF of the PMMW imaging system by a variational Bayesian blind restoration algorithm. Secondly, we focus on mainly four algorithms, including Conjugate-Gradient algorithm (CG), Adaptive Projected Landweber super-resolution algorithm(APL), and Undedicated Steerable Pyramid Transform Projected Landweber algorithm (USPTPL), and Two-step Projection Iteration Thresholding (tw-PIT) supper resolution using compressed sensing architecture. Finally, we have verified the system model and super-resolution algorithms by

Passive millimeter-wave (PMMW) imaging is a method of forming images through the passive detection ofnaturally occurring millimeter-wave radiation from a scene.According toblackbody radiation theory[1], all substances with a finite absolute temperature will radiate electromagnetic energy. The radiated energy spectral intensitycan be described as a

> <sup>3</sup> 2 1 1 *<sup>f</sup> hf KT*

where <sup>34</sup> *h* 6.63 10 (J) is the Plank's Constant; *f* (Hz) is the frequency; <sup>23</sup> *K* 1.38 10 (J.K-1) is Boltzmann's Constant; *c* is velocity of light and the *T* represents the absolute temperature. The brightness temperature can be assigned a gray level to generate a millimeter wave image. For the convenience of analyzing, we introduce a simplified focal plane model to calculate the imaging process, as illustrated in Fig.1 and Fig.2. The noise power can be received by sensor in the data Plane or in the image plane. Thus the millimeter wave image can be formed in the data plane or in the image plane.

Fig.1 shows the imaging process of PMMW Focal Plane Array (FPA) imaging. The process is Space-variant by the non-uniformity of the antenna beam and inconsistencies of channels. Fig.2 shows the principle of imaging in different planes. System will obtain the higher

(1)

*c e* 

*hf <sup>B</sup>*

translates into a cost reduction of an imager by more than 5 times.

experiment in different plane using differentsystem.

**2. PMMW image formulation model** 

Brightness Temperature B*<sup>f</sup>*

Practically, the system model above can be improved appropriately according to certain conditions. For example, if the gain consistency of the channels is well, the model can be expressed space-variant model

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 263

The analytical continuation theory is the theoretical basis for to achieve super-resolution. Analytic continuation theory includes two aspects, 1) The Fourier transform of any airspace bounded function is analytic functions. 2) For any analytic function, as long as we can accurately know the part information in a limited range , we can uniquely determine the entire function. Given the values of analytic function in a range, the overall reconstruction of

In passive millimeter wave imaging signal processing, using accurate point spread function (PSF) is important for getting high quality restored image. In the process of imaging, the PSF is decided by antenna beam, atmospheric transmission, system noise etc. So the real PSF can not be substituted by a simple model. We can't use a simple model to replace the real PSF. And the PSF will change due to the specific imaging environment so that it is difficult to acquire the exact PSF. The common methods to obtain the PSF include direct measurement and model parameter estimation. In order to improve the quality of image restoration, a variational Bayesian blind restoration algorithm for PMMW imaging is proposed in this section, which combines the posterior probability model of the PMMW imaging to obtain

The variational Bayesian estimation algorithm is based on Bayesian framework [22]. The idea of this algorithm is according to the known priori information and assumptions establish a posteriori distribution model of the imaging system first. And then under a certain criterion, use the variational [23,24] method to obtain a optimal distribution which approximates with the posteriori distribution, so that reducing the complexity of the model and make the problem more analytic. Finally work out the estimation under the restriction of the cost function.

According to the Bayesian theorem, the posteriori distribution of passive millimeter wave imaging system is decided by the prior distribution of original image, the prior distribution of PSF, noise probability distribution and likelihood probability distribution. Because of the prior distribution of original image is established on its gradient value of the statistical distribution, according to the linear relationship of the above imaging model we can get

The is the gradient operator. Assuming that every point value of original image is independent, , (, ) *i j f F i j* is the gradient value of each point. The prior distribution of original image can be expressed by c-th order zero mean mixture gaussian distribution, the

> , , 1 ( ) () ( |0, ) *C i j c i j c*

Also, assuming that every point value of PSF is independent , *h Hmn m n* ( ,) is the point value of the PSF. In passive millimeter wave imaging, the receive antenna's power pattern is

*G f v*

(9)

*<sup>c</sup>* is the weight of the c-th

*i j i j c*

*PF P f*

Where *<sup>c</sup> v* is the variance of the c-th Gaussian distribution,

G( , ) ( , ) ( , ) ( , ) *xy Hxy Fxy Nxy* (8)

**3. Variational bayesian estimate the PSF of the imaging system** 

accurate point spread function by variational Bayesian estimation

the function is called analytic continuation [21].

mathematical expressed as

Gaussian distribution.

$$\mathbf{g}(\mu, \upsilon) = \mathbf{S}\_0 \left[ \iint f(\mathbf{x}, y) h(\mu, \upsilon; \mathbf{x}, y) d\mathbf{x} dy + n(\mu, \upsilon) \right] \tag{3}$$

Where *S*0 is the gain consistency of the receiver-channels. This model only considers the non-uniform effects of array antenna beams.

The imaging system is linear space-invariant when the non-uniformity of antenna beams and the inconsistencies of channels can be ignored together. Fig.2 can be shown as a simplified convolution model

$$\mathcal{S}(\mu, \upsilon) = f(\mu, \upsilon) \otimes h(\mu, \upsilon) + n(\mu, \upsilon) \tag{4}$$

Where the function *huv* (,) is the PSF of the imaging, which determines the radiant energy distribution in the image plane from a point source of radiant energy located in the object plane.

In matrix and vector form, we can write the problem as

$$g = Hf + n\tag{5}$$

Where *g* , *f* , *n* are lexicographically ordered vectors created from the observed image, original image, noise image respectively and *H* is the matrix resulting from the PSF. If the original image is represented by a *N N* matrix, these vectors represent image of <sup>2</sup> *N* 1 and the size of *H* is 2 2 *N N* .

The corresponding imaging model in frequency domain is

$$
\hat{G}(\mu, \upsilon) = \hat{H}(\mu, \upsilon)\hat{F}(\mu, \upsilon) + \hat{N}(\mu, \upsilon) \tag{6}
$$

Where *u* , *v* are the discrete frequency variables, ˆ *Guv*, , ˆ *Fuv* (,) , ˆ*N uv*, signify the FFT transform of *Gxy* (,) , *Fxy* (,) , *Nxy* (,) respectively.

The so-called image restoration problem is solving for *fxy* (,) based on the above mathematic model, namely the inverse problem of equation (5). And the inverse problem is usually bizarre or morbid. According to equation (6), there is:

$$
\hat{\mathcal{F}}(\mu, \upsilon) = \hat{\mathcal{G}}(\mu, \upsilon) / \hat{H}(\mu, \upsilon) + \hat{N}(\mu, \upsilon) / \hat{H}(\mu, \upsilon) \tag{7}
$$

From the equation (6) we know, when using the wrong or inaccurate PSF, the solution of ˆ *Fuv* (,) would be wrong. And under the effect of system noise, using the inaccurate PSF can seriously influence the imaging quality. So on condition that it's unable to accurately determine the PSF of the imaging system, the way which according to the obtained blurred image, using blind processing method to estimate more accurate PSF, and then utilizing the classic image restoration algorithm to recover the image, can improve the passive millimeter wave imaging quality.

As the ˆ *Huv* (,) is zero outside the cut-off frequency of the imaging system. Only from the frequency domain point of view, to restore high-frequency components outside the cutoff frequency appears to be impossible.

The analytical continuation theory is the theoretical basis for to achieve super-resolution. Analytic continuation theory includes two aspects, 1) The Fourier transform of any airspace bounded function is analytic functions. 2) For any analytic function, as long as we can accurately know the part information in a limited range , we can uniquely determine the entire function. Given the values of analytic function in a range, the overall reconstruction of the function is called analytic continuation [21].

## **3. Variational bayesian estimate the PSF of the imaging system**

262 Image Restoration – Recent Advances and Applications

Where *S*0 is the gain consistency of the receiver-channels. This model only considers the

The imaging system is linear space-invariant when the non-uniformity of antenna beams and the inconsistencies of channels can be ignored together. Fig.2 can be shown as a

Where the function *huv* (,) is the PSF of the imaging, which determines the radiant energy distribution in the image plane from a point source of radiant energy located in the object

Where *g* , *f* , *n* are lexicographically ordered vectors created from the observed image, original image, noise image respectively and *H* is the matrix resulting from the PSF. If the original image is represented by a *N N* matrix, these vectors represent image of <sup>2</sup> *N* 1

The so-called image restoration problem is solving for *fxy* (,) based on the above mathematic model, namely the inverse problem of equation (5). And the inverse problem is

From the equation (6) we know, when using the wrong or inaccurate PSF, the solution of

*Fuv* (,) would be wrong. And under the effect of system noise, using the inaccurate PSF can seriously influence the imaging quality. So on condition that it's unable to accurately determine the PSF of the imaging system, the way which according to the obtained blurred image, using blind processing method to estimate more accurate PSF, and then utilizing the classic image restoration algorithm to recover the image, can improve the passive millimeter

*Huv* (,) is zero outside the cut-off frequency of the imaging system. Only from the

frequency domain point of view, to restore high-frequency components outside the cutoff

non-uniform effects of array antenna beams.

In matrix and vector form, we can write the problem as

The corresponding imaging model in frequency domain is

Where *u* , *v* are the discrete frequency variables, ˆ

usually bizarre or morbid. According to equation (6), there is:

transform of *Gxy* (,) , *Fxy* (,) , *Nxy* (,) respectively.

simplified convolution model

and the size of *H* is 2 2 *N N* .

plane.

ˆ

As the ˆ

wave imaging quality.

frequency appears to be impossible.

<sup>0</sup> *<sup>g</sup>*(, ) (, )(, ; , ) ( , ) *uv S <sup>f</sup> <sup>x</sup> <sup>y</sup> huvx <sup>y</sup> dxdy nuv* (3)

*guv f uv huv nuv* (,) (,) (,) (,) (4)

ˆ ˆˆ ˆ *Guv HuvFuv Nuv* (,) (,)(,) (,) (6)

*Guv*, , ˆ

ˆ ˆˆˆ <sup>ˆ</sup> *Fuv Guv Huv Nuv Huv* ( , ) ( , )/ ( , ) ( , )/ ( , ) (7)

*g Hf n* (5)

*Fuv* (,) , ˆ*N uv*, signify the FFT

In passive millimeter wave imaging signal processing, using accurate point spread function (PSF) is important for getting high quality restored image. In the process of imaging, the PSF is decided by antenna beam, atmospheric transmission, system noise etc. So the real PSF can not be substituted by a simple model. We can't use a simple model to replace the real PSF. And the PSF will change due to the specific imaging environment so that it is difficult to acquire the exact PSF. The common methods to obtain the PSF include direct measurement and model parameter estimation. In order to improve the quality of image restoration, a variational Bayesian blind restoration algorithm for PMMW imaging is proposed in this section, which combines the posterior probability model of the PMMW imaging to obtain accurate point spread function by variational Bayesian estimation

The variational Bayesian estimation algorithm is based on Bayesian framework [22]. The idea of this algorithm is according to the known priori information and assumptions establish a posteriori distribution model of the imaging system first. And then under a certain criterion, use the variational [23,24] method to obtain a optimal distribution which approximates with the posteriori distribution, so that reducing the complexity of the model and make the problem more analytic. Finally work out the estimation under the restriction of the cost function.

According to the Bayesian theorem, the posteriori distribution of passive millimeter wave imaging system is decided by the prior distribution of original image, the prior distribution of PSF, noise probability distribution and likelihood probability distribution. Because of the prior distribution of original image is established on its gradient value of the statistical distribution, according to the linear relationship of the above imaging model we can get

$$
\nabla \mathbf{G}(\mathbf{x}, \mathbf{y}) = H(\mathbf{x}, \mathbf{y}) \otimes \nabla F(\mathbf{x}, \mathbf{y}) + \nabla N(\mathbf{x}, \mathbf{y}) \tag{8}
$$

The is the gradient operator. Assuming that every point value of original image is independent, , (, ) *i j f F i j* is the gradient value of each point. The prior distribution of original image can be expressed by c-th order zero mean mixture gaussian distribution, the mathematical expressed as

$$P(\nabla F) = \prod\_{i,j} P(f\_{ij}) = \prod\_{i,j} \sum\_{c=1}^{C} \pi\_c G(f\_{ij} \mid 0, v\_c) \tag{9}$$

Where *<sup>c</sup> v* is the variance of the c-th Gaussian distribution, *<sup>c</sup>* is the weight of the c-th Gaussian distribution.

Also, assuming that every point value of PSF is independent , *h Hmn m n* ( ,) is the point value of the PSF. In passive millimeter wave imaging, the receive antenna's power pattern is

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 265

*PH F G*

*QH F P G QH F dHd Fd*

(, , ) ( || ) log (, | )

*QH F DQP PH F G*

Due to it is uncorrelated between *P G* ( ) and <sup>2</sup> *H F* , ,

equation (13), so that we can get the ( ) *Q H opt* , the expression is:

*KL*

*KL*

iteration times, and estimate the PSF faster.

distance is:

obtained by ( ) *EQ H opt* .

scene.

(, , ) ( , , )log (, | )

 

( , , )( ) ( , , )log ( | , )( , )

(, , ) ( || ) log ( | , )( , )

*QH F DQP P G H FPH F*

*Q H PH P G H F <sup>Z</sup>*

The estimation ( ) *Q H opt* is not analytical, it has the form of ( ) exp( ( )) *i i*

use the iteration method to get the ( ) *Q H opt* . The solution procedure is as follows:

the ( ) *Q H opt* .Get the expected value of ( ) *Q H opt* so that get the estimation of the PSF.

() ( ) ( ) log log log () ( ) ( |, )

In order to minimize *DKL* , the minimum value can be deserved by partial derivate the

<sup>1</sup> ( ) ( )exp( log[ ( | , )] ) *opt QF H <sup>H</sup>*

Where *ZH* is a normalized variable. And then the optimum estimation of PSF can be

Obtain the parametric expression of *Q H*( ) from the form of the known *P H*( ) . Factor *Q H*( ) into *DKL* , and get the update equations.Use the gradient descent method to get

For PMMW imaging, the power pattern of the antenna is similar to gaussian function, so at the beginning of the iteration we could use the gaussian function as its initial value, and then through variational bayesian estimation to get more accurate PSF. That can reduce the

According to the above analysis and derivation, the process of variational bayesian blind restoration algorithm as shown in figure 3. First input blurred image and compute its gradient value. Second set the initial value of variational bayesian iteration algorithm. Then begin the variational bayesian estimation through the known priori information, when the K-L distance less than the threshold, stop the iteration and output the estimated PSF. At last use the Lucy-Richardson algorithm to restore the image, get the millimeter image of the

*PH P F P G H F*

 

 

*QH Q F Q*

*QH F QH F dHd Fd*

2

2 2 2

*Q*

2 2 2

2

 

2

(18)

*Q*

, so the new definition of K-L

*Q*

*i*

*Px a <sup>f</sup> <sup>x</sup>* , so we can

<sup>2</sup> ( , |)

(16)

(17)

*P G H FPH F*

nonnegative, its side lobe attenuate quickly, and can be treated as limited support domain. So The prior distribution of the PSF can be expressed by modified Gaussian distribution, the mathematical expressed as

$$P(H) = \prod\_{m,n \in S} P(h\_{mn}) = \prod\_{m,n \in S} G^{(\mathbb{R})}(h\_{mn} \mid 0, \upsilon\_{\mathbb{R}}) = \begin{cases} \prod\_{m,n \in S} \sqrt{\frac{2\upsilon\_{\mathbb{R}}}{\pi}} \exp(-\frac{\upsilon\_{\mathbb{R}}}{2} h\_{mn}^{-2}) & h\_{mn} \ge 0 \\ 0 & h\_{mn} < 0 \end{cases} \tag{10}$$

Where *Rv* is the variance of the Gaussian distribution, *S* is the support domain of the PSF.

The probability distribution of the system noise's gradient can be expressed as a Gaussian distribution the mean is zero and the variance is <sup>2</sup> . Due to the different scene, the variance of the noise distribution is different, so the Gamma distribution is used to simulate the variance's distribution of different scene, the mathematical expressed as

$$P(\sigma^{-2} \mid \alpha, \beta) = \text{Gamma}(\sigma^{-2} \mid \alpha, \beta) \tag{11}$$

 , is the parameters of Gamma distribution. The likelihood distribution of the system can be obtained through the formulation *G FH N* , assumption that , (, ) *i j g G i j* is the gradient of the blurred image, so the likelihood distribution is

$$P\{\nabla G \mid H, \nabla F\} = \prod\_{i,j} G(b\_{ij} \mid (\nabla F \otimes H)\_{i j \cdot}, \sigma^2) \tag{12}$$

Combine the equation (5), (6), (7), (8), the posteriori distribution of the passive millimeter wave imaging system is

$$\begin{aligned} \operatorname{P}(H, \nabla F \mid \nabla G) &\propto \operatorname{P}(\nabla G \mid H, \nabla F) \operatorname{P}(\nabla F) \operatorname{P}(H) \operatorname{P}(\sigma^{-2}) \\ &= \prod\_{i,j} \operatorname{G}(b\_{ij} \mid (\nabla F \otimes H)\_{ij}, \sigma^{2}) \prod\_{i,j} \sum\_{c=1}^{\mathcal{C}} \pi\_{c} \operatorname{G}(f\_{ij} \mid 0, \upsilon\_{c}) \\ &\prod\_{m,n} \operatorname{G}^{(\mathbb{R})}(b\_{mn} \mid 0, \upsilon\_{n}) \operatorname{Gamma}(\sigma^{-2} \mid \alpha, \beta) \end{aligned} \tag{13}$$

After getting the posteriori distribution of the system, use the variational method to find an optical distribution <sup>2</sup> (, , ) *Q HF opt* that minimizes the Kullback-Leibler (K-L) distance *DKL* (K-L distance is the value that describes the difference between two probability distributions, *DKL* is nonnegative, and ( )0 *D QP KL* if and only if *P Q* )between <sup>2</sup> *QH F* (, , ) and *PH F G* (, , ) . <sup>2</sup> *QH F* (, , ) is the approximation of the real posteriori distribution *PH F G* (, ) . Assumption that <sup>2</sup> *H F* , ,is independent of each other, namely:

$$P(H, \nabla F \mid \nabla G) = P(H \mid \nabla G)P(\nabla F \mid \nabla G) \tag{14}$$

$$Q(H, \nabla F, \sigma^{-2}) = Q(H)Q(\nabla F)Q(\sigma^{-2}) \tag{15}$$

The K-L distance between <sup>2</sup> *QH F* (, , ) and *PH F G* (, ) can be deserved by the definition of *DKL* :

264 Image Restoration – Recent Advances and Applications

nonnegative, its side lobe attenuate quickly, and can be treated as limited support domain. So The prior distribution of the PSF can be expressed by modified Gaussian distribution, the

<sup>2</sup> exp( ) 0 ( ) ( ) ( |0, ) <sup>2</sup>

Where *Rv* is the variance of the Gaussian distribution, *S* is the support domain of the PSF.

The probability distribution of the system noise's gradient can be expressed as a Gaussian

of the noise distribution is different, so the Gamma distribution is used to simulate the

2 2 *P*( |,) ( |,)

can be obtained through the formulation *G FH N* , assumption that , (, ) *i j g G i j*

Combine the equation (5), (6), (7), (8), the posteriori distribution of the passive millimeter

( , | ) ( | , )( )( )( )

2

*Gb F H Gf v*

( |0, ) ( | , )

After getting the posteriori distribution of the system, use the variational method to find an

*DKL* (K-L distance is the value that describes the difference between two probability distributions, *DKL* is nonnegative, and ( )0 *D QP KL* if and only if *P Q* )between

2 2 *QH F QHQ FQ* ( , , ) ( )( )( )

( |( ) , ) ( |0, )

*i j i j c i j c*

*C*

 

 

 

that minimizes the Kullback-Leibler (K-L) distance

*PH F G PH GP F G* ( , | ) ( | )( | ) (14)

and *PH F G* (, ) can be deserved by the

 (15)

, ( | , ) ( |( ) , ) *i j i j i j P G H F Gb F H*

*PH F G P G H FP FPHP*

, , 1 ( ) 2

*i j i j c*

*G h v Gamma*

*mn R*

is the parameters of Gamma distribution. The likelihood distribution of the system

 *Gamma* (11)

,

(10)

 

*R mn mn*

( )

variance's distribution of different scene, the mathematical expressed as

is the gradient of the blurred image, so the likelihood distribution is

,

*m n*

and *PH F G* (, , ) . <sup>2</sup> *QH F* (, , )

distribution *PH F G* (, ) . Assumption that <sup>2</sup> *H F* , ,

The K-L distance between <sup>2</sup> *QH F* (, , )

optical distribution <sup>2</sup> (, , ) *Q HF opt*

*R*

*mn mn R mn S*

2

*mn*

*h*

(12)

(13)

0 0

. Due to the different scene, the variance

*v v h h*

*R R*

2

2

is the approximation of the real posteriori

is independent of each other, namely:

mathematical expressed as

wave imaging system is

<sup>2</sup> *QH F* (, , ) 

definition of *DKL* :

 ,  , ,

*mn S mn S*

*PH Ph G h v*

distribution the mean is zero and the variance is <sup>2</sup>

$$\begin{split} \mathcal{D}\_{\text{KL}}(Q \mid P) &= \left\langle \log \left[ \frac{Q(H, \nabla F, \sigma^{-2})}{P(H, \nabla F \mid \nabla G)} \right] \right\rangle\_Q \\ &= \iiint Q(H, \nabla F, \sigma^{-2}) \log \left[ \frac{Q(H, \nabla F, \sigma^{-2})}{P(H, \nabla F \mid \nabla G)} \right] dH d\nabla F d\sigma^{-2} \\ &= \iiint Q(H, \nabla F, \sigma^{-2}) \log \left[ \frac{Q(H, \nabla F, \sigma^{-2})P(\nabla G)}{P(\nabla G \mid H, \nabla F)P(H, \nabla F)} \right] dH d\nabla F d\sigma^{-2} \end{split} \tag{16}$$

Due to it is uncorrelated between *P G* ( ) and <sup>2</sup> *H F* , , , so the new definition of K-L distance is:

$$\begin{aligned} \left| D\_{KL}(Q \mid \mid P) = \left\langle \log \left[ \frac{Q(H, \nabla F, \sigma^{-2})}{P(\nabla G \mid H, \nabla F)P(H, \nabla F)} \right] \right\rangle\_Q \\ = \left\langle \log \left[ \frac{Q(H)}{P(H)} \right] + \log \left[ \frac{Q(\nabla F)}{P(\nabla F)} \right] + \log \left[ \frac{Q(\sigma^{-2})}{P(\nabla G \mid H, \nabla F)} \right] \right\rangle\_Q \end{aligned} \tag{17}$$

In order to minimize *DKL* , the minimum value can be deserved by partial derivate the equation (13), so that we can get the ( ) *Q H opt* , the expression is:

$$Q\_{\rm opt}(H) = \frac{1}{Z\_H} P(H) \exp\{ \left\{ \log \left[ P(\nabla G \mid H, \nabla F) \right] \right\}\_{Q(\nabla F, \sigma^{-2} \mid H)} \tag{18}$$

Where *ZH* is a normalized variable. And then the optimum estimation of PSF can be obtained by ( ) *EQ H opt* .

The estimation ( ) *Q H opt* is not analytical, it has the form of ( ) exp( ( )) *i i i Px a <sup>f</sup> <sup>x</sup>* , so we can

use the iteration method to get the ( ) *Q H opt* . The solution procedure is as follows:

Obtain the parametric expression of *Q H*( ) from the form of the known *P H*( ) . Factor *Q H*( ) into *DKL* , and get the update equations.Use the gradient descent method to get the ( ) *Q H opt* .Get the expected value of ( ) *Q H opt* so that get the estimation of the PSF.

For PMMW imaging, the power pattern of the antenna is similar to gaussian function, so at the beginning of the iteration we could use the gaussian function as its initial value, and then through variational bayesian estimation to get more accurate PSF. That can reduce the iteration times, and estimate the PSF faster.

According to the above analysis and derivation, the process of variational bayesian blind restoration algorithm as shown in figure 3. First input blurred image and compute its gradient value. Second set the initial value of variational bayesian iteration algorithm. Then begin the variational bayesian estimation through the known priori information, when the K-L distance less than the threshold, stop the iteration and output the estimated PSF. At last use the Lucy-Richardson algorithm to restore the image, get the millimeter image of the scene.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 267

The image restoration effect is assessed by sum of squared differences (SSD) between

1 1 ( , ) ( ( , ) ( , )) *n n*

M, N are two images. The SSD error is smaller the recovery image is more approximate to the original one, namely the restoration effect is better. The SSD error of fig3-a is 179, the SSD error of fig3-b is 184. We can find that the SSD errors of the images which restored by

*i j SSD M N M i j N i j* 

(a) experimental PSF recovery (b) estimated PSF recovery

The second experiment is the variational bayesian blind restoration of the simulated passive millimeter wave image. Assumption that the view field is 30 50 , the 3*dB* power beam

a. The blurred image obtained through simulation is also 210 350 pixels, showed in fig6-b.

The PSF that deserved through variational bayesian estimation, showed in fig7-a, and then use the Lucy-Richardson algorithm through 20 iteration to get the recovery image, showed in fig7-b. And for PMMW image processing, we can also get the estimated PSF through the parameter model method that according to the diffractiFon cut-off characteristics of the image system, utilizing the spectrum of the blurred image to get the parameters of the gaussian-form PSF. The result showed in fig7-c. Then uses the same restoration algorithm to

<sup>3</sup>*dB* is 0.57 degree. Due to diffraction cut-off characteristics of the

*dB* , so the original image of the scene is 210 350 pixels, showed in fig6-

*dB* . Considering the follow-up image processing, we set

two kind of PSF are proximate, so verifying the correct of the estimated PSF.

2

(19)

recovery image and original scene, the formula is as follows:

Fig. 5. The cooperation of two PSF recovery effect

Fig. 6. Blurred image through simulated passive millimeter

angle of the scan antenna

pixel interval <sup>3</sup> / 4 

get the recovery image.

system, the sample interval is <sup>3</sup> / 2

Fig. 3. Variational Bayesian Blind Restoration

In order to testify the effect of variational bayesian algorithm, we hold three experimentsto verify the accurate of the variational bayesian blind restoration algorithm. Fig4-a is stimulated original image, the pixel size is 110 231 . Fig4-b is the experimental PSF, the pixel size is 21 21 . Let the experimental PSF convolute with the original image to get the blurred image which is showed in fig4-c. Utilize the variational bayesian estimate algorithm to get the PSF, the result is showed in fig4-d. Compare the estimated PSF with experimental one, we can find that the shapes are roughly same, only a few details are different.

In order to illustrate the correct of the variational bayesian estimation, we restore the image through the Lucy-Richardson algorithm using experimental PSF and estimated PSF separately. Each restoration process iterates 20 times, the recovery results are showed in fig5.

The image restoration effect is assessed by sum of squared differences (SSD) between recovery image and original scene, the formula is as follows:

$$SSD(M, N) = \sum\_{i=1}^{n} \sum\_{j=1}^{n} \left( M(i, j) - N(i, j) \right)^2 \tag{19}$$

M, N are two images. The SSD error is smaller the recovery image is more approximate to the original one, namely the restoration effect is better. The SSD error of fig3-a is 179, the SSD error of fig3-b is 184. We can find that the SSD errors of the images which restored by two kind of PSF are proximate, so verifying the correct of the estimated PSF.

(a) experimental PSF recovery (b) estimated PSF recovery

266 Image Restoration – Recent Advances and Applications

Input blurred image G

Obtained the gradient of the G

variational bayesian estimation

Estimate of H

Lucy-Richardson algorithm

Restored image output

In order to testify the effect of variational bayesian algorithm, we hold three experimentsto verify the accurate of the variational bayesian blind restoration algorithm. Fig4-a is stimulated original image, the pixel size is 110 231 . Fig4-b is the experimental PSF, the pixel size is 21 21 . Let the experimental PSF convolute with the original image to get the blurred image which is showed in fig4-c. Utilize the variational bayesian estimate algorithm to get the PSF, the result is showed in fig4-d. Compare the estimated PSF with experimental

In order to illustrate the correct of the variational bayesian estimation, we restore the image through the Lucy-Richardson algorithm using experimental PSF and estimated PSF separately.

one, we can find that the shapes are roughly same, only a few details are different.

Each restoration process iterates 20 times, the recovery results are showed in fig5.

(c) blurred image (d) estimated PSF

(a) original image (b) experimental PSF

Fig. 3. Variational Bayesian Blind Restoration

Fig. 4. Experiment 1 verify the accurate

set the initial value

The second experiment is the variational bayesian blind restoration of the simulated passive millimeter wave image. Assumption that the view field is 30 50 , the 3*dB* power beam angle of the scan antenna <sup>3</sup>*dB* is 0.57 degree. Due to diffraction cut-off characteristics of the system, the sample interval is <sup>3</sup> / 2 *dB* . Considering the follow-up image processing, we set pixel interval <sup>3</sup> / 4 *dB* , so the original image of the scene is 210 350 pixels, showed in fig6 a. The blurred image obtained through simulation is also 210 350 pixels, showed in fig6-b.

Fig. 6. Blurred image through simulated passive millimeter

The PSF that deserved through variational bayesian estimation, showed in fig7-a, and then use the Lucy-Richardson algorithm through 20 iteration to get the recovery image, showed in fig7-b. And for PMMW image processing, we can also get the estimated PSF through the parameter model method that according to the diffractiFon cut-off characteristics of the image system, utilizing the spectrum of the blurred image to get the parameters of the gaussian-form PSF. The result showed in fig7-c. Then uses the same restoration algorithm to get the recovery image.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 269

Projection Iteration Thresholding (*tw*-PIT) supper resolution using compressed sensing

(a) optical image (b) blurred image

(c) estimated PSF (d) recovery image

Stochastic super-resolution image restoration, typically a Bayesian approach, provides a flexible and convenient way to model a priori knowledge concerning the solution. Bayesian estimation methods are used when the a posteriori probability density function (PDF) of the original image can be established. Using maximum-a-posteriori estimation, we can obtain an

The Conjugate-gradient (CG) algorithm is an effective Krylov subspace method of solving an unconstrained large-scale optimization problem, which is equivalent to solve the

1 1 ( ) 2 2

( ( )) *T T grad J f H Hf H g* (22)

The *kf* is the k-th iteration estimate of the original scene, it generates a descent direction *<sup>k</sup> d* , n~ is conjugate to all previous search directions: 1 2 , ...... *<sup>n</sup> dd d* with respect to matrix *<sup>T</sup> H H* ;

*n k d H Hd* , k.=0,1,…n-1. In other words, conjugate gradient algorithm deals with

2

<sup>1</sup> min , 0 <sup>2</sup> *J Hf g f* (20)

*TT T T J f f H Hf g Hf H H* (21)

Fig. 8. Measured data restoration

exact solution of the formula.

And the gradient function is

problem of 1D searching:

that is *T T*

**4.1 Conjugate-gradient algorithm** 

quadratic minimization problem[25, 26].

<sup>2</sup> The target function can be denoted as

architecture algorithm. In this section,wepresentthe image restoration algorithms.

Fig. 7. The restoration of two methods

According to the results of the above experiment, the PSF estimated by variational bayesian method is more approximate to the antenna power pattern, the recovery image is more clearer. Also compute the SSD error between recovery image and original image. For fig7-b the SSD error is 878, and for fig7-d the SSD error is 1088. We can find that the SSD error is smaller and the recovery image is more approximate to the original scene when using the variational bayesian blind restoration algorithm rather than the parameter model method.

The third experiment uses the measured data that from PMMW imaging system to verify the algorithms availability. We use a single-channel scanning radiometer for imaging. Due to the limitation of scanning angle in pitch direction, the PMMW image isstitched together.Fig8-a is the optical image of the scene, fig8-b is the obtained millimeter wave image which is blurred, fig8-c is the estimated PSF, and fig8-d is the recovery image from variational bayesian blind restoration algorithm. We can find that the recovery image's contour and details are clearer, the imaging quality is improved effectively.

## **4. Super-resolution restoration and image reconstruction**

In order to improve the resolution of the image, some methods of image restoration will be needed. The image restoration is an inverse problem in general, which is always ill posed. Traditional de-convolution approaches restore the information of the pass band and eliminate the effect of additive noise components. Therefore, these methods have only limited resolution enhancement capabilities. Greater resolution improvements can only be achieved through a class of more sophisticated algorithms, called super-resolution algorithms, including Lucy-Richardson algorithm, Conjugate-gradient (CG) algorithm,Adaptive Projected Landweber (APL) super-resolution algorithm, Undecimated Steerable Pyramid Transform Projected Landweber (USPTPL) algorithm and Two-step Projection Iteration Thresholding (*tw*-PIT) supper resolution using compressed sensing architecture algorithm. In this section,wepresentthe image restoration algorithms.

#### Fig. 8. Measured data restoration

268 Image Restoration – Recent Advances and Applications

According to the results of the above experiment, the PSF estimated by variational bayesian method is more approximate to the antenna power pattern, the recovery image is more clearer. Also compute the SSD error between recovery image and original image. For fig7-b the SSD error is 878, and for fig7-d the SSD error is 1088. We can find that the SSD error is smaller and the recovery image is more approximate to the original scene when using the variational bayesian blind restoration algorithm rather than the parameter model method. The third experiment uses the measured data that from PMMW imaging system to verify the algorithms availability. We use a single-channel scanning radiometer for imaging. Due to the limitation of scanning angle in pitch direction, the PMMW image isstitched together.Fig8-a is the optical image of the scene, fig8-b is the obtained millimeter wave image which is blurred, fig8-c is the estimated PSF, and fig8-d is the recovery image from variational bayesian blind restoration algorithm. We can find that the recovery image's

In order to improve the resolution of the image, some methods of image restoration will be needed. The image restoration is an inverse problem in general, which is always ill posed. Traditional de-convolution approaches restore the information of the pass band and eliminate the effect of additive noise components. Therefore, these methods have only limited resolution enhancement capabilities. Greater resolution improvements can only be achieved through a class of more sophisticated algorithms, called super-resolution algorithms, including Lucy-Richardson algorithm, Conjugate-gradient (CG) algorithm,Adaptive Projected Landweber (APL) super-resolution algorithm, Undecimated Steerable Pyramid Transform Projected Landweber (USPTPL) algorithm and Two-step

contour and details are clearer, the imaging quality is improved effectively.

**4. Super-resolution restoration and image reconstruction** 

Fig. 7. The restoration of two methods

Stochastic super-resolution image restoration, typically a Bayesian approach, provides a flexible and convenient way to model a priori knowledge concerning the solution. Bayesian estimation methods are used when the a posteriori probability density function (PDF) of the original image can be established. Using maximum-a-posteriori estimation, we can obtain an exact solution of the formula.

## **4.1 Conjugate-gradient algorithm**

The Conjugate-gradient (CG) algorithm is an effective Krylov subspace method of solving an unconstrained large-scale optimization problem, which is equivalent to solve the quadratic minimization problem[25, 26].

$$\min J = \frac{1}{2} \left\| Hf - g \right\|\_{2'} \quad f > 0 \tag{20}$$

<sup>2</sup> The target function can be denoted as

$$J(f) = \frac{1}{2} f^T H^T H f - g^T H f + \frac{1}{2} H^T H \tag{21}$$

And the gradient function is

$$\text{grad}(\mathbf{J}(f)) = \mathbf{H}^T \mathbf{H} \mathbf{f} - \mathbf{H}^T \mathbf{g} \tag{22}$$

The *kf* is the k-th iteration estimate of the original scene, it generates a descent direction *<sup>k</sup> d* , n~ is conjugate to all previous search directions: 1 2 , ...... *<sup>n</sup> dd d* with respect to matrix *<sup>T</sup> H H* ; that is *T T n k d H Hd* , k.=0,1,…n-1. In other words, conjugate gradient algorithm deals with problem of 1D searching:

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 271

structure (BCCB) by the above mentioned matrix-vector computing and Fast Fourier

To verify the super-resolution capability of the CG algorithm, representative results of restoring blurred images is shown in Figs. 10a-c. The simulation image is a 256 256 synthetic image composed of a series of concentric disks with the background

The simulation image and its spectrum are shown in Fig.10a and Fig.11a, respectively. To simulate the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a circular aperture antenna which is equivalent to a low-pass filter. Zero-mean Gaussian noise is added to the blurred image to get the noisy blurred image. The blurred image and its spectrum are shown in Fig.4b and Fig.5b, respectively. The simulation

results of the CG algorithms are shown Fig.4c, and its spectrum are shown in Fig.5c.

Fig. 9. (a) Original Image, (b) Blurred Image (c) Conjugate gradient after 50 iterations

From Fig.9 and Fig.10, we can see that the super resolution capability (spectralextrapolation) of the CG algorithm is improvement remarkable. Furthermore, the CG

To evaluate the capacity of the recovery algorithm, the simplest common assessment criteria is to calculate the *L*<sup>2</sup> norms of the deviation between the original image and restore image,

Fig. 11 plots the MSE of the CG algorithms versus iteration numbers. Obviously, the CG algorithm has a fast convergence of the MSE. Generally, the MSE will be below 0.065 when the iterations number is 6. Clearly, the CG algorithm has the good convergence performance.

In the second experiment, a small number of 32x32 gun images are collected by 91.5GHz mechanically scanned mono-channel radiometer with horn antenna in the lab and around surroundings, as shown in Fig.12. The captured PMMW image is shown in Fig.12b. The restored image is shown in Fig.12c. From Fig.12, we can see that the super-resolution

2

<sup>2</sup> *<sup>M</sup>* <sup>1</sup> *<sup>k</sup> SE f f <sup>N</sup>* (29)

represents the k-th restoration result, 2 is *L*2 norms.

Transform (FFT). Thus, the CG algorithm has quick convergence.

(dark) at intensity value zero and the disks (bright) at intensity value one.

**4.1.1 Simulation and experiments** 

algorithm can effectively reduce rings effects.

where *f* denotes the ideal image, *<sup>k</sup> f*

that is the Mean Square Error (MSE), which is expressed as

performance and spatial resolution are enhanced by the algorithm.

$$f\_{k+1} = f\_k + \mathcal{A}\_k d\_k \tag{23}$$

where *<sup>k</sup>* denotes optimal size at searching directions, which is decided by min ( ); ( ) *k kk k k Jf d d Jf* .

Because the standard conjugate gradient algorithm is a linear restoration method, it has only limited capability of super-resolution, that is, spectral extrapolation. Furthermore, during each iteration, we can not guarantee the nonnegative constraint of estimated images. An interesting feature of the CG method is that it can be modified in order to take into account the additional priori-information about the solution. The information can be expressed as a number of closed convex sets.

An example of a constraint which is rather natural in many problems of image restoration is the positivity of the solution. The constraint can also be projected as a closed convex set. Thus we can impose the constraint that the image is nonnegative on the iteration. Because the projection operation is non-linear operations, it introduces frequencies beyond the passband. Thus, the modified CG algorithm has the capacity of spectrum extrapolation. It can be shown as

$$f\_P^{k+1} = P\_\mathbb{C} \left[ f\_{k+1} = f\_k + \mathcal{A}\_k d\_k \right] \tag{24}$$

where *Pc* denotes the projection operator on the constraint set*C* . Then the projection operator is given by

$$P\_C f = \begin{cases} f & \text{if } f > 0 \\ 0 & \text{if } f \le 0 \end{cases} \tag{25}$$

Other convex and closed sets include finite support constraint, band limited constraint, and spatial limited constraint[27]. The super-resolution performance of the modified CG algorithm can be verified from subsequent experiments.

Along with the increase of image size( *n n* ), the dimensions of matrix *H* is larger( 2 2 *n n* ), it is bad for calculating and storage. However, *H* is Toeplitz matrix, a circulant convolution matrix *Bh* can be produced by *H* , it has features as follow[26]:

$$B\_h = \mathcal{V}\Lambda\Lambda\_h\mathcal{V}\mathcal{V}^{-1}, \; B\_h^T = \mathcal{V}\Lambda\Lambda\_h^\*\mathcal{V}\mathcal{V}^{-1} \tag{26}$$

where

$$\mathcal{W}(m,n) = \frac{1}{L^2} \exp(-\frac{j \times 2\pi mn}{L}), m, n \in 0, 1, \dots \\ L-1, \Lambda\_h = \text{diag}\{DFT(h)\} \tag{27}$$

The relationship between W, W-1and Discrete Fourier Transform (DFT) is as follows:

$$\text{V}\text{V}p = \text{IDFT}(p(n)) \,, \text{ } \text{V}^{-1}\text{p} = \text{DFT}(p(n)) \tag{28}$$

In image processing, n is selected in the 2-power, such as 28-2128, etc. The amount of calculation of image direct iteration restoring is very great at this time. However, *H* is Toeplitz matrix(BTTB), so the matrix *H* (BTTB) can be continued to circulant matrix structure (BCCB) by the above mentioned matrix-vector computing and Fast Fourier Transform (FFT). Thus, the CG algorithm has quick convergence.

## **4.1.1 Simulation and experiments**

270 Image Restoration – Recent Advances and Applications

*k k kk* <sup>1</sup> *f f d* 

Because the standard conjugate gradient algorithm is a linear restoration method, it has only limited capability of super-resolution, that is, spectral extrapolation. Furthermore, during each iteration, we can not guarantee the nonnegative constraint of estimated images. An interesting feature of the CG method is that it can be modified in order to take into account the additional priori-information about the solution. The information can be expressed as a

An example of a constraint which is rather natural in many problems of image restoration is the positivity of the solution. The constraint can also be projected as a closed convex set. Thus we can impose the constraint that the image is nonnegative on the iteration. Because the projection operation is non-linear operations, it introduces frequencies beyond the passband. Thus, the modified CG algorithm has the capacity of spectrum extrapolation. It can be

> <sup>1</sup> [ ] *<sup>k</sup> P C k k kk f P f f*

where *Pc* denotes the projection operator on the constraint set*C* . Then the projection

0 0 *<sup>C</sup> f if f P f if f* 

Other convex and closed sets include finite support constraint, band limited constraint, and spatial limited constraint[27]. The super-resolution performance of the modified CG

Along with the increase of image size( *n n* ), the dimensions of matrix *H* is larger( 2 2 *n n* ), it is bad for calculating and storage. However, *H* is Toeplitz matrix, a circulant convolution

, *<sup>T</sup>* \* 1 *B WW h h*

0

, ( ( )) *<sup>h</sup> diag DFT h* (27)

*Wp IDFT p n* ( ( )) , <sup>1</sup> *W p DFT*( ( )) *p n* (28)

*d* (24)

(25)

(26)

1

algorithm can be verified from subsequent experiments.

2

matrix *Bh* can be produced by *H* , it has features as follow[26]:

*L L*

<sup>1</sup> *B WW h h*

<sup>1</sup> <sup>2</sup> ( , ) exp( ), , 0,1,... 1 *j mn Wmn mn L*

The relationship between W, W-1and Discrete Fourier Transform (DFT) is as follows:

In image processing, n is selected in the 2-power, such as 28-2128, etc. The amount of calculation of image direct iteration restoring is very great at this time. However, *H* is Toeplitz matrix(BTTB), so the matrix *H* (BTTB) can be continued to circulant matrix

 

*<sup>k</sup>* denotes optimal size at searching directions, which is decided by

where 

shown as

where

operator is given by

min ( ); ( ) *k kk k k Jf d d Jf* 

number of closed convex sets.

.

(23)

To verify the super-resolution capability of the CG algorithm, representative results of restoring blurred images is shown in Figs. 10a-c. The simulation image is a 256 256 synthetic image composed of a series of concentric disks with the background (dark) at intensity value zero and the disks (bright) at intensity value one.

The simulation image and its spectrum are shown in Fig.10a and Fig.11a, respectively. To simulate the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a circular aperture antenna which is equivalent to a low-pass filter. Zero-mean Gaussian noise is added to the blurred image to get the noisy blurred image. The blurred image and its spectrum are shown in Fig.4b and Fig.5b, respectively. The simulation results of the CG algorithms are shown Fig.4c, and its spectrum are shown in Fig.5c.

Fig. 9. (a) Original Image, (b) Blurred Image (c) Conjugate gradient after 50 iterations

From Fig.9 and Fig.10, we can see that the super resolution capability (spectralextrapolation) of the CG algorithm is improvement remarkable. Furthermore, the CG algorithm can effectively reduce rings effects.

To evaluate the capacity of the recovery algorithm, the simplest common assessment criteria is to calculate the *L*<sup>2</sup> norms of the deviation between the original image and restore image, that is the Mean Square Error (MSE), which is expressed as

$$MSE = \frac{1}{N^2} \sum \left| f - \tilde{f}^k \right|^2 \tag{29}$$

where *f* denotes the ideal image, *<sup>k</sup> f* represents the k-th restoration result, 2 is *L*2 norms. Fig. 11 plots the MSE of the CG algorithms versus iteration numbers. Obviously, the CG algorithm has a fast convergence of the MSE. Generally, the MSE will be below 0.065 when the iterations number is 6. Clearly, the CG algorithm has the good convergence performance.

In the second experiment, a small number of 32x32 gun images are collected by 91.5GHz mechanically scanned mono-channel radiometer with horn antenna in the lab and around surroundings, as shown in Fig.12. The captured PMMW image is shown in Fig.12b. The restored image is shown in Fig.12c. From Fig.12, we can see that the super-resolution performance and spatial resolution are enhanced by the algorithm.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 273

regularization parameter because semi-convergence holds true in the case of noisy images. Iterative methods of image restoration have an advantage over one-step methods in that the partial solution may be examined at each step of the iteration and any constraints on the

The Landweber method (successive-approximation method) [29] is the simplest iterative regularizing algorithm to solve linear ill-posed problems. Because standard Landweber algorithm is a linear restoration method, it has only limited capability of super-resolution (spectral extrapolation). Furthermore, during each iteration, we cannot guarantee the nonnegative constraint of estimated image. Other disadvantages of the Standard Landweber method are slow convergence and the difficulty of choosing proper update parameter.

Landweber algorithm is the simplest iterative method for approximating the least-square

<sup>1</sup> ( ) *k kT k f f* 

Where *<sup>T</sup> H* is the transpose of the blurring operator *H* ; superscript *k* denotes the

are given by

<sup>2</sup> <sup>0</sup> 

Because standard Landweber algorithm is a linear restoration method, it has only limited capability of super-resolution, that is, spectral extrapolation. Furthermore, during each

An interesting feature of the Landweber method is that it can be modified in order to take into account additional priori information about the solution. Fundamental of reliable estimation of high frequencies is the utilization of a priori known information during the processing iteration. In fact, since image restoration is inherently an ill-posed problem, the quality of restoration and the extent for achievable super-resolution depend on the accuracy and the amount of a priori information. As shown there, many physical constraints on the unknown object can be expressed by requiring that it belong to some given closed and convex sets. While efforts at the modeling of constraint sets and the use of these in projection-based set theoretic image recovery constitutes a popular direction for current research, it seems that the idea of combining the strong point of Landweber schemes and that of the projection-based methods has not been paid much attention to. The modified

Landweber method, also called the Projected Landweber Algorithm[31] is as follows

1 \*( ) *kk k <sup>C</sup> f P f H g Hf* 

(32)

is a relaxation parameter controlling the convergence, in order to guarantee

2 1

1 is the largest singular value of matrix H [30]. The initial guess <sup>0</sup> *f* is usually set to 0.

*H g Hf* (30)

(31)

solutions of equation (5). It is characterized by the equation

**4.2.2 Projected Landweber super-resolution algorithm** 

iteration, we cannot guarantee the nonnegative constraint of estimated image.

solution can be enforced at that time.

**4.2.1 Landweber algorithm** 

*k*th iteration;

Where the convergence of *<sup>k</sup> f* , the value of

Fig. 10. Spectrum of Fig. 10 (a) Original Image, (b) Blurred Image (c) Result after CG

Fig. 11. MSE of the CG

Fig. 12. (a) Visible Image (b) Captured Image (c) Result of CG after 50 iterations

Experiment results demonstrate that the CG super-resolution algorithm has fast convergent rate and the good spectral-extrapolation capacity. The CG algorithm improves the spatial resolution and reduce the ringing effects which are caused by regularizing the image restoration. Thus, the CG algorithm can be used in image restoration and PMMWI to enhance the super-resolution performance and eliminate most of the effects of blurring.

## **4.2 Adaptive projected Landweber super-resolution algorithm**

It is well-known that the problem of image restoration is the computation of *f* , given the image data *g* and the PSF *h* . Some iterative methods have been introduced to solve equation (5) [28]. Thebasic feature is that the number of iterations plays the role of a regularization parameter because semi-convergence holds true in the case of noisy images. Iterative methods of image restoration have an advantage over one-step methods in that the partial solution may be examined at each step of the iteration and any constraints on the solution can be enforced at that time.

## **4.2.1 Landweber algorithm**

272 Image Restoration – Recent Advances and Applications

Fig. 10. Spectrum of Fig. 10 (a) Original Image, (b) Blurred Image (c) Result after CG

(a) (b) (c)

Experiment results demonstrate that the CG super-resolution algorithm has fast convergent rate and the good spectral-extrapolation capacity. The CG algorithm improves the spatial resolution and reduce the ringing effects which are caused by regularizing the image restoration. Thus, the CG algorithm can be used in image restoration and PMMWI to enhance the super-resolution performance and eliminate most of the effects of blurring.

It is well-known that the problem of image restoration is the computation of *f* , given the image data *g* and the PSF *h* . Some iterative methods have been introduced to solve equation (5) [28]. Thebasic feature is that the number of iterations plays the role of a

Fig. 12. (a) Visible Image (b) Captured Image (c) Result of CG after 50 iterations

**4.2 Adaptive projected Landweber super-resolution algorithm** 

Fig. 11. MSE of the CG

The Landweber method (successive-approximation method) [29] is the simplest iterative regularizing algorithm to solve linear ill-posed problems. Because standard Landweber algorithm is a linear restoration method, it has only limited capability of super-resolution (spectral extrapolation). Furthermore, during each iteration, we cannot guarantee the nonnegative constraint of estimated image. Other disadvantages of the Standard Landweber method are slow convergence and the difficulty of choosing proper update parameter.

Landweber algorithm is the simplest iterative method for approximating the least-square solutions of equation (5). It is characterized by the equation

$$f^{k+1} = f^k + \tau H^T (\mathbf{g} - H \mathbf{f}^k) \tag{30}$$

Where *<sup>T</sup> H* is the transpose of the blurring operator *H* ; superscript *k* denotes the *k*th iteration; is a relaxation parameter controlling the convergence, in order to guarantee the convergence of *<sup>k</sup> f* , the value of are given by

$$0 < \tau < \frac{2}{\sigma\_1^2} \tag{31}$$

Where1 is the largest singular value of matrix H [30]. The initial guess <sup>0</sup> *f* is usually set to 0.

## **4.2.2 Projected Landweber super-resolution algorithm**

Because standard Landweber algorithm is a linear restoration method, it has only limited capability of super-resolution, that is, spectral extrapolation. Furthermore, during each iteration, we cannot guarantee the nonnegative constraint of estimated image.

An interesting feature of the Landweber method is that it can be modified in order to take into account additional priori information about the solution. Fundamental of reliable estimation of high frequencies is the utilization of a priori known information during the processing iteration. In fact, since image restoration is inherently an ill-posed problem, the quality of restoration and the extent for achievable super-resolution depend on the accuracy and the amount of a priori information. As shown there, many physical constraints on the unknown object can be expressed by requiring that it belong to some given closed and convex sets. While efforts at the modeling of constraint sets and the use of these in projection-based set theoretic image recovery constitutes a popular direction for current research, it seems that the idea of combining the strong point of Landweber schemes and that of the projection-based methods has not been paid much attention to. The modified Landweber method, also called the Projected Landweber Algorithm[31] is as follows

$$f^{k+1} = P\_{\mathbb{C}} \left[ f^k + \tau H^\*(g - Hf^k) \right] \tag{32}$$

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 275

Where *<sup>k</sup> f* denotes the first order derivative of *<sup>k</sup> f* . The algorithm hence adaptively updates

each cycle of implementation. A flow-chart depicting the various step is shown in Fig. 13.

Fig. 13. Flow-chart for implementation of Adaptive Projected Landweber algorithm

In order to verify the super-resolution capability of this algorithm mentioned above, two images have been adopted in simulation experiments, one is a 256x256 synthetic image composed of a series of concentric disks, and the other is an 32x32 gun image captured by

In the first experiment, the ideal image and its spectrum are shown in Fig.14a and Fig.16a, respectively. For simulating the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a low-pass filter that simulates a sensor with a circular aperture of diameter 8 pixels. The blurred image and its spectrum are shown in Fig.14b and Fig.15b respectively. the simulation results of these algorithms are shown Fig.14c-14e, and their spectrum are shown in Fig.15c-15e, respectively. It is clear that from Fig.14 and Fig.15 the results obtained by the Adaptive Porjected Landweber algorithm are better than the results obtained by the Porjected Landweber algorithm and the standard Landweber algorithm. The super-resolution capabilities (spectrum extrapolation) of these projectionbased methods are obvious. However, the convergence of the standard Landweber method is slow compared to the Adaptive Porjected Landweber algorithm and the Porjected Landweber algorithm, mainly because the standard Landweber algorithm is a linear

to speed up convergence and obtain improved object estimation after

the relax parameter

**4.2.4 Simulation and experiments** 

91.5 GHz mechanically scanned mono channel radiometer.

method, and it has hardly super-resolution capability.

Where *PC* denotes the projection operator onto the constraint set *C* .

An example of a constraint which is rather natural in many problems of image restoration is the positivity of the solution. We can impose the constraint that the image is nonnegative on the iteration. Because this constraint is non-linear operation, it introduces frequencies beyond the passband. The super-resolution performance of the Projected Landweber algorithm can be verified from subsequent experiments. Then the projection operator is given by

$$P\_C f = \begin{cases} f & \text{if } f > 0 \\ 0 & \text{if } f \le 0 \end{cases} \tag{33}$$

Other convex and closed set *C* include finite support constraint, band limited constraint and spatial limited constraint[27].

### **4.2.3 Adaptive projected Landweber Super-resolution alogithm**

One disadvantage of the Landweber method is slow convergence and the difficulty to choose proper update parameter. If is too large, the iterative process may diverge. If is too small, the iterative process would be slow. Therefore, it is necessary to determine a suitable for the iterative algorithm. Lie Liang and Yuanchang Xu proposed modification of this method, Adaptive Landweber method [32], in which constant is calculated adaptively in each iteration. The Adaptive Landweber algorithm has a better result and faster convergence than standard Landweber algorithm. Instead of using a constant update parameter, the Adaptive Landweber method computes the update parameter at each iteration and chooses the maximum of the computed parameter and the preset constantparameter to use in the next iteration.

Then we proposed a hybrid algorithm that attempt to combine the strong points of both Projected Landweber scheme (simplicity of execution, Super-resolution, etc.) and the Adaptive adjustments relax parameter (faster convergence rate, lower mean square error, etc.). For a brief description, each cycle of this "Adaptive Projected Landweber algorithm" consists of executing the three steps:

**Step 1.** Implement standard Landweber algorithm with initial relax parameter 0 and <sup>0</sup> *f* 0

$$f^{k+1} = f^k + \tau\_k H^T (\mathcal{g} - H f^k) \tag{34}$$

**Step 2.** Implement projection onto the convex set *C*

$$f^{k+1} = P\_{\mathbb{C}} f^{k+1} \tag{35}$$

**Step 3.** Implement relax parameter *<sup>k</sup>* <sup>1</sup> updating algorithm

$$\tau\_{k+1} = \max(\tau\_0, \frac{\left\|\nabla f^{k+1}\right\|\_2}{\left\|\nabla f^k\right\|\_2})\tag{36}$$

Where *<sup>k</sup> f* denotes the first order derivative of *<sup>k</sup> f* . The algorithm hence adaptively updates the relax parameter to speed up convergence and obtain improved object estimation after each cycle of implementation. A flow-chart depicting the various step is shown in Fig. 13.

Fig. 13. Flow-chart for implementation of Adaptive Projected Landweber algorithm

## **4.2.4 Simulation and experiments**

274 Image Restoration – Recent Advances and Applications

An example of a constraint which is rather natural in many problems of image restoration is the positivity of the solution. We can impose the constraint that the image is nonnegative on the iteration. Because this constraint is non-linear operation, it introduces frequencies beyond the passband. The super-resolution performance of the Projected Landweber algorithm can be verified from subsequent experiments. Then the projection operator is

> 0 0 *<sup>C</sup> f if f P f if f*

Other convex and closed set *C* include finite support constraint, band limited constraint

One disadvantage of the Landweber method is slow convergence and the difficulty to

too small, the iterative process would be slow. Therefore, it is necessary to determine a

adaptively in each iteration. The Adaptive Landweber algorithm has a better result and faster convergence than standard Landweber algorithm. Instead of using a constant update parameter, the Adaptive Landweber method computes the update parameter at each iteration and chooses the maximum of the computed parameter and the preset

Then we proposed a hybrid algorithm that attempt to combine the strong points of both Projected Landweber scheme (simplicity of execution, Super-resolution, etc.) and the

etc.). For a brief description, each cycle of this "Adaptive Projected Landweber algorithm"

<sup>1</sup> ( ) *kk T k*

*k k* 1 1

updating algorithm

max( , ) *k*

*f f*

1 2

2

**Step 1.** Implement standard Landweber algorithm with initial relax parameter 0

for the iterative algorithm. Lie Liang and Yuanchang Xu proposed modification

0

is too large, the iterative process may diverge. If

(faster convergence rate, lower mean square error,

*H g Hf* (34)

*<sup>C</sup> <sup>f</sup> <sup>P</sup> <sup>f</sup>* (35)

(36)

(33)

is

is calculated

and

Where *PC* denotes the projection operator onto the constraint set *C* .

**4.2.3 Adaptive projected Landweber Super-resolution alogithm** 

of this method, Adaptive Landweber method [32], in which constant

*<sup>k</sup> f f*

1 0

*k k*

 

given by

suitable

and spatial limited constraint[27].

choose proper update parameter. If

constantparameter to use in the next iteration.

**Step 2.** Implement projection onto the convex set *C*

Adaptive adjustments relax parameter

**Step 3.** Implement relax parameter *<sup>k</sup>* <sup>1</sup>

consists of executing the three steps:

<sup>0</sup> *f* 0

In order to verify the super-resolution capability of this algorithm mentioned above, two images have been adopted in simulation experiments, one is a 256x256 synthetic image composed of a series of concentric disks, and the other is an 32x32 gun image captured by 91.5 GHz mechanically scanned mono channel radiometer.

In the first experiment, the ideal image and its spectrum are shown in Fig.14a and Fig.16a, respectively. For simulating the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a low-pass filter that simulates a sensor with a circular aperture of diameter 8 pixels. The blurred image and its spectrum are shown in Fig.14b and Fig.15b respectively. the simulation results of these algorithms are shown Fig.14c-14e, and their spectrum are shown in Fig.15c-15e, respectively. It is clear that from Fig.14 and Fig.15 the results obtained by the Adaptive Porjected Landweber algorithm are better than the results obtained by the Porjected Landweber algorithm and the standard Landweber algorithm. The super-resolution capabilities (spectrum extrapolation) of these projectionbased methods are obvious. However, the convergence of the standard Landweber method is slow compared to the Adaptive Porjected Landweber algorithm and the Porjected Landweber algorithm, mainly because the standard Landweber algorithm is a linear method, and it has hardly super-resolution capability.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 277

<sup>0</sup> <sup>10</sup> <sup>20</sup> <sup>30</sup> <sup>40</sup> <sup>50</sup> <sup>60</sup> <sup>70</sup> <sup>80</sup> <sup>90</sup> <sup>100</sup> 0.11

In the second experiment, the visual image is shown in Fig.17a. The blurred image is shown in Fig.17b acquired by the PMMW radiometer. The restored images are shown in Fig.17c-17e respectively. The Adaptive Projected Landweber algorithm gives better result than standard Landweber algorithm. The results obtained by the Adaptive Projected Landweber algorithm are very similar to (slightly better than) the results obtained by the Projected Landweber algorithm. Furthermore, Gibbs rings of standard Landweber algorithm aggravate as the

Fig. 16. MSE of the Adaptive Projected Landweber(solid line). MSE of the Projected

Fig. 17. (a) Visible Image (b) Captured Image (c) Standard Landweber (d) Projected

The APL algorithm, which iteratively applies a cycle of Projected Landweber algorithm followed by a relax parameter adaptive adjustment, combines the strong points of the two approaches and hence possesses a number of implementation benefits. The adaptive update parameter aims to emphasize speed and stability. Experiment results demonstrate that the

Landweber (e) Adaptive Projected Landweber

Landweber(dash-dot line). MSE of the Standard Landweber(dash line).

0.112

iteration increases.

0.114

0.116

0.118

0.12

0.122

Fig. 14. (a) Original Image (b) Blurred (c) Standard Landweber after 100 iterations (d)Projected Landweber (e) Adaptive Projected Landweber

Fig. 15. Spectrum of Fig. 1 (a) Original Image, (b) Blurred (c) Standard Landweber after 100 iterations (d) Projected Landweber (e) Adaptive Projected Landweber

Fig. 16 plots the MSE of the three algorithms. It can be observed that the Adaptive Porjected Landweber algorithm has a faster decrease of the MSE than the other two algorithms. Also, the MSE of the Adaptive Porjected Landweber algorithm is lower than that of the other two algorithms.

276 Image Restoration – Recent Advances and Applications

Fig. 14. (a) Original Image (b) Blurred (c) Standard Landweber after 100 iterations

Fig. 15. Spectrum of Fig. 1 (a) Original Image, (b) Blurred (c) Standard Landweber after 100

Fig. 16 plots the MSE of the three algorithms. It can be observed that the Adaptive Porjected Landweber algorithm has a faster decrease of the MSE than the other two algorithms. Also, the MSE of the Adaptive Porjected Landweber algorithm is lower than that of the other two

iterations (d) Projected Landweber (e) Adaptive Projected Landweber

algorithms.

(d)Projected Landweber (e) Adaptive Projected Landweber

Fig. 16. MSE of the Adaptive Projected Landweber(solid line). MSE of the Projected Landweber(dash-dot line). MSE of the Standard Landweber(dash line).

In the second experiment, the visual image is shown in Fig.17a. The blurred image is shown in Fig.17b acquired by the PMMW radiometer. The restored images are shown in Fig.17c-17e respectively. The Adaptive Projected Landweber algorithm gives better result than standard Landweber algorithm. The results obtained by the Adaptive Projected Landweber algorithm are very similar to (slightly better than) the results obtained by the Projected Landweber algorithm. Furthermore, Gibbs rings of standard Landweber algorithm aggravate as the iteration increases.

The APL algorithm, which iteratively applies a cycle of Projected Landweber algorithm followed by a relax parameter adaptive adjustment, combines the strong points of the two approaches and hence possesses a number of implementation benefits. The adaptive update parameter aims to emphasize speed and stability. Experiment results demonstrate that the

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 279

To perform the multi-scale decomposition, we construct non-subsampled pyramids by iterated non-subsampled filter banks. For the next level, we upsample all filters by 2 in both dimensions. Therefore, they also satisfy the perfect reconstruction condition. The cascading

Seen from frequency domain, the UPT decomposes an image into different frequency bands. Suppose *N* is the order of the cascading decomposition, *HN*0 is a low frequency band and *Hn*<sup>1</sup> (1 ) *n N* corresponds to high frequency component of the *th n* stage. Let *n* be the passband of *Hn*<sup>0</sup> , we have 123 ... . If the UTP is designed to acquire low frequency component contained in , we can easily determine *N* by the inclusion relation

In above mentionProjection Landweber algorithm, essentially the property of semiconvergence indicates that the Landweber algorithm is a regularization method. Suppose a

> *Ax b*

Where *A* is an *m n* system matrix, which describes the system geometry, *x* is an 1 *n*


amplified with the increase of iterative steps *k* . The algorithm stops at the semiconvergence point when the data error reaches the magnitude of approaching error

Known from equation (36), the Landweber stops more quickly if it is used to restore an

may have not plenty iterative steps to restore or re-create the frequency components that we need. Unfortunately, signal-to-noise ratio of PMMW image is quite low due to limited integral time and bandwidth. Because of this, the PL algorithms can not provide a satisfied resolution improvement in most practical applications. The practical performance of the PL

The denoising technique can avoid the fast termination of the Landweber algorithm. Because PSF of a PMMW imaging system is approximately band-limited, a low passband filter can perform the function. By such pre-procession, the high-frequency component of

is attenuated thus the Landweber algorithm avoid the adverse effect of the amplification of

**4.3.2 Undecimated Pyramid Projected Landweber (UPPL) super-resolution algorithm** The basic strategy of image super-resolution based on the UPT is to use a band-limit frequency selection rule to construct a multilevel Pyramid-like restoration model from the

be the solution of equation (35) acquired by the Landweber after the *th k* iteration,

, the Landweber algorithm has [37]

. In the restoration of a badly contaminated image, the Landweber

Where *A* is the Moore-Penrose inverse of *A* . For a given

(38)

(39)

, the data error || || *k k x x*

is

is an 1 *m* vector of the measured data with perturbation

of the analysis part is shown in Fig. 18(b).

in frequency domain: *N N* <sup>1</sup> .

perturbed linear equation is given as below

*<sup>k</sup> x* be the solution of *Ax b* . If ( ) *b DA*

vector of the image pixels and *b*


algorithm is shown in Fig. 23(c).

image with a bigger

high-frequency noise.

 . Let *<sup>k</sup> x*

results of Adaptive Projected Landweber algorithm are better than those of standard Landweber algorithm and Projected Landweber algorithm. The Adaptive Projected Landweber Super-resolution algorithm has lower MSE and produces sharper images. These constraints and adaptive character speed up the convergence of the Landweber estimation process. The superior restoration of the object features observed in the image domain and the significant extrapolation of spatial frequencies observed in the spectral domain lead to the conclusion that the Adaptive Prejected Landweber super-resolution algorithm can be used for restoration and super-resolution processing for PMMW imaging.

#### **4.3 Undecimated steerable pyramid transform projected Landweber algorithm**

Super resolution algorithms have two tasks: restoring the spectrum components within the passband (by reversing the effects of convolution with the point spread function of imaging system) and re-create the lost frequencies due to the imposition of sensor diffraction limits by spectral extrapolation. Recently, a more sophisticated spectrum decomposition technique is adopted. Using multi-scale technique, an image is decomposed into a hierarchical manner where each level corresponds to a reduced-resolution approximation of the image. It is equivalent to a filter bank that decomposes an image into different frequency components. By such decomposition, one can restore the passband firstly and then extrapolate high frequency components stage by stage. The most commonly used multi-scale methods are based on the Pyramid transform (such as Laplacian Pyramid, steerable Pyramid), the wavelets transform, the contourlets transform and so on [33-36]. The multi-scale transforms mentioned above belong to linear transform, but they are not shift invariant due to the down sampling. The lack of shift-invariance is a problem for many applications such as image restoration and image denoising because it causes pseudo-Gibbs phenomena around singularities [33]. Undecimated multi-scale methods can avoid such problem thus it has been introduced in several studies [34-36].

#### **4.3.1 Undecimated pyramid transform**

The Undecimated Pyramid transform (UPT) uses filter bank ( *H*<sup>10</sup> , *H*<sup>11</sup> ) in the analysis part and ( *G*<sup>10</sup> , *G*<sup>11</sup> ) in the synthesis part. The ideal frequency response of the building block of the UPT is shown in Fig. 18(a). The perfect reconstruction condition is given as

$$H\_{10}G\_{10} + H\_{11}G\_{11} = 1\tag{37}$$

Filters of the UPT do not need to be orthogonal or bi-orthogonal and this lack of the need for orthogonality or bi-orthogonality is beneficial for design freedom.

Fig. 18. Undecimated Pyramid transform. (a) The structure of the two-channel undecimated filter bank (b) Two stage Pyramid decomposition.

278 Image Restoration – Recent Advances and Applications

results of Adaptive Projected Landweber algorithm are better than those of standard Landweber algorithm and Projected Landweber algorithm. The Adaptive Projected Landweber Super-resolution algorithm has lower MSE and produces sharper images. These constraints and adaptive character speed up the convergence of the Landweber estimation process. The superior restoration of the object features observed in the image domain and the significant extrapolation of spatial frequencies observed in the spectral domain lead to the conclusion that the Adaptive Prejected Landweber super-resolution algorithm can be

used for restoration and super-resolution processing for PMMW imaging.

been introduced in several studies [34-36].

**4.3.1 Undecimated pyramid transform** 

*x*

**4.3 Undecimated steerable pyramid transform projected Landweber algorithm** 

Super resolution algorithms have two tasks: restoring the spectrum components within the passband (by reversing the effects of convolution with the point spread function of imaging system) and re-create the lost frequencies due to the imposition of sensor diffraction limits by spectral extrapolation. Recently, a more sophisticated spectrum decomposition technique is adopted. Using multi-scale technique, an image is decomposed into a hierarchical manner where each level corresponds to a reduced-resolution approximation of the image. It is equivalent to a filter bank that decomposes an image into different frequency components. By such decomposition, one can restore the passband firstly and then extrapolate high frequency components stage by stage. The most commonly used multi-scale methods are based on the Pyramid transform (such as Laplacian Pyramid, steerable Pyramid), the wavelets transform, the contourlets transform and so on [33-36]. The multi-scale transforms mentioned above belong to linear transform, but they are not shift invariant due to the down sampling. The lack of shift-invariance is a problem for many applications such as image restoration and image denoising because it causes pseudo-Gibbs phenomena around singularities [33]. Undecimated multi-scale methods can avoid such problem thus it has

The Undecimated Pyramid transform (UPT) uses filter bank ( *H*<sup>10</sup> , *H*<sup>11</sup> ) in the analysis part and ( *G*<sup>10</sup> , *G*<sup>11</sup> ) in the synthesis part. The ideal frequency response of the building block of

Filters of the UPT do not need to be orthogonal or bi-orthogonal and this lack of the need for

*x*

*x*ˆ

Fig. 18. Undecimated Pyramid transform. (a) The structure of the two-channel undecimated

10 10 11 11 *HG HG* 1 (37)

1 *y*

2 *y* 3 *y*

*H*<sup>21</sup>

*H*<sup>20</sup>

*b*

*H*<sup>11</sup>

*H*<sup>10</sup>

the UPT is shown in Fig. 18(a). The perfect reconstruction condition is given as

orthogonality or bi-orthogonality is beneficial for design freedom.

1 *y*

2 *y*

*a*

*G*<sup>10</sup>

*G*<sup>11</sup>

*H*<sup>11</sup>

filter bank (b) Two stage Pyramid decomposition.

*H*<sup>10</sup>

To perform the multi-scale decomposition, we construct non-subsampled pyramids by iterated non-subsampled filter banks. For the next level, we upsample all filters by 2 in both dimensions. Therefore, they also satisfy the perfect reconstruction condition. The cascading of the analysis part is shown in Fig. 18(b).

Seen from frequency domain, the UPT decomposes an image into different frequency bands. Suppose *N* is the order of the cascading decomposition, *HN*0 is a low frequency band and *Hn*<sup>1</sup> (1 ) *n N* corresponds to high frequency component of the *th n* stage. Let *n* be the passband of *Hn*<sup>0</sup> , we have 123 ... . If the UTP is designed to acquire low frequency component contained in , we can easily determine *N* by the inclusion relation in frequency domain: *N N* <sup>1</sup> .

In above mentionProjection Landweber algorithm, essentially the property of semiconvergence indicates that the Landweber algorithm is a regularization method. Suppose a perturbed linear equation is given as below

$$A\mathfrak{x} = b^{\delta} \tag{38}$$

Where *A* is an *m n* system matrix, which describes the system geometry, *x* is an 1 *n* vector of the image pixels and *b* is an 1 *m* vector of the measured data with perturbation . Let *<sup>k</sup> x* be the solution of equation (35) acquired by the Landweber after the *th k* iteration, *<sup>k</sup> x* be the solution of *Ax b* . If ( ) *b DA* , the Landweber algorithm has [37]

$$\|\|\mathbf{x}\_k^\delta - \mathbf{x}\_k\|\| \le \sqrt{k\tau}\delta\tag{39}$$

Where *A* is the Moore-Penrose inverse of *A* . For a given , the data error || || *k k x x* is amplified with the increase of iterative steps *k* . The algorithm stops at the semiconvergence point when the data error reaches the magnitude of approaching error || || *<sup>k</sup> x x* , where *x Ab* .

Known from equation (36), the Landweber stops more quickly if it is used to restore an image with a bigger . In the restoration of a badly contaminated image, the Landweber may have not plenty iterative steps to restore or re-create the frequency components that we need. Unfortunately, signal-to-noise ratio of PMMW image is quite low due to limited integral time and bandwidth. Because of this, the PL algorithms can not provide a satisfied resolution improvement in most practical applications. The practical performance of the PL algorithm is shown in Fig. 23(c).

The denoising technique can avoid the fast termination of the Landweber algorithm. Because PSF of a PMMW imaging system is approximately band-limited, a low passband filter can perform the function. By such pre-procession, the high-frequency component of is attenuated thus the Landweber algorithm avoid the adverse effect of the amplification of high-frequency noise.

## **4.3.2 Undecimated Pyramid Projected Landweber (UPPL) super-resolution algorithm**

The basic strategy of image super-resolution based on the UPT is to use a band-limit frequency selection rule to construct a multilevel Pyramid-like restoration model from the

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 281

In comparison with PL algorithm, two images have been adopted in simulation experiments, one is a 256x256 synthetic image composed of a series of disks, and the other is an 32x32 gun image captured by 91.5 GHz mechanically scanned mono channel radiometer.

(a) (b)

(c) (d)

In the first experiment, the ideal image and its spectrum are shown in Fig. 20(a) and Fig. 20(b), respectively. For simulating the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a low-pass filter that simulates a sensor with a circular aperture of diameter 16 pixels. Zero-mean Gaussian noise was added to the blurred image to get the observed noisy blurred image at 30 dB Blurred Signal-to-Noise Ratio (BSNR). The blurred image and its spectrum are shown in Fig. 20(c) and Fig. 20(d),

algorithm are shown in Fig. 21(a) and 21(c) after 200 iterations, and their spectrum are shown in Fig. 21(b) and 21(d), respectively. It is clear that the results obtained by the UPPL algorithm are better than the results of the PL algorithm. The super-resolution capability

Fig. 22 plots the MSE of the two algorithms. It is clear that the UPPL algorithm has a faster decrease of the MSE than the PL algorithm. The MSE of UPPL algorithm is lower than that

0.4 respectively. The simulation results of the PL algorithm and UPPL

 1.5 , 2 1.2 ,

Fig. 20. (a) Original Image, (b) Original Image Spectrum, (c) Blurred Image, (d) Blurred

In the experiment, scale number of UPT is 3. The update parameters are <sup>1</sup>

(spectrum extrapolation) of the UPPL algorithm is more obvious.

**4.3.3 Simulation and experiments** 

Image Spectrum

respectively.

1.0 and 4

of the PL algorithm.

3 

Pyramid representations of the original data. The UPT of an image can be described as collection of low- or band-pass copies of an original image. For a band-limited image, the decomposition of UPT can implement the function of image denoising.

The framework of the UPPL algorithm is shown in Fig. 19. First the UPPL algorithm decomposes a PMMW images by the UPT. Sub images *yy y* 12 1 , ,, *<sup>j</sup>* acquired by the decomposition correspond to the frequency bands shown in Fig. 18(b). Then the UPPL restores these frequency components from the lowest frequency band to the highest. Because the UPT has the capability to attenuate noise in higher frequency bands, the UPPL is able to improve the restoration of lower frequency bands by providing the PL algorithm plenty iterate steps. In each stage the initial guess of the current PL iterations is the restored result of the last stage. So the restoration of current frequency band is always based on a sufficient restoration of lower frequency components.

Fig. 19. The UPPL algorithm framework

The UPPL algorithm is summarized as below:


In each stage, the UPPL restores a frequency band by an independent PL algorithm, which has its own update parameter. Known from equation (36), a big update parameter accelerates the amplification of the data error || || *k k x x* thus it decreases the number of iterative steps. Because of this, the UPPL enjoys the flexibility of controlling the convergence speed of different frequency bands. The principle for choosing a suitable *<sup>j</sup>* is that the number of iterative steps should be sufficient but not excessive. Commonly the concrete value of *<sup>j</sup>* is acquired by a number of experiments.

## **4.3.3 Simulation and experiments**

280 Image Restoration – Recent Advances and Applications

Pyramid representations of the original data. The UPT of an image can be described as collection of low- or band-pass copies of an original image. For a band-limited image, the

The framework of the UPPL algorithm is shown in Fig. 19. First the UPPL algorithm decomposes a PMMW images by the UPT. Sub images *yy y* 12 1 , ,, *<sup>j</sup>* acquired by the decomposition correspond to the frequency bands shown in Fig. 18(b). Then the UPPL restores these frequency components from the lowest frequency band to the highest. Because the UPT has the capability to attenuate noise in higher frequency bands, the UPPL is able to improve the restoration of lower frequency bands by providing the PL algorithm plenty iterate steps. In each stage the initial guess of the current PL iterations is the restored result of the last stage. So the restoration of current frequency band is always based on a

**Initial guess** 

**PL** 

**PL** 

, <sup>0</sup>

denotes the restored image of the ( 1)*th j* scale,

2 *y*

<sup>1</sup>*f* 0 and 1 1 *g y* .

, <sup>0</sup> <sup>1</sup>

1 *Kj <sup>j</sup> <sup>j</sup> f f* 

thus it decreases the number of

is that the

**PL** 

decomposition of UPT can implement the function of image denoising.

1*y*

*<sup>J</sup>* <sup>1</sup> *y*

**Step 1.** Compute the UPT of the input image for *J* levels *yy y* 12 1 , ,, *<sup>J</sup>* .

**Step 3.** Implement *Kj* PL iterations with initial update parameter *<sup>j</sup>*

1 *Kj <sup>j</sup> f*

speed of different frequency bands. The principle for choosing a suitable *<sup>j</sup>*

In each stage, the UPPL restores a frequency band by an independent PL algorithm, which has its own update parameter. Known from equation (36), a big update parameter

iterative steps. Because of this, the UPPL enjoys the flexibility of controlling the convergence

number of iterative steps should be sufficient but not excessive. Commonly the concrete

**Step 2.** Implement *K*1 PL iterations with initial update parameter <sup>1</sup>

sufficient restoration of lower frequency components.

**UPT PMMW** 

Fig. 19. The UPPL algorithm framework

The UPPL algorithm is summarized as below:

and *jjj* <sup>1</sup> *gyg* , where 1

**Step 4.** Repeat step 3 to the highest scale of *J* 1 .

accelerates the amplification of the data error || || *k k x x*

is acquired by a number of experiments.

*j* 1,2, ,*J* .

value of *<sup>j</sup>* 

**image** 

**Super-resolution result** 

In comparison with PL algorithm, two images have been adopted in simulation experiments, one is a 256x256 synthetic image composed of a series of disks, and the other is an 32x32 gun image captured by 91.5 GHz mechanically scanned mono channel radiometer.

Fig. 20. (a) Original Image, (b) Original Image Spectrum, (c) Blurred Image, (d) Blurred Image Spectrum

In the first experiment, the ideal image and its spectrum are shown in Fig. 20(a) and Fig. 20(b), respectively. For simulating the blurring caused by a diffraction limited imaging sensor, we blur the ideal image by the PSF of a low-pass filter that simulates a sensor with a circular aperture of diameter 16 pixels. Zero-mean Gaussian noise was added to the blurred image to get the observed noisy blurred image at 30 dB Blurred Signal-to-Noise Ratio (BSNR). The blurred image and its spectrum are shown in Fig. 20(c) and Fig. 20(d), respectively.

In the experiment, scale number of UPT is 3. The update parameters are <sup>1</sup> 1.5 , 2 1.2 , 3 1.0 and 4 0.4 respectively. The simulation results of the PL algorithm and UPPL algorithm are shown in Fig. 21(a) and 21(c) after 200 iterations, and their spectrum are shown in Fig. 21(b) and 21(d), respectively. It is clear that the results obtained by the UPPL algorithm are better than the results of the PL algorithm. The super-resolution capability (spectrum extrapolation) of the UPPL algorithm is more obvious.

Fig. 22 plots the MSE of the two algorithms. It is clear that the UPPL algorithm has a faster decrease of the MSE than the PL algorithm. The MSE of UPPL algorithm is lower than that of the PL algorithm.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 283

(a) (b)

(c) (d)

Undecimated steerable pyramid decomposition of the image generates a number of different sub-band frequency images. As these details and the differences between the useful signals and noise in the approximate sub-band images, we can use different relaxation parameters and iterations in order to suppress the noise. The UPPL algorithm combines the advantages of the PL algorithms and multi-level pyramid recovery methods. The acquired image are first decomposed by non-sampling pyramid transform into some sub-images. Then we operate the super-resolution process to each grade low-pass image using the PL algorithm. Since each sub-image contains different frequency content, we select different relaxation parameters and iterations in the super-resolution processing. The algorithm improves the ability of spectral extrapolation, and makes the restored

**4.4 Two-step projection iteration thresholding supper-resolution using compressed** 

For passive millimeter wave image, it has certain structure, which can be sparse decomposed in a particular base. Therefore, we can use the sparse prior information of PMMW images for the image reconstruction process. Because the noise is not sparse, the supper-resolution based on sparse prior information can suppress noise. The low-pass effect of the PMMW imaging system make the space resolution very poor. The *tw*-PIT algorithm uses the sparse prior information and the non-negative finite value information of PMMW images, which can separate the noise from the signals in the iteration process. The algorithm is very effective when the image noise is existed which can also has good

Fig. 23. (a) Visible Image, (b) Captured Image, (c) Result of PL, (d) Result of UPPL.

image more sharpen.

**sensing architecture** 

super-resolution performance.

Fig. 21. (a) Result of PL, (b) Spectrum of PL, (c) Result of UPPL, (d) Spectrum of UPPL

Fig. 22. MSE vs Iterations

In the second experiment, the visual image is shown in Fig. 23(a). The blurred image is shown in Fig. 23(b) acquired by a 91.5 GHz mechanically scanned radiometer. We use the same experimental parameter as the first experiment. The restored images are shown in Fig. 23(c) and 23(d) respectively. It is clear that a significant resolution improvement is achieved by the UPPL algorithm.

A reasonable frequency decomposition scheme, the UPT is also presented. Experiment results demonstrate that the result of UPPL is better than that of the PL algorithm. The UPPL algorithm has lower MSE and produces sharper images. The effectiveness of the UPPL algorithm for practical PMMW images is also validated.

282 Image Restoration – Recent Advances and Applications

Fig. 21. (a) Result of PL, (b) Spectrum of PL, (c) Result of UPPL, (d) Spectrum of UPPL

<sup>0</sup> <sup>20</sup> <sup>40</sup> <sup>60</sup> <sup>80</sup> <sup>100</sup> <sup>120</sup> <sup>140</sup> <sup>160</sup> <sup>180</sup> <sup>200</sup> 0.045

In the second experiment, the visual image is shown in Fig. 23(a). The blurred image is shown in Fig. 23(b) acquired by a 91.5 GHz mechanically scanned radiometer. We use the same experimental parameter as the first experiment. The restored images are shown in Fig. 23(c) and 23(d) respectively. It is clear that a significant resolution improvement is achieved

A reasonable frequency decomposition scheme, the UPT is also presented. Experiment results demonstrate that the result of UPPL is better than that of the PL algorithm. The UPPL algorithm has lower MSE and produces sharper images. The effectiveness of the

iterations

PL algorithm UPPL algorithm

0.05

UPPL algorithm for practical PMMW images is also validated.

0.055

0.06

MSE

Fig. 22. MSE vs Iterations

by the UPPL algorithm.

0.065

0.07

0.075

Fig. 23. (a) Visible Image, (b) Captured Image, (c) Result of PL, (d) Result of UPPL.

Undecimated steerable pyramid decomposition of the image generates a number of different sub-band frequency images. As these details and the differences between the useful signals and noise in the approximate sub-band images, we can use different relaxation parameters and iterations in order to suppress the noise. The UPPL algorithm combines the advantages of the PL algorithms and multi-level pyramid recovery methods. The acquired image are first decomposed by non-sampling pyramid transform into some sub-images. Then we operate the super-resolution process to each grade low-pass image using the PL algorithm. Since each sub-image contains different frequency content, we select different relaxation parameters and iterations in the super-resolution processing. The algorithm improves the ability of spectral extrapolation, and makes the restored image more sharpen.

### **4.4 Two-step projection iteration thresholding supper-resolution using compressed sensing architecture**

For passive millimeter wave image, it has certain structure, which can be sparse decomposed in a particular base. Therefore, we can use the sparse prior information of PMMW images for the image reconstruction process. Because the noise is not sparse, the supper-resolution based on sparse prior information can suppress noise. The low-pass effect of the PMMW imaging system make the space resolution very poor. The *tw*-PIT algorithm uses the sparse prior information and the non-negative finite value information of PMMW images, which can separate the noise from the signals in the iteration process. The algorithm is very effective when the image noise is existed which can also has good super-resolution performance.

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 285

According the definition of C, *<sup>H</sup> CI K K* is a strictly positive definite matrix. So function *sur f a* (;) is a strictly convex function for any value a. For image degradation, we can always

*f a Kf Ka f a*

To acquire the solution of the equation above, the process of iteration is shown as follows.

, arg min( ( ; ) 1,2,... *n sur <sup>n</sup>*

In equation (43), when **w 0** , it means the objective function doesn't include any sparse priori information. Therefore, the algorithm is degraded to a process of landweber iteration

2 ( ) 2( [ ( ] ) *<sup>H</sup> f w sign f a K g Ka*

 

>0, the solution is:

 

 

, when *f*

 

[ ( )] / 2 *<sup>H</sup> f a K g Ka w*

[ ( )] / 2 *<sup>H</sup> f a K g Ka w*

otherwise, when the equation do not satisfy the two conditions mentioned above, which is

*<sup>w</sup> f S a K g Ka*

 

*w i*

*S x if x w*

,1

In conclusion, the iterating thresholding algorithm based on sparse prior is described as

,1( [ ( )] ) *<sup>H</sup>*

() 0 / 2

 

*x if x w*

*x if x w*

/ 2 / 2

*i*

*i*

/ 2 / 2

2 2

*H H p*

 

*w p f f <sup>f</sup> <sup>n</sup>* (46)

(47)

(48)

(49)

(51)

(52)

<0, the solution turns to:

(50)

(45)

[ 2( ) ]

1

*f f a K g K Ka w f*

have *K* 1 . So let C equal to value 1, and we can replace objective function by:

2 2 2

 

*g a Ka*

When p=1, the variational equation of the objective function is expressed as:

 

, ,

 

 

For the process of symbolic function, when *f*

Under the condition of [ ( )] / 2 *<sup>H</sup> a K g Ka w* 

0 *f*

is defined as follow:

*sur wp wp*

For any initial point <sup>0</sup> *f* ,

algorithm.

follow

Where *Sw* ,1 

2

( ;)

#### **4.4.1 Projection Iteration Threshold (PIT) supper-resolution**

In the compressed sensing(CS) theory, we use sparsity to describe a signal's feature or structure. Passive millimeter wave(PMMW) images have a certain structure so that they can have sparse representation on some basis. Therefore, during the processing of superresolution, we can use the prior that the PMMW images can be sparse represented to reconstruct them. And in image restoration, the introduction of l1-norm optimization can proform a more effective way to recover a original image.

In the CS theory, we take advantage of the signal's sparsity by bringing p-norm restrict condition in solving the objective function. So far, using iterative way to solve nonlinear reconstruct problem has achieved remarkable results in the field of CS. So in the field of image restoration, we can also bring in p-norm restriction to obtain images' sparse prior information. The process of image degradation as shown in section 2, for the model of PMMW imaging system which is supposed space-invariable, the operator K is simplified a convolution process. And the solution of this problem requires minimizing the difference between the optimal solution and the real one:

$$
\Delta f = \left\| Kf - g \right\|^2 \tag{40}
$$

Unlike classical regularization way, we add regularization to the sparse prior information of image, where the constraint is not quadric, but the lp-norm( 1 2 *p* ) of signal f. It is here that the introduction of p-norm make the solution of objective function have sparsity.

The objective function to be optimized is:

$$\begin{split} \left\| \Phi\_{w,p}(f) = \Delta(f) + \sum\_{\lambda \in \Gamma} w\_{\gamma} \left| \left\langle f, \phi\_{\gamma} \right\rangle \right|^{p} \\ = \left\| Kf - g \right\|^{2} + \sum\_{\lambda \in \Gamma} w\_{\gamma} \left| \left\langle f, \phi\_{\gamma} \right\rangle \right|^{p} \end{split} \tag{41}$$

Where is the orthogonal basis which f could be sparse represented with, and ( ) *w w* is the weight of the coefficients in transform-domain. When K is a identitymatrix, and is a wavelet basii function, the objective function mentioned above turns to denoising via wavelet transform based on Besov priori information .

Then, the corresponding variational equation is:

$$\forall \gamma \in \Gamma: \quad \left\langle K^H K f, \rho\_\gamma \right\rangle - \left\langle K^H g, \rho\_\gamma \right\rangle + \frac{w\_\gamma p}{2} \left| \left\langle f, \rho\_\gamma \right\rangle \right|^{p-1} \operatorname{sign}(\left\langle f, \rho\_\gamma \right\rangle) = 0 \tag{42}$$

The nonlinear equation shown above which involved symbolic function is a tricky one in practice. Define constant C satisfies:

$$\left\|\mathbf{K}^{H}\mathbf{K}\right\| < \mathbf{C} \tag{43}$$

And the function as follow:

$$\text{svar}(f; a) = \mathbb{C} \left\| f - a \right\|^2 - \left\| Kf - Ka \right\|^2 \tag{44}$$

According the definition of C, *<sup>H</sup> CI K K* is a strictly positive definite matrix. So function *sur f a* (;) is a strictly convex function for any value a. For image degradation, we can always have *K* 1 . So let C equal to value 1, and we can replace objective function by:

$$\begin{split} \boldsymbol{\Phi}\_{w,p}^{sur} &= \boldsymbol{\Phi}\_{w,p} \left( f; \boldsymbol{a} \right) - \left\| \boldsymbol{K}f - \boldsymbol{K}\boldsymbol{a} \right\|^2 + \left\| f - \boldsymbol{a} \right\|^2 \\ &= \sum\_{\boldsymbol{\mathcal{V}}} \left\| f\_{\boldsymbol{\mathcal{V}}}^2 - 2f\_{\boldsymbol{\mathcal{V}}} \left( \boldsymbol{a} + \boldsymbol{K}^H \boldsymbol{g} - \boldsymbol{K}^H \boldsymbol{K} \boldsymbol{a} \right)\_{\boldsymbol{\mathcal{V}}} + \boldsymbol{w}\_{\boldsymbol{\mathcal{V}}} \left| f\_{\boldsymbol{\mathcal{V}}} \right|^p \right) \\ &+ \left\| \boldsymbol{g} \right\|^2 + \left\| \boldsymbol{a} \right\|^2 + \left\| \boldsymbol{K} \boldsymbol{a} \right\|^2 \end{split} \tag{45}$$

To acquire the solution of the equation above, the process of iteration is shown as follows. For any initial point <sup>0</sup> *f* ,

$$f^n = \arg\!-\min\{\Phi^{sur}\_{w,p}(f; f^{n-1})\} \quad n = 1, 2, \dots \tag{46}$$

In equation (43), when **w 0** , it means the objective function doesn't include any sparse priori information. Therefore, the algorithm is degraded to a process of landweber iteration algorithm.

When p=1, the variational equation of the objective function is expressed as:

$$\mathcal{Q}f\_{\gamma} + w\_{\gamma} \text{sign}(f\_{\gamma}) = \mathcal{Q}(a\_{\gamma} + [K^H(\mathcal{g} - Ka)\_{\gamma}])\tag{47}$$

For the process of symbolic function, when *f*>0, the solution is:

$$f\_{\gamma} = a\_{\gamma} + [K^H (g - \text{Ka})]\_{\gamma} - w\_{\gamma} \tag{48}$$

Under the condition of [ ( )] / 2 *<sup>H</sup> a K g Ka w* , when *f*<0, the solution turns to:

$$f\_{\gamma} = a\_{\gamma} + [K^H (g - Ka)]\_{\gamma} + w\_{\gamma} \tag{49}$$

otherwise, when the equation do not satisfy the two conditions mentioned above, which is

$$f\_{\gamma} = 0\tag{50}$$

In conclusion, the iterating thresholding algorithm based on sparse prior is described as follow

$$f\_{\gamma} = \mathbf{S}\_{w\_{\gamma}, 1}(\mathbf{a}\_{\gamma} + [\mathbf{K}^H(\mathbf{g} - \mathbf{K}a)]\_{\gamma})\tag{51}$$

Where *Sw* ,1 is defined as follow:

284 Image Restoration – Recent Advances and Applications

In the compressed sensing(CS) theory, we use sparsity to describe a signal's feature or structure. Passive millimeter wave(PMMW) images have a certain structure so that they can have sparse representation on some basis. Therefore, during the processing of superresolution, we can use the prior that the PMMW images can be sparse represented to reconstruct them. And in image restoration, the introduction of l1-norm optimization can

In the CS theory, we take advantage of the signal's sparsity by bringing p-norm restrict condition in solving the objective function. So far, using iterative way to solve nonlinear reconstruct problem has achieved remarkable results in the field of CS. So in the field of image restoration, we can also bring in p-norm restriction to obtain images' sparse prior information. The process of image degradation as shown in section 2, for the model of PMMW imaging system which is supposed space-invariable, the operator K is simplified a convolution process. And the solution of this problem requires minimizing the difference

Unlike classical regularization way, we add regularization to the sparse prior information of image, where the constraint is not quadric, but the lp-norm( 1 2 *p* ) of signal f. It is here

2

a wavelet basii function, the objective function mentioned above turns to denoising via

The nonlinear equation shown above which involved symbolic function is a tricky one in

*Kf g w f*

is the orthogonal basis which f could be sparse represented with, and ( ) *w w*

: , , , (, )0 <sup>2</sup> *<sup>p</sup> H H w p K Kf K g f sign f* 

,

 

 

*p*

1

*sur*(;) *f a C f a Kf Ka* (44)

*<sup>H</sup> KK C* (43)

 

 

 

 

2 2

(42)

(41)

 

is

*p*

() () ,

is the weight of the coefficients in transform-domain. When K is a identitymatrix, and

*f f wf*

that the introduction of p-norm make the solution of objective function have sparsity.

<sup>2</sup> *f Kf g* (40)

**4.4.1 Projection Iteration Threshold (PIT) supper-resolution** 

proform a more effective way to recover a original image.

between the optimal solution and the real one:

The objective function to be optimized is:

Where ,

*w p*

wavelet transform based on Besov priori information .

Then, the corresponding variational equation is:

And the function as follow:

practice. Define constant C satisfies:

$$S\_{w\_r,1}(\mathbf{x}) = \begin{cases} \mathbf{x} + \mu / 2 & \text{if } |\mathbf{x}\_i < -w / 2 \\ 0 & \text{if } |\mathbf{x}\_i| \le w / 2 \\ \mathbf{x} - \mu / 2 & \text{if } |\mathbf{x}\_i > w / 2 \end{cases} \tag{52}$$

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 287

1 1 ( ) *k k <sup>C</sup> f P*

Passive millimeter wave images can be sparsely represented. Assuming the orthogonal basis

*f* 

A flow chart depicting the various steps of Projected Iterating Thresh-holding (PIT) super-

(a) (b)

(c) (d)

Fig. 25. (a) Original Image, (b) BlurredImage, (c) After Projected Landweber , (d) After *tw*PIT

1 1 1

 

*k k k*

1 1

*k k*

*k k*

/ 2 / 2

 

(55)

(56)

(54)

is the coefficient of

 

 

 

/ 2 / 2

1 1

is the sparse coefficient in orthogonal basis.

( ,) 0 / 2

2. Computing soft threshold by the coefficients acquired above

*u*

 

Where the definition of operator *Su* is soft threshold operation, and

is 1 2 [ , ,..., ] *<sup>N</sup>* , the sparsity of image is expressed as follows:

the image's sparse representation on a certain orthogonal basis function.

*S*

3. Projecting onto a non-negative limited convex set

Where, *f* is the scene signal,

resolution algorithm is shown as Fig. 24.

Fig. 24. Flow-chart for PIT super-resolution algorithm

The sparse prior iterating thresholding algorithm will achieve a better effect in image restoration, which is introducted l1-norm restricted condition and the sparse priori information. And in the PMMW imaging system, the non-negative limitation of the image can be used as a prior in image super-resolution process. So we add that in our algorithm, and we have the sparse prior super-resolution algorithm restricted by l1-norm – projected iterating thresholding(PIT) algorithm. The iterative formula is shown as follows:

1. Updating sparse coefficients of image by a additive term

$$
\boldsymbol{\theta}^{k+1} = \boldsymbol{\theta}^k + \left(\boldsymbol{H}\boldsymbol{\Psi}\right)^T \left(\boldsymbol{\varrho} - \boldsymbol{H}\boldsymbol{\Psi}\boldsymbol{\theta}^k\right) \tag{53}
$$

2. Computing soft threshold by the coefficients acquired above

$$\boldsymbol{\theta}^{k+1} = \boldsymbol{S}\_{\boldsymbol{u}}(\boldsymbol{\theta}^{k+1}, \boldsymbol{\mu}) = \begin{cases} \boldsymbol{\theta}^{k+1} + \boldsymbol{\mu} / 2 & \boldsymbol{\theta}^{k+1} < -\boldsymbol{\mu} / 2 \\ 0 & \left| \boldsymbol{\theta}^{k+1} \right| \le \boldsymbol{\mu} / 2 \\ \boldsymbol{\theta}^{k+1} - \boldsymbol{\mu} / 2 & \boldsymbol{\theta}^{k+1} > \boldsymbol{\mu} / 2 \end{cases} \tag{54}$$

3. Projecting onto a non-negative limited convex set

286 Image Restoration – Recent Advances and Applications

<sup>0</sup> *f* 0

The sparse prior iterating thresholding algorithm will achieve a better effect in image restoration, which is introducted l1-norm restricted condition and the sparse priori information. And in the PMMW imaging system, the non-negative limitation of the image can be used as a prior in image super-resolution process. So we add that in our algorithm, and we have the sparse prior super-resolution algorithm restricted by l1-norm – projected

<sup>1</sup> ( )( ) *kk T k*

 

*H gH* (53)

iterating thresholding(PIT) algorithm. The iterative formula is shown as follows:

Fig. 24. Flow-chart for PIT super-resolution algorithm

1. Updating sparse coefficients of image by a additive term

$$f^{k+1} = P\_{\mathbb{C}}(\Psi \,\theta^{k+1})\tag{55}$$

Where the definition of operator *Su* is soft threshold operation, and is the coefficient of the image's sparse representation on a certain orthogonal basis function.

Passive millimeter wave images can be sparsely represented. Assuming the orthogonal basis is 1 2 [ , ,..., ] *<sup>N</sup>* , the sparsity of image is expressed as follows:

$$f = \Psi' \theta \tag{56}$$

Where, *f* is the scene signal, is the sparse coefficient in orthogonal basis.

A flow chart depicting the various steps of Projected Iterating Thresh-holding (PIT) superresolution algorithm is shown as Fig. 24.

Fig. 25. (a) Original Image, (b) BlurredImage, (c) After Projected Landweber , (d) After *tw*PIT

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 289

*Tw*-*PIT* supper-resolution algorithm can finish one iteration though the above four steps.

In order to verify the super-resolution capability of the*tw*-*PIT*algorithm mentioned above, two images have been adopted in simulation and experiments, one is a lena image, and the other is an 210×96 University of ElectronicScience and Technology of China (UESTC) library's image captured by 94 GHz mechanically scanned mono channel radiometer.

In the first experiment, the optical lena image and the blurred image are shown in Fig.25a and Fig.25b respectively. The blurred image is from the lena image via a 5×5 Gaussian templates convolution. Zero-mean Gaussian white noise is added to the blurred image. The noise variance is 10. The Blurred Signal-to-Noise Ratio (BSNR) is 30 dB of the noisy blurred

> , 10 2

It is clear from Fig.25 that the results obtained by the *tw*PIT algorithm are better than the

When the ideal image is available for comparison, various distance metrics can be readily postulated to compare the images and their spectra. Straightforward measure is the Mean

It can be observed that the *tw*PIT algorithm has a faster decrease of the MSE than the PL algorithm and PIT algorithm from Fig.27. Also, the MSE of the *tw*PIT algorithm is lower than that of the other two algorithms. Clearly, the *tw*PIT algorithm has the best convergence performance and lowest MSE. This is because the *tw*PIT algorithm suppress the noise in an iterative process. The experiments show that the *tw*PIT algorithm convergence faster, which is suitable for the required real-time applications, such as airport security and Concealed

In the second experiment, the visual image is shown in Fig.28a. The PMMW image is shown in Fig.28b, which is captured by a 3mm Passive millimeter wave focal plane linear array scanning imaging system. The restored images are shown in Fig.28c-28e, respectively. Fig.27

10 log ( ) *i j yi j yi j MN*

Landweber and *tw*PIT are shown in Fig.25c and Fig.25d. This metric is given by[42]

Where, *yij gij nij* ,,, is the noise free blurred image, *y*( ,) *mn E y* , <sup>2</sup>

1 is 0.1. The results obtained by the Projected

<sup>2</sup>

2

<sup>2</sup> *<sup>M</sup>* <sup>1</sup> *<sup>k</sup> SE f f <sup>N</sup>* (62)

represents the th *<sup>k</sup>* restoration result, 2 is *L*2 norms.

<sup>1</sup> (, ) (, )

1 is much smaller.

(61)

is the

The algorithm converges very fast. The convergence is much faster with

**4.4.3 Simulation and experiments** 

image. The experimental parameters

results obtained by the PL algorithm.

Square Error (MSE), which given by

Where *f* denotes the ideal image, *<sup>k</sup> f*

Fig. 26 plots the MSE of the three algorithms.

additive noise variance.

Weapons Detection.

*BSNR*

But it does not guarantee optimal performance of the reconstruction algorithm.

After the above steps, one iteration is finished. Depending on the specific situation, the algorithm can perform multiple iterations until reaching design requirement. The PIT algorithm taking advantage of the sparse prior of the image, can have a better recovery performance. Because of the isotropy, the noise has no sparsity in transform-domain. Since we keep projecting to the l1-ball in every iteration, the algorithm can effectively eliminate the impact of the noise's amplification. We will verify it in the later simulation.

In one iteration, there are two convolutions of images, two wavelet transform operations, two plus-minus operation and thresholding operation. The convolutions of images can perform in frequency-domain, which could reduce calculations by using FFT, and wavelet transform could achieve quickly by using Mallat algorithm.

### **4.4.2 tw-PIT supper-resolution**

To improve the above algorithm, we can modify equation (52) by updating algorithm twostep projection iteration [38-41]. Thus we can get two-step Projected Iterating Threshholding, which is *tw*-*PIT* supper-resolution.

For a brief description, each cycle of this "*tw*-*PIT* supper-resolution algorithm" consists of executing the four steps:

**Step 1.** The iterative principal process

$$
\boldsymbol{\theta}^{k+1} = \boldsymbol{\theta}^k + \left(\boldsymbol{H}\boldsymbol{\Psi}\right)^T \left(\boldsymbol{\varrho} - \boldsymbol{H}\boldsymbol{\Psi}\boldsymbol{\theta}^k\right) \tag{57}
$$

Where *H* is the PSF of system, *g* is the Captured PMMW image, is the sparse coefficient in orthogonal basis .

**Step 2.** Soft threshold procedure:

$$\boldsymbol{\theta}\_{i}^{k+1} = \begin{vmatrix} \boldsymbol{\theta}\_{i}^{k} + \boldsymbol{\mu} / \, \mathbf{2} & \boldsymbol{\theta}\_{i}^{k} < -\boldsymbol{\mu} / \, \mathbf{2} \\ \boldsymbol{0} & \left| \boldsymbol{\theta}\_{i}^{k} \right| \le \boldsymbol{\mu} / \, \mathbf{2} \\ \boldsymbol{\theta}\_{i}^{k} - \boldsymbol{\mu} / \, \mathbf{2} & \boldsymbol{\theta}\_{i}^{k} > \boldsymbol{\mu} / \, \mathbf{2} \end{vmatrix} \tag{58}$$

is the given parameters.

**Step 3.** Updating algorithm two-step projection iteration

$$f^{k+1} = (1 - a)f^{k-1} + (a - \beta)f^k + \beta \Psi \theta^{k+1} \tag{59}$$

The designation "two-step" stems from the fact that depends on both *<sup>k</sup> f* and *<sup>k</sup>* <sup>1</sup> *f* , rather than only on *<sup>k</sup> f* .

**Step 4.** Projection process

$$f^{k+1} = P\_C(f^{k+1})\tag{60}$$

Where <sup>2</sup> <sup>1</sup> 2 2 , 1 1 *<sup>N</sup>* , 1 1 , 1 *<sup>N</sup> N N* , <sup>1</sup> can be 0.1,0.01,0.001,0.0001, *et al*. *Tw*-*PIT* supper-resolution algorithm can finish one iteration though the above four steps. The algorithm converges very fast. The convergence is much faster with 1 is much smaller. But it does not guarantee optimal performance of the reconstruction algorithm.

## **4.4.3 Simulation and experiments**

288 Image Restoration – Recent Advances and Applications

After the above steps, one iteration is finished. Depending on the specific situation, the algorithm can perform multiple iterations until reaching design requirement. The PIT algorithm taking advantage of the sparse prior of the image, can have a better recovery performance. Because of the isotropy, the noise has no sparsity in transform-domain. Since we keep projecting to the l1-ball in every iteration, the algorithm can effectively eliminate

In one iteration, there are two convolutions of images, two wavelet transform operations, two plus-minus operation and thresholding operation. The convolutions of images can perform in frequency-domain, which could reduce calculations by using FFT, and wavelet

To improve the above algorithm, we can modify equation (52) by updating algorithm twostep projection iteration [38-41]. Thus we can get two-step Projected Iterating Thresh-

For a brief description, each cycle of this "*tw*-*PIT* supper-resolution algorithm" consists of

<sup>1</sup> ( )( ) *kk T k*

*k k i i k k i i*

 

*k k i i*

1 1 <sup>1</sup> (1 ) ( ) *k k kk ff f*

The designation "two-step" stems from the fact that depends on both *<sup>k</sup> f* and *<sup>k</sup>* <sup>1</sup> *f* , rather

1 1 ( ) *k k*

1 , 1 *<sup>N</sup> N*

,

 

    

 

 

 (59)

*<sup>C</sup> f Pf* (60)

<sup>1</sup> can be 0.1,0.01,0.001,0.0001, *et al*.

/2 /2

/2 /2

 

0 / 2

 

*H gH* (57)

is the sparse coefficient

(58)

the impact of the noise's amplification. We will verify it in the later simulation.

transform could achieve quickly by using Mallat algorithm.

Where *H* is the PSF of system, *g* is the Captured PMMW image,

1

, 1

 

*N* 

**Step 3.** Updating algorithm two-step projection iteration

<sup>2</sup> <sup>1</sup> 2 2 , 1 1 *<sup>N</sup>*

 

**4.4.2 tw-PIT supper-resolution** 

executing the four steps:

in orthogonal basis .

holding, which is *tw*-*PIT* supper-resolution.

**Step 1.** The iterative principal process

**Step 2.** Soft threshold procedure:

is the given parameters.

than only on *<sup>k</sup> f* .

Where

**Step 4.** Projection process

In order to verify the super-resolution capability of the*tw*-*PIT*algorithm mentioned above, two images have been adopted in simulation and experiments, one is a lena image, and the other is an 210×96 University of ElectronicScience and Technology of China (UESTC) library's image captured by 94 GHz mechanically scanned mono channel radiometer.

In the first experiment, the optical lena image and the blurred image are shown in Fig.25a and Fig.25b respectively. The blurred image is from the lena image via a 5×5 Gaussian templates convolution. Zero-mean Gaussian white noise is added to the blurred image. The noise variance is 10. The Blurred Signal-to-Noise Ratio (BSNR) is 30 dB of the noisy blurred image. The experimental parameters 1 is 0.1. The results obtained by the Projected Landweber and *tw*PIT are shown in Fig.25c and Fig.25d. This metric is given by[42]

$$BSNR = 10\log\_{10}\frac{\frac{1}{MN}\sum\_{i,j}\left[y(i,j) - \overline{y}(i,j)\right]^2}{\sigma^2} \tag{61}$$

Where, *yij gij nij* ,,, is the noise free blurred image, *y*( ,) *mn E y* , <sup>2</sup> is the additive noise variance.

It is clear from Fig.25 that the results obtained by the *tw*PIT algorithm are better than the results obtained by the PL algorithm.

When the ideal image is available for comparison, various distance metrics can be readily postulated to compare the images and their spectra. Straightforward measure is the Mean Square Error (MSE), which given by

$$MSE = \frac{1}{N^2} \sum \left| f - \tilde{f}^k \right|^2 \tag{62}$$

Where *f* denotes the ideal image, *<sup>k</sup> f* represents the th *<sup>k</sup>* restoration result, 2 is *L*2 norms. Fig. 26 plots the MSE of the three algorithms.

It can be observed that the *tw*PIT algorithm has a faster decrease of the MSE than the PL algorithm and PIT algorithm from Fig.27. Also, the MSE of the *tw*PIT algorithm is lower than that of the other two algorithms. Clearly, the *tw*PIT algorithm has the best convergence performance and lowest MSE. This is because the *tw*PIT algorithm suppress the noise in an iterative process. The experiments show that the *tw*PIT algorithm convergence faster, which is suitable for the required real-time applications, such as airport security and Concealed Weapons Detection.

In the second experiment, the visual image is shown in Fig.28a. The PMMW image is shown in Fig.28b, which is captured by a 3mm Passive millimeter wave focal plane linear array scanning imaging system. The restored images are shown in Fig.28c-28e, respectively. Fig.27

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 291

<sup>1</sup> (,) (2 1)(2 1) *P Q*

Where, (, ) *i j* indicates the location of the image. , *P Q* is the local variance calculation

2. The equally spaced intervals are created between the maximum local variance and minimum local variance. The each local variance is enclosed into the appropriate interval. The local variance average of which contains the largest number of extents is used as the image noise variance. For the simulation image and the actual millimeter

Fig. 28. (a) Visible Image, (b) Captured PMMW image, (c) Result of Projected Landweber,

result of PL, PIT and *tw*PIT are shown in Fig.28c-28e after 20 iterations, respectively.

algorithm, while the 2 *P Q* and the window size of the local variance is 5×5.

In our experiments, the three kinds of super-resolution algorithm including PL, PIT and *tw*PIT are executed for low resolution PMMW images.The symmetric extension technology is adopted in the image boundaries convolution for eliminating shock ringing. The restored

The calculated local noise variance is 55.3552, 24.2451 and 34.6997 by PL, PIT and *tw*PIT

From above the results, we can see that PL algorithm can effectively perform super-resolution processing, but the noise is magnified in the recovery process. PIT and *tw*PIT algorithm can

*k Pl Q <sup>y</sup> i kj l P Q*

(64)

 

*<sup>y</sup>* is the local mean, which is defined as:

*y*

wave image, we set the block number is 100.

(d) Result of PIT, (e) Result of *tw*PIT

The 

window size.

is the PSF which is estimated by variational bayesian. In captured image, the image are assembled together by three times mechanically scanned mono-channel radiometer, as a single mechanical scan pitch direction field limit.

Fig. 26. Mean square error of the reconstructed image

Fig. 27. Variational bayesian estimated PSF

The MSE is a global evaluation criterion for super-resolution algorithm. We can use the local image variance method to estimate the image noise level. This approach is based on the assumption, which the image is a large number of small pieces of uniform composition, and image noise to additive noise based. The captured passive millimeter wave images meet this requirement, which there are many flat background and noise is mainly additive noise, basically meet the above assumptions. Local variance calculated as follows [43]:

1. The image is divided into many small block, such as 4×4, 5×5. Then, we calculated the local mean and local variance of each block. The local variance is defined as:

$$\sigma\_y^2(i,j) = \frac{1}{(2P+1)(2Q+1)} \sum\_{k=-P}^{P} \sum\_{l=-Q}^{Q} \left[ y(i+k, j+l) - \mu\_y(i,j) \right]^2 \tag{63}$$

The *<sup>y</sup>* is the local mean, which is defined as:

290 Image Restoration – Recent Advances and Applications

is the PSF which is estimated by variational bayesian. In captured image, the image are assembled together by three times mechanically scanned mono-channel radiometer, as a

The MSE is a global evaluation criterion for super-resolution algorithm. We can use the local image variance method to estimate the image noise level. This approach is based on the assumption, which the image is a large number of small pieces of uniform composition, and image noise to additive noise based. The captured passive millimeter wave images meet this requirement, which there are many flat background and noise is mainly additive noise,

1. The image is divided into many small block, such as 4×4, 5×5. Then, we calculated the

<sup>2</sup> <sup>1</sup> <sup>2</sup> (, ) [ ( , ) ( , )] (2 1)(2 1) *P Q y y k Pl Q i j yi k j l i j P Q*

 

(63)

basically meet the above assumptions. Local variance calculated as follows [43]:

local mean and local variance of each block. The local variance is defined as:

single mechanical scan pitch direction field limit.

Fig. 26. Mean square error of the reconstructed image

Fig. 27. Variational bayesian estimated PSF

$$\mu\_y = \frac{1}{(2P+1)(2Q+1)} \sum\_{k=-P}^{P} \sum\_{l=-Q}^{Q} y(i+k, j+l) \tag{64}$$

Where, (, ) *i j* indicates the location of the image. , *P Q* is the local variance calculation window size.

2. The equally spaced intervals are created between the maximum local variance and minimum local variance. The each local variance is enclosed into the appropriate interval. The local variance average of which contains the largest number of extents is used as the image noise variance. For the simulation image and the actual millimeter wave image, we set the block number is 100.

Fig. 28. (a) Visible Image, (b) Captured PMMW image, (c) Result of Projected Landweber, (d) Result of PIT, (e) Result of *tw*PIT

In our experiments, the three kinds of super-resolution algorithm including PL, PIT and *tw*PIT are executed for low resolution PMMW images.The symmetric extension technology is adopted in the image boundaries convolution for eliminating shock ringing. The restored result of PL, PIT and *tw*PIT are shown in Fig.28c-28e after 20 iterations, respectively.

The calculated local noise variance is 55.3552, 24.2451 and 34.6997 by PL, PIT and *tw*PIT algorithm, while the 2 *P Q* and the window size of the local variance is 5×5.

From above the results, we can see that PL algorithm can effectively perform super-resolution processing, but the noise is magnified in the recovery process. PIT and *tw*PIT algorithm can

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 293

parameter *p* is set to 0.5. All the algorithms are iterative 100 times, the experimental results

is settled to 2, the

is chosen as 1. For *tw*-PIT algorithm, the parameter

 (d) (e)

(d) The denoised image (e) After tw-PIT super-resolution

iterative process, while maintaining a high-resolution images.

Fig. 30. (a) Visible Image, (b) The visual image under the fog, (c) Captured PMMW image,

The *tw*PIT algorithm is introduced for the PMMW supper resolution process. The update equation depends on the two previous estimates (thus, the term two-step), rather than only on the previous one. This class contains and extends the iterative thresholding methods recently introduced. This algorithm uses the sparse prior information and the non-negative finite value information of PMMW images, which can separate the noise from the signals in the iteration process. Our algorithm is very effective when the image noise is existed, which can also has good super-resolution performance. The experimental results have shown that PL, PIT and *tw*PIT algorithm can effectively be super-resolution processing, but the noise is magnified in the PL algorithm recovery process. The PIT and *tw*PIT algorithm can effectively suppress high frequency noise smooth the region and maintain its flatness in an

And the *tw*PIT algorithm can in fact be tuned to converge much faster than PL and PIT algorithm, specially in severely ill-conditioned problems. The *tw*PIT algorithm has minimum MSE, which restore original signal more accurate. The reason is that noise is restrained in the iteration process. Sparse prior super-resolution algorithm has good MSE performance is mainly due to which can separate the noise in the iterative process and

(b) (c)

(a)

parameter

as shown below:

effectively suppress high frequency noise and make the flat region of original image maintain its flatness in the iterative process and maintain high-resolution of the images.

In the second experiment, the scene is relatively simple millimeter-wave imaging image processing. The Point Spread Function of the imaging system is estimated by the variational Bayesian method. For the PIT algorithm, the parameter is chosen as 1. For *tw*-PIT algorithm, the parameter is settled to 2, the parameter *p* is set to 0.5. All the algorithms are iterative 100 times, the experimental results as shown below: the covered the car visual image is shown in Fig. 29(a),the unobstructed visual image is shown in Fig. 29(b). The obtainedPMMW image is shown in Fig. 29(c), which is acquired by a W band mechanically scanned radiometer. The spectrum of the PMMW image is shown in Fig. 29(d). The result and its spectrum after100 iterations PL algorithm are shown in Fig. 29(e) and Fig. 29(f). The result and its spectrum after100 iterations PIT are shown in Fig. 29(g) and Fig.29 h). The result and its spectrum after100 iterations *tw*-PIT are shown in Fig. 29(i) and Fig. 29(j).

Fig. 29. (a) Visible Image the covered the car, (b) The unobstructed visual image, (c) (d) Captured PMMW image and its spectrum, (e) (f)Result of Projected Landweber and its spectrum, (g) (h) Result of PIT and its spectrum , (i) (j)Result of twPIT and its spectrum

In the third experiment, we process the obtained PMMW image is the UESTC Qingshuihe campus's Classroom Building, as shown in Fig.30a-30e. The Point Spread Function of the system is estimated by the variational Bayesian method. For the PIT algorithm, the 292 Image Restoration – Recent Advances and Applications

effectively suppress high frequency noise and make the flat region of original image maintain

In the second experiment, the scene is relatively simple millimeter-wave imaging image processing. The Point Spread Function of the imaging system is estimated by the variational

iterative 100 times, the experimental results as shown below: the covered the car visual image is shown in Fig. 29(a),the unobstructed visual image is shown in Fig. 29(b). The obtainedPMMW image is shown in Fig. 29(c), which is acquired by a W band mechanically scanned radiometer. The spectrum of the PMMW image is shown in Fig. 29(d). The result and its spectrum after100 iterations PL algorithm are shown in Fig. 29(e) and Fig. 29(f). The result and its spectrum after100 iterations PIT are shown in Fig. 29(g) and Fig.29 h). The result and its spectrum after100 iterations *tw*-PIT are shown in Fig. 29(i) and Fig. 29(j).

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Fig. 29. (a) Visible Image the covered the car, (b) The unobstructed visual image, (c) (d) Captured PMMW image and its spectrum, (e) (f)Result of Projected Landweber and its spectrum, (g) (h) Result of PIT and its spectrum , (i) (j)Result of twPIT and its spectrum

In the third experiment, we process the obtained PMMW image is the UESTC Qingshuihe campus's Classroom Building, as shown in Fig.30a-30e. The Point Spread Function of the system is estimated by the variational Bayesian method. For the PIT algorithm, the

(i) (j)

is settled to 2, the parameter *p* is set to 0.5. All the algorithms are

is chosen as 1. For *tw*-PIT

its flatness in the iterative process and maintain high-resolution of the images.

Bayesian method. For the PIT algorithm, the parameter

algorithm, the parameter

parameter is chosen as 1. For *tw*-PIT algorithm, the parameter is settled to 2, the parameter *p* is set to 0.5. All the algorithms are iterative 100 times, the experimental results as shown below:

Fig. 30. (a) Visible Image, (b) The visual image under the fog, (c) Captured PMMW image, (d) The denoised image (e) After tw-PIT super-resolution

The *tw*PIT algorithm is introduced for the PMMW supper resolution process. The update equation depends on the two previous estimates (thus, the term two-step), rather than only on the previous one. This class contains and extends the iterative thresholding methods recently introduced. This algorithm uses the sparse prior information and the non-negative finite value information of PMMW images, which can separate the noise from the signals in the iteration process. Our algorithm is very effective when the image noise is existed, which can also has good super-resolution performance. The experimental results have shown that PL, PIT and *tw*PIT algorithm can effectively be super-resolution processing, but the noise is magnified in the PL algorithm recovery process. The PIT and *tw*PIT algorithm can effectively suppress high frequency noise smooth the region and maintain its flatness in an iterative process, while maintaining a high-resolution images.

And the *tw*PIT algorithm can in fact be tuned to converge much faster than PL and PIT algorithm, specially in severely ill-conditioned problems. The *tw*PIT algorithm has minimum MSE, which restore original signal more accurate. The reason is that noise is restrained in the iteration process. Sparse prior super-resolution algorithm has good MSE performance is mainly due to which can separate the noise in the iterative process and

Super–Resolution Restoration and Image Reconstruction for Passive Millimeter Wave Imaging 295

Zelong Xiao, Jianzhong Xu, Shusheng Peng, et al. "Super resolution image restoration of a

L. C. Li, Research on Passive Millimeter Wave Imaging Theory and Super-resolution Signal

L. C. Li, J. Y. Yang, J. T. Xiong, et al., "W Band Dicke-Radiometer For Imaging," Int J

L. C. Li, J. Y. Yang, et al., "Method of passive mmW image detection and identification for

L. C. Li, J. Y. Yang, et al., Research on 3mm radiometric imaging, Journal of infrared and

L. C. Li, J. Y. Yang, et al., An Focal Plane Array Space Variant Model For PMMW Imaging,

Goodman, J.W. Introduction to Fourier Optics 2nd.(McGraw-Hill Book Co, New York, 1996) Su Bing-hua, Jin Wei-qi, Niu Li-hong, NIU Li-hong, LIU Guang-rong. Super-resolution

Steven M. Kay.Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory

Sina Farsiu, M. Dirk Robinson,and et al. "Fast and Robust Multiframe Super Resolution",

A. Chambolle. "Total variation minimization and a class of binary MRF models", Proc. Int.

Kyung Sub Joo, Tamal Bose, and Guo Fang Xu. Image restoration using a conjugate gradient

Zou Mouyan. Deconvolution and Signal Recovery [M. Beijing: National Defence Industry

J. Biemond, R. L. Lagendijk, R.M.Mersereau, "Iterative methods for image deblurring,"

L. Landweber. "An iteration formula for Fredholm integral equations of the first kind,"

R. W. Schafer, R. M. Mersereau, and M. A. Richards, "Constrained iterative restoration

Piana M, Bertero M. "Projected Landweber method and preconditioning," Inverse Problems,

Lei Liang and Yuanchang Xu, " Adaptive Landweber method to deblur images," IEEE

Marta Peracaula, Joan Martí, Jordi Freixenet, etc. Multi-Scale Image Analysis Applied to

Radioastronomical Interferometric Data, Pattern Recognition and Image Analysis,

Stark H. "Image recovery: theory and application," New York:Academic Press, 1987.

American Journal of Mathematics, vol 73, pp 615–624, 1951.

algorithms," Proc. IEEE, vol. 69, pp. 432-450, 1981.

Signal Processing Letters, Vol. 10, No. 5, 2003.

and Volume :Detection Theory [M]. Beijing: Publishing House of Electronics

Workshop on Energy Minimization Methods in Computer Vision and Pattern

based adaptive filtering algorithm. Circuits system signal processing, 1997,16 (2)

image restoration and progress. Optical Technique, 2001, 27(1):6-9.

IEEE Trans. On Image Process, 2004, 13(10), pp. 1327–1344.

University of Electronic Science and Technology of China, 2009.

close target," Int. J. Infrared Milli. Waves, (2011) 32: 102-115.

Astronautics, " 2006. ISSCAA 2006, 2006, 680-683

Infrared Milli Waves, 29:879–888(2008).

millimeter waves, Vol.28(1):11-15, 2009.

IET International Radar Conference 2009.

Recognition, 2005,Vol 3757,pp.136-152.

Proc.of IEEE, Vol.78, pp.856-883, 1990.

Volume 13, Issue 2, pp. 441-463,1997.

Vol. 5524, 2009, pp. 192-199.

Industry, 2003.

:197-206.

Press, 2001

PMMW sensor based on POCS algorithm. Systems and Control in Aerospace and

Processing Technology, Ph.D. dissertation, Dept. of Electronic Engineering,

minimize the impact of noise for image restoration. Thus, the *tw*PIT algorithm can be used in the occasion which requires PMMW imaging quality and real-time such as airports and Concealed Weapons Detection (CWD).

In the second experiment, due to the limited field of view, the single-channel scanning process has introduced a certain artifacts. So eliminate and abate the artifacts is the problem that need to be researched and solved in our follow-up work.

## **5. Acknowledgment**

This work is supported by the National Natural Science Foundation of China.

## **6. References**


294 Image Restoration – Recent Advances and Applications

minimize the impact of noise for image restoration. Thus, the *tw*PIT algorithm can be used in the occasion which requires PMMW imaging quality and real-time such as airports and

In the second experiment, due to the limited field of view, the single-channel scanning process has introduced a certain artifacts. So eliminate and abate the artifacts is the problem

F.T Ulaby, R.K. Moore,A.K. Fung(1981).Microwave Remote Sensing: Fundamentals and

Larry Yujiri, Merit Shoucri, and Philip Moffa.Passive Millimeter Wave Imaging.*IEEE* 

Richard J. Lang, Laurence F. Ward, John W. Cunningham. Close-range high-resolution W

Thomas Lüthi, Christian Mätzler.Stereoscopic Passive Millimeter-Wave Imaging and

Stuart E. Clark, John A. Lovberg, and Christopher A. Martin. Passive Millimeter-Wave Imaging for Airborne and Security Applications.*Proc. of SPIE*, 5077:16-21(2003). P. Moffa, L.Yujiri,H.H.Agravante,et al, "Large-aperture passive millimeter wave push

Chritopher A. Martin, Will Manning, and Vladimir G. Koliano, "Flight Test of a Passive

L. B. Lucy. "An iteration technique for the rectification of observed distributions,"

W. H. Richardson. "Bayesian-based iterative method of image restoration," Journal of the

B. R. Hunt, P. Sementilli, "Description of a poisson imagery super resolution algorithm," Astronomical Data Analysis Software and Systems 1, Ch123, pp.196-199, 1992. L. C. Li, J. Y. Yang, J. T. Xiong, et al.,Super-resolution Processing of passive millimeter wave

Millimeter-Wave Imaging System", Proc. of SPIE, 5789:24-34(2005). G.Gleed, A.H.Lettington.High-speed super-resolution techniques for passive millimeter-

Astronomical Journal, vol. 79, no.6, pp.745–754, 1974.

Optical Society of America, vol.62, pp55–59, 1972.

Electronics, Vol. 20, No. 4, 2009, pp.762–767.

*SPIE, Passive Millimeter-Wave Imaging Technology IV*, 4032:34-39(2000). Evelyn J. Boettcher, Keith Krapels, Ron Driggers, et al.Modeling passive millimeter wave

band radiometric imaging system for security screening applications.*Proceedings of* 

imaging sensor Modeling passive millimeter wave imaging sensor.*APPLIED* 

Ranging.*IEEE Transaction on Microwave Theory and Techniques*,53(8):2594-2599(2005).

broom camera", Passive Millimeter-wave Imaging Technology V, Proc.of SPIE ,

wave imaging systems. SPIE Conference on Image and Video Processing II, Vol.

image based on conjugate-gradient algorithm, Journal of Systems Engineering and

Concealed Weapons Detection (CWD).

**5. Acknowledgment** 

**6. References** 

that need to be researched and solved in our follow-up work.

Radiometry.*Artech House* I:402-403, Norwood.

*OPTICS*, 49(19) :E58-E66( 2010).

4373:1-6(2001).

2182:255-265 (1994)

This work is supported by the National Natural Science Foundation of China.

*Microwave magazine*, 1527-3342/02,September,39-50(2003).


**13** 

*México* 

**Blind Image Restoration for** 

José Luis López-Martínez1 and Vitaly Kober2 *1Mathematics School, UADY, Mérida, Yucatán, 2Computer Science Department, CICESE,* 

**a Microscanning Imaging System** 

Image restoration is an important topic in the area of image processing because its techniques are useful to recover images degraded during a capturing process (Bovik, 2005). There are a wide range of degradations in real world, such as blurring (i.e. camera motion capture process), nonuniform illumination, cloud environments (fog, clouds, smoke), noise (white noise, impulse noise, etc.), and damaged elements in imaging array sensors (Gonzalez & Woods, 2008; Hautiere & Aubert, 2005; Jain, 1989; Narasimhan & Nayar, 2002). Common restoration methods are based on *a priori* knowledge of the degradation process. They usually use the degradation model and a single observed scene to carry out restoration (Banham & Katsaggelos, 1997; Kundur & Hatzinakos, 1996). There are also methods of restoration based on unknown functions of degradation referred to as blind methods (Jain,1989). Although there are numerous algorithms, the process of restoration is still open

Image restoration methods described in this chapter belong to the class of blind adaptive methods. The techniques use camera microscanning (Shi et al., 2006) for the restoration of images degraded with nonuniform additive, nonuniform multiplicative interferences, and

The spatial nonuniform additive interference is present in infrared focal-plane array sensors (IFPA), because each photodetector has a variation in its photoresponse as intrinsic result of the IFPA's fabrication stage (Hayat et al., 1999; Ratliff et al., 2002). On the other hand, the nonuniform illumination may be characterized as multiplicative interference. Nonuniform illumination limits the performance of others algorithms of image processing such as pattern

Microscanning is a technique to acquire time-sequential images of the same scene with a slight shifting between the scene and camera. Recently, restoration methods for different models of observed scenes using three images anisotropically captured with a microscanning imaging system were investigated (López-Martínez & Kober, 2008, 2010; López-Martínez et al., 2010). In order to carry out restoration, we consider three degraded images captured with a microscanning imaging system. Next, an explicit system of equations is derived and solved.

**1. Introduction** 

problem.

sensor noise.

recognition (Lee & Kim, 2009).


## **Blind Image Restoration for a Microscanning Imaging System**

José Luis López-Martínez1 and Vitaly Kober2 *1Mathematics School, UADY, Mérida, Yucatán, 2Computer Science Department, CICESE, México* 

## **1. Introduction**

296 Image Restoration – Recent Advances and Applications

Eslami R, Radha H. Translation-invariant contourlet transform and its application to image

A. L. Cunha, J. Zhou, and M. N. Do, The non-subsampled contourlet transform: Theory,

J.-L. Starck, J. Fadili, and F. Murtagh, The undecimated wavelet decomposition and its reconstruction, IEEE Trans. Image Processing, Vol. 16, No. 2, 2007, pp. 297-308. Wang Yanfei, Computational methods for inverse problems and their applications. Higher

I Daubechies, M Defrise, C De Mol.An iterative thresholding algorithm for linear inverse

Emmanuel Candes and Terence Tao. "Near optimal signal recovery from random

J. Bioucas-Dias and M. Figueiredo, "A new TwIST: Two-step iterative

A. Beck and M. Teboulle. "A fast iterative shrinkage-thresholding algorithm for linear inverse problems". SIAM Journal on Imaging Sciences, Mar 2009, 2(1):183-202. Bahnam M A , Katsaggelos A K. Digital image restoration[J]. IEEE Signal Process. 1997,

Gao B C. An OperationalMethod for Estimating Signal to Noise Ratios from Data Acquired with Imaging Spectrometers. Remote Sensing of Environment, 1993, 43 (1):23—33.

3374.

1457.

2006, pp. 3089-3101.

2006, 52(12), pp. 1-39.

14(2): 24–41.

education press of China, 2005.

Process, 2007, 16(12), pp. 2992–3004.

denoising. IEEE Transactions on image processing, Vol. 15, No. 11, 2006, pp. 3362 –

design, and applications, IEEE Transactions on Image Processing, Vol. 15, No. 10,

problems with a sparsity constraint[J]. Comm.Pure Appl.Math., 2004, 57(11):1413-

projections Universal encoding strategies", IEEE Trans. on Information Theory,

shrinkage/thresholding algorithms for image restoration", IEEE Trans. Image

Image restoration is an important topic in the area of image processing because its techniques are useful to recover images degraded during a capturing process (Bovik, 2005). There are a wide range of degradations in real world, such as blurring (i.e. camera motion capture process), nonuniform illumination, cloud environments (fog, clouds, smoke), noise (white noise, impulse noise, etc.), and damaged elements in imaging array sensors (Gonzalez & Woods, 2008; Hautiere & Aubert, 2005; Jain, 1989; Narasimhan & Nayar, 2002).

Common restoration methods are based on *a priori* knowledge of the degradation process. They usually use the degradation model and a single observed scene to carry out restoration (Banham & Katsaggelos, 1997; Kundur & Hatzinakos, 1996). There are also methods of restoration based on unknown functions of degradation referred to as blind methods (Jain,1989). Although there are numerous algorithms, the process of restoration is still open problem.

Image restoration methods described in this chapter belong to the class of blind adaptive methods. The techniques use camera microscanning (Shi et al., 2006) for the restoration of images degraded with nonuniform additive, nonuniform multiplicative interferences, and sensor noise.

The spatial nonuniform additive interference is present in infrared focal-plane array sensors (IFPA), because each photodetector has a variation in its photoresponse as intrinsic result of the IFPA's fabrication stage (Hayat et al., 1999; Ratliff et al., 2002). On the other hand, the nonuniform illumination may be characterized as multiplicative interference. Nonuniform illumination limits the performance of others algorithms of image processing such as pattern recognition (Lee & Kim, 2009).

Microscanning is a technique to acquire time-sequential images of the same scene with a slight shifting between the scene and camera. Recently, restoration methods for different models of observed scenes using three images anisotropically captured with a microscanning imaging system were investigated (López-Martínez & Kober, 2008, 2010; López-Martínez et al., 2010). In order to carry out restoration, we consider three degraded images captured with a microscanning imaging system. Next, an explicit system of equations is derived and solved.

Blind Image Restoration for a Microscanning Imaging System 299

The additive interference and the original image are spatially displaced by the

2 1 2 1 *ri j s i j s i j f i j f i j n i j n i j i M j N* ( , ) ( , ) ( 1, ) ( , ) ( 1, ) ( , ) ( 1, ), 1 1,1 , (4)

3 1 3 1 *ci j s i j s i j f i j f i j n i j n i j i M j N* ( , ) ( , ) ( , 1) ( , ) ( , 1) ( , ) ( , 1), 1 ,1 1. (5)

21 21 2 1 21 21 2 1

(1,1) (2,1) (1,2) (2,2) (1, ) (2, ) (2,1) (3,1) (2,2) (3,2) (2, ) (3, )

 

  (6)

(7)

(8)

*ss ss s Ns N ss ss s Ns N*

 

( 1,1) ( ,1) ( 1,2) ( ,2) ( 1, ) ( , ) 00 0

(1,1) (1,2) (1,2) (1,3) (1, 1) (1, ) 0

 

 

( ,1) ( ,2) ( ,2) ( ,3) ( , 1) ( , ) 0

(, ) ( , (, ) (, ) ( , 1)

( , ) ( , 1) ,) ( , , )

**Ax u** , (9)

*Mj M j i riN fiN N f j*

1 1 2 2

where the first terms takes into account the noise information present in the most part of the image, and the last two terms are inserted to the objective function in order to take into account the noise information in the bottom row and the right column of the image, respectively. To solve the minimization problem, we differentiate the objective function with respect to elements of the image *f* (, ) *i j* and set derivatives equal to zero. The minimization of the objective function leads to a linear system of equations. In matrix-vector

where matrix **A** has the size *MN MN* , **x** is a vector version of *f* (, ) *i j* of size 1 *MN* , and vector **uu u** *r c* has the size 1 *MN* . The vectors **u***<sup>r</sup>* and **u***<sup>c</sup>* are computed as follows

( ,) (, ) (

) ( 1, )

*F ri j f f ci j f i j f i j*

*ij i j*

1 1 2 2

*s M s M s M s M s M N s MN*

 

2 12 1 2 1

31 31 3 1 31 31 3 1

(2,1) (2,2) (2,2) (2,3) (2, 1) (2, ) 0 (, ) .

*s M s M s M s M s MN s MN*

We want to minimize the additive noise variance contained in these matrices. So, the objective function to be minimized using the least-squares approach (Kay, 1993), is given as

*s s s s sN sN s s s s sN sN*

3131 3 1

(, ) ,

microscanning. Let us compute gradient matrices as follows:

and

Hence

and

*ri j*

*ci j*

1 1

*M N*

*i j*

notation the linear system is given by

1 1

*N M*

*cMj f f*

*j i*

3 3 *s ai j f i j n i j i M* (*i N* , )*j j* ( , 1) ( , ) ( , ), 1 , 1 . (3)

The proposed method is analyzed in terms of restoration accuracy, execution time, and computational complexity. Experimental results are also provided.

## **2. Image restoration methods for a microscanning system**

Microscanning acquires multiple images of the same scene by slight shifting image acquisition system (Shi et al., 2006). Microscanning can be implemented either with a controlled movement of a sensor array that captures images or with a controlled motion of a light source, for example in the case of nonuniform illumination.

Microscanning is usually used for supper-resolution (Milanfar, 2011). However, controlled camera microscanning can be also used for image restoration, if the original image and interferences are spatially displaced relatively each other during the microscanning process. At least three observed images should be captured. The first image is taken without any displacement, the second one is captured with shift of one pixel down, and finally the third one is obtained with shift of one pixel to the right. In practice, microscanning may be implemented using a piezoelectric actuator to precision positioning of a sensor array. In the case of nonuniform illumination, controlled motion performs a light source.

Let us introduce some useful notation and definitions. Let *s i <sup>t</sup>*( , ), 1,2,.., *j t T* be a set of observed images, where *t* is the index of time-sequential images captured during microscanning, *T* is the number of observed images captured around the origin with a small displacement of a camera, and (i, j) are the pixel coordinates. Without loss of generality, suppose that each image has the size of *M N* pixels. Let *f* (, ) *i j* ,*a i*(, )*j* and*b i*(, )*j* denote an original image, an additive interference, and a multiplicative interference, respectively. Assume that these images are time-invariant during the capture process. Let *n i <sup>t</sup>*(, )*j* be a time-varying zero-mean white Gaussian noise.

### **2.1 Additive degradation model**

An example of spatially nonuniform additive interference is IFPA with a low gain variation (Hayat et al., 1999; Ratliff et al., 2002). IFPA sensor is a mosaic of photodetectors placed at the focal plane of an imaging system. It is known that the performance of IFPA sensors is affected by the presence of fixed-pattern noise (spatially nonuniform noise). The nonuniform noise occurs because each detector has the photoresponse slightly different from that of its neighbors.

When image degradation is caused by additive nonuniform interference and additive noise, the observed scene can be described as

$$s\_1(i,j) = a(i,j) + f(i,j) + n\_1(i,j), \quad 1 \le i \le M, 1 \le j \le N. \tag{1}$$

With a help of the technique of microscanning, two frames with vertical and horizontal displacements of one pixel can be obtained as follows:

$$s\_2(i,j) = a(i+1,j) + f(i,j) + n\_2(i,j), \quad 1 \le i < M, 1 \le j \le N\_2 \tag{2}$$

and

$$s\_3(i,j) = a(i,j+1) + f(i,j) + n\_3(i,j), \quad 1 \le i \le M, 1 \le j < N. \tag{3}$$

The additive interference and the original image are spatially displaced by the microscanning. Let us compute gradient matrices as follows:

$$r(i,j) = s\_2(i,j) - s\_1(i+1,j) = f(i,j) - f(i+1,j) + n\_2(i,j) - n\_1(i+1,j), \quad 1 \le i \le M-1, 1 \le j \le N, \text{(4)}$$

and

298 Image Restoration – Recent Advances and Applications

The proposed method is analyzed in terms of restoration accuracy, execution time, and

Microscanning acquires multiple images of the same scene by slight shifting image acquisition system (Shi et al., 2006). Microscanning can be implemented either with a controlled movement of a sensor array that captures images or with a controlled motion of a

Microscanning is usually used for supper-resolution (Milanfar, 2011). However, controlled camera microscanning can be also used for image restoration, if the original image and interferences are spatially displaced relatively each other during the microscanning process. At least three observed images should be captured. The first image is taken without any displacement, the second one is captured with shift of one pixel down, and finally the third one is obtained with shift of one pixel to the right. In practice, microscanning may be implemented using a piezoelectric actuator to precision positioning of a sensor array. In the

Let us introduce some useful notation and definitions. Let *s i <sup>t</sup>*( , ), 1,2,.., *j t T* be a set of observed images, where *t* is the index of time-sequential images captured during microscanning, *T* is the number of observed images captured around the origin with a small displacement of a camera, and (i, j) are the pixel coordinates. Without loss of generality, suppose that each image has the size of *M N* pixels. Let *f* (, ) *i j* ,*a i*(, )*j* and*b i*(, )*j* denote an original image, an additive interference, and a multiplicative interference, respectively. Assume that these images are time-invariant during the capture process. Let

An example of spatially nonuniform additive interference is IFPA with a low gain variation (Hayat et al., 1999; Ratliff et al., 2002). IFPA sensor is a mosaic of photodetectors placed at the focal plane of an imaging system. It is known that the performance of IFPA sensors is affected by the presence of fixed-pattern noise (spatially nonuniform noise). The nonuniform noise occurs because each detector has the photoresponse slightly different

When image degradation is caused by additive nonuniform interference and additive noise,

With a help of the technique of microscanning, two frames with vertical and horizontal

1 1 *s i j ai j f i j n i j i M j N* ( , ) ( , ) ( , ) ( , ), 1 ,1 . (1)

2 2 *s ai j f i j n i* (, ) *i j* ( 1, ) ( , ) , 1 ,1 , (, ) *i j M j N* (2)

computational complexity. Experimental results are also provided.

light source, for example in the case of nonuniform illumination.

*n i <sup>t</sup>*(, )*j* be a time-varying zero-mean white Gaussian noise.

**2.1 Additive degradation model** 

from that of its neighbors.

and

the observed scene can be described as

displacements of one pixel can be obtained as follows:

case of nonuniform illumination, controlled motion performs a light source.

**2. Image restoration methods for a microscanning system** 

$$\text{rc}(i,j) = \text{s}\_3(i,j) - \text{s}\_1(i,j+1) = f(i,j) - f(i,j+1) + n\_3(i,j) - n\_1(i,j+1), \quad 1 \le i \le M, 1 \le j \le N-1. \text{ (5)}$$

Hence

$$r(i,j) = \begin{bmatrix} s\_2(1,1) - s\_1(2,1) & s\_2(1,2) - s\_1(2,2) & \cdots & s\_2(1,N) - s\_1(2,N) \\ s\_2(2,1) - s\_1(3,1) & s\_2(2,2) - s\_1(3,2) & \cdots & s\_2(2,N) - s\_1(3,N) \\ \vdots & \vdots & \vdots & \vdots \\ s\_2(M-1,1) - s\_1(M,1) & s\_2(M-1,2) - s\_1(M,2) & \cdots & s\_2(M-1,N) - s\_1(M,N) \\ 0 & 0 & \cdots & 0 \end{bmatrix},\tag{6}$$

and

$$\mathbf{c}(i,j) = \begin{bmatrix} s\_3(1,1) - s\_1(1,2) & s\_3(1,2) - s\_1(1,3) & \cdots & s\_3(1,N-1) - s\_1(1,N) & 0\\ s\_3(2,1) - s\_1(2,2) & s\_3(2,2) - s\_1(2,3) & \cdots & s\_3(2,N-1) - s\_1(2,N) & 0\\ \vdots & \vdots & \vdots & \vdots & \vdots\\ s\_3(M,1) - s\_1(M,2) & s\_3(M,2) - s\_1(M,3) & \cdots & s\_3(M,N-1) - s\_1(M,N) & 0 \end{bmatrix} \tag{7}$$

We want to minimize the additive noise variance contained in these matrices. So, the objective function to be minimized using the least-squares approach (Kay, 1993), is given as

$$\begin{aligned} \tilde{F} &= \left\{ \sum\_{i=1}^{M-1} \sum\_{j=1}^{N-1} \left[ r(i,j) - f(i,j) + f(i+1,j) \right]^2 + \left[ c(i,j) - f(i,j) + f(i,j+1) \right]^2 \right\} \\ &+ \sum\_{j=1}^{N-1} \left[ c(M,j) - f(M,j) + f(M,j+1) \right]^2 + \sum\_{i=1}^{M-1} \left[ r(i,N) - f(i,N) + f(i+N,j) \right]^2 \end{aligned} \tag{8}$$

where the first terms takes into account the noise information present in the most part of the image, and the last two terms are inserted to the objective function in order to take into account the noise information in the bottom row and the right column of the image, respectively. To solve the minimization problem, we differentiate the objective function with respect to elements of the image *f* (, ) *i j* and set derivatives equal to zero. The minimization of the objective function leads to a linear system of equations. In matrix-vector notation the linear system is given by

$$\mathbf{Ax} = \mathbf{u}\_{\prime\prime} \tag{9}$$

where matrix **A** has the size *MN MN* , **x** is a vector version of *f* (, ) *i j* of size 1 *MN* , and vector **uu u** *r c* has the size 1 *MN* . The vectors **u***<sup>r</sup>* and **u***<sup>c</sup>* are computed as follows

Blind Image Restoration for a Microscanning Imaging System 301

becomes symmetric and positive-definite. In this case there exists a unique solution. After solving the linear system, the obtained image is point-wise processed to have the same mean value (assumed to be known) with original image. To solve the linear system, iterative

The computational complexity of the algorithm is given by the execution order of conjugate gradient and the size of an image to be restored. It is estimated as *kp* operations, where *p* is the number of nonzero entries in the matrix associated to lineal system, and *k* is the number of iterations required for solving the system of equations. Without loss of generality, we assume that *M N* . Therefore, *p*= <sup>2</sup> 5*M* and *k qM* where *q* depends on precision of the solution. The computational complexity of the method can be estimated as

Impulse noise is caused by sensor failures in a camera or transmission through a noisy channel. The proposed method is able to interpolate implicitly the pixel values corrupted with impulsive noise based on the information contained in neighboring pixels. This is because, during the microscanning the information of each pixel of the original image is captured in three different observed images. If one of sensors is damaged, partial information about the pixel intensity of the original image could be available in the other

A typical example of multiplicative interference is nonuniform illumination. When an input image degraded by a multiplicative nonuniform interference and additive noise, the

Suppose that microscanning is able to separate the original image and the interference, and

Now we define two quotient matrices using spatial information between rows and columns

(, ) ( , ) , 1 , 1 , ( 1, ) 0 ( 1, ) *s ij hi j i M j N si j si j*

(, ) ( , ) , 1 , 1 , ( , 1) 0. ( , 1) *s ij vi j i M j N s ij s ij*

1 1 *s i j bi j f i j n i j i M j N* ( , ) ( , ) ( , ) ( , ), 1 ,1 (20)

2 2 *s bi j f i j n i M j N* (, ) *i j* ( 1, ) ( , ) (, , 1 *i j*) ,1 , (21)

3 3 *s bi j f i j n i* ( ,*i j*) ( , 1) ( , ) ( , ), 1 , 1 . *j iM jN* (22)

1

1

(24)

, (23)

conjugate gradient method is used (Golub & Van Loan, 1996).

<sup>3</sup> 5*qM* .

observed images.

of the images,

**2.2 Multiplicative degradation model** 

observed scene can be described as

then the shifted frames are obtained as follows:

2

3

1

1

Hence, the matrices *h i*(, )*j* and *v i*(, )*j* as defined as

$$\mathfrak{u}\_r(j) = r(1, j), \quad 1 \le j \le M \,\,\,\,\,\tag{10}$$

$$\mathbf{u}\_r(iN+j) = r(i+1,j) - r(i,j), \; 1 \le i \le M-2, \; 1 \le j \le N \; \; \; \tag{11}$$

$$\mathbf{u}\_r(NM - j) = -r(M - 1, N - j), \quad 0 \le j \le N - 1,\tag{12}$$

$$\mathbf{u}\_c \text{(} \mathbf{i}M + \mathbf{1} \text{)}=\mathbf{c} \text{(} i+1, \mathbf{1} \text{)}, \quad \mathbf{0} \le i \le M - \mathbf{1}, \tag{13}$$

$$\mathbf{u}\_c(iN+j) = c(i+1,j) - c(i+1,j-1), \, 2 \le j \le N-1, 0 \le i \le M-1,\tag{14}$$

$$\mathbf{u}\_c(iN) = -\mathbf{c}(i, N-1), \quad 1 \le i \le M. \tag{15}$$

The matrix A is sparse, and it is calculated as

$$\mathbf{A} = \begin{bmatrix} \mathbf{A}\_1 & \mathbf{A}\_3 & 0 & 0 & 0 & 0 \\ \mathbf{A}\_3 & \mathbf{A}\_2 & \mathbf{A}\_3 & 0 & 0 & 0 \\ 0 & \mathbf{A}\_3 & \mathbf{A}\_2 & \mathbf{A}\_3 & 0 & 0 \\ 0 & 0 & \ddots & \ddots & \ddots & 0 \\ 0 & 0 & 0 & \mathbf{A}\_3 & \mathbf{A}\_2 & \mathbf{A}\_3 \\ 0 & 0 & 0 & 0 & \mathbf{A}\_3 & \mathbf{A}\_1 \end{bmatrix} \tag{16}$$

where the matrices **A**<sup>1</sup> , **A**<sup>2</sup> , and **A**<sup>3</sup> , of the size *N N* , are given by

$$\mathbf{A}\_1 = \begin{bmatrix} 4 & -2 & 0 & 0 & 0 & 0 \\ -2 & 6 & -2 & 0 & 0 & 0 \\ 0 & -2 & 6 & -2 & 0 & 0 \\ 0 & 0 & \ddots & \ddots & \ddots & 0 \\ 0 & 0 & 0 & -2 & 6 & -2 \\ 0 & 0 & 0 & 0 & -2 & 4 \end{bmatrix}, \tag{17}$$

$$\mathbf{A}\_2 = \begin{bmatrix} 6 & -2 & 0 & 0 & 0 & 0 \\ -2 & 8 & -2 & 0 & 0 & 0 \\ 0 & -2 & 8 & -2 & 0 & 0 \\ 0 & 0 & \ddots & \ddots & \ddots & 0 \\ 0 & 0 & 0 & -2 & 8 & -2 \\ 0 & 0 & 0 & 0 & -2 & 6 \end{bmatrix}, \tag{18}$$

and

$$\mathbf{A}\_3 = \text{diag}\{-\mathbf{2}, -\mathbf{2}, \dots, -\mathbf{2}\}.\tag{19}$$

The rank of the matrix **A** is 1 *MN* , therefore the original image can be restored if one pixel of the image is *a priori* assigned to a constant, for instance the last pixel of the image is set to zero. So, the matrix **A** associated to the lineal system has the size 1 1 *MN MN* , and it 300 Image Restoration – Recent Advances and Applications

1 3 323

**A A AAA**

where the matrices **A**<sup>1</sup> , **A**<sup>2</sup> , and **A**<sup>3</sup> , of the size *N N* , are given by

1

2

and

**A**

000 0000

323

**AAA**

0 0 0 0 0 0

4 20 0 0 0 26 20 0 0 0 26 20 0 0 0 0 0 0 0 26 2 0 0 0 0 24

6 20 0 0 0 28 20 0 0 0 28 20 0 0 0 0 0 0 0 28 2 0 0 0 0 26

The rank of the matrix **A** is 1 *MN* , therefore the original image can be restored if one pixel of the image is *a priori* assigned to a constant, for instance the last pixel of the image is set to zero. So, the matrix **A** associated to the lineal system has the size 1 1 *MN MN* , and it

The matrix A is sparse, and it is calculated as

( ) (1, ), 1 *<sup>r</sup>* **u** *j r j jM* , (10)

( ) ( 1, ) ( , ), 1 2, 1 *<sup>r</sup>* **u** *iN j r i j r i j i M j N* , (11)

( ) ( 1, ) ( 1, 1), 2 1, 0 1, *<sup>c</sup>* **u** *iN j c i j c i j j N i M* (14)

0000 000

> 323 3 1

**<sup>A</sup>** , (17)

**<sup>A</sup>** , (18)

**A**<sup>3</sup> *diag* 2, 2,..., 2 . (19)

**AAA A A**

( ) ( 1, ), 0 1, **u***<sup>r</sup> NM j r M N j j N* (12)

( 1) ( 1,1), 0 1, *<sup>c</sup>* **u** *iM c i i M* (13)

( ) ( , 1), 1 . *<sup>c</sup>* **u** *iN c i N i M* (15)

, (16)

becomes symmetric and positive-definite. In this case there exists a unique solution. After solving the linear system, the obtained image is point-wise processed to have the same mean value (assumed to be known) with original image. To solve the linear system, iterative conjugate gradient method is used (Golub & Van Loan, 1996).

The computational complexity of the algorithm is given by the execution order of conjugate gradient and the size of an image to be restored. It is estimated as *kp* operations, where *p* is the number of nonzero entries in the matrix associated to lineal system, and *k* is the number of iterations required for solving the system of equations. Without loss of generality, we assume that *M N* . Therefore, *p*= <sup>2</sup> 5*M* and *k qM* where *q* depends on precision of the solution. The computational complexity of the method can be estimated as <sup>3</sup> 5*qM* .

Impulse noise is caused by sensor failures in a camera or transmission through a noisy channel. The proposed method is able to interpolate implicitly the pixel values corrupted with impulsive noise based on the information contained in neighboring pixels. This is because, during the microscanning the information of each pixel of the original image is captured in three different observed images. If one of sensors is damaged, partial information about the pixel intensity of the original image could be available in the other observed images.

#### **2.2 Multiplicative degradation model**

A typical example of multiplicative interference is nonuniform illumination. When an input image degraded by a multiplicative nonuniform interference and additive noise, the observed scene can be described as

$$s\_1(i,j) = b(i,j)f(i,j) + n\_1(i,j), \quad 1 \le i \le M, 1 \le j \le N \tag{20}$$

Suppose that microscanning is able to separate the original image and the interference, and then the shifted frames are obtained as follows:

$$s\_2(i,j) = b(i+1,j)f(i,j) + n\_2(i,j), \quad 1 \le i < M, 1 \le j \le N\_\prime \tag{21}$$

$$s\_3(i,j) = b(i,j+1)f(i,j) + n\_3(i,j), \quad 1 \le i \le M, 1 \le j < N. \tag{22}$$

Now we define two quotient matrices using spatial information between rows and columns of the images,

$$h(i,j) = \frac{s\_2(i,j)}{s\_1(i+1,j)}, \quad 1 \le i < M, \quad 1 \le j \le N, \quad s\_1(i+1,j) \ne 0 \tag{23}$$

$$w(i,j) = \frac{s\_3(i,j)}{s\_1(i,j+1)}, \quad 1 \le i \le M, \quad 1 \le j < N, \quad s\_1(i,j+1) \ne 0. \tag{24}$$

Hence, the matrices *h i*(, )*j* and *v i*(, )*j* as defined as

Blind Image Restoration for a Microscanning Imaging System 303

In a similar manner, the iterative conjugate gradient method can be used for solving this linear system. Finally, the restored image is obtained by applying the exponential function to the solution of the linear system. Since this method is based on the conjugate gradient method, its computational complexity is close to that of for the additive degradation.

In this section computer simulation results for restoration of images degraded by additive and multiplicative interferences are presented. The root mean square error (RMSE) criterion is used for comparison of quality of restoration. Additionally, a subjective visual criterion is

*M N*

*i j*

where *fij* (, ) is the original image and *f* (, ) *i j* is the restored image. The size of images used in our experiments is 256 256 pixels. The intensity values are in the range of [0,255]. The experiments were performed using a laptop computer (Intel Core 2 Duo 2.26 GHz with 2GB of RAM). To guarantee statistically correct results, 30 statistical trials of each experiment for different realizations of the random noise process were performed. The conjugate gradient method is used to solve the linear system. The convergence criterion is when the residual value drops below <sup>10</sup> 10 . The subjective visual criterion is defined as an enhanced difference between original and restored images. This enhanced difference

1 2 *EDF i j f i j f i j* ( , ) ( ( , ) ( , )) ,

2 are predetermined constants. In our experiments we set <sup>1</sup>

gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white (with intensity values of 0 and 255,

The linear minimum mean square error method is a popular technique in image restoration. In the case of stationary processes and in the absence of any blur, the method takes a simplified form of the Wiener smoothing filtering (Jain, 1989). The frequency response of the

(,) (,) (,) (,)

*S*

*P uv*

*S Noise Wiener*

*P uv P uv H uv*

where (,) *P uv <sup>s</sup>* is the power spectral density of the observed degraded input scene and *P uv Noise*(,) is the power spectral density of additive interference. It is assumed that all degradation parameters for the Wiener filter are exactly known. Note that the proposed

for additive and multiplicative models, respectively, and <sup>2</sup>

*RMSE f f MN*

 <sup>2</sup> (, ) (, )

> 

(30)

, (31)

128 . A pixel is displayed as

4 and 1

1

(29)

*fij fij*

, ,

Additional expenses are required for logarithm and exponential transformations.

**3. Computer simulation results** 

used. The RMSE is given by

(Kober, 2001) is defined as follows:

where 1 and 

respectively).

empirical Wiener filter is

$$h(i,j) = \begin{pmatrix} \left(\frac{s\_2(1,1)}{s\_1(2,1)}\right) & \left(\frac{s\_2(1,2)}{s\_1(2,2)}\right) & \cdots & \left(\frac{s\_2(1,N)}{s\_1(2,N)}\right) \\ \left(\frac{s\_2(2,1)}{s\_1(3,1)}\right) & \left(\frac{s\_2(2,2)}{s\_1(3,2)}\right) & \cdots & \left(\frac{s\_2(2,N)}{s\_1(3,N)}\right) \\ \vdots & \vdots & \vdots & \vdots \\ \left(\frac{s\_2(M-1,1)}{s\_1(M,1)}\right) & \left(\frac{s\_2(M-1,2)}{s\_1(M,2)}\right) & \cdots & \left(\frac{s\_2(M-1,N)}{s\_1(M,N)}\right) \\ 0 & 0 & \cdots & 0 \end{pmatrix} \tag{25}$$

$$
\mathbf{v}(i,j) = \begin{bmatrix}
\left(\frac{s\_3(1,1)}{s\_1(1,2)}\right) & \left(\frac{s\_3(1,2)}{s\_1(1,3)}\right) & \cdots & \left(\frac{s\_3(1,N-1)}{s\_1(1,N)}\right) & 0\\ 
\left(\frac{s\_3(2,1)}{s\_1(2,2)}\right) & \left(\frac{s\_3(2,2)}{s\_1(2,3)}\right) & \cdots & \left(\frac{s\_3(2,N-1)}{s\_1(2,N)}\right) & 0\\ 
\vdots & \vdots & \vdots & \vdots & \vdots\\ 
\left(\frac{s\_3(M,1)}{s\_1(M,2)}\right) & \left(\frac{s\_3(M,2)}{s\_1(M,3)}\right) & \cdots & \left(\frac{s\_3(M,N-1)}{s\_1(M,N)}\right) & 0
\end{bmatrix}.\tag{26}
$$

00 0

and

The multiplicative interference in the matrices *hi j* (, ) and *vi j* (, ) is eliminated when the observed images have no additive noise. However, for small standard deviation of noise the matrices are close to the correspondent quotient matrices constructed with the original image and its shifted versions. In a similar manner, the objective function can be written as

$$\begin{split} \tilde{F} &= \left[ \sum\_{i=1}^{M-1} \sum\_{j=1}^{N-1} \left[ h(i,j) - \left( \frac{f(i,j)}{f(i+1,j)} \right) \right]^2 + \left[ v(i,j) - \left( \frac{f(i,j)}{f(i,j+1)} \right) \right]^2 \right] \\ &+ \sum\_{j=1}^{N-1} \left[ v(M,j) - \left( \frac{f(M,j)}{f(M,j+1)} \right) \right]^2 + \sum\_{i=1}^{M-1} \left[ h(i,N) - \left( \frac{f(i,N)}{f(i+N,j)} \right) \right]^2. \end{split} \tag{27}$$

Since it is difficult to solve the system of nonlinear equation then a logarithm transformation to the system of nonlinear equations is applied. In this way the system can be converted to the linear system as follows:

$$\begin{split} \log(\tilde{F}) &= \begin{bmatrix} \sum\_{i=1}^{M-1} \sum\_{j=1}^{N-1} \left[ \log\left(h(i,j)\right) - \log\left(f(i,j) - f(i+1,j)\right) \right]^2 \\ + \left[ \log\left(v(i,j)\right) - \log\left(f(i,j) - f(i,j+1)\right) \right]^2 \\ + \sum\_{j=1}^{N-1} \left[ \log\left(v(M,j)\right) - \log\left(f(M,j) - f(M,j+1)\right) \right]^2 \\ &+ \sum\_{i=1}^{M-1} \left[ \log\left(h(i,N)\right) - \log\left(f(i,N) - f(i+N,j)\right) \right]^2 . \end{split} \tag{28}$$

In a similar manner, the iterative conjugate gradient method can be used for solving this linear system. Finally, the restored image is obtained by applying the exponential function to the solution of the linear system. Since this method is based on the conjugate gradient method, its computational complexity is close to that of for the additive degradation. Additional expenses are required for logarithm and exponential transformations.

## **3. Computer simulation results**

302 Image Restoration – Recent Advances and Applications

22 2 11 1 22 2 11 1

(1,1) (1,2) (1, ) (2,1) (2,2) (2, ) (2,1) (2,2) (2, )

,

.

0

1)

2

( , ) ( 1, )

*ij i j*

( , ) ( , 1)

*Mj Mj*

(

*i Nj*

(25)

(26)

(27)

(28)

*s s s N s s s N s s s N*

 

> ( 1,1) ( 1,2) ( 1, ) ( ,1) ( ,2) ( , )

*sM sM sM N sM sM s MN*

 

00 0

(1,1) (1,2) (1, 1) <sup>0</sup> (1,2) (1,3) (1, )

22 2 11 1

33 3 11 1 33 3 11 1

(2,1) (2,2) (2, 1) <sup>0</sup> (, ) (2,2) (2,3) (2, )

*s M s M s MN sM sM s MN*

The multiplicative interference in the matrices *hi j* (, ) and *vi j* (, ) is eliminated when the observed images have no additive noise. However, for small standard deviation of noise the matrices are close to the correspondent quotient matrices constructed with the original image and its shifted versions. In a similar manner, the objective function can be written as

*s s s N s s s N s s s N*

( ,1) ( ,2) ( , 1) ( ,2) ( ,3) ( , )

2 2 1 1

(, ) (, ) (, ) ( ,

(, ) ( ,) (, ) ( ,) .

log ( , ) log ( , ) ( , 1)

*vi j f i j f i j*

og

*hiN f iN f*

*vM j f f*

1 1 <sup>2</sup>

*hi j f f*

 

<sup>1</sup> <sup>2</sup>

log ( , ) l ) , ) og ( , .

<sup>1</sup> <sup>2</sup>

*hiN f i f N j*

2 2 1 1

 

*f f <sup>i</sup> <sup>N</sup> vM j*

Since it is difficult to solve the system of nonlinear equation then a logarithm transformation to the system of nonlinear equations is applied. In this way the system can be converted to

log ( , ) log

33 3 11 1

> (, ) ( 1, )

*i j i j*

*f f <sup>i</sup> <sup>j</sup> F hi j vi j f f <sup>i</sup> <sup>j</sup>*

*vi j s s s N*

1 1

*M N*

the linear system as follows:

*i j*

log( )

*F*

1 1

*N M*

*j i*

1 1

*M N*

*i j*

1

*N*

*j M*

 

*i*

1

( ,) ( , 1)

*M j M j*

log ( , ) l

(3,1) (3,2) (3, ) (, )

*s s s N hi j*

and

In this section computer simulation results for restoration of images degraded by additive and multiplicative interferences are presented. The root mean square error (RMSE) criterion is used for comparison of quality of restoration. Additionally, a subjective visual criterion is used. The RMSE is given by

$$RMSE\left(f,\tilde{f}\right) = \sqrt{\frac{\sum\_{i}^{M} \sum\_{j}^{N} \left(f(i,j) - \tilde{f}(i,j)\right)^{2}}{MN}},\tag{29}$$

where *fij* (, ) is the original image and *f* (, ) *i j* is the restored image. The size of images used in our experiments is 256 256 pixels. The intensity values are in the range of [0,255]. The experiments were performed using a laptop computer (Intel Core 2 Duo 2.26 GHz with 2GB of RAM). To guarantee statistically correct results, 30 statistical trials of each experiment for different realizations of the random noise process were performed. The conjugate gradient method is used to solve the linear system. The convergence criterion is when the residual value drops below <sup>10</sup> 10 . The subjective visual criterion is defined as an enhanced difference between original and restored images. This enhanced difference (Kober, 2001) is defined as follows:

$$\text{EDF}(\mathbf{i}, \mathbf{j}) = o\_1(f(\mathbf{i}, j) - \mathbf{\bar{f}}(\mathbf{i}, j)) + o\_{2, \prime} \tag{30}$$

where 1 and 2 are predetermined constants. In our experiments we set <sup>1</sup> 4 and 1 1 for additive and multiplicative models, respectively, and <sup>2</sup> 128 . A pixel is displayed as gray if there is no error between the original image and the restored image. For maximum error, the pixel is displayed either black or white (with intensity values of 0 and 255, respectively).

The linear minimum mean square error method is a popular technique in image restoration. In the case of stationary processes and in the absence of any blur, the method takes a simplified form of the Wiener smoothing filtering (Jain, 1989). The frequency response of the empirical Wiener filter is

$$H\_{\text{Minerr}}\left(\mu,\upsilon\right) = \frac{P\_S\left(\mu,\upsilon\right) - P\_{\text{Noise}}\left(\mu,\upsilon\right)}{P\_S\left(\mu,\upsilon\right)}\,,\tag{31}$$

where (,) *P uv <sup>s</sup>* is the power spectral density of the observed degraded input scene and *P uv Noise*(,) is the power spectral density of additive interference. It is assumed that all degradation parameters for the Wiener filter are exactly known. Note that the proposed

Blind Image Restoration for a Microscanning Imaging System 305

(a) (b) Fig. 2. Performance of the proposed method for additive degradation and additive noise with standard deviation of 2: (a) restored image, (b) enhanced difference between the

Fig. 3 shows the performance in terms of the RMSE of the proposed methods using three images (Am3), and the Wiener filter versus the standard deviation of additive noise. It can be seen that the performance of the proposed method is much better than that of the Wiener filtering with known parameters. It happens because the additive interference is spatially inhomogeneous, and therefore, it cannot be considered as a realization of a stationary process and correctly used in the filtering. The time required to restore the image with the proposed method is approximately 46 sec. The iterative conjugate gradient algorithm

Fig. 3. Performance of the proposed method for additive degradation: RMSE versus a

original image and the restored image.

requires about 1070 iterations.

standard deviation of additive noise.

method does not need any information about the degradation function. When observed images are degraded by multiplicative interference, first we apply a logarithm function to the degraded images to convert the multiplicative interference to additive one (ignoring the sensor noise). Then the empiric Wiener filtering with known parameters is utilized. Finally, the exponential function is applied to the Wiener restored image to obtain the output image.

## **3.1 Restoration of noisy image degraded with additive interference**

Figs. 1(a), 1(b), and 1(c) show a test original image, a nonuniform additive interference, and the original image degraded with the interference and a zero-mean white Gaussian noise with a standard deviation of 2, respectively. The mean value and standard deviation of the interference are 118.8 and 59.7, respectively. Fig. 2(a) shows restored images with the proposed method. Fig. 2(b) shows enhanced difference between the original image and the restored image.

(a) (b)

(c)

Fig. 1. (a) Original image, (b) additive interference, (c) observed image degraded with additive interference and white noise with standard deviation of 2.

304 Image Restoration – Recent Advances and Applications

method does not need any information about the degradation function. When observed images are degraded by multiplicative interference, first we apply a logarithm function to the degraded images to convert the multiplicative interference to additive one (ignoring the sensor noise). Then the empiric Wiener filtering with known parameters is utilized. Finally, the exponential function is applied to the Wiener restored image to obtain the

Figs. 1(a), 1(b), and 1(c) show a test original image, a nonuniform additive interference, and the original image degraded with the interference and a zero-mean white Gaussian noise with a standard deviation of 2, respectively. The mean value and standard deviation of the interference are 118.8 and 59.7, respectively. Fig. 2(a) shows restored images with the proposed method. Fig. 2(b) shows enhanced difference between the original image and the

(a) (b)

(c)

Fig. 1. (a) Original image, (b) additive interference, (c) observed image degraded with

additive interference and white noise with standard deviation of 2.

**3.1 Restoration of noisy image degraded with additive interference** 

output image.

restored image.

Fig. 2. Performance of the proposed method for additive degradation and additive noise with standard deviation of 2: (a) restored image, (b) enhanced difference between the original image and the restored image.

Fig. 3 shows the performance in terms of the RMSE of the proposed methods using three images (Am3), and the Wiener filter versus the standard deviation of additive noise. It can be seen that the performance of the proposed method is much better than that of the Wiener filtering with known parameters. It happens because the additive interference is spatially inhomogeneous, and therefore, it cannot be considered as a realization of a stationary process and correctly used in the filtering. The time required to restore the image with the proposed method is approximately 46 sec. The iterative conjugate gradient algorithm requires about 1070 iterations.

Fig. 3. Performance of the proposed method for additive degradation: RMSE versus a standard deviation of additive noise.

Blind Image Restoration for a Microscanning Imaging System 307

(a) (b)

(c) Fig. 5. (a) Observed image degraded with additive interference, white noise with standard deviation of 2, and impulse noise cluster (15x15 pixels), (b) restored image, and (c) enhanced

Note that outside of damaged elements the restoration performance of the proposed method is good. At the location of the damaged elements the method carries out a smooth

Nonuniform illumination is an example of multiplicative interference. In our experiments we use the Lambertian model of illumination, which reflects light equally in all directions.

0 0 1/2 2 2

 

( , ) cos arctan , <sup>2</sup> cos( ) ( tan( )cos( ) ) ( tan( )sin( ) )

 

0 0

(32)

 

*r pr q*

difference between the original image and restored image.

interpolation using information containing in neighboring pixels.

**3.2 Restoration of noisy image degraded with nonuniform illumination** 

Its reflectance map (Diaz-Ramirez & Kober, 2009) can be expressed as

*<sup>r</sup> Ip q*

Fig. 4(a) shows the observed scene degraded by nonuniform additive interference, zeromean white Gaussian noise with a standard deviation of 2, and impulse noise with the occurrence probability of 0.03. The value of impulse noise is zero (physical meaning is defective sensor element). Figs. 4(b) and 4(c) show the restored image and the enhanced difference between the original and restored images, respectively.

(c)

Fig. 4. (a) Observed image degraded with additive interference, white noise with standard deviation of 2, and impulse noise with probability of 0.03, (b) restored image, and (c) enhanced difference between the original image and restored image.

Finally, we show computer simulation results when the original image additional degraded by impulse noise cluster of size 15x15 elements. Figs. 5(a), 5(b), and 5(c) show the observed image, the restored image, and enhanced difference between the original and restored images, respectively.

306 Image Restoration – Recent Advances and Applications

Fig. 4(a) shows the observed scene degraded by nonuniform additive interference, zeromean white Gaussian noise with a standard deviation of 2, and impulse noise with the occurrence probability of 0.03. The value of impulse noise is zero (physical meaning is defective sensor element). Figs. 4(b) and 4(c) show the restored image and the enhanced

(a) (b)

(c) Fig. 4. (a) Observed image degraded with additive interference, white noise with standard deviation of 2, and impulse noise with probability of 0.03, (b) restored image, and (c)

Finally, we show computer simulation results when the original image additional degraded by impulse noise cluster of size 15x15 elements. Figs. 5(a), 5(b), and 5(c) show the observed image, the restored image, and enhanced difference between the original and restored

enhanced difference between the original image and restored image.

images, respectively.

difference between the original and restored images, respectively.

Fig. 5. (a) Observed image degraded with additive interference, white noise with standard deviation of 2, and impulse noise cluster (15x15 pixels), (b) restored image, and (c) enhanced difference between the original image and restored image.

Note that outside of damaged elements the restoration performance of the proposed method is good. At the location of the damaged elements the method carries out a smooth interpolation using information containing in neighboring pixels.

## **3.2 Restoration of noisy image degraded with nonuniform illumination**

Nonuniform illumination is an example of multiplicative interference. In our experiments we use the Lambertian model of illumination, which reflects light equally in all directions. Its reflectance map (Diaz-Ramirez & Kober, 2009) can be expressed as

$$I(p\_0, q\_0) = \cos\left[\frac{\pi}{2} - \arctan\left[\frac{r}{\cos(\tau)\left[\left(r\tan(\tau)\cos(a) - p\_0\right)^2 + \left(r\tan(\tau)\sin(a) - q\_0\right)^2\right]^{1/2}}\right]\right] \tag{32}$$

Blind Image Restoration for a Microscanning Imaging System 309

(a) (b)

(c) (d) Fig. 7. Nonuniform illumination correction with the proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 1.1 *r* , (c) restored image, (d) enhanced difference

> 5 ,

245 , and 1.1 *r* : RMSE

between the original image and the restored image.

Fig. 8. Performance of the proposed method when

versus a standard deviation of additive noise.

where is the slant angle, is the tilt angle, and r is the magnitude of the vector from pointlight source to surface. These parameters define the position of the light source with respect to the surface origin. In our simulations, we set 5 , 245 , and *r* 1.1,1.5,2 . Table 1 shows the values taken by the illumination function in our experiments.


Table 1. Values of the illumination function.

The mean value and standard deviation of the original image are 112.3 and 50, respectively, with maximum and minimum values of 237 and 17, respectively. Fig. 6 shows a test original image.

Fig. 6. Test original image.

Figs. 7(a), 9(a), and 11(a) show degraded images with different illuminations functions shown in Figs. 7(b), 9(b), and 11(b) (*r* 1.1,1.5,2 ). The degraded image also contains a zero-mean Gaussian noise with a standard deviation of 1. Figs. 7(c), 9(c), and 11(c) show the restored images using the proposed method. Figs. 7(d), 9(d), and 11(d) show enhanced difference between the original image and the restored images.

Figs. 8, 10, and 12 show the performance in terms of the RMSE of the proposed methods using three images (Mm3) and the Wiener filter versus the standard deviation of additive noise with different parameters of illumination. One can observe that the proposed method is useful when the standard deviation of additive noise is low. Note that the performance of the proposed method is essentially better than that of the Wiener filter, which is designed with known parameters. Time required to restore the image using the Mm3 is approximately 51 sec. In this case, the iterative conjugate gradient algorithm requires about 1070 iterations.

308 Image Restoration – Recent Advances and Applications

where is the slant angle, is the tilt angle, and r is the magnitude of the vector from pointlight source to surface. These parameters define the position of the light source with respect

1.1 0.35-1 0.60 0.14 65% 1.5 0.45-1 0.70 0.12 55% 2 0.55-1 0.79 0.10 45%

The mean value and standard deviation of the original image are 112.3 and 50, respectively, with maximum and minimum values of 237 and 17, respectively. Fig. 6 shows a test original

Figs. 7(a), 9(a), and 11(a) show degraded images with different illuminations functions shown in Figs. 7(b), 9(b), and 11(b) (*r* 1.1,1.5,2 ). The degraded image also contains a zero-mean Gaussian noise with a standard deviation of 1. Figs. 7(c), 9(c), and 11(c) show the restored images using the proposed method. Figs. 7(d), 9(d), and 11(d) show enhanced

Figs. 8, 10, and 12 show the performance in terms of the RMSE of the proposed methods using three images (Mm3) and the Wiener filter versus the standard deviation of additive noise with different parameters of illumination. One can observe that the proposed method is useful when the standard deviation of additive noise is low. Note that the performance of the proposed method is essentially better than that of the Wiener filter, which is designed with known parameters. Time required to restore the image using the Mm3 is approximately 51 sec. In this case, the iterative conjugate gradient algorithm requires about

difference between the original image and the restored images.

shows the values taken by the illumination function in our experiments.

5 ,

Mean value

245 , and *r* 1.1,1.5,2 . Table 1

Standard

deviation Shading

to the surface origin. In our simulations, we set

Table 1. Values of the illumination function.

function

Fig. 6. Test original image.

1070 iterations.

image.

*<sup>r</sup>* Range of values taken by the illumination

Fig. 7. Nonuniform illumination correction with the proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 1.1 *r* , (c) restored image, (d) enhanced difference between the original image and the restored image.

Fig. 8. Performance of the proposed method when 5 , 245 , and 1.1 *r* : RMSE versus a standard deviation of additive noise.

Blind Image Restoration for a Microscanning Imaging System 311

(a) (b)

(c) (d) Fig. 11. Nonuniform illumination correction with proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 2.0 *r* , (c) restored image, (d) enhanced difference

Fig. 12. Performance of the proposed method for multiplicative degradation with

245 , and 2.0 *r* : RMSE versus a standard

 5 , 

between the original image and the restored image.

parameters of illumination of

deviation of additive noise.

Fig. 9. Nonuniform illumination correction with the proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 1.5 *r* , (c) restored image, (d) enhanced difference between the original image and the restored image.

Fig. 10. Performance of the proposed method when 5 , 245 , and 1.5 *r* : RMSE versus a standard deviation of additive noise.

310 Image Restoration – Recent Advances and Applications

(a) (b)

(c) (d) Fig. 9. Nonuniform illumination correction with the proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 1.5 *r* , (c) restored image, (d) enhanced difference

> 5 ,

245 , and 1.5 *r* : RMSE

between the original image and the restored image.

Fig. 10. Performance of the proposed method when

versus a standard deviation of additive noise.

Fig. 11. Nonuniform illumination correction with proposed method: (a) observed image degraded with multiplicative interference and white noise with a standard deviation of 1, (b) multiplicative interference with 2.0 *r* , (c) restored image, (d) enhanced difference between the original image and the restored image.

Fig. 12. Performance of the proposed method for multiplicative degradation with parameters of illumination of 5 , 245 , and 2.0 *r* : RMSE versus a standard deviation of additive noise.

Blind Image Restoration for a Microscanning Imaging System 313

In this chapter we presented methods for restoration of images degraded with additive and multiplicative interferences, and corrupted by sensor noise. Using three observed images taken with a microscanning imaging system, an explicit system of equations for additive and multiplicative signal models was derived. The restored image is a solution of the system. With the help of computer simulations we demonstrated the performance of the proposed method in terms of restoration accuracy and execution time. The performance of the proposed method is essentially better than that of the Wiener filter, which is designed

Banham, M. & Katsaggelos, A. (1997) Digital image restoration. *IEEE Signal Processing* 

Bovik, A. (2005). *Handbook of image and video processing* (2nd ed.), Academic Press, ISBN 0-12-

Diaz-Ramirez, V. & Kober, V. (2009) Target recognition under nonuniform illumination

Golub, G. & Van Loan, C. (1996). *Matrix computations*, (3rd ed.), The Johns Hopkins

González, R. & Woods, R. (2008). *Digital image processing* (3rd ed.), Prentice Hall, ISBN 0-13-

Hautiere, N. & Aubert, D. (2005). Contrast restoration of foggy images through use of an

Jain, A. (1989). *Fundamentals of Digital Image Processing*. Prentice Hall, ISBN 0-13-336165-9,

Kay, S. (1993). *Fundamentals of statistical signal processing : estimation theory*, Prentice Hall,

Kober, V., Mozero, M. & Alvarez-Borrego, J.(2001). Nonlinear filters with spatially

Kundur, D. & Hatzinakos, D. (1996). Blind image deconvolution. *IEEE Signal Processing* 

Lee, H. & Kim, J. (2009). Retrospective correction of nonuniform illumination on bi-level

López-Martínez, J. & Kober, V. (2008). Image restoration based on camera microscanning,

*Magazine,* Vol. 13, No. 3, (May 1996), pp. 73-76, ISSN 1053-5888

1090-1095, ISBN 0-7803-9215-9, Versailles, France, September 13-15, 2005 Hayat, M., Torres, S., Armstrong, E., Cain, S. & Yasuda, B. (1999). Statistical algorithm for

conditions. *Applied Optics,* Vol. 48, No. 7, (March 2009), pp.1408–1418, ISSN 1559-

onboard camera, *Proceedings of IEEE Conference Intelligent Transportation System,* pp.

nonuniformity correction in focal-plane arrays. *Applied Optics*, Vol. 38, No. 8,

connected neighborhoods. *Optical Engineering*, Vol. 40, No. 6, (June 2001), pp. 971-

images. *Optics Express*, Vol. 17, No. 26, (December 2009), pp. 23880-23893, eISSN

*Proceedings of Applications of Digital Image Processing XXXI* , ISBN 9780819472939,

*Magazine,* Vol. 14, No.2, (March 1997), pp. 24-41, ISSN 1053-5888

University Press, ISBN 0-8018-5414-8, Baltimore, Maryland, USA

(March 1999), pp. 772-780, ISSN 1559-128X

San Diego, CA, USA, August 10-14, 2008.

Englewood Cliffs, NJ, USA

ISBN 0-13-345711-7, NJ, USA

983, ISSN 0091-3286

1094-4087

**5. Conclusion** 

with known parameters

128X

119792-1, NJ, USA

1687288, NJ, USA

**6. References** 

## **4. Experimental results**

Here we present experimental results with a real-life image degraded by a multiplicative interference. The observed images were obtained as follows. A test image was displayed on a LCD screen. A printed transparency was placed between a camera and the screen in order to simulate a multiplicative degradation. Microscanning was performed by shifting the test image on the screen. Finally, the three observed images were captured with the camera. The observed images have the size of 256 × 256 pixels. First, the observed images were passed through the logarithmic transformation. Next, the proposed method was utilized to obtain a resulting image. Finally, the exponential transformation was applied to the resulting image to restore the original image. The original image, multiplicative degradation, and one of the observed images taken by a camera are shown in Figs. 13(a), 13(b), and 13(c), respectively. The restored image is presented in Fig. 13(d).

Since in the experiment the level of additive noise is low, the quality of the restoration with the proposed method is very good.

Fig. 13. (a) Original image, (b) multiplicative interference, (c) observed image degraded with multiplicative interference, and (d) restored image.

## **5. Conclusion**

312 Image Restoration – Recent Advances and Applications

Here we present experimental results with a real-life image degraded by a multiplicative interference. The observed images were obtained as follows. A test image was displayed on a LCD screen. A printed transparency was placed between a camera and the screen in order to simulate a multiplicative degradation. Microscanning was performed by shifting the test image on the screen. Finally, the three observed images were captured with the camera. The observed images have the size of 256 × 256 pixels. First, the observed images were passed through the logarithmic transformation. Next, the proposed method was utilized to obtain a resulting image. Finally, the exponential transformation was applied to the resulting image to restore the original image. The original image, multiplicative degradation, and one of the observed images taken by a camera are shown in Figs. 13(a), 13(b), and 13(c), respectively.

Since in the experiment the level of additive noise is low, the quality of the restoration with

(a) (b)

(c) (d) Fig. 13. (a) Original image, (b) multiplicative interference, (c) observed image degraded with

**4. Experimental results** 

The restored image is presented in Fig. 13(d).

multiplicative interference, and (d) restored image.

the proposed method is very good.

In this chapter we presented methods for restoration of images degraded with additive and multiplicative interferences, and corrupted by sensor noise. Using three observed images taken with a microscanning imaging system, an explicit system of equations for additive and multiplicative signal models was derived. The restored image is a solution of the system. With the help of computer simulations we demonstrated the performance of the proposed method in terms of restoration accuracy and execution time. The performance of the proposed method is essentially better than that of the Wiener filter, which is designed with known parameters

## **6. References**


Daniel Kekrt and Miloš Klíma

**Performance Analysis** 

*Czech Republic*

**14** 

**14**

*Dept. of Radio Engineering, Czech Technical University in Prague*

**2D Iterative Detection Network Based Image** 

**Restoration: Principles, Applications and** 

Image capturing is corrupted by numerous perturbing influences. These influences are divisible to time-invariant and temporal. The typical time-invariant influence is an image blurring, rising from various causes, that can be mathematically understood as deterministic 2D ISI channel or FSM (Finite state machine). Among temporal influences pertain especially noises on the other hand. There are four significant noise sources in the case of a camera with CCD (CMOS) sensor: photon noise (signal dependent additive Poisson stochastic process), thermal noise (additive Poisson stochastic process), readout noise (additive Gaussian stochastic process) and quantization noise (J. van Vliet L. et al., 1998). The photon noise is caused by the time inhomogeneous photon emission, incident to the lens in individual par-axial light rays, with the mean value and squared standard deviation *μ<sup>P</sup>* equal to the averaged intensity of these rays. The photon noise cannot be compensated because its origin is located front of the lens. Therefore we will not take it into account. The thermal noise, readout noise and quantization noise together create one composite noise of the CCD/CMOS sensor that affects on the captured blurred image as the random IECS-ML channel. Such channel is biased by three parameters *μR*, *σ<sup>R</sup>* (mean value and standard deviation of the readout noise), depending on the sensor readout rate, and *μ<sup>T</sup>* (mean value and squared standard deviation of the thermal noise), exponentially raising according to the sensor temperature *Ts*. All mentioned influences can be eliminated by iterative detection network (IDN). Such system solves effectively the 2D MAP criterion through feedback process based on the exchange and precision of certain probability density functions (PDFs). Similar networks (simpler one-dimensional alternatives) have found utilization in the sphere of Turbo code detection (Chugg et al., 2001; Vucetic & Juan, 2003; 2000). There will be discussed de facto theirs generalization to the two-dimensional form. The explanation begins by decomposition of the 2D MAP criterion that elucidates the essential principle of IDNs functioning. The necessary conditions for this decomposition will be defined too. Consequently, we focus closer to the IDNs dedicated for restoration of dichromatic images and using PDFs marginalization at the

We assume the model of image capturing system in Fig. 1 including three major sections. The first and second section are emulating all perturbing influences affecting the captured

**1. Introduction**

symbol level and symbol block level.

**2. System model**


Milanfar, P. (2011). *Super-resolution imaging*, CRC Press, ISBN 13-978-1-4398-1931-9, NJ, USA


## **2D Iterative Detection Network Based Image Restoration: Principles, Applications and Performance Analysis**

Daniel Kekrt and Miloš Klíma *Dept. of Radio Engineering, Czech Technical University in Prague Czech Republic*

## **1. Introduction**

314 Image Restoration – Recent Advances and Applications

López-Martínez, J. & Kober, V. (2010). Image restoration of nonuniformly illuminated

Milanfar, P. (2011). *Super-resolution imaging*, CRC Press, ISBN 13-978-1-4398-1931-9, NJ, USA Narasimhan, S. & Nayar, S. (2002). Vision and the atmosphere. *International Journal of Computer Vision,* Vol. 48, No.3, (July-August 2002), pp. 233–254, ISSN 0920-5691 Ratliff, B., Hayat, M. & Hardie, R. (2002). An algebraic algorithm for nonuniformity

Shi, J., Reichenbach, S. & Howe, J. (2006) Small-kernel superresolution methods for

2010), pp.370–375, ISSN 1054-6618

1203–1214, ISSN 1559-128X

(September 2002), pp. 1737–1747, ISSN 1084-7529

images with camera microscanning, *Proceedings of Applications of Digital Image Processing XXXIII,* ISBN 9780819482945 , San Diego, CA, USA, August 2-4 , 2010 López-Martínez, J., Kober, V. & Ovseyevich, I. (2010). *Image restoration based on camera* 

*microscanning*. Pattern Recognition and Image Analysis, Vol. 20, No. 3,(September

correction in focal-plane arrays. *Journal Optical Society of America A*, Vol. 19, No. 9,

microscanning imaging systems. *Applied Optics,* Vol. 45, No. 6,(February 2006), pp.

Image capturing is corrupted by numerous perturbing influences. These influences are divisible to time-invariant and temporal. The typical time-invariant influence is an image blurring, rising from various causes, that can be mathematically understood as deterministic 2D ISI channel or FSM (Finite state machine). Among temporal influences pertain especially noises on the other hand. There are four significant noise sources in the case of a camera with CCD (CMOS) sensor: photon noise (signal dependent additive Poisson stochastic process), thermal noise (additive Poisson stochastic process), readout noise (additive Gaussian stochastic process) and quantization noise (J. van Vliet L. et al., 1998). The photon noise is caused by the time inhomogeneous photon emission, incident to the lens in individual par-axial light rays, with the mean value and squared standard deviation *μ<sup>P</sup>* equal to the averaged intensity of these rays. The photon noise cannot be compensated because its origin is located front of the lens. Therefore we will not take it into account. The thermal noise, readout noise and quantization noise together create one composite noise of the CCD/CMOS sensor that affects on the captured blurred image as the random IECS-ML channel. Such channel is biased by three parameters *μR*, *σ<sup>R</sup>* (mean value and standard deviation of the readout noise), depending on the sensor readout rate, and *μ<sup>T</sup>* (mean value and squared standard deviation of the thermal noise), exponentially raising according to the sensor temperature *Ts*. All mentioned influences can be eliminated by iterative detection network (IDN). Such system solves effectively the 2D MAP criterion through feedback process based on the exchange and precision of certain probability density functions (PDFs). Similar networks (simpler one-dimensional alternatives) have found utilization in the sphere of Turbo code detection (Chugg et al., 2001; Vucetic & Juan, 2003; 2000). There will be discussed de facto theirs generalization to the two-dimensional form. The explanation begins by decomposition of the 2D MAP criterion that elucidates the essential principle of IDNs functioning. The necessary conditions for this decomposition will be defined too. Consequently, we focus closer to the IDNs dedicated for restoration of dichromatic images and using PDFs marginalization at the symbol level and symbol block level.

## **2. System model**

We assume the model of image capturing system in Fig. 1 including three major sections. The first and second section are emulating all perturbing influences affecting the captured

sensor cells. Thereafter, the discrete convolution kernel can be got by the integration *a*(*i*, *j*) =

*PSF*(*x*, *y*) over the individual sensor cells. They are marked in Fig. 2 by the carnation lines and adjusted, without detriment to generality, as the unit size areas. Concretely, the results of

marked in the given figure by red stem graphs. The first is 9-ray kernel and the second one is

(a) **A**GBC,3/10. (b) **A**SGBC,6/10.

a(0, 0) = 0.2 a(0, 0) = 0.3 a(0, 0) = 0.4 a(0, 0) = 0.5 a(0, 0) = 0.7 a(0, 0) = 1

We have an object moving on the certain known trajectory as well as in Fig. 4. Each point

**s***<sup>e</sup>* = **s**(*t* + *Te*) point. These points are projected through the lens (it is not included to Fig. 4 for simplicity) to the plane {**b***x*, **b***y*} of the CCD/CMOS sensor, where *Te* is the exposure time. We denote impulse response of the channel emulating blurring due to object moving as **A**BOM,*<sup>α</sup>* and with respect to discrete character of this response let us approximate the

� ) *sy*(*t* �

Fig. 2. The two examples of GBCs with corresponding convolutional kernels.

Fig. 3. Examples of the defocusing by GBC on the QR code snapshot.

�

) = [*sx*(*t*

**3.2 Blurring due to object moving**

of this object pass the curve **s**(*t*

⎦ , **A**SGBC,6/10 =

⎤

<sup>4</sup> (erf(*i* − 1/2) − erf(*i* + 1/2))(erf(*j* − 1/2) − erf(*j* + 1/2)) of the

⎡ ⎣

0 0.0872 0 0.0872 0.6000 0.0872 0 0.0872 0

)] with the starting **s***<sup>s</sup>* = **s**(*t*) and ending

⎤

⎦ , (2)

317

� *<sup>i</sup>*+1/2 *i*−1/2

� *<sup>j</sup>*+1/2

**A**GBC,3/10 =

*<sup>j</sup>*−1/2 *PSF*(*x*, *<sup>y</sup>*)*dxdy* <sup>=</sup> <sup>Δ</sup>

2D Iterative Detection Network Based

⎡ ⎣

5-ray kernel with suppressed insignificant rays.

the mentioned integration in our examples are two kernels

Image Restoration: Principles, Applications and Performance Analysis

0.0458 0.1172 0.0458 0.1172 0.3000 0.1172 0.0458 0.1172 0.0458

image. The first section presents the hypothetical model of the time-invariant image blurring (2D ISI channel). The second one is image capturing section (IECS-ML channel) modeling all significant noises incident this kind of capturing systems. Such noises rise especially in the camera sensor with on-chip electronics, but no only there. The last section is the MAP criterion based iterative detector (restorative circuits) containing IDN with its front-end, so-called soft output demodulator (SODEM).

Fig. 1. Block diagram of a CCD/CMOS camera with dichromatic image restorative section.

## **3. Deterministic 2D ISI channel (blurring model)**

A time-invariant image blurring can be emulated by the signal transmission through the 2D ISI channel defined via convolution

$$\begin{split} \tilde{q}(i,j) &= f(\mathbf{A}, \mathcal{N}\_d(i,j) \subset \mathbf{D}) \\ &= \sum\_{0 \le i', j' \le H\_\mathbf{A}, \mathcal{W}\_\mathbf{A}} a(i', j') d(i + i', j + j'), \end{split} \tag{1}$$

where N*d*(*i*, *j*) = {*d*(*i* + *i* � , *j* + *j* � )}0≤*i*� ,*j*�≤*H***A**,*W***<sup>A</sup>** denotes the convolution region and **<sup>D</sup>** = [*d*(*i*, *<sup>j</sup>*)]*i*,*<sup>j</sup>* the page of black and white pixels *<sup>d</sup>*(*i*, *<sup>j</sup>*) ∈ {*d*()}*Md* <sup>=</sup><sup>1</sup> = {0, 1} (relative photon quantity impacting at the lens in several par-axial rays). The discrete 2D finite impulse response **A** = [*a*(*i* � , *j* � )]0≤*i*� ,*j*�≤*H***A**,*W***<sup>A</sup>** with the high *<sup>H</sup>***<sup>A</sup>** and width *<sup>W</sup>***<sup>A</sup>** model time-invariant image blurring. There we focus closer to the two basic time-invariant distortion — defocusing in the imperfectly adjusted lens and blurring due to object moving.

#### **3.1 Defocusing in the inperfectly adjusted lens**

The image blurring in a defocused lens can be mathematically imitated as the Gaussian blurring channel (GBC). It is defined by the point spread function *PSF*(*x*, *y*) = <sup>Δ</sup> *<sup>π</sup> <sup>e</sup>*−Δ(*x*<sup>2</sup>+*y*<sup>2</sup>), where Δ is the spread parameter. Two examples of *PSF*(*x*, *y*) with different Δ shows Fig. 2.

If we want transform such PSF to the 2D ISI channel, we have to assume approximation, that the distribution of light is uniform on the flats with the size and shape congruous to the 2 Will-be-set-by-IN-TECH

image. The first section presents the hypothetical model of the time-invariant image blurring (2D ISI channel). The second one is image capturing section (IECS-ML channel) modeling all significant noises incident this kind of capturing systems. Such noises rise especially in the camera sensor with on-chip electronics, but no only there. The last section is the MAP criterion based iterative detector (restorative circuits) containing IDN with its front-end, so-called soft

Restorative

), (1)

<sup>=</sup><sup>1</sup> = {0, 1} (relative photon

*<sup>π</sup> <sup>e</sup>*−Δ(*x*<sup>2</sup>+*y*<sup>2</sup>),

circuits

SO[dˇ(i, j)]

SI[ˇq(i, j)]

dˆ(i, j)

Restored picture

DEC

SODEM

(FSM−1) IDN

Quantization noise

,*j*�≤*H***A**,*W***<sup>A</sup>** denotes the convolution region and **<sup>D</sup>** =

,*j*�≤*H***A**,*W***<sup>A</sup>** with the high *<sup>H</sup>***<sup>A</sup>** and width *<sup>W</sup>***<sup>A</sup>** model time-invariant

SI[dˇ(i, j)]

SO[ˇq(i, j)]

rd(i, j)

converter A/D

r(i, j)

Fig. 1. Block diagram of a CCD/CMOS camera with dichromatic image restorative section.

A time-invariant image blurring can be emulated by the signal transmission through the 2D

*a*(*i* � , *j* � )*d*(*i* + *i* � , *j* + *j* �

quantity impacting at the lens in several par-axial rays). The discrete 2D finite impulse

image blurring. There we focus closer to the two basic time-invariant distortion — defocusing

The image blurring in a defocused lens can be mathematically imitated as the Gaussian

where Δ is the spread parameter. Two examples of *PSF*(*x*, *y*) with different Δ shows Fig. 2. If we want transform such PSF to the 2D ISI channel, we have to assume approximation, that the distribution of light is uniform on the flats with the size and shape congruous to the

blurring channel (GBC). It is defined by the point spread function *PSF*(*x*, *y*) = <sup>Δ</sup>

output demodulator (SODEM).

Photon noise

(Blurring) channel 2D ISI

Thermal noise Readout noise (Sensor noises)

ISI channel defined via convolution

� , *j* � )]0≤*i*�

**3.1 Defocusing in the inperfectly adjusted lens**

where N*d*(*i*, *j*) = {*d*(*i* + *i*

response **A** = [*a*(*i*

q˜(i, j)

**3. Deterministic 2D ISI channel (blurring model)**

� , *j* + *j* � )}0≤*i*�

[*d*(*i*, *<sup>j</sup>*)]*i*,*<sup>j</sup>* the page of black and white pixels *<sup>d</sup>*(*i*, *<sup>j</sup>*) ∈ {*d*()}*Md*

in the imperfectly adjusted lens and blurring due to object moving.

CCD (CMOS) Sensor

(FSM) network Encoding

> converter q → V

*q*˜(*i*, *j*) = *f*(**A**, N*d*(*i*, *j*) ⊂ **D**) = ∑ 0≤*i*�

,*j*�≤*H***A**,*W***<sup>A</sup>**

IECS-ML channel

d(i, j)

Light rays (image)

sensor cells. Thereafter, the discrete convolution kernel can be got by the integration *a*(*i*, *j*) = � *<sup>i</sup>*+1/2 *i*−1/2 � *<sup>j</sup>*+1/2 *<sup>j</sup>*−1/2 *PSF*(*x*, *<sup>y</sup>*)*dxdy* <sup>=</sup> <sup>Δ</sup> <sup>4</sup> (erf(*i* − 1/2) − erf(*i* + 1/2))(erf(*j* − 1/2) − erf(*j* + 1/2)) of the *PSF*(*x*, *y*) over the individual sensor cells. They are marked in Fig. 2 by the carnation lines and adjusted, without detriment to generality, as the unit size areas. Concretely, the results of the mentioned integration in our examples are two kernels

$$\mathbf{A}\_{\text{GBC}, \%/10} = \begin{bmatrix} 0.0458 \ 0.1172 \ 0.0458 \\ 0.1172 \ 0.3000 \ 0.1172 \\ 0.0458 \ 0.1172 \ 0.0458 \end{bmatrix}, \begin{array}{c} \mathbf{A}\_{\text{SGBC}, \%/10} = \begin{bmatrix} 0 & 0.0872 & 0 \\ 0.0872 \ 0.6000 \ 0.0872 \\ 0 & 0.0872 & 0 \end{bmatrix}, \end{array} \tag{2}$$

marked in the given figure by red stem graphs. The first is 9-ray kernel and the second one is 5-ray kernel with suppressed insignificant rays.

Fig. 2. The two examples of GBCs with corresponding convolutional kernels.

Fig. 3. Examples of the defocusing by GBC on the QR code snapshot.

## **3.2 Blurring due to object moving**

We have an object moving on the certain known trajectory as well as in Fig. 4. Each point of this object pass the curve **s**(*t* � ) = [*sx*(*t* � ) *sy*(*t* � )] with the starting **s***<sup>s</sup>* = **s**(*t*) and ending **s***<sup>e</sup>* = **s**(*t* + *Te*) point. These points are projected through the lens (it is not included to Fig. 4 for simplicity) to the plane {**b***x*, **b***y*} of the CCD/CMOS sensor, where *Te* is the exposure time. We denote impulse response of the channel emulating blurring due to object moving as **A**BOM,*<sup>α</sup>* and with respect to discrete character of this response let us approximate the

If the object expansion on the plane {**b***x*, **b***y*}, between the times *t* and *t* + *Te*, is not insignificant

is analogical to Eq. 4, only with the difference that size of sliding sensor cell will not be equal

<sup>2</sup> *Lc* (*t*�) *fp*(*i*(**s***s*) + *<sup>i</sup>*

� in compliance with the movement of scanning object.

�

�

, *j*(**s***s*) + *j*

ss a(0, 0) a(0, 1)

> a(1, 1) a(2, 1)

a(1, 2)

a(0, 2)

a(2, 2)

(b) The example with **s***<sup>s</sup>* situated on the edge of the pixel (*i*(**s***s*), *j*(**s***s*)). Negligible radial

a(3, 2) a(3, 3)

j = j(ss)

a(1, 0)

component of the velocity.

Lc(t)

Lc(t + Te)

a(3, 3)

a(2, 3)

a(1, 3)

se

� , *x*� , *y*� )*dx*� *dy*�

) linearly (Fig. 5c) or non-linearly

se

a(2, 3)

*dt*� (6)

319

i = i(ss)

the situation in Fig. 5c occurs. Computation of the constants

Image Restoration: Principles, Applications and Performance Analysis

<sup>2</sup> *Lc* (*t*�)

to pixel size *Lp* anymore, but it will present the function *Lc*(*t*

<sup>a</sup>(1, 3) <sup>a</sup>(1, 2)

Pattern

i = i(ss)

the velocity.

component of the velocity **v**).

a(2, 2)

se

a(2, 3)

 *sx* (*t* � )+ <sup>1</sup> <sup>2</sup> *Lc* (*t* � )

*sx* (*t*�)<sup>−</sup> <sup>1</sup>

i = i(ss)

<sup>a</sup>(1, 1) <sup>a</sup>(1, 0) ss

a(2, 0) a(2, 1)

j = j(ss)

<sup>a</sup>(0, 2) <sup>a</sup>(0, 1) <sup>a</sup>(0, 0)

a(3, 1) a(3, 2) a(2, 2)

(c) The example with **s***<sup>s</sup>* situated on the edge of the pixel (*i*(**s***s*), *j*(**s***s*)). Indispensable radial component of

Fig. 5. The equivalent movement trajectory of the sensor sensing cell sliding on the pattern **D**

(snapshot of the straight-line moving object with negligible and indispensable radial

a(1, 2)

 *sy* (*t* � )+ <sup>1</sup> <sup>2</sup> *Lc* (*t* � )

*sy* (*t*�)<sup>−</sup> <sup>1</sup>

*a*(*i* � , *j* � ) ∝ *Te* 0

dependent on the time *t*

<sup>L</sup><sup>p</sup> <sup>ϕ</sup>

of the velocity.

Sliding CCD cell

2D Iterative Detection Network Based

j = j(ss)

ss

a(1, 0) a(0, 0) a(0, 2)

a(0, 1)

a(1, 1)

a(2, 1)

(a) The example with **s***<sup>s</sup>* situated inside of the pixel (*i*(**s***s*), *j*(**s***s*)). Negligible radial component

realization of par-axial intensity on the plane {**b***x*, **b***y*}, in the time *t*, as the mosaic *D*(*x*, *y*) = <sup>∑</sup>*i*,*<sup>j</sup> <sup>d</sup>*(*i*, *<sup>j</sup>*)*fp*(*i*, *<sup>j</sup>*, *<sup>x</sup>*, *<sup>y</sup>*) of pixels *<sup>d</sup>*(*i*, *<sup>j</sup>*) ∈ {*d*()}*Md* <sup>=</sup><sup>1</sup> shaped by square function

$$f\_p(i, j, \mathbf{x}, y) = \begin{cases} 1 \text{ iff } |\mathbf{x} - i\mathbf{L}\_p| \le \frac{\mathbf{L}\_p}{2} \land |y - j\mathbf{L}\_p| \le \frac{\mathbf{L}\_p}{2} \\ 0 & \text{otherwise} \end{cases} \tag{3}$$

where *Lp* establish their size, that is equal or less then size *Lc* of the sensor cell (potential well).

Fig. 4. The projection of the moving object point to the CCD/CMOS sensor plane.

On the basis of declared definitions and approximations, we can subsequently express useful signal impacting the sensor by the convolution Eq. 1 emulating the situation, when the snapshot **D**, gained in the time *t*, is slided on the sensor surface and stepwise trapped to the its cells. The said fact can be also conceived from the opposite side, when potential well sliding on the immovable pattern **D**. If the velocity radial component is insignificant in the comparison with axial component (expansion of the object is negligible between times *t* and *t* + *Te*) and starting time *t* is equated to zero, we can obtain the impulse response coefficients by the integration

$$a(\mathbf{i'}, \mathbf{j'}) \propto \int\_0^{T\_\varepsilon} \int\_{\mathbf{s}\_\mathbf{y}(t') - \frac{1}{2}L\_\rho}^{\mathbf{s}\_\mathbf{y}(t') + \frac{1}{2}L\_\rho} \int\_{\mathbf{s}\_\mathbf{x}(t') - \frac{1}{2}L\_\rho}^{\mathbf{s}\_\mathbf{x}(t') + \frac{1}{2}L\_\rho} f\_\mathbf{p}(\mathbf{i(s\_\mathbf{s})} + \mathbf{i'}, \mathbf{j(s\_\mathbf{s})} + \mathbf{j'}, \mathbf{x'}, \mathbf{y'}) d\mathbf{x'} d\mathbf{y'} dt'. \tag{4}$$

This situation illustrate Fig. 5a or Fig. 5b. The responses

$$\mathbf{A}\_{\text{BOM},0} = [\begin{array}{cc} \frac{1}{12} & \frac{1}{6} & \frac{1}{6} & \frac{1}{6} & \frac{1}{6} & \frac{1}{12} \end{array}]^T \text{ } \mathbf{A}\_{\text{BOM},\pi/4} = \begin{bmatrix} \frac{1}{6} & \frac{1}{12} & 0 \\\\ \frac{1}{12} & \frac{1}{5} & \frac{1}{12} \\\\ 0 & \frac{1}{12} & \frac{1}{6} \end{bmatrix} \tag{5}$$

can be stated as the examples of such distortion channels that come from Eq. 4 biased by angles *ϕ* = 0, *ϕ* = *<sup>π</sup>* <sup>4</sup> and with confinement only to equable movement, when **s**(*t* � ) is linear function of time *t* � .

If the object expansion on the plane {**b***x*, **b***y*}, between the times *t* and *t* + *Te*, is not insignificant the situation in Fig. 5c occurs. Computation of the constants

$$a(\mathbf{i'}, \mathbf{j'}) \propto \int\_0^{T\_\varepsilon} \int\_{\mathbf{s}\_y(t') - \frac{1}{2}L\_\varepsilon(t')}^{\mathbf{s}\_y(t') + \frac{1}{2}L\_\varepsilon(t')} \int\_{\mathbf{s}\_x(t') - \frac{1}{2}L\_\varepsilon(t')}^{\mathbf{s}\_x(t') + \frac{1}{2}L\_\varepsilon(t')} f\_p(\mathbf{i(s\_s) + \mathbf{i'}}, \mathbf{j(s\_s) + \mathbf{j'}}, \mathbf{x'}, \mathbf{y'}) d\mathbf{x'} d\mathbf{y'} dt' \tag{6}$$

is analogical to Eq. 4, only with the difference that size of sliding sensor cell will not be equal to pixel size *Lp* anymore, but it will present the function *Lc*(*t* � ) linearly (Fig. 5c) or non-linearly dependent on the time *t* � in compliance with the movement of scanning object.

4 Will-be-set-by-IN-TECH

realization of par-axial intensity on the plane {**b***x*, **b***y*}, in the time *t*, as the mosaic *D*(*x*, *y*) =

1 iff <sup>|</sup>*<sup>x</sup>* <sup>−</sup> *iLp*| ≤ *Lp*

where *Lp* establish their size, that is equal or less then size *Lc* of the sensor cell (potential well).

Light rays

v

Fig. 4. The projection of the moving object point to the CCD/CMOS sensor plane.

� *sx* (*t* � )+ <sup>1</sup> <sup>2</sup> *Lp*

> *sx* (*t*�)<sup>−</sup> <sup>1</sup> <sup>2</sup> *Lp*

On the basis of declared definitions and approximations, we can subsequently express useful signal impacting the sensor by the convolution Eq. 1 emulating the situation, when the snapshot **D**, gained in the time *t*, is slided on the sensor surface and stepwise trapped to the its cells. The said fact can be also conceived from the opposite side, when potential well sliding on the immovable pattern **D**. If the velocity radial component is insignificant in the comparison with axial component (expansion of the object is negligible between times *t* and *t* + *Te*) and starting time *t* is equated to zero, we can obtain the impulse response coefficients

*fp*(*i*(**s***s*) + *i*

�

*<sup>T</sup>*, **A**BOM,*π*/4 =

can be stated as the examples of such distortion channels that come from Eq. 4 biased by

<sup>4</sup> and with confinement only to equable movement, when **s**(*t*

, *j*(**s***s*) + *j*

⎡ ⎢ ⎢ ⎢ ⎢ ⎣ 1 6 1 <sup>12</sup> 0 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦

1 12 1 3 1 12

0 <sup>1</sup> 12 1 6

� , *x*� , *y*� )*dx*� *dy*� *dt*�

. (4)

�

) is linear

(5)

by bx

<sup>=</sup><sup>1</sup> shaped by square function

<sup>2</sup> ∧ |*<sup>y</sup>* <sup>−</sup> *jLp*| ≤ *Lp*

<sup>0</sup> otherwise , (3)

Sensor

ss

se

2

<sup>∑</sup>*i*,*<sup>j</sup> <sup>d</sup>*(*i*, *<sup>j</sup>*)*fp*(*i*, *<sup>j</sup>*, *<sup>x</sup>*, *<sup>y</sup>*) of pixels *<sup>d</sup>*(*i*, *<sup>j</sup>*) ∈ {*d*()}*Md*

*fp*(*i*, *j*, *x*, *y*) =

Object

t + Te

t

� *sy* (*t* � )+ <sup>1</sup> <sup>2</sup> *Lp*

**A**BOM,0 = [ <sup>1</sup>

*sy* (*t*�)<sup>−</sup> <sup>1</sup> <sup>2</sup> *Lp*

This situation illustrate Fig. 5a or Fig. 5b. The responses

12 1 6 1 6 1 6 1 6 1 6 1 <sup>12</sup> ]

by the integration

*a*(*i* � , *j* � ) ∝ � *Te* 0

angles *ϕ* = 0, *ϕ* = *<sup>π</sup>*

� .

function of time *t*

�

(a) The example with **s***<sup>s</sup>* situated inside of the pixel (*i*(**s***s*), *j*(**s***s*)). Negligible radial component of the velocity.

(b) The example with **s***<sup>s</sup>* situated on the edge of the pixel (*i*(**s***s*), *j*(**s***s*)). Negligible radial component of the velocity.

(c) The example with **s***<sup>s</sup>* situated on the edge of the pixel (*i*(**s***s*), *j*(**s***s*)). Indispensable radial component of the velocity.

Fig. 5. The equivalent movement trajectory of the sensor sensing cell sliding on the pattern **D** (snapshot of the straight-line moving object with negligible and indispensable radial component of the velocity **v**).

the random IECS-ML channel with I/O equation *r*(*i*, *j*) = *Cq*˜(*i*, *j*) + *wT*(*i*, *j*) + *wR*(*i*, *j*), where *r*(*i*, *j*) is the channel output, *wT*(*i*, *j*) is the total thermal noise accumulated with the useful signal *q*(*i*, *j*) = *Cq*˜(*i*, *j*) as a charge in the CCD sensor cell at the position (*i*, *j*) and *wR*(*i*, *j*) is the readout noise. The constant *C* (above limited by a full well capacity FWC of the CCD sensor) defines a quantity of generated charge and it is directly proportional to the exposure

The output *r*(*i*, *j*) (IECS-ML channel output also registrable in the matrix form **R** = **Q** + **W***<sup>T</sup>* + **W***<sup>R</sup>* for entire sensor) is further quantized in the *Nb*-bit A/D converter that can be approximated by the random channel *rd*(*i*, *j*) = *r*(*i*, *j*) + *wQ*(*i*, *j*) with uniformly distributed noise *wQ*(*i*, *j*) in light of probability theory. We denote the output of this converter as

Heaviside step function. For simplicity we will assume the linear quantization only with

and quantization step <sup>Δ</sup>*Q*, where for the highest quantization level *tQ*(2*Nb* ) <sup>≤</sup> *FWC* is applied.

The channel has independent eliminated states (IECS), if noise sources in CCD sensor cells are mutually independent. It makes joint probability density function (PDF) *p***W**(Ξ) = ∏*i*,*<sup>j</sup> pw*(*ξ*) as the product of marginal densities. The channel is memory-less (ML), if the current *x*(*i*, *j*) depends only on the corresponding *q*(*i*, *j*). The fulfillment of both conditions creates the

*<sup>p</sup>***R**|**Q**,**W**(Ξ|**Qˇ** , <sup>Ξ</sup>�

iff iff

)*p***W**(Ξ�

*pr*|*q*,*w*(*ξ*|*q*ˇ(*i*, *<sup>j</sup>*), *<sup>ξ</sup>*�

 *pr*(*ξ*|*q*ˇ(*i*,*j*))

S[**R***d*, **D**ˇ ]

)*d*Ξ�

)*pw*(*ξ*�

)*dξ*�

, (9)

� = 1

(� − 1)Δ*<sup>Q</sup>* + 1/2

1

<sup>1</sup> < � <sup>≤</sup> <sup>2</sup>*Nb* (7)

iff *ξ*<0

otherwise is a

321

(8)

*rd*(*i*, *<sup>j</sup>*) = *<sup>Q</sup>*(*r*(*i*, *<sup>j</sup>*)), where *<sup>Q</sup>*(*ξ*) = <sup>∑</sup>1≤�≤2*Nb <sup>H</sup>*(*<sup>ξ</sup>* <sup>−</sup> *tQ*(�)) and *<sup>H</sup>*(*ξ*) = { <sup>0</sup>

*tQ*(�) = <sup>0</sup>

Image Restoration: Principles, Applications and Performance Analysis

**5. Image restoration — Symbol and page 2D MAP detection**

*<sup>p</sup>***R**(Ξ|**Qˇ** ) =

The optimal 2D MAP detector is based on the criterion

**W**

*w*(*i*,*j*)

factorizable that is necessary to transition from a single-stage detector to an IDN.

ˇ*d*(*i*,*j*)

where ˆ*d*(*i*, *j*) denotes the wanted estimation, ˇ*d*(*i*, *j*) denotes a testing estimator (takes individual values from data alphabet), **D**ˇ : ˇ*d*(*i*, *j*) denotes the set of possible image realizations containing the estimator ˇ*d*(*i*, *j*) and **<sup>M</sup>** with <sup>M</sup> denote certain types of marginalization

M **D**ˇ : ˇ*d*(*i*,*j*)

ˆ*d*(*i*, *j*) = arg **<sup>M</sup>**

= ∏ *i*,*j*

time *Te*.

threshold values

**5.1 Optimal MAP detection**

likeli-hood function

operators.

**4.2 Quantization noise of the A/D converter**

2D Iterative Detection Network Based

#### **3.3 Hypothetical cellular model of a 2D ISI channel — Encoding network (EN)**

The convolution Eq. 1 can be emulated by certain 2D hypothetical encoding network (EN). The assembly of this EN is variable and contains specific, relatively simple, functional blocks *fG*(). We denote these functional blocks as general processing elements (GPEs) and one such elements is illustrated in Fig. 6a.

In principle, a GPE presents a simple FSM whose inputs as well as outputs are variables S(*k*) discrete in values, that are derivable from alphabets {S(�)(*k*)}�. The inputs together with the outputs consequently create the set N = *<sup>k</sup>* <sup>S</sup>(*k*) flowing from the alphabet {N (�)}�.

(c) Structure of a SISO module.

Fig. 6. A general processing element (GPE) and its soft inversion (SISO module).

#### **4. Random IECS-ML channel (noise model)**

#### **4.1 Composite noise of the CCD/CMOS sensor**

The presence of the signal independent composite CCD/CMOS sensor noise in mutually correlated rays 0 ≤ *q*˜(*i*, *j*) ≤ 1 can be expressed as the blurred image transmission through the random IECS-ML channel with I/O equation *r*(*i*, *j*) = *Cq*˜(*i*, *j*) + *wT*(*i*, *j*) + *wR*(*i*, *j*), where *r*(*i*, *j*) is the channel output, *wT*(*i*, *j*) is the total thermal noise accumulated with the useful signal *q*(*i*, *j*) = *Cq*˜(*i*, *j*) as a charge in the CCD sensor cell at the position (*i*, *j*) and *wR*(*i*, *j*) is the readout noise. The constant *C* (above limited by a full well capacity FWC of the CCD sensor) defines a quantity of generated charge and it is directly proportional to the exposure time *Te*.

### **4.2 Quantization noise of the A/D converter**

6 Will-be-set-by-IN-TECH

The convolution Eq. 1 can be emulated by certain 2D hypothetical encoding network (EN). The assembly of this EN is variable and contains specific, relatively simple, functional blocks *fG*(). We denote these functional blocks as general processing elements (GPEs) and one such

In principle, a GPE presents a simple FSM whose inputs as well as outputs are variables S(*k*) discrete in values, that are derivable from alphabets {S(�)(*k*)}�. The inputs together with the

*<sup>k</sup>* <sup>S</sup>(*k*) flowing from the alphabet {N (�)}�.

f−<sup>1</sup> <sup>G</sup> ( )

(b) A SISO module.

SO[Sˇ(<sup>K</sup> <sup>−</sup> 1) SI[Sˇ(<sup>K</sup> <sup>−</sup> 1)]

SI[Sˇ(K)] SO[Sˇ(K)]

SO[Sˇ(<sup>k</sup> + 1)] SI[Sˇ(<sup>k</sup> + 1)]

SO[Sˇ(k)] SI[Sˇ(k)]

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

()�<sup>C</sup> ()

−1

{S[N (�)]}�

<sup>−</sup><sup>1</sup> �<sup>M</sup> ()�<sup>C</sup> ()

SI[Sˇ(<sup>k</sup> + 1)] SO[Sˇ(<sup>k</sup> + 1)]

SI[Sˇ(<sup>K</sup> <sup>−</sup> 1)] SO[Sˇ(<sup>K</sup> <sup>−</sup> 1)]

SO[Sˇ(K)] SI[Sˇ(K)]

SI[Sˇ(1)] SO[Sˇ(1)]

SI[Sˇ(2)] SO[Sˇ(2)]

**3.3 Hypothetical cellular model of a 2D ISI channel — Encoding network (EN)**

elements is illustrated in Fig. 6a.

/

S(2) S(K − 1)

fG( )

(a) A GPE.

S(k)

S(1)

outputs consequently create the set N =

S(K)

S(k + 1)

SI[Sˇ(1)]

SI[Sˇ(2)] SO[Sˇ(2)]

SO[Sˇ(k)]

SI[Sˇ(k)]

**4. Random IECS-ML channel (noise model) 4.1 Composite noise of the CCD/CMOS sensor**

SO[Sˇ(1)]

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

S[N()]

�<sup>C</sup> <sup>S</sup>ˇ(k)∈N<sup>ˇ</sup>

(c) Structure of a SISO module.

The presence of the signal independent composite CCD/CMOS sensor noise in mutually correlated rays 0 ≤ *q*˜(*i*, *j*) ≤ 1 can be expressed as the blurred image transmission through

Fig. 6. A general processing element (GPE) and its soft inversion (SISO module).

<sup>N</sup><sup>ˇ</sup> :Sˇ(k)

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

<sup>−</sup><sup>1</sup> ()�<sup>C</sup> ()

The output *r*(*i*, *j*) (IECS-ML channel output also registrable in the matrix form **R** = **Q** + **W***<sup>T</sup>* + **W***<sup>R</sup>* for entire sensor) is further quantized in the *Nb*-bit A/D converter that can be approximated by the random channel *rd*(*i*, *j*) = *r*(*i*, *j*) + *wQ*(*i*, *j*) with uniformly distributed noise *wQ*(*i*, *j*) in light of probability theory. We denote the output of this converter as *rd*(*i*, *<sup>j</sup>*) = *<sup>Q</sup>*(*r*(*i*, *<sup>j</sup>*)), where *<sup>Q</sup>*(*ξ*) = <sup>∑</sup>1≤�≤2*Nb <sup>H</sup>*(*<sup>ξ</sup>* <sup>−</sup> *tQ*(�)) and *<sup>H</sup>*(*ξ*) = { <sup>0</sup> 1 iff *ξ*<0 otherwise is a Heaviside step function. For simplicity we will assume the linear quantization only with threshold values

$$\Lambda\_Q(\ell) = \begin{cases} 0 & \text{iff} \quad \ell = 1\\ (\ell - 1)\Lambda\_Q + 1/2 & \text{iff } 1 < \ell \le 2^{N\_b} \end{cases} \tag{7}$$

and quantization step <sup>Δ</sup>*Q*, where for the highest quantization level *tQ*(2*Nb* ) <sup>≤</sup> *FWC* is applied.

## **5. Image restoration — Symbol and page 2D MAP detection**

#### **5.1 Optimal MAP detection**

The channel has independent eliminated states (IECS), if noise sources in CCD sensor cells are mutually independent. It makes joint probability density function (PDF) *p***W**(Ξ) = ∏*i*,*<sup>j</sup> pw*(*ξ*) as the product of marginal densities. The channel is memory-less (ML), if the current *x*(*i*, *j*) depends only on the corresponding *q*(*i*, *j*). The fulfillment of both conditions creates the likeli-hood function

$$\begin{split} p\_{\mathbf{R}}(\boldsymbol{\Xi}|\check{\mathbf{Q}}) &= \int\_{\mathbf{W}} p\_{\mathbf{R}|\mathbf{Q},\mathbf{W}}(\boldsymbol{\Sigma}|\check{\mathbf{Q}},\boldsymbol{\Sigma}') p\_{\mathbf{W}}(\boldsymbol{\Sigma}') d\boldsymbol{\Sigma}' \\ &= \prod\_{i,j} \underbrace{\int\_{w(i,j)} p\_{r|q,w}(\boldsymbol{\xi}|\boldsymbol{\eta}(i,j),\boldsymbol{\xi}') p\_{w}(\boldsymbol{\xi}') d\boldsymbol{\xi}'}\_{p\_{r}(\boldsymbol{\xi}|\boldsymbol{\eta}(i,j))} \end{split} \tag{8}$$

factorizable that is necessary to transition from a single-stage detector to an IDN.

The optimal 2D MAP detector is based on the criterion

$$\hat{d}(i,j) = \arg \underset{d(i,j)}{\oplus} \left[ \underset{\dot{\mathbf{D}}: d(i,j)}{\oplus} \mathbf{S}[\mathbf{R}\_{d\prime}\dot{\mathbf{D}}] \right],\tag{9}$$

where ˆ*d*(*i*, *j*) denotes the wanted estimation, ˇ*d*(*i*, *j*) denotes a testing estimator (takes individual values from data alphabet), **D**ˇ : ˇ*d*(*i*, *j*) denotes the set of possible image realizations containing the estimator ˇ*d*(*i*, *j*) and **<sup>M</sup>** with <sup>M</sup> denote certain types of marginalization operators.

emulating the image blurring. The fulfillment of the condition IECS-ML enables a usage of such GPEs that compose the hypothetical realization **D**ˇ of a sensing image even by smallest parts (by individual pixels ˇ*d*(*i*, *j*)) if this yields a implementation benefits in light of concrete

The IDN output can be regarded as a optimal (identical with the output of single-stage detector) after the execution of infinite number of iterations (information interchange between inversions). It is not practicable. Therefore, the IDN with the finite number of iterations is sub-optimal detector, that generally provides an inferior estimation to the single-stage detector. Roughly speaking the IDN contains simpler SISO modules, the IDN is more numerically effective and more sub-optimal (it includes more iteratively refining variables). In the course of each iteration *I* of the IDN, from count *NI*, is every SISO module once activated at the least. The activation rests in the reading of soft measures (whole probability or metric densities {SI[S()(*k*)]}, corresponding to certain random variable <sup>S</sup>(*k*) in the hypothetical model) on inputs of a SISO module, followed by the enumeration of output soft measures (whole densities {SO[S()(*k*)]}). The current iteration concludes the exchange of

> → ←

among the neighboring modules. After execution of all iteration *NI*, the estimations of all

An IDN presents a soft inversion of an arbitrary EN formed from certain mutually concatenated GPEs that jointly execute an arbitrary processing with input signal. In the our case, as was said in the paragraph 3.3, such EN executes the convolution Eq. 1 and due to fulfillment of the IECS-ML condition can contain the simplest GPEs working on the level of individual pixels. The IDN topology exactly agrees to the EN topology only with the difference, that each of GPE is substituted by its SISO module, just like in Fig. 6a, 6b. The signal processing in the soft inversion is implied in Fig. 6c and takes place in two steps. Firstly,

{SI[S()(*k*)]} {SO[S()(*k*)]}

[SI[Sˇ(*k*)]<sup>C</sup> SO[Sˇ(*k*)]]. (13)

SI[Sˇ(*k*)] for ∀N<sup>ˇ</sup> . (14)

<sup>C</sup> <sup>−</sup>1SI[Sˇ(*k*)] for ∀Sˇ(*k*) (15)

*<sup>G</sup>* () **(SISO module) of general processing element**

(12)

323

{SO[S()(*k*)]} {SI[S()(*k*)]}

wanted (output) random variables are performed from the formula

<sup>S</sup>ˆ(*k*) = arg **<sup>M</sup>**

the inputs {SI[S()(*k*)]},*<sup>k</sup>* are combined to particular joint soft measures

<sup>S</sup>[N<sup>ˇ</sup> ] = **<sup>C</sup>**

M <sup>N</sup><sup>ˇ</sup> :Sˇ(*k*)

realizations <sup>N</sup> (), are marginalized to outputs {SO[S()(*k*)]},*<sup>k</sup>*

SO[Sˇ(*k*)] =

<sup>S</sup>ˇ(*k*)∈N<sup>ˇ</sup>

<sup>S</sup>[N<sup>ˇ</sup> ] 

Consequently, all joint measures {S[<sup>N</sup> ()]}, corresponding to the set ∀N<sup>ˇ</sup> of possible

<sup>S</sup>ˇ(*k*)

We express this operation as a hard decision provided by a decision block (DEC).

modeled distortion.

2D Iterative Detection Network Based

Image Restoration: Principles, Applications and Performance Analysis

soft measures

**6. Iterative detection networks**

*fG*() **(FSM)**

**6.1 IDN topology and soft inversion** *f* <sup>−</sup><sup>1</sup>

The quantity S[**R***d*, **D**ˇ ] is to be understood as some kind of the joint soft measure and due to IECS-ML condition fulfillment

$$\begin{aligned} \mathbb{S}[\mathbf{R}\_d|\check{\mathbf{D}}] &= \underset{i,j}{\mathbb{S}} \mathbb{S}[r\_d(i,j)|\check{\mathbf{D}}] \\ &= \underset{i,j}{\mathbb{S}} \mathbb{S}[r\_d(i,j)|\check{\mathcal{N}}\_d(i,j) \subset \mathbf{D}] \end{aligned} \tag{10}$$

with assumption of a statistically independent data S[**D**ˇ ] = **<sup>C</sup>** *i*,*j* S[ ˇ*d*(*i*, *j*)] can be decomposed to

the form

$$\begin{aligned} \mathbb{S}[\mathbf{R}\_d \mid \dot{\mathbf{D}}] &= \mathbb{S}[\mathbf{R}\_d | \dot{\mathbf{D}}] \circledS \mathbf{S}[\dot{\mathbf{D}}] \\ &= \left( \underset{i,j}{\mathbb{S}} \mathbb{S}[r\_d(i,j) | \check{\mathcal{N}}\_d(i,j) \subset \mathbf{D}] \right) \odot \left( \underset{i,j}{\mathbb{S}} \mathbb{S}[d(i,j)] \right) \end{aligned} \tag{11}$$

that is joined by certain types of combination operators (**<sup>C</sup>** , <sup>C</sup> ). On the basis of marginalization and combination operators, we can split detectors into four groups in light of detection technique (symbol or page) and implementing domain (probability P[ ] or equivalent logarithmic metric M[]= − ln(P[ ])). All possibilities are summarized in Table 1.

The symbol technique (SyD) seeks to minimize of the actual symbol detection error only. The page detection (PgD) has tendency to minimization the entire page detection error. The most numerically effective is the Md-PgD alternative, because it contains the simple combination operator as well as the simple marginalization operator. Close to the Md-PgD, the Md-SyD conjunction with the relatively simple marginalization operator min∗(*x*, *y*) = min(*x*, *<sup>y</sup>*) <sup>−</sup> ln(<sup>1</sup> <sup>+</sup> *<sup>e</sup>*−|*x*−*y*<sup>|</sup> ).


Table 1. Summary of combination and marginalization operators.

#### **5.2 Suboptimal MAP detection**

The direct evaluation of the **D**ˆ from the criterion (9) (single-stage detection) is impossible, because it requires a sequent substitution of all potential image realizations **D**ˇ . But Eq. 10 (IECS-ML condition) makes possible decomposition of the detection problem from the entire page **D** to the level of individual (mutually overlapping) convolution regions N*d*(*i*, *j*), corresponding to individual captured pixels. Therefore, we can substitute the single-stage MAP detector by the sub-optimal iterative detection network (IDN).

Such network is formed from a definite number of functional blocks, so-called soft inversions (SISO modules), that exchange the soft measures with each other. The SISO modules present statistical devices complementary to the GPEs in the appropriately designed hypothetical EN 8 Will-be-set-by-IN-TECH

The quantity S[**R***d*, **D**ˇ ] is to be understood as some kind of the joint soft measure and due to

<sup>S</sup>[*rd*(*i*, *<sup>j</sup>*)|**D**<sup>ˇ</sup> ]

<sup>S</sup>[*rd*(*i*, *<sup>j</sup>*)|N<sup>ˇ</sup>

*<sup>d</sup>*(*i*, *<sup>j</sup>*) <sup>⊂</sup> **<sup>D</sup>**<sup>ˇ</sup> ]

that is joined by certain types of combination operators (**<sup>C</sup>** , <sup>C</sup> ). On the basis of marginalization and combination operators, we can split detectors into four groups in light of detection technique (symbol or page) and implementing domain (probability P[ ] or equivalent logarithmic metric M[]= − ln(P[ ])). All possibilities are summarized in Table 1. The symbol technique (SyD) seeks to minimize of the actual symbol detection error only. The page detection (PgD) has tendency to minimization the entire page detection error. The most numerically effective is the Md-PgD alternative, because it contains the simple combination operator as well as the simple marginalization operator. Close to the Md-PgD, the Md-SyD conjunction with the relatively simple marginalization operator min∗(*x*, *y*) =

Domain S[ ] Detection **<sup>M</sup>** <sup>M</sup> **<sup>C</sup>** <sup>C</sup> <sup>C</sup> <sup>−</sup><sup>1</sup>

Probability (Pd) P[ ] Page (PgD) max max Π × ÷ Probability (Pd) P[ ] Symbol (SyD) max Σ Π × ÷ Metric (Md) M[ ] Page (PgD) min min Σ + − Metric (Md) M[ ] Symbol (SyD) min min<sup>∗</sup> Σ + −

The direct evaluation of the **D**ˆ from the criterion (9) (single-stage detection) is impossible, because it requires a sequent substitution of all potential image realizations **D**ˇ . But Eq. 10 (IECS-ML condition) makes possible decomposition of the detection problem from the entire page **D** to the level of individual (mutually overlapping) convolution regions N*d*(*i*, *j*), corresponding to individual captured pixels. Therefore, we can substitute the single-stage

Such network is formed from a definite number of functional blocks, so-called soft inversions (SISO modules), that exchange the soft measures with each other. The SISO modules present statistical devices complementary to the GPEs in the appropriately designed hypothetical EN

*<sup>d</sup>*(*i*, *<sup>j</sup>*) <sup>⊂</sup> **<sup>D</sup>**<sup>ˇ</sup> ] (10)

S[ ˇ*d*(*i*, *j*)]

S[ ˇ*d*(*i*, *j*)] can be decomposed to

(11)

*i*,*j*

 C **<sup>C</sup>** *i*,*j*

<sup>S</sup>[**R***d*|**D**<sup>ˇ</sup> ] = **<sup>C</sup>**

with assumption of a statistically independent data S[**D**ˇ ] = **<sup>C</sup>**

<sup>S</sup>[**R***d*, **<sup>D</sup>**<sup>ˇ</sup> ] = <sup>S</sup>[**R***d*|**D**<sup>ˇ</sup> ]<sup>C</sup> <sup>S</sup>[**D**<sup>ˇ</sup> ]

= **<sup>C</sup>** *i*,*j*

).

Table 1. Summary of combination and marginalization operators.

MAP detector by the sub-optimal iterative detection network (IDN).

*i*,*j*

= **<sup>C</sup>** *i*,*j*

<sup>S</sup>[*rd*(*i*, *<sup>j</sup>*)|N<sup>ˇ</sup>

IECS-ML condition fulfillment

min(*x*, *<sup>y</sup>*) <sup>−</sup> ln(<sup>1</sup> <sup>+</sup> *<sup>e</sup>*−|*x*−*y*<sup>|</sup>

**5.2 Suboptimal MAP detection**

the form

emulating the image blurring. The fulfillment of the condition IECS-ML enables a usage of such GPEs that compose the hypothetical realization **D**ˇ of a sensing image even by smallest parts (by individual pixels ˇ*d*(*i*, *j*)) if this yields a implementation benefits in light of concrete modeled distortion.

The IDN output can be regarded as a optimal (identical with the output of single-stage detector) after the execution of infinite number of iterations (information interchange between inversions). It is not practicable. Therefore, the IDN with the finite number of iterations is sub-optimal detector, that generally provides an inferior estimation to the single-stage detector. Roughly speaking the IDN contains simpler SISO modules, the IDN is more numerically effective and more sub-optimal (it includes more iteratively refining variables).

In the course of each iteration *I* of the IDN, from count *NI*, is every SISO module once activated at the least. The activation rests in the reading of soft measures (whole probability or metric densities {SI[S()(*k*)]}, corresponding to certain random variable <sup>S</sup>(*k*) in the hypothetical model) on inputs of a SISO module, followed by the enumeration of output soft measures (whole densities {SO[S()(*k*)]}). The current iteration concludes the exchange of soft measures

$$\begin{aligned} \{\text{SO}[\mathcal{S}^{(\ell)}(k)]\}\_{\ell} &\to \{\text{SI}[\mathcal{S}^{(\ell)}(k)]\}\_{\ell} \\ \{\text{SI}[\mathcal{S}^{(\ell)}(k)]\}\_{\ell} &\leftarrow \{\text{SO}[\mathcal{S}^{(\ell)}(k)]\}\_{\ell} \end{aligned} \tag{12}$$

among the neighboring modules. After execution of all iteration *NI*, the estimations of all wanted (output) random variables are performed from the formula

$$\mathcal{S}(k) = \arg \sup\_{\mathcal{S}(k)} \left[ \text{SI}[\mathcal{S}(k)] \ominus \text{SO}[\mathcal{S}(k)] \right]. \tag{13}$$

We express this operation as a hard decision provided by a decision block (DEC).

## **6. Iterative detection networks**

#### **6.1 IDN topology and soft inversion** *f* <sup>−</sup><sup>1</sup> *<sup>G</sup>* () **(SISO module) of general processing element** *fG*() **(FSM)**

An IDN presents a soft inversion of an arbitrary EN formed from certain mutually concatenated GPEs that jointly execute an arbitrary processing with input signal. In the our case, as was said in the paragraph 3.3, such EN executes the convolution Eq. 1 and due to fulfillment of the IECS-ML condition can contain the simplest GPEs working on the level of individual pixels. The IDN topology exactly agrees to the EN topology only with the difference, that each of GPE is substituted by its SISO module, just like in Fig. 6a, 6b. The signal processing in the soft inversion is implied in Fig. 6c and takes place in two steps. Firstly, the inputs {SI[S()(*k*)]},*<sup>k</sup>* are combined to particular joint soft measures

$$\mathbf{S}[\check{\mathcal{N}}] = \underset{\mathcal{S}(k)\in\mathcal{N}}{\ominus} \mathbf{SI}[\check{\mathcal{S}}(k)] \text{ for } \forall \check{\mathcal{N}}.\tag{14}$$

Consequently, all joint measures {S[<sup>N</sup> ()]}, corresponding to the set ∀N<sup>ˇ</sup> of possible realizations <sup>N</sup> (), are marginalized to outputs {SO[S()(*k*)]},*<sup>k</sup>*

$$\mathbf{SO}[\mathring{\mathcal{S}}(k)] = \left( \underset{\check{\mathcal{N}} : \check{\mathcal{S}}(k)}{\otimes} \mathbf{S}[\check{\mathcal{N}}] \right) \odot^{-1} \mathbf{SI}[\check{\mathcal{S}}(k)] \text{ for } \forall \check{\mathcal{S}}(k) \tag{15}$$

<sup>0</sup> <sup>10</sup> <sup>20</sup> <sup>30</sup> <sup>40</sup> <sup>50</sup> <sup>0</sup>

2D Iterative Detection Network Based

ξ

for such purpose, which represents the solution of the equation

Image Restoration: Principles, Applications and Performance Analysis

*d dμ*

*K* ∑ *k*=1 ln *e* <sup>−</sup>*<sup>μ</sup> <sup>μ</sup>w*(*k*) *w*(*k*)!

<sup>0</sup> <sup>10</sup> <sup>20</sup> <sup>30</sup> <sup>40</sup> <sup>50</sup> <sup>0</sup>

ξ

= 0 (19)

325

0.05 0.1 0.15 0.2 0.25 0.3 0.35

> *μ*=*μ*ˆ

pwR (ξ)

Fig. 7. The example of the thermal and readout noise PDF for *μ<sup>T</sup>* = 1, *μ<sup>R</sup>* = 5, *σ<sup>R</sup>* = 5 and

derived from the ML criterion. The set of values {*w*(1), *w*(2),..., *w*(*k*),..., *w*(*K*)} is the realization of the composite noise on the blacked out CCD sensor strip. The average value of the composite noise *μ*ˆ, obtained by the measurement from the blacked out area is used for the computation of *μ*ˆ*<sup>T</sup>* = *μ*ˆ − *μR*. That, substituted back to the density *pwT* (*ξ*), produces its

Statistical properties of the composite noise *w*(*i*, *j*) presents the wanted noise model, that controls the IECS-ML channel (channel with additive noise) behavior and results from the

0

0 0.1 0.2 0.3 0.4 0.5

ˆprd (ξ|ˇq(i, j))

Fig. 8. The *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) and *p*ˆ*rd* (*ξ*|*q*ˇ(*i*, *j*)) PDF corresponding to the given examples densities

5

10

ξ

15 0

10 20 30 40

qˇ(i, j)

0.05 0.1 0.15 0.2 0.25 0.3 0.35

FWC = 50.

estimation *p*ˆ*wT* (*ξ*).

0

0 0.05 0.1 0.15 0.2

ˆpr(ξ|ˇq(i, j))

20

40

ξ

0 10 20 30 40

of the thermal and readout noise. *Nb* = 4 and Δ*<sup>Q</sup>* = 3.

qˇ(i, j)

ˆpwT (ξ)

that are connected to the inputs {SI[S(�)(*k*)]}�,*<sup>k</sup>* of neighboring soft inversions. This update of soft information is known as a soft inversion activation. The activation accomplishment of all soft inversions in the IDN afterwards makes one iteration of the IDN.

#### **6.2 Soft-output demodulator**

The IDN closely cooperates with the SODEM (Soft output demodulator) providing fundamental hypothesis about captured signal **R***d*. This functional block presents a gateway between the domain of a real realizations (realizations of certain random variables), where are operated with scalars (factual realizations), and the probability (metric) domain of the IDN, where are operated with whole densities. The SODEM includes the likeli-hood function (or transfer function) of the IECS-ML channel emulating all noises incident in the image capturing system chain (in the our case it is CCD/CMOS camera). The SODEM input forms realization *rd*(*i*, *<sup>j</sup>*) that is inside transformed to the discrete a posteriori density {SI[*q*(�)(*i*, *<sup>j</sup>*)]}� presenting the input (sufficient statistic) of the IDN.

Before the sufficient statistic derivation as such, it should be noted that the derivation is purely theoretical and does not correspond to any particular type of CCD / CMOS sensor. We start the SODEM derivation with the definition of discrete cut off Gaussian PDF

$$\begin{split} p\_{\mathcal{N}\_0^{\mathrm{FWC}}}(\boldsymbol{\xi}, \boldsymbol{\mu}, \sigma) \approx & \frac{1}{\sqrt{2\pi}\sigma} \sum\_{\ell=0}^{\mathrm{FWC}} \delta(\boldsymbol{\xi} - \ell) \exp\left(-\frac{(\ell - \mu)^2}{2\sigma^2}\right) \\ & + \frac{1}{2} \delta(\boldsymbol{\xi} - \mathrm{FWC}) \mathrm{erfc}\left(\frac{2\mathrm{FWC} + 1 - 2\mu}{\sqrt{8}\sigma}\right) \\ & + \delta(\boldsymbol{\xi}) \left(1 - \frac{1}{2} \mathrm{erfc}\left(-\frac{1 + 2\mu}{\sqrt{8}\sigma}\right)\right). \end{split} \tag{16}$$

All parameters FWC, *ξ*, *μ*, *σ*, etc. in all densities, we will assume in terms of number of electrons [-] either directly or as equivalent quantities related to number of electrons (for example: the quantization noise occurring in the A/D converter, we will consider as if the equivalent electron noise source deteriorative in the sensor, etc.). On the basis of the mentioned definition can be formed thermal and readout noise PDFs as

$$p\_{\textit{w}\uparrow}(\xi) \approx \begin{cases} 0 & \text{iff} \quad \frac{\mu\_{\textit{T}} < 40}{\xi \ge 70} \\ e^{-\mu\_{\textit{T}}} \sum\_{\ell \in \mathbb{N}\_{0}} \frac{\mu\_{\textit{T}}^{\ell}}{\ell!} \delta(\xi - \ell) & \text{iff} \quad \frac{\mu\_{\textit{T}} < 40}{\xi < 70} \\ p\_{\textit{N}\_{0}^{\text{FWC}}}(\xi, \mu\_{\textit{T}'}, \sqrt{\mu\_{\textit{T}}}) & \text{iff} \; \mu\_{\textit{T}} \ge 40 \end{cases} \tag{17}$$

and

$$p\_{\mathfrak{w}\_{\mathbb{R}}}(\xi) = p\_{\mathcal{N}\_0^{\mathsf{FWC}}}(\mathfrak{f}\_{\prime}\,\mu\_{\mathbb{R}}\,\sigma\_{\mathbb{R}}).\tag{18}$$

Both noises together then create the composite noise *w*(*i*, *j*) = *wT*(*i*, *j*) + *wR*(*i*, *j*) and examples of theirs densities, for certain parameters *μT*, *μ<sup>R</sup>* and *σR*, is shown in Fig. 7.

The mentioned PDFs are biased by three parameters *μR*, *σ<sup>R</sup>* and *μT*. First two parameters are known, because depend on the sensor readout rate that is also known. The last one is the nuisance unknown parameter consisting in the unknown CCD/CMOS sensor temperature and has to be estimated by a suitable way. A ML estimation *μ*ˆ = *N*−<sup>1</sup> ∑*<sup>K</sup> <sup>k</sup>*=<sup>1</sup> *w*(*k*) can be used 10 Will-be-set-by-IN-TECH

that are connected to the inputs {SI[S(�)(*k*)]}�,*<sup>k</sup>* of neighboring soft inversions. This update of soft information is known as a soft inversion activation. The activation accomplishment of all

The IDN closely cooperates with the SODEM (Soft output demodulator) providing fundamental hypothesis about captured signal **R***d*. This functional block presents a gateway between the domain of a real realizations (realizations of certain random variables), where are operated with scalars (factual realizations), and the probability (metric) domain of the IDN, where are operated with whole densities. The SODEM includes the likeli-hood function (or transfer function) of the IECS-ML channel emulating all noises incident in the image capturing system chain (in the our case it is CCD/CMOS camera). The SODEM input forms realization *rd*(*i*, *<sup>j</sup>*) that is inside transformed to the discrete a posteriori density {SI[*q*(�)(*i*, *<sup>j</sup>*)]}� presenting

Before the sufficient statistic derivation as such, it should be noted that the derivation is purely theoretical and does not correspond to any particular type of CCD / CMOS sensor. We start

*δ*(*ξ* − FWC)erfc

All parameters FWC, *ξ*, *μ*, *σ*, etc. in all densities, we will assume in terms of number of electrons [-] either directly or as equivalent quantities related to number of electrons (for example: the quantization noise occurring in the A/D converter, we will consider as if the equivalent electron noise source deteriorative in the sensor, etc.). On the basis of the

> *μ*� *T*

(*ξ*, *μT*,

0

Both noises together then create the composite noise *w*(*i*, *j*) = *wT*(*i*, *j*) + *wR*(*i*, *j*) and examples

The mentioned PDFs are biased by three parameters *μR*, *σ<sup>R</sup>* and *μT*. First two parameters are known, because depend on the sensor readout rate that is also known. The last one is the nuisance unknown parameter consisting in the unknown CCD/CMOS sensor temperature

*δ*(*ξ* − �) exp

�

−<sup>1</sup> <sup>+</sup> <sup>2</sup>*<sup>μ</sup>* <sup>√</sup>8*<sup>σ</sup>*

0 iff *<sup>μ</sup>T*<<sup>40</sup>

�! *<sup>δ</sup>*(*<sup>ξ</sup>* <sup>−</sup> �) iff *<sup>μ</sup>T*<<sup>40</sup>

<sup>√</sup>*μT*) iff *<sup>μ</sup><sup>T</sup>* <sup>≥</sup> <sup>40</sup>

−(� <sup>−</sup> *<sup>μ</sup>*)<sup>2</sup> 2*σ*<sup>2</sup>

��

*ξ*≥70

*ξ*<70

(*ξ*, *μR*, *σR*). (18)

�2FWC <sup>+</sup> <sup>1</sup> <sup>−</sup> <sup>2</sup>*<sup>μ</sup>* <sup>√</sup>8*<sup>σ</sup>*

�

�

. (16)

*<sup>k</sup>*=<sup>1</sup> *w*(*k*) can be used

(17)

FWC ∑ �=0

the SODEM derivation with the definition of discrete cut off Gaussian PDF

<sup>√</sup>2*πσ*

+*δ*(*ξ*) � <sup>1</sup> <sup>−</sup> <sup>1</sup> 2 erfc �

+ 1 2

mentioned definition can be formed thermal and readout noise PDFs as

*<sup>e</sup>*−*μ<sup>T</sup>* <sup>∑</sup>�∈**N**<sup>0</sup>

*<sup>p</sup>*<sup>N</sup> FWC 0

*pwR* (*ξ*) = *<sup>p</sup>*<sup>N</sup> FWC

of theirs densities, for certain parameters *μT*, *μ<sup>R</sup>* and *σR*, is shown in Fig. 7.

and has to be estimated by a suitable way. A ML estimation *μ*ˆ = *N*−<sup>1</sup> ∑*<sup>K</sup>*

⎧ ⎪⎪⎨

⎪⎪⎩

*pwT* (*ξ*) ≈

and

(*ξ*, *<sup>μ</sup>*, *<sup>σ</sup>*) <sup>≈</sup> <sup>1</sup>

soft inversions in the IDN afterwards makes one iteration of the IDN.

**6.2 Soft-output demodulator**

the input (sufficient statistic) of the IDN.

*<sup>p</sup>*<sup>N</sup> FWC 0

Fig. 7. The example of the thermal and readout noise PDF for *μ<sup>T</sup>* = 1, *μ<sup>R</sup>* = 5, *σ<sup>R</sup>* = 5 and FWC = 50.

for such purpose, which represents the solution of the equation

$$\left. \frac{d}{d\mu} \sum\_{k=1}^{K} \ln \left( e^{-\mu} \frac{\mu^{w(k)}}{w(k)!} \right) \right|\_{\mu=\mu} = 0 \tag{19}$$

derived from the ML criterion. The set of values {*w*(1), *w*(2),..., *w*(*k*),..., *w*(*K*)} is the realization of the composite noise on the blacked out CCD sensor strip. The average value of the composite noise *μ*ˆ, obtained by the measurement from the blacked out area is used for the computation of *μ*ˆ*<sup>T</sup>* = *μ*ˆ − *μR*. That, substituted back to the density *pwT* (*ξ*), produces its estimation *p*ˆ*wT* (*ξ*).

Statistical properties of the composite noise *w*(*i*, *j*) presents the wanted noise model, that controls the IECS-ML channel (channel with additive noise) behavior and results from the

Fig. 8. The *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) and *p*ˆ*rd* (*ξ*|*q*ˇ(*i*, *j*)) PDF corresponding to the given examples densities of the thermal and readout noise. *Nb* = 4 and Δ*<sup>Q</sup>* = 3.

**7. Distributed iterative detection networks**

2D Iterative Detection Network Based

variables are established by the condition

*H***A**, *j* + *W***A**) and it is possible when *a*(*H***A**, *W***A**) �= 0.

One of possible results is shown in Fig. 12b.

Fig. 11b, performing the combination

� )] =

� , *<sup>j</sup>*)] =

SO[ <sup>ˇ</sup>*d*(*<sup>i</sup>* <sup>+</sup> *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**)] =

with marginalizations

SO[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> *<sup>j</sup>*

SO[Cˇ(*<sup>i</sup>* <sup>+</sup> *<sup>i</sup>*

and

Eq. 13.

**7.1 Distributed IDN marginalizing at the symbol block level**

Image Restoration: Principles, Applications and Performance Analysis

Firstly we focus on the IDNs marginalizing at the symbol block level with horizontal and vertical state variable. These IDNs result from the EN [1] in Fig. 10a where each node (GPE), shown in Fig. 11b, creates a one functional block with the fixed system of inputs and outputs N*f*(*i*, *j*) = {R(*i*, *j*), C(*i*, *j*), *d*(*i* + *H***A**, *j* + *W***A**), R(*i*, *j* + 1), C(*i* + 1, *j*), *q*(*i*, *j*)}, where R(*i*, *j*) and C(*i*, *j*) present auxiliary state variables containing more symbols together. Shapes of these

N*d*(*i*, *j*) R(*i*, *j*) ∪ C(*i*, *j*) ∪ *d*(*i* + *H***A**, *j* + *W***A**)

The optimal decomposition of the convolution region is N*d*(*i*, *j*) = R(*i*, *j*) ∪ C(*i*, *j*) ∪ *d*(*i* +

In the case of a channel **A**GBC,*a*(0,0), with regard to the condition Eq. 24, the variables are R(*i*, *j*) = {*d*(*i*, *j*), *d*(*i*, *j* + 1), *d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 2, *j*), *d*(*i* + 2, *j* + 1)} and C(*i*, *j*) = {*d*(*i*, *j*), *d*(*i*, *j* + 1), *d*(*i*, *j* + 2), *d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 1, *j* + 2)}. Their shapes you can see in Fig. 12a. In a similar way we can get decomposition of the N*d*(*i*, *j*) for a channel **A**GBC,*a*(0,0).

The IDN has same topology as the EN and each its node forms SISO module, illustrated in

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*(*i*, *<sup>j</sup>*)] = SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>*)]<sup>C</sup> SI[Cˇ(*i*, *<sup>j</sup>*)]<sup>C</sup> SI[ <sup>ˇ</sup>*d*(*<sup>i</sup>* <sup>+</sup> *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**)]<sup>C</sup> SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*()(*i*, *<sup>j</sup>*)]

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*()(*i*, *<sup>j</sup>*)]

M <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*):R<sup>ˇ</sup> (*i*,*j*+*j*�)

M <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*):Cˇ(*i*+*i*�

SO[*q*ˇ(*i*, *<sup>j</sup>*)] =

,*j*)

M <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*): <sup>ˇ</sup>*d*(*i*+*H***A**,*j*+*W***A**)

> M <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*):*q*ˇ(*i*,*j*)

⊃ R(*i*, *j* + 1), C(*i* + 1, *j*). (24)

<sup>C</sup> <sup>−</sup>1SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> *<sup>j</sup>*

<sup>C</sup> <sup>−</sup>1SI[Cˇ(*<sup>i</sup>* <sup>+</sup> *<sup>i</sup>*

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*()(*i*, *<sup>j</sup>*)]

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*()(*i*, *<sup>j</sup>*)]

One iteration of such IDN rests in the serial activation of each SISO module from the upper left corner to the lower right corner in the IDN as is marked by the light blue curve in Fig. 10a. After finishing all iterations, of the chosen count *NI*, the IDN makes hard decisions ˆ*d*(*i*, *j*) per

� )] for *j*

, *j*)] for *i*

�

<sup>C</sup> SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)]<sup>C</sup> SI[*q*ˇ(*i*, *<sup>j</sup>*)] (25)

<sup>C</sup> <sup>−</sup>1SI[ ˇ*d*(*i* + *H***A**, *j* + *W***A**)], (28)

<sup>C</sup> <sup>−</sup>1SI[*q*ˇ(*i*, *j*)]. (29)

� ∈ {0, 1}, (26)

327

� ∈ {0, 1}, (27)

cut off PDF

$$\mathfrak{p}\_{w}(\xi) \approx \mathfrak{p}\_{w}^{\prime}(\xi) + \delta(\xi - \text{FWC}) \left( 1 - \int\_{-\frac{1}{2}}^{\text{FWC} + \frac{1}{2}} \mathfrak{p}\_{w}^{\prime}(\xi^{\prime}) d\xi^{\prime} \right), \tag{20}$$

based on the convolution

$$\mathfrak{p}'\_{w}(\mathfrak{F}) = \sum\_{\ell=0}^{\text{FWC}} \delta(\mathfrak{F} - \ell) \int\_{-\frac{1}{2}}^{\mathfrak{x} + \frac{1}{2}} \mathfrak{p}\_{w\tau}(\ell - \mathfrak{F}') p\_{w\mathfrak{x}}(\mathfrak{F}') d\mathfrak{F}' \tag{21}$$

of densities *p*ˆ*wT* (*ξ*) and *pwT* (*ξ*).

The procedure obtaining the metric SI[*q*ˇ(*i*, *j*)] from the derived noise model begins with the elimination Eq. 10 of the noise density *p*ˆ*w*(*ξ*) out of the likeli-hood function of a channel with additive noise *<sup>p</sup>***R**|**Q**,**W**(Ξ|**Q**<sup>ˇ</sup> , <sup>Ξ</sup>� ) = *<sup>δ</sup>*(<sup>Ξ</sup> <sup>−</sup> (**Q**<sup>ˇ</sup> <sup>+</sup> <sup>Ξ</sup>� )) or *pr*|*q*,*w*(*ξ*|*q*ˇ(*i*, *<sup>j</sup>*), *<sup>ξ</sup>*� ) = *δ*(*ξ* − (*q*ˇ(*i*, *j*) + *ξ*� )), assuming perfect knowledge of the noise realization **W**. The result of this elimination is the density *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) = *p*ˆ*w*(*q*ˇ(*i*, *j*) − *ξ*). Along with the thermal noise, another source of noise is found in the chain. It is the quantization noise with the uniform PDF *pwQ* which is added to the signal in the A/D converter. It can be eliminated out of the density *p*ˆ*<sup>r</sup>* by the integrating

$$
\begin{split}
\mathfrak{p}\_{\mathfrak{r}\_{\delta}}(\xi|\mathfrak{j}(i,j)) &= \delta(\xi - 2^{N\_{\mathfrak{b}}}) \lim\_{\varepsilon \to 0^{+}} \int\_{t\_{\mathbb{Q}}(2^{N\_{\mathfrak{b}}}) - \varepsilon}^{\mathrm{FWC} + \frac{1}{2}} \mathfrak{p}\_{\mathfrak{r}}(\xi'|\mathfrak{j}(i,j)) d\xi' \\ &+ \sum\_{\ell=1}^{2^{N\_{\mathfrak{b}}} - 1} \delta(\xi - \ell) \lim\_{\varepsilon \to 0^{+}} \int\_{t\_{\mathbb{Q}}(\ell) - \varepsilon}^{t\_{\mathbb{Q}}(\ell+1) - \varepsilon} \mathfrak{p}\_{\mathfrak{r}}(\xi'|\mathfrak{j}(i,j)) d\xi' \end{split} \tag{22}
$$

over individual quantization steps (by averaging with the density *pwQ* ) of the size Δ*<sup>Q</sup>* expressed in the number of electrons and defined, in our case, by Eq. 7. In Fig. 8 you can see the examples of the densities *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) and *p*ˆ*rd* (*ξ*|*q*ˇ(*i*, *j*)) coming out of the given demonstrations in Fig. 7 if and only if *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*) ∈ {*q*()} *Mq* <sup>=</sup><sup>1</sup> = {0, 10, 20, 30, 40}. We would like to emphasize that it is just a simple example and actually the cardinality of set {*q*()} *Mq* <sup>=</sup><sup>1</sup> = *f*(**A**, {N*d*(*i*, *j*)} *M*N*<sup>d</sup>* <sup>=</sup><sup>1</sup> ) is much larger then 5. Blue lines on Fig. 8 represent boundaries of the individual quantization steps.

The SODEM output can be obtained from the derived density by the integration

$$\text{PI}[\check{\eta}(i,j)] = \int\_{r\_d(i,j)-\frac{1}{2}}^{r\_d(i,j)+\frac{1}{2}} \hat{p}\_{r\_d}(\mathcal{J}'|\check{\eta}(i,j))d\xi'.\tag{23}$$

On the following Fig. 9a, 9c, 9e, 9g are introduced four examples of the SODEM transfer function constituting the derived composite CCD/CMOS noise model for the blurring channel **A**GBC,3/10 (Eq. 2) and **A**BOM,*π*/4 (Eq. 5). For the purpose of higher lucidity was chosen continuous plotting, although the transfer functions have a discrete domain of definition. The individual demonstrations of transfer functions afterwards construe with samples (realizations) of captured images on Fig. 9b, 9d, 9f, 9h.

## **7. Distributed iterative detection networks**

12 Will-be-set-by-IN-TECH

The procedure obtaining the metric SI[*q*ˇ(*i*, *j*)] from the derived noise model begins with the elimination Eq. 10 of the noise density *p*ˆ*w*(*ξ*) out of the likeli-hood function of a channel with

assuming perfect knowledge of the noise realization **W**. The result of this elimination is the density *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) = *p*ˆ*w*(*q*ˇ(*i*, *j*) − *ξ*). Along with the thermal noise, another source of noise is found in the chain. It is the quantization noise with the uniform PDF *pwQ* which is added to the signal in the A/D converter. It can be eliminated out of the density *p*ˆ*<sup>r</sup>* by the integrating

> FWC+ <sup>1</sup> 2 *tQ*(2*Nb* )−*�*

*�*→0<sup>+</sup>

*�*→0<sup>+</sup>

*δ*(*ξ* − ) lim

to emphasize that it is just a simple example and actually the cardinality of set {*q*()}

 *rd* (*i*,*j*)+ <sup>1</sup> 2

> *rd* (*i*,*j*)<sup>−</sup> <sup>1</sup> 2

On the following Fig. 9a, 9c, 9e, 9g are introduced four examples of the SODEM transfer function constituting the derived composite CCD/CMOS noise model for the blurring channel **A**GBC,3/10 (Eq. 2) and **A**BOM,*π*/4 (Eq. 5). For the purpose of higher lucidity was chosen continuous plotting, although the transfer functions have a discrete domain of definition. The individual demonstrations of transfer functions afterwards construe with

The SODEM output can be obtained from the derived density by the integration

PI[*q*ˇ(*i*, *j*)] =

samples (realizations) of captured images on Fig. 9b, 9d, 9f, 9h.

over individual quantization steps (by averaging with the density *pwQ* ) of the size Δ*<sup>Q</sup>* expressed in the number of electrons and defined, in our case, by Eq. 7. In Fig. 8 you can see the examples of the densities *p*ˆ*r*(*ξ*|*q*ˇ(*i*, *j*)) and *p*ˆ*rd* (*ξ*|*q*ˇ(*i*, *j*)) coming out of the given

 1 −

 FWC+ <sup>1</sup> 2

)) or *pr*|*q*,*w*(*ξ*|*q*ˇ(*i*, *<sup>j</sup>*), *<sup>ξ</sup>*�

*p*ˆ*r*(*ξ*�

 *tQ*(+1)−*� tQ*()−*�*

*Mq*

<sup>=</sup><sup>1</sup> ) is much larger then 5. Blue lines on Fig. 8 represent boundaries of the

*p*ˆ*rd* (*ξ*�



*p*ˆ*r*(*ξ*�

*p*ˆ � *w*(*ξ*� )*dξ*� 

)*pwR* (*ξ*�

, (20)

)*dξ*� (21)

) = *δ*(*ξ* − (*q*ˇ(*i*, *j*) + *ξ*�


. (23)

*Mq* <sup>=</sup><sup>1</sup> =

<sup>=</sup><sup>1</sup> = {0, 10, 20, 30, 40}. We would like

)),

− 1 2

*p*ˆ*wT* ( − *ξ*�

cut off PDF

based on the convolution

of densities *p*ˆ*wT* (*ξ*) and *pwT* (*ξ*).

additive noise *<sup>p</sup>***R**|**Q**,**W**(Ξ|**Q**<sup>ˇ</sup> , <sup>Ξ</sup>�

*f*(**A**, {N*d*(*i*, *j*)}

*M*N*<sup>d</sup>*

individual quantization steps.

*p*ˆ*w*(*ξ*) ≈ *p*ˆ

*p*ˆ � *<sup>w</sup>*(*ξ*) =

�

FWC ∑ =0

*<sup>p</sup>*ˆ*rd* (*ξ*|*q*ˇ(*i*, *<sup>j</sup>*)) = *<sup>δ</sup>*(*<sup>ξ</sup>* <sup>−</sup> <sup>2</sup>*Nb* ) lim

+ <sup>2</sup>*Nb* <sup>−</sup><sup>1</sup> ∑ =1

demonstrations in Fig. 7 if and only if *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*) ∈ {*q*()}

*<sup>w</sup>*(*ξ*) + *δ*(*ξ* − FWC)

*δ*(*ξ* − )

) = *<sup>δ</sup>*(<sup>Ξ</sup> <sup>−</sup> (**Q**<sup>ˇ</sup> <sup>+</sup> <sup>Ξ</sup>�

## **7.1 Distributed IDN marginalizing at the symbol block level**

Firstly we focus on the IDNs marginalizing at the symbol block level with horizontal and vertical state variable. These IDNs result from the EN [1] in Fig. 10a where each node (GPE), shown in Fig. 11b, creates a one functional block with the fixed system of inputs and outputs N*f*(*i*, *j*) = {R(*i*, *j*), C(*i*, *j*), *d*(*i* + *H***A**, *j* + *W***A**), R(*i*, *j* + 1), C(*i* + 1, *j*), *q*(*i*, *j*)}, where R(*i*, *j*) and C(*i*, *j*) present auxiliary state variables containing more symbols together. Shapes of these variables are established by the condition

$$\begin{split} \mathcal{N}\_d(i,j) &\subseteq \mathcal{R}(i,j) \cup \mathcal{C}(i,j) \cup d(i+H\_\mathbf{A}, j+W\_\mathbf{A}) \\ &\supset \mathcal{R}(i,j+1), \mathcal{C}(i+1,j). \end{split} \tag{24}$$

The optimal decomposition of the convolution region is N*d*(*i*, *j*) = R(*i*, *j*) ∪ C(*i*, *j*) ∪ *d*(*i* + *H***A**, *j* + *W***A**) and it is possible when *a*(*H***A**, *W***A**) �= 0.

In the case of a channel **A**GBC,*a*(0,0), with regard to the condition Eq. 24, the variables are R(*i*, *j*) = {*d*(*i*, *j*), *d*(*i*, *j* + 1), *d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 2, *j*), *d*(*i* + 2, *j* + 1)} and C(*i*, *j*) = {*d*(*i*, *j*), *d*(*i*, *j* + 1), *d*(*i*, *j* + 2), *d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 1, *j* + 2)}. Their shapes you can see in Fig. 12a. In a similar way we can get decomposition of the N*d*(*i*, *j*) for a channel **A**GBC,*a*(0,0). One of possible results is shown in Fig. 12b.

The IDN has same topology as the EN and each its node forms SISO module, illustrated in Fig. 11b, performing the combination

$$\begin{split} \text{S[\tilde{N}\_f(i,j)]} = \text{SI}[\dot{\mathcal{R}}(i,j)] \odot \text{SI}[\dot{\mathcal{L}}(i,j)] \odot \text{SI}[\dot{d}(i+H\_{\mathbf{A}\prime}j+W\_{\mathbf{A}})] \odot \text{SI}[\dot{\mathcal{R}}(i,j+1)] \\ & \odot \text{SI}[\dot{\mathcal{L}}(i+1,j)] \odot \text{SI}[\dot{\eta}(i,j)] \quad \text{(25)} \end{split} \tag{25}$$

with marginalizations

$$\mathbf{SO}[\check{\mathcal{R}}(i, j + j')] = \left( \underset{\check{\mathcal{N}}\_{\hat{f}}(i, j); \check{\mathcal{R}}(i, j + j')}{\oplus} \mathbb{S}[\check{\mathcal{N}}\_{f(j)}(i, j)] \right) \odot^{-1} \mathbb{S}[\check{\mathcal{R}}(i, j + j')] \text{ for } j' \in \{0, 1\}, \tag{26}$$

$$\mathbf{SO}[\mathcal{C}(\mathbf{i}+\mathbf{i}',j)] = \left( \underset{\mathcal{N}\_f(\mathbf{i},j): \mathcal{X}(\mathbf{i}+\mathbf{i}',j)}{\oplus} \mathbf{S}[\mathcal{N}\_{f(\cdot)}(\mathbf{i},j)] \right) \odot^{-1} \mathbf{SI}[\mathcal{C}(\mathbf{i}+\mathbf{i}',j)] \text{ for } \mathbf{i}' \in \{0,1\}, \tag{27}$$

$$\mathbf{SO}[\check{d}(i+H\_{\mathbf{A}},j+W\_{\mathbf{A}})] = \begin{pmatrix} \oplus & \mathbf{S}[\check{N}\_{f(\cdot)}(i,j)] \\ \boldsymbol{\mathcal{N}}\_{f(\cdot),\check{\mathbf{u}}(i+H\_{\mathbf{A}},j+W\_{\mathbf{A}})} & \mathbf{S}[\check{N}\_{f(\cdot)}(i,j)] \end{pmatrix} \oplus \prescript{-1}{}{\mathbf{SI}[\check{d}(i+H\_{\mathbf{A}},j+W\_{\mathbf{A}})]} \tag{28}$$

and

$$\mathbf{SO}[\mathfrak{j}(i,j)] = \left( \underset{\mathcal{N}\_f(i,j):\mathfrak{j}(i,j)}{\oplus} \mathbf{S}[\check{\mathcal{N}}\_{f(\cdot)}(i,j)] \right) \odot^{-1} \mathbf{SI}[\mathfrak{j}(i,j)].\tag{29}$$

One iteration of such IDN rests in the serial activation of each SISO module from the upper left corner to the lower right corner in the IDN as is marked by the light blue curve in Fig. 10a. After finishing all iterations, of the chosen count *NI*, the IDN makes hard decisions ˆ*d*(*i*, *j*) per Eq. 13.

(a) The PDF for the blurring channel **A**GBC,3/10: *C* = 612, *μ<sup>T</sup>* = 60, *μ<sup>R</sup>* = 40, *σ<sup>R</sup>* = 35.

(e) The PDF for the blurring channel **A**BOM,*π*/4: *C* = 612, *μ<sup>T</sup>* = 60, *μ<sup>R</sup>* = 40, *σ<sup>R</sup>* = 35.

(g) (h)

SI[ ˇ

IDN SO[ ˇ

> SO[ ˇ

block level.

SI[ ˇ

C(i + 1, j)] C(i + 1, j)]

i + 2

R(i, j)

d(i + HA, j + WA)

**A**GBC,*a*(0,0) .

The quantities

i + 1

i

C(i + 1, j)]

C(i + 1, j)]

EN

C(i + 1, j)

SI[Rˇ(i, j + 1)] SO[Rˇ(i, j + 1)]

2D Iterative Detection Network Based

j j + 1

(a) The EN and IDN marginalizing at the symbol

C(i, j)

f( )

C(i + 1, j)

(a) The EN cell.

q(i, j) R(i, j + 1)

SO[Rˇ(i, j + 1)] SI[Rˇ(i, j + 1)]

Image Restoration: Principles, Applications and Performance Analysis

schedule Activation

Fig. 10. The topology of the EN and IDN with denotation of the IDN activation schedule.

Fig. 11. The cells in the node (*i*, *j*) of the EN and IDN marginalizing at the symbol block level.

R(i, j)

Fig. 12. The primary layout of auxiliary state variables R(*i*, *j*) and C(*i*, *j*) for the ISI channel

<sup>ˇ</sup>*d*(*i*,*j*)∈R<sup>ˇ</sup> (*i*,*j*)

SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>*)] = �**<sup>C</sup>**

SO[dˇ(i + HA, j + WA)] SI[dˇ(i + HA, j + WA)]

> C(i, j)

d(i+2,j+2)

AGBC,a(0,0)

SI[Rˇ(i, j)] SO[Rˇ(i, j)]

i + 2

block level with bias state variables.

i + 1

i

j j + 1

(b) The EN and IDN marginalizing at the symbol

SO[ ˇ

SI[ ˇ

SI[ ˇ*d*(*i*, *j*)] (30)

SO[ ˇ

(b) The IDN cell (SISO module).

C(i + 1, j)] C(i + 1, j)]

SI[ ˇ

f−1( )

C(i, j)] C(i, j)] j + 2

SO[Rˇ(i, j + 1)] SI[Rˇ(i, j + 1)]

SI[ˇq(i, j)] SO[ˇq(i, j)] schedule Activation 329

j + 2 R(i, j + 1)

EN

IDN

(b) The PDF for the blurring channel **A**GBC,3/10: *C* = 542, *μ<sup>T</sup>* = 150, *μ<sup>R</sup>* = 50, *σ<sup>R</sup>* = 25.

(c) (d)

(f) The PDF for the blurring channel **A**BOM,*π*/4: *C* = 542, *μ<sup>T</sup>* = 150, *μ<sup>R</sup>* = 50, *σ<sup>R</sup>* = 25.

Fig. 9. The examples of SODEM transfer (likeli-hood) functions with the corresponding realization **<sup>R</sup>***<sup>d</sup>* of the A/D converter output: *Nb* <sup>=</sup> 8, <sup>Δ</sup>*<sup>Q</sup>* <sup>=</sup> 3. All alphabets {*q*()} *Mq* <sup>=</sup><sup>1</sup> are sorted according to the size (*q*(1) is the least element and *q*(*Mq* ) is the greatest element).

14 Will-be-set-by-IN-TECH

(c) (d)

(g) (h)

Fig. 9. The examples of SODEM transfer (likeli-hood) functions with the corresponding

sorted according to the size (*q*(1) is the least element and *q*(*Mq* ) is the greatest element).

realization **<sup>R</sup>***<sup>d</sup>* of the A/D converter output: *Nb* <sup>=</sup> 8, <sup>Δ</sup>*<sup>Q</sup>* <sup>=</sup> 3. All alphabets {*q*()}

(b) The PDF for the blurring channel **A**GBC,3/10: *C* = 542, *μ<sup>T</sup>* = 150, *μ<sup>R</sup>* =

(f) The PDF for the blurring channel **A**BOM,*π*/4: *C* = 542, *μ<sup>T</sup>* = 150, *μ<sup>R</sup>* =

> *Mq* <sup>=</sup><sup>1</sup> are

50, *σ<sup>R</sup>* = 25.

50, *σ<sup>R</sup>* = 25.

(a) The PDF for the blurring channel **A**GBC,3/10: *C* = 612, *μ<sup>T</sup>* = 60, *μ<sup>R</sup>* = 40,

(e) The PDF for the blurring channel **A**BOM,*π*/4: *C* = 612, *μ<sup>T</sup>* = 60, *μ<sup>R</sup>* = 40,

*σ<sup>R</sup>* = 35.

*σ<sup>R</sup>* = 35.

(a) The EN and IDN marginalizing at the symbol block level.

(b) The EN and IDN marginalizing at the symbol block level with bias state variables.

Fig. 10. The topology of the EN and IDN with denotation of the IDN activation schedule.

Fig. 11. The cells in the node (*i*, *j*) of the EN and IDN marginalizing at the symbol block level.

Fig. 12. The primary layout of auxiliary state variables R(*i*, *j*) and C(*i*, *j*) for the ISI channel **A**GBC,*a*(0,0) .

The quantities

$$\text{SI}[\check{\mathcal{R}}(i,j)] = \underset{\check{d}(i,j) \in \check{\mathcal{R}}(i,j)}{\ominus} \text{SI}[\check{d}(i,j)] \tag{30}$$

B(i + 1, j + 1)

AGBC,a(0,0)

SO[dˇ(i + HA, j + WA)] SI[dˇ(i + HA, j + WA)]

Fig. 13. The cells in the node (*i*, *j*) of the EN and IDN marginalizing at the symbol block level

Fig. 14. The two possible layouts of auxiliary state variables R(*i*, *j*), C(*i*, *j*) and B(*i*, *j*) for the

centered on the central (dominant) coefficient *a*(0, 0). Each node (GPE) in this EN contains two functional blocks (broadcaster and combining element) with the generally variable system of inputs and outputs. In the our current example, each node has the framework shown in Fig. 15b and it is connected to the eight nearest neighbors. All these cells together creating the convolutional region N*d*(*i* − 1, *j* − 1). An input and outputs of each broadcaster make the set

The outputs of nine neighboring broadcasters create inputs for one combining element. These

*a*(�1/2 + sin(*πk*/4)� , �1/2 − cos(*πk*/4)�)*c*(*i* + �1/2 + sin(*πk*/4)� ,

N*f*(*i*, *<sup>j</sup>*) = {*c*(*<sup>i</sup>* + �1/2 + sin(*πk*/4)� , *<sup>j</sup>* + �1/2 − cos(*πk*/4)� , *<sup>k</sup>* + <sup>1</sup>), *<sup>c</sup>*(*i*, *<sup>j</sup>*, 9), *<sup>q</sup>*(*i*, *<sup>j</sup>*)}0≤*k*≤7. (36)

N*B*(*i*, *<sup>j</sup>*) = {*d*(*i*, *<sup>j</sup>*), *<sup>c</sup>*(*i*, *<sup>j</sup>*, *<sup>k</sup>*) = *<sup>d</sup>*(*i*, *<sup>j</sup>*)}1≤*k*≤9. (34)

*j* + �1/2 − cos(*πk*/4)� , *k* + 1) + *Ca*(0, 0)*c*(*i*, *j*, 9) (35)

SI[Rˇ(i, j)] SO[Rˇ(i, j)] SI[ˇq(i, j)] SO[ˇq(i, j)] SO[Rˇ(i, j + 1)] SI[Rˇ(i, j + 1)]

SI[ ˇ

SO[ ˇB(i + 1, j + 1)]

B(i, j)

R(i, j)

C(i, j)

AGBC,a(0,0)

331

d(i+2,j+2)

C(i + 1, j + 1)]

SO[ ˇ

SI[ ˇ

SO[ ˇ

(b) The IDN cell (SISO module).

C(i + 1, j)] C(i + 1, j)]

SI[ ˇ

f−1( )

SO[ ˇB(i, j)]

SI[ ˇB(i, j)] C(i, j)] C(i, j)]

q(i, j) R(i, j + 1)

Image Restoration: Principles, Applications and Performance Analysis

B(i, j)

2D Iterative Detection Network Based

C(i, j)

f( )

C(i + 1, j)

(a) The EN cell.

R(i, j)

B(i, j)

C(i, j)

d(i+2,j+2)

R(i, j)

with bias state variables.

ISI channel **A**GBC,*a*(0,0).

inputs with the output

0≤*k*≤7

*q*(*i*, *j*) = *C* ∑

make the set

d(i + HA, j + WA)

and

$$\text{SI}[\check{\mathcal{C}}(i,j)] = \underset{\check{d}(i,j) \in \mathcal{C}(i,j)}{\text{GE}} \text{SI}[\check{d}(i,j)],\tag{31}$$

before the first iteration of the IDN, are adjusted via combination of the a priori measures SI[ <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)] corresponding to testing estimators <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*), that form block estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*) and <sup>C</sup>ˇ(*i*, *<sup>j</sup>*).

### **7.2 Distributed IDN marginalizing at the symbol block level with bias state variables**

The fundamental disadvantage of IDNs described in the previous paragraph rests in cardinality of state variables. For example decomposition in Fig. 12a has cardinality for dichromatic pictures 26 per variable and therefore both bidirectional concatenations of all SISO modules have to transfer 2 <sup>×</sup> 26 <sup>=</sup> 128 soft measures in the each direction. However, this is not only one disadvantage. The another rests in potential absence of the optimal decomposition for irregular kernels which can extremely gross up computational exigencies of the algorithm.

Impacts of both mentioned disadvantages can be reduced by the suggestion of EN structure in Fig. 10b that is expanded by the bias state variable B(*i*, *j*). All inputs and outputs of each node, shown in Fig. 13a, forms the set N*f*(*i*, *j*) = {R(*i*, *j*), C(*i*, *j*), B(*i*, *j*), *d*(*i* + *H***A**, *j* + *W***A**), R(*i*, *j* + 1), C(*i* + 1, *j*), B(*i*, *j*), *q*(*i*, *j*)}, where state variables are shaped by more liberal condition

$$\begin{split} \mathcal{N}\_d(i,j) &\subseteq \mathcal{R}(i,j) \cup \mathcal{C}(i,j) \cup \mathcal{B}(i,j) \cup d(i+H\_{\mathbf{A}\prime}j+W\_{\mathbf{A}})\\ &\supset \mathcal{R}(i,j+1), \mathcal{C}(i+1,j), \mathcal{B}(i+1,j+1). \end{split} \tag{32}$$

In Fig. 12 are expressed three examples of decompositions for two different cores. Concretely, in the case of 1st version of kernel **<sup>A</sup>**GBC,*a*(0,0) partitioning, where state variables are <sup>R</sup>(*i*, *<sup>j</sup>*) = {*d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 2, *j*), *d*(*i* + 2, *j* + 1)}, C(*i*, *j*) = {*d*(*i*, *j* + 1), *d*(*i*, *j* + 2), *d*(*i* + 1, *j* + 1), *d*(*i* + 1, *j* + 2)} and B(*i*, *j*) = {*d*(*i*, *j*), *d*(*i* + 1, *j*), *d*(*i*, *j* + 1), *d*(*i* + 1, *j* + 1)}, is necessary to transfer and store only 3 <sup>×</sup> 24 <sup>=</sup> 48 soft measures in the each direction. In the 2nd version for same kernel this number is smaller, namely 2 <sup>×</sup> 23 <sup>+</sup> 22 <sup>=</sup> 20.

The IDN topology copies the EN topology as in the previous case and activation schedule is identical too. A SISO module used in this IDN is approached in Fig. 13b.

#### **7.3 Distributed IDN marginalizing at the symbol level**

The topology of the IDN marginalizing at the symbol level is dependent on the shape of the response **A**. It is the basic difference from the IDN marginalizing at the symbol block level, whose topology is invariable. There are two possible kinds of the topology. Either the topology centered on the response central coefficient or the shifted topology, that is fixed on the coefficient other than the central coefficient.

Firstly, we focus on the IDN example with the centered topology for the ISI channel **A**GBC,*a*(0,0) with order *L* = *H***<sup>A</sup>** + 1 = *W***<sup>A</sup>** + 1 = 3. Let us suppose the source EN in Fig. 15a emulating the convolution

$$\begin{split} q(i,j) &= \mathbb{C}f(\mathbf{A}\_{\text{GBC},a(0,0)'}N\_d(i-1,j-1) \subset \mathbf{D}) \\ &= \mathbb{C} \sum\_{|i'|, |j'| \le 1} a(i',j')d(i+i',j+j') \end{split} \tag{33}$$

16 Will-be-set-by-IN-TECH

<sup>ˇ</sup>*d*(*i*,*j*)∈Cˇ(*i*,*j*)

before the first iteration of the IDN, are adjusted via combination of the a priori measures SI[ <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)] corresponding to testing estimators <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*), that form block estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*) and

The fundamental disadvantage of IDNs described in the previous paragraph rests in cardinality of state variables. For example decomposition in Fig. 12a has cardinality for dichromatic pictures 26 per variable and therefore both bidirectional concatenations of all SISO modules have to transfer 2 <sup>×</sup> 26 <sup>=</sup> 128 soft measures in the each direction. However, this is not only one disadvantage. The another rests in potential absence of the optimal decomposition for irregular kernels which can extremely gross up computational exigencies of the algorithm. Impacts of both mentioned disadvantages can be reduced by the suggestion of EN structure in Fig. 10b that is expanded by the bias state variable B(*i*, *j*). All inputs and outputs of each node, shown in Fig. 13a, forms the set N*f*(*i*, *j*) = {R(*i*, *j*), C(*i*, *j*), B(*i*, *j*), *d*(*i* + *H***A**, *j* + *W***A**), R(*i*, *j* + 1), C(*i* + 1, *j*), B(*i*, *j*), *q*(*i*, *j*)}, where state variables are shaped by more liberal condition

N*d*(*i*, *j*) R(*i*, *j*) ∪ C(*i*, *j*) ∪ B(*i*, *j*) ∪ *d*(*i* + *H***A**, *j* + *W***A**)

In Fig. 12 are expressed three examples of decompositions for two different cores. Concretely, in the case of 1st version of kernel **<sup>A</sup>**GBC,*a*(0,0) partitioning, where state variables are <sup>R</sup>(*i*, *<sup>j</sup>*) = {*d*(*i* + 1, *j*), *d*(*i* + 1, *j* + 1), *d*(*i* + 2, *j*), *d*(*i* + 2, *j* + 1)}, C(*i*, *j*) = {*d*(*i*, *j* + 1), *d*(*i*, *j* + 2), *d*(*i* + 1, *j* + 1), *d*(*i* + 1, *j* + 2)} and B(*i*, *j*) = {*d*(*i*, *j*), *d*(*i* + 1, *j*), *d*(*i*, *j* + 1), *d*(*i* + 1, *j* + 1)}, is necessary to transfer and store only 3 <sup>×</sup> 24 <sup>=</sup> 48 soft measures in the each direction. In the 2nd version for

The IDN topology copies the EN topology as in the previous case and activation schedule is

The topology of the IDN marginalizing at the symbol level is dependent on the shape of the response **A**. It is the basic difference from the IDN marginalizing at the symbol block level, whose topology is invariable. There are two possible kinds of the topology. Either the topology centered on the response central coefficient or the shifted topology, that is fixed on

Firstly, we focus on the IDN example with the centered topology for the ISI channel **A**GBC,*a*(0,0) with order *L* = *H***<sup>A</sup>** + 1 = *W***<sup>A</sup>** + 1 = 3. Let us suppose the source EN in Fig. 15a emulating

*q*(*i*, *j*) = *C f*(**A**GBC,*a*(0,0), N*d*(*i* − 1, *j* − 1) ⊂ **D**)

*a*(*i* � , *j* � )*d*(*i* + *i* � , *j* + *j* �

same kernel this number is smaller, namely 2 <sup>×</sup> 23 <sup>+</sup> 22 <sup>=</sup> 20.

**7.3 Distributed IDN marginalizing at the symbol level**

the coefficient other than the central coefficient.

the convolution

identical too. A SISO module used in this IDN is approached in Fig. 13b.

= *C* ∑ |*i*�|,|*j*�|≤1

⊃ R(*i*, *j* + 1), C(*i* + 1, *j*), B(*i* + 1, *j* + 1). (32)

SI[ ˇ*d*(*i*, *j*)], (31)

) (33)

SI[Cˇ(*i*, *<sup>j</sup>*)] = **<sup>C</sup>**

**7.2 Distributed IDN marginalizing at the symbol block level with bias state variables**

and

<sup>C</sup>ˇ(*i*, *<sup>j</sup>*).

Fig. 13. The cells in the node (*i*, *j*) of the EN and IDN marginalizing at the symbol block level with bias state variables.

Fig. 14. The two possible layouts of auxiliary state variables R(*i*, *j*), C(*i*, *j*) and B(*i*, *j*) for the ISI channel **A**GBC,*a*(0,0).

centered on the central (dominant) coefficient *a*(0, 0). Each node (GPE) in this EN contains two functional blocks (broadcaster and combining element) with the generally variable system of inputs and outputs. In the our current example, each node has the framework shown in Fig. 15b and it is connected to the eight nearest neighbors. All these cells together creating the convolutional region N*d*(*i* − 1, *j* − 1). An input and outputs of each broadcaster make the set

$$\mathcal{N}\_B(i,j) = \{d(i,j), c(i,j,k) = d(i,j)\}\_{1 \le k \le 9}.\tag{34}$$

The outputs of nine neighboring broadcasters create inputs for one combining element. These inputs with the output

$$q(i,j) = \mathbb{C} \sum\_{0 \le k \le 7} a(\lfloor 1/2 + \sin(\pi k/4) \rfloor, \lfloor 1/2 - \cos(\pi k/4) \rfloor) c(i + \lfloor 1/2 + \sin(\pi k/4) \rfloor)$$

$$j + \lfloor 1/2 - \cos(\pi k/4) \rfloor, k + 1 \rangle + \mathbb{C}a(0,0)c(i,j,9) \quad \text{(35)}$$

make the set

$$\mathcal{N}\_f(\mathbf{i}, \mathbf{j}) = \{ \mathbf{c}(\mathbf{i} + \lfloor 1/2 + \sin(\pi k/4) \rfloor, j + \lfloor 1/2 - \cos(\pi k/4) \rfloor, k + 1), \mathbf{c}(\mathbf{i}, \mathbf{j}, \mathbf{9}), \mathbf{q}(\mathbf{i}, \mathbf{j}) \}\_{0 \le k \le 7}. \tag{36}$$

i − 1

j − 1 j j + 1

Image Restoration: Principles, Applications and Performance Analysis

2D Iterative Detection Network Based

/

d(i, j)

c(i, j, 4)

c(i, j − 1, 1)

c(i − 1, j − 1, 8)

c(i, j, 6)

c(i, j, 9)

c(i, j, 5)

c(i + 1, j

SI[ˇc(i, j + 1, 5)] SO[ˇc(i, j + 1, 5)]

SI[ˇq(i, j)] SO[ˇq(i, j)]

SI[ˇc(i, j, 9)] SO[ˇc(i, j, 9)]

SO[ˇc(i, j, 1)]

SO[ˇc(i, j, 2)] SI[ˇc(i, j, 2)]

SO[ˇc(i + 1, j + 1, 4)] SI[ˇc(i + 1, j + 1, 4)]

SO[ˇc(i, j, 8)]

SI[ˇc(i + 1, j, 3)]

f−1( )

SO[ˇc(i + 1, j, 3)]

(c) The IDN cell (SISO module) in the node (*i*, *j*).

If the response **A** has a dominant coefficient and the IDN topology is centered on this coefficient, then such IDN can be further numerically simplified by the approximation SO[ <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)] <sup>≈</sup> SO[*c*ˇ(*i*, *<sup>j</sup>*, max(*k*))] (Chugg et al., 2001). It reduces a number of marginalizations

Fig. 15. The EN and IDN marginalizing at the symbol level with the centered topology for

SI[ˇc(i, j, 7)]

SO[ˇc(i, j, 7)] SI[ˇc(i, j, 8)]

− 1, 2) c(i, j, 3)

B

c(i − 1, j, 7)

f( )

c(i, j, 7) c(i + 1, j, 3)

(b) The EN cell in the node (*i*, *j*). The connection of the broadcaster () and combining element ().

c(i, j, 1)

c(i, j + 1, 5)

c(i, j, 2) c(i − 1, j + 1, 6)

q(i, j)

c(i + 1, j + 1, 4)

333

c(i, j, 8)

(a) The topology of the EN and IDN with denotation of the IDN activation schedule.

> SO[ˇc(i, j, 5)] SI[ˇc(i, j, 5)]

SI[dˇ(i, j)] SO[dˇ(i, j)]

SI[ˇc(i, j, 9)] SO[ˇc(i, j, 9)] SO[ˇc(i, j, 6)] SI[ˇc(i, j, 6)]

SI[ˇc(i, j − 1, 1)] SO[ˇc(i, j − 1, 1)] SO[ˇc(i, j, 4)]

SI[ˇc(i, j, 3)] SO[ˇc(i, j, 3)] SO[ˇc(i − 1, j, 7)] SI[ˇc(i − 1, j + 1, 6)]

B<sup>−</sup> SI[ˇc(i, j, 1)] <sup>1</sup>

SO[ˇc(i − 1, j + 1, 6)]

SI[ˇc(i − 1, j, 7)]

SI[ˇc(i, j, 4)]

SO[ˇc(i + 1, j − 1, 2)]

**7.4 Simplified distributed IDN marginalizing at the symbol level**

SI[ˇc(i + 1, j − 1, 2)]

SI[ˇc(i − 1, j − 1, 8)] SO[ˇc(i − 1, j − 1, 8)]

i

i + 1

the ISI channel**A**GBC,*a*(0,0).

As is known, the IDN has topology corresponding with the EN topology. One of the composite SISO modules creating the present IDN is illustrated in Fig. 15c, where the SISO module *B*−<sup>1</sup> performing the combination

$$\text{SO}[\check{d}(i,j)] = \underset{k}{\ominus} \text{SI}[\check{\varepsilon}(i,j,k) = \check{d}(i,j)] \tag{37}$$

with the marginalization

$$\text{SO}[\check{\varepsilon}(i,j,k)] = \left(\underset{k' \neq k}{\odot} \text{SI}[\check{\varepsilon}(i,j,k') = \check{\varepsilon}(i,j,k)]\right) \odot \text{SI}[\check{d}(i,j) = \check{\varepsilon}(i,j,k)] \tag{38}$$

and the SISO module *f* <sup>−</sup>1() performing combination

$$\mathbf{S}[\check{\mathcal{N}}\_f(i,j)] = \left( \bigoplus\_{\mathbb{X}(i',j',k) \in \check{\mathcal{N}}\_f(i,j)} \mathbf{SI}[\check{\mathfrak{X}}(i',j',k)] \right) \odot \mathbf{SI}[\check{\mathfrak{q}}(i,j)] \tag{39}$$

with the marginalizations

$$\mathbf{SO}[\mathfrak{i}(i',j',k)] = \begin{pmatrix} \oplus & \mathbf{S}[\check{\mathcal{N}}\_f(i,j)] \\ \mathcal{N}\_f(i,j)\check{\mathfrak{i}}(i',j',k) \end{pmatrix} \odot^{-1} \mathbf{S}[\check{\mathfrak{i}}(i',j',k)] \tag{40}$$

and SO[*q*ˇ(*i*, *j*)] acquisitionable from Eq. 29.

A soft measure processing in the composite SISO modules is mirrored to the signal processing in the EN nodes (from *d*(*i*, *j*) to *q*(*i*, *j*)) and begins with activation all SISO modules *f* <sup>−</sup>1(). Consequently, the activations of the SISO modules *B*−<sup>1</sup> take place. Both activations, intimated by activation loops in Fig. 15a, form one iteration of the entire IDN. After finishing of all required iterations, the IDN makes hard decisions ˆ*d*(*i*, *j*) in accordance with Eq. 13.

The quantities SI[*c*ˇ(*i*, *j*, *k*)], before the first iteration of the IDN, are adjusted pursuant to the a priori measures SI[ ˇ*d*(*i*, *j*)], because the *c*(*i*, *j*, *k*) constitute copies of the *d*(*i*, *j*). Henceforth, we come up to the IDN example again for the **A**GBC,3/10 , but now it will be the IDN with the shifted topology. In this case, the source EN in Fig. 16a implements the convolution Eq. 1 centered on the upper left coefficient. One of the EN cells is shown in Fig. 16b. In comparison with the previous example, there is a difference only in the combining element. It has the I/O set

$$\mathcal{N}\_f(i,j) = \{ \mathfrak{c}(i-\lceil k/3 \rceil + 3, j + 3, \lceil k/3 \rceil - k, k), q(i,j) \}\_{1 \le k \le 9} \tag{41}$$

where

$$q(i,j) = \mathbb{C} \sum\_{1 \le k \le 9} a(i - \lceil k/\mathfrak{z} \rceil + 3, j + 3\lceil k/\mathfrak{z} \rceil - k)c(i - \lceil k/\mathfrak{z} \rceil + 3, j + 3\lceil k/\mathfrak{z} \rceil - k, k). \tag{42}$$

The soft inversion of the mentioned EN cell is represented by Fig. 16c and along with other cooperative SISO modules forming the resulting IDN. A similar IDN can be synthesized by the analogical way for any other ISI channel.

18 Will-be-set-by-IN-TECH

As is known, the IDN has topology corresponding with the EN topology. One of the composite SISO modules creating the present IDN is illustrated in Fig. 15c, where the SISO module *B*−<sup>1</sup>

) = *c*ˇ(*i*, *j*, *k*)]

SI[*c*ˇ(*i* � , *j* � , *k*)] 

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*(*i*, *<sup>j</sup>*)]

N*f*(*i*, *<sup>j</sup>*) = {*c*(*<sup>i</sup>* − �*k*/3� + 3, *<sup>j</sup>* + <sup>3</sup> �*k*/3� − *<sup>k</sup>*, *<sup>k</sup>*), *<sup>q</sup>*(*i*, *<sup>j</sup>*)}1≤*k*≤<sup>9</sup> (41)

*a*(*i* − �*k*/3� + 3, *j* + 3 �*k*/3� − *k*)*c*(*i* − �*k*/3� + 3, *j* + 3 �*k*/3� − *k*, *k*). (42)

<sup>C</sup> <sup>−</sup>1SI[*c*ˇ(*i*

� , *j* �

SI[*c*ˇ(*i*, *j*, *k*) = ˇ*d*(*i*, *j*)] (37)

<sup>C</sup> SI[ ˇ*d*(*i*, *j*) = *c*ˇ(*i*, *j*, *k*)] (38)

<sup>C</sup> SI[*q*ˇ(*i*, *j*)] (39)

, *k*)] (40)

*k*

SO[ ˇ*d*(*i*, *j*)] = **<sup>C</sup>**

SI[*c*ˇ(*i*, *j*, *k*�

**<sup>C</sup>**

M <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*):*c*ˇ(*i*�

required iterations, the IDN makes hard decisions ˆ*d*(*i*, *j*) in accordance with Eq. 13.

,*j*� ,*k*)

A soft measure processing in the composite SISO modules is mirrored to the signal processing in the EN nodes (from *d*(*i*, *j*) to *q*(*i*, *j*)) and begins with activation all SISO modules *f* <sup>−</sup>1(). Consequently, the activations of the SISO modules *B*−<sup>1</sup> take place. Both activations, intimated by activation loops in Fig. 15a, form one iteration of the entire IDN. After finishing of all

The quantities SI[*c*ˇ(*i*, *j*, *k*)], before the first iteration of the IDN, are adjusted pursuant to the a priori measures SI[ ˇ*d*(*i*, *j*)], because the *c*(*i*, *j*, *k*) constitute copies of the *d*(*i*, *j*). Henceforth, we come up to the IDN example again for the **A**GBC,3/10 , but now it will be the IDN with the shifted topology. In this case, the source EN in Fig. 16a implements the convolution Eq. 1 centered on the upper left coefficient. One of the EN cells is shown in Fig. 16b. In comparison with the previous example, there is a difference only in the combining element. It has the I/O

The soft inversion of the mentioned EN cell is represented by Fig. 16c and along with other cooperative SISO modules forming the resulting IDN. A similar IDN can be synthesized by

,*k*)∈N<sup>ˇ</sup> *<sup>f</sup>*(*i*,*j*)

*c*ˇ(*i*� ,*j*�

performing the combination

with the marginalization

with the marginalizations

set

where

*q*(*i*, *j*) = *C* ∑

1≤*k*≤9

the analogical way for any other ISI channel.

SO[*c*ˇ(*i*, *j*, *k*)] =

SO[*c*ˇ(*i* � , *j* � , *k*)] =

and SO[*q*ˇ(*i*, *j*)] acquisitionable from Eq. 29.

 **<sup>C</sup>** *k*��=*k*

and the SISO module *f* <sup>−</sup>1() performing combination

<sup>S</sup>[N<sup>ˇ</sup> *<sup>f</sup>*(*i*, *<sup>j</sup>*)] =

(a) The topology of the EN and IDN with denotation of the IDN activation schedule.

(b) The EN cell in the node (*i*, *j*). The connection of the broadcaster () and combining element ().

/

(c) The IDN cell (SISO module) in the node (*i*, *j*).

## **7.4 Simplified distributed IDN marginalizing at the symbol level**

If the response **A** has a dominant coefficient and the IDN topology is centered on this coefficient, then such IDN can be further numerically simplified by the approximation SO[ <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)] <sup>≈</sup> SO[*c*ˇ(*i*, *<sup>j</sup>*, max(*k*))] (Chugg et al., 2001). It reduces a number of marginalizations

SO[ˇq(i, j)] SI[ˇq(i, j)]

SO[ˇc(i, j, 2)]

SO[ˇc(i, j, 9)]

SO[ˇc(i, j, 1)]

SI[ˇc(i, j + 1, 5)]

SI[ˇc(i + 1, j + 1, 4)]

SI[ˇc(i − 1, j + 1, 6)]

SO[ˇc(i, j, 8)]

SO[dˇ(i, j)] SI[dˇ(i, j)]

Image Restoration: Principles, Applications and Performance Analysis

SI[ˇc(i − 1, j − 1, 8)]

2D Iterative Detection Network Based

SO[ˇc(i, j, 4)]

SI[ˇc(i + 1, j − 1, 2)] SO[ˇc(i, j, 3)]

B

SI[ˇc(i − 1, j, 7)]

f−1( )

SI[ˇc(i + 1, j, 3)]

SO[ˇc(i, j, 7)]

Fig. 17. The cell in the node (*i*, *j*) of the simplified IDN marginalizing at the symbol level

Finally, we focus on the Layered IDN for ISI channels with a decomposition-able impulse response to the horizontal **h** = [ *h*� *h h*� ] and vertical **g** = [ *g*� *g g*� ] direction. For example, the GBC has this property and therefore we assume, without detriment to generality, the GBC

*h*�

*h*�

with size *L* × *L* = 3 × 3 that will be used for exemplary construction of the layered EN, shown

The signal processing (two-step convolution) starts on the EN bottom layer *c*(*i*, *j*) =

*c*(*i*, *j*)). Each row-wise or column-wise concatenated node (GPE) creates a one functional block with the fixed system of inputs and outputs N*fR* (*i*, *j*) = {R(*i*, *j*), *d*(*i*, *j*), R(*i*, *j* +1), *c*(*i*, *j*)} or N*fC* (*i*, *j*) = {C(*i*, *j*), *c*(*i*, *j*), C(*i*, *j* + 1), *q*(*i*, *j*)}, where R(*i*, *j*) = {*d*(*i*, *j* − 2), *d*(*i*, *j* − 1)} and C(*i*, *j*) = {*c*(*i* − 2, *j*), *c*(*i* − 1, *j*)} present auxiliary state variables containing two symbols together, as in the case of the distributed IDN marginalizing at the symbol block level.

This specific case of the 2D ISI channel makes 2D detection, through the chosen EN topology, decomposition-able into double 1D detection, when in separate rows and columns can be used the well known Fixed interval forward-backward algorithm (FI FBA) Chugg et al. (2001). Let us elucidate the FI FBA principle on the IDN top layer. Its current recursion, in the node

*g*� *hg*� *h*�

*g hg h*�

*g*� *hg*� *h*�

*g*�

⎤

⎦ (43)

*d*(*i*, *j*) and continues on

335

*c*(*i* − 2, *j*) + *gc*(*i* − 1, *j*) +

*g*

*g*�

*d*(*i*, *j* − 2) + *hd*(*i*, *j* − 1) + *h*�

**7.5 Layered IDN for an ISI channel with a decomposition-able impulse response**

**A**GBC,*gh* = **hg***<sup>T</sup>*

in Fig. 18a, and mutually corresponding IDN, shown in Fig. 18b.

the EN top layer *q*˜(*i*, *j*) = *fC*(**g**, {*c*(*i* − 2, *j*), *c*(*i* − 1, *j*), *c*(*i*, *j*)}) = *g*�

*fR*(**h**, {*d*(*i*, *j* − 2), *d*(*i*, *j* − 1), *d*(*i*, *j*)}) = *h*�

*g*�

= ⎡ ⎣ *h*�

SO[ˇc(i, j, 5)]

SO[ˇc(i, j, 6)]

with centered topology for the ISI channel **A**GBC,*a*(0,0).

SI[ˇc(i, j, 9)]

SI[ˇc(i, j − 1, 1)]

(c) The IDN cell in the node (*i*, *j*).

Fig. 16. The EN and IDN marginalizing at the symbol level with the shifted topology for ISI channel **A**GBC,*a*(0,0).

in the SISO modules *f* <sup>−</sup>1() only to the variable *c*ˇ(*i*, *j*, max(*k*)) and enables removing of all SISO modules *B*−1.

Let us approach the principle of this approximation at the first IDN example in the paragraph 7.3. There had **A** the dominant coefficient *a*(0, 0). Therefore, the SO[ ˇ*d*(*i*, *j*)] can be approximated by the SO[*c*ˇ(*i*, *j*, 9)] and inputs {SI[*c*ˇ(*i* + �1/2 + sin(*πk*/4)� , *j* + �1/2 − cos(*πk*/4)� , *<sup>k</sup>* + <sup>1</sup>)]}0≤*k*≤<sup>7</sup> of associated SISO modules will be directly equal to SO[*c*ˇ(*i*, *j*, 9)], as is shown in Fig. 17.

20 Will-be-set-by-IN-TECH

c(i, j,

f( )

c(i, j, 6)

B

c(i, j, 2)

c(i, j, 1)

9)

c(i + 1, j + 1, 5)

c(i + 2, j + 1, 2)

c(i + 2, j, 3)

c(i + 1, j, 6)

(b) The EN cell in the node (*i*, *j*).

c(i, j, 3)

q(i, j)

c(i, j + 1, 8) c(i, j + 2, 7) c(i + 1, j + 2, 4)

c(i + 2, j + 2, 1)

d(i, j)

c(i, j, 7)

SI[ˇc(i, j + 2, 7)] SO[ˇc(i, j + 2, 7)] SI[ˇc(i, j + 1, 8)] SO[ˇc(i, j + 1, 8)]

SI[ˇq(i, j)] SO[ˇq(i, j)]

SO[ˇc(i + 1, j + 2, 4)] SI[ˇc(i + 1, j + 2, 4)]

SI[ˇc(i + 2, j + 2, 1)] SO[ˇc(i + 2, j + 2, 1)]

SO[ˇc(i + 1, j + 1, 5)]

SI[ˇc(i + 2, j + 1, 2)]

SO[ˇc(i + 2, j + 1, 2)]

(c) The IDN cell in the node (*i*, *j*).

Fig. 16. The EN and IDN marginalizing at the symbol level with the shifted topology for ISI

in the SISO modules *f* <sup>−</sup>1() only to the variable *c*ˇ(*i*, *j*, max(*k*)) and enables removing of all SISO

Let us approach the principle of this approximation at the first IDN example in the paragraph 7.3. There had **A** the dominant coefficient *a*(0, 0). Therefore, the SO[ ˇ*d*(*i*, *j*)] can be approximated by the SO[*c*ˇ(*i*, *j*, 9)] and inputs {SI[*c*ˇ(*i* + �1/2 + sin(*πk*/4)� , *j* + �1/2 − cos(*πk*/4)� , *<sup>k</sup>* + <sup>1</sup>)]}0≤*k*≤<sup>7</sup> of associated SISO modules will be directly equal to

SO[ˇc(i, j, 9)]

f−1( )

SI[ˇc(i, j, 9)]

SO[ˇc(i + 2, j, 3)]

SO[ˇc(i + 1, j, 6)]

SO[ˇc(i, j, 9)]

B−<sup>1</sup>

SI[ˇc(i, j, 9)]

> SI[ˇc(i + 1, j, 6)]

SI[ˇc(i + 2, j, 3)] SI[ˇc(i + 1, j + 1, 5)] c(i, j, 4) c(i, j, 5) c(i, j, 8)

i

j j + 1 j + 2

(a) The topology of the EN and IDN.

SI[ˇc(i, j, 7)] SO[ˇc(i, j, 7)]

SI[ˇc(i, j, 8)] SO[ˇc(i, j, 8)] SO[ˇc(i, j, 5)] SI[ˇc(i, j, 5)]

SI[dˇ(i, j)] SO[dˇ(i, j)]

SO[ˇc(i, j, 4)] SI[ˇc(i, j, 4)]

SI[ˇc(i, j, 1)]

SO[ˇc(i, j, 1)] SO[ˇc(i, j, 2)]

SI[ˇc(i, j, 6)]

SO[ˇc(i, j, 6)] SO[ˇc(i, j, 3)]

SI[ˇc(i, j, 3)]

SI[ˇc(i, j, 2)]

i + 1

i + 2

channel **A**GBC,*a*(0,0).

SO[*c*ˇ(*i*, *j*, 9)], as is shown in Fig. 17.

modules *B*−1.

Fig. 17. The cell in the node (*i*, *j*) of the simplified IDN marginalizing at the symbol level with centered topology for the ISI channel **A**GBC,*a*(0,0).

#### **7.5 Layered IDN for an ISI channel with a decomposition-able impulse response**

Finally, we focus on the Layered IDN for ISI channels with a decomposition-able impulse response to the horizontal **h** = [ *h*� *h h*� ] and vertical **g** = [ *g*� *g g*� ] direction. For example, the GBC has this property and therefore we assume, without detriment to generality, the GBC

$$\begin{aligned} \mathbf{A}\_{\text{GBC}, \text{g}}{\text{h}} &= \mathbf{h} \mathbf{g}^{T} \\ &= \begin{bmatrix} h' \mathbf{g'} \text{ h} \mathbf{g'} \text{ h'} \mathbf{g'} \\ h' \mathbf{g} \text{ hg} \text{ h'} \mathbf{g} \\ h' \mathbf{g'} \text{ hg'} \text{ h'} \mathbf{g'} \end{bmatrix} \end{aligned} \tag{43}$$

with size *L* × *L* = 3 × 3 that will be used for exemplary construction of the layered EN, shown in Fig. 18a, and mutually corresponding IDN, shown in Fig. 18b.

The signal processing (two-step convolution) starts on the EN bottom layer *c*(*i*, *j*) = *fR*(**h**, {*d*(*i*, *j* − 2), *d*(*i*, *j* − 1), *d*(*i*, *j*)}) = *h*� *d*(*i*, *j* − 2) + *hd*(*i*, *j* − 1) + *h*� *d*(*i*, *j*) and continues on the EN top layer *q*˜(*i*, *j*) = *fC*(**g**, {*c*(*i* − 2, *j*), *c*(*i* − 1, *j*), *c*(*i*, *j*)}) = *g*� *c*(*i* − 2, *j*) + *gc*(*i* − 1, *j*) + *g*� *c*(*i*, *j*)). Each row-wise or column-wise concatenated node (GPE) creates a one functional block with the fixed system of inputs and outputs N*fR* (*i*, *j*) = {R(*i*, *j*), *d*(*i*, *j*), R(*i*, *j* +1), *c*(*i*, *j*)} or N*fC* (*i*, *j*) = {C(*i*, *j*), *c*(*i*, *j*), C(*i*, *j* + 1), *q*(*i*, *j*)}, where R(*i*, *j*) = {*d*(*i*, *j* − 2), *d*(*i*, *j* − 1)} and C(*i*, *j*) = {*c*(*i* − 2, *j*), *c*(*i* − 1, *j*)} present auxiliary state variables containing two symbols together, as in the case of the distributed IDN marginalizing at the symbol block level.

This specific case of the 2D ISI channel makes 2D detection, through the chosen EN topology, decomposition-able into double 1D detection, when in separate rows and columns can be used the well known Fixed interval forward-backward algorithm (FI FBA) Chugg et al. (2001). Let us elucidate the FI FBA principle on the IDN top layer. Its current recursion, in the node

with their distribution to cooperative SISO modules). On the bottom layer is done identical process, directed by the rotated activation schedule merging the forward recursion <sup>1</sup> and the backward recursion <sup>2</sup> . The several iteration of the entire layered IDN creates firstly the FBA initiation on the top layer, terminated by the inter-marginalization {SO[*c*ˇ(*i*, *j*)]}*i*,*j*, and

The hard decision <sup>ˆ</sup>*d*(*i*, *<sup>j</sup>*) and primary adjustment of the input soft measures SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>*)] and SI[Cˇ(*i*, *<sup>j</sup>*)] (before first iteration of the IDN) is similar as in the previous cases and they are

The introduced IDNs can be evaluated in four angles: computation exigences, implementation

A distributed IDN marginalizing at the symbol level in comparison with the distributed IDN marginalizing at the symbol block level has less computation exigences and it is effectively applicable at whatever kind of the impulse response **A**. However it has plenty of jumpers on the other hand and its structure changes in accordance with the shape of the **A**. Each jumper presents one inside (auxiliary) variable, thus this IDN is highly suboptimal and with respect to marginalizations at the level of individual symbols (pixels) it fails on lower signal to noise

The structure of the distributed IDN marginalizing at the symbol block level is invariable for all responses **A**. Each SISO module always has the same inputs and outputs. Only shapes of the estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*), <sup>C</sup>ˇ(*i*, *<sup>j</sup>*) and <sup>B</sup>(*i*, *<sup>j</sup>*) are various and for certain special responses **A** don't have to exist in the optimal form that leads to the computationally simplest IDN. Therefore this IDN can have considerably more exacting computational complexity than IDN marginalizing at the symbol level, but due to marginalizations at the level of symbol (pixel)

The advantage of the layered IDN for an ISI channels with a decomposition-able impulse responses rests in the inter-marginalization between both layers, that makes its computational

IDNs. The worst properties has in the angle of application flexibility, because it can be applied only to some few ISI channels. In term of implementation complexity it is a structure relatively

We should use the tree-structured enumeration everywhere it is possible. It represents de facto a pipeline signal processing based upon an intermediate data usage. Fig. 19 shows this

As the example using tree-structured implementation, let us expose the simple SISO module *<sup>f</sup>* <sup>−</sup>1() in Fig. 20, where <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*, *<sup>j</sup>*) = {*c*ˇ(*i*, *<sup>j</sup>* <sup>−</sup> 1, 1), *<sup>c</sup>*ˇ(*i*, *<sup>j</sup>*, 3), *<sup>c</sup>*ˇ(*i*, *<sup>j</sup>* <sup>+</sup> 1, 2), *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*)}. Such soft

inversion is a component of the IDN marginalizing at the symbol level in Fig. 21.

<sup>2</sup> )*<sup>L</sup>* lower than complexity ∝ *ML*<sup>2</sup>

*<sup>d</sup>* of the both distributed

337

blocks it offers a quality output, even as the signal to noise ratio is very low.

*L*−1

**8.1 Tree-structured enumeration of combinations and marginalizations**

subsequently the same procedure on the bottom layer.

Image Restoration: Principles, Applications and Performance Analysis

complexity, application flexibility and performance.

*<sup>d</sup>* + (*Md*(2*Md* − 1)

**8. Implementation and complexity reduction issue**

directed by Eq. 13, Eq. 30 and Eq. 31.

2D Iterative Detection Network Based

**7.6 Summary of IDNs properties**

ratios.

complexity ∝ *M<sup>L</sup>*

simple.

principle.

(a) The topology of the layered EN with illustration of individual EN cells. (b) The topology of the layered IDN with illustration of individual IDN cells (SISO modules) and denotation of activation schedule.

Fig. 18. The layered EN and IDN for an ISI channel with a decomposition-able impulse response.

(*i*, *j*), is performed by following the sequence of operations: the combination (preprocessing — enumeration of auxiliary variables

$$\mathbb{S}[\check{N}\_{f\_{\mathbb{C}}}(i,j) \mid \{\check{\mathcal{L}}(i,j), \check{\mathcal{L}}(i+1,j)\}] = \mathbb{S}[\check{\varepsilon}(i,j)] \odot \text{SI}[\check{\eta}(i,j)] \tag{44}$$

and

$$\mathbb{S}[\check{\mathcal{N}}\_{f\_{\mathbb{C}}}(i,j) \mid \{\check{\varepsilon}(i,j), \check{\mathfrak{q}}(i,j)\}] = \mathbb{S}[\check{\mathcal{C}}(i,j)] \odot \mathbb{S}[\check{\mathcal{C}}(i+1,j)] \tag{45}$$

with their storage), the forward recursion �<sup>3</sup> (top-down)

$$\mathbf{SO}[\mathcal{C}(i+1,j)] = \underset{\mathcal{N}\_{\mathbf{f}\_{\mathbf{C}}}(i,j), \mathcal{C}(i+1,j)}{\otimes} \mathbb{S}[\mathcal{N}\_{f\_{\mathbf{C}}}(i,j) \mid \{\mathcal{C}(i,j), \mathcal{C}(i+1,j)\}] \odot \mathbf{SI}[\mathcal{C}(i,j)],\tag{46}$$

the backward recursion �<sup>4</sup> (bottom-up)

$$\text{SO}[\mathcal{L}(i,j)] = \underset{\mathcal{N}\_{\mathcal{f}\_{\mathbb{C}}}(i,j)\mathcal{L}(i,j)}{\text{@}} \text{S}[\check{\mathcal{N}}\_{f\_{\mathbb{C}}}(i,j) \mid \{\mathcal{L}(i,j), \mathcal{L}(i+1,j)\}] \odot \text{SI}[\mathcal{C}(i+1,j)] \tag{47}$$

and the completion operation (postprocessing — enumeration of output variables

$$\text{SO}[\mathbb{X}(i,j)] = \underset{\mathcal{N}\_{\mathbb{f}\_{\mathbb{C}}}(i,j): \mathbb{X}(i,j)}{\text{\"\S\"}} \text{S}[\check{\mathcal{N}}\_{f\_{\mathbb{C}}}(i,j) \text{\"\}} \left\{ \mathbb{X}(i,j), \not q(i,j) \right\} [\mathbb{G}\text{SI}[\not](i,j)] \tag{48}$$

and

$$\text{SO}[\mathfrak{q}(i,j)] = \underset{\tilde{N}\_{\mathbb{C}}(i,j): \mathfrak{q}(i,j)}{\text{\uph}} \text{S}[\check{N}\_{f\mathbb{C}}(i,j) \text{ } \{\mathfrak{k}(i,j), \mathfrak{q}(i,j)\}] \odot \text{SI}[\mathfrak{k}(i,j)] \text{ (optional)}\tag{49}$$

with their distribution to cooperative SISO modules). On the bottom layer is done identical process, directed by the rotated activation schedule merging the forward recursion <sup>1</sup> and the backward recursion <sup>2</sup> . The several iteration of the entire layered IDN creates firstly the FBA initiation on the top layer, terminated by the inter-marginalization {SO[*c*ˇ(*i*, *j*)]}*i*,*j*, and subsequently the same procedure on the bottom layer.

The hard decision <sup>ˆ</sup>*d*(*i*, *<sup>j</sup>*) and primary adjustment of the input soft measures SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>*)] and SI[Cˇ(*i*, *<sup>j</sup>*)] (before first iteration of the IDN) is similar as in the previous cases and they are directed by Eq. 13, Eq. 30 and Eq. 31.

## **7.6 Summary of IDNs properties**

22 Will-be-set-by-IN-TECH

i

Fig. 18. The layered EN and IDN for an ISI channel with a decomposition-able impulse

(*i*, *j*), is performed by following the sequence of operations: the combination (preprocessing

SI[ˇq(i, j)] SO[ˇq(i, j)]

�<sup>2</sup>

�<sup>1</sup>

SO[Cˇ(i, j)] SI[Cˇ(i, j)]

f−<sup>1</sup> <sup>R</sup> ( )

SI[Cˇ(i, j)] SO[Cˇ(i, j)]

(b) The topology of the layered IDN with illustration of individual IDN cells (SISO modules) and denotation

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {Cˇ(*i*, *<sup>j</sup>*), <sup>C</sup>ˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)}]�<sup>C</sup> SI[Cˇ(*i*, *<sup>j</sup>*)], (46)

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {Cˇ(*i*, *<sup>j</sup>*), <sup>C</sup>ˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)}]�<sup>C</sup> SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)] (47)

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {*c*ˇ(*i*, *<sup>j</sup>*), *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*)}]�<sup>C</sup> SI[*q*ˇ(*i*, *<sup>j</sup>*)] (48)

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {*c*ˇ(*i*, *<sup>j</sup>*), *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*)}]�<sup>C</sup> SI[*c*ˇ(*i*, *<sup>j</sup>*)] (optional) (49)

j

f−<sup>1</sup> <sup>C</sup> ( )

f−<sup>1</sup> <sup>C</sup> ( )

f−<sup>1</sup> <sup>C</sup> ( )

�<sup>4</sup> �<sup>3</sup>

SI[ ˇ

f−<sup>1</sup> <sup>R</sup> ( )

SO[ˇc(i, j)] SI[ˇc(i, j)]

SO[ ˇR(i, j)]

SI[ ˇR(i, j)]

SI[ ˇR(i, j)] SO[ ˇR(i, j)]

SI[ˇc(i, j)]

of activation schedule.

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {Cˇ(*i*, *<sup>j</sup>*), <sup>C</sup>ˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)}] = <sup>S</sup>[*c*ˇ(*i*, *<sup>j</sup>*)]�<sup>C</sup> SI[*q*ˇ(*i*, *<sup>j</sup>*)] (44)

<sup>S</sup>[N<sup>ˇ</sup> *fC* (*i*, *<sup>j</sup>*) \ {*c*ˇ(*i*, *<sup>j</sup>*), *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*)}] = <sup>S</sup>[Cˇ(*i*, *<sup>j</sup>*)]�<sup>C</sup> <sup>S</sup>[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)] (45)

f−<sup>1</sup> <sup>R</sup> ( )

SO[ˇc(i, j)]

SO[ ˇ

d(i, j)]

d(i, j)]

d(i, j)

fR( ) fR( ) fR( )

(a) The topology of the layered EN with

illustration of individual EN cells.

— enumeration of auxiliary variables

response.

and

and

R(i, j)

i

c(i, j) q(i, j)

f<sup>C</sup> ( )

f<sup>C</sup> ( )

C(i, j)

with their storage), the forward recursion �<sup>3</sup> (top-down)

Nˇ *f*

*<sup>C</sup>* (*i*,*j*):Cˇ(*i*+1,*j*)

and the completion operation (postprocessing — enumeration of output variables

SO[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)] = �<sup>M</sup>

the backward recursion �<sup>4</sup> (bottom-up)

SO[Cˇ(*i*, *<sup>j</sup>*)] = �<sup>M</sup>

SO[*q*ˇ(*i*, *j*)] = �<sup>M</sup>

Nˇ *f*

SO[*c*ˇ(*i*, *j*)] = �<sup>M</sup>

Nˇ *f*

*<sup>C</sup>* (*i*,*j*):Cˇ(*i*,*j*)

Nˇ *f*

*<sup>C</sup>* (*i*,*j*):*q*ˇ(*i*,*j*)

*<sup>C</sup>* (*i*,*j*):*c*ˇ(*i*,*j*)

j

f<sup>C</sup> ( )

The introduced IDNs can be evaluated in four angles: computation exigences, implementation complexity, application flexibility and performance.

A distributed IDN marginalizing at the symbol level in comparison with the distributed IDN marginalizing at the symbol block level has less computation exigences and it is effectively applicable at whatever kind of the impulse response **A**. However it has plenty of jumpers on the other hand and its structure changes in accordance with the shape of the **A**. Each jumper presents one inside (auxiliary) variable, thus this IDN is highly suboptimal and with respect to marginalizations at the level of individual symbols (pixels) it fails on lower signal to noise ratios.

The structure of the distributed IDN marginalizing at the symbol block level is invariable for all responses **A**. Each SISO module always has the same inputs and outputs. Only shapes of the estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*), <sup>C</sup>ˇ(*i*, *<sup>j</sup>*) and <sup>B</sup>(*i*, *<sup>j</sup>*) are various and for certain special responses **A** don't have to exist in the optimal form that leads to the computationally simplest IDN. Therefore this IDN can have considerably more exacting computational complexity than IDN marginalizing at the symbol level, but due to marginalizations at the level of symbol (pixel) blocks it offers a quality output, even as the signal to noise ratio is very low.

The advantage of the layered IDN for an ISI channels with a decomposition-able impulse responses rests in the inter-marginalization between both layers, that makes its computational complexity ∝ *M<sup>L</sup> <sup>d</sup>* + (*Md*(2*Md* − 1) *L*−1 <sup>2</sup> )*<sup>L</sup>* lower than complexity ∝ *ML*<sup>2</sup> *<sup>d</sup>* of the both distributed IDNs. The worst properties has in the angle of application flexibility, because it can be applied only to some few ISI channels. In term of implementation complexity it is a structure relatively simple.

## **8. Implementation and complexity reduction issue**

## **8.1 Tree-structured enumeration of combinations and marginalizations**

We should use the tree-structured enumeration everywhere it is possible. It represents de facto a pipeline signal processing based upon an intermediate data usage. Fig. 19 shows this principle.

As the example using tree-structured implementation, let us expose the simple SISO module *<sup>f</sup>* <sup>−</sup>1() in Fig. 20, where <sup>N</sup><sup>ˇ</sup> *<sup>f</sup>*(*i*, *<sup>j</sup>*) = {*c*ˇ(*i*, *<sup>j</sup>* <sup>−</sup> 1, 1), *<sup>c</sup>*ˇ(*i*, *<sup>j</sup>*, 3), *<sup>c</sup>*ˇ(*i*, *<sup>j</sup>* <sup>+</sup> 1, 2), *<sup>q</sup>*ˇ(*i*, *<sup>j</sup>*)}. Such soft inversion is a component of the IDN marginalizing at the symbol level in Fig. 21.

(a) Combinations.

(b) Marginalizations.

Fig. 19. The tree-structured enumeration of combinations and marginalizations.

#### **8.2 Fixation of the arithmetics**

The most sensitive arithmetics have the IDNs implemented in the Pd. A large number of quantities less than 1 are multiplied in the SISO modules of such IDNs and the underflow of the arithmetics can happen. Therefore the Pd requires the regular scaling of the output measures

$$\text{PO}[\mathcal{S}^{(\ell)}(k)] = \frac{\text{PO}[\mathcal{S}^{(\ell)}(k)]}{\sum\_{\ell} \text{PO}[\mathcal{S}^{(\ell)}(k)]} \tag{50}$$

S[ ˇNf (i, j) = {1, 1, 1, 2h + h}]

�<sup>M</sup>

SO[ˇc(i, j + 1, 2) = 1]

*c*(*i*, *j* − 1, 1) + *hc*(*i*, *j*, 3)

SI[ˇq(i, j)=2h + h]

SO[ˇq(i, j)=2h + h] SI[ˇc(i, j + 1, 2) = 1] 339

SI[ˇc(i, j, 3) = 1]

SI[ˇc(i, j

− 1, 1) = 1]

S[ ˇNf (i, j) = {0, 1, 1, h + h}]

()�<sup>C</sup> () <sup>−</sup><sup>1</sup> <sup>−</sup><sup>1</sup> ()�<sup>C</sup> () <sup>−</sup><sup>1</sup> ()�<sup>C</sup> ()

SO[ˇc(i, j, 3) = 1]

SO[ˇc(i, j

− 1, 1) = 1] �<sup>M</sup>

S[ ˇNf (i, j) = {1, 0, 1, 2h

S[ ˇNf (i, j) = {0, 0, 1, h

�<sup>M</sup> �<sup>M</sup> �<sup>M</sup>

�<sup>M</sup> �<sup>M</sup> �<sup>M</sup> �<sup>M</sup> �<sup>M</sup> �<sup>M</sup>

�<sup>M</sup> �<sup>M</sup>

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

SO[ˇq(i, j) = h + h]

S[ ˇNf (i, j) = {1, 1, 0, h + h}]

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

]

SO[ˇq(i, j) = h

Fig. 20. The implementation of the SISO module *f* <sup>−</sup>1() using tree-structured enumeration of

<sup>−</sup><sup>1</sup> ()�<sup>C</sup> ()

SO[ˇc(i, j

− 1, 1) = 0]

S[ ˇNf (i, j) = {0, 1, 0, h}]

S[ ˇNf (i, j) = {1, 0, 0, h

()�<sup>C</sup> () <sup>−</sup><sup>1</sup>

SO[ˇc(i, j, 3) = 0]

combinations and marginalizations, where *f*(): *q*(*i*, *j*) = *h*�

�<sup>M</sup>

<sup>−</sup><sup>1</sup> ()�<sup>C</sup> ()

SO[ˇc(i, j + 1, 2) = 0]

+*h*�

*c*(*i*, *j* + 1, 2)

�<sup>M</sup>

S[ ˇNf (i, j) = {0, 0, 0, 0}]

�<sup>M</sup>

SI[ˇc(i, j

> SI[ˇq(i, j) = 0]

SO[ˇq(i, j) = 0]

�<sup>C</sup>

SI[ˇc(i, j, 3) = 0]

SI[ˇc(i, j + 1, 2) = 0]

− 1, 1) = 0]

2D Iterative Detection Network Based

}]

SI[ˇq(i, j) = h]

�<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup>

Image Restoration: Principles, Applications and Performance Analysis

SO[ˇq(i, j) = h]

}]

SI[ˇq(i, j) = h + h]

�<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup>

SI[ˇq(i, j) = h

]

}]

SO[ˇq(i, j)=2h

]

�<sup>C</sup>

�<sup>C</sup> �<sup>C</sup>

SI[ˇq(i, j)=2h

]

�<sup>C</sup> �<sup>C</sup> �<sup>C</sup> �<sup>C</sup>

and so, the Md is preferable to a real implementation of the IDN. The arithmetics of the Md is relatively stable and can be protected from the incidental overflow by the scaling

$$\text{MO}[\mathcal{S}^{(\ell)}(k)] = \text{MO}[\mathcal{S}^{(\ell)}(k)] - \min\_{\ell} \text{MO}[\mathcal{S}^{(\ell)}(k)].\tag{51}$$

But since the overflow is rare as the better scaling is MO[S()(*k*)] = MO[S()(*k*)] <sup>−</sup> MO[S(1)(*k*)] that allows discount the number of swapped measures by the fixed measure MO[S(1)(*k*)] = 0.

#### **8.3 Additional sub-optimality embedding complexity reduction**

The distributed IDNs can be simplified in addition by the approximation neglecting least significant rays in the original response of the blurring channel. For example, the

$$\mathbf{A}\_{\text{GBC},0.2574} = \begin{bmatrix} 0.0607 \ 0.1250 \ 0.0607\\ 0.1250 \ 0.2574 \ 0.1250\\ 0.0607 \ 0.1250 \ 0.0607 \end{bmatrix} . \tag{52}$$

24 Will-be-set-by-IN-TECH

<sup>S</sup>[<sup>N</sup> (1) <sup>=</sup> {S(1)(1), <sup>S</sup>(1)(2), <sup>S</sup>(1) SI[<sup>S</sup> (3)}] (1)(3)]

(a) Combinations.

<sup>S</sup>[<sup>N</sup> ()]

<sup>S</sup>[<sup>N</sup> (2) <sup>=</sup> {S(1)(1), <sup>S</sup>(1)(2), <sup>S</sup>(2)(3)}]

�<sup>M</sup>

�<sup>M</sup>

SO[S(1)(1)]

SO[S(1)(2)]

<sup>∑</sup> PO[S()(*k*)] (50)

(*k*)]. (51)

⎦ . (52)

�<sup>M</sup>

�<sup>M</sup>

{S[<sup>N</sup> ()]}

(b) Marginalizations.

The most sensitive arithmetics have the IDNs implemented in the Pd. A large number of quantities less than 1 are multiplied in the SISO modules of such IDNs and the underflow of the arithmetics can happen. Therefore the Pd requires the regular scaling of the output

and so, the Md is preferable to a real implementation of the IDN. The arithmetics of the Md is

But since the overflow is rare as the better scaling is MO[S()(*k*)] = MO[S()(*k*)] <sup>−</sup> MO[S(1)(*k*)] that allows discount the number of swapped measures by the fixed measure

The distributed IDNs can be simplified in addition by the approximation neglecting least

(*k*)] = PO[S()(*k*)]

(*k*)] − min 

0.0607 0.1250 0.0607 0.1250 0.2574 0.1250 0.0607 0.1250 0.0607

MO[S()

⎤

Fig. 19. The tree-structured enumeration of combinations and marginalizations.

relatively stable and can be protected from the incidental overflow by the scaling

significant rays in the original response of the blurring channel. For example, the

⎡ ⎣

(*k*)] = MO[S()

PO[S()

MO[S()

**8.3 Additional sub-optimality embedding complexity reduction**

**A**GBC,0.2574 =

�<sup>M</sup>

o[h]

**8.2 Fixation of the arithmetics**

measures

MO[S(1)(*k*)] = 0.

SI[S(2)(3)]

∪S(1)(2)} {S[N<sup>ˇ</sup> ]}N<sup>ˇ</sup> :{S(1)(1)

{S[N<sup>ˇ</sup> ]}N<sup>ˇ</sup> :S(1)(2)

{S[N<sup>ˇ</sup> ]}N<sup>ˇ</sup> :S(1)(1)

�<sup>C</sup>

�<sup>C</sup>

�<sup>C</sup>

SI[S(1)(2)]

SI[S(1)(1)]

Fig. 20. The implementation of the SISO module *f* <sup>−</sup>1() using tree-structured enumeration of combinations and marginalizations, where *f*(): *q*(*i*, *j*) = *h*� *c*(*i*, *j* − 1, 1) + *hc*(*i*, *j*, 3) +*h*� *c*(*i*, *j* + 1, 2)

j − 1 j j + 1

Image Restoration: Principles, Applications and Performance Analysis

2D Iterative Detection Network Based

c(i, j, 2) c(i − 1, j, 4)

f( )

c(i + 1, j, 2)

(b) The EN cell in the node (*i*, *j*).

c(i, j, 4) c(i, j + 1, 3)

341

q(i, j)

B c(i, j, 1)

c(i, j, 5)

SO[ˇc(i, j, 5)] SI[ˇc(i, j, 5)]

SO[ˇc(i, j, 1)]

SI[ˇc(i, j + 1, 3)] SO[ˇc(i, j + 1, 3)]

SI[ˇq(i, j)] SO[ˇq(i, j)]

c(i, j − 1, 1) c(i, j, 3)

d(i, j)

(a) The topology of the EN and IDN.

SI[ˇc(i, j, 5)] SO[ˇc(i, j, 5)]

SI[ˇc(i, j, 3)] SI[dˇ(i, j)] SO[dˇ(i, j)]

SI[ˇc(i, j − 1, 1)] SO[ˇc(i, j − 1, 1)]

the ISI channel **A**SGBC,*a*(0,0).

that lead to the computationally simplest IDN.

SI[ˇc(i, j, 4)]

SI[ˇc(i, j, 1)] SO[ˇc(i, j, 3)]

SI[ˇc(i, j, 2)]

B−<sup>1</sup>

SO[ˇc(i, j, 2)] SO[ˇc(i − 1, j, 4)]

SI[ˇc(i − 1, j, 4)]

> SI[ˇc(i + 1, j, 2)]

f−1( )

SO[ˇc(i + 1, j, 2)]

(c) The IDN cell in the node (*i*, *j*).

Fig. 23. The EN and IDN marginalizing at the symbol level with the centered topology for

Can happen the situation, when the channel truncation causes the original 2D ISI channel atypical. Therefore, such kind of approximation is suitable especially for the IDN marginalizing at the symbol level, because it is a very flexible structure, applicable to the absolutely arbitrary 2D ISI channel. In case of the IDN marginalizing at the symbol block level can occur the problem with fulfillment of the condition N*d*(*i*, *j*) = R(*i*, *j*) ∪ C(*i*, *j*) ∪ *d*(*i* + *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**) and the estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*) and <sup>C</sup>ˇ(*i*, *<sup>j</sup>*) don't have to exist in the optimal shapes

SO[ˇc(i, j, 4)]

i + 1

i

i − 1

can be simplified to the form

$$\mathbf{A}\_{\text{SGBC},0.3399} = \begin{bmatrix} 0 & 0.1650 & 0\\ 0.1650 \, 0.3399 \, 0.1650\\ 0 & 0.1650 & 0 \end{bmatrix} \, . \tag{53}$$

(a) The topology of the EN and IDN.

(c) The IDN cell in the node (*i*, *j*).

Fig. 21. The EN and IDN marginalizing at the symbol level with the centered topology for the horizontal ISI channel **A** = [ *h*� *h h*� ].

by the truncation of last four insignificant rays. The consequences of such approximation are illustrated in Fig. 23 and Fig. 22. In the both cases it reduces computational complexity to <sup>1</sup> 16 . Moreover, it lowers the number of jumpers to one half in the case of the IDN marginalizing at the symbol level. The price paid for this rapid simplification rests in a quality degradation of the reconstructed picture.

(a) Layouts for the IDN without bias state variable. (b) Layouts for the IDN with bias state variable.

Fig. 22. The layouts of auxiliary state variables R(*i*, *j*), C(*i*, *j*) and B(*i*, *j*) in the IDN marginalizing at the symbol block level for the ISI channel **A**SGBC,*a*(0,0).

26 Will-be-set-by-IN-TECH

SI[ˇc(i, j, 1)] SI[dˇ(i, j)]

(c) The IDN cell in the node (*i*, *j*).

by the truncation of last four insignificant rays. The consequences of such approximation are illustrated in Fig. 23 and Fig. 22. In the both cases it reduces computational complexity to <sup>1</sup>

Moreover, it lowers the number of jumpers to one half in the case of the IDN marginalizing at the symbol level. The price paid for this rapid simplification rests in a quality degradation of

variable.

ASGBC,a(0,0)

Fig. 22. The layouts of auxiliary state variables R(*i*, *j*), C(*i*, *j*) and B(*i*, *j*) in the IDN

Fig. 21. The EN and IDN marginalizing at the symbol level with the centered topology for

f−1( )

B−<sup>1</sup>

0 0.1650 0 0.1650 0.3399 0.1650 0 0.1650 0

> c(i, j − 1, 1) c(i, j, 2)

> > c(i, j, 3)

d(i, j)

⎤

B

SI[ˇc(i, j + 1, 2)] SO[ˇc(i, j + 1, 2)]

SI[ˇq(i, j)] SO[ˇq(i, j)]

SI[ˇc(i, j, 3)] SO[ˇc(i, j, 3)]

SO[ˇc(i, j, 1)]

f( )

C(i, j)

d(i+2,j+2)

ASGBC,a(0,0)

B(i, j)

R(i, j)

(b) Layouts for the IDN with bias state

(b) The EN cell in the node (*i*, *j*).

⎦ . (53)

q(i, j)

c(i, j, 1)

c(i, j + 1, 2)

16 .

⎡ ⎣

**A**SGBC,0.3399 =

j − 1 j j + 1

(a) The topology of the EN and IDN.

SO[ˇc(i, j, 2)] SI[ˇc(i, j, 2)]

SO[dˇ(i, j)]

SI[ˇc(i, j, 3)] SO[ˇc(i, j, 3)]

d(i+2,j+2)

marginalizing at the symbol block level for the ISI channel **A**SGBC,*a*(0,0).

C(i, j)

R(i, j)

(a) Layouts for the IDN without bias state

SI[ˇc(i, j − 1, 1)] SO[ˇc(i, j − 1, 1)]

the horizontal ISI channel **A** = [ *h*� *h h*� ].

the reconstructed picture.

variable.

can be simplified to the form

i

(a) The topology of the EN and IDN.

(b) The EN cell in the node (*i*, *j*).

(c) The IDN cell in the node (*i*, *j*).

Fig. 23. The EN and IDN marginalizing at the symbol level with the centered topology for the ISI channel **A**SGBC,*a*(0,0).

Can happen the situation, when the channel truncation causes the original 2D ISI channel atypical. Therefore, such kind of approximation is suitable especially for the IDN marginalizing at the symbol level, because it is a very flexible structure, applicable to the absolutely arbitrary 2D ISI channel. In case of the IDN marginalizing at the symbol block level can occur the problem with fulfillment of the condition N*d*(*i*, *j*) = R(*i*, *j*) ∪ C(*i*, *j*) ∪ *d*(*i* + *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**) and the estimators <sup>R</sup><sup>ˇ</sup> (*i*, *<sup>j</sup>*) and <sup>C</sup>ˇ(*i*, *<sup>j</sup>*) don't have to exist in the optimal shapes that lead to the computationally simplest IDN.

Mentioned BER curves leads to conclusion, that the application of IDNs with bias state variables is better than standard IDNs marginalizing at the symbol block level, because bring evident implementation advantages beside performance preservation. In the case of 1st version it is consensus in the output state variables R(*i*, *j* + 1) = C(*i* + 1, *j*) = B(*i* + 1, *j* + 1), that warrants equation SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]<sup>C</sup> SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)] = SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)]<sup>C</sup> SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)] = SI[Bˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]<sup>C</sup> SI[Bˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)] and allows only one and same marginalization for whole triplet of the state variables. On the other hand, the 2nd version is visible significant reduction

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

Fig. 26. The BER curves (1st, 2nd, 3th and 7th iteration) of the distributed IDN marginalizing at the symbol block level (Pd-SyD, black line with squares) and distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 1st version — red line with triangles, 2nd version — black line with triangles): Kernel **<sup>A</sup>**GBC,*a*(0,0), *Nb* = 8, <sup>Δ</sup>*<sup>Q</sup>* = 3, *C* = 612. The comparison with BER curves (dot line with circles) of the threshold detector.

a(0, 0)[−]

a(0, 0)[−]

a(0, 0)[−]

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

a(0, 0)[−]

a(0, 0)[−]

a(0, 0)[−]

343

10−4

10−4

10−4

BER(a(0, 0), μT = 75, μR = 50, σR = 20)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 75, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 75, μR = 12, σR = 5)[−]

10−3

10−2

10−1

100

10−4

10−4

10−4

BER(a(0, 0), μT = 50, μR = 50, σR = 20)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 50, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 50, μR = 12, σR = 5)[−]

10−3

10−2

10−1

100

Image Restoration: Principles, Applications and Performance Analysis

of state variable cardinalities.

2D Iterative Detection Network Based

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

a(0, 0)[−]

a(0, 0)[−]

a(0, 0)[−]

10−4

10−4

10−4

BER(a(0, 0), μT = 25, μR = 50, σR = 20)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 25, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 25, μR = 12, σR = 5)[−]

10−3

10−2

10−1

100

## **9. Examples of dichromaric picture restoration, performance analyses and conclusions**

## **9.1 Suppression of the defocusing in the imperfectly adjusted lens (GBC) with BER performance analyses**

We will demonstrate the IDNs functionality on QR code snapshot restorations by the IDNs marginalizing at the symbol block level that were described in paragraphs 7.1 and 7.2. In Fig. 24 and Fig. 25 are shown example restorations of two different QR codes by the 1st and 2nd version of IDNs with bias state variables. Additionally, all versions of these IDNs has been tested with the Monte Carlo method for performance and compared with simple threshold detector, based on the relation <sup>ˆ</sup>*dTD*(*i*, *<sup>j</sup>*) = arg min <sup>ˇ</sup>*d*(*i*,*j*)∈{0,1} <sup>|</sup> *xd* (*i*,*j*)−*μ*ˆ*<sup>T</sup> <sup>C</sup>* <sup>−</sup> <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)|. The result of this analysis is the set of BER curves shown in Fig. 26 and Fig. 27. As we can see in the BER curves, all tested IDNs have almost same performance and especially in the area of higher defocusing the usage is expedient. However, the performance slightly falls down in the focusation rising and at the beginning of iterative process. It is caused by diminishing corelation among individual neighboring pixels when iteratively precised state variables do not carry so fundamental and strong information worth to the current node of the network.

Fig. 24. The example of the QR code restoration by the distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 1st version): Kernel **A**GBC,2/10, *Nb* = 8, Δ*<sup>Q</sup>* = 3, *C* = 612, *μ<sup>T</sup>* = 50, *μ<sup>R</sup>* = 25, *σ<sup>R</sup>* = 10.

Fig. 25. The example of the QR code restoration by the distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 2nd version): Kernel **A**GBC,2/10, *Nb* = 8, Δ*<sup>Q</sup>* = 3, *C* = 612, *μ<sup>T</sup>* = 50, *μ<sup>R</sup>* = 25, *σ<sup>R</sup>* = 10.

In other words, the output {SI[*d*()(*<sup>i</sup>* <sup>+</sup> *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**)]} of current node strongly depends only on the input {SI[*q*()(*i*, *<sup>j</sup>*)]} (product of the SODEM) and not so much on the state information {SI[R()(*i*, *<sup>j</sup>*)]}, {SI[R()(*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]}, {SI[C()(*i*, *<sup>j</sup>*)]}, {SI[C()(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)]}, {SI[B()(*i*, *<sup>j</sup>*)]} and {SI[B()(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]} from other nodes as in the case of high defocusing.

28 Will-be-set-by-IN-TECH

We will demonstrate the IDNs functionality on QR code snapshot restorations by the IDNs marginalizing at the symbol block level that were described in paragraphs 7.1 and 7.2. In Fig. 24 and Fig. 25 are shown example restorations of two different QR codes by the 1st and 2nd version of IDNs with bias state variables. Additionally, all versions of these IDNs has been tested with the Monte Carlo method for performance and compared with simple

result of this analysis is the set of BER curves shown in Fig. 26 and Fig. 27. As we can see in the BER curves, all tested IDNs have almost same performance and especially in the area of higher defocusing the usage is expedient. However, the performance slightly falls down in the focusation rising and at the beginning of iterative process. It is caused by diminishing corelation among individual neighboring pixels when iteratively precised state variables do not carry so fundamental and strong information worth to the current node of the network.

> **D**ˆ , *I* = 5, BER = 0.022

> **D**ˆ , *I* = 5, BER = 0.006

*xd* (*i*,*j*)−*μ*ˆ*<sup>T</sup>*

**D**ˆ , *I* = 7, BER = 0.018

**D**ˆ , *I* = 7, BER = 0.004

*<sup>C</sup>* <sup>−</sup> <sup>ˇ</sup>*d*(*i*, *<sup>j</sup>*)|. The

**D**ˆ *TD*, BER = 0.141

**D**ˆ *TD*, BER = 0.085

**9. Examples of dichromaric picture restoration, performance analyses and**

**9.1 Suppression of the defocusing in the imperfectly adjusted lens (GBC) with BER**

threshold detector, based on the relation <sup>ˆ</sup>*dTD*(*i*, *<sup>j</sup>*) = arg min <sup>ˇ</sup>*d*(*i*,*j*)∈{0,1} <sup>|</sup>

**D**ˆ , *I* = 3, BER = 0.033

**D**ˆ , *I* = 3, BER = 0.021

{SI[B()(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]} from other nodes as in the case of high defocusing.

Fig. 24. The example of the QR code restoration by the distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 1st version): Kernel **A**GBC,2/10, *Nb* = 8,

Fig. 25. The example of the QR code restoration by the distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 2nd version): Kernel **A**GBC,2/10, *Nb* = 8,

In other words, the output {SI[*d*()(*<sup>i</sup>* <sup>+</sup> *<sup>H</sup>***A**, *<sup>j</sup>* <sup>+</sup> *<sup>W</sup>***A**)]} of current node strongly depends only on the input {SI[*q*()(*i*, *<sup>j</sup>*)]} (product of the SODEM) and not so much on the state information {SI[R()(*i*, *<sup>j</sup>*)]}, {SI[R()(*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]}, {SI[C()(*i*, *<sup>j</sup>*)]}, {SI[C()(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)]}, {SI[B()(*i*, *<sup>j</sup>*)]} and

**conclusions**

**performance analyses**

**R***<sup>d</sup>* **D**ˆ , *I* = 1,

**R***<sup>d</sup>* **D**ˆ , *I* = 1,

BER = 0.208

Δ*<sup>Q</sup>* = 3, *C* = 612, *μ<sup>T</sup>* = 50, *μ<sup>R</sup>* = 25, *σ<sup>R</sup>* = 10.

BER = 0.168

Δ*<sup>Q</sup>* = 3, *C* = 612, *μ<sup>T</sup>* = 50, *μ<sup>R</sup>* = 25, *σ<sup>R</sup>* = 10.

Mentioned BER curves leads to conclusion, that the application of IDNs with bias state variables is better than standard IDNs marginalizing at the symbol block level, because bring evident implementation advantages beside performance preservation. In the case of 1st version it is consensus in the output state variables R(*i*, *j* + 1) = C(*i* + 1, *j*) = B(*i* + 1, *j* + 1), that warrants equation SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]<sup>C</sup> SI[R<sup>ˇ</sup> (*i*, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)] = SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)]<sup>C</sup> SI[Cˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>*)] = SI[Bˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)]<sup>C</sup> SI[Bˇ(*<sup>i</sup>* <sup>+</sup> 1, *<sup>j</sup>* <sup>+</sup> <sup>1</sup>)] and allows only one and same marginalization for whole triplet of the state variables. On the other hand, the 2nd version is visible significant reduction of state variable cardinalities.

Fig. 26. The BER curves (1st, 2nd, 3th and 7th iteration) of the distributed IDN marginalizing at the symbol block level (Pd-SyD, black line with squares) and distributed IDN marginalizing at the symbol block level with bias state variables (Pd-SyD, 1st version — red line with triangles, 2nd version — black line with triangles): Kernel **<sup>A</sup>**GBC,*a*(0,0), *Nb* = 8, <sup>Δ</sup>*<sup>Q</sup>* = 3, *C* = 612. The comparison with BER curves (dot line with circles) of the threshold detector.

mentioned conditions are fulfilled, the IDN produces a high quality output (estimation **D**ˆ )

d(i+2,j+2)

R(i, j)

Fig. 28. The shapes of auxiliary state variables R(*i*, *j*) and C(*i*, *j*) in the IDN marginalizing at

The example of the IDN malfunction, caused by the wrong movement parameters adjustment,

This response makes the blurring in Fig. 32b. If the IDN is consequently fed by the wrong impulse response **A**<sup>0</sup> coming from the shifted point **s***<sup>s</sup>* , then its failure, shown in Fig. 32c, is going to happen. It presents a serious problem, because we mostly have not available so accurate information about the movement of the scanned object. Thus, in the overwhelming majority of real applications, the IDN will have been supplemented by the

The section vehicle speed measurement appears to be the most suitable application of the described method, because it is based on the signplate detection at the beginning and at the end of the monitored section, where the scanned vehicles have the same direction of the movement. So, the synchronization of the IDN will not represent a serious issue in this

⎡ ⎣

1 24 1 12 1 12 1 12 1 12 1 24

1 24 1 12 1 12 1 12 1 12 1 24

(b) **A**BOM,*π*/4.

⎤ ⎦ *T*

*<sup>s</sup>* of the **s**�

C(i, j)

ABOM,π/<sup>4</sup>

345

*<sup>s</sup>* and corresponds with the

, (54)

*<sup>s</sup>* placement inside the

on the other hand.

2D Iterative Detection Network Based

C(i, j)

ABOM,<sup>0</sup>

Image Restoration: Principles, Applications and Performance Analysis

is demonstrated in Fig. 32. Suppose that the true starting point is **s**�

**A**� BOM,0 =

auxiliary synchronizer performing the reliable estimation **s**ˆ�

d(i+6,j)

(a) **A**BOM,0.

the symbol block level for the ISI channel **A**BOM,*ϕ*.

correct impulse response

blue square in Fig. 32a.

application.

Fig. 27. The BER curves (1st, 2nd, 3th and 7th iteration) of the distributed IDN marginalizing at the symbol block level (Pd-PgD, black line with squares) and distributed IDN marginalizing at the symbol block level with bias state variables (Pd-PgD, 1st version — red line with triangles, 2nd version — black line with triangles): Kernel **<sup>A</sup>**GBC,*a*(0,0), *Nb* = 8, <sup>Δ</sup>*<sup>Q</sup>* = 3, *C* = 612. The comparison with BER curves (dot line with circles) of the threshold detector.

#### **9.2 Suppression of the blurring due to object moving (BOM)**

In this case we use for functionality demonstration the IDNs marginalizing at the symbol level. The following Fig. 31 shows the sample pictures restored by these IDNs for two different combination of the BOM. Used IDNs (theirs topologies) are represented in Fig 29 and Fig 30. Of course the IDNs marginalizing at the symbol block level can be used for this issue too. Theirs state variables are shown in Fig. 28.

The usage without whatever other supporting aids (external synchronization) demands the perfect knowledge about the scanned moving object (**s***<sup>s</sup>* and velocity vector). But, if all 30 Will-be-set-by-IN-TECH

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

Fig. 27. The BER curves (1st, 2nd, 3th and 7th iteration) of the distributed IDN marginalizing at the symbol block level (Pd-PgD, black line with squares) and distributed IDN marginalizing at the symbol block level with bias state variables (Pd-PgD, 1st version — red line with triangles, 2nd version — black line with triangles): Kernel **<sup>A</sup>**GBC,*a*(0,0), *Nb* = 8, <sup>Δ</sup>*<sup>Q</sup>* = 3, *C* = 612. The comparison with BER curves (dot line with circles) of the threshold detector.

In this case we use for functionality demonstration the IDNs marginalizing at the symbol level. The following Fig. 31 shows the sample pictures restored by these IDNs for two different combination of the BOM. Used IDNs (theirs topologies) are represented in Fig 29 and Fig 30. Of course the IDNs marginalizing at the symbol block level can be used for this issue too.

The usage without whatever other supporting aids (external synchronization) demands the perfect knowledge about the scanned moving object (**s***<sup>s</sup>* and velocity vector). But, if all

a(0, 0)[−]

a(0, 0)[−]

a(0, 0)[−]

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

a(0, 0)[−]

a(0, 0)[−]

a(0, 0)[−]

10−4

10−4

10−4

BER(a(0, 0), μT = 75, μR = 50, σR = 20)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 75, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 75, μR = 12, σR = 5)[−]

10−3

10−2

10−1

100

10−4

10−4

10−4

BER(a(0, 0), μT = 50, μR = 50, σR = 20)[−]

**9.2 Suppression of the blurring due to object moving (BOM)**

10−3

10−2

10−1

100

BER(a(0, 0), μT = 50, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 50, μR = 12, σR = 5)[−]

10−3

10−2

10−1

100

0.2 0.3 0.4 0.5 0.6 10−6

0.2 0.3 0.4 0.5 0.6 10−5

0.2 0.3 0.4 0.5 0.6 10−5

a(0, 0)[−]

Theirs state variables are shown in Fig. 28.

a(0, 0)[−]

a(0, 0)[−]

10−5 10−4 10−3 10−2 10−1 100

10−4

10−4

BER(a(0, 0), μT = 25, μR = 50, σR = 20)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 25, μR = 25, σR = 10)[−]

10−3

10−2

10−1

100

BER(a(0, 0), μT = 25, μR = 12, σR = 5)[−] mentioned conditions are fulfilled, the IDN produces a high quality output (estimation **D**ˆ ) on the other hand.

Fig. 28. The shapes of auxiliary state variables R(*i*, *j*) and C(*i*, *j*) in the IDN marginalizing at the symbol block level for the ISI channel **A**BOM,*ϕ*.

The example of the IDN malfunction, caused by the wrong movement parameters adjustment, is demonstrated in Fig. 32. Suppose that the true starting point is **s**� *<sup>s</sup>* and corresponds with the correct impulse response

$$\mathbf{A}'\_{\text{BOM},0} = \begin{bmatrix} \frac{1}{24} & \frac{1}{12} & \frac{1}{12} & \frac{1}{12} & \frac{1}{12} & \frac{1}{24} \\\\ \frac{1}{24} & \frac{1}{12} & \frac{1}{12} & \frac{1}{12} & \frac{1}{12} & \frac{1}{24} \end{bmatrix}^T \tag{54}$$

This response makes the blurring in Fig. 32b. If the IDN is consequently fed by the wrong impulse response **A**<sup>0</sup> coming from the shifted point **s***<sup>s</sup>* , then its failure, shown in Fig. 32c, is going to happen. It presents a serious problem, because we mostly have not available so accurate information about the movement of the scanned object. Thus, in the overwhelming majority of real applications, the IDN will have been supplemented by the auxiliary synchronizer performing the reliable estimation **s**ˆ� *<sup>s</sup>* of the **s**� *<sup>s</sup>* placement inside the blue square in Fig. 32a.

The section vehicle speed measurement appears to be the most suitable application of the described method, because it is based on the signplate detection at the beginning and at the end of the monitored section, where the scanned vehicles have the same direction of the movement. So, the synchronization of the IDN will not represent a serious issue in this application.

j − 1 j j + 1

2D Iterative Detection Network Based

Image Restoration: Principles, Applications and Performance Analysis

c(i, j, 3)

> c(i, j, 5)

c(i + 1, j, 2)

(b) The EN cell in the node (*i*, *j*).

c(i − 1, j, 5)

f( )

c(i, j, 2)

B

c(i, j, 6) c(i, j + 1, 4)

c(i, j, 1)

q(i, j)

c(i + 1, j + 1, 3)

347

c(i, j, 4)

c(i, j − 1, 1) c(i, j, 7)

d(i, j)

SO[ˇc(i, j, 7)] SI[ˇc(i, j, 7)]

SI[ˇc(i, j, 1)] SO[ˇc(i, j, 1)]

SO[ˇq(i, j)] SI[ˇq(i, j)] SO[ˇc(i, j + 1, 4)] SI[ˇc(i, j + 1, 4)]

SI[ˇc(i + 1, j + 1, 3)] SO[ˇc(i + 1, j + 1, 3)]

c(i − 1, j − 1, 6)

(a) The topology of the EN and IDN.

SO[ˇc(i − 1, j − 1, 6)] SI[ˇc(i − 1, j − 1, 6)] SO[dˇ(i, j)] SI[dˇ(i, j)] SI[ˇc(i, j, 4)] SO[ˇc(i, j, 4)]

> SO[ˇc(i, j − 1, 1)] SI[ˇc(i, j − 1, 1)] SO[ˇc(i, j, 7)] SI[ˇc(i, j, 7)]

> > SO[ˇc(i, j, 5)]

SI[ˇc(i, j, 5)]

(c) The IDN cell in the node (*i*, *j*).

Fig. 30. The EN and IDN marginalizing at the symbol level with the centered topology for

SO[ˇc(i + 1, j, 2)]

SI[ˇc(i − 1, j, 5)]

SO[ˇc(i, j, 2)]

SI[ˇc(i, j, 2)]

B−<sup>1</sup>

SI[ˇc(i, j, 3)] SO[ˇc(i, j, 3)] SO[ˇc(i − 1, j, 5)]

SI[ˇc(i + 1, j, 2)]

f−1( )

SI[ˇc(i, j, 6)]

SO[ˇc(i, j, 6)]

i + 1

the ISI channel **A**BOM,*π*/4.

i

i − 1

(a) The topology of the EN and IDN.

(b) The EN cell in the node (*i*, *j*).

(c) The IDN cell in the node (*i*, *j*).

32 Will-be-set-by-IN-TECH

c(i, j, 4)

c(i, j, 3)

d(i, j)

c(i, j, 7)

(*i*, *j*).

SI[ˇc(i + 2, j, 1)] SI[ˇq(i, j)] SO[ˇq(i, j)]

SO[ˇc(i, j, 7)] SI[ˇc(i, j, 7)]

SO[ˇc(i + 2, j, 1)]

SO[ˇc(i + 1, j, 2)]

f−1( )

SO[ˇc(i + 3, j, 3)]

(c) The IDN cell in the node (*i*, *j*).

Fig. 29. The EN and IDN marginalizing at the symbol level with the centered topology for

SI[ˇc(i, j, 4)] SI[ˇc(i, j, 5)] SO[ˇc(i, j, 6)]

SI[ˇc(i, j, 6)]

SO[ˇc(i, j, 5)]

SO[ˇc(i, j, 4)]

SI[ˇc(i, j, 3)]

SO[ˇc(i, j, 3)] SI[ˇc(i, j, 2)]

B−<sup>1</sup>

SO[ˇc(i, j, 2)] SI[ˇc(i, j, 1)]

SO[ˇc(i, j, 1)] SI[ˇc(i − 1, j, 5)]

SO[ˇc(i − 1, j, 5)]

SI[ˇc(i − 3, j, 6)]

SO[ˇc(i − 3, j, 6)]

SO[ˇc(i − 2, j, 4)]

SI[ˇc(i − 2, j, 4)]

SI[ˇc(i, j, 7)] SO[ˇc(i, j, 7)] SI[dˇ(i, j)] SO[dˇ(i, j)]

> SI[ˇc(i + 3, j, 3)]

SI[ˇc(i + 1, j, 2)] c(i, j, 2)

B

c(i, j, 1) c(i − 3, j, 6)

c(i − 1, j, 5)

f( )

c(i − 2, j, 4)

c(i, j, 5)

c(i, j, 6)

(b) The EN cell in the node

c(i + 2, j, 1) q(i, j)

c(i + 1, j, 2)

c(i + 3, j, 3)

i − 3

IDN.

(a) The topology of the EN and

the ISI channel **A**BOM,0.

i − 2

i − 1

i

i + 1

i + 2

i + 3

j

Fig. 30. The EN and IDN marginalizing at the symbol level with the centered topology for the ISI channel **A**BOM,*π*/4.

**9.3 Conclusions and open problems**

Image Restoration: Principles, Applications and Performance Analysis

2D Iterative Detection Network Based

monitoring system), etc.

**10. References**

the algorithm can be found in (Noels et al., 2003).

The IDNs are based on the optimal (the best) MAP detector and so they are mostly able to obtain the optimal estimation. The IDN, analogous to the optimal detector, forms all possible variants of the input image and compares them with the corrupted image by the decision metric perfectly matched to the noise distribution. There is the difference only in fact, that the IDN solves this issue smartly by the suitable image segmentation and with minimal computational exigencies. This can be considered as the greatest advantage of the IDNs.

349

The important disadvantage rests in the application limitations, because the contemporary IDNs are able to restore a dichromatic (or black and white) patterns only (texts, car sign plates, QR codes, etc.). For restoration of grayscale or color image restorations and larger 2D ISI channels, where the number of all possible realizations of current convolution region {N ()} extremely grows, the IDN will require the another sub-optimality embedding simplification (generalization). Although the IDNs perform the segmental image processing, the numerical complexity is extreme. Individual SISO modules in the IDN can not be implemented directly as was shown on the simple example in Fig. 20, but must be realized also as the certain iterative system. This would establish an iterative detection network where each inner cell presents iterative subnet. The question remains how a single-shot SISO can be approximated to the iterative subnet. If it is possible and what level of sub-optimality this establish and whether this sub-optimality allows the good and fast convergence of the entire system to the correct solution. At present it is only surmise without concrete and functional results. But it is clear, that such network will be more suboptimal and its performace will not be so good as in the case of black & white images, where SISO module can be implemented as a single-shot (optimal) system. The next problem rests in fact, that is not possible analytically predicate the behaviour of such network due to extreme quantity of functional blocks and interconnections. Therefore the computer simulation and debuging will be very difficult and based on the labour principle. However, this method has very good application in the case of black & white images and it is completely different from classical methods like adaptive filtration, minimum mean square errors, etc. Because it is perfectly matched to the noise distribution and reconstruct image from all possible images by the intelligent way. Thanks to this ability the IDN is very powerful and can find good use in the area of image halftoning (Chugg et al., 2001), text detection, QR code detection, number plate detection of cars (traffic

The last problem rests in the iterative detection network synchronization. In all cases it was considered that the IDN has perfect knowledge about kernel of the 2D ISI channel. This information, however, in reality it is not known and must be estimated. A Soft decision directed (SDD) Channel state estimator (CSE) can be used for this purposes. One of the most suitable CSEs for an IDN synchronization is the Expectation-Maximization (EM) algorithm. Its greatest benefit rests in the implementation simplicity and additional information about

Chugg, K.; Anastasopoulos, A. & Chen, X. (2001). *Iterative detection : Adaptivity, Complexity reduction and Applications*, Kluwer Academic Publishers, ISBN 0-470-84757-3.

Fig. 31. The examples of image restorations by the distributed IDN marginalizing at the symbol level (Pd-PgD) for the channels **A**BOM,0(a) and **A**BOM,*π*/4(b) merged with the simple noise model including the thermal noise and the quantization noise: *Nb* = 8, Δ*<sup>Q</sup>* = 3, *C* = 768, *μ<sup>T</sup>* = 200.

(a) The trajectory starting point displacement.

(b) The realization **R***<sup>d</sup>* of the A/D converter output.

(c) The hard decision **Dˆ** after 7th iteration of the wrong adjusted IDN.

Fig. 32. The example of the wrong image restoration by the distributed IDN marginalizing at the symbol level (Pd-PgD), that was incorrectly set by the **A**BOM,0 at the true blurring **A** BOM,0 merged with the simple noise model including the thermal noise and the quantization noise: *C* = 768, *Nb* = 8, Δ*<sup>d</sup>* = 3, *μ<sup>T</sup>* = 200.

## **9.3 Conclusions and open problems**

34 Will-be-set-by-IN-TECH

(a.2) The hard decision **Dˆ** after 3th iteration of the IDN.

(b.2) The hard decision **Dˆ** after 3th iteration of the IDN.

Fig. 31. The examples of image restorations by the distributed IDN marginalizing at the symbol level (Pd-PgD) for the channels **A**BOM,0(a) and **A**BOM,*π*/4(b) merged with the simple noise model including the thermal noise and the quantization noise: *Nb* = 8, Δ*<sup>Q</sup>* = 3,

> (b) The realization **R***<sup>d</sup>* of the A/D converter output.

Fig. 32. The example of the wrong image restoration by the distributed IDN marginalizing at the symbol level (Pd-PgD), that was incorrectly set by the **A**BOM,0 at the true blurring **A**

merged with the simple noise model including the thermal noise and the quantization noise:

(a.3) The hard decision **Dˆ** after 7th iteration of the IDN.

(b.3) The hard decision **Dˆ** after 7th iteration of the IDN.

(c) The hard decision **Dˆ** after 7th iteration of the wrong adjusted

BOM,0

IDN.

(a.1) The realization **R***<sup>d</sup>* of the A/D converter output.

(b.1) The realization **R***<sup>d</sup>* of the A/D converter output.

> ss s s

(a) The trajectory starting point displacement.

*C* = 768, *Nb* = 8, Δ*<sup>d</sup>* = 3, *μ<sup>T</sup>* = 200.

*C* = 768, *μ<sup>T</sup>* = 200.

The IDNs are based on the optimal (the best) MAP detector and so they are mostly able to obtain the optimal estimation. The IDN, analogous to the optimal detector, forms all possible variants of the input image and compares them with the corrupted image by the decision metric perfectly matched to the noise distribution. There is the difference only in fact, that the IDN solves this issue smartly by the suitable image segmentation and with minimal computational exigencies. This can be considered as the greatest advantage of the IDNs.

The important disadvantage rests in the application limitations, because the contemporary IDNs are able to restore a dichromatic (or black and white) patterns only (texts, car sign plates, QR codes, etc.). For restoration of grayscale or color image restorations and larger 2D ISI channels, where the number of all possible realizations of current convolution region {N ()} extremely grows, the IDN will require the another sub-optimality embedding simplification (generalization). Although the IDNs perform the segmental image processing, the numerical complexity is extreme. Individual SISO modules in the IDN can not be implemented directly as was shown on the simple example in Fig. 20, but must be realized also as the certain iterative system. This would establish an iterative detection network where each inner cell presents iterative subnet. The question remains how a single-shot SISO can be approximated to the iterative subnet. If it is possible and what level of sub-optimality this establish and whether this sub-optimality allows the good and fast convergence of the entire system to the correct solution. At present it is only surmise without concrete and functional results. But it is clear, that such network will be more suboptimal and its performace will not be so good as in the case of black & white images, where SISO module can be implemented as a single-shot (optimal) system. The next problem rests in fact, that is not possible analytically predicate the behaviour of such network due to extreme quantity of functional blocks and interconnections. Therefore the computer simulation and debuging will be very difficult and based on the labour principle. However, this method has very good application in the case of black & white images and it is completely different from classical methods like adaptive filtration, minimum mean square errors, etc. Because it is perfectly matched to the noise distribution and reconstruct image from all possible images by the intelligent way. Thanks to this ability the IDN is very powerful and can find good use in the area of image halftoning (Chugg et al., 2001), text detection, QR code detection, number plate detection of cars (traffic monitoring system), etc.

The last problem rests in the iterative detection network synchronization. In all cases it was considered that the IDN has perfect knowledge about kernel of the 2D ISI channel. This information, however, in reality it is not known and must be estimated. A Soft decision directed (SDD) Channel state estimator (CSE) can be used for this purposes. One of the most suitable CSEs for an IDN synchronization is the Expectation-Maximization (EM) algorithm. Its greatest benefit rests in the implementation simplicity and additional information about the algorithm can be found in (Noels et al., 2003).

## **10. References**

Chugg, K.; Anastasopoulos, A. & Chen, X. (2001). *Iterative detection : Adaptivity, Complexity reduction and Applications*, Kluwer Academic Publishers, ISBN 0-470-84757-3.

**Part 3** 

**Interdisciplinarity** 


## **Part 3**

**Interdisciplinarity** 

36 Will-be-set-by-IN-TECH

350 Image Restoration – Recent Advances and Applications

Vucetic, B. & Juan, J. (2003). *Space-Time coding*, Kluwer Academic Publishers, ISBN

Vucetic, B. & Juan, J. (2000). *Turbo codes : Principles and Applications*, Kluwer Academic

Kekrt, D. & Klíma, M. (2008). A black & white picture reconstruction by iterative detection

Kekrt, D. & Klíma, M. (2008). The iterative detection network based recovery of black &

Kekrt, D. & Klíma, M. (2008). Layered Iterative detection network based text recognition in

Kekrt, D.; Klíma, M. & Podgorny, R. (2008). The iterative detection network suppression of

Kekrt, D.; Klíma, M. & Fliegel, K. (2008). The iterative detection network based suppression

Kekrt, D.; Klíma, M. & Podgorny, R. (2008). Aspects of Image Quality Enhancement in Security

J. van Vliet L.; Boddeke F. R.; Sudar D. & Young I.T. (1998). Image detectors for digital

Anastasopoulos, A. & Chugg, K. (2000). Adaptive soft-input soft-output algorithms for

Noels, N.; Herzet, C.; Dejonghe, A.; Lottici, V. ; Steendam, H.; Moeneclaey, M.; Luise, M.;

*IEEE International Conference on Communications - ICC*, 2003.

142-149, ISBN 978-1-4244-1816-9, Prague - Czech Republic, October 2008. Kekrt, D. & Klíma, M. (2008). The advanced noise model for an IDN based restoration of black

ISBN 978-1-4244-2087-2, Prague, Czech Republic, April 2008.

network in the image capturing system with CCD (CMOS) sensor, *50th International Symposium ELMAR-2008*, Vol. 1, pp. 125-128, ISBN 978-953-7044-06-0, Zadar, Croatia,

white pictures shot by camera — implementation and complexity reduction issue, *In Proceedings of the 18th International Conference Radioelektronika 2008*, pp. 219-222,

the snapshot gained by camera with CCD (CMOS) sensor, *In Proceedings of the 12th International Student Conference on Electrical Engineering POSTER 2008* [CD-ROM],

defocusing and thermal noise in black and white pictures shot by a camera with CCD/CMOS sensor, *6th International Conference on Photonics, Devices and Systems*, Vol. 7138, pp. 71381Z-1-71381Z-7, ISBN 978-0-8194-7379-0, Prague, Czech Republic,

of the thermal noise and blurring due to object moving in black and white pictures shot by a camera with CCD/CMOS sensor, *In Proceedings of SPIE Optics + Photonics*, Vol. 7076, pp. 70760M-1-70760M-9, ISBN 978-0-8194-7296-0, San Diego, USA, August

Technology, *IEEE International Carnahan Conference on Security Technology 2008*, pp.

and white pictures captured by a camera with CCD/CMOS sensor, *IEEE International Carnahan Conference on Security Technology 2008*, pp. 126-130, ISBN 978-1-4244-1816-9,

image microscopy. *M.H.F. Wilkinson, F. Schut (eds.), Digital Image Analysis of Microbes; Imaging, Morphometry, Fluorometry and Motility Techniques and Applications, Modern Microbiological Methods*, pp. 37-64, John Wiley & Sons, Chichester, The United

iterative detection with parametric uncertainty. *IEEE Trans. Commun.*, Vol. 48, pp.

Vandendorpe, L. (2003). Turbo synchronization : an EM algorithm interpretation,

0-470-84757-3.

September 2008.

August 2008.

Kingdom, 1998.

1638-1649, Oct. 2000.

2008.

Publishers, ISBN 0-792-37868-7.

Prague, Czech Republic, May 2008.

Prague, Czech Republic, October 2008.

**0**

**15**

Pål Ellingsen

*Norway*

*Department of Computer Science University College of Bergen*

**An Application of Digital Image Restoration**

Digital image restoration has been a field of very active research for many years, and digital image restoration techniques has been put to use in a lot of different contexts including astronomy, medicine, intelligence work and many others (Banham & Katsaggelos (1997)). Common to these fields of application is that the restoration techniques are applied to image data of some kind true to the original intentions of the algorithms. In this text we present an application of principles from digital image restoration to the field of coding theory, and the

Information can be represented in many different ways. A typical approach in information theory is to represent information as binary vectors, but there are many situations where information can rather be represented as a matrix or grid containing the information symbols giving rise to the concept of two-dimensional channels. Good examples of this can be found

When transmitting information of any kind, a central problem is how to deal with errors that result from the transmission process, and a solution to this problem is to add redundancy to the information in such a way that it is possible to detect and eventually also correct the errors that occur. Adding this redundancy is called error control coding, and the techniques for doing so is called error correcting or detecting codes. There exists a huge variety of error control coding techniques for channels with different characteristics and for fulfilling different sets of requirements. However, most of the channel coding techniques assumes the information that is to be encoded, are one-dimensional vectors or a stream of information symbols. Channel coding for two-dimensional channels on the other hand, is a part of coding

theory that has only recently attracted attention from the coding theory community.

control coding for a one-dimensional channel with an equal information rate.

There are different models for describing how errors occur in a two-dimensional

• The errors can be modeled as independent and identically distributed over the information symbols. In this case the problem of error control coding is reduced to the case of error

• The errors can be modeled by considering two-dimensional intersymbol interference, which is the effect of the information symbols interfering with their neighboring information symbols. This error model applies to many practical communication channels, most notably magnetic and optical storage media (which can be seen as

objects of application are not images but rather general information data.

in the fields of magnetic and optical storage, bar codes and others.

**1. Introduction**

communication channel.

**Techniques to Error Control Coding**

## **An Application of Digital Image Restoration Techniques to Error Control Coding**

## Pål Ellingsen

*Department of Computer Science University College of Bergen Norway*

## **1. Introduction**

Digital image restoration has been a field of very active research for many years, and digital image restoration techniques has been put to use in a lot of different contexts including astronomy, medicine, intelligence work and many others (Banham & Katsaggelos (1997)). Common to these fields of application is that the restoration techniques are applied to image data of some kind true to the original intentions of the algorithms. In this text we present an application of principles from digital image restoration to the field of coding theory, and the objects of application are not images but rather general information data.

Information can be represented in many different ways. A typical approach in information theory is to represent information as binary vectors, but there are many situations where information can rather be represented as a matrix or grid containing the information symbols giving rise to the concept of two-dimensional channels. Good examples of this can be found in the fields of magnetic and optical storage, bar codes and others.

When transmitting information of any kind, a central problem is how to deal with errors that result from the transmission process, and a solution to this problem is to add redundancy to the information in such a way that it is possible to detect and eventually also correct the errors that occur. Adding this redundancy is called error control coding, and the techniques for doing so is called error correcting or detecting codes. There exists a huge variety of error control coding techniques for channels with different characteristics and for fulfilling different sets of requirements. However, most of the channel coding techniques assumes the information that is to be encoded, are one-dimensional vectors or a stream of information symbols. Channel coding for two-dimensional channels on the other hand, is a part of coding theory that has only recently attracted attention from the coding theory community.

There are different models for describing how errors occur in a two-dimensional communication channel.


El-Gabali who developed the applied techniques in Shridhar et al. (1989), El-Gabali et al. (1987), El-Gabali et al. (1988) and El-Gabali et al. (1990) but similar techniques are also presented in Geman & Geman (1993), Zhang (1993), Jeng & Woods (1991) and Chalmond (1988). The basis for all of these image restoration techniques is that simulated annealing is used to produce an estimate of the maximum a posteriori probability of the original image and the techniques has been extensively used for different purposes within the field of image processing, including image restoration, image segmentation, object identification and texture analysis. Using a system model based on Markov random field theory, one wants to find the joint distribution of the variables representing the image (e.g. pixels) and then use this distribution as basis for detecting the original scene and eliminate or reduce noise in the image. However, finding the joint distribution directly from a Markov random field model is mathematically intractable, so one needs to compute the distribution by aid of the so-called Hammersley-Clifford theorem which states the equivalence between the joint distribution of the variables in a Markov random field and Gibbs distribution which can be treated mathematically in an efficient way. Given this distribution one common method for performing the actual detection of the original image is to find the maximum a priori

An Application of Digital Image Restoration Techniques to Error Control Coding 355

probability for the image given the observed output.

exploit this extra information given in the error model.

there is a significant gain in performance.

**3. General overview of the iterative decoding and detection process**

two-dimensional channels are subjected to different kinds of errors, but in this setting we are interested in sources of errors that will affect spatially limited parts of a two-dimensional codeword. This is called an error cluster or equivalently a burst error. Burst error correction is a well known and much studied problem, but none of the classical techniques in this field are able to take into account the fact that such spatially correlated error clusters gives rise to a statistical correlation on the error probability for neighboring positions in the codeword. Using the above mentioned techniques from digital image restoration is one way one can

Several different approaches are possible when trying to use the information gleaned from the image restoration to enhance the decoding process. However, our approach is based on the use of so-called soft input - soft output (SISO) decoding. The principle behind this decoding strategy is that the decoder should accept input values in the form of conditional probabilities as a measure of the reliability of the corresponding channel value, and as output produce a new measure of reliability for the corresponding channel value. Such a decoder can take advantage of the information produced by the restoration process in a very natural way by supplying conditional probabilities resulting from the estimation process described below. Based on results from our papers Ellingsen et al. (2004) Ellingsen & Kvamme (2010) we show how this principle can be used to implement an actual decoding system based on the techniques described above. We study two different channel models, the two-dimensional binary asymmetric channel and the two-dimensional binary symmetric channel, and use LDPC-codes for error correction. Then we show how the redundant information of the code can be used to provide prior information to an image restoration process, while the results from the image restoration process is used to assist the decoding process by providing a priori information to the decoder. Thus we construct an iterative decoding process where estimates in the form of conditional probabilities for each bit in the codeword is exchanged back and forth between the LDPC-decoder and the image restoration module. The results from this process are compared to the results when using the LDPC decoder alone and we see that

communication channels). This error model has been studied extensively, see for example Singla & O'Sullivan (2005) and Kurkoski (2008).

• The errors can be modeled as spatially contagious areas bounding a cluster of errors. The underlying channel model assumes that the information symbols are affected by some physical process that affects a limited part of the information. Error that result from such processes would form error bursts that may take the form of clusters or be concentrated to a limited area. Such error clusters are defined differently in the literature, but a very common approach is to define an error cluster as a rectangular area of a given size *n*<sup>1</sup> × *n*2. Code constructions for this type of cluster errors can be found in Farrell (1982) Schwartz & Etzion (2005) and Breitbach et al. (1998). More recently, error clusters of arbitrary form has been considered in several works, and most of these use interleaving strategies to correct cluster errors. This approach is used in Blaum et al. (1998), Schwartz & Etzion (2003) and Xu & Golomb (2007).

In the following we take the latter perspective on the nature of errors on a two-dimensional channel, and we apply techniques from digital image restoration to support the decoding process since these methods can exploit the information inherent in the two-dimensional cluster error model. The core element in this application of digital image restoration is looking at the encoded information as "noise" and the areas affected by burst errors as the "original image" that we want to restore. The strategy is to first encode the information at the source using ordinary error control coding. At the receiver, ordinary decoding and detection of the error clusters are combined in an iterative system where the decoding process produces an estimate of the probability of error in each information symbol, and this estimate is then used as input to a component that produces an estimate of the size and shape of the error burst based on image restoration techniques. The information that is extracted from this process is then used to support the decoding process as a priori information about the error clusters.

Error cluster estimate

Fig. 1. Basic principle of iterative process

#### **2. General overview over relevant image restoration techniques**

In our application we are concerned with describing the statistical properties of context dependent entities such as neighboring bits in a two-dimensional representation of digital information. One technique for describing such properties is the use of Markov random field theory which uses conditional probabilities to describe spatial dependencies in an n-dimensional system. This approach is based on the results of Shridhar, Ahmadi and 2 Image Restoration

• The errors can be modeled as spatially contagious areas bounding a cluster of errors. The underlying channel model assumes that the information symbols are affected by some physical process that affects a limited part of the information. Error that result from such processes would form error bursts that may take the form of clusters or be concentrated to a limited area. Such error clusters are defined differently in the literature, but a very common approach is to define an error cluster as a rectangular area of a given size *n*<sup>1</sup> × *n*2. Code constructions for this type of cluster errors can be found in Farrell (1982) Schwartz & Etzion (2005) and Breitbach et al. (1998). More recently, error clusters of arbitrary form has been considered in several works, and most of these use interleaving strategies to correct cluster errors. This approach is used in Blaum et al. (1998), Schwartz & Etzion (2003) and

In the following we take the latter perspective on the nature of errors on a two-dimensional channel, and we apply techniques from digital image restoration to support the decoding process since these methods can exploit the information inherent in the two-dimensional cluster error model. The core element in this application of digital image restoration is looking at the encoded information as "noise" and the areas affected by burst errors as the "original image" that we want to restore. The strategy is to first encode the information at the source using ordinary error control coding. At the receiver, ordinary decoding and detection of the error clusters are combined in an iterative system where the decoding process produces an estimate of the probability of error in each information symbol, and this estimate is then used as input to a component that produces an estimate of the size and shape of the error burst based on image restoration techniques. The information that is extracted from this process is then used to support the decoding process as a priori information about the error clusters.

Decoder Detector

Error cluster estimate

In our application we are concerned with describing the statistical properties of context dependent entities such as neighboring bits in a two-dimensional representation of digital information. One technique for describing such properties is the use of Markov random field theory which uses conditional probabilities to describe spatial dependencies in an n-dimensional system. This approach is based on the results of Shridhar, Ahmadi and

**2. General overview over relevant image restoration techniques**

Error probability estimate

Singla & O'Sullivan (2005) and Kurkoski (2008).

Xu & Golomb (2007).

Fig. 1. Basic principle of iterative process

communication channels). This error model has been studied extensively, see for example

El-Gabali who developed the applied techniques in Shridhar et al. (1989), El-Gabali et al. (1987), El-Gabali et al. (1988) and El-Gabali et al. (1990) but similar techniques are also presented in Geman & Geman (1993), Zhang (1993), Jeng & Woods (1991) and Chalmond (1988). The basis for all of these image restoration techniques is that simulated annealing is used to produce an estimate of the maximum a posteriori probability of the original image and the techniques has been extensively used for different purposes within the field of image processing, including image restoration, image segmentation, object identification and texture analysis. Using a system model based on Markov random field theory, one wants to find the joint distribution of the variables representing the image (e.g. pixels) and then use this distribution as basis for detecting the original scene and eliminate or reduce noise in the image. However, finding the joint distribution directly from a Markov random field model is mathematically intractable, so one needs to compute the distribution by aid of the so-called Hammersley-Clifford theorem which states the equivalence between the joint distribution of the variables in a Markov random field and Gibbs distribution which can be treated mathematically in an efficient way. Given this distribution one common method for performing the actual detection of the original image is to find the maximum a priori probability for the image given the observed output.

## **3. General overview of the iterative decoding and detection process**

two-dimensional channels are subjected to different kinds of errors, but in this setting we are interested in sources of errors that will affect spatially limited parts of a two-dimensional codeword. This is called an error cluster or equivalently a burst error. Burst error correction is a well known and much studied problem, but none of the classical techniques in this field are able to take into account the fact that such spatially correlated error clusters gives rise to a statistical correlation on the error probability for neighboring positions in the codeword. Using the above mentioned techniques from digital image restoration is one way one can exploit this extra information given in the error model.

Several different approaches are possible when trying to use the information gleaned from the image restoration to enhance the decoding process. However, our approach is based on the use of so-called soft input - soft output (SISO) decoding. The principle behind this decoding strategy is that the decoder should accept input values in the form of conditional probabilities as a measure of the reliability of the corresponding channel value, and as output produce a new measure of reliability for the corresponding channel value. Such a decoder can take advantage of the information produced by the restoration process in a very natural way by supplying conditional probabilities resulting from the estimation process described below.

Based on results from our papers Ellingsen et al. (2004) Ellingsen & Kvamme (2010) we show how this principle can be used to implement an actual decoding system based on the techniques described above. We study two different channel models, the two-dimensional binary asymmetric channel and the two-dimensional binary symmetric channel, and use LDPC-codes for error correction. Then we show how the redundant information of the code can be used to provide prior information to an image restoration process, while the results from the image restoration process is used to assist the decoding process by providing a priori information to the decoder. Thus we construct an iterative decoding process where estimates in the form of conditional probabilities for each bit in the codeword is exchanged back and forth between the LDPC-decoder and the image restoration module. The results from this process are compared to the results when using the LDPC decoder alone and we see that there is a significant gain in performance.

the center node *ai* and all pairs of (*ai*, *aj*) where *aj* is a neighbor of *ai*. In a second order

An Application of Digital Image Restoration Techniques to Error Control Coding 357

✤ ✤ ✤

❅ � � ❴ ❴ ❴

❅ ❅❅❅ ❅❅ ❅❅

> ✤ ✤ ✤

❅❅❅❅❅❅❅❅ �

�

�

� ❴ ❴ ❴

❅ ❅❅❅❅❅❅

�

✤ ✤ ✤

*ai ai* �

The collection of all cliques of size *i* in a neighborhood system N is called C*i*. The set C of all

Now, based on the concept of neighborhoods we can then proceed to define a Markov Random

*<sup>P</sup>*(*ai*|*ai*−1, *ai*−2,...) = *<sup>P</sup>*(*ai*|*ai*−1, *ai*−2,..., *ai*−*n*)

*<sup>P</sup>*(*ai*|*aI*−{*i*}) = *<sup>P</sup>*(*ai* | N*i*)

The fact that the errors of our channel can be represented by an MRF does not immediately enable us to analyze the error patterns statistically. By assuming that the dependencies in a collection of random variables can be represented by an MRF, the joint probability of the

where *I* is the set of indices of *a* and N*<sup>i</sup>* is the neighborhood of *ai* as defined above.

*ai* � *ai*

�

⑦⑦⑦⑦⑦⑦⑦⑦

✤ ✤ ✤

⑦⑦⑦⑦⑦⑦⑦⑦ � ❴ ❴ ❴

neighborhood, all nodes within distance <sup>√</sup>2 are defined as neighbors:

✤ ✤ ✤

❴ ❴ ❴ � *ai*

✤ ✤ ✤

❄❄❄❄❄❄❄

cliques in a graph can then be partitioned into the subsets C*<sup>i</sup>* for 1 ≤ *i* ≤ *n*

Field. Just as a Markov chain {..., *ak*, *ak*−1, *ak*−2,...} satisfies

for some *n*, a Markov Random Field should satisfy

variables is given by the so called *Gibbs distribution*.

**4.3 Probability distribution**

❴ ❴ ❴ �

❴ ❴ ❴ �

and in this case the cliques becomes any configuration of

#### **4. Details of restoration technique**

#### **4.1 Modelling a two-dimensional channel using Markov Random Fields**

A channel with memory is characterized by the existence of dependencies in the noise generating process. Such dependencies can e.g. be described by a Markov chain in the case of one-dimensional channels as is the case for the Gilbert-Elliott channel Gilbert et al. (1960). This implies that the channel will have characteristics that are varying with time. We want to extend this line of thinking to the case of two-dimensional channels and look at spatial dependencies in the noise generating process rather than temporal dependencies as in the one-dimensional case. Such spatial dependencies can be modeled using a Markov Random Field (MRF).

An MRF can be seen as a generalization of Markov chains, but while a Markov chain is often defined over a domain of time as a sequence of random variables, an MRF can be defined in space to describe dependencies between variables on a grid of dimension 2 or higher.

#### **4.2 Markov Random Fields**

Consider a set of random variables *A* = {*Ai*|*i* ∈ *I*} for some index set *I*, where the variables are organized in a two dimensional grid. Let the variables correspond to the vertices and the statistical dependencies between the variables correspond to edges in an undirected graph *G*. We shall use this setup to model both codewords and errors in our system. Two connected vertices in *G* are said to be *neighbors*, and a *neighborhood* N*<sup>i</sup>* of a vertex *ai* can be defined as the set of vertices that are connected to *ai* in *G*. Different sizes of neighborhoods can be defined for an MRF. By convention, a node is not a neighbor of itself. On a regular lattice we define the first order neighborhood to be the four closest neighbors of a node as seen below, the second order neighborhood as the eight closest neighbors and so on. The collection of all neighborhoods N = {N*<sup>i</sup>* | ∀*i* ∈ *I*} in a graph, is called a *neighborhood system*.

Within the neighborhood of a vertex *ai*, we define a *clique* to be any collection of vertices that contains *ai* and forms a fully connected subgraph of *G*, i. e. that the vertices are mutual neighbors relative to the neighborhood system N . In the case of a first order neighborhood, all nodes within distance 1 of the center are said to be neighbors, and the cliques become 4 Image Restoration

A channel with memory is characterized by the existence of dependencies in the noise generating process. Such dependencies can e.g. be described by a Markov chain in the case of one-dimensional channels as is the case for the Gilbert-Elliott channel Gilbert et al. (1960). This implies that the channel will have characteristics that are varying with time. We want to extend this line of thinking to the case of two-dimensional channels and look at spatial dependencies in the noise generating process rather than temporal dependencies as in the one-dimensional case. Such spatial dependencies can be modeled using a Markov Random

An MRF can be seen as a generalization of Markov chains, but while a Markov chain is often defined over a domain of time as a sequence of random variables, an MRF can be defined in

Consider a set of random variables *A* = {*Ai*|*i* ∈ *I*} for some index set *I*, where the variables are organized in a two dimensional grid. Let the variables correspond to the vertices and the statistical dependencies between the variables correspond to edges in an undirected graph *G*. We shall use this setup to model both codewords and errors in our system. Two connected vertices in *G* are said to be *neighbors*, and a *neighborhood* N*<sup>i</sup>* of a vertex *ai* can be defined as the set of vertices that are connected to *ai* in *G*. Different sizes of neighborhoods can be defined for an MRF. By convention, a node is not a neighbor of itself. On a regular lattice we define the first order neighborhood to be the four closest neighbors of a node as seen below, the second order neighborhood as the eight closest neighbors and so on. The collection of all

> ✤ ✤ ✤

✤ ✤ ✤

✤ ✤ ✤

✤ ✤ ✤

✤ ✤ ✤

� ❴ ❴ ❴

� ❴ ❴ ❴ � ❴ ❴ ❴

*ai* � ❴ ❴ ❴

� ❴ ❴ ❴

✤ ✤ ✤

Within the neighborhood of a vertex *ai*, we define a *clique* to be any collection of vertices that contains *ai* and forms a fully connected subgraph of *G*, i. e. that the vertices are mutual neighbors relative to the neighborhood system N . In the case of a first order neighborhood, all nodes within distance 1 of the center are said to be neighbors, and the cliques become

space to describe dependencies between variables on a grid of dimension 2 or higher.

neighborhoods N = {N*<sup>i</sup>* | ∀*i* ∈ *I*} in a graph, is called a *neighborhood system*.

✤ ✤ ✤

❴ ❴ ❴ � ❴ ❴ ❴

✤ ✤ ✤

✤ ✤ ✤

❴ ❴ ❴ � ❴ ❴ ❴

✤ ✤ ✤

❴ ❴ ❴ �

**4.1 Modelling a two-dimensional channel using Markov Random Fields**

**4. Details of restoration technique**

Field (MRF).

**4.2 Markov Random Fields**

the center node *ai* and all pairs of (*ai*, *aj*) where *aj* is a neighbor of *ai*. In a second order neighborhood, all nodes within distance <sup>√</sup>2 are defined as neighbors:

and in this case the cliques becomes any configuration of

The collection of all cliques of size *i* in a neighborhood system N is called C*i*. The set C of all cliques in a graph can then be partitioned into the subsets C*<sup>i</sup>* for 1 ≤ *i* ≤ *n*

Now, based on the concept of neighborhoods we can then proceed to define a Markov Random Field. Just as a Markov chain {..., *ak*, *ak*−1, *ak*−2,...} satisfies

$$P(a\_i|a\_{i-1}, a\_{i-2}, \dots) = P(a\_i|a\_{i-1}, a\_{i-2}, \dots, a\_{i-n})$$

for some *n*, a Markov Random Field should satisfy

$$P(a\_i | a\_{I - \{i\}}) = P(a\_i \mid \mathcal{N}\_i).$$

where *I* is the set of indices of *a* and N*<sup>i</sup>* is the neighborhood of *ai* as defined above.

#### **4.3 Probability distribution**

The fact that the errors of our channel can be represented by an MRF does not immediately enable us to analyze the error patterns statistically. By assuming that the dependencies in a collection of random variables can be represented by an MRF, the joint probability of the variables is given by the so called *Gibbs distribution*.

MAP estimation of the error pattern *X* based on the received word *Y* can be formulated as the optimization of the *a posteriori* probability *P*(*X* = *x*|*Y* = *y*) with respect to *x*. That is, we want to find a configuration *x* that makes the probability *P*(*X* = *x*|*Y* = *y*) as high as possible.

An Application of Digital Image Restoration Techniques to Error Control Coding 359

*<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*|*<sup>Y</sup>* <sup>=</sup> *<sup>y</sup>*) = *<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*)*P*(*<sup>Y</sup>* <sup>=</sup> *<sup>y</sup>*|*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*)

To find the probabilities *P*(*Y* = *y*|*X* = *x*), we must take care to remember that the error pattern *X* is now considered as the original information that we want to estimate, and the codeword *C* is to be considered as errors obscuring the information. In the following, we shall make some

• The variables are bipolar, with 1 corresponding to 0 and −1 corresponding to 1 in the

• The codeword *C*, when treated as errors, can be seen as random bipolar variables so that

*i*

The conditional probabilities must then be expressed by by using the characteristics of the channel and this expression must then be optimized with respect to the input configuration. In chapter 5.3 and 6.3 we show two examples of how this optimization can be done for a given channel. In both of our cases, finding a global maximum of the conditional probabilities with respect to *x* would become computationally infeasible as the size of *x* increases. Instead, we use the local dependencies between bits to do a local optimization along the lines of the PDFE in Neifeld & King (1998); Neifield et al. (1996) or the partial binary segmentation algorithm of

Generation of two dimensional burst errors for simulation purposes is done by the use of a Monte Carlo Markov chain technique called the Metropolis algorithm. We do not have very strict requirements for the generated sample configurations, other than that they should be "somewhat likely" to occur given the condition that the variables' distribution is given by the

The Metropolis algorithm is a general method for generating samples from a joint distribution of two or more variables, and can be applied to distributions that are either continuous or discrete as long as it is possible to compute the difference of the likelihoods for two

We would like to sample the joint distribution *A* = {*A*1,..., *An*}. This is achieved by generating random changes to the components *Ai* of *A*, and accepting or rejecting these changes based on how they affect the likelihood of the configuration. In our case, the natural

change to a component of a configuration would be to flip the bit value.

*<sup>P</sup>*(*ci*)=( <sup>1</sup>

2 )*n*

*P*(*C* = *c*) = ∏

Since *P*(*Y* = *y*) does not depend on *P*(*X* = *x*), we can maximize over

*P*(*Y* = *y*)

*P*(*X* = *x*)*P*(*Y* = *y*|*X* = *x*) (4)

Bayes rule gives us

assumptions about *X* and *C*.

channel model.

Shridhar et al. (1989).

**4.5 Error generation**

Gibbs distribution.

configurations of the variables.

**Definition 1** (Gibbs distribution)**.** *A set of random variables is said to be a* Gibbs random field *(GRF) if the joint distribution of the variables takes the following form:*

$$P(X=\mathbf{x}) = \frac{1}{Z} \exp\left[-\frac{1}{T} \mathcal{U}(X)\right] \tag{1}$$

*This distribution is called a* Gibbs distribution*.*


$$\mathcal{U}(\mathbf{x}) = \sum\_{\mathbf{c} \in \mathcal{C}} V\_{\mathbf{c}}(\mathbf{x}) \tag{2}$$

We can expand (2) further by summing over the cliques of the same degree separately

$$\sum\_{\mathbf{c}\in\mathcal{C}} V\_{\mathbf{c}}(\mathbf{x}) = \sum\_{a\in\mathcal{C}\_{1}} V\_{1}(a) + \sum\_{a,b\in\mathcal{C}\_{2}} V\_{2}(a,b) + \sum\_{a,b,c\in\mathcal{C}\_{3}} V\_{3}(a,b,c) + \dots \tag{3}$$

where C*<sup>i</sup>* is the collection of all cliques of degree *i*, so that *Vi* is a function of *i* variables forming a clique, and <sup>∑</sup>C*<sup>i</sup> Vi* mean that we sum over all possible cliques in the field of degree *i*.

• *T* is called the *temperature* (this is a legacy from the distribution's origin in statistical physics). The parameter *T* influences the degree of cohesion between the variables on a grid, so that a higher temperature corresponds to a lower degree of cohesion in the sense that the values of the variables becomes more and more independent, while a lower temperature gives a higher probability of the formation of large clusters of variables with the same value. We shall assume that the temperature is 1 in our simulations, even if the parameter will be used in the theoretical treatment of the decoding algorithm.

The Clifford-Hammersley theorem states that for a set of variables F with a neighborhood system N , F is an MRF with respect to N if and only if F is a GRF with respect to N . See Kindermann & Snell (1980).

Unfortunately, *Z* is very hard to compute. Since we have to consider all possible of values of *x* in order to find *Z*, the computational complexity of the task is a formidable *O*(2*n*), effectively preventing us from computing the absolute probabilities for the configurations of *X*. It is nevertheless possible to use the Gibbs distribution to find an estimate of the error patterns generated by the channel.

#### **4.4 MAP estimation**

We want to find an estimate of the error pattern that was added to the codeword, based on the assumptions about the dependence between errors given in the previous sections. In order to avoid computing the constant *Z* in the Gibbs distribution, we will do a MAP estimation of the errors. That is, given a received word *Y*, we want to find an estimate of the most likely error pattern *X* that was added to *C*. Some terminology is needed in order to develop this.

Let *A* be a set of random variables defined on the set L, and let the elements of *A* be indexed by 1 ≤ *i* ≤ *n*. If *Ai* = *ai* for each variable *Ai*, where *ai* ∈ L, we call {*a*1,..., *an*} = *a* a *configuration* of *A*

MAP estimation of the error pattern *X* based on the received word *Y* can be formulated as the optimization of the *a posteriori* probability *P*(*X* = *x*|*Y* = *y*) with respect to *x*. That is, we want to find a configuration *x* that makes the probability *P*(*X* = *x*|*Y* = *y*) as high as possible.

Bayes rule gives us

6 Image Restoration

**Definition 1** (Gibbs distribution)**.** *A set of random variables is said to be a* Gibbs random field

*<sup>Z</sup>* exp

• *U*(*x*) is called *the energy function* and is a function of the values of the variables forming

*U*(*x*) = ∑

We can expand (2) further by summing over the cliques of the same degree separately

*a*,*b*∈C<sup>2</sup>

*V*1(*a*) + ∑

*c*∈C

where C*<sup>i</sup>* is the collection of all cliques of degree *i*, so that *Vi* is a function of *i* variables forming a clique, and <sup>∑</sup>C*<sup>i</sup> Vi* mean that we sum over all possible cliques in the field of

• *T* is called the *temperature* (this is a legacy from the distribution's origin in statistical physics). The parameter *T* influences the degree of cohesion between the variables on a grid, so that a higher temperature corresponds to a lower degree of cohesion in the sense that the values of the variables becomes more and more independent, while a lower temperature gives a higher probability of the formation of large clusters of variables with the same value. We shall assume that the temperature is 1 in our simulations, even if the

The Clifford-Hammersley theorem states that for a set of variables F with a neighborhood system N , F is an MRF with respect to N if and only if F is a GRF with respect to N . See

Unfortunately, *Z* is very hard to compute. Since we have to consider all possible of values of *x* in order to find *Z*, the computational complexity of the task is a formidable *O*(2*n*), effectively preventing us from computing the absolute probabilities for the configurations of *X*. It is nevertheless possible to use the Gibbs distribution to find an estimate of the error patterns

We want to find an estimate of the error pattern that was added to the codeword, based on the assumptions about the dependence between errors given in the previous sections. In order to avoid computing the constant *Z* in the Gibbs distribution, we will do a MAP estimation of the errors. That is, given a received word *Y*, we want to find an estimate of the most likely error

Let *A* be a set of random variables defined on the set L, and let the elements of *A* be indexed by 1 ≤ *i* ≤ *n*. If *Ai* = *ai* for each variable *Ai*, where *ai* ∈ L, we call {*a*1,..., *an*} = *a* a

pattern *X* that was added to *C*. Some terminology is needed in order to develop this.

parameter will be used in the theoretical treatment of the decoding algorithm.

*V*2(*a*, *b*) + ∑

*a*,*b*,*c*∈C<sup>3</sup>

• *<sup>Z</sup>* is a constant called *the partition function* and can be expressed as *<sup>Z</sup>* <sup>=</sup> <sup>∑</sup>*x*∈**<sup>X</sup>** *<sup>e</sup>*<sup>−</sup> <sup>1</sup>

 − 1 *<sup>T</sup> <sup>U</sup>*(*X*)

*Vc*(*x*) (2)

*V*3(*a*, *b*, *c*) + . . . (3)

(1)

*<sup>T</sup> U*(*x*) , so

*(GRF) if the joint distribution of the variables takes the following form:*

that *Z*−<sup>1</sup> becomes a normalizing constant in the expression.

*This distribution is called a* Gibbs distribution*.*

cliques in the field. It can be written as

*Vc* (*x*) = ∑

*a*∈C<sup>1</sup>

∑ *c*∈C

degree *i*.

Kindermann & Snell (1980).

generated by the channel.

**4.4 MAP estimation**

*configuration* of *A*

*<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*) = <sup>1</sup>

$$P(X=\mathfrak{x}|Y=y) = \frac{P(X=\mathfrak{x})P(Y=y|X=\mathfrak{x})}{P(Y=y)}$$

Since *P*(*Y* = *y*) does not depend on *P*(*X* = *x*), we can maximize over

$$P(X=x)P(Y=y|X=x)\tag{4}$$

To find the probabilities *P*(*Y* = *y*|*X* = *x*), we must take care to remember that the error pattern *X* is now considered as the original information that we want to estimate, and the codeword *C* is to be considered as errors obscuring the information. In the following, we shall make some assumptions about *X* and *C*.


$$P(\mathsf{C} = c) = \prod\_{i} P(c\_i) = (\frac{1}{2})^n$$

The conditional probabilities must then be expressed by by using the characteristics of the channel and this expression must then be optimized with respect to the input configuration. In chapter 5.3 and 6.3 we show two examples of how this optimization can be done for a given channel. In both of our cases, finding a global maximum of the conditional probabilities with respect to *x* would become computationally infeasible as the size of *x* increases. Instead, we use the local dependencies between bits to do a local optimization along the lines of the PDFE in Neifeld & King (1998); Neifield et al. (1996) or the partial binary segmentation algorithm of Shridhar et al. (1989).

#### **4.5 Error generation**

Generation of two dimensional burst errors for simulation purposes is done by the use of a Monte Carlo Markov chain technique called the Metropolis algorithm. We do not have very strict requirements for the generated sample configurations, other than that they should be "somewhat likely" to occur given the condition that the variables' distribution is given by the Gibbs distribution.

The Metropolis algorithm is a general method for generating samples from a joint distribution of two or more variables, and can be applied to distributions that are either continuous or discrete as long as it is possible to compute the difference of the likelihoods for two configurations of the variables.

We would like to sample the joint distribution *A* = {*A*1,..., *An*}. This is achieved by generating random changes to the components *Ai* of *A*, and accepting or rejecting these changes based on how they affect the likelihood of the configuration. In our case, the natural change to a component of a configuration would be to flip the bit value.

likelihood values *L* from the SISO are then sent to the detector. Based on these values, the detector produces an estimate of the error pattern that was multiplied with the codeword. This information is subsequently fed back into the SISO in the next iteration. The decoding process continues until either the SISO finds a valid codeword, or a set of stopping criteria is

An Application of Digital Image Restoration Techniques to Error Control Coding 361

C

reached. The final estimate of the codeword is produced by the SISO. See figure 2.

X

Noise gen.

SISO

Y

Fig. 2. System model

**5.3 MAP estimation**

given in Table 1.

Table 1. Transition probabilities

Decoder

*C*ˆ

To optimize the maximum a priori probability, we need to find some expression for the conditional probability *P*(*Y* = *y*|*X* = *x*) as noted in chapter 4.4. Based on the assumptions in chapter 4.4, the conditional probabilities *P*(*Yi* = *yi*|*Xi* = *xi*) for each information symbol is

The conditional probabilities in the table can be expressed as an exponential function by

→0 1 <sup>2</sup> exp

We can then express the probability of a given *y* conditioned on a configuration *x* by

*i* lim →0 1 <sup>2</sup> exp

−*yi*(1 − *xi*)ln 2 1 − *yi* +

−*yi*(1 − *xi*)ln 2 1 − *yi* +

(5)

*P*(*Yi* = *yi*|*Xi* = *xi*) = lim

*<sup>P</sup>*(*<sup>Y</sup>* = *<sup>y</sup>*|*<sup>X</sup>* = *<sup>x</sup>*) = ∏

*X*ˆ

L

Encoder I

Detector

Given an initial configuration *A*, a new configuration *A*∗ is obtained as explained above by flipping a bit. Then, the difference of the likelihood of the new configuration and the old configuration is calculated by

$$\begin{aligned} \Delta \mathcal{U} = \mathcal{U}(A^\*) - \mathcal{U}(A) &= \sum\_{a^\* \in \mathcal{C}\_1} V\_1(a^\*) + \sum\_{a^\*, b^\* \in \mathcal{C}\_2} V\_2(a^\*, b^\*) + \sum\_{a^\*, b^\*, c^\* \in \mathcal{C}\_3} V\_3(a^\*, b^\*, c^\*) + \dots \\ &- \sum\_{a \in \mathcal{C}\_1} V\_1(a) + \sum\_{a, b \in \mathcal{C}\_2} V\_2(a, b) + \sum\_{a, b, c \in \mathcal{C}\_3} V\_3(a, b, c) + \dots \end{aligned}$$

and the new configuration is accepted with probability 1 if the new likelihood is higher than the old one. Otherwise, the new configuration is accepted with probability *e*−Δ*U*/*T*, so the probability of accepting the new configuration becomes:

$$P(A \to A^\*) = \begin{cases} 1 & \Delta U \ge 0\\ e^{-\Delta U/T} & \Delta U < 0 \end{cases}$$

A pass through all the components in *A* in this way is called a *sweep* over the variables in *A*. In our case, we generate a sample from the distribution by doing 4 sweeps over *A*, resulting in the evaluation of a total of 4*n* new configurations. This should result in a sample that has high enough probability to be detected by the estimation algorithm described above.

#### **5. Application to the two-dimensional binary asymmetric channel**

#### **5.1 Channel model**

In this section, we will apply this general method for cluster error detection and correction to the binary asymmetric channel As the name of the channel implies, we will assume that errors are asymmetric so that only the transition −1 → 1 occurs in a received codeword.

**Definition 2** (Matrix OR)**.** *Assume A and B are matrices with dimensions d*<sup>1</sup> × *d*<sup>2</sup> *where d*<sup>1</sup> · *d*<sup>2</sup> = *n, and with coordinates ai and bi respectively. Then the OR of these matrices is defined as*

$$A \lor B \stackrel{\Delta}{=} a\_{\dot{i}} \lor b\_{\dot{i}\nu} \qquad 1 \le \dot{i} \le n$$

The received word *Y* can then be defined as the combination of

$$\mathcal{Y} = \mathsf{C} \vee \mathsf{X}$$

*Y* - received word

*C* - original codeword

*X* - error pattern

∨ - OR operator on matrices as defined above

#### **5.2 System model**

Information *I* is encoded, producing a codeword *C*. For each generated codeword *C*, the channel induces two-dimensionally correlated noise *X* and the resulting word *Y* is passed to the decoder. Information is converted to likelihood ratios and sent to the SISO. The output likelihood values *L* from the SISO are then sent to the detector. Based on these values, the detector produces an estimate of the error pattern that was multiplied with the codeword. This information is subsequently fed back into the SISO in the next iteration. The decoding process continues until either the SISO finds a valid codeword, or a set of stopping criteria is reached. The final estimate of the codeword is produced by the SISO. See figure 2.

Fig. 2. System model

8 Image Restoration

Given an initial configuration *A*, a new configuration *A*∗ is obtained as explained above by flipping a bit. Then, the difference of the likelihood of the new configuration and the old

and the new configuration is accepted with probability 1 if the new likelihood is higher than the old one. Otherwise, the new configuration is accepted with probability *e*−Δ*U*/*T*, so the

A pass through all the components in *A* in this way is called a *sweep* over the variables in *A*. In our case, we generate a sample from the distribution by doing 4 sweeps over *A*, resulting in the evaluation of a total of 4*n* new configurations. This should result in a sample that has

In this section, we will apply this general method for cluster error detection and correction to the binary asymmetric channel As the name of the channel implies, we will assume that errors

**Definition 2** (Matrix OR)**.** *Assume A and B are matrices with dimensions d*<sup>1</sup> × *d*<sup>2</sup> *where d*<sup>1</sup> · *d*<sup>2</sup> = *n,*

*A* ∨ *B ai* ∨ *bi*, 1 ≤ *i* ≤ *n*

*Y* = *C* ∨ *X*

Information *I* is encoded, producing a codeword *C*. For each generated codeword *C*, the channel induces two-dimensionally correlated noise *X* and the resulting word *Y* is passed to the decoder. Information is converted to likelihood ratios and sent to the SISO. The output

high enough probability to be detected by the estimation algorithm described above.

are asymmetric so that only the transition −1 → 1 occurs in a received codeword.

*and with coordinates ai and bi respectively. Then the OR of these matrices is defined as*

The received word *Y* can then be defined as the combination of

*V*1(*a*) + ∑

*<sup>V</sup>*2(*a*∗, *<sup>b</sup>*∗) + <sup>∑</sup> *<sup>a</sup>*∗,*b*∗,*c*∗∈C<sup>3</sup>

*V*2(*a*, *b*) + ∑

*a*,*b*,*c*∈C<sup>3</sup>

*a*,*b*∈C<sup>2</sup>

 <sup>1</sup> <sup>Δ</sup>*<sup>U</sup>* <sup>≥</sup> <sup>0</sup> *e*−Δ*U*/*<sup>T</sup>* Δ*U* < 0 *V*3(*a*∗, *b*∗, *c*∗) + ...

*V*3(*a*, *b*, *c*) + ...

*<sup>V</sup>*1(*a*∗) + <sup>∑</sup>*<sup>a</sup>*∗,*b*∗∈C<sup>2</sup>

− ∑ *a*∈C<sup>1</sup>

*P*(*A* → *A*∗) =

**5. Application to the two-dimensional binary asymmetric channel**

configuration is calculated by

<sup>Δ</sup>*<sup>U</sup>* = *<sup>U</sup>*(*A*∗) − *<sup>U</sup>*(*A*) = ∑

**5.1 Channel model**

*Y* - received word *C* - original codeword *X* - error pattern

**5.2 System model**

∨ - OR operator on matrices as defined above

*a*∗∈C<sup>1</sup>

probability of accepting the new configuration becomes:

#### **5.3 MAP estimation**

To optimize the maximum a priori probability, we need to find some expression for the conditional probability *P*(*Y* = *y*|*X* = *x*) as noted in chapter 4.4. Based on the assumptions in chapter 4.4, the conditional probabilities *P*(*Yi* = *yi*|*Xi* = *xi*) for each information symbol is given in Table 1.


Table 1. Transition probabilities

The conditional probabilities in the table can be expressed as an exponential function by

$$P(Y\_i = y\_i | X\_i = x\_i) = \lim\_{\varepsilon \to 0} \frac{1}{2} \exp\left[\frac{-y\_i(1-x\_i)\ln 2}{1-y\_i+\varepsilon}\right]$$

We can then express the probability of a given *y* conditioned on a configuration *x* by

$$P(Y=y|X=x) = \prod\_{i} \lim\_{\epsilon \to 0} \frac{1}{2} \exp\left[\frac{-y\_i(1-x\_i)\ln 2}{1-y\_i+\epsilon}\right] \tag{5}$$

of the error pattern, it can be used to find likelihood ratios for input to the decoder. For each

An Application of Digital Image Restoration Techniques to Error Control Coding 363

*Li* <sup>=</sup> *<sup>P</sup>*(*Ci* <sup>=</sup> <sup>−</sup>1|*Yi*, *<sup>X</sup>*ˆ*i*) *<sup>P</sup>*(*Ci* <sup>=</sup> <sup>1</sup>|*Yi*, *<sup>X</sup>*ˆ*i*)

The resulting probabilities can be seen in Table 2. In the table, *ρ* is the probability that a bit

belonging to the error pattern is incorrectly estimated as a 1-bit. The parameter *ρ* must be estimated by simulation, but should in general be small, indicating a relatively certain −1-bit.

The performance of the estimation algorithm depends heavily on the value of *β*, which determines the degree of clustering in the error pattern. A critical performance parameter is the probability *ε* that not all bits in the error pattern are detected by the estimation algorithm. A bit that belongs to the error pattern, but is not detected as such, is given a high probability of being correct, and can hence be the source of errors that are hard to correct. Therefore, *ε* is an important measure of the reliability of the algorithm. As can be seen in Fig. 3, *P*(*ε*) is high for *β* < 0.5, reflecting the fact that small and very irregular error clusters appear in this range. *P*(*ε*) drops sharply initially, but levels out when *β* > 1 as a result of the clusters becoming bigger and more coherent. As we shall see later, this is also reflected in the performance of the

0 0.5 1 1.5 2

β

bit, we set the likelihood ratio to

Table 2. Input probabilities to the decoder

**5.4 Performance of estimation algorithm**

10-3

Fig. 3. *P*(*ε*) for fixed *β* = 0.2 in the estimation algorithm.

10-2

P(ε)

10-1

100

algorithm.

Substituting (1) and (5) into (4), we can find the joint probability by

$$P(X=x, Y=y) = \left[ Z^{-1} e^{-\frac{1}{\tau}II(x)} \right] \prod\_{i} \lim\_{\varepsilon \to 0} \frac{1}{2} \exp\left[ \frac{-y\_i(1-x\_i)\ln 2}{1-y\_i+\varepsilon} \right] \tag{6}$$

Since the natural logarithm is strictly increasing, the following equality holds:

$$\arg\max\_{X}(P(X,Y)) = \arg\max\_{X}(\ln(P(X,Y)))\tag{7}$$

In order to avoid computing *Z* in (6), we take the logarithm of both sides and eliminate constants to get

$$V(\mathbf{x}) = \mathcal{U}(\mathbf{x}) + \sum\_{i} \lim\_{\varepsilon \to 0} \left[ \frac{-y\_i (1 - x\_i) \ln 2}{1 - y\_i + \varepsilon} \right],$$

where *V*(*x*) = ln [*P*(*X* = *x*,*Y* = *y*)].

We define the partial functions *Vi* of *U*(*x*) according to Li (2000); Shridhar et al. (1989)

$$V\_1(\mathbf{x}\_{i}) = \alpha \mathbf{x}\_{i}$$

$$V\_2(\mathbf{x}\_{i\prime}\mathbf{x}\_{i\prime}) = \beta\_{i,i\prime}\mathbf{x}\_{i}\mathbf{x}\_{i\prime}$$

$$V\_3(\mathbf{x}\_{i\prime}\mathbf{x}\_{i\prime\prime}\mathbf{x}\_{i\prime\prime}) = \dots = 0$$

Note that the expression for *V*<sup>2</sup> implies that *V*2(*xi*, *xi*�) = *βi*,*i*� for *xi* = *xi*�and *V*2(*xi*, *xi*�) = −*βi*,*i*�for *xi* �= *xi*� .

From this we get a new expression for *V*(*x*):

$$V(\mathbf{x}) = \sum\_{\bar{i}} \left[ \alpha \mathbf{x}\_{\bar{i}} + \beta\_{\bar{i},\bar{i}'} \sum\_{\bar{i}' \in \mathcal{N}\_{\bar{i}}} \mathbf{x}\_{\bar{i}} \mathbf{x}\_{\bar{i}'} + \lim\_{\varepsilon \to 0} \left[ \frac{-y\_{\bar{i}}(1-\mathbf{x}\_{\bar{i}}) \ln 2}{1-y\_{\bar{i}}+\varepsilon} \right] \right], \quad \bar{i}$$

and splitting the last term into a constant and a non-constant term yields

$$V(\mathbf{x}) = \sum\_{i} \left[ a\mathbf{x}\_{i} + \beta\_{i,\bar{i}'} \sum\_{\bar{i}' \in \mathcal{N}\_{l}} \mathbf{x}\_{i}\mathbf{x}\_{\bar{i}'} + \lim\_{\varepsilon \to 0} \left[ \frac{\mathbf{x}\_{i} y\_{i} \ln 2}{1 - y\_{i} + \varepsilon} \right] + \lim\_{\varepsilon \to 0} \left[ \frac{-y\_{i} \ln 2}{1 - y\_{i} + \varepsilon} \right] \right].$$

Since the last term only depends on *y*, we can find the MAP configuration by simplifying the expression to:

$$\mathcal{V}(\mathbf{x}) = \sum\_{i} \left[ \alpha \mathbf{x}\_{i} + \beta \sum\_{i' \in \mathcal{N}\_{i}} \mathbf{x}\_{i} \mathbf{x}\_{i'} + \lim\_{\epsilon \to 0} \left[ \frac{\mathbf{x}\_{i} y\_{i} \ln 2}{1 - y\_{i} + \epsilon} \right] \right]$$

$$= \sum\_{i} \left[ \alpha + \beta \sum\_{i' \in \mathcal{N}\_{i}} \mathbf{x}\_{i'} + \lim\_{\epsilon \to 0} \left[ \frac{y\_{i} \ln 2}{1 - y\_{i} + \epsilon} \right] \right] \mathbf{x}\_{i} \tag{8}$$

Having obtained an estimate

$$\hat{X} = \{\hat{X}\_1, \dots, \hat{X}\_{\hat{\nu}}, \dots, \hat{X}\_n\}.$$

of the error pattern, it can be used to find likelihood ratios for input to the decoder. For each bit, we set the likelihood ratio to

$$L\_i = \frac{P(\mathbf{C}\_i = -1 | Y\_{i\nu} \vec{X}\_i)}{P(\mathbf{C}\_i = 1 | Y\_{i\nu} \hat{X}\_i)}$$

The resulting probabilities can be seen in Table 2. In the table, *ρ* is the probability that a bit

Table 2. Input probabilities to the decoder

10 Image Restoration

In order to avoid computing *Z* in (6), we take the logarithm of both sides and eliminate

*V*1(*xi*) = *αxi*

*V*2(*xi*, *xi*�) = *βi*,*i*� *xixi*�

*xixi*� + lim *�*→0  *xiyi* ln 2 1 − *yi* + *�*

*xixi*� + lim *�*→0

*xi*� + lim *�*→0

*<sup>X</sup>*<sup>ˆ</sup> <sup>=</sup> {*X*<sup>ˆ</sup> 1,..., *<sup>X</sup>*ˆ*i*,..., *<sup>X</sup>*<sup>ˆ</sup> *<sup>n</sup>*}

−*yi*(1 − *xi*)ln 2 1 − *yi* + *�*

> + lim *�*→0

 *xiyi* ln 2 1 − *yi* + *�*

 *yi* ln 2 1 − *yi* + *�*  ,

 <sup>−</sup>*yi* ln 2 1 − *yi* + *�*

 .

*xi* (8)

*V*3(*xi*, *xi*� , *xi*��) = ··· = 0 Note that the expression for *V*<sup>2</sup> implies that *V*2(*xi*, *xi*�) = *βi*,*i*� for *xi* = *xi*�and *V*2(*xi*, *xi*�) =

*i* lim *�*→0

We define the partial functions *Vi* of *U*(*x*) according to Li (2000); Shridhar et al. (1989)

−*yi*(1 − *xi*)ln 2 1 − *yi* + *�*

arg max *<sup>X</sup>* (*P*(*X*, *<sup>Y</sup>*)) = arg max *<sup>X</sup>* (ln(*P*(*X*, *<sup>Y</sup>*))) (7)

−*yi*(1 − *xi*)ln 2 1 − *yi* + *�*

(6)

*<sup>T</sup> U*(*x*) ∏ *i* lim *�*→0 1 <sup>2</sup> exp 

Since the natural logarithm is strictly increasing, the following equality holds:

Substituting (1) and (5) into (4), we can find the joint probability by

 *Z*−1*e*<sup>−</sup> <sup>1</sup>

*V*(*x*) = *U*(*x*) + ∑

*P*(*X* = *x*, *Y* = *y*) =

where *V*(*x*) = ln [*P*(*X* = *x*,*Y* = *y*)].

From this we get a new expression for *V*(*x*):

*V*(*x*) = ∑

*V*(*x*) = ∑

Having obtained an estimate

*i*

*i*

*αxi* + *βi*,*i*� ∑ *i*� ∈N*<sup>i</sup>*

V(*x*) = ∑

*i*

= ∑ *i*

*αxi* + *βi*,*i*� ∑

and splitting the last term into a constant and a non-constant term yields

*i*�∈N*<sup>i</sup>*

*xixi*� + lim *�*→0

*αxi* + *β* ∑

*α* + *β* ∑ *i*�∈N*<sup>i</sup>*

*i*�∈N*<sup>i</sup>*

Since the last term only depends on *y*, we can find the MAP configuration by simplifying the

constants to get

−*βi*,*i*�for *xi* �= *xi*� .

expression to:

belonging to the error pattern is incorrectly estimated as a 1-bit. The parameter *ρ* must be estimated by simulation, but should in general be small, indicating a relatively certain −1-bit.

#### **5.4 Performance of estimation algorithm**

The performance of the estimation algorithm depends heavily on the value of *β*, which determines the degree of clustering in the error pattern. A critical performance parameter is the probability *ε* that not all bits in the error pattern are detected by the estimation algorithm. A bit that belongs to the error pattern, but is not detected as such, is given a high probability of being correct, and can hence be the source of errors that are hard to correct. Therefore, *ε* is an important measure of the reliability of the algorithm. As can be seen in Fig. 3, *P*(*ε*) is high for *β* < 0.5, reflecting the fact that small and very irregular error clusters appear in this range. *P*(*ε*) drops sharply initially, but levels out when *β* > 1 as a result of the clusters becoming bigger and more coherent. As we shall see later, this is also reflected in the performance of the algorithm.

Fig. 3. *P*(*ε*) for fixed *β* = 0.2 in the estimation algorithm.

on the parameter *β* in the Gibbs distribution in a way that makes it hard to predict the average error probability over codewords, we fix the value of *β* to *β* = 0.2 and *β* = 1.0 which gives an average error rate of about 0.12 and 0.02 respectively, and study the performance of joint LDPC - MRF decoding for different code rates using these parameters. We see in Fig. 4 that the effect of the MRF estimator gives very good results in combination with the LDPC code when the code rate is sufficiently low, while the performance gap between the two decoders gets smaller as the code rate grows. This occurs because the MRF-LDPC decoder needs a certain amount of information from the code itself to determine the value of the bits in the

An Application of Digital Image Restoration Techniques to Error Control Coding 365

In this section we will see how the outlined cluster detection technique can be applied to the two-dimensional binary symmetric channel. On this channel, both the transition 1 → −1 and

**Definition 3** (Componentwise product)**.** *Assume A and B are matrices with the same dimensions d*<sup>1</sup> × *d*<sup>2</sup> *where d*<sup>1</sup> · *d*<sup>2</sup> = *n, and with coordinates ai and bi respectively. Then the componentwise*

*A* ∗ *B* = *ai* · *bi*, 1 ≤ *i* ≤ *n*

Now assuming *X*, *Y* and *C* are *d*<sup>1</sup> × *d*<sup>2</sup> matrices with bipolar coordinates, the effect of the

*Y* = *C* ∗ *X*

Like in chapter 5.2, information *I* is encoded, producing a codeword *C*. Two-dimensionally correlated noise *X* is applied to the codeword and the result *Y* is passed to the decoder. The decoding process is different in this case, however. The likelihood values *L* from the SISO are used to find some values *δ* that measures the distance between the received input and the output of the SISO, multiplied by the channel value. The *δ* value can be seen as an estimate of the value of the corresponding bit in *X*. These values are sent to the cluster detector to be used as basis for producing an estimate of the error cluster which is in turn fed back to the SISO. As in 5.2, the iterative process continues until a valid codeword is found or a set of stopping

The values *Y* received from the channel is used to compute likelihood ratios *L<sup>I</sup>* =

*<sup>n</sup>*} for the bits. Since we assume that the variables are bipolar, the likelihood

error pattern, even if the MRF estimator provides a perfect estimate of the errors.

**6. Application to the two-dimensional binary symmetric channel**

**6.1 Channel model**

−1 → 1 may take place.

*product of these matrices is defined as*

∗ - multiplicative operator on matrices defined as above

channel can be described as:

where

*Y* - received word *C* - original codeword *X* - error pattern

**6.2 System model**

criteria is met.

{*LI* <sup>1</sup>, *<sup>L</sup><sup>I</sup>*

**6.3 MAP estimation 6.3.1 distribution of** Δ*<sup>i</sup>*

<sup>2</sup>,..., *<sup>L</sup><sup>I</sup>*

#### **5.5 Results**

A regular LDPC code was used as the error correcting code component , with different values of *β* in the simulations, and *α* = *γ* = ··· = 0 in the estimation algorithm. We assumed that the receiver does not know the value of *β* used by the noise generating process. The components of the simulator was then connected as shown in Fig. 6 The simulations show that there is a large performance gain for some choices of parameter using the LDPC-MRF combination described above. The value of *β* has great influence over the relative performance of the two decoding methods. Looking in Fig. 5 at the performance of a code in combination with the MAP error estimate and alone, under varying *β*, we can observe that the performance difference between the two decoders increases as *β* increases. This is due to the effect described in Section 5.4: as *β* increases, the reliability of the error estimate also increases. We also notice that the drop in BER levels off at about *β* = 1 corresponding to the reliability of the estimate leveling off from the same point. The performance of the decoder could also be measured under varying bit error probabilities, but because the bit error probability depends

Fig. 4. Performance under varying rate with *β* = 0.2 and *β* = 1.0

Fig. 5. Performance under varying *β* with rate <sup>3</sup> 8

on the parameter *β* in the Gibbs distribution in a way that makes it hard to predict the average error probability over codewords, we fix the value of *β* to *β* = 0.2 and *β* = 1.0 which gives an average error rate of about 0.12 and 0.02 respectively, and study the performance of joint LDPC - MRF decoding for different code rates using these parameters. We see in Fig. 4 that the effect of the MRF estimator gives very good results in combination with the LDPC code when the code rate is sufficiently low, while the performance gap between the two decoders gets smaller as the code rate grows. This occurs because the MRF-LDPC decoder needs a certain amount of information from the code itself to determine the value of the bits in the error pattern, even if the MRF estimator provides a perfect estimate of the errors.

## **6. Application to the two-dimensional binary symmetric channel**

## **6.1 Channel model**

12 Image Restoration

A regular LDPC code was used as the error correcting code component , with different values of *β* in the simulations, and *α* = *γ* = ··· = 0 in the estimation algorithm. We assumed that the receiver does not know the value of *β* used by the noise generating process. The components of the simulator was then connected as shown in Fig. 6 The simulations show that there is a large performance gain for some choices of parameter using the LDPC-MRF combination described above. The value of *β* has great influence over the relative performance of the two decoding methods. Looking in Fig. 5 at the performance of a code in combination with the MAP error estimate and alone, under varying *β*, we can observe that the performance difference between the two decoders increases as *β* increases. This is due to the effect described in Section 5.4: as *β* increases, the reliability of the error estimate also increases. We also notice that the drop in BER levels off at about *β* = 1 corresponding to the reliability of the estimate leveling off from the same point. The performance of the decoder could also be measured under varying bit error probabilities, but because the bit error probability depends

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

Rate

LDPC MRF-LDPC

0.1 1

8

β

**5.5 Results**

10-4

10-5

Fig. 5. Performance under varying *β* with rate <sup>3</sup>

10-4

10-3

BER

10-2

10-1

100

Fig. 4. Performance under varying rate with *β* = 0.2 and *β* = 1.0

10-3

10-2

BER

10-1

100

'data' u (\$1/400):2 'data' u (\$1/400):3 In this section we will see how the outlined cluster detection technique can be applied to the two-dimensional binary symmetric channel. On this channel, both the transition 1 → −1 and −1 → 1 may take place.

**Definition 3** (Componentwise product)**.** *Assume A and B are matrices with the same dimensions d*<sup>1</sup> × *d*<sup>2</sup> *where d*<sup>1</sup> · *d*<sup>2</sup> = *n, and with coordinates ai and bi respectively. Then the componentwise product of these matrices is defined as*

$$A \ast B = a\_i \cdot b\_{i\nu} \qquad 1 \le i \le n$$

Now assuming *X*, *Y* and *C* are *d*<sup>1</sup> × *d*<sup>2</sup> matrices with bipolar coordinates, the effect of the channel can be described as:

*Y* = *C* ∗ *X*

where

*Y* - received word

*C* - original codeword

*X* - error pattern

∗ - multiplicative operator on matrices defined as above

#### **6.2 System model**

Like in chapter 5.2, information *I* is encoded, producing a codeword *C*. Two-dimensionally correlated noise *X* is applied to the codeword and the result *Y* is passed to the decoder. The decoding process is different in this case, however. The likelihood values *L* from the SISO are used to find some values *δ* that measures the distance between the received input and the output of the SISO, multiplied by the channel value. The *δ* value can be seen as an estimate of the value of the corresponding bit in *X*. These values are sent to the cluster detector to be used as basis for producing an estimate of the error cluster which is in turn fed back to the SISO. As in 5.2, the iterative process continues until a valid codeword is found or a set of stopping criteria is met.

#### **6.3 MAP estimation**

## **6.3.1 distribution of** Δ*<sup>i</sup>*

The values *Y* received from the channel is used to compute likelihood ratios *L<sup>I</sup>* = {*LI* <sup>1</sup>, *<sup>L</sup><sup>I</sup>* <sup>2</sup>,..., *<sup>L</sup><sup>I</sup> <sup>n</sup>*} for the bits. Since we assume that the variables are bipolar, the likelihood

The values Δ*<sup>i</sup>* = *Yi*(

with mean *xiμ* so that

*LI i* − *LO*

0.3 0.25 0.2

0.15 0.1

0.05 0

Fig. 9. Distribution of the Δ*i*'s

Bayes rule gives us

instead of (10).

probabilities as:


(a) *Xi* = +1 and *μ* = 0.55

*<sup>i</sup>* ) are sent to the detector, which computes the MAP estimate of the

<sup>2</sup>*σ*<sup>2</sup> . (9)

0 15

5

(b) *Xi* = −1 and *μ* = −0.55

*P*(*X* = *x*)*P*(Δ = *δ*|*X* = *x*) (11)

1 *σ* <sup>√</sup>2*<sup>π</sup> e* <sup>−</sup> (*δ<sup>i</sup>* <sup>−</sup>*xiμ*)<sup>2</sup>

> 1 *σ* <sup>√</sup>2*<sup>π</sup> e* <sup>−</sup> (*δ<sup>i</sup>* <sup>−</sup>*xiμ*)<sup>2</sup> 2*σ*2

10

*<sup>P</sup>*(<sup>Δ</sup> <sup>=</sup> *<sup>δ</sup>*) (10)

<sup>2</sup>*σ*<sup>2</sup> (12)

configuration of *X*, i.e. the most likely *X* to produce the observed values. As is shown below, knowing the distribution of *P*(Δ*<sup>i</sup>* = *δ<sup>i</sup>* | *Xi* = *xi*) is essential to the MAP estimate calculation, so we shall make the assumption that the information from the SISO, Δ*i*, conditioned on *Xi* has a normal distribution with variance *σ* and mean *Xiμ*. The conditional probabilities can then be expressed as an exponential function by approximating them with the normal distribution

An Application of Digital Image Restoration Techniques to Error Control Coding 367

As an example of this, we can see in Fig. 9(a) and Fig. 9(b) the distribution of the Δ*i*'s for *Xi* = −1 and *Xi* = +1 respectively, compared with the normal distribution with *μ* = ±0.55 and *σ* = 1.4. We can see that this approximation is reasonably good. MAP estimation of the

0.25

0.2 0.15

0.1 0.05 0 -15 -10 -5

*σ* <sup>√</sup>2*<sup>π</sup> e* <sup>−</sup> (*δ<sup>i</sup>* −*xi μ*)2

*<sup>P</sup>*(Δ*<sup>i</sup>* <sup>=</sup> *<sup>δ</sup><sup>i</sup>* <sup>|</sup> *Xi* <sup>=</sup> *xi*) = <sup>1</sup>

15

error pattern *X* based on the values Δ from the SISO, can be formulated as the maximization of the *a posteriori* probability *P*(*X* = *x*|Δ = *δ*) with respect to *x*. That is, we want to find a configuration *x* that makes the probability *P*(*X* = *x*|Δ = *δ*) as high as possible for a given *δ*.

*<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*|<sup>Δ</sup> <sup>=</sup> *<sup>δ</sup>*) = *<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*)*P*(<sup>Δ</sup> <sup>=</sup> *<sup>δ</sup>*|*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*)

Based on the assumption that the distribution of *P*(Δ*<sup>i</sup>* = *δ<sup>i</sup>* | *Xi* = *xi*) is given by (9), we can

Substituting (1) and (12) into (11), and taking *T* = 1, we can express the product of the

*<sup>P</sup>*(*δ<sup>i</sup>* | *xi*) = ∏

*Z*−1*e*−*U*(*x*)

*i*

 ∏ *i*

Since we are optimizing the expression for a given value of *δ*, we can maximize over

*i*

express the probability of a given *δ* conditioned on a configuration *x* by

*<sup>P</sup>*(<sup>Δ</sup> = *<sup>δ</sup>* | *<sup>X</sup>* = *<sup>x</sup>*) = ∏

*<sup>P</sup>*(*<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*)*P*(<sup>Δ</sup> <sup>=</sup> *<sup>δ</sup>* <sup>|</sup> *<sup>X</sup>* <sup>=</sup> *<sup>x</sup>*) =

10

Fig. 6. System model

ratios can be expressed as

$$L\_i^I = \frac{P(C\_i = -1 | \mathcal{Y}\_i)}{P(C\_i = +1 | \mathcal{Y}\_i)}.$$

These values are the channel values used in the SISO. The soft output from the SISO-component in the decoder is *<sup>L</sup><sup>O</sup>* <sup>=</sup> {*L<sup>O</sup>* <sup>1</sup> , *<sup>L</sup><sup>O</sup>* <sup>2</sup> ,..., *<sup>L</sup><sup>O</sup> <sup>n</sup>* }. The output *<sup>L</sup><sup>O</sup>* of the SISO also has the form of likelihood ratios. We now take the logarithm of *L<sup>I</sup>* and *L<sup>O</sup>* giving us the values *<sup>L</sup><sup>I</sup>* and *<sup>L</sup>O*. These variables are now real valued in the range �−∞, <sup>∞</sup>�, with a negative value indicating a possible −1 bit and a positive value indicating a possible +1 bit. The difference *LI i* − *LO <sup>i</sup>* measures the distance between the input- and output values, and multiplication by *Yi* gives the relative direction Δ*<sup>i</sup>* of the change.

Fig. 7. Negative Δ*<sup>i</sup>*

As an example, in figure 7 the distance between the input and the output *d* is positive, but *Yi* = −1, so the relative direction Δ*<sup>i</sup>* = −1 · *d* = −*d* is negative. This corresponds to a higher probability that the bit was flipped by the channel.

Fig. 8. Positive Δ*<sup>i</sup>*

In figure 8 on the other hand, the distance *d* is negative and *Yi* = −1 so the relative distance Δ*<sup>i</sup>* = −*d* is positive, indicating a higher probability that the bit is correct.

The values Δ*<sup>i</sup>* = *Yi*( *LI i* − *LO <sup>i</sup>* ) are sent to the detector, which computes the MAP estimate of the configuration of *X*, i.e. the most likely *X* to produce the observed values. As is shown below, knowing the distribution of *P*(Δ*<sup>i</sup>* = *δ<sup>i</sup>* | *Xi* = *xi*) is essential to the MAP estimate calculation, so we shall make the assumption that the information from the SISO, Δ*i*, conditioned on *Xi* has a normal distribution with variance *σ* and mean *Xiμ*. The conditional probabilities can then be expressed as an exponential function by approximating them with the normal distribution with mean *xiμ* so that

$$P(\Delta\_i = \delta\_i \mid X\_i = x\_i) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(\delta\_i - x\_i\mu)^2}{2\sigma^2}}.\tag{9}$$

As an example of this, we can see in Fig. 9(a) and Fig. 9(b) the distribution of the Δ*i*'s for *Xi* = −1 and *Xi* = +1 respectively, compared with the normal distribution with *μ* = ±0.55 and *σ* = 1.4. We can see that this approximation is reasonably good. MAP estimation of the

Fig. 9. Distribution of the Δ*i*'s

error pattern *X* based on the values Δ from the SISO, can be formulated as the maximization of the *a posteriori* probability *P*(*X* = *x*|Δ = *δ*) with respect to *x*. That is, we want to find a configuration *x* that makes the probability *P*(*X* = *x*|Δ = *δ*) as high as possible for a given *δ*. Bayes rule gives us

$$P(X = \mathbf{x} | \Delta = \delta) = \frac{P(X = \mathbf{x})P(\Delta = \delta | X = \mathbf{x})}{P(\Delta = \delta)}\tag{10}$$

Since we are optimizing the expression for a given value of *δ*, we can maximize over

$$P(X = x)P(\Delta = \delta | X = x) \tag{11}$$

instead of (10).

14 Image Restoration

*LI*

*<sup>i</sup>* <sup>=</sup> *<sup>P</sup>*(*Ci* <sup>=</sup> <sup>−</sup>1|*Yi*) *P*(*Ci* = +1|*Yi*)

These values are the channel values used in the SISO. The soft output from the

the form of likelihood ratios. We now take the logarithm of *L<sup>I</sup>* and *L<sup>O</sup>* giving us the values

As an example, in figure 7 the distance between the input and the output *d* is positive, but *Yi* = −1, so the relative direction Δ*<sup>i</sup>* = −1 · *d* = −*d* is negative. This corresponds to a higher

In figure 8 on the other hand, the distance *d* is negative and *Yi* = −1 so the relative distance

Δ*<sup>i</sup>* = −*d* is positive, indicating a higher probability that the bit is correct.

<sup>2</sup> ,..., *<sup>L</sup><sup>O</sup>*

*<sup>L</sup>O*. These variables are now real valued in the range �−∞, <sup>∞</sup>�, with a negative value indicating a possible −1 bit and a positive value indicating a possible +1 bit. The difference

*<sup>i</sup>* measures the distance between the input- and output values, and multiplication by

<sup>1</sup> , *<sup>L</sup><sup>O</sup>*

.

*<sup>n</sup>* }. The output *<sup>L</sup><sup>O</sup>* of the SISO also has

Fig. 6. System model

Fig. 7. Negative Δ*<sup>i</sup>*

Fig. 8. Positive Δ*<sup>i</sup>*

 *<sup>L</sup><sup>I</sup>* and

 *LI i* − *LO*

ratios can be expressed as

SISO-component in the decoder is *<sup>L</sup><sup>O</sup>* <sup>=</sup> {*L<sup>O</sup>*

*Yi* gives the relative direction Δ*<sup>i</sup>* of the change.

probability that the bit was flipped by the channel.

Based on the assumption that the distribution of *P*(Δ*<sup>i</sup>* = *δ<sup>i</sup>* | *Xi* = *xi*) is given by (9), we can express the probability of a given *δ* conditioned on a configuration *x* by

$$P(\Delta = \delta \mid X = \mathbf{x}) = \prod\_{i} P(\delta\_{i} \mid \mathbf{x}\_{i}) = \prod\_{i} \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{(\delta\_{i} - \mathbf{x}\_{i}\mu)^{2}}{2\sigma^{2}}} \tag{12}$$

Substituting (1) and (12) into (11), and taking *T* = 1, we can express the product of the probabilities as:

$$P(X=\mathbf{x})P(\Delta=\delta \mid X=\mathbf{x}) = \left[Z^{-1}e^{-U(\mathbf{x})}\right] \prod\_{i} \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(\delta\_i - x\_i\mu)^2}{2\sigma^2}}$$

<sup>−</sup> *<sup>μ</sup>*<sup>2</sup>

**6.3.2 Optimization of** V(*x*)

we compute the value of

Having obtained an estimate

we set the likelihood ratio to

Fig. 10. Input probabilities to the decoder

relatively certain −1-bit.

<sup>2</sup>*σ*<sup>2</sup> , and the expression to maximize over becomes:

V(*x*) = ∑ *i*

> = ∑ *i*

⎡

⎡

<sup>⎣</sup>*αxi* + *<sup>β</sup>* ∑

<sup>⎣</sup>*<sup>α</sup>* + *<sup>β</sup>* ∑

*αxi* + *β* ∑ *xi* � ∈N*<sup>i</sup>*

*xi*� ∈N*<sup>i</sup>*

*xi*� ∈N*<sup>i</sup>*

An Application of Digital Image Restoration Techniques to Error Control Coding 369

To do a global optimization of the expression above with respect to *x* would become computationally infeasible as the size of *x* increases. Instead we can use the local dependencies between bits to do a local optimization along the lines of the PDFE in Neifeld & King (1998); Neifield et al. (1996) or the partial binary segmentation algorithm of Shridhar et al. (1989). It is apparent that when V(*x*) is expressed as in (14), we can always choose the value of *xi* so that each term in the sum becomes positive, and thus the sum is non-decreasing. For each node

and set the value of *xi* so that the product is positive. This procedure is iterated until we converge on a solution where all terms in the sum are positive, or a maximum number of iterations is reached. Normally the process arrives at a solution after less than 10 iterations.

*<sup>X</sup>*<sup>ˆ</sup> <sup>=</sup> {*X*<sup>ˆ</sup> 1,..., *<sup>X</sup>*ˆ*i*,..., *<sup>X</sup>*<sup>ˆ</sup> *<sup>n</sup>*} of the error pattern, it can be used to find likelihood ratios for input to the SISO. For each bit,

*Li* <sup>=</sup> *<sup>P</sup>*(*Ci* <sup>=</sup> <sup>−</sup>1|*Yi*, *<sup>X</sup>*ˆ*i*) *<sup>P</sup>*(*Ci* <sup>=</sup> <sup>1</sup>|*Yi*, *<sup>X</sup>*ˆ*i*) The resulting probabilities can be seen in Fig. 10. In the table, *ρ* is the probability that a bit

detected as belonging to the error pattern is actually a <sup>−</sup>1-bit, i.e. *<sup>ρ</sup>* <sup>=</sup> *<sup>P</sup>*(*Xi* <sup>=</sup> <sup>−</sup><sup>1</sup> <sup>|</sup> *<sup>X</sup>*ˆ*<sup>i</sup>* <sup>=</sup> <sup>−</sup>1). For a given channel, the parameter *ρ* must be found experimentally by sending some known codewords over the channel. When the receiver knows the value of *X*, the value of *ρ* can be computed based on the value of the estimates *X*ˆ . *ρ* should in general be large indicating a

*xi*� +

*δiμ σ*2

*xixi*� +

*xi*� <sup>+</sup> *<sup>μ</sup> <sup>σ</sup>*<sup>2</sup> *<sup>δ</sup><sup>i</sup>*

2*xiδiμ* 2*σ*<sup>2</sup>

⎤

⎤ ⎦

⎦ *xi* (14)

In order to avoid computing *Z* in the above expression, we take the logarithm of both sides and eliminate the constants to get

$$V(\mathbf{x}) = U(\mathbf{x}) - \sum\_{i} \frac{(\delta\_{i} - \mathbf{x}\_{i}\mu)^{2}}{2\sigma^{2}}.\tag{13}$$

Since the natural logarithm is strictly increasing, optimizing (13) with respect to *x* also optimizes (5).

We define the partial functions *Vt* of *U*(*x*) in (3) according to Li (2000) and Shridhar et al. (1989):

$$\begin{aligned} \sum\_{c \in \mathcal{C}\_1} V\_1(c) &= \sum\_{i} \alpha \mathbf{x}\_i \\ \sum\_{c \in \mathcal{C}\_2} V\_2(c) &= \sum\_{i} \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_i} \beta \mathbf{x}\_i \mathbf{x}\_{i'} \quad i \neq i' \end{aligned}$$

$$V\_t(\cdot) = 0 \quad \forall t > 2$$

This implies that we let the total probability depend on the value of each bit represented by *αxi* and the value of each neighboring bit represented by *βxixi*� , while we do not consider more complex dependencies like three-ways dependencies and up. Note that the expression for *V*<sup>2</sup> implies that *V*2(*xi*, *xi*�) = *β* for *xi* = *xi*�and *V*2(*xi*, *xi*�) = −*β* for *xi* �= *xi*� .

From this we get a new expression for *V*(*x*):

$$V(\mathbf{x}) = \sum\_{i} \left[ \alpha \mathbf{x}\_{i} + \beta \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_{i}} \mathbf{x}\_{i} \mathbf{x}\_{i'} - \frac{(\delta\_{i} - \mathbf{x}\_{i}\mu)^{2}}{2\sigma^{2}} \right] \mathbf{x}$$

and expanding the last term of the sum yields

$$V(\mathbf{x}) = \sum\_{i} \left[ \alpha \mathbf{x}\_i + \beta \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_i} \mathbf{x}\_i \mathbf{x}\_{i'} - \frac{(\delta\_i^2 - 2\mathbf{x}\_i \delta\_i \mu + \mathbf{x}\_i^2 \mu^2)}{2\sigma^2} \right].$$

The variables *xi* are bipolar so *x*<sup>2</sup> *<sup>i</sup>* = 1 and we can simplify the expression to:

$$\mathcal{V}(\mathbf{x}) = \sum\_{i} \left[ a\mathbf{x}\_{i} + \boldsymbol{\beta} \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_{i}} \mathbf{x}\_{i}\mathbf{x}\_{i'} - \frac{(\delta\_{i}^{2} - 2\mathbf{x}\_{i}\delta\_{i}\mu + \mu^{2})}{2\sigma^{2}} \right] $$

$$= \sum\_{i} \left[ a\mathbf{x}\_{i} + \boldsymbol{\beta} \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_{i}} \mathbf{x}\_{i}\mathbf{x}\_{i'} - \frac{\delta\_{i}^{2}}{2\sigma^{2}} + \frac{2\mathbf{x}\_{i}\delta\_{i}\mu}{2\sigma^{2}} - \frac{\mu^{2}}{2\sigma^{2}} \right].$$

As we are doing a maximization with respect to *xi*, any term in the sum that does not contain or otherwise depend on *xi* will not affect the result, and hence we cancel the terms <sup>−</sup> *<sup>δ</sup>*<sup>2</sup> *i* <sup>2</sup>*σ*<sup>2</sup> and <sup>−</sup> *<sup>μ</sup>*<sup>2</sup> <sup>2</sup>*σ*<sup>2</sup> , and the expression to maximize over becomes:

$$\mathcal{V}(\mathbf{x}) = \sum\_{i} \left[ \alpha \mathbf{x}\_{i} + \beta \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_{i}} \mathbf{x}\_{i} \mathbf{x}\_{i'} + \frac{2 \mathbf{x}\_{i} \delta\_{i} \mu}{2 \sigma^{2}} \right]$$

$$= \sum\_{i} \left[ \alpha + \beta \sum\_{\mathbf{x}\_{i'} \in \mathcal{N}\_{i}} \mathbf{x}\_{i'} + \frac{\mu}{\sigma^{2}} \delta\_{i} \right] \mathbf{x}\_{i} \tag{14}$$

## **6.3.2 Optimization of** V(*x*)

16 Image Restoration

In order to avoid computing *Z* in the above expression, we take the logarithm of both sides

*i*

Since the natural logarithm is strictly increasing, optimizing (13) with respect to *x* also

We define the partial functions *Vt* of *U*(*x*) in (3) according to Li (2000) and Shridhar et al.

∑ *xi* �∈N*<sup>i</sup>*

This implies that we let the total probability depend on the value of each bit represented by *αxi* and the value of each neighboring bit represented by *βxixi*� , while we do not consider more complex dependencies like three-ways dependencies and up. Note that the expression

(*δ<sup>i</sup>* <sup>−</sup> *xiμ*)<sup>2</sup>

*βxixi*� *i* �= *i*

*xixi*� <sup>−</sup> (*δ<sup>i</sup>* <sup>−</sup> *xiμ*)<sup>2</sup> 2*σ*<sup>2</sup>

*<sup>i</sup>* <sup>−</sup> <sup>2</sup>*xiδi<sup>μ</sup>* <sup>+</sup> *<sup>x</sup>*<sup>2</sup>

2*σ*<sup>2</sup>

*<sup>i</sup>* <sup>−</sup> <sup>2</sup>*xiδi<sup>μ</sup>* <sup>+</sup> *<sup>μ</sup>*2) 2*σ*<sup>2</sup>

> 2*xiδiμ* <sup>2</sup>*σ*<sup>2</sup> <sup>−</sup> *<sup>μ</sup>*<sup>2</sup> 2*σ*<sup>2</sup>

�

⎤ ⎦ ,

*<sup>i</sup> <sup>μ</sup>*2)

� .

⎤ ⎦

> ⎤ ⎦

> > *i* <sup>2</sup>*σ*<sup>2</sup> and

<sup>2</sup>*σ*<sup>2</sup> . (13)

*<sup>V</sup>*(*x*) = *<sup>U</sup>*(*x*) − ∑

*V*1(*c*) = ∑

*V*2(*c*) = ∑

for *V*<sup>2</sup> implies that *V*2(*xi*, *xi*�) = *β* for *xi* = *xi*�and *V*2(*xi*, *xi*�) = −*β* for *xi* �= *xi*� .

<sup>⎣</sup>*αxi* + *<sup>β</sup>* ∑ *xi* � ∈N*<sup>i</sup>*

*xi*� ∈N*<sup>i</sup>*

*xixi*� <sup>−</sup> (*δ*<sup>2</sup>

*xixi*� <sup>−</sup> (*δ*<sup>2</sup>

*xixi*� <sup>−</sup> *<sup>δ</sup>*<sup>2</sup> *i* <sup>2</sup>*σ*<sup>2</sup> <sup>+</sup>

As we are doing a maximization with respect to *xi*, any term in the sum that does not contain or otherwise depend on *xi* will not affect the result, and hence we cancel the terms <sup>−</sup> *<sup>δ</sup>*<sup>2</sup>

*<sup>i</sup>* = 1 and we can simplify the expression to:

⎡

<sup>⎣</sup>*αxi* + *<sup>β</sup>* ∑

<sup>⎣</sup>*αxi* + *<sup>β</sup>* ∑ *xi* � ∈N*<sup>i</sup>*

<sup>⎣</sup>*αxi* + *<sup>β</sup>* ∑ *xi* � ∈N*<sup>i</sup>*

*i αxi*

*i*

*Vt*(·) = 0 ∀*t* > 2

∑ *c*∈C<sup>1</sup>

∑ *c*∈C<sup>2</sup>

From this we get a new expression for *V*(*x*):

and expanding the last term of the sum yields

*V*(*x*) = ∑ *i*

V(*x*) = ∑

*i*

= ∑ *i*

⎡

⎡

The variables *xi* are bipolar so *x*<sup>2</sup>

*V*(*x*) = ∑ *i*

⎡

and eliminate the constants to get

optimizes (5).

(1989):

To do a global optimization of the expression above with respect to *x* would become computationally infeasible as the size of *x* increases. Instead we can use the local dependencies between bits to do a local optimization along the lines of the PDFE in Neifeld & King (1998); Neifield et al. (1996) or the partial binary segmentation algorithm of Shridhar et al. (1989). It is apparent that when V(*x*) is expressed as in (14), we can always choose the value of *xi* so that each term in the sum becomes positive, and thus the sum is non-decreasing. For each node we compute the value of

$$
\alpha \mathbf{x}\_{i} + \beta \sum\_{\mathbf{x}\_{l'} \in \mathcal{N}\_{l}} \mathbf{x}\_{l'} + \frac{\delta\_{i} \mu}{\sigma^{2}},
$$

and set the value of *xi* so that the product is positive. This procedure is iterated until we converge on a solution where all terms in the sum are positive, or a maximum number of iterations is reached. Normally the process arrives at a solution after less than 10 iterations.

Having obtained an estimate

$$\hat{X} = \{\hat{X}\_1, \dots, \hat{X}\_{\hat{\nu}}, \dots, \hat{X}\_n\}.$$

of the error pattern, it can be used to find likelihood ratios for input to the SISO. For each bit, we set the likelihood ratio to

$$L\_i = \frac{P(\mathbf{C}\_i = -1 | Y\_{i\nu} \hat{X}\_i)}{P(\mathbf{C}\_i = 1 | Y\_{i\nu} \hat{X}\_i)}$$

The resulting probabilities can be seen in Fig. 10. In the table, *ρ* is the probability that a bit


Fig. 10. Input probabilities to the decoder

detected as belonging to the error pattern is actually a <sup>−</sup>1-bit, i.e. *<sup>ρ</sup>* <sup>=</sup> *<sup>P</sup>*(*Xi* <sup>=</sup> <sup>−</sup><sup>1</sup> <sup>|</sup> *<sup>X</sup>*ˆ*<sup>i</sup>* <sup>=</sup> <sup>−</sup>1). For a given channel, the parameter *ρ* must be found experimentally by sending some known codewords over the channel. When the receiver knows the value of *X*, the value of *ρ* can be computed based on the value of the estimates *X*ˆ . *ρ* should in general be large indicating a relatively certain −1-bit.

10-6

Fig. 12. Performance under varying rate with *Eh* = 0.2250

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

An Application of Digital Image Restoration Techniques to Error Control Coding 371

Rate

By applying principles known form digital image restoration, we have introduced a channel model for two-dimensional channels with memory based on Markov random fields which allows us to describe spatially dependent errors. We have showed that a significant performance gain over an ordinary error correcting code can be achieved , for both the symmetric and the asymmetric binary two-dimensional channel by combining an error-correcting component with an MRF-based burst detection algorithm. We have also demonstrated that this decoding technique gives the most gain for larger clusters and for

Banham, M. & Katsaggelos, A. (1997). Digital image restoration, *Signal Processing Magazine,*

Blaum, M., Bruck, J. & Vardy, A. (1998). Interleaving schemes for multidimensional cluster

Breitbach, M., Bossert, M., Zyablov, V. & Sidorenko, V. (1998). Array codes correcting

Chalmond, B. (1988). Image restoration using an estimated markov model, *Signal Processing*

El-Gabali, M., Shridhar, M. & Ahmadi, M. (1987). Segmentation of noisy images modelled by

El-Gabali, M., Shridhar, M., Ahmadi, M. & Kekre, J. (n.d.). Restoration of blurred images,

*IEEE International Conference on ICASSP'87.*, Vol. 12, IEEE, pp. 551–554. El-Gabali, M., Shridhar, M. & Ahmadi, M. (1988). Restoration of degraded images using

a two-dimensional cluster of errors, *Information Theory, IEEE Transactions on*

markov random fields with gibbs distribution, *Acoustics, Speech, and Signal Processing,*

markov random fields, *Acoustics, Speech, and Signal Processing, 1988. ICASSP-88., 1988*

*Signals, Systems and Computers, 1990. 1990 Conference Record Twenty-Fourth Asilomar*

errors, *Information Theory, IEEE Transactions on* 44(2): 730–743.

*International Conference on*, IEEE, pp. 1004–1007.

*Conference on*, Vol. 2, IEEE, p. 549.

10-5

10-4

BER

**7. Conclusion**

lower information rates.

*IEEE* 14(2): 24–41.

44(5): 2025–2031.

15(2): 115–129.

**8. References**

10-3

10-2

10-1

 LDPC MRF-LDPC

#### **6.4 Results**

We have implemented the system with an LDPC decoder as the SISO component, and we assume a priori knowledge of the values *α*, *β* and *ρ*. The error-simulation process allows upper bounding of the overall error rate of the channel, and the performance of the decoder is measured for two different upper bounds on the error rate, denoted *Eh*, to investigate the effect of varying error rates on the performance of the joint decoding and estimation. The parameter *β* in the Gibbs distribution should ideally depend on the the bit error probability of the channel, but in the simulations we have chosen to fix the value of *β* to *β* = 0.2 which corresponds to an average error rate of about 0.12, and study the performance of joint LDPC - MRF decoding for different code rates under this assumption. This is a rather "harsh" assumption in the sense that in practice it should be possible to find an estimate of the value of *β* before transmission takes place. The performance of the joint decoding and estimation algorithm is compared to decoding of the same received information using the same LDPC decoder component but under a random error assumption, i.e., the decoder assumes that there are no dependencies in the error generating process. The LDPC codes used in the simulations are generic regular LDPC codes that were not optimized for use on this particluar type of channel. We see in Fig. 11 that the effect of the MRF estimator gives very good results in combination with the LDPC code when the code rate is sufficiently low, while the performance gap between the two decoders gets smaller as the code rate grows. This indicates that even if the MRF estimator provides a perfect estimate of the errors, the MRF-LDPC decoder still needs a certain amount of information from the code itself to determine the value of the bits in the error pattern, and therefore the mutual gain when exchanging information between the LDPC component and the channel detector component decreases as the code rate increases.

Fig. 12 shows that the performance of an LDPC-decoder in combination with the MAP error estimate, relative to an LDPC code alone. We see that the gain is even greater for a higher upper bound on the error rate, mainly because the MRF-estimate makes the most difference in face of a high number of errors.

Fig. 11. Performance under varying rate with *Eh* = 0.1250

Fig. 12. Performance under varying rate with *Eh* = 0.2250

#### **7. Conclusion**

18 Image Restoration

We have implemented the system with an LDPC decoder as the SISO component, and we assume a priori knowledge of the values *α*, *β* and *ρ*. The error-simulation process allows upper bounding of the overall error rate of the channel, and the performance of the decoder is measured for two different upper bounds on the error rate, denoted *Eh*, to investigate the effect of varying error rates on the performance of the joint decoding and estimation. The parameter *β* in the Gibbs distribution should ideally depend on the the bit error probability of the channel, but in the simulations we have chosen to fix the value of *β* to *β* = 0.2 which corresponds to an average error rate of about 0.12, and study the performance of joint LDPC - MRF decoding for different code rates under this assumption. This is a rather "harsh" assumption in the sense that in practice it should be possible to find an estimate of the value of *β* before transmission takes place. The performance of the joint decoding and estimation algorithm is compared to decoding of the same received information using the same LDPC decoder component but under a random error assumption, i.e., the decoder assumes that there are no dependencies in the error generating process. The LDPC codes used in the simulations are generic regular LDPC codes that were not optimized for use on this particluar type of channel. We see in Fig. 11 that the effect of the MRF estimator gives very good results in combination with the LDPC code when the code rate is sufficiently low, while the performance gap between the two decoders gets smaller as the code rate grows. This indicates that even if the MRF estimator provides a perfect estimate of the errors, the MRF-LDPC decoder still needs a certain amount of information from the code itself to determine the value of the bits in the error pattern, and therefore the mutual gain when exchanging information between the LDPC component and the channel detector component decreases as the code rate increases. Fig. 12 shows that the performance of an LDPC-decoder in combination with the MAP error estimate, relative to an LDPC code alone. We see that the gain is even greater for a higher upper bound on the error rate, mainly because the MRF-estimate makes the most difference

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

Rate

**6.4 Results**

in face of a high number of errors.

10-5

Fig. 11. Performance under varying rate with *Eh* = 0.1250

10-4

10-3

BER

10-2

10-1

100

 LDPC MRF-LDPC By applying principles known form digital image restoration, we have introduced a channel model for two-dimensional channels with memory based on Markov random fields which allows us to describe spatially dependent errors. We have showed that a significant performance gain over an ordinary error correcting code can be achieved , for both the symmetric and the asymmetric binary two-dimensional channel by combining an error-correcting component with an MRF-based burst detection algorithm. We have also demonstrated that this decoding technique gives the most gain for larger clusters and for lower information rates.

#### **8. References**


20 Image Restoration

372 Image Restoration – Recent Advances and Applications

Ellingsen, P. & Kvamme, A. (2010). Iterative decoding and estimation of two-dimensional

Ellingsen, P., Ytrehus, O. & Siegel, P. (n.d.). Enhanced decoding by error detection on a

Farrell, P. (1982). Array codes for correcting cluster-error patterns, *Proc. IEE Conf. Elect. Signal*

Geman, S. & Geman, D. (1993). Stochastic relaxation, gibbs distributions and the bayesian

Kurkoski, B. M. (2008). Towards efficient detection of two-dimensional intersymbol

Li, S. Z. (2000). Modeling image analysis problems using markov random fields, Vol. 20,

Neifeld, M. . & King, B. M. (1998 ). Parallel detection algorithms for page oriented optical

Neifield, M. A., Chugg, K. M. & King, B. M. (1996). Parallel data detection in page-oriented

Schwartz, M. & Etzion, T. (2003). Optimal 2-dimensional 3-dispersion lattices, *Applied Algebra,*

Schwartz, M. & Etzion, T. (2005). Two-dimensional cluster-correcting codes, *IEEE Trans. Info.*

Shridhar, M., Ahmadi, M. & El-Gabali, M. (1989). Restoration of noisy images modeled by

Singla, N. & O'Sullivan, J. A. (2005). Joint equalization and decoding for nonlinear

Xu, W. & Golomb, S. (2007). Optimal interleaving schemes for correcting two-dimensional

Zhang, J. (1993). The mean field theory in em procedures for blind markov random field image

markov random fields with gibbs distribution, *Circuits and Systems, IEEE Transactions*

Gilbert, E. et al. (1960). Capacity of a burst-noise channel, *Bell Syst. Tech. J* 39(9): 1253–1265. Jeng, F. & Woods, J. (1991). Compound gauss-markov random fields for image estimation,

restoration of images, *Journal of Applied Statistics* 20(5-6): 25–62.

Kindermann, R. & Snell, J. L. (1980). Markov random fields and their applications.

*Signal Processing, IEEE Transactions on* 39(3): 683–697.

memories. *Applied optics*(37(26) pp. 6275–6298

optical memory, *Optics Letters* 21(18): 1481–1483.

two-dimensional intersymbol interference channels.

cluster errors, *Discrete applied mathematics* 155(10): 1200–1212.

restoration, *Image Processing, IEEE Transactions on* 2(1): 27–40.

*Algebraic Algorithms and Error-Correcting Codes* pp. 606–606.

*Reliability, and Quality of Service*, IEEE, pp. 43–48.

*IEEE*, IEEE, pp. 17–21.

interference channels.

*Theory* 51: 2121–2132.

*on* 36(6): 884–890.

pp. 1–43.

*Processing (York, England)*.

channels with memory, *2010 Third International Conference on Communication Theory,*

channel with correlated two-dimensional errors, *Information Theory Workshop, 2004.*

## *Edited by Aymeric Histace*

This book represents a sample of recent contributions of researchers all around the world in the field of image restoration. The book consists of 15 chapters organized in three main sections (Theory, Applications, Interdisciplinarity). Topics cover some different aspects of the theory of image restoration, but this book is also an occasion to highlight some new topics of research related to the emergence of some original imaging devices. From this arise some real challenging problems related to image reconstruction/restoration that open the way to some new fundamental scientific questions closely related with the world we interact with.

Photo by LiliGraphie / iStock

Image Restoration - Recent Advances and Applications

Image Restoration

Recent Advances and Applications