**2. FPGA‐based SHWFS evaluation**

response of the deformable mirror, the wavefront needs to be measured accurately. To compensate for wavefront distortions, e.g., time‐varying disturbances with or without a stochastical and/or dynamical model, the disturbance has to be measured with adequate precision. For the quasi‐continuous measurement of the wavefront in AO systems, Shack‐ Hartmann wavefront sensors (SHWFSs) have widely been employed for measuring the

**Figure 1.** General scheme of adaptive optics, consisting of deformable mirror, wavefront sensor, and control system for

The SHWFS has shown some performance benefits when compared to interferometers as the SHWFS does not require a reference wave during the measurement process. Furthermore, the measurement sensitivity of an SHWFS primarily depends on the read‐out noise of the detector, the luminosity of the wavefront and, hence, the intensity of the spots, and on the algorithm to

The most commonly used wavefront measurement sensors, together with their advantages and disadvantages are discussed in Ref. [10]. The SHWFS itself relies decisively on the

closed‐loop operation [9].

178 Field - Programmable Gate Array

find and assign the centroids, respectively [6–8].

wavefront, thus, the phase of the electromagnetic wave [2–5].

For controlling the wavefront in an AO system, the wavefront itself has to be measured in an appropriate way. Several methods have been developed for that purpose, e.g., Pyramid, Shack‐ Hartmann (SHWFS), Curvature, or Holographic wavefront sensors [3, 16, 17]. Until now, an SHWFS is typically used for this objective as it may offer the best trade‐off between perform‐ ance, flexibility, and price. Since the SHWFS is based on capturing the intensities on an image plane (in general, a complementary metal‐oxide semiconductor (CMOS) or charge‐coupled device (CCD) image sensor), the evaluation of the SHWFS may be seen as some kind of image processing, calculating image moments. **Figure 2** depicts the basic principle of an SHWFS. Generally, an SHWFS will consist of an array of these lenses, called lenslet array.

**Figure 2.** Single convex lens; the gray dot marks the incident flat wavefront, the red dot marks the tilted incident wave‐ front.

The lenslet array is positioned in parallel to the image plane with a distance of the focal length of the lenses such that the focal point is on the image plane. If a flat wavefront is incident on the lenslet, the spot lies in the projected center of the convex lens in the image plane (marked with the gray dot in **Figure 2**). Due to the nature of the convex lens, the partial derivative of the wavefront with respect to *x*‐ and *y*‐direction is averaged over the area of the lens. Thus, the deviation of the focal spot on the image plane with respect to the projected center of the convex lens denotes the local derivative of the wavefront. If a lenslet array is used then one would define a given area on the image plane in which the spot must lie. This may limit, of course, the possible dynamic range because the steepness of the partially tilted wavefront is limited by the area of the image plane and the focal length of the lenslet array.

The task is to determine the position or the deviation and of the spots, as shown in **Figure 2**. For a given area with a predetermined number of pixels, this can be formulated as

device (CCD) image sensor), the evaluation of the SHWFS may be seen as some kind of image processing, calculating image moments. **Figure 2** depicts the basic principle of an SHWFS.

**Figure 2.** Single convex lens; the gray dot marks the incident flat wavefront, the red dot marks the tilted incident wave‐

The lenslet array is positioned in parallel to the image plane with a distance of the focal length of the lenses such that the focal point is on the image plane. If a flat wavefront is incident on the lenslet, the spot lies in the projected center of the convex lens in the image plane (marked with the gray dot in **Figure 2**). Due to the nature of the convex lens, the partial derivative of the wavefront with respect to *x*‐ and *y*‐direction is averaged over the area of the lens. Thus, the deviation of the focal spot on the image plane with respect to the projected center of the convex lens denotes the local derivative of the wavefront. If a lenslet array is used then one would define a given area on the image plane in which the spot must lie. This may limit, of course, the possible dynamic range because the steepness of the partially tilted wavefront is limited

by the area of the image plane and the focal length of the lenslet array.

front.

180 Field - Programmable Gate Array

Generally, an SHWFS will consist of an array of these lenses, called lenslet array.

$$\mathbf{x}\_c = \frac{\sum\_{i=0}^{M-1} \sum\_{j=0}^{N-1} \mathbf{x}\_i I(\mathbf{x}\_i, \mathbf{y}\_j)}{\sum\_{i=0}^{M-1} \sum\_{j=0}^{N-1} I(\mathbf{x}\_i, \mathbf{y}\_j)} \tag{1}$$

$$\mathbf{y}\_c = \frac{\sum\_{i=0}^{M-1} \sum\_{j=0}^{N-1} \mathbf{y}\_i I(\mathbf{x}\_i, \mathbf{y}\_j)}{\sum\_{i=0}^{M-1} \sum\_{j=0}^{N-1} I(\mathbf{x}\_i, \mathbf{y}\_j)} \tag{2}$$

where and denote the number of pixel in *x*‐ and *y*‐direction, respectively. ( , ) is the intensity of the pixel at the coordinate ( , ).

In the past, several methods have been developed for extending the dynamic range of the SHWFS, such as hardware modification, tracking, similarity approaches, to name but a few. We may now calculate <sup>=</sup> − , where is the *x*‐coordinate of the projected center of the convex lens. The corresponding calculation can be done for too. For the work presented here, the SHWFS with its CCD camera is directly connected to an FPGA to the end of lowest possible latency.

Evaluating the pixel information of the SHWFS for determining the phase of the wavefront may be divided into two problems: First, determine the individual centroids, i.e., calculate the centroid of the connected areas and, second, the ordering of the centroids to the lenslet for calculating the deviation with respect to the default position and, thus, computing the local derivatives of the wavefront.

As mentioned previously, as long as a predefined area is given in which the spots have to stay, the dynamic range of the SHWFS is limited. Since the approach of the connected areas does not use any predefined area, the restriction is no longer prevalent. However, to be fair, the default algorithm performs the determination and ordering of the centroids in a single step whereas the ordering is a subsequent step which is discussed in the following.

The determination of the connected areas for calculation of the centroids may be based on different methods. These methods mainly differ in their ability for online calculation of the connected areas, meaning that the pixel stream is processed sequentially at the end of it. Such methods are called single‐pass algorithms, emphasizing that only a single pass is required without necessarily storing the complete pixel information. The methods have extensively been studied, e.g., in road sign detection or line tracking systems for lane assistance. The general name for these algorithms is connected‐component labeling (CCL). CCL—also denoted by connected‐component analysis, region labeling—is an algorithmic application of graph theory. The subsets of connected components are often denoted as "blobs." Blobs are uniquely labeled, based on a predetermined heuristic, mostly along the neighbor relationship. For this approach, the labeling used for the CCL is based on an eight‐point neighborhood system, see **Figure 4**. Another popular neighborhood system is the four‐point neighborhood system which is presented in **Figure 3**. In **Figures 3** and **4**, the symbol "*s*" marks the actual pixel and the corresponding neighbors of pixel *s* are marked in gray.


**Figure 3.** Four‐point neighborhood system.


**Figure 4.** Eight‐point neighborhood system.

The procedure for CCL is straightforward. The pixels (the intensity information for each pixel) are streamed sequentially, typically from left to right and top to bottom. If the intensity information is larger than a threshold value, the pixel is assumed to be "1" else "0." This step is called binarization. In **Figure 5**, the boxes with a gray background have already been processed and the actual pixel carries the symbol "?."


**Figure 5.** Label collision due to nonconvex blob.

In the case when two sets are connected, but due to the sequential processing receive two different numbers, a label collision may occur. As long as the blobs are convex sets, a label

collision cannot occur when using an eight‐point neighborhood. Experiments have shown that the assumption of convex blobs is not valid for the typical application scenario of an SHWFS. This may be caused by a disturbed pixel intensity information, recording the noise of the camera sensor, the photon noise, nonperfect lenses, and other effects. Due to thresholding with a fixed value, a single count in terms of the digitalized intensity information can lead to nonconvex blobs, see e.g., **Figure 6**.


**Figure 6.** Resolved label collision due to blob merging.

For this approach, the labeling used for the CCL is based on an eight‐point neighborhood system, see **Figure 4**. Another popular neighborhood system is the four‐point neighborhood system which is presented in **Figure 3**. In **Figures 3** and **4**, the symbol "*s*" marks the actual

The procedure for CCL is straightforward. The pixels (the intensity information for each pixel) are streamed sequentially, typically from left to right and top to bottom. If the intensity information is larger than a threshold value, the pixel is assumed to be "1" else "0." This step is called binarization. In **Figure 5**, the boxes with a gray background have already been

In the case when two sets are connected, but due to the sequential processing receive two different numbers, a label collision may occur. As long as the blobs are convex sets, a label

pixel and the corresponding neighbors of pixel *s* are marked in gray.

**Figure 3.** Four‐point neighborhood system.

182 Field - Programmable Gate Array

**Figure 4.** Eight‐point neighborhood system.

**Figure 5.** Label collision due to nonconvex blob.

processed and the actual pixel carries the symbol "?."

Application of morphology methods, such as dilation or erosion, is not possible without storing large parts of the image, thus are not single‐pass compliant. Additionally, morphology methods significantly increase the delay.

The handling of label collisions can be accomplished by using a label stack which allows label reusing after a label collision has occurred. By means of label reusing the number of provided labels can be kept to a minimum; otherwise, under some circumstances, twice the number or even more labels must be provided. More information is given in Refs. [9, 18, 19].

After having determined the blobs, thus the connected areas, the division of the numerator and denominator may be performed for each valid blob found. The numerator and denomi‐ nator have to be stored separately as the division step can only be performed when the connected set is maximum. In the block diagram given in **Figure 7**, this step is done in the "centroid calculation/feature extraction" block which also performs the assignment of the centroids to the lenslets.

**Figure 7.** Block diagram of the complete CCL FPGA implementation [18].

One of the key elements of the drafted implementation in **Figure 7** is that solely the former line of the pixel stream has to be stored, not the whole pixel stream. For the applied camera, this results in storing 224 pixels where each pixel is one bit wide because only the binarized value has to be stored. Furthermore, only parts of the former line have to be accessed in parallel such that a small row register is sufficient which is automatically loaded from a Block RAM (BRAM). Using a BRAM has the advantage that the consumption of logic cells is reduced as the BRAM is a dedicated peripheral offered by most FPGAs. The overall logic consumption can be kept at a minimum [18].

**Figure 8.** Exemplary centroids for a 4 × 4 lenslet array; some spots are missing.

The assignment or segmentation of the centroids is visualized in **Figures 8** and **9**. This idea has been presented in Ref. [18] and behaves similar to the standard approach for the regular case, that is, the wavefront is not strongly disturbed. However, the advantage of this approach appears whenever a large defocus is present in the wavefront to be measured since shrinking or increasing the overall distance between two neighbored centroids is not a problem for the segmentation method.

The fundamental principle is that the centroids are ordered in parallel with respect to their *x*‐ and *y*‐value such that two separately ordered lists exist. Then, straight lines are used to segment the centroids in *x*‐ and *y*‐direction by using their distance between each other. As **Figure 9** illustrates, this method is working well also for the case when some centroids are missing due to shadowing or insufficient light intensities. When a very large shearing occurs, however, the method will not be ideal because straight vertical lines are used. But if this problem appears, the standard approach is also not applicable anymore. This algorithm is called simple straight line segmentation.

**Figure 9.** Segmented centroids applying the method presented in [18].

One of the key elements of the drafted implementation in **Figure 7** is that solely the former line of the pixel stream has to be stored, not the whole pixel stream. For the applied camera, this results in storing 224 pixels where each pixel is one bit wide because only the binarized value has to be stored. Furthermore, only parts of the former line have to be accessed in parallel such that a small row register is sufficient which is automatically loaded from a Block RAM (BRAM). Using a BRAM has the advantage that the consumption of logic cells is reduced as the BRAM is a dedicated peripheral offered by most FPGAs. The overall logic consumption

can be kept at a minimum [18].

184 Field - Programmable Gate Array

segmentation method.

line segmentation.

**Figure 8.** Exemplary centroids for a 4 × 4 lenslet array; some spots are missing.

The assignment or segmentation of the centroids is visualized in **Figures 8** and **9**. This idea has been presented in Ref. [18] and behaves similar to the standard approach for the regular case, that is, the wavefront is not strongly disturbed. However, the advantage of this approach appears whenever a large defocus is present in the wavefront to be measured since shrinking or increasing the overall distance between two neighbored centroids is not a problem for the

The fundamental principle is that the centroids are ordered in parallel with respect to their *x*‐ and *y*‐value such that two separately ordered lists exist. Then, straight lines are used to segment the centroids in *x*‐ and *y*‐direction by using their distance between each other. As **Figure 9** illustrates, this method is working well also for the case when some centroids are missing due to shadowing or insufficient light intensities. When a very large shearing occurs, however, the method will not be ideal because straight vertical lines are used. But if this problem appears, the standard approach is also not applicable anymore. This algorithm is called simple straight The described algorithm is very simple and straightforward. In Ref. [19] the so‐called spiral method has been extended to be deterministic and real‐time capable using the centroids gathered by employing CCL. It is obvious that depending on the specific application other methods may be better suited. The CCL may be enhanced by making the thresholding adaptive to compensate the natural intensity inhomogeneity [9, 20]. Another enhancement is the adaptive positioning which for most cases may solve the problem when the number of rows and columns after assignment of the centroids are not the same as with the lenslet array. This circumstance, in general, will lead to ambiguity of the assignment. The adaptive positioning, however, uses an approach based on the similarity of the shape of the segmented centroids and minimizes the shift. Based on this information, the assignment is shifted by one row or column to reduce the offset.

**Figure 10.** Evolution of processing time, each step rounded up to 25 µs, during SHWFS evaluation applying the FPGA approach.

**Figure 10** holds the timeline for the whole evaluation of the SHWFS beginning with the exposure until the assignment of the centroids to the lenslets. The "Imperx ICL‐B0620M" camera, on which the "Imagine Optics HASO™ 3 Fast" wavefront sensor is based, is used for this setup. The camera has a maximum frame rate of approximately 900 Hz at 224 × 224 pixel which corresponds to 1111 ms. Thus, the proposed method has a delay equal or less than one single frame.
