**4.2 Data fusion by machine learning**

Currently, what exists in the field of data fusion is a collection of routines/algorithms that can be linked and embedded for various applications. A very well-known

#### **Figure 4.**

*TerraSAR-X image of Venice, Italy: (left) a quick-look view of the image and (right) the corresponding classification map generated by CANDELA.*

**83**

**Figure 6.**

*(bottom-right) for the Venice image.*

**Figure 5.**

*CANDELA.*

*Artificial Intelligence Data Science Methodology for Earth Observation*

*Sentinel-1 image of Venice, Italy (after selecting the area that is covered by TerraSAR-X from the full Sentinel-1 image): (bottom-left) a quick-look view of the image and (bottom-right) the classification map generated by* 

*Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-left) and Sentinel-1* 

*DOI: http://dx.doi.org/10.5772/intechopen.86886*

*Artificial Intelligence Data Science Methodology for Earth Observation DOI: http://dx.doi.org/10.5772/intechopen.86886*

#### **Figure 5.**

*Advanced Analytics and Artificial Intelligence Applications*

general machine learning.

racy are given in **Figures 10** and **11**.

**4.2 Data fusion by machine learning**

focused on the everyday needs of a broad category of users [31]. A very popular satellite image data mining system is Tomnod from DigitalGlobe or Google Earth, which is targeting general user topics. Especially for EO, there are systems such as LandEX [32] which is a land cover management system, while GeoIRIS [33] is a system that allows the user to refine a given query by iteratively specifying a set of relevant and a set of nonrelevant images. A similar system is IKONA [34] which is using relevance feedback in order to analyze the content of very high-resolution EO images. Further, the knowledge-driven information mining (KIM) system [41] is an example of an active learning system providing semantic interpretation of image content. The KIM concept evolved into the TELEIOS prototype [36], complementing the scope of searching EO images with additional geo-information and in situ data. Finally, a cascaded active learning prototype [21] has been integrated into an operational EO system [20] to interpret the archives of TerraSAR-X images [37]. CANDELA is improving this cascaded active learning system by searching for dedicated algorithms for typical Earth observation images. Its implementation, test, and validation aim at automated knowledge extraction and image content interpretation. The targeted performance characteristics are verified for several typical use cases and tell us more about the potential of dedicated algorithms with respect to

**Figures 4–9** depict typical classification maps for TerraSAR-X and Sentinel-1 images together with their respective accuracy (e.g., precision/recall) for the cities of Venice, Italy, and Munich, Germany. Another example is the Dutch part of the Wadden Sea in the Netherlands. The results of the classification map and their accu-

Currently, what exists in the field of data fusion is a collection of routines/algorithms that can be linked and embedded for various applications. A very well-known

*TerraSAR-X image of Venice, Italy: (left) a quick-look view of the image and (right) the corresponding* 

**82**

**Figure 4.**

*classification map generated by CANDELA.*

*Sentinel-1 image of Venice, Italy (after selecting the area that is covered by TerraSAR-X from the full Sentinel-1 image): (bottom-left) a quick-look view of the image and (bottom-right) the classification map generated by CANDELA.*

*Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-left) and Sentinel-1 (bottom-right) for the Venice image.*

**Figure 7.**

*TerraSAR-X image of Munich, Germany: (left) a quick-look view of the image and (right) the classification map generated by CANDELA.*

#### **Figure 8.**

*Sentinel-1 image of Munich, Germany (after selecting the area that is also covered by TerraSAR-X): (bottomleft) a quick-look view of the image and (bottom-right) the classification map generated by CANDELA.*

open-source toolbox is Orfeo [38] which provides a large number of state-of-the-art algorithms to process SAR and multispectral images for different applications. Another one is Google Earth [31] that includes a large image database and an expandable number of algorithms that can be used for image processing.

In our case, we need to recognize different target area details in overlapping SAR and multispectral images. For doing this, we selected a number of cities from all over the world. The cities are Bucharest in Romania, Munich in Germany, Venice in Italy, and Washington in the USA. The selection criteria of these cities were the simultaneous availability of these cities covered by the two satellites and the variety of categories that can be found. A difficulty arises when trying to co-align these images, for

**85**

**Figure 10.**

**Figure 9.**

*(bottom-left) for the Munich image.*

*(right) the classification map generated by CANDELA.*

*Artificial Intelligence Data Science Methodology for Earth Observation*

example, images provided by TerraSAR-X and WorldView-2, because the original data have different pixel spacing. To solve this problem, we resampled the panchromatic

*Sentinel-2 quadrant image of an area of the Dutch Wadden Sea: (left) a quick-look view of the image and* 

*Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-right) and Sentinel-1* 

WorldView-2 image in order to co-align it with the TerraSAR-X image [27].

*DOI: http://dx.doi.org/10.5772/intechopen.86886*

*Artificial Intelligence Data Science Methodology for Earth Observation DOI: http://dx.doi.org/10.5772/intechopen.86886*

#### **Figure 9.**

*Advanced Analytics and Artificial Intelligence Applications*

open-source toolbox is Orfeo [38] which provides a large number of state-of-the-art algorithms to process SAR and multispectral images for different applications. Another one is Google Earth [31] that includes a large image database and an expand-

*Sentinel-1 image of Munich, Germany (after selecting the area that is also covered by TerraSAR-X): (bottomleft) a quick-look view of the image and (bottom-right) the classification map generated by CANDELA.*

*TerraSAR-X image of Munich, Germany: (left) a quick-look view of the image and (right) the classification* 

In our case, we need to recognize different target area details in overlapping SAR and multispectral images. For doing this, we selected a number of cities from all over the world. The cities are Bucharest in Romania, Munich in Germany, Venice in Italy, and Washington in the USA. The selection criteria of these cities were the simultaneous availability of these cities covered by the two satellites and the variety of categories that can be found. A difficulty arises when trying to co-align these images, for

able number of algorithms that can be used for image processing.

**84**

**Figure 8.**

**Figure 7.**

*map generated by CANDELA.*

*Classification accuracy (precision/recall) by comparison between TerraSAR-X (top-right) and Sentinel-1 (bottom-left) for the Munich image.*

#### **Figure 10.**

*Sentinel-2 quadrant image of an area of the Dutch Wadden Sea: (left) a quick-look view of the image and (right) the classification map generated by CANDELA.*

example, images provided by TerraSAR-X and WorldView-2, because the original data have different pixel spacing. To solve this problem, we resampled the panchromatic WorldView-2 image in order to co-align it with the TerraSAR-X image [27].

#### **Figure 11.**

*Classification accuracy (precision/recall) for the Sentinel-2 quadrant image covering an area of the Wadden Sea.*

In the case of Sentinel-1 and Sentinel-2, the images can be rectified and coaligned by publicly available toolbox routines [39]; this allowed us a straightforward image comparison.

While we are accustomed to image fusion as a radiometric combination of multispectral images, a comparably mature level of semantic fusion of SAR images has not been reached yet. In order to remedy the situation, we propose a semantic fusion concept for SAR images, where we combine the semantic image content of two data sets with different characteristics. By exploiting the specific imaging details and the retrievable semantic categories of the two image types, we obtained

#### **Figure 12.**

*A multi-sensor data set: multispectral image (top-left side), panchromatic image (top-right side), and TerraSAR-X image (bottom-center) for the city of Venice, Italy.*

**87**

different categories.

*images (bottom-right side).*

**Figure 13.**

TerraSAR-X image.

**Figures 15**–**17** apply to the city of Munich.

*Artificial Intelligence Data Science Methodology for Earth Observation*

semantically fused image classification maps that allow us to differentiate between

*Classification maps generated using the CANDELA platform for the city of Venice: multispectral image (topleft side), panchromatic image (top-right side), TerraSAR-X image (bottom-left side), and fusion of all three* 

together with their accuracy (e.g., precision/recall) for the city of Venice, while

annotations have a different semantic classification when compared to the

For a quantitative assessment, we compared the semantic annotation results with the given reference data set and computed precision/recall for each category and sensor. Analyzing the figures separately, we observed that the average of precision/recall obtained for fused sensor images is higher than the precision/ recall of individual sensor images. Unfortunately, there are also cases in which for corresponding image patches tiled from different sensor images, the WorldView-2

TerraSAR-X results or when a category is missing for one sensor. In our case, in the Venice image, the category "buoys" is only detected in the TerraSAR-X image, and not in the WorldView-2 image. This has a noticeable impact on the performance of the category "boats." Another example is the category "clouds" that appears in the case of the Munich image that is detected in the WorldView-2 image, but not in the

**Figures 12**–**14** present the classification maps for each sensor and the fused ones

*DOI: http://dx.doi.org/10.5772/intechopen.86886*
